Need some help with python-cybox to parse indicators

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Need some help with python-cybox to parse indicators

Trey Darley
Hi -

I¹m trying to use python-cybox to parse CybOX indicators. Considering the
fact that python-cybox weighs in at something like 143k LOC it seems like
there must be value there. Unfortunately there isn¹t any documentation and
example code is focused on generating indicators, not parsing them.

After having spent way too much time pulling my hair out with python-cybox
I¹m ready to give up and write my own lxml parser.

Is anyone else out there parsing CybOX with python-cybox? If so, would you
be so kind as to hit me with a cluestick?

Thanks in advance!

Cheers,
Trey
--
Trey Darley
Minister of Foreign Affairs
Brussels > Cupertino Seattle London Hong Kong Washington D.C.
Plano San Francisco Singapore Munich Paris
Splunk > because specialisation is for insects
Reply | Threaded
Open this post in threaded view
|

RE: Need some help with python-cybox to parse indicators

Shields, Wesley
We parse it inside CRITs. Sadly, Mike and I had to figure out the labyrinth that is python-cybox and python-stix somewhat on our own. Luckily we had Greg Back to help us when we got lost. I wish I could tell you more than that but if you do get stuck, Greg has been more than helpful with us.

What issues are you having specifically?

-- WXS

>-----Original Message-----
>From: [hidden email] [mailto:owner-cybox-
>[hidden email]] On Behalf Of Trey Darley
>Sent: Friday, October 25, 2013 8:33 AM
>To: cybox-discussion-list Cyber Observable Expression/CybOX Discussi
>Subject: Need some help with python-cybox to parse indicators
>
>Hi -
>
>I¹m trying to use python-cybox to parse CybOX indicators. Considering the
>fact that python-cybox weighs in at something like 143k LOC it seems like
>there must be value there. Unfortunately there isn¹t any documentation and
>example code is focused on generating indicators, not parsing them.
>
>After having spent way too much time pulling my hair out with python-cybox
>I¹m ready to give up and write my own lxml parser.
>
>Is anyone else out there parsing CybOX with python-cybox? If so, would you
>be so kind as to hit me with a cluestick?
>
>Thanks in advance!
>
>Cheers,
>Trey
>--
>Trey Darley
>Minister of Foreign Affairs
>Brussels > Cupertino Seattle London Hong Kong Washington D.C.
>Plano San Francisco Singapore Munich Paris
>Splunk > because specialisation is for insects
Reply | Threaded
Open this post in threaded view
|

RE: Need some help with python-cybox to parse indicators

Worrell, Bryan A.
In reply to this post by Trey Darley
Hi Everybody,

Thanks for the questions and feedback! We just added an example parsing script to the python-cybox repository, so you can check that out here: https://github.com/CybOXProject/python-cybox/blob/master/examples/parse_xml.py.

The python-cybox team has recognized that the current method of parsing XML source documents/streams could certainly be improved, so we have added an issue to our tracker to improve and simplify the process that abstracts away the calls to our underlying, machine-generated bindings (https://github.com/CybOXProject/python-cybox/issues/55). For the moment, developers leveraging python-cybox can refer to the parse_xml.py example document when parsing CybOX Observables documents.

The volume of code in python-cybox can certainly seem daunting at first--143k LOC is definitely a lot! However, much of that is code that was generated from the CybOX schemas using generateDS, which we try to shield developers from as much as possible. In fact, the machine-generated bindings represent about 92% of the codebase in python-cybox (132k LOC), so we're trying hard to keep developers from having to write code against it. Trying to write against machine-generated code isn't always the most fun experience :)

Thank you again for the feedback! Hopefully this will help yourself as well as other developers out there who may be scratching their heads with the same issue of parsing. We will continue to work at providing better documentation and examples for the community to leverage. As always, any and all feedback is welcomed so if something isn't working right or seems confusing, please feel free to bring it up on our GitHub issue tracker or send an email across the discussion list!

Thanks,
Bryan Worrell
The MITRE Corporation

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Trey Darley
Sent: Friday, October 25, 2013 8:33 AM
To: cybox-discussion-list Cyber Observable Expression/CybOX Discussi
Subject: Need some help with python-cybox to parse indicators

Hi -

I¹m trying to use python-cybox to parse CybOX indicators. Considering the
fact that python-cybox weighs in at something like 143k LOC it seems like
there must be value there. Unfortunately there isn¹t any documentation and
example code is focused on generating indicators, not parsing them.

After having spent way too much time pulling my hair out with python-cybox
I¹m ready to give up and write my own lxml parser.

Is anyone else out there parsing CybOX with python-cybox? If so, would you
be so kind as to hit me with a cluestick?

Thanks in advance!

Cheers,
Trey
--
Trey Darley
Minister of Foreign Affairs
Brussels > Cupertino Seattle London Hong Kong Washington D.C.
Plano San Francisco Singapore Munich Paris
Splunk > because specialisation is for insects
Reply | Threaded
Open this post in threaded view
|

RE: Need some help with python-cybox to parse indicators

Shields, Wesley
Thanks Bryan. I should clarify my earlier message...

When we first started integrating these libraries into our code it was a daunting task. Nothing was clear to me. The more I read from the standards and the code the more clear things became and once things clicked in my brain I found it not nearly as daunting as before. I wish I had a better description on how to get started but for us it was very much a "read the standard, read the examples, read the code and ask questions" way of doing things. The biggest boon we had to understanding things was the access to the developers of the code. I encourage anyone who has questions to speak up. It helps tremendously to have people like Ivan, Greg, Bryan and the others on this list.

-- WXS

>-----Original Message-----
>From: [hidden email] [mailto:owner-cybox-
>[hidden email]] On Behalf Of Worrell, Bryan A.
>Sent: Friday, October 25, 2013 10:59 AM
>To: cybox-discussion-list Cyber Observable Expression/CybOX Discussi
>Subject: RE: Need some help with python-cybox to parse indicators
>
>Hi Everybody,
>
>Thanks for the questions and feedback! We just added an example parsing
>script to the python-cybox repository, so you can check that out here:
>https://github.com/CybOXProject/python-
>cybox/blob/master/examples/parse_xml.py.
>
>The python-cybox team has recognized that the current method of parsing
>XML source documents/streams could certainly be improved, so we have
>added an issue to our tracker to improve and simplify the process that
>abstracts away the calls to our underlying, machine-generated bindings
>(https://github.com/CybOXProject/python-cybox/issues/55). For the
>moment, developers leveraging python-cybox can refer to the parse_xml.py
>example document when parsing CybOX Observables documents.
>
>The volume of code in python-cybox can certainly seem daunting at first--143k
>LOC is definitely a lot! However, much of that is code that was generated from
>the CybOX schemas using generateDS, which we try to shield developers from
>as much as possible. In fact, the machine-generated bindings represent about
>92% of the codebase in python-cybox (132k LOC), so we're trying hard to keep
>developers from having to write code against it. Trying to write against
>machine-generated code isn't always the most fun experience :)
>
>Thank you again for the feedback! Hopefully this will help yourself as well as
>other developers out there who may be scratching their heads with the same
>issue of parsing. We will continue to work at providing better documentation
>and examples for the community to leverage. As always, any and all feedback
>is welcomed so if something isn't working right or seems confusing, please
>feel free to bring it up on our GitHub issue tracker or send an email across the
>discussion list!
>
>Thanks,
>Bryan Worrell
>The MITRE Corporation
>
>-----Original Message-----
>From: [hidden email] [mailto:owner-cybox-
>[hidden email]] On Behalf Of Trey Darley
>Sent: Friday, October 25, 2013 8:33 AM
>To: cybox-discussion-list Cyber Observable Expression/CybOX Discussi
>Subject: Need some help with python-cybox to parse indicators
>
>Hi -
>
>I¹m trying to use python-cybox to parse CybOX indicators. Considering the
>fact that python-cybox weighs in at something like 143k LOC it seems like
>there must be value there. Unfortunately there isn¹t any documentation and
>example code is focused on generating indicators, not parsing them.
>
>After having spent way too much time pulling my hair out with python-cybox
>I¹m ready to give up and write my own lxml parser.
>
>Is anyone else out there parsing CybOX with python-cybox? If so, would you
>be so kind as to hit me with a cluestick?
>
>Thanks in advance!
>
>Cheers,
>Trey
>--
>Trey Darley
>Minister of Foreign Affairs
>Brussels > Cupertino Seattle London Hong Kong Washington D.C.
>Plano San Francisco Singapore Munich Paris
>Splunk > because specialisation is for insects