Quantcast

CEE Python / AMQP Test Code

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

CEE Python / AMQP Test Code

Burnes, James - NRCS, Fort Collins, CO
All,
 
In my spare time I wrote a CEEEvent log record class for Python that works within the Python logging library.  It operates very similarly to how a similar log record would work within log4net and log4j.
 
I also wrote an AMQP appender for Python logging that you can point to an AMQP exchange broker (like RabbitMQ) with a selected topic and your CEE events will automagically be rendered using the CEE JSON syntax and published to the broker.  Whoever is subscribed will receive the JSON CEE events.
 
 
log.ERROR(CEEEvent(id='cee_generic',p_prod_id='myapp',action='allocate',status='failed',text='filesystem /mnt/xyz inodes exhausted'))
 
The CEEEvent is automatically timestamped at creation.
 
 
You can also create the event and modify it between logging operations like this.  In this example I included preselected fields from a group.
 
(Notice that you can select from CEE field groups (tuples).  I arbitrarily organized them).
 
myev = CEEEvent(id='cee_generic',p_prod_id='VFS',action='allocate',status='failed',use_fields=disk_base + file_base + file_path)
 
for fsystem in filesystems:
        if fsystem.inodes < 1:
                myev.field.text = 'filesystem %s, inodes exhausted' % (fsystem.name)
                myev.field.inode_devid = fsystem.dev
                myev.field.inode_id = 0
                myev.field.file_name = file_name
                myev.field.file_path = file_path
                myev.mark()             # timestamp it
                log.ERROR(myev)
 
 
You can also do the usual things in the Python shell like just type the name of the event.  When you do that it shows the field dictionary and values.  If you do a "print myev" you'll get the JSON serialized form the event (same as gets sent to the log appender).
 
It's not perfect, but I'd say it's at 0.9.  Only the core fields have the proper types assigned to them, so that needs to be updated. I haven't implemented the XML renderer in the class, but that would be easy enough.
 
Otherwise, you just create a logger, attach an AMQP appender and select the correct topic exchange to send it to.
 
I verified the design with:
        - Ubuntu Linux 11.04
        - Python 2.7
        - RabbitMQ 2.6.1.1
        - Logstash 1.0.17
        - ElasticSearch 0.17.8
 
With the above setup I'm getting about 12k messages per second routed to the elasticsearch backend (of a couple hundred bytes each).
 
If anyone wants to help out, or do a little customization on the currently existing log4j/log4net AMQP appenders that would be great.
 
I haven't pushed the code to a repository yet, but if someone wants to host it that would be fine.  I haven't applied a license to it yet.  I think it's simple enough that I could just put a zlib license on it and be done.
 
Jim Burnes
Application Security Engineer
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CEE Python / AMQP Test Code

Anton Chuvakin
Thanks for developing this!!

Can it make both XML or JSON/text CEE events? I think JSON alone would be pretty useful and can easily be converted to XML, if needed.

Given that you are planning an open license for it, maybe syslog-ng/rsyslog folks can make some use of it?

On Thu, Oct 27, 2011 at 4:06 PM, Burnes, James - NRCS, Fort Collins, CO <[hidden email]> wrote:
All,
 
In my spare time I wrote a CEEEvent log record class for Python that works within the Python logging library.  It operates very similarly to how a similar log record would work within log4net and log4j.
 
I also wrote an AMQP appender for Python logging that you can point to an AMQP exchange broker (like RabbitMQ) with a selected topic and your CEE events will automagically be rendered using the CEE JSON syntax and published to the broker.  Whoever is subscribed will receive the JSON CEE events.
 
 
log.ERROR(CEEEvent(id='cee_generic',p_prod_id='myapp',action='allocate',status='failed',text='filesystem /mnt/xyz inodes exhausted'))
 
The CEEEvent is automatically timestamped at creation.
 
 
You can also create the event and modify it between logging operations like this.  In this example I included preselected fields from a group.
 
(Notice that you can select from CEE field groups (tuples).  I arbitrarily organized them).
 
myev = CEEEvent(id='cee_generic',p_prod_id='VFS',action='allocate',status='failed',use_fields=disk_base + file_base + file_path)
 
for fsystem in filesystems:
        if fsystem.inodes < 1:
                myev.field.text = 'filesystem %s, inodes exhausted' % (fsystem.name)
                myev.field.inode_devid = fsystem.dev
                myev.field.inode_id = 0
                myev.field.file_name = file_name
                myev.field.file_path = file_path
                myev.mark()             # timestamp it
                log.ERROR(myev)
 
 
You can also do the usual things in the Python shell like just type the name of the event.  When you do that it shows the field dictionary and values.  If you do a "print myev" you'll get the JSON serialized form the event (same as gets sent to the log appender).
 
It's not perfect, but I'd say it's at 0.9.  Only the core fields have the proper types assigned to them, so that needs to be updated. I haven't implemented the XML renderer in the class, but that would be easy enough.
 
Otherwise, you just create a logger, attach an AMQP appender and select the correct topic exchange to send it to.
 
I verified the design with:
        - Ubuntu Linux 11.04
        - Python 2.7
        - RabbitMQ 2.6.1.1
        - Logstash 1.0.17
        - ElasticSearch 0.17.8
 
With the above setup I'm getting about 12k messages per second routed to the elasticsearch backend (of a couple hundred bytes each).
 
If anyone wants to help out, or do a little customization on the currently existing log4j/log4net AMQP appenders that would be great.
 
I haven't pushed the code to a repository yet, but if someone wants to host it that would be fine.  I haven't applied a license to it yet.  I think it's simple enough that I could just put a zlib license on it and be done.
 
Jim Burnes
Application Security Engineer
 
 



--
Dr. Anton Chuvakin
Site: http://www.chuvakin.org
Twitter: @anton_chuvakin
Work: http://www.linkedin.com/in/chuvakin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CEE Python / AMQP Test Code

Burnes, James - NRCS, Fort Collins, CO

Anton,

 

I’ll answer your questions inline:

 

Ø  Thanks for developing this!!

 

No problem.  Necessity is the mother of invention and all that.  Most of the time taken was just organizing the CEE events into groups and typing them  in.  I guess a little design time deciding how to implement mixing-in the attribute groups.   In Python the attribute groups are just tuples that you append together and pass to the class during creation.

 

Ø  Can it make both XML or JSON/text CEE events? I think JSON alone would be pretty useful and can easily be converted to XML, if needed.

 

Yes, the CEEEvent class is designed to be syntax agnostic, though it emits JSON syntax by default.  It has a stub XML serializer method that just emits “</not implemented>” J   You can simply fill in the straightforward Python code to output the event in XML format.  Then you tell CEEEvent that you want to use the XML serializer rather than the JSON serializer.   That way, the Python appender (listener) can be told at runtime to switch formats and it will (assuming the downstream listeners don’t care).  In theory you could have one appender emitting JSON and another emitting XML but sent to a different destination.

 

Originally I was going to implement the XML serializer first because I was going to send the events via XMPP messaging.  Injecting it into an XMPP message as an extra stanza just seemed to make a lot of sense.   When I realized that AMQP avoided a lot of XMPP issues I’d have to correct for later, added to the fact that modern cloud-based logging solutions like logstash, loggly and ElasticSearch actually made excellent use of JSON during parsing and object storage I changed my approach.

 

Ø  Given that you are planning an open license for it, maybe syslog-ng/rsyslog folks can make some use of it?

 

Hey, if it’s ZLIB license it’s downstream compatible with nearly every existing license including syslog*.  Mostly the Python code would be useful to those projects in seeing an example of what is contained in a CEE event.  That way if they consume or generate them, they’ll have a reference.

 

I’ve done the Python work.   The only thing really required for other generators is to create Java/C/ObjectiveC/C#/VB/JavaScript/PERL etc versions of CEEEvent.  (AMQP appenders already exist for log4j/log4net)

 

Speaking of CEE standard attributes:

 

1.      It initially took me a lot of noodling around in relatively abstruse documentation to get down to the critical attribute/syntax documentation.  I realize this is the nature of standards-track documentation, but the web site doesn’t have to be structured the same as a standards track RFC.

2.      What is the standards body that will eventually publish CEE?

3.      I think an attribute group / action type for application-layer audit logging would be good.   There are a few attributes/actions that get close, but I’ll expand more on that later.

 

Regards,

 

Jim Burnes
Application Security Engineer

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CEE Python / AMQP Test Code

heinbockel
In reply to this post by Burnes, James - NRCS, Fort Collins, CO
James,

I would love to take a look at it.

I have been investigating developing a libcee C library to do all of the CEE
encoding/decoding, which sounds a lot like your python library.

William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Burnes, James - NRCS, Fort Collins, CO
>[mailto:[hidden email]]
>Sent: Thursday, 27 October, 2011 19:06
>To: cee-discussion-list CEE-Related Discussion
>Subject: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
>
>All,
>
>In my spare time I wrote a CEEEvent log record class for Python that works
>within the Python logging library.  It operates very similarly to how a
>similar log record would work within log4net and log4j.
>
>I also wrote an AMQP appender for Python logging that you can point to an
>AMQP exchange broker (like RabbitMQ) with a selected topic and your CEE
>events will automagically be rendered using the CEE JSON syntax and
>published to the broker.  Whoever is subscribed will receive the JSON CEE
>events.
>
>
>log.ERROR(CEEEvent(id='cee_generic',p_prod_id='myapp',action='allocate',sta
>tus='failed',text='filesystem /mnt/xyz inodes exhausted'))
>
>The CEEEvent is automatically timestamped at creation.
>
>
>You can also create the event and modify it between logging operations like
>this.  In this example I included preselected fields from a group.
>
>(Notice that you can select from CEE field groups (tuples).  I arbitrarily
>organized them).
>
>myev =
>CEEEvent(id='cee_generic',p_prod_id='VFS',action='allocate',status='failed'
>,use_fields=disk_base + file_base + file_path)
>
>for fsystem in filesystems:
>        if fsystem.inodes < 1:
>                myev.field.text = 'filesystem %s, inodes exhausted' %
>(fsystem.name)
>                myev.field.inode_devid = fsystem.dev
>                myev.field.inode_id = 0
>                myev.field.file_name = file_name
>                myev.field.file_path = file_path
>                myev.mark()             # timestamp it
>                log.ERROR(myev)
>
>
>You can also do the usual things in the Python shell like just type the
>name of the event.  When you do that it shows the field dictionary and
>values.  If you do a "print myev" you'll get the JSON serialized form the
>event (same as gets sent to the log appender).
>
>It's not perfect, but I'd say it's at 0.9.  Only the core fields have the
>proper types assigned to them, so that needs to be updated. I haven't
>implemented the XML renderer in the class, but that would be easy enough.
>
>Otherwise, you just create a logger, attach an AMQP appender and select the
>correct topic exchange to send it to.
>
>I verified the design with:
>        - Ubuntu Linux 11.04
>        - Python 2.7
>        - RabbitMQ 2.6.1.1
>        - Logstash 1.0.17
>        - ElasticSearch 0.17.8
>
>With the above setup I'm getting about 12k messages per second routed to
>the elasticsearch backend (of a couple hundred bytes each).
>
>If anyone wants to help out, or do a little customization on the currently
>existing log4j/log4net AMQP appenders that would be great.
>
>I haven't pushed the code to a repository yet, but if someone wants to host
>it that would be fine.  I haven't applied a license to it yet.  I think
>it's simple enough that I could just put a zlib license on it and be done.
>
>Jim Burnes
>Application Security Engineer
>
>

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CEE Python / AMQP Test Code

Rainer Gerhards
In reply to this post by Burnes, James - NRCS, Fort Collins, CO
> -----Original Message-----
> From: Heinbockel, Bill [mailto:[hidden email]]
> Sent: Monday, November 07, 2011 4:48 PM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
>
> James,
>
> I would love to take a look at it.
>
> I have been investigating developing a libcee C library to do all of the
CEE
> encoding/decoding, which sounds a lot like your python library.

It would probably be useful to consider a kind of (semi-) standard API for
such libraries. Also, I think we need to refine the profile requirements. I
have not yet updated libee, because it already covers most of the new specs,
but what really is missing is to ensure that the emitted CEE is correct
beyond pure syntax...

If you like, I can check what needs to be done to make libee emit the exact
correct format.

rainer

>
> William Heinbockel
> The MITRE Corporation
>
>
> >-----Original Message-----
> >From: Burnes, James - NRCS, Fort Collins, CO
> >[mailto:[hidden email]]
> >Sent: Thursday, 27 October, 2011 19:06
> >To: cee-discussion-list CEE-Related Discussion
> >Subject: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
> >
> >All,
> >
> >In my spare time I wrote a CEEEvent log record class for Python that works
> >within the Python logging library.  It operates very similarly to how a
> >similar log record would work within log4net and log4j.
> >
> >I also wrote an AMQP appender for Python logging that you can point to an
> >AMQP exchange broker (like RabbitMQ) with a selected topic and your CEE
> >events will automagically be rendered using the CEE JSON syntax and
> >published to the broker.  Whoever is subscribed will receive the JSON CEE
> >events.
> >
> >
> >log.ERROR(CEEEvent(id='cee_generic',p_prod_id='myapp',action='allocate',
> sta
> >tus='failed',text='filesystem /mnt/xyz inodes exhausted'))
> >
> >The CEEEvent is automatically timestamped at creation.
> >
> >
> >You can also create the event and modify it between logging operations
like

> >this.  In this example I included preselected fields from a group.
> >
> >(Notice that you can select from CEE field groups (tuples).  I arbitrarily
> >organized them).
> >
> >myev =
>
>CEEEvent(id='cee_generic',p_prod_id='VFS',action='allocate',status='failed'
> >,use_fields=disk_base + file_base + file_path)
> >
> >for fsystem in filesystems:
> >        if fsystem.inodes < 1:
> >                myev.field.text = 'filesystem %s, inodes exhausted' %
> >(fsystem.name)
> >                myev.field.inode_devid = fsystem.dev
> >                myev.field.inode_id = 0
> >                myev.field.file_name = file_name
> >                myev.field.file_path = file_path
> >                myev.mark()             # timestamp it
> >                log.ERROR(myev)
> >
> >
> >You can also do the usual things in the Python shell like just type the
> >name of the event.  When you do that it shows the field dictionary and
> >values.  If you do a "print myev" you'll get the JSON serialized form the
> >event (same as gets sent to the log appender).
> >
> >It's not perfect, but I'd say it's at 0.9.  Only the core fields have the
> >proper types assigned to them, so that needs to be updated. I haven't
> >implemented the XML renderer in the class, but that would be easy
> enough.
> >
> >Otherwise, you just create a logger, attach an AMQP appender and select
> the
> >correct topic exchange to send it to.
> >
> >I verified the design with:
> >        - Ubuntu Linux 11.04
> >        - Python 2.7
> >        - RabbitMQ 2.6.1.1
> >        - Logstash 1.0.17
> >        - ElasticSearch 0.17.8
> >
> >With the above setup I'm getting about 12k messages per second routed to
> >the elasticsearch backend (of a couple hundred bytes each).
> >
> >If anyone wants to help out, or do a little customization on the currently
> >existing log4j/log4net AMQP appenders that would be great.
> >
> >I haven't pushed the code to a repository yet, but if someone wants to
host
> >it that would be fine.  I haven't applied a license to it yet.  I think
> >it's simple enough that I could just put a zlib license on it and be done.
> >
> >Jim Burnes
> >Application Security Engineer
> >
> >
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CEE Python / AMQP Test Code

Burnes, James - NRCS, Fort Collins, CO
I've taken the approach that:

1. The event is an object with attribute fields which can be set at construction time or after the fact.
2. The default values for the attributes are "reasonable", timestamped with UTC/Zulu time, host set.
3. The object can be constructed and emitted in a single call for the simplest possible case.
4. The event object can be re-used in a loop without re-construction.
4. The rendering syntax (JSON/XML/etc) can be specified at construction time and changed on-the-fly

Most importantly, there is no *overt* API per-se except possibly in the constructor and attribute access.   The event is constructed as an instance of a class.  All the programmer has to do is create a CEE Event and log.WARN(myEvent).   Everything else is abstracted away inside the logger. This is standard logging and configuration pattern for log4j, log4net and python logging.  

In that pattern, an "appender" (which is really an observer/listener) is configured (either through config files or at run-time), to attach to your log stream, select an output format, and send it out via some pre-determined transport like Syslog, log files or, as in my case AMQP.  The really great thing is the way this de-couples the event from how it's routed, formatted, filtered and transported -- all before it leaves the event generator.  Also it delays event rendering until the last possible moment.

This allows you to do certain things easily.  You can construct a certain listener/appender to only forward an event over an encrypted connection if its event.PII field is True.  (BTW: That might be a nice attribute to add, hint).   The standard way for an event to then be rendered in the log4x pattern is to call the event object's default serializer, unless overridden by an explicit formatter in the appender.

It also allows you to setup logging in your app and the operations staff can add transport/format/filter-specific appenders in config files without having to restart the app.

In my case I created CEEEvent so that it's default serializer (called when you say 'print myevent' in Python), calls the JSON renderer.  That can be changed easily at construct time or afterwards.

For example:

... (log stream setup code) ....

nuke_time = 10
myEvent = CEEEvent(<attributes to use during construction>)
myEvent.useJSON(assertTypes=True)  # render as JSON, string types are explicit
(or)
myEvent.useXML()   # string types will not be asserted by default
myEvent.id = 'weyland_utani-emergency'
for i in (10,9,8,7,6,5,4,3,2,1):
   myEvent.text = 'You have %d minutes to reach minimum safe distance.' % (nuke_time)
   log.WARN(myEvent)
   sleep(1)
coolant.shutdown()


Or you could send a single warning explicitly like this:

log.WARN(CEEEvent('You have 10 minutes to reach minimum safe distance.',id='weyland_utani-emergency',render='xml'))

In any case, specifying an actual API is probably not appropriate at the standards level as it's implementation dependent.

** Bill, do you want to host it at Mitre?   I can also drop it at BitBucket.

Jim



 
Jim Burnes
Application Security Engineer
USDA/NRCS/ITC/Fort Collins



-----Original Message-----
From: Rainer Gerhards [mailto:[hidden email]]
Sent: Monday, November 07, 2011 8:58 AM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code

> -----Original Message-----
> From: Heinbockel, Bill [mailto:[hidden email]]
> Sent: Monday, November 07, 2011 4:48 PM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
>
> James,
>
> I would love to take a look at it.
>
> I have been investigating developing a libcee C library to do all of the
CEE
> encoding/decoding, which sounds a lot like your python library.

It would probably be useful to consider a kind of (semi-) standard API for
such libraries. Also, I think we need to refine the profile requirements. I
have not yet updated libee, because it already covers most of the new specs,
but what really is missing is to ensure that the emitted CEE is correct
beyond pure syntax...

If you like, I can check what needs to be done to make libee emit the exact
correct format.

rainer

>
> William Heinbockel
> The MITRE Corporation
>
>
> >-----Original Message-----
> >From: Burnes, James - NRCS, Fort Collins, CO
> >[mailto:[hidden email]]
> >Sent: Thursday, 27 October, 2011 19:06
> >To: cee-discussion-list CEE-Related Discussion
> >Subject: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
> >
> >All,
> >
> >In my spare time I wrote a CEEEvent log record class for Python that works
> >within the Python logging library.  It operates very similarly to how a
> >similar log record would work within log4net and log4j.
> >
> >I also wrote an AMQP appender for Python logging that you can point to an
> >AMQP exchange broker (like RabbitMQ) with a selected topic and your CEE
> >events will automagically be rendered using the CEE JSON syntax and
> >published to the broker.  Whoever is subscribed will receive the JSON CEE
> >events.
> >
> >
> >log.ERROR(CEEEvent(id='cee_generic',p_prod_id='myapp',action='allocate',
> sta
> >tus='failed',text='filesystem /mnt/xyz inodes exhausted'))
> >
> >The CEEEvent is automatically timestamped at creation.
> >
> >
> >You can also create the event and modify it between logging operations
like

> >this.  In this example I included preselected fields from a group.
> >
> >(Notice that you can select from CEE field groups (tuples).  I arbitrarily
> >organized them).
> >
> >myev =
>
>CEEEvent(id='cee_generic',p_prod_id='VFS',action='allocate',status='failed'
> >,use_fields=disk_base + file_base + file_path)
> >
> >for fsystem in filesystems:
> >        if fsystem.inodes < 1:
> >                myev.field.text = 'filesystem %s, inodes exhausted' %
> >(fsystem.name)
> >                myev.field.inode_devid = fsystem.dev
> >                myev.field.inode_id = 0
> >                myev.field.file_name = file_name
> >                myev.field.file_path = file_path
> >                myev.mark()             # timestamp it
> >                log.ERROR(myev)
> >
> >
> >You can also do the usual things in the Python shell like just type the
> >name of the event.  When you do that it shows the field dictionary and
> >values.  If you do a "print myev" you'll get the JSON serialized form the
> >event (same as gets sent to the log appender).
> >
> >It's not perfect, but I'd say it's at 0.9.  Only the core fields have the
> >proper types assigned to them, so that needs to be updated. I haven't
> >implemented the XML renderer in the class, but that would be easy
> enough.
> >
> >Otherwise, you just create a logger, attach an AMQP appender and select
> the
> >correct topic exchange to send it to.
> >
> >I verified the design with:
> >        - Ubuntu Linux 11.04
> >        - Python 2.7
> >        - RabbitMQ 2.6.1.1
> >        - Logstash 1.0.17
> >        - ElasticSearch 0.17.8
> >
> >With the above setup I'm getting about 12k messages per second routed to
> >the elasticsearch backend (of a couple hundred bytes each).
> >
> >If anyone wants to help out, or do a little customization on the currently
> >existing log4j/log4net AMQP appenders that would be great.
> >
> >I haven't pushed the code to a repository yet, but if someone wants to
host
> >it that would be fine.  I haven't applied a license to it yet.  I think
> >it's simple enough that I could just put a zlib license on it and be done.
> >
> >Jim Burnes
> >Application Security Engineer
> >
> >
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CEE Python / AMQP Test Code

Burnes, James - NRCS, Fort Collins, CO
In reply to this post by heinbockel
I've uploaded the CEE Event and AMQP Appender in Python to a repository at google code.  

Here's the link: http://code.google.com/p/cee-appender/downloads/list

Just download the tar.gz file, or feel free to clone the source code if you wish.

I've given it the once over.   The code, comments and documentation are all of the "gitt'r done" variety.  Not too ugly, didn't spend too much time prettying up but tried to create a minimal elegance. :-)

Have at it.   Hack away.  Etc.   There are a couple micro-snippets of code that I found on the web which I imagine are so small as to be public domain.   I've zlib open sourced the thing.

Test away and I'll do what I can to make changes, improvements etc.  

If you follow the README and INSTALL, you'll have a full CEE/AMQP Python logging test rig, from the generator to the aggregator(s) to a cloud/cluster-capable logging backend.  Play around.

Enjoy.

Jim Burnes
Application Security Engineer

-----Original Message-----
From: Heinbockel, Bill [mailto:[hidden email]]
Sent: Monday, November 07, 2011 8:48 AM
To: Burnes, James - NRCS, Fort Collins, CO; cee-discussion-list CEE-Related Discussion
Subject: RE: CEE Python / AMQP Test Code

James,

I would love to take a look at it.

I have been investigating developing a libcee C library to do all of the CEE
encoding/decoding, which sounds a lot like your python library.

William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Burnes, James - NRCS, Fort Collins, CO
>[mailto:[hidden email]]
>Sent: Thursday, 27 October, 2011 19:06
>To: cee-discussion-list CEE-Related Discussion
>Subject: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
>
>All,
>
>In my spare time I wrote a CEEEvent log record class for Python that works
>within the Python logging library.  It operates very similarly to how a
>similar log record would work within log4net and log4j.
>
>I also wrote an AMQP appender for Python logging that you can point to an
>AMQP exchange broker (like RabbitMQ) with a selected topic and your CEE
>events will automagically be rendered using the CEE JSON syntax and
>published to the broker.  Whoever is subscribed will receive the JSON CEE
>events.
>
>
>log.ERROR(CEEEvent(id='cee_generic',p_prod_id='myapp',action='allocate',sta
>tus='failed',text='filesystem /mnt/xyz inodes exhausted'))
>
>The CEEEvent is automatically timestamped at creation.
>
>
>You can also create the event and modify it between logging operations like
>this.  In this example I included preselected fields from a group.
>
>(Notice that you can select from CEE field groups (tuples).  I arbitrarily
>organized them).
>
>myev =
>CEEEvent(id='cee_generic',p_prod_id='VFS',action='allocate',status='failed'
>,use_fields=disk_base + file_base + file_path)
>
>for fsystem in filesystems:
>        if fsystem.inodes < 1:
>                myev.field.text = 'filesystem %s, inodes exhausted' %
>(fsystem.name)
>                myev.field.inode_devid = fsystem.dev
>                myev.field.inode_id = 0
>                myev.field.file_name = file_name
>                myev.field.file_path = file_path
>                myev.mark()             # timestamp it
>                log.ERROR(myev)
>
>
>You can also do the usual things in the Python shell like just type the
>name of the event.  When you do that it shows the field dictionary and
>values.  If you do a "print myev" you'll get the JSON serialized form the
>event (same as gets sent to the log appender).
>
>It's not perfect, but I'd say it's at 0.9.  Only the core fields have the
>proper types assigned to them, so that needs to be updated. I haven't
>implemented the XML renderer in the class, but that would be easy enough.
>
>Otherwise, you just create a logger, attach an AMQP appender and select the
>correct topic exchange to send it to.
>
>I verified the design with:
>        - Ubuntu Linux 11.04
>        - Python 2.7
>        - RabbitMQ 2.6.1.1
>        - Logstash 1.0.17
>        - ElasticSearch 0.17.8
>
>With the above setup I'm getting about 12k messages per second routed to
>the elasticsearch backend (of a couple hundred bytes each).
>
>If anyone wants to help out, or do a little customization on the currently
>existing log4j/log4net AMQP appenders that would be great.
>
>I haven't pushed the code to a repository yet, but if someone wants to host
>it that would be fine.  I haven't applied a license to it yet.  I think
>it's simple enough that I could just put a zlib license on it and be done.
>
>Jim Burnes
>Application Security Engineer
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CEE Python / AMQP Test Code

Rainer Gerhards
In reply to this post by Burnes, James - NRCS, Fort Collins, CO
> -----Original Message-----
> From: Burnes, James - NRCS, Fort Collins, CO
> [mailto:[hidden email]]
> Sent: Monday, November 07, 2011 8:27 PM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
>
> I've taken the approach that:
>
> 1. The event is an object with attribute fields which can be set at
> construction time or after the fact.
> 2. The default values for the attributes are "reasonable", timestamped
> with UTC/Zulu time, host set.
> 3. The object can be constructed and emitted in a single call for the
> simplest possible case.
> 4. The event object can be re-used in a loop without re-construction.
> 4. The rendering syntax (JSON/XML/etc) can be specified at construction
> time and changed on-the-fly
>
> Most importantly, there is no *overt* API per-se except possibly in the
> constructor and attribute access.   The event is constructed as an
> instance of a class.  All the programmer has to do is create a CEE
> Event and log.WARN(myEvent).   Everything else is abstracted away
> inside the logger. This is standard logging and configuration pattern
> for log4j, log4net and python logging.


Indeed, you are right: it obviously makes most sense to create
implementations adapted to the individual frameworks in question.

>
> In that pattern, an "appender" (which is really an observer/listener)
> is configured (either through config files or at run-time), to attach
> to your log stream, select an output format, and send it out via some
> pre-determined transport like Syslog, log files or, as in my case AMQP.
> The really great thing is the way this de-couples the event from how
> it's routed, formatted, filtered and transported -- all before it
> leaves the event generator.  Also it delays event rendering until the
> last possible moment.
>
> This allows you to do certain things easily.  You can construct a
> certain listener/appender to only forward an event over an encrypted
> connection if its event.PII field is True.  (BTW: That might be a nice
> attribute to add, hint).   The standard way for an event to then be
> rendered in the log4x pattern is to call the event object's default
> serializer, unless overridden by an explicit formatter in the appender.
>
> It also allows you to setup logging in your app and the operations
> staff can add transport/format/filter-specific appenders in config
> files without having to restart the app.
>
> In my case I created CEEEvent so that it's default serializer (called
> when you say 'print myevent' in Python), calls the JSON renderer.  That
> can be changed easily at construct time or afterwards.
>
> For example:
>
> ... (log stream setup code) ....
>
> nuke_time = 10
> myEvent = CEEEvent(<attributes to use during construction>)
> myEvent.useJSON(assertTypes=True)  # render as JSON, string types are
> explicit
> (or)
> myEvent.useXML()   # string types will not be asserted by default
> myEvent.id = 'weyland_utani-emergency'
> for i in (10,9,8,7,6,5,4,3,2,1):
>    myEvent.text = 'You have %d minutes to reach minimum safe distance.'
> % (nuke_time)
>    log.WARN(myEvent)
>    sleep(1)
> coolant.shutdown()
>
>
> Or you could send a single warning explicitly like this:
>
> log.WARN(CEEEvent('You have 10 minutes to reach minimum safe
> distance.',id='weyland_utani-emergency',render='xml'))
>
> In any case, specifying an actual API is probably not appropriate at
> the standards level as it's implementation dependent.

Yeah, that's why I wrote (semi-)standard. But, again, that doesn't make much
sense as you say ;)

Rainer

>
> ** Bill, do you want to host it at Mitre?   I can also drop it at
> BitBucket.
>
> Jim
>
>
>
>
> Jim Burnes
> Application Security Engineer
> USDA/NRCS/ITC/Fort Collins
>
>
>
> -----Original Message-----
> From: Rainer Gerhards [mailto:[hidden email]]
> Sent: Monday, November 07, 2011 8:58 AM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
>
> > -----Original Message-----
> > From: Heinbockel, Bill [mailto:[hidden email]]
> > Sent: Monday, November 07, 2011 4:48 PM
> > To: [hidden email]
> > Subject: Re: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
> >
> > James,
> >
> > I would love to take a look at it.
> >
> > I have been investigating developing a libcee C library to do all of
> the
> CEE
> > encoding/decoding, which sounds a lot like your python library.
>
> It would probably be useful to consider a kind of (semi-) standard API
> for
> such libraries. Also, I think we need to refine the profile
> requirements. I
> have not yet updated libee, because it already covers most of the new
> specs,
> but what really is missing is to ensure that the emitted CEE is correct
> beyond pure syntax...
>
> If you like, I can check what needs to be done to make libee emit the
> exact
> correct format.
>
> rainer
> >
> > William Heinbockel
> > The MITRE Corporation
> >
> >
> > >-----Original Message-----
> > >From: Burnes, James - NRCS, Fort Collins, CO
> > >[mailto:[hidden email]]
> > >Sent: Thursday, 27 October, 2011 19:06
> > >To: cee-discussion-list CEE-Related Discussion
> > >Subject: [CEE-DISCUSSION-LIST] CEE Python / AMQP Test Code
> > >
> > >All,
> > >
> > >In my spare time I wrote a CEEEvent log record class for Python that
> works
> > >within the Python logging library.  It operates very similarly to
> how a
> > >similar log record would work within log4net and log4j.
> > >
> > >I also wrote an AMQP appender for Python logging that you can point
> to an
> > >AMQP exchange broker (like RabbitMQ) with a selected topic and your
> CEE
> > >events will automagically be rendered using the CEE JSON syntax and
> > >published to the broker.  Whoever is subscribed will receive the
> JSON CEE
> > >events.
> > >
> > >
> >
> >log.ERROR(CEEEvent(id='cee_generic',p_prod_id='myapp',action='allocate
> ',
> > sta
> > >tus='failed',text='filesystem /mnt/xyz inodes exhausted'))
> > >
> > >The CEEEvent is automatically timestamped at creation.
> > >
> > >
> > >You can also create the event and modify it between logging
> operations
> like
> > >this.  In this example I included preselected fields from a group.
> > >
> > >(Notice that you can select from CEE field groups (tuples).  I
> arbitrarily
> > >organized them).
> > >
> > >myev =
> >
> >CEEEvent(id='cee_generic',p_prod_id='VFS',action='allocate',status='fa
> iled'
> > >,use_fields=disk_base + file_base + file_path)
> > >
> > >for fsystem in filesystems:
> > >        if fsystem.inodes < 1:
> > >                myev.field.text = 'filesystem %s, inodes exhausted'
> %
> > >(fsystem.name)
> > >                myev.field.inode_devid = fsystem.dev
> > >                myev.field.inode_id = 0
> > >                myev.field.file_name = file_name
> > >                myev.field.file_path = file_path
> > >                myev.mark()             # timestamp it
> > >                log.ERROR(myev)
> > >
> > >
> > >You can also do the usual things in the Python shell like just type
> the
> > >name of the event.  When you do that it shows the field dictionary
> and
> > >values.  If you do a "print myev" you'll get the JSON serialized
> form the
> > >event (same as gets sent to the log appender).
> > >
> > >It's not perfect, but I'd say it's at 0.9.  Only the core fields
> have the
> > >proper types assigned to them, so that needs to be updated. I
> haven't
> > >implemented the XML renderer in the class, but that would be easy
> > enough.
> > >
> > >Otherwise, you just create a logger, attach an AMQP appender and
> select
> > the
> > >correct topic exchange to send it to.
> > >
> > >I verified the design with:
> > >        - Ubuntu Linux 11.04
> > >        - Python 2.7
> > >        - RabbitMQ 2.6.1.1
> > >        - Logstash 1.0.17
> > >        - ElasticSearch 0.17.8
> > >
> > >With the above setup I'm getting about 12k messages per second
> routed to
> > >the elasticsearch backend (of a couple hundred bytes each).
> > >
> > >If anyone wants to help out, or do a little customization on the
> currently
> > >existing log4j/log4net AMQP appenders that would be great.
> > >
> > >I haven't pushed the code to a repository yet, but if someone wants
> to
> host
> > >it that would be fine.  I haven't applied a license to it yet.  I
> think
> > >it's simple enough that I could just put a zlib license on it and be
> done.
> > >
> > >Jim Burnes
> > >Application Security Engineer
> > >
> > >
Loading...