Similar effort to formally discribe event data

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Similar effort to formally discribe event data

Philip Black-Knight

We’ve been working on a similar effort for describing an enumerating events. Our architecture is comprised of 3 distinct documents:

-       EventProfileList (profiles.xsd) – An EventProfileList is an enumeration of events for a class of device (e.g. router, Cisco router, UNIX system, Solaris). The EventProfileList can be for a generic device (e.g. router, switch, UNIX system) or specific to a vender/product (e.g. Cisco Router, Juniper switch, Oracle Solaris). This is similar to how SNMP works. The type for each field is recorded in the EventProfile, and would be one of the existing CEE dictionary types.

-       Events (events.xsd) – An Events document is a listing of one or more event instances (called records). Each record references an Event from a specific EventProfile.

-       Translations (translations.xsd) – A Translation document provides a human readable, language specific, version of the events enumerated in an EventProfileList. To display the details of an Event, you would use a record from an Events document, combined with the translation from a Translations document, to render a human readable string in the user’s language of choice.

For example, an EventProfileList could include an event that indicates a user logged on to a particular system. In the EventProfileList, the descriptor for this login event would indicate the required fields: user_name, hostname, and tty. When a login event occurs, the system would record the event in an Events document, including the values for these fields. A Translations document (for US English) might include the format string: “%user_name% logged on to %hostname% on %tty% at %timestamp%”. Using this format string, and the event record from the Events document, a human readable string can be rendered for the user.

One of the primary goals is to move away from loosely formatted event messages which are hard to parse, and can change from one version of software package to the next. Using a common format to enumerate event records makes processing and correlating data simpler. This is all part of a larger effort to define data formats and protocols for reliably transferring Journal/Audit/Logging events. Additionally, we’d like to be able to align various software products that perform similar tasks (i.e. routers, or network switches, etc) to all use a common set of events so that a router built by Cisco & a router built by Juniper Networks would both generate the same events. Each vendor would be free to augment their list of events (by providing their own EventProfileLists) and would be able to define new events, and/or extend existing events.

Existing syslog data is easily recorded using this approach. The EventProfile for syslog data indicates the required fields for msg (the actual syslog message), facility, priority, hostname, applicationName, etc that are part of standard syslog messages. The attached zip includes an EventProfileList & sample Events document for syslog.

 

Another important goal was to support existing logging systems. While creating these schemas, we studied Syslog, Windows Event Structure, Solaris Audit, and Linux Audit and believe that all can be represented in our Event structures with no loss of information or context.

Other features:

-       Support optional digital signatures of all XML files to meet government requirement (Sarbanes-Oxley, Frank-Dodd, HIPAA, etc...) s for audit log integrity and non-repudiation

-       Updates to language translations (both new languages and to fix errors in translations) can be added without the need to update a single line of code.

-       Both the EventProfileLists and the Event descriptors within the EventProfile list are versioned, allowing event records & translations to target a specific version of an event.

-       Documents containing Event records may be in XML or JSON format

-       Limited support for Event inheritance. Each Event descriptor can inherit from some other Event descriptor (multiple inheritance is not supported). For example, a generic profile could exist that describes a “logon” event, and specifies the required fields username, hostname, and time. The EventProfileList for Windows would identify a login event that inherits from this base, and adds the fields first_name, last_name, guid, etc. The EventProfileList for Linux would add fields for uid, tty, etc. When a Linux Logon event is recorded, in addition to the uid and tty, it must also specify the fields from the inherited class (i.e. username, hostname, and time). This is similar to current CEE approach to taxonomies.

 

To further support this effort, we’ve been working on a network protocol to reliably and securely transfer Event records off box. These protocols are designed to support raw XML as well as compressed formats such as Efficient XML Interchange
Attached are a draft set of schemas and sample documents (it is just a renamed zip file).

-Phil


eventSchemas.1814d42.piz (14K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Similar effort to formally discribe event data

zrlram
Hi Phil,

When you say "we have been ...", who is we?

The goals you list are pretty much the ones we are addressing with CEE. Glad we are all on the same page. Would love to hear your specific input on CEE.

Thanks

  -raffy

--
  Raffael Marty
  @raffaelmarty                                          http://raffy.ch

On Aug 30, 2012, at 5:15 AM, Philip Black-Knight <[hidden email]> wrote:

> We’ve been working on a similar effort for describing an enumerating events. Our architecture is comprised of 3 distinct documents:
>
> -       EventProfileList (profiles.xsd) – An EventProfileList is an enumeration of events for a class of device (e.g. router, Cisco router, UNIX system, Solaris). The EventProfileList can be for a generic device (e.g. router, switch, UNIX system) or specific to a vender/product (e.g. Cisco Router, Juniper switch, Oracle Solaris). This is similar to how SNMP works. The type for each field is recorded in the EventProfile, and would be one of the existing CEE dictionary types.
>
> -       Events (events.xsd) – An Events document is a listing of one or more event instances (called records). Each record references an Event from a specific EventProfile.
>
> -       Translations (translations.xsd) – A Translation document provides a human readable, language specific, version of the events enumerated in an EventProfileList. To display the details of an Event, you would use a record from an Events document, combined with the translation from a Translations document, to render a human readable string in the user’s language of choice.
>
> For example, an EventProfileList could include an event that indicates a user logged on to a particular system. In the EventProfileList, the descriptor for this login event would indicate the required fields: user_name, hostname, and tty. When a login event occurs, the system would record the event in an Events document, including the values for these fields. A Translations document (for US English) might include the format string: “%user_name% logged on to %hostname% on %tty% at %timestamp%”. Using this format string, and the event record from the Events document, a human readable string can be rendered for the user.
>
> One of the primary goals is to move away from loosely formatted event messages which are hard to parse, and can change from one version of software package to the next. Using a common format to enumerate event records makes processing and correlating data simpler. This is all part of a larger effort to define data formats and protocols for reliably transferring Journal/Audit/Logging events. Additionally, we’d like to be able to align various software products that perform similar tasks (i.e. routers, or network switches, etc) to all use a common set of events so that a router built by Cisco & a router built by Juniper Networks would both generate the same events. Each vendor would be free to augment their list of events (by providing their own EventProfileLists) and would be able to define new events, and/or extend existing events.
>
> Existing syslog data is easily recorded using this approach. The EventProfile for syslog data indicates the required fields for msg (the actual syslog message), facility, priority, hostname, applicationName, etc that are part of standard syslog messages. The attached zip includes an EventProfileList & sample Events document for syslog.
>  
>
> Another important goal was to support existing logging systems. While creating these schemas, we studied Syslog, Windows Event Structure, Solaris Audit, and Linux Audit and believe that all can be represented in our Event structures with no loss of information or context.
>
> Other features:
>
> -       Support optional digital signatures of all XML files to meet government requirement (Sarbanes-Oxley, Frank-Dodd, HIPAA, etc...) s for audit log integrity and non-repudiation
>
> -       Updates to language translations (both new languages and to fix errors in translations) can be added without the need to update a single line of code.
>
> -       Both the EventProfileLists and the Event descriptors within the EventProfile list are versioned, allowing event records & translations to target a specific version of an event.
>
> -       Documents containing Event records may be in XML or JSON format
>
> -       Limited support for Event inheritance. Each Event descriptor can inherit from some other Event descriptor (multiple inheritance is not supported). For example, a generic profile could exist that describes a “logon” event, and specifies the required fields username, hostname, and time. The EventProfileList for Windows would identify a login event that inherits from this base, and adds the fields first_name, last_name, guid, etc. The EventProfileList for Linux would add fields for uid, tty, etc. When a Linux Logon event is recorded, in addition to the uid and tty, it must also specify the fields from the inherited class (i.e. username, hostname, and time). This is similar to current CEE approach to taxonomies.
>
>  
> To further support this effort, we’ve been working on a network protocol to reliably and securely transfer Event records off box. These protocols are designed to support raw XML as well as compressed formats such as Efficient XML Interchange
> Attached are a draft set of schemas and sample documents (it is just a renamed zip file).
>
> -Phil
>
> <eventSchemas.1814d42.piz>
Reply | Threaded
Open this post in threaded view
|

Re: Similar effort to formally discribe event data

Philip Black-Knight
I'm working on a government funded project (JALoP). The primary goals are to author & implement a protocol to reliably transfer what we refer to as journal, audit, and log data. Much of our efforts are related to what CEE labels the CLT.

However, we've also done some work (i.e. the schemas I just sent out) related to describing log data in a strict format. Because our schemas are similar (and have similar goals) we were hoping that it would be possible to combine our efforts.

We feel that our schemas are a lot simpler and easier to understand than the current CEE approach, and would like to see the CEE format transformed into a something a little simpler and easier to follow.

-Phil

-----Original Message-----
From: Raffael Marty [mailto:[hidden email]]
Sent: Thursday, August 30, 2012 12:12 PM
To: Philip Black-Knight
Cc: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Similar effort to formally discribe event data

Hi Phil,

When you say "we have been ...", who is we?

The goals you list are pretty much the ones we are addressing with CEE. Glad we are all on the same page. Would love to hear your specific input on CEE.

Thanks

  -raffy

--
  Raffael Marty
  @raffaelmarty                                          http://raffy.ch

On Aug 30, 2012, at 5:15 AM, Philip Black-Knight <[hidden email]> wrote:

> We've been working on a similar effort for describing an enumerating events. Our architecture is comprised of 3 distinct documents:
>
> -       EventProfileList (profiles.xsd) - An EventProfileList is an enumeration of events for a class of device (e.g. router, Cisco router, UNIX system, Solaris). The EventProfileList can be for a generic device (e.g. router, switch, UNIX system) or specific to a vender/product (e.g. Cisco Router, Juniper switch, Oracle Solaris). This is similar to how SNMP works. The type for each field is recorded in the EventProfile, and would be one of the existing CEE dictionary types.
>
> -       Events (events.xsd) - An Events document is a listing of one or more event instances (called records). Each record references an Event from a specific EventProfile.
>
> -       Translations (translations.xsd) - A Translation document provides a human readable, language specific, version of the events enumerated in an EventProfileList. To display the details of an Event, you would use a record from an Events document, combined with the translation from a Translations document, to render a human readable string in the user's language of choice.
>
> For example, an EventProfileList could include an event that indicates a user logged on to a particular system. In the EventProfileList, the descriptor for this login event would indicate the required fields: user_name, hostname, and tty. When a login event occurs, the system would record the event in an Events document, including the values for these fields. A Translations document (for US English) might include the format string: "%user_name% logged on to %hostname% on %tty% at %timestamp%". Using this format string, and the event record from the Events document, a human readable string can be rendered for the user.
>
> One of the primary goals is to move away from loosely formatted event messages which are hard to parse, and can change from one version of software package to the next. Using a common format to enumerate event records makes processing and correlating data simpler. This is all part of a larger effort to define data formats and protocols for reliably transferring Journal/Audit/Logging events. Additionally, we'd like to be able to align various software products that perform similar tasks (i.e. routers, or network switches, etc) to all use a common set of events so that a router built by Cisco & a router built by Juniper Networks would both generate the same events. Each vendor would be free to augment their list of events (by providing their own EventProfileLists) and would be able to define new events, and/or extend existing events.
>
> Existing syslog data is easily recorded using this approach. The EventProfile for syslog data indicates the required fields for msg (the actual syslog message), facility, priority, hostname, applicationName, etc that are part of standard syslog messages. The attached zip includes an EventProfileList & sample Events document for syslog.
>  
>
> Another important goal was to support existing logging systems. While creating these schemas, we studied Syslog, Windows Event Structure, Solaris Audit, and Linux Audit and believe that all can be represented in our Event structures with no loss of information or context.
>
> Other features:
>
> -       Support optional digital signatures of all XML files to meet government requirement (Sarbanes-Oxley, Frank-Dodd, HIPAA, etc...) s for audit log integrity and non-repudiation
>
> -       Updates to language translations (both new languages and to fix errors in translations) can be added without the need to update a single line of code.
>
> -       Both the EventProfileLists and the Event descriptors within the EventProfile list are versioned, allowing event records & translations to target a specific version of an event.
>
> -       Documents containing Event records may be in XML or JSON format
>
> -       Limited support for Event inheritance. Each Event descriptor can inherit from some other Event descriptor (multiple inheritance is not supported). For example, a generic profile could exist that describes a "logon" event, and specifies the required fields username, hostname, and time. The EventProfileList for Windows would identify a login event that inherits from this base, and adds the fields first_name, last_name, guid, etc. The EventProfileList for Linux would add fields for uid, tty, etc. When a Linux Logon event is recorded, in addition to the uid and tty, it must also specify the fields from the inherited class (i.e. username, hostname, and time). This is similar to current CEE approach to taxonomies.
>
>  
> To further support this effort, we've been working on a network
> protocol to reliably and securely transfer Event records off box. These protocols are designed to support raw XML as well as compressed formats such as Efficient XML Interchange Attached are a draft set of schemas and sample documents (it is just a renamed zip file).
>
> -Phil
>
> <eventSchemas.1814d42.piz>
Reply | Threaded
Open this post in threaded view
|

Re: Similar effort to formally discribe event data

Wunder, John A.
I've been working with Phil and the JALoP team to help them understand CEE and where we're coming from as well as to understand more about where they're coming from. So at this point I'm thinking I have a pretty good understanding of the differences.

In my opinion the most important distinction is that we have different focus areas. JALoP is focusing on syntax and (outside of the audit schema) the transport. CEE has the syntax component but also has more of a focus on defining the event taxonomy and core field dictionary. So to that extent if we move towards the JALoP schemas our existing work on the taxonomy and field dictionary wouldn't be significantly affected. The caveat is that we'll have to go back and forth a little bit on how the taxonomy ends up being represented, since although their concept is kind of a taxonomy it's implemented very differently from ours.
 
In the syntax specifically, the main philosophical difference is that CEE's primary focus has been JSON and lightweight uses while JALoPs has been XML and more "heavy" uses (that require validation, integrity, etc). That ends up being reflected in some of the structures that we use. The proposal Phil has here includes both JSON and XML, just like CEE does, but because it started as XML the JSON is influenced a little by that. Take a look at what he sent (the .piz file is a .zip file w/ the schemas) and compare it to what we have now.

I think it's unlikely that we'd end up doing a full replacement of the CEE syntax with the JALoP audit syntax, although of course that's definitely an option. But in any case we should look at what's out there and take the best pieces of each. If it would help I could set up a telecom where Phil and the team can explain what they've done and you all can ask questions.

I'd especially want to get the opinion of event record PRODUCERS here, since they're the one that will be using the syntax.

John

-----Original Message-----
From: Philip Black-Knight [mailto:[hidden email]]
Sent: Thursday, August 30, 2012 1:17 PM
To: cee-discussion-list CEE-Related Discussion
Subject: Re: [CEE-DISCUSSION-LIST] Similar effort to formally discribe event data

I'm working on a government funded project (JALoP). The primary goals are to author & implement a protocol to reliably transfer what we refer to as journal, audit, and log data. Much of our efforts are related to what CEE labels the CLT.

However, we've also done some work (i.e. the schemas I just sent out) related to describing log data in a strict format. Because our schemas are similar (and have similar goals) we were hoping that it would be possible to combine our efforts.

We feel that our schemas are a lot simpler and easier to understand than the current CEE approach, and would like to see the CEE format transformed into a something a little simpler and easier to follow.

-Phil

-----Original Message-----
From: Raffael Marty [mailto:[hidden email]]
Sent: Thursday, August 30, 2012 12:12 PM
To: Philip Black-Knight
Cc: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Similar effort to formally discribe event data

Hi Phil,

When you say "we have been ...", who is we?

The goals you list are pretty much the ones we are addressing with CEE. Glad we are all on the same page. Would love to hear your specific input on CEE.

Thanks

  -raffy

--
  Raffael Marty
  @raffaelmarty                                          http://raffy.ch

On Aug 30, 2012, at 5:15 AM, Philip Black-Knight <[hidden email]> wrote:

> We've been working on a similar effort for describing an enumerating events. Our architecture is comprised of 3 distinct documents:
>
> -       EventProfileList (profiles.xsd) - An EventProfileList is an enumeration of events for a class of device (e.g. router, Cisco router, UNIX system, Solaris). The EventProfileList can be for a generic device (e.g. router, switch, UNIX system) or specific to a vender/product (e.g. Cisco Router, Juniper switch, Oracle Solaris). This is similar to how SNMP works. The type for each field is recorded in the EventProfile, and would be one of the existing CEE dictionary types.
>
> -       Events (events.xsd) - An Events document is a listing of one or more event instances (called records). Each record references an Event from a specific EventProfile.
>
> -       Translations (translations.xsd) - A Translation document provides a human readable, language specific, version of the events enumerated in an EventProfileList. To display the details of an Event, you would use a record from an Events document, combined with the translation from a Translations document, to render a human readable string in the user's language of choice.
>
> For example, an EventProfileList could include an event that indicates a user logged on to a particular system. In the EventProfileList, the descriptor for this login event would indicate the required fields: user_name, hostname, and tty. When a login event occurs, the system would record the event in an Events document, including the values for these fields. A Translations document (for US English) might include the format string: "%user_name% logged on to %hostname% on %tty% at %timestamp%". Using this format string, and the event record from the Events document, a human readable string can be rendered for the user.
>
> One of the primary goals is to move away from loosely formatted event messages which are hard to parse, and can change from one version of software package to the next. Using a common format to enumerate event records makes processing and correlating data simpler. This is all part of a larger effort to define data formats and protocols for reliably transferring Journal/Audit/Logging events. Additionally, we'd like to be able to align various software products that perform similar tasks (i.e. routers, or network switches, etc) to all use a common set of events so that a router built by Cisco & a router built by Juniper Networks would both generate the same events. Each vendor would be free to augment their list of events (by providing their own EventProfileLists) and would be able to define new events, and/or extend existing events.
>
> Existing syslog data is easily recorded using this approach. The EventProfile for syslog data indicates the required fields for msg (the actual syslog message), facility, priority, hostname, applicationName, etc that are part of standard syslog messages. The attached zip includes an EventProfileList & sample Events document for syslog.
>  
>
> Another important goal was to support existing logging systems. While creating these schemas, we studied Syslog, Windows Event Structure, Solaris Audit, and Linux Audit and believe that all can be represented in our Event structures with no loss of information or context.
>
> Other features:
>
> -       Support optional digital signatures of all XML files to meet government requirement (Sarbanes-Oxley, Frank-Dodd, HIPAA, etc...) s for audit log integrity and non-repudiation
>
> -       Updates to language translations (both new languages and to fix errors in translations) can be added without the need to update a single line of code.
>
> -       Both the EventProfileLists and the Event descriptors within the EventProfile list are versioned, allowing event records & translations to target a specific version of an event.
>
> -       Documents containing Event records may be in XML or JSON format
>
> -       Limited support for Event inheritance. Each Event descriptor can inherit from some other Event descriptor (multiple inheritance is not supported). For example, a generic profile could exist that describes a "logon" event, and specifies the required fields username, hostname, and time. The EventProfileList for Windows would identify a login event that inherits from this base, and adds the fields first_name, last_name, guid, etc. The EventProfileList for Linux would add fields for uid, tty, etc. When a Linux Logon event is recorded, in addition to the uid and tty, it must also specify the fields from the inherited class (i.e. username, hostname, and time). This is similar to current CEE approach to taxonomies.
>
>  
> To further support this effort, we've been working on a network
> protocol to reliably and securely transfer Event records off box. These protocols are designed to support raw XML as well as compressed formats such as Efficient XML Interchange Attached are a draft set of schemas and sample documents (it is just a renamed zip file).
>
> -Phil
>
> <eventSchemas.1814d42.piz>