Proposed Changes to CEE XSD

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposed Changes to CEE XSD

Keith Robertson
All,

In a previous thread, we discussed the idea of adding a free-form
message element to the CEE event.  I have made some small changes to a
version of the CEE XSD that would allow the optional inclusion of such
an element.   In this email you will find the enhanced CEE XSD, sample
XML that is both valid and well-formed to the new CEE XSD, and a
test-harness that demonstrates how a stream of CEE events containing
these extensions can be successfully parsed.

- Background:
The CEE XSD attached to this email is based off 0.6 and fixes some
serious flaws in that version.  The most notable issues in 0.6 are the
clunky extension mechanism and the fact that extension elements exist
outside the <Event/> element (i.e. they are not children).  This is a
serious issue that could make it very complex for event consumers to
correlate extensions added by relays to their parent events.  See [1]
for an illustration of the problem.

- Summary of Improvements to the 0.6 CEE XSD:
1: The base attributes that we would like to see in every good CEE event
are now *required* (i.e. crit, id,p_app, p_proc, p_proc_id, etc.) and
failure to supply these minimal attributes produces invalid XML or, by
extension, an invalid CEE event.
2. In the 0.6 version of the XSD, there was an element called
<EventExtension/>.  This element has been renamed <Module/> and is an
immediate child of <Event/>.  This fact will make parsing and
correlation significantly easier (see [1]).
3. In the 0.6 version of the CEE XSD, there was the notion of <Field/>s
to facilitate the transport of program specific data.  This concept has
been replaced with proper OOP abstraction.  The new CEE XSD defines an
abstract element called "BaseProfile".  Any projects/programs that wish
to extend the CEE event with their own schema *must* extend this profile
element in true OOP form.  The fact that BaseProfile is both abstract
and that it is optional ensures that CEE parsers do not need to be aware
of every possible program extension.  This very important scenario is
exercised in the attached test harness.

- Test Harness Explanation:
The attached test harness demonstrates how an application would extend
the base CEE event through the <BaseProfile/> abstract element.  An
important consideration when allowing extensions is the assurance that a
parser generated from the base CEE XSD need *not* be aware of
application/vendor extensions to properly parse a stream of CEE events.  
Because we have made the <BaseProfile/> element both abstract and
optional this is possible and is demonstrated in the test harness.

The test harness is written in Java and uses JAXB to create bindings
(classes) to the XSD elements.  Two sets of bindings were created for
this test harness.  The first set of bindings live in the directory
CEE-Test/src/com/mitre/cee.  These bindings (classes) are generated from
the base CEE XSD (i.e. cee.xsd in com/mitre/cee/schemas).  The second
set of bindings demonstrate how an application would extend the base CEE
event and provide product specific data.  The product I selected for
this example is netfilter.org's iptables and its bindings live in the
directory CEE-Test/src/com/netfilter/cee.  This extension is described
in CEE-Test/src/com/mitre/schemas/iptables.xsd.  Please review it and
notice how easy it is to extend <BaseProfile/>.

The harness is a small Java program
(CEE-Test/src/com/mitre/cee/test/CEETest.java) that reads in a stream of
CEE events which are described in
CEE-Test/src/com/mitre/cee/samplexml/ceesample.xml.  The harness
demonstrates that it is possible/easy to parse a CEE stream with product
specific extensions without knowing them "a priori".  The harness
demonstrates how an application can include free-form text via the
<SimpleMessageProfile/>.

- Test Harness File Locations:
The modified CEE XSD is here: CEE-Test/src/com/mitre/schemas/cee.xsd
The netfilter.org extension of the CEE XSD is here:
CEE-Test/src/com/mitre/schemas/iptables.xsd
The base CEE bindings are here: CEE-Test/src/com/mitre/cee
The netfilter/iptables CEE bindings are here: CEE-Test/src/com/netfilter/cee

- Test Harness Usage:
0. The attached zip file is an Eclipse project.  If you have Eclipse,
you can simply import it. If you want to run the project you will need
Java 1.6.
1. Unzip CEE-Test.zip
2. cwd into CEE-Test
3.  java -classpath ./bin/ com.mitre.cee.test.CEETest


[1] Extensions should live inside the <Event/> element:
Sample flow:
   [Emitter]--> <Event N/>... <Event 2/> ... <Event 1/> --> [Relay] -->
<Event N/>+<Extension for Event N/>... <Event 2/>+<Extension for Event
2/> --> [Parser]
Problem description:
   The [Parser] will have the unenviable task of correlating
<Extension/>s to <Event/>s.  This could be completely eliminated if
there [Relay] could augment the <Event/> element directly.  In fact, it
might be really nasty if the<Event/>s were delivered to a server socked
over UDP where the server is receiving multiple event streams at once.






CEE-Test.zip (95K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposed Changes to CEE XSD

heinbockel
Keith,

Great!

There are a few differences between your schemas and mine. I've rolled my
latest versions into your proposed schemas.

Generally, the only differences were minor, mainly a usage of attributes vs.
elements. As I spoke to you on the phone, there are some inconsistencies
with marshaling between JSON and XML when using XML elements/attributes.

The only two consistent solutions are, either:

1. Require that attribute names in JSON be prepended with "@" (a la XPath)
2. Do not allow the use of XML attributes

Currently, we are favoring the second option -- require all CEE event fields
to be encoded in XML as XML elements.


William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Keith Robertson [mailto:[hidden email]]
>Sent: Sunday, 05 February, 2012 22:38
>To: cee-discussion-list CEE-Related Discussion; Heinbockel, Bill
>Cc: Jack Rieden; Denise Dumas
>Subject: Proposed Changes to CEE XSD
>
>All,
>
>In a previous thread, we discussed the idea of adding a free-form
>message element to the CEE event.  I have made some small changes to a
>version of the CEE XSD that would allow the optional inclusion of such
>an element.   In this email you will find the enhanced CEE XSD, sample
>XML that is both valid and well-formed to the new CEE XSD, and a
>test-harness that demonstrates how a stream of CEE events containing
>these extensions can be successfully parsed.
>
>- Background:
>The CEE XSD attached to this email is based off 0.6 and fixes some
>serious flaws in that version.  The most notable issues in 0.6 are the
>clunky extension mechanism and the fact that extension elements exist
>outside the <Event/> element (i.e. they are not children).  This is a
>serious issue that could make it very complex for event consumers to
>correlate extensions added by relays to their parent events.  See [1]
>for an illustration of the problem.
>
>- Summary of Improvements to the 0.6 CEE XSD:
>1: The base attributes that we would like to see in every good CEE event
>are now *required* (i.e. crit, id,p_app, p_proc, p_proc_id, etc.) and
>failure to supply these minimal attributes produces invalid XML or, by
>extension, an invalid CEE event.
>2. In the 0.6 version of the XSD, there was an element called
><EventExtension/>.  This element has been renamed <Module/> and is an
>immediate child of <Event/>.  This fact will make parsing and
>correlation significantly easier (see [1]).
>3. In the 0.6 version of the CEE XSD, there was the notion of <Field/>s
>to facilitate the transport of program specific data.  This concept has
>been replaced with proper OOP abstraction.  The new CEE XSD defines an
>abstract element called "BaseProfile".  Any projects/programs that wish
>to extend the CEE event with their own schema *must* extend this profile
>element in true OOP form.  The fact that BaseProfile is both abstract
>and that it is optional ensures that CEE parsers do not need to be aware
>of every possible program extension.  This very important scenario is
>exercised in the attached test harness.
>
>- Test Harness Explanation:
>The attached test harness demonstrates how an application would extend
>the base CEE event through the <BaseProfile/> abstract element.  An
>important consideration when allowing extensions is the assurance that a
>parser generated from the base CEE XSD need *not* be aware of
>application/vendor extensions to properly parse a stream of CEE events.
>Because we have made the <BaseProfile/> element both abstract and
>optional this is possible and is demonstrated in the test harness.
>
>The test harness is written in Java and uses JAXB to create bindings
>(classes) to the XSD elements.  Two sets of bindings were created for
>this test harness.  The first set of bindings live in the directory
>CEE-Test/src/com/mitre/cee.  These bindings (classes) are generated from
>the base CEE XSD (i.e. cee.xsd in com/mitre/cee/schemas).  The second
>set of bindings demonstrate how an application would extend the base CEE
>event and provide product specific data.  The product I selected for
>this example is netfilter.org's iptables and its bindings live in the
>directory CEE-Test/src/com/netfilter/cee.  This extension is described
>in CEE-Test/src/com/mitre/schemas/iptables.xsd.  Please review it and
>notice how easy it is to extend <BaseProfile/>.
>
>The harness is a small Java program
>(CEE-Test/src/com/mitre/cee/test/CEETest.java) that reads in a stream of
>CEE events which are described in
>CEE-Test/src/com/mitre/cee/samplexml/ceesample.xml.  The harness
>demonstrates that it is possible/easy to parse a CEE stream with product
>specific extensions without knowing them "a priori".  The harness
>demonstrates how an application can include free-form text via the
><SimpleMessageProfile/>.
>
>- Test Harness File Locations:
>The modified CEE XSD is here: CEE-Test/src/com/mitre/schemas/cee.xsd
>The netfilter.org extension of the CEE XSD is here:
>CEE-Test/src/com/mitre/schemas/iptables.xsd
>The base CEE bindings are here: CEE-Test/src/com/mitre/cee
>The netfilter/iptables CEE bindings are here: CEE-
>Test/src/com/netfilter/cee
>
>- Test Harness Usage:
>0. The attached zip file is an Eclipse project.  If you have Eclipse,
>you can simply import it. If you want to run the project you will need
>Java 1.6.
>1. Unzip CEE-Test.zip
>2. cwd into CEE-Test
>3.  java -classpath ./bin/ com.mitre.cee.test.CEETest
>
>
>[1] Extensions should live inside the <Event/> element:
>Sample flow:
>   [Emitter]--> <Event N/>... <Event 2/> ... <Event 1/> --> [Relay] -->
><Event N/>+<Extension for Event N/>... <Event 2/>+<Extension for Event
>2/> --> [Parser]
>Problem description:
>   The [Parser] will have the unenviable task of correlating
><Extension/>s to <Event/>s.  This could be completely eliminated if
>there [Relay] could augment the <Event/> element directly.  In fact, it
>might be really nasty if the<Event/>s were delivered to a server socked
>over UDP where the server is receiving multiple event streams at once.
>
>
>
>


cee-schemas-20120206.zip (12K) Download Attachment
smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposed Changes to CEE XSD

STOECKP
Bill,

Is the test harness available for us to look at?

Paul



-----Original Message-----
From: Heinbockel, Bill [mailto:[hidden email]]
Sent: Monday, February 06, 2012 10:55 AM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Proposed Changes to CEE XSD

Keith,

Great!

There are a few differences between your schemas and mine. I've rolled my
latest versions into your proposed schemas.

Generally, the only differences were minor, mainly a usage of attributes vs.
elements. As I spoke to you on the phone, there are some inconsistencies
with marshaling between JSON and XML when using XML elements/attributes.

The only two consistent solutions are, either:

1. Require that attribute names in JSON be prepended with "@" (a la XPath)
2. Do not allow the use of XML attributes

Currently, we are favoring the second option -- require all CEE event fields
to be encoded in XML as XML elements.


William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Keith Robertson [mailto:[hidden email]]
>Sent: Sunday, 05 February, 2012 22:38
>To: cee-discussion-list CEE-Related Discussion; Heinbockel, Bill
>Cc: Jack Rieden; Denise Dumas
>Subject: Proposed Changes to CEE XSD
>
>All,
>
>In a previous thread, we discussed the idea of adding a free-form
>message element to the CEE event.  I have made some small changes to a
>version of the CEE XSD that would allow the optional inclusion of such
>an element.   In this email you will find the enhanced CEE XSD, sample
>XML that is both valid and well-formed to the new CEE XSD, and a
>test-harness that demonstrates how a stream of CEE events containing
>these extensions can be successfully parsed.
>
>- Background:
>The CEE XSD attached to this email is based off 0.6 and fixes some
>serious flaws in that version.  The most notable issues in 0.6 are the
>clunky extension mechanism and the fact that extension elements exist
>outside the <Event/> element (i.e. they are not children).  This is a
>serious issue that could make it very complex for event consumers to
>correlate extensions added by relays to their parent events.  See [1]
>for an illustration of the problem.
>
>- Summary of Improvements to the 0.6 CEE XSD:
>1: The base attributes that we would like to see in every good CEE event
>are now *required* (i.e. crit, id,p_app, p_proc, p_proc_id, etc.) and
>failure to supply these minimal attributes produces invalid XML or, by
>extension, an invalid CEE event.
>2. In the 0.6 version of the XSD, there was an element called
><EventExtension/>.  This element has been renamed <Module/> and is an
>immediate child of <Event/>.  This fact will make parsing and
>correlation significantly easier (see [1]).
>3. In the 0.6 version of the CEE XSD, there was the notion of <Field/>s
>to facilitate the transport of program specific data.  This concept has
>been replaced with proper OOP abstraction.  The new CEE XSD defines an
>abstract element called "BaseProfile".  Any projects/programs that wish
>to extend the CEE event with their own schema *must* extend this profile
>element in true OOP form.  The fact that BaseProfile is both abstract
>and that it is optional ensures that CEE parsers do not need to be aware
>of every possible program extension.  This very important scenario is
>exercised in the attached test harness.
>
>- Test Harness Explanation:
>The attached test harness demonstrates how an application would extend
>the base CEE event through the <BaseProfile/> abstract element.  An
>important consideration when allowing extensions is the assurance that a
>parser generated from the base CEE XSD need *not* be aware of
>application/vendor extensions to properly parse a stream of CEE events.
>Because we have made the <BaseProfile/> element both abstract and
>optional this is possible and is demonstrated in the test harness.
>
>The test harness is written in Java and uses JAXB to create bindings
>(classes) to the XSD elements.  Two sets of bindings were created for
>this test harness.  The first set of bindings live in the directory
>CEE-Test/src/com/mitre/cee.  These bindings (classes) are generated from
>the base CEE XSD (i.e. cee.xsd in com/mitre/cee/schemas).  The second
>set of bindings demonstrate how an application would extend the base CEE
>event and provide product specific data.  The product I selected for
>this example is netfilter.org's iptables and its bindings live in the
>directory CEE-Test/src/com/netfilter/cee.  This extension is described
>in CEE-Test/src/com/mitre/schemas/iptables.xsd.  Please review it and
>notice how easy it is to extend <BaseProfile/>.
>
>The harness is a small Java program
>(CEE-Test/src/com/mitre/cee/test/CEETest.java) that reads in a stream of
>CEE events which are described in
>CEE-Test/src/com/mitre/cee/samplexml/ceesample.xml.  The harness
>demonstrates that it is possible/easy to parse a CEE stream with product
>specific extensions without knowing them "a priori".  The harness
>demonstrates how an application can include free-form text via the
><SimpleMessageProfile/>.
>
>- Test Harness File Locations:
>The modified CEE XSD is here: CEE-Test/src/com/mitre/schemas/cee.xsd
>The netfilter.org extension of the CEE XSD is here:
>CEE-Test/src/com/mitre/schemas/iptables.xsd
>The base CEE bindings are here: CEE-Test/src/com/mitre/cee
>The netfilter/iptables CEE bindings are here: CEE-
>Test/src/com/netfilter/cee
>
>- Test Harness Usage:
>0. The attached zip file is an Eclipse project.  If you have Eclipse,
>you can simply import it. If you want to run the project you will need
>Java 1.6.
>1. Unzip CEE-Test.zip
>2. cwd into CEE-Test
>3.  java -classpath ./bin/ com.mitre.cee.test.CEETest
>
>
>[1] Extensions should live inside the <Event/> element:
>Sample flow:
>   [Emitter]--> <Event N/>... <Event 2/> ... <Event 1/> --> [Relay] -->
><Event N/>+<Extension for Event N/>... <Event 2/>+<Extension for Event
>2/> --> [Parser]
>Problem description:
>   The [Parser] will have the unenviable task of correlating
><Extension/>s to <Event/>s.  This could be completely eliminated if
>there [Relay] could augment the <Event/> element directly.  In fact, it
>might be really nasty if the<Event/>s were delivered to a server socked
>over UDP where the server is receiving multiple event streams at once.
>
>
>
>


smime.p7s (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposed Changes to CEE XSD

Keith Robertson
Paul,

The test harness is attached to my original message[1].  Bill has suggested some slight modifications to my updated XSD.  They will not substantially alter the harness but, I will update it soon and repost.

[1] http://making-security-measurable.1364806.n2.nabble.com/Proposed-Changes-to-CEE-XSD-td7257476.html

Cheers,
Keith


Keith Robertson
1-919-754-4004


From: "Paul Stoecker" <[hidden email]>
To: [hidden email]
Sent: Wednesday, February 8, 2012 10:08:16 AM
Subject: Re: [CEE-DISCUSSION-LIST] Proposed Changes to CEE XSD

Bill,

Is the test harness available for us to look at?

Paul



-----Original Message-----
From: Heinbockel, Bill [mailto:[hidden email]]
Sent: Monday, February 06, 2012 10:55 AM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Proposed Changes to CEE XSD

Keith,

Great!

There are a few differences between your schemas and mine. I've rolled my
latest versions into your proposed schemas.

Generally, the only differences were minor, mainly a usage of attributes vs.
elements. As I spoke to you on the phone, there are some inconsistencies
with marshaling between JSON and XML when using XML elements/attributes.

The only two consistent solutions are, either:

1. Require that attribute names in JSON be prepended with "@" (a la XPath)
2. Do not allow the use of XML attributes

Currently, we are favoring the second option -- require all CEE event fields
to be encoded in XML as XML elements.


William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Keith Robertson [mailto:[hidden email]]
>Sent: Sunday, 05 February, 2012 22:38
>To: cee-discussion-list CEE-Related Discussion; Heinbockel, Bill
>Cc: Jack Rieden; Denise Dumas
>Subject: Proposed Changes to CEE XSD
>
>All,
>
>In a previous thread, we discussed the idea of adding a free-form
>message element to the CEE event.  I have made some small changes to a
>version of the CEE XSD that would allow the optional inclusion of such
>an element.   In this email you will find the enhanced CEE XSD, sample
>XML that is both valid and well-formed to the new CEE XSD, and a
>test-harness that demonstrates how a stream of CEE events containing
>these extensions can be successfully parsed.
>
>- Background:
>The CEE XSD attached to this email is based off 0.6 and fixes some
>serious flaws in that version.  The most notable issues in 0.6 are the
>clunky extension mechanism and the fact that extension elements exist
>outside the <Event/> element (i.e. they are not children).  This is a
>serious issue that could make it very complex for event consumers to
>correlate extensions added by relays to their parent events.  See [1]
>for an illustration of the problem.
>
>- Summary of Improvements to the 0.6 CEE XSD:
>1: The base attributes that we would like to see in every good CEE event
>are now *required* (i.e. crit, id,p_app, p_proc, p_proc_id, etc.) and
>failure to supply these minimal attributes produces invalid XML or, by
>extension, an invalid CEE event.
>2. In the 0.6 version of the XSD, there was an element called
><EventExtension/>.  This element has been renamed <Module/> and is an
>immediate child of <Event/>.  This fact will make parsing and
>correlation significantly easier (see [1]).
>3. In the 0.6 version of the CEE XSD, there was the notion of <Field/>s
>to facilitate the transport of program specific data.  This concept has
>been replaced with proper OOP abstraction.  The new CEE XSD defines an
>abstract element called "BaseProfile".  Any projects/programs that wish
>to extend the CEE event with their own schema *must* extend this profile
>element in true OOP form.  The fact that BaseProfile is both abstract
>and that it is optional ensures that CEE parsers do not need to be aware
>of every possible program extension.  This very important scenario is
>exercised in the attached test harness.
>
>- Test Harness Explanation:
>The attached test harness demonstrates how an application would extend
>the base CEE event through the <BaseProfile/> abstract element.  An
>important consideration when allowing extensions is the assurance that a
>parser generated from the base CEE XSD need *not* be aware of
>application/vendor extensions to properly parse a stream of CEE events.
>Because we have made the <BaseProfile/> element both abstract and
>optional this is possible and is demonstrated in the test harness.
>
>The test harness is written in Java and uses JAXB to create bindings
>(classes) to the XSD elements.  Two sets of bindings were created for
>this test harness.  The first set of bindings live in the directory
>CEE-Test/src/com/mitre/cee.  These bindings (classes) are generated from
>the base CEE XSD (i.e. cee.xsd in com/mitre/cee/schemas).  The second
>set of bindings demonstrate how an application would extend the base CEE
>event and provide product specific data.  The product I selected for
>this example is netfilter.org's iptables and its bindings live in the
>directory CEE-Test/src/com/netfilter/cee.  This extension is described
>in CEE-Test/src/com/mitre/schemas/iptables.xsd.  Please review it and
>notice how easy it is to extend <BaseProfile/>.
>
>The harness is a small Java program
>(CEE-Test/src/com/mitre/cee/test/CEETest.java) that reads in a stream of
>CEE events which are described in
>CEE-Test/src/com/mitre/cee/samplexml/ceesample.xml.  The harness
>demonstrates that it is possible/easy to parse a CEE stream with product
>specific extensions without knowing them "a priori".  The harness
>demonstrates how an application can include free-form text via the
><SimpleMessageProfile/>.
>
>- Test Harness File Locations:
>The modified CEE XSD is here: CEE-Test/src/com/mitre/schemas/cee.xsd
>The netfilter.org extension of the CEE XSD is here:
>CEE-Test/src/com/mitre/schemas/iptables.xsd
>The base CEE bindings are here: CEE-Test/src/com/mitre/cee
>The netfilter/iptables CEE bindings are here: CEE-
>Test/src/com/netfilter/cee
>
>- Test Harness Usage:
>0. The attached zip file is an Eclipse project.  If you have Eclipse,
>you can simply import it. If you want to run the project you will need
>Java 1.6.
>1. Unzip CEE-Test.zip
>2. cwd into CEE-Test
>3.  java -classpath ./bin/ com.mitre.cee.test.CEETest
>
>
>[1] Extensions should live inside the <Event/> element:
>Sample flow:
>   [Emitter]--> <Event N/>... <Event 2/> ... <Event 1/> --> [Relay] -->
><Event N/>+<Extension for Event N/>... <Event 2/>+<Extension for Event
>2/> --> [Parser]
>Problem description:
>   The [Parser] will have the unenviable task of correlating
><Extension/>s to <Event/>s.  This could be completely eliminated if
>there [Relay] could augment the <Event/> element directly.  In fact, it
>might be really nasty if the<Event/>s were delivered to a server socked
>over UDP where the server is receiving multiple event streams at once.
>
>
>
>