Event Record Size Limits

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Event Record Size Limits

heinbockel
CEE is looking to recommend sizing constraints for event records to better
enable compatibility.

What are your thoughts on reasonable maximum limits for the following:
 +  Event record size (1MB? 10MB? 100KB?)
 +  Number of values for a single event field (e.g, a packet sent to 1000 IP
addresses) (128? 255?)
 +  Value (string) size (1KB? 10KB?)


Thank you,

William Heinbockel
Infosec Engineer, Sr.
The MITRE Corporation
202 Burlington Rd. MS S145
Bedford, MA 01730
[hidden email]
781-271-2615



smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Adam Montville
This may be a newb type of question, but are we constrained to IT audit
events or is there some wider applicability for CEE in the future?  I can
conceive a physical sensor sending image and/or video data as part of a
record, for example.  In that case, I could see the event record size
being large.  If we are constrained to IT audit events, I can't see
anything larger than 1MB being realistic (if we see something like that,
it's anomalous in its own right).

Also, it seems that the moment we truncate data, we're walking into the
realm of assumptions or at least guessing.  If a packet is sent to 1000 IP
addresses, but we only report 128 or even 255 of them, we don't make the
job of containment/remediation for issues any easier.  Of course, I could
be reading that question wrong.

Not sure about the last question.

Adam

On 11/22/10 2:08 PM, "Heinbockel, Bill" <[hidden email]> wrote:

>CEE is looking to recommend sizing constraints for event records to better
>enable compatibility.
>
>What are your thoughts on reasonable maximum limits for the following:
> +  Event record size (1MB? 10MB? 100KB?)
> +  Number of values for a single event field (e.g, a packet sent to 1000
>IP
>addresses) (128? 255?)
> +  Value (string) size (1KB? 10KB?)
>
>
>Thank you,
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
>[hidden email]
>781-271-2615
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Ruben Oliva
In reply to this post by heinbockel

Hello to all:
 
My role in the board is that of promoting the cause.  But I figure to throw in a cent to this issue.
 
If XML does not place a limit to the size of any data, would it be reasonable to specify that size of the log to be the same size as that allowed in XML?
 
David Oliva
ISSEP, CISSP, CAP, CISM, Security+, Network+

Nov 23, 2010 04:45:52 PM, [hidden email] wrote:
This may be a newb type of question, but are we constrained to IT audit
events or is there some wider applicability for CEE in the future? I can
conceive a physical sensor sending image and/or video data as part of a
record, for example. In that case, I could see the event record size
being large. If we are constrained to IT audit events, I can't see
anything larger than 1MB being realistic (if we see something like that,
it's anomalous in its own right).

Also, it seems that the moment we truncate data, we're walking into the
realm of assumptions or at least guessing. If a packet is sent to 1000 IP
addresses, but we only report 128 or even 255 of them, we don't make the
job of containment/remediation for issues any easier. Of course, I could
be reading that question wrong.

Not sure about the last question.

Adam

On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:

>CEE is looking to recommend sizing constraints for event records to better
>enable compatibility.
>
>What are your thoughts on reasonable maximum limits for the following:
> + Event record size (1MB? 10MB? 100KB?)
> + Number of values for a single event field (e.g, a packet sent to 1000
>IP
>addresses) (128? 255?)
> + Value (string) size (1KB? 10KB?)
>
>
>Thank you,
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
>[hidden email]
>781-271-2615
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Adam Montville
That's an interesting way to look at the issue.  Assuming that the requirement would be limitless size and the technology used to express the requirement (for now XML, but conceivably something else in the future), that could be OK.

What issues would this raise for potential implementations?  Parser attacks or anything along those lines? Is that something with which we should be concerned at this level (I would argue not).


-Adam

On Nov 23, 2010, at 3:57 PM, "[hidden email]<mailto:[hidden email]>" <[hidden email]<mailto:[hidden email]>> wrote:


Hello to all:

My role in the board is that of promoting the cause.  But I figure to throw in a cent to this issue.

If XML does not place a limit to the size of any data, would it be reasonable to specify that size of the log to be the same size as that allowed in XML?

David Oliva
ISSEP, CISSP, CAP, CISM, Security+, Network+

Nov 23, 2010 04:45:52 PM, [hidden email]<mailto:[hidden email]> wrote:
This may be a newb type of question, but are we constrained to IT audit
events or is there some wider applicability for CEE in the future? I can
conceive a physical sensor sending image and/or video data as part of a
record, for example. In that case, I could see the event record size
being large. If we are constrained to IT audit events, I can't see
anything larger than 1MB being realistic (if we see something like that,
it's anomalous in its own right).

Also, it seems that the moment we truncate data, we're walking into the
realm of assumptions or at least guessing. If a packet is sent to 1000 IP
addresses, but we only report 128 or even 255 of them, we don't make the
job of containment/remediation for issues any easier. Of course, I could
be reading that question wrong.

Not sure about the last question.

Adam

On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:

>CEE is looking to recommend sizing constraints for event records to better
>enable compatibility.
>
>What are your thoughts on reasonable maximum limits for the following:
> + Event record size (1MB? 10MB? 100KB?)
> + Number of values for a single event field (e.g, a packet sent to 1000
>IP
>addresses) (128? 255?)
> + Value (string) size (1KB? 10KB?)
>
>
>Thank you,
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
><mailto:[hidden email]>[hidden email]<mailto:[hidden email]>
>781-271-2615
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Forbes, R R
Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
Would the problem be at that point how do you properly store those items in a DB. If you start allowing fields to be unlimited size then you are into using CLOB and NCLOB datatypes for storing these large objects and that isn't always conducive to DB design or determing database sizing requirements


From: Adam Montville [mailto:[hidden email]]
Sent: Tue 11/23/2010 9:07 PM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

That's an interesting way to look at the issue.  Assuming that the requirement would be limitless size and the technology used to express the requirement (for now XML, but conceivably something else in the future), that could be OK.

What issues would this raise for potential implementations?  Parser attacks or anything along those lines? Is that something with which we should be concerned at this level (I would argue not).


-Adam

On Nov 23, 2010, at 3:57 PM, "[hidden email]<[hidden email]>" <[hidden email]<[hidden email]>> wrote:


Hello to all:

My role in the board is that of promoting the cause.  But I figure to throw in a cent to this issue.

If XML does not place a limit to the size of any data, would it be reasonable to specify that size of the log to be the same size as that allowed in XML?

David Oliva
ISSEP, CISSP, CAP, CISM, Security+, Network+

Nov 23, 2010 04:45:52 PM, [hidden email]<[hidden email]> wrote:
This may be a newb type of question, but are we constrained to IT audit
events or is there some wider applicability for CEE in the future? I can
conceive a physical sensor sending image and/or video data as part of a
record, for example. In that case, I could see the event record size
being large. If we are constrained to IT audit events, I can't see
anything larger than 1MB being realistic (if we see something like that,
it's anomalous in its own right).

Also, it seems that the moment we truncate data, we're walking into the
realm of assumptions or at least guessing. If a packet is sent to 1000 IP
addresses, but we only report 128 or even 255 of them, we don't make the
job of containment/remediation for issues any easier. Of course, I could
be reading that question wrong.

Not sure about the last question.

Adam

On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:


>CEE is looking to recommend sizing constraints for event records to better
>enable compatibility.
>
>What are your thoughts on reasonable maximum limits for the following:
> + Event record size (1MB? 10MB? 100KB?)
> + Number of values for a single event field (e.g, a packet sent to 1000
>IP
>addresses) (128? 255?)
> + Value (string) size (1KB? 10KB?)
>
>
>Thank you,
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
><[hidden email]>[hidden email]<[hidden email]>
>781-271-2615
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Adam Montville
If the architecture of a system is to in fact store data of that size in the DB, then yes that's a concern.  It seems that a better way to go would be to store off the event unscathed as it were and then split the large data out to somewhere other than the DB for analysis purposes.  But, yes, that seems like something to consider for sure.

But, I'm not sure that my original question was explicitly answered.  Is CEE really intended to capture any kind of event or simply "traditional" IT log events?  If the latter, then this discussion is probably academic anyway.

Adam

From: "Forbes, R R" <[hidden email]<mailto:[hidden email]>>
Date: Tue, 23 Nov 2010 19:34:20 -0800
To: Adam Montville <[hidden email]<mailto:[hidden email]>>, "[hidden email]<mailto:[hidden email]>" <[hidden email]<mailto:[hidden email]>>
Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits

Would the problem be at that point how do you properly store those items in a DB. If you start allowing fields to be unlimited size then you are into using CLOB and NCLOB datatypes for storing these large objects and that isn't always conducive to DB design or determing database sizing requirements

________________________________
From: Adam Montville [mailto:[hidden email]]
Sent: Tue 11/23/2010 9:07 PM
To: [hidden email]<mailto:[hidden email]>
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits


That's an interesting way to look at the issue.  Assuming that the requirement would be limitless size and the technology used to express the requirement (for now XML, but conceivably something else in the future), that could be OK.

What issues would this raise for potential implementations?  Parser attacks or anything along those lines? Is that something with which we should be concerned at this level (I would argue not).


-Adam

On Nov 23, 2010, at 3:57 PM, "[hidden email]<mailto:[hidden email]><mailto:[hidden email]>" <[hidden email]<mailto:[hidden email]><mailto:[hidden email]>> wrote:


Hello to all:

My role in the board is that of promoting the cause.  But I figure to throw in a cent to this issue.

If XML does not place a limit to the size of any data, would it be reasonable to specify that size of the log to be the same size as that allowed in XML?

David Oliva
ISSEP, CISSP, CAP, CISM, Security+, Network+

Nov 23, 2010 04:45:52 PM, [hidden email]<mailto:[hidden email]><mailto:[hidden email]> wrote:
This may be a newb type of question, but are we constrained to IT audit
events or is there some wider applicability for CEE in the future? I can
conceive a physical sensor sending image and/or video data as part of a
record, for example. In that case, I could see the event record size
being large. If we are constrained to IT audit events, I can't see
anything larger than 1MB being realistic (if we see something like that,
it's anomalous in its own right).

Also, it seems that the moment we truncate data, we're walking into the
realm of assumptions or at least guessing. If a packet is sent to 1000 IP
addresses, but we only report 128 or even 255 of them, we don't make the
job of containment/remediation for issues any easier. Of course, I could
be reading that question wrong.

Not sure about the last question.

Adam

On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:

>CEE is looking to recommend sizing constraints for event records to better
>enable compatibility.
>
>What are your thoughts on reasonable maximum limits for the following:
> + Event record size (1MB? 10MB? 100KB?)
> + Number of values for a single event field (e.g, a packet sent to 1000
>IP
>addresses) (128? 255?)
> + Value (string) size (1KB? 10KB?)
>
>
>Thank you,
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
><mailto:[hidden email]>[hidden email]<mailto:[hidden email]><mailto:[hidden email]>
>781-271-2615
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Forbes, R R
Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
That's a good thought on dealing with large events. It would be easy enough to generate logic to store events that were larger than a certain size in a seperate tablespace thus elminating the performance impact when dealing with more traditionally sized "IT Events"
 
Robert


From: Adam Montville [mailto:[hidden email]]
Sent: Tue 11/23/2010 10:40 PM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

If the architecture of a system is to in fact store data of that size in the DB, then yes that's a concern.  It seems that a better way to go would be to store off the event unscathed as it were and then split the large data out to somewhere other than the DB for analysis purposes.  But, yes, that seems like something to consider for sure.

But, I'm not sure that my original question was explicitly answered.  Is CEE really intended to capture any kind of event or simply "traditional" IT log events?  If the latter, then this discussion is probably academic anyway.

Adam

From: "Forbes, R R" <[hidden email]<[hidden email]>>
Date: Tue, 23 Nov 2010 19:34:20 -0800
To: Adam Montville <[hidden email]<[hidden email]>>, "[hidden email]<[hidden email]>" <[hidden email]<[hidden email]>>
Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits

Would the problem be at that point how do you properly store those items in a DB. If you start allowing fields to be unlimited size then you are into using CLOB and NCLOB datatypes for storing these large objects and that isn't always conducive to DB design or determing database sizing requirements

________________________________
From: Adam Montville [[hidden email]]
Sent: Tue 11/23/2010 9:07 PM
To: [hidden email]<[hidden email]>
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits


That's an interesting way to look at the issue.  Assuming that the requirement would be limitless size and the technology used to express the requirement (for now XML, but conceivably something else in the future), that could be OK.

What issues would this raise for potential implementations?  Parser attacks or anything along those lines? Is that something with which we should be concerned at this level (I would argue not).


-Adam

On Nov 23, 2010, at 3:57 PM, "[hidden email]<[hidden email]><[hidden email]>" <[hidden email]<[hidden email]><[hidden email]>> wrote:


Hello to all:

My role in the board is that of promoting the cause.  But I figure to throw in a cent to this issue.

If XML does not place a limit to the size of any data, would it be reasonable to specify that size of the log to be the same size as that allowed in XML?

David Oliva
ISSEP, CISSP, CAP, CISM, Security+, Network+

Nov 23, 2010 04:45:52 PM, [hidden email]<[hidden email]><[hidden email]> wrote:
This may be a newb type of question, but are we constrained to IT audit
events or is there some wider applicability for CEE in the future? I can
conceive a physical sensor sending image and/or video data as part of a
record, for example. In that case, I could see the event record size
being large. If we are constrained to IT audit events, I can't see
anything larger than 1MB being realistic (if we see something like that,
it's anomalous in its own right).

Also, it seems that the moment we truncate data, we're walking into the
realm of assumptions or at least guessing. If a packet is sent to 1000 IP
addresses, but we only report 128 or even 255 of them, we don't make the
job of containment/remediation for issues any easier. Of course, I could
be reading that question wrong.

Not sure about the last question.

Adam

On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:


>CEE is looking to recommend sizing constraints for event records to better
>enable compatibility.
>
>What are your thoughts on reasonable maximum limits for the following:
> + Event record size (1MB? 10MB? 100KB?)
> + Number of values for a single event field (e.g, a packet sent to 1000
>IP
>addresses) (128? 255?)
> + Value (string) size (1KB? 10KB?)
>
>
>Thank you,
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
><[hidden email]>[hidden email]<[hidden email]><[hidden email]>
>781-271-2615
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Mark Poepping
In reply to this post by Adam Montville
Seems to me that CEE (with the OAS approach) is strongly biased to current
IT-based logs, and I think it's worth that focus given the problems there.  

I would generalize your question to: is this about "IT log events" or
"situational awareness" (*way more general*)?  I talked with Chuvakin about
this three years ago, and my take at the time was that this effort was
mostly about normalizing firewall and syslog information, since that's 95%
of the market and more importantly, it's 98% of the *understanding of
current practitioners*.  In that, I think there is good value in continuing
to support this effort, and it's probably important to *not* make it too
general, because that might jeopardize the possibility of improving stuff
we've already decided we need to do (my opinion).

Of course, I find the general question of situational awareness far more
interesting.  I think it will be increasingly important to capture and
incorporate video, network traffic flows, component health, physical sensor
information, application/service transaction logs, "you-name-it" into the
diagnostic management framework, but that's the stuff we're not so good at
and we generally don't understand the value or approach to bringing it
together.  The only thing I can tell you is that we're certainly *not* going
to mine it all from a database:-).

At present, many business sectors are experimenting with data from their own
sensors and building analytics for the information they have, that's better
than nothing, but they rarely share analytics and almost never cross that
data with information from related infrastructures (e.g. when my motion
sensor is unreachable, did it die, did the network connectivity to it go
away, or did the service that's supposed to ping it go postal?)..  I think
you should understand a problem before you standardize and that's why we try
to assist local researchers with experimental analysis into this kind of
thing.  At present however, as I said, 98% of the understanding (even here)
doesn't yet *see* this level of correlation in their situational awareness
picture - they're still trying to get data from their own stuff...

Mark.


>-----Original Message-----
>From: Adam Montville [mailto:[hidden email]]
>Sent: Tuesday, November 23, 2010 10:40 PM
>To: [hidden email]
>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>If the architecture of a system is to in fact store data of that size in
the
>DB, then yes that's a concern.  It seems that a better way to go would be
to
>store off the event unscathed as it were and then split the large data out
to
>somewhere other than the DB for analysis purposes.  But, yes, that seems
like
>something to consider for sure.
>
>But, I'm not sure that my original question was explicitly answered.  Is
CEE
>really intended to capture any kind of event or simply "traditional" IT log
>events?  If the latter, then this discussion is probably academic anyway.
>
>Adam
>
>From: "Forbes, R R" <[hidden email]<mailto:[hidden email]>>
>Date: Tue, 23 Nov 2010 19:34:20 -0800
>To: Adam Montville
<[hidden email]<mailto:[hidden email]>>,
>"[hidden email]<mailto:CEE-DISCUSSION-
>[hidden email]>" <[hidden email]<mailto:CEE-
>[hidden email]>>
>Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>Would the problem be at that point how do you properly store those items in
a
>DB. If you start allowing fields to be unlimited size then you are into
using
>CLOB and NCLOB datatypes for storing these large objects and that isn't
always

>conducive to DB design or determing database sizing requirements
>
>________________________________
>From: Adam Montville [mailto:[hidden email]]
>Sent: Tue 11/23/2010 9:07 PM
>To: [hidden email]<mailto:CEE-DISCUSSION-
>[hidden email]>
>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>
>That's an interesting way to look at the issue.  Assuming that the
requirement
>would be limitless size and the technology used to express the requirement
(for
>now XML, but conceivably something else in the future), that could be OK.
>
>What issues would this raise for potential implementations?  Parser attacks
or
>anything along those lines? Is that something with which we should be
concerned
>at this level (I would argue not).
>
>
>-Adam
>
>On Nov 23, 2010, at 3:57 PM,
>"[hidden email]<mailto:[hidden email]><mailto:david.oliva
@ver
>izon.net>"
><[hidden email]<mailto:[hidden email]><mailto:david.oliva
@ver
>izon.net>> wrote:
>
>
>Hello to all:
>
>My role in the board is that of promoting the cause.  But I figure to throw
in
>a cent to this issue.
>
>If XML does not place a limit to the size of any data, would it be
reasonable
>to specify that size of the log to be the same size as that allowed in XML?
>
>David Oliva
>ISSEP, CISSP, CAP, CISM, Security+, Network+
>
>Nov 23, 2010 04:45:52 PM,
>[hidden email]<mailto:[hidden email]><mailto:amontville@T
RIPW

>IRE.COM> wrote:
>This may be a newb type of question, but are we constrained to IT audit
>events or is there some wider applicability for CEE in the future? I can
>conceive a physical sensor sending image and/or video data as part of a
>record, for example. In that case, I could see the event record size
>being large. If we are constrained to IT audit events, I can't see
>anything larger than 1MB being realistic (if we see something like that,
>it's anomalous in its own right).
>
>Also, it seems that the moment we truncate data, we're walking into the
>realm of assumptions or at least guessing. If a packet is sent to 1000 IP
>addresses, but we only report 128 or even 255 of them, we don't make the
>job of containment/remediation for issues any easier. Of course, I could
>be reading that question wrong.
>
>Not sure about the last question.
>
>Adam
>
>On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:
>
>>CEE is looking to recommend sizing constraints for event records to better
>>enable compatibility.
>>
>>What are your thoughts on reasonable maximum limits for the following:
>> + Event record size (1MB? 10MB? 100KB?)
>> + Number of values for a single event field (e.g, a packet sent to 1000
>>IP
>>addresses) (128? 255?)
>> + Value (string) size (1KB? 10KB?)
>>
>>
>>Thank you,
>>
>>William Heinbockel
>>Infosec Engineer, Sr.
>>The MITRE Corporation
>>202 Burlington Rd. MS S145
>>Bedford, MA 01730
>><mailto:[hidden email]>[hidden email]<mailto:heinbockel@mitre.
org>
><mailto:[hidden email]>
>>781-271-2615
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Chris Blask
In reply to this post by Forbes, R R
Hi folks,

I second that emotion.

Future size forecasts are notoriously incorrect.  Rich content of god-knows what size is almost inevitable and should probably be encouraged. We should want to be able to say: "this card-swipe related to this video coverage related to this workstation login related to this anomalous network behavior are worth knowing about".

As I've thought about the collision of IT and CIP the shape of the solution has always seemed to include a "pointer to (video content, etc)" in messaging streams. Obviously you can't have a 100MB video capture in every (or any) syslog message, so the future solution has to include some sort of pointer-to for related content. This would seem to be a critical part of any messaging standard.

-cheers

-chris



On 23 November 2010 19:59, Forbes, R R <[hidden email]> wrote:
That's a good thought on dealing with large events. It would be easy enough to generate logic to store events that were larger than a certain size in a seperate tablespace thus elminating the performance impact when dealing with more traditionally sized "IT Events"
 
Robert


From: Adam Montville [mailto:[hidden email]]
Sent: Tue 11/23/2010 10:40 PM
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

If the architecture of a system is to in fact store data of that size in the DB, then yes that's a concern.  It seems that a better way to go would be to store off the event unscathed as it were and then split the large data out to somewhere other than the DB for analysis purposes.  But, yes, that seems like something to consider for sure.

But, I'm not sure that my original question was explicitly answered.  Is CEE really intended to capture any kind of event or simply "traditional" IT log events?  If the latter, then this discussion is probably academic anyway.

Adam

From: "Forbes, R R" <[hidden email]<[hidden email]>>
Date: Tue, 23 Nov 2010 19:34:20 -0800
To: Adam Montville <[hidden email]<[hidden email]>>, "[hidden email]<[hidden email]>" <[hidden email]<[hidden email]>>
Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits

Would the problem be at that point how do you properly store those items in a DB. If you start allowing fields to be unlimited size then you are into using CLOB and NCLOB datatypes for storing these large objects and that isn't always conducive to DB design or determing database sizing requirements

________________________________
From: Adam Montville [[hidden email]]
Sent: Tue 11/23/2010 9:07 PM
To: [hidden email]<[hidden email]>
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits


That's an interesting way to look at the issue.  Assuming that the requirement would be limitless size and the technology used to express the requirement (for now XML, but conceivably something else in the future), that could be OK.

What issues would this raise for potential implementations?  Parser attacks or anything along those lines? Is that something with which we should be concerned at this level (I would argue not).


-Adam

On Nov 23, 2010, at 3:57 PM, "[hidden email]<[hidden email]><[hidden email]>" <[hidden email]<[hidden email]><[hidden email]>> wrote:


Hello to all:

My role in the board is that of promoting the cause.  But I figure to throw in a cent to this issue.

If XML does not place a limit to the size of any data, would it be reasonable to specify that size of the log to be the same size as that allowed in XML?

David Oliva
ISSEP, CISSP, CAP, CISM, Security+, Network+

Nov 23, 2010 04:45:52 PM, [hidden email]<[hidden email]><[hidden email]> wrote:
This may be a newb type of question, but are we constrained to IT audit
events or is there some wider applicability for CEE in the future? I can
conceive a physical sensor sending image and/or video data as part of a
record, for example. In that case, I could see the event record size
being large. If we are constrained to IT audit events, I can't see
anything larger than 1MB being realistic (if we see something like that,
it's anomalous in its own right).

Also, it seems that the moment we truncate data, we're walking into the
realm of assumptions or at least guessing. If a packet is sent to 1000 IP
addresses, but we only report 128 or even 255 of them, we don't make the
job of containment/remediation for issues any easier. Of course, I could
be reading that question wrong.

Not sure about the last question.

Adam

On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:

>CEE is looking to recommend sizing constraints for event records to better
>enable compatibility.
>
>What are your thoughts on reasonable maximum limits for the following:
> + Event record size (1MB? 10MB? 100KB?)
> + Number of values for a single event field (e.g, a packet sent to 1000
>IP
>addresses) (128? 255?)
> + Value (string) size (1KB? 10KB?)
>
>
>Thank you,
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
><[hidden email]>[hidden email]<[hidden email]><[hidden email]>
>781-271-2615
>
>




--
Chris Blask
VP Marketing
AlienVault

1901 S Bascom Avenue
Suite 220
Campbell, CA, 95008
[hidden email]
+1 408 465-9989
Skype: chrisblask

Author: Security Information and Event Management (SIEM) Implementation
McGraw Hill  ISBN-10: 0071701095 ISBN-13: 978-0071701099

Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Gabrielson, Bruce C.
In reply to this post by Adam Montville
Adam

The system I have been working on captures many kinds of data elements,
not all traditional log events.  Where possible we have normalized to
existing CEE data elements and for those not in the current dictionary
we have defined new elements.

Bruce

-----Original Message-----
From: Adam Montville [mailto:[hidden email]]
Sent: Tuesday, November 23, 2010 10:40 PM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

If the architecture of a system is to in fact store data of that size in
the DB, then yes that's a concern.  It seems that a better way to go
would be to store off the event unscathed as it were and then split the
large data out to somewhere other than the DB for analysis purposes.
But, yes, that seems like something to consider for sure.

But, I'm not sure that my original question was explicitly answered.  Is
CEE really intended to capture any kind of event or simply "traditional"
IT log events?  If the latter, then this discussion is probably academic
anyway.

Adam

From: "Forbes, R R" <[hidden email]<mailto:[hidden email]>>
Date: Tue, 23 Nov 2010 19:34:20 -0800
To: Adam Montville
<[hidden email]<mailto:[hidden email]>>,
"[hidden email]<mailto:[hidden email]
TRE.ORG>"
<[hidden email]<mailto:[hidden email]
TRE.ORG>>
Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits

Would the problem be at that point how do you properly store those items
in a DB. If you start allowing fields to be unlimited size then you are
into using CLOB and NCLOB datatypes for storing these large objects and
that isn't always conducive to DB design or determing database sizing
requirements

________________________________
From: Adam Montville [mailto:[hidden email]]
Sent: Tue 11/23/2010 9:07 PM
To:
[hidden email]<mailto:[hidden email]
RE.ORG>
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits


That's an interesting way to look at the issue.  Assuming that the
requirement would be limitless size and the technology used to express
the requirement (for now XML, but conceivably something else in the
future), that could be OK.

What issues would this raise for potential implementations?  Parser
attacks or anything along those lines? Is that something with which we
should be concerned at this level (I would argue not).


-Adam

On Nov 23, 2010, at 3:57 PM,
"[hidden email]<mailto:[hidden email]><mailto:david.ol
[hidden email]>"
<[hidden email]<mailto:[hidden email]><mailto:david.ol
[hidden email]>> wrote:


Hello to all:

My role in the board is that of promoting the cause.  But I figure to
throw in a cent to this issue.

If XML does not place a limit to the size of any data, would it be
reasonable to specify that size of the log to be the same size as that
allowed in XML?

David Oliva
ISSEP, CISSP, CAP, CISM, Security+, Network+

Nov 23, 2010 04:45:52 PM,
[hidden email]<mailto:[hidden email]><mailto:amontvill
[hidden email]> wrote:
This may be a newb type of question, but are we constrained to IT audit
events or is there some wider applicability for CEE in the future? I can
conceive a physical sensor sending image and/or video data as part of a
record, for example. In that case, I could see the event record size
being large. If we are constrained to IT audit events, I can't see
anything larger than 1MB being realistic (if we see something like that,
it's anomalous in its own right).

Also, it seems that the moment we truncate data, we're walking into the
realm of assumptions or at least guessing. If a packet is sent to 1000
IP
addresses, but we only report 128 or even 255 of them, we don't make the
job of containment/remediation for issues any easier. Of course, I could
be reading that question wrong.

Not sure about the last question.

Adam

On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:

>CEE is looking to recommend sizing constraints for event records to
better
>enable compatibility.
>
>What are your thoughts on reasonable maximum limits for the following:
> + Event record size (1MB? 10MB? 100KB?)
> + Number of values for a single event field (e.g, a packet sent to
1000

>IP
>addresses) (128? 255?)
> + Value (string) size (1KB? 10KB?)
>
>
>Thank you,
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
><mailto:[hidden email]>[hidden email]<mailto:heinbockel@mit
re.org><mailto:[hidden email]>
>781-271-2615
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Rainer Gerhards
In reply to this post by Mark Poepping
On 11/24/2010 06:24 AM, Mark Poepping wrote:

> Seems to me that CEE (with the OAS approach) is strongly biased to current
> IT-based logs, and I think it's worth that focus given the problems there.
>
> I would generalize your question to: is this about "IT log events" or
> "situational awareness" (*way more general*)?  I talked with Chuvakin about
> this three years ago, and my take at the time was that this effort was
> mostly about normalizing firewall and syslog information, since that's 95%
> of the market and more importantly, it's 98% of the *understanding of
> current practitioners*.  In that, I think there is good value in continuing
> to support this effort, and it's probably important to *not* make it too
> general, because that might jeopardize the possibility of improving stuff
> we've already decided we need to do (my opinion).

IMHO the current effort is about "IT log events". The effort was created
due to the problem of the many ways to say essentially the same thing in
IT logs and the many problems that result from this fact.

So I would strongly opt to keep at least the initial version focused on
that topic. HOWEVER, I, too, find situational awareness more interesting
and useful as an ultimate goal. But it probably is not yet mature enough
(as you say below) to be incorporated into the current CEE effort.

To me, the best compromise is to try to make sure CEE is sufficiently
versioned and extensible, so that in the future, we can add features
(and lift [size] restrictions) as need arises. But this must happen in a
well-defined way and without breaking applications obeying to older CEE
versions.

Rainer

> Of course, I find the general question of situational awareness far more
> interesting.  I think it will be increasingly important to capture and
> incorporate video, network traffic flows, component health, physical sensor
> information, application/service transaction logs, "you-name-it" into the
> diagnostic management framework, but that's the stuff we're not so good at
> and we generally don't understand the value or approach to bringing it
> together.  The only thing I can tell you is that we're certainly *not* going
> to mine it all from a database:-).
>
> At present, many business sectors are experimenting with data from their own
> sensors and building analytics for the information they have, that's better
> than nothing, but they rarely share analytics and almost never cross that
> data with information from related infrastructures (e.g. when my motion
> sensor is unreachable, did it die, did the network connectivity to it go
> away, or did the service that's supposed to ping it go postal?)..  I think
> you should understand a problem before you standardize and that's why we try
> to assist local researchers with experimental analysis into this kind of
> thing.  At present however, as I said, 98% of the understanding (even here)
> doesn't yet *see* this level of correlation in their situational awareness
> picture - they're still trying to get data from their own stuff...
>
> Mark.
>
>
>> -----Original Message-----
>> From: Adam Montville [mailto:[hidden email]]
>> Sent: Tuesday, November 23, 2010 10:40 PM
>> To: [hidden email]
>> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>> If the architecture of a system is to in fact store data of that size in
> the
>> DB, then yes that's a concern.  It seems that a better way to go would be
> to
>> store off the event unscathed as it were and then split the large data out
> to
>> somewhere other than the DB for analysis purposes.  But, yes, that seems
> like
>> something to consider for sure.
>>
>> But, I'm not sure that my original question was explicitly answered.  Is
> CEE
>> really intended to capture any kind of event or simply "traditional" IT log
>> events?  If the latter, then this discussion is probably academic anyway.
>>
>> Adam
>>
>> From: "Forbes, R R"<[hidden email]<mailto:[hidden email]>>
>> Date: Tue, 23 Nov 2010 19:34:20 -0800
>> To: Adam Montville
> <[hidden email]<mailto:[hidden email]>>,
>> "[hidden email]<mailto:CEE-DISCUSSION-
>> [hidden email]>"<[hidden email]<mailto:CEE-
>> [hidden email]>>
>> Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>> Would the problem be at that point how do you properly store those items in
> a
>> DB. If you start allowing fields to be unlimited size then you are into
> using
>> CLOB and NCLOB datatypes for storing these large objects and that isn't
> always
>> conducive to DB design or determing database sizing requirements
>>
>> ________________________________
>> From: Adam Montville [mailto:[hidden email]]
>> Sent: Tue 11/23/2010 9:07 PM
>> To: [hidden email]<mailto:CEE-DISCUSSION-
>> [hidden email]>
>> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>
>> That's an interesting way to look at the issue.  Assuming that the
> requirement
>> would be limitless size and the technology used to express the requirement
> (for
>> now XML, but conceivably something else in the future), that could be OK.
>>
>> What issues would this raise for potential implementations?  Parser attacks
> or
>> anything along those lines? Is that something with which we should be
> concerned
>> at this level (I would argue not).
>>
>>
>> -Adam
>>
>> On Nov 23, 2010, at 3:57 PM,
>> "[hidden email]<mailto:[hidden email]><mailto:david.oliva
> @ver
>> izon.net>"
>> <[hidden email]<mailto:[hidden email]><mailto:david.oliva
> @ver
>> izon.net>>  wrote:
>>
>>
>> Hello to all:
>>
>> My role in the board is that of promoting the cause.  But I figure to throw
> in
>> a cent to this issue.
>>
>> If XML does not place a limit to the size of any data, would it be
> reasonable
>> to specify that size of the log to be the same size as that allowed in XML?
>>
>> David Oliva
>> ISSEP, CISSP, CAP, CISM, Security+, Network+
>>
>> Nov 23, 2010 04:45:52 PM,
>> [hidden email]<mailto:[hidden email]><mailto:amontville@T
> RIPW
>> IRE.COM>  wrote:
>> This may be a newb type of question, but are we constrained to IT audit
>> events or is there some wider applicability for CEE in the future? I can
>> conceive a physical sensor sending image and/or video data as part of a
>> record, for example. In that case, I could see the event record size
>> being large. If we are constrained to IT audit events, I can't see
>> anything larger than 1MB being realistic (if we see something like that,
>> it's anomalous in its own right).
>>
>> Also, it seems that the moment we truncate data, we're walking into the
>> realm of assumptions or at least guessing. If a packet is sent to 1000 IP
>> addresses, but we only report 128 or even 255 of them, we don't make the
>> job of containment/remediation for issues any easier. Of course, I could
>> be reading that question wrong.
>>
>> Not sure about the last question.
>>
>> Adam
>>
>> On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:
>>
>>> CEE is looking to recommend sizing constraints for event records to better
>>> enable compatibility.
>>>
>>> What are your thoughts on reasonable maximum limits for the following:
>>> + Event record size (1MB? 10MB? 100KB?)
>>> + Number of values for a single event field (e.g, a packet sent to 1000
>>> IP
>>> addresses) (128? 255?)
>>> + Value (string) size (1KB? 10KB?)
>>>
>>>
>>> Thank you,
>>>
>>> William Heinbockel
>>> Infosec Engineer, Sr.
>>> The MITRE Corporation
>>> 202 Burlington Rd. MS S145
>>> Bedford, MA 01730
>>> <mailto:[hidden email]>[hidden email]<mailto:heinbockel@mitre.
> org>
>> <mailto:[hidden email]>
>>> 781-271-2615
>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Adam Montville
This is a very useful discussion from my perspective.  I completely
appreciate the sentiment that we should focus on IT log events, but I also
believe that where we are able we should refrain from over-specification
in order to accommodate the unknown in the future (I.e. What situational
awareness will look like).

CEE seems set up for this already in that the specification itself defines
the framework in which OAS has been defined.  As Bruce pointed out
previously on this thread, they extended the OAS model.  In fact, this is
what we are doing as well.

From what I have been reading, it seems that we all agree that CEE should
be focused on defining a framework that is initially suited toward IT log
events, but not specified to the point where it will choke the larger
future effort of handling events related to situational awareness.

Does that sound about right?

Adam

On 11/24/10 6:08 AM, "Rainer Gerhards" <[hidden email]> wrote:

>On 11/24/2010 06:24 AM, Mark Poepping wrote:
>> Seems to me that CEE (with the OAS approach) is strongly biased to
>>current
>> IT-based logs, and I think it's worth that focus given the problems
>>there.
>>
>> I would generalize your question to: is this about "IT log events" or
>> "situational awareness" (*way more general*)?  I talked with Chuvakin
>>about
>> this three years ago, and my take at the time was that this effort was
>> mostly about normalizing firewall and syslog information, since that's
>>95%
>> of the market and more importantly, it's 98% of the *understanding of
>> current practitioners*.  In that, I think there is good value in
>>continuing
>> to support this effort, and it's probably important to *not* make it too
>> general, because that might jeopardize the possibility of improving
>>stuff
>> we've already decided we need to do (my opinion).
>
>IMHO the current effort is about "IT log events". The effort was created
>due to the problem of the many ways to say essentially the same thing in
>IT logs and the many problems that result from this fact.
>
>So I would strongly opt to keep at least the initial version focused on
>that topic. HOWEVER, I, too, find situational awareness more interesting
>and useful as an ultimate goal. But it probably is not yet mature enough
>(as you say below) to be incorporated into the current CEE effort.
>
>To me, the best compromise is to try to make sure CEE is sufficiently
>versioned and extensible, so that in the future, we can add features
>(and lift [size] restrictions) as need arises. But this must happen in a
>well-defined way and without breaking applications obeying to older CEE
>versions.
>
>Rainer
>
>> Of course, I find the general question of situational awareness far more
>> interesting.  I think it will be increasingly important to capture and
>> incorporate video, network traffic flows, component health, physical
>>sensor
>> information, application/service transaction logs, "you-name-it" into
>>the
>> diagnostic management framework, but that's the stuff we're not so good
>>at
>> and we generally don't understand the value or approach to bringing it
>> together.  The only thing I can tell you is that we're certainly *not*
>>going
>> to mine it all from a database:-).
>>
>> At present, many business sectors are experimenting with data from
>>their own
>> sensors and building analytics for the information they have, that's
>>better
>> than nothing, but they rarely share analytics and almost never cross
>>that
>> data with information from related infrastructures (e.g. when my motion
>> sensor is unreachable, did it die, did the network connectivity to it go
>> away, or did the service that's supposed to ping it go postal?)..  I
>>think
>> you should understand a problem before you standardize and that's why
>>we try
>> to assist local researchers with experimental analysis into this kind of
>> thing.  At present however, as I said, 98% of the understanding (even
>>here)
>> doesn't yet *see* this level of correlation in their situational
>>awareness
>> picture - they're still trying to get data from their own stuff...
>>
>> Mark.
>>
>>
>>> -----Original Message-----
>>> From: Adam Montville [mailto:[hidden email]]
>>> Sent: Tuesday, November 23, 2010 10:40 PM
>>> To: [hidden email]
>>> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>>
>>> If the architecture of a system is to in fact store data of that size
>>>in
>> the
>>> DB, then yes that's a concern.  It seems that a better way to go would
>>>be
>> to
>>> store off the event unscathed as it were and then split the large data
>>>out
>> to
>>> somewhere other than the DB for analysis purposes.  But, yes, that
>>>seems
>> like
>>> something to consider for sure.
>>>
>>> But, I'm not sure that my original question was explicitly answered.
>>>Is
>> CEE
>>> really intended to capture any kind of event or simply "traditional"
>>>IT log
>>> events?  If the latter, then this discussion is probably academic
>>>anyway.
>>>
>>> Adam
>>>
>>> From: "Forbes, R R"<[hidden email]<mailto:[hidden email]>>
>>> Date: Tue, 23 Nov 2010 19:34:20 -0800
>>> To: Adam Montville
>> <[hidden email]<mailto:[hidden email]>>,
>>> "[hidden email]<mailto:CEE-DISCUSSION-
>>> [hidden email]>"<[hidden email]<mailto:CEE-
>>> [hidden email]>>
>>> Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>>
>>> Would the problem be at that point how do you properly store those
>>>items in
>> a
>>> DB. If you start allowing fields to be unlimited size then you are into
>> using
>>> CLOB and NCLOB datatypes for storing these large objects and that isn't
>> always
>>> conducive to DB design or determing database sizing requirements
>>>
>>> ________________________________
>>> From: Adam Montville [mailto:[hidden email]]
>>> Sent: Tue 11/23/2010 9:07 PM
>>> To: [hidden email]<mailto:CEE-DISCUSSION-
>>> [hidden email]>
>>> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>>
>>>
>>> That's an interesting way to look at the issue.  Assuming that the
>> requirement
>>> would be limitless size and the technology used to express the
>>>requirement
>> (for
>>> now XML, but conceivably something else in the future), that could be
>>>OK.
>>>
>>> What issues would this raise for potential implementations?  Parser
>>>attacks
>> or
>>> anything along those lines? Is that something with which we should be
>> concerned
>>> at this level (I would argue not).
>>>
>>>
>>> -Adam
>>>
>>> On Nov 23, 2010, at 3:57 PM,
>>>
>>>"[hidden email]<mailto:[hidden email]><mailto:david.ol
>>>iva
>> @ver
>>> izon.net>"
>>>
>>><[hidden email]<mailto:[hidden email]><mailto:david.ol
>>>iva
>> @ver
>>> izon.net>>  wrote:
>>>
>>>
>>> Hello to all:
>>>
>>> My role in the board is that of promoting the cause.  But I figure to
>>>throw
>> in
>>> a cent to this issue.
>>>
>>> If XML does not place a limit to the size of any data, would it be
>> reasonable
>>> to specify that size of the log to be the same size as that allowed in
>>>XML?
>>>
>>> David Oliva
>>> ISSEP, CISSP, CAP, CISM, Security+, Network+
>>>
>>> Nov 23, 2010 04:45:52 PM,
>>>
>>>[hidden email]<mailto:[hidden email]><mailto:amontvill
>>>e@T
>> RIPW
>>> IRE.COM>  wrote:
>>> This may be a newb type of question, but are we constrained to IT audit
>>> events or is there some wider applicability for CEE in the future? I
>>>can
>>> conceive a physical sensor sending image and/or video data as part of a
>>> record, for example. In that case, I could see the event record size
>>> being large. If we are constrained to IT audit events, I can't see
>>> anything larger than 1MB being realistic (if we see something like
>>>that,
>>> it's anomalous in its own right).
>>>
>>> Also, it seems that the moment we truncate data, we're walking into the
>>> realm of assumptions or at least guessing. If a packet is sent to 1000
>>>IP
>>> addresses, but we only report 128 or even 255 of them, we don't make
>>>the
>>> job of containment/remediation for issues any easier. Of course, I
>>>could
>>> be reading that question wrong.
>>>
>>> Not sure about the last question.
>>>
>>> Adam
>>>
>>> On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:
>>>
>>>> CEE is looking to recommend sizing constraints for event records to
>>>>better
>>>> enable compatibility.
>>>>
>>>> What are your thoughts on reasonable maximum limits for the following:
>>>> + Event record size (1MB? 10MB? 100KB?)
>>>> + Number of values for a single event field (e.g, a packet sent to
>>>>1000
>>>> IP
>>>> addresses) (128? 255?)
>>>> + Value (string) size (1KB? 10KB?)
>>>>
>>>>
>>>> Thank you,
>>>>
>>>> William Heinbockel
>>>> Infosec Engineer, Sr.
>>>> The MITRE Corporation
>>>> 202 Burlington Rd. MS S145
>>>> Bedford, MA 01730
>>>>
>>>><mailto:[hidden email]>[hidden email]<mailto:heinbockel@mit
>>>>re.
>> org>
>>> <mailto:[hidden email]>
>>>> 781-271-2615
>>>>
>>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

David Corlette
In reply to this post by Chris Blask
FWIW, the XDAS v1 specification included a specific field that was supposed to be a URL pointer back to the original event source as a place to get more information (detail) about the described activity.  We expect XDAS v2 will do something similar.

- This obviously imposes a requirement on the event source to embed some sort of request engine that can respond to queries for that data - this could be HTTP, like RESTful interfaces, or it could be any other supported protocol (JDBC, whatever).

- I think in general it seems pretty silly to me to embed large data files in event data, since the point of the event to some extent is to summarize the activity that took place. If the event consumer wants more detail, it can and should go look it up.

- As examples of the types of data that could be referenced for more information, think images, raw packet captures, videos, live feeds....  This last one is most interesting, imagine you receive an event that says, essentially "someone is trying to break into your ATM" and you could click on a link to get a live video feed from the ATM's camera.

In other words, think of the event as a summary, and add metadata that will allow you to mash it up with data from other sources, rather than trying to shove everything under the sun into the event schema. This is what SIEMs do; they add contextual metadata to a baseline event schema (well, usually).

My 2c, HTH.

>>> On 24.11.2010 at 00:35, in message
<AANLkTi=[hidden email]>, Chris Blask
<[hidden email]> wrote:

> Hi folks,
>
> I second that emotion.
>
> Future size forecasts are notoriously incorrect.  Rich content of god-knows
> what size is almost inevitable and should probably be encouraged. We should
> want to be able to say: "this card-swipe related to this video coverage
> related to this workstation login related to this anomalous network behavior
> are worth knowing about".
>
> As I've thought about the collision of IT and CIP the shape of the solution
> has always seemed to include a "pointer to (video content, etc)" in
> messaging streams. Obviously you can't have a 100MB video capture in every
> (or any) syslog message, so the future solution has to include some sort of
> pointer-to for related content. This would seem to be a critical part of any
> messaging standard.
>
> -cheers
>
> -chris
>
>
>
> On 23 November 2010 19:59, Forbes, R R <[hidden email]> wrote:
>
>>  That's a good thought on dealing with large events. It would be easy
>> enough to generate logic to store events that were larger than a certain
>> size in a seperate tablespace thus elminating the performance impact when
>> dealing with more traditionally sized "IT Events"
>>
>> Robert
>>
>> ------------------------------
>> *From:* Adam Montville [mailto:[hidden email]]
>> *Sent:* Tue 11/23/2010 10:40 PM
>>
>> *To:* [hidden email]
>> *Subject:* Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>  If the architecture of a system is to in fact store data of that size in
>> the DB, then yes that's a concern.  It seems that a better way to go would
>> be to store off the event unscathed as it were and then split the large data
>> out to somewhere other than the DB for analysis purposes.  But, yes, that
>> seems like something to consider for sure.
>>
>> But, I'm not sure that my original question was explicitly answered.  Is
>> CEE really intended to capture any kind of event or simply "traditional" IT
>> log events?  If the latter, then this discussion is probably academic
>> anyway.
>>
>> Adam
>>
>> From: "Forbes, R R" <[hidden email]<mailto:[hidden email]<[hidden email]>
>> >>
>> Date: Tue, 23 Nov 2010 19:34:20 -0800
>> To: Adam Montville
> <[hidden email]<mailto:[hidden email]<[hidden email]
>om>>>,
>> "[hidden email]<
>> mailto:[hidden email]<[hidden email]>>"
>> <[hidden email]<
>> mailto:[hidden email]<[hidden email]>
>> >>
>> Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>> Would the problem be at that point how do you properly store those items in
>> a DB. If you start allowing fields to be unlimited size then you are into
>> using CLOB and NCLOB datatypes for storing these large objects and that
>> isn't always conducive to DB design or determing database sizing
>> requirements
>>
>> ________________________________
>> From: Adam Montville [mailto:[hidden email]<[hidden email]>
>> ]
>> Sent: Tue 11/23/2010 9:07 PM
>> To: [hidden email]<
>> mailto:[hidden email]<[hidden email]>
>> >
>> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>
>> That's an interesting way to look at the issue.  Assuming that the
>> requirement would be limitless size and the technology used to express the
>> requirement (for now XML, but conceivably something else in the future),
>> that could be OK.
>>
>> What issues would this raise for potential implementations?  Parser attacks
>> or anything along those lines? Is that something with which we should be
>> concerned at this level (I would argue not).
>>
>>
>> -Adam
>>
>> On Nov 23, 2010, at 3:57 PM, "[hidden email]<
>> mailto:[hidden email] <[hidden email]>><
>> mailto:[hidden email] <[hidden email]>>" <
>> [hidden email]<mailto:[hidden email]<[hidden email]>
>> ><mailto:[hidden email] <[hidden email]>>> wrote:
>>
>>
>> Hello to all:
>>
>> My role in the board is that of promoting the cause.  But I figure to throw
>> in a cent to this issue.
>>
>> If XML does not place a limit to the size of any data, would it be
>> reasonable to specify that size of the log to be the same size as that
>> allowed in XML?
>>
>> David Oliva
>> ISSEP, CISSP, CAP, CISM, Security+, Network+
>>
>> Nov 23, 2010 04:45:52 PM, [hidden email]<
>> mailto:[hidden email] <[hidden email]>><
>> mailto:[hidden email] <[hidden email]>> wrote:
>> This may be a newb type of question, but are we constrained to IT audit
>> events or is there some wider applicability for CEE in the future? I can
>> conceive a physical sensor sending image and/or video data as part of a
>> record, for example. In that case, I could see the event record size
>> being large. If we are constrained to IT audit events, I can't see
>> anything larger than 1MB being realistic (if we see something like that,
>> it's anomalous in its own right).
>>
>> Also, it seems that the moment we truncate data, we're walking into the
>> realm of assumptions or at least guessing. If a packet is sent to 1000 IP
>> addresses, but we only report 128 or even 255 of them, we don't make the
>> job of containment/remediation for issues any easier. Of course, I could
>> be reading that question wrong.
>>
>> Not sure about the last question.
>>
>> Adam
>>
>> On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:
>>
>> >CEE is looking to recommend sizing constraints for event records to better
>> >enable compatibility.
>> >
>> >What are your thoughts on reasonable maximum limits for the following:
>> > + Event record size (1MB? 10MB? 100KB?)
>> > + Number of values for a single event field (e.g, a packet sent to 1000
>> >IP
>> >addresses) (128? 255?)
>> > + Value (string) size (1KB? 10KB?)
>> >
>> >
>> >Thank you,
>> >
>> >William Heinbockel
>> >Infosec Engineer, Sr.
>> >The MITRE Corporation
>> >202 Burlington Rd. MS S145
>> >Bedford, MA 01730
>> ><mailto:[hidden email] <[hidden email]>>[hidden email]<
>> mailto:[hidden email] <[hidden email]>><
>> mailto:[hidden email] <[hidden email]>>
>> >781-271-2615
>> >
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Lawrence Smith
Regarding the size of the event record... I think we should keep in mind that a common requirement of SIEM systems is forensically-sound storage for event records, ostensibly to be able to prove in a court of law what actually transpired. If the event record contains only an external pointer to key evidence (e.g. recorded video), the integrity of that evidence may be called into question. In other words, I think imposing a limit on the size of the event record will limit the value of the standard. 

Regarding the size of value (string)... In my work I am looking at incorporating execution of SQL queries of certain critical databases into our SIEM solution. I would like to include the full text of the query in the actual event record, and I've seen queries that are incredibly long. A size limit on this type will undoubtedly cut off some information. However, I recognize that many if not all event records will wind up in a database of some sort which will have limits on such fields, or at least those that are indexable. Perhaps the standard can define two field types: one limited, the other unlimited. As far as what an appropriate length would be for the former, we could again look to databases: a quick check of various databases suggests that an appropriate limit would be 4000 bytes.

On Wed, Nov 24, 2010 at 1:41 PM, David Corlette <[hidden email]> wrote:
FWIW, the XDAS v1 specification included a specific field that was supposed to be a URL pointer back to the original event source as a place to get more information (detail) about the described activity.  We expect XDAS v2 will do something similar.

- This obviously imposes a requirement on the event source to embed some sort of request engine that can respond to queries for that data - this could be HTTP, like RESTful interfaces, or it could be any other supported protocol (JDBC, whatever).

- I think in general it seems pretty silly to me to embed large data files in event data, since the point of the event to some extent is to summarize the activity that took place. If the event consumer wants more detail, it can and should go look it up.

- As examples of the types of data that could be referenced for more information, think images, raw packet captures, videos, live feeds....  This last one is most interesting, imagine you receive an event that says, essentially "someone is trying to break into your ATM" and you could click on a link to get a live video feed from the ATM's camera.

In other words, think of the event as a summary, and add metadata that will allow you to mash it up with data from other sources, rather than trying to shove everything under the sun into the event schema. This is what SIEMs do; they add contextual metadata to a baseline event schema (well, usually).

My 2c, HTH.

>>> On 24.11.2010 at 00:35, in message
<AANLkTi=[hidden email]>, Chris Blask
<[hidden email]> wrote:
> Hi folks,
>
> I second that emotion.
>
> Future size forecasts are notoriously incorrect.  Rich content of god-knows
> what size is almost inevitable and should probably be encouraged. We should
> want to be able to say: "this card-swipe related to this video coverage
> related to this workstation login related to this anomalous network behavior
> are worth knowing about".
>
> As I've thought about the collision of IT and CIP the shape of the solution
> has always seemed to include a "pointer to (video content, etc)" in
> messaging streams. Obviously you can't have a 100MB video capture in every
> (or any) syslog message, so the future solution has to include some sort of
> pointer-to for related content. This would seem to be a critical part of any
> messaging standard.
>
> -cheers
>
> -chris
>
>
>
> On 23 November 2010 19:59, Forbes, R R <[hidden email]> wrote:
>
>>  That's a good thought on dealing with large events. It would be easy
>> enough to generate logic to store events that were larger than a certain
>> size in a seperate tablespace thus elminating the performance impact when
>> dealing with more traditionally sized "IT Events"
>>
>> Robert
>>
>> ------------------------------
>> *From:* Adam Montville [mailto:[hidden email]]
>> *Sent:* Tue 11/23/2010 10:40 PM
>>
>> *To:* [hidden email]
>> *Subject:* Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>  If the architecture of a system is to in fact store data of that size in
>> the DB, then yes that's a concern.  It seems that a better way to go would
>> be to store off the event unscathed as it were and then split the large data
>> out to somewhere other than the DB for analysis purposes.  But, yes, that
>> seems like something to consider for sure.
>>
>> But, I'm not sure that my original question was explicitly answered.  Is
>> CEE really intended to capture any kind of event or simply "traditional" IT
>> log events?  If the latter, then this discussion is probably academic
>> anyway.
>>
>> Adam
>>
>> From: "Forbes, R R" <[hidden email]<mailto:[hidden email]<[hidden email]>
>> >>
>> Date: Tue, 23 Nov 2010 19:34:20 -0800
>> To: Adam Montville
> <[hidden email]<mailto:[hidden email]<[hidden email]
>om>>>,
>> Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>> Would the problem be at that point how do you properly store those items in
>> a DB. If you start allowing fields to be unlimited size then you are into
>> using CLOB and NCLOB datatypes for storing these large objects and that
>> isn't always conducive to DB design or determing database sizing
>> requirements
>>
>> ________________________________
>> From: Adam Montville [mailto:[hidden email]<[hidden email]>
>> ]
>> Sent: Tue 11/23/2010 9:07 PM
>> To: [hidden email]<
>> mailto:[hidden email]<[hidden email]>
>> >
>> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>
>> That's an interesting way to look at the issue.  Assuming that the
>> requirement would be limitless size and the technology used to express the
>> requirement (for now XML, but conceivably something else in the future),
>> that could be OK.
>>
>> What issues would this raise for potential implementations?  Parser attacks
>> or anything along those lines? Is that something with which we should be
>> concerned at this level (I would argue not).
>>
>>
>> -Adam
>>
>> On Nov 23, 2010, at 3:57 PM, "[hidden email]<
>> mailto:[hidden email] <[hidden email]>><
>> ><mailto:[hidden email] <[hidden email]>>> wrote:
>>
>>
>> Hello to all:
>>
>> My role in the board is that of promoting the cause.  But I figure to throw
>> in a cent to this issue.
>>
>> If XML does not place a limit to the size of any data, would it be
>> reasonable to specify that size of the log to be the same size as that
>> allowed in XML?
>>
>> David Oliva
>> ISSEP, CISSP, CAP, CISM, Security+, Network+
>>
>> Nov 23, 2010 04:45:52 PM, [hidden email]<
>> mailto:[hidden email] <[hidden email]>><
>> mailto:[hidden email] <[hidden email]>> wrote:
>> This may be a newb type of question, but are we constrained to IT audit
>> events or is there some wider applicability for CEE in the future? I can
>> conceive a physical sensor sending image and/or video data as part of a
>> record, for example. In that case, I could see the event record size
>> being large. If we are constrained to IT audit events, I can't see
>> anything larger than 1MB being realistic (if we see something like that,
>> it's anomalous in its own right).
>>
>> Also, it seems that the moment we truncate data, we're walking into the
>> realm of assumptions or at least guessing. If a packet is sent to 1000 IP
>> addresses, but we only report 128 or even 255 of them, we don't make the
>> job of containment/remediation for issues any easier. Of course, I could
>> be reading that question wrong.
>>
>> Not sure about the last question.
>>
>> Adam
>>
>> On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:
>>
>> >CEE is looking to recommend sizing constraints for event records to better
>> >enable compatibility.
>> >
>> >What are your thoughts on reasonable maximum limits for the following:
>> > + Event record size (1MB? 10MB? 100KB?)
>> > + Number of values for a single event field (e.g, a packet sent to 1000
>> >IP
>> >addresses) (128? 255?)
>> > + Value (string) size (1KB? 10KB?)
>> >
>> >
>> >Thank you,
>> >
>> >William Heinbockel
>> >Infosec Engineer, Sr.
>> >The MITRE Corporation
>> >202 Burlington Rd. MS S145
>> >Bedford, MA 01730
>> ><mailto:[hidden email] <[hidden email]>>[hidden email]<
>> mailto:[hidden email] <[hidden email]>><
>> mailto:[hidden email] <[hidden email]>>
>> >781-271-2615
>> >
>> >
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Bill Frank

I agree with Lawrence Smith that such attributes as complete SQL queries must be supported.

 

In addition, a way to get back to the event source for more detailed information is also needed. David Corlette’s examples are good – full packet capture and video clips. Another example I’ve seen is email. The email metadata is captured in log events and there is a pointer back to the original full email that is stored outside of the log repository.

 

The ability to query the event source for more information raises access control issues. In other words, the person viewing the logs may not have the right to access the original event source for more information. My email example above is a case in point. The person conducting the investigation of the logs may not be allowed to read the actual email messages.

 

Bill Frank

 

From: Lawrence Smith [mailto:[hidden email]]
Sent: Saturday, November 27, 2010 12:26 AM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

 

Regarding the size of the event record... I think we should keep in mind that a common requirement of SIEM systems is forensically-sound storage for event records, ostensibly to be able to prove in a court of law what actually transpired. If the event record contains only an external pointer to key evidence (e.g. recorded video), the integrity of that evidence may be called into question. In other words, I think imposing a limit on the size of the event record will limit the value of the standard. 

 

Regarding the size of value (string)... In my work I am looking at incorporating execution of SQL queries of certain critical databases into our SIEM solution. I would like to include the full text of the query in the actual event record, and I've seen queries that are incredibly long. A size limit on this type will undoubtedly cut off some information. However, I recognize that many if not all event records will wind up in a database of some sort which will have limits on such fields, or at least those that are indexable. Perhaps the standard can define two field types: one limited, the other unlimited. As far as what an appropriate length would be for the former, we could again look to databases: a quick check of various databases suggests that an appropriate limit would be 4000 bytes.

 

On Wed, Nov 24, 2010 at 1:41 PM, David Corlette <[hidden email]> wrote:

FWIW, the XDAS v1 specification included a specific field that was supposed to be a URL pointer back to the original event source as a place to get more information (detail) about the described activity.  We expect XDAS v2 will do something similar.

- This obviously imposes a requirement on the event source to embed some sort of request engine that can respond to queries for that data - this could be HTTP, like RESTful interfaces, or it could be any other supported protocol (JDBC, whatever).

- I think in general it seems pretty silly to me to embed large data files in event data, since the point of the event to some extent is to summarize the activity that took place. If the event consumer wants more detail, it can and should go look it up.

- As examples of the types of data that could be referenced for more information, think images, raw packet captures, videos, live feeds....  This last one is most interesting, imagine you receive an event that says, essentially "someone is trying to break into your ATM" and you could click on a link to get a live video feed from the ATM's camera.

In other words, think of the event as a summary, and add metadata that will allow you to mash it up with data from other sources, rather than trying to shove everything under the sun into the event schema. This is what SIEMs do; they add contextual metadata to a baseline event schema (well, usually).

My 2c, HTH.

>>> On 24.11.2010 at 00:35, in message
<AANLkTi=[hidden email]>, Chris Blask

<[hidden email]> wrote:


> Hi folks,
>
> I second that emotion.
>
> Future size forecasts are notoriously incorrect.  Rich content of god-knows
> what size is almost inevitable and should probably be encouraged. We should
> want to be able to say: "this card-swipe related to this video coverage
> related to this workstation login related to this anomalous network behavior
> are worth knowing about".
>
> As I've thought about the collision of IT and CIP the shape of the solution
> has always seemed to include a "pointer to (video content, etc)" in
> messaging streams. Obviously you can't have a 100MB video capture in every
> (or any) syslog message, so the future solution has to include some sort of
> pointer-to for related content. This would seem to be a critical part of any
> messaging standard.
>
> -cheers
>
> -chris
>
>
>
> On 23 November 2010 19:59, Forbes, R R <[hidden email]> wrote:
>
>>  That's a good thought on dealing with large events. It would be easy
>> enough to generate logic to store events that were larger than a certain
>> size in a seperate tablespace thus elminating the performance impact when
>> dealing with more traditionally sized "IT Events"
>>
>> Robert
>>
>> ------------------------------
>> *From:* Adam Montville [mailto:[hidden email]]
>> *Sent:* Tue 11/23/2010 10:40 PM
>>
>> *To:* [hidden email]
>> *Subject:* Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>  If the architecture of a system is to in fact store data of that size in
>> the DB, then yes that's a concern.  It seems that a better way to go would
>> be to store off the event unscathed as it were and then split the large data
>> out to somewhere other than the DB for analysis purposes.  But, yes, that
>> seems like something to consider for sure.
>>
>> But, I'm not sure that my original question was explicitly answered.  Is
>> CEE really intended to capture any kind of event or simply "traditional" IT
>> log events?  If the latter, then this discussion is probably academic
>> anyway.
>>
>> Adam
>>

>> From: "Forbes, R R" <[hidden email]<mailto:[hidden email]<[hidden email]>

>> >>
>> Date: Tue, 23 Nov 2010 19:34:20 -0800
>> To: Adam Montville

> <[hidden email]<mailto:[hidden email]<[hidden email]
>om>>>,

>> Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>> Would the problem be at that point how do you properly store those items in
>> a DB. If you start allowing fields to be unlimited size then you are into
>> using CLOB and NCLOB datatypes for storing these large objects and that
>> isn't always conducive to DB design or determing database sizing
>> requirements
>>
>> ________________________________

>> From: Adam Montville [mailto:[hidden email]<[hidden email]>

>> ]
>> Sent: Tue 11/23/2010 9:07 PM
>> To: [hidden email]<

>> mailto:[hidden email]<[hidden email]>

>> >
>> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>
>> That's an interesting way to look at the issue.  Assuming that the
>> requirement would be limitless size and the technology used to express the
>> requirement (for now XML, but conceivably something else in the future),
>> that could be OK.
>>
>> What issues would this raise for potential implementations?  Parser attacks
>> or anything along those lines? Is that something with which we should be
>> concerned at this level (I would argue not).
>>
>>
>> -Adam
>>
>> On Nov 23, 2010, at 3:57 PM, "[hidden email]<

>> mailto:[hidden email] <[hidden email]>><

>> ><mailto:[hidden email] <[hidden email]>>> wrote:
>>
>>
>> Hello to all:
>>
>> My role in the board is that of promoting the cause.  But I figure to throw
>> in a cent to this issue.
>>
>> If XML does not place a limit to the size of any data, would it be
>> reasonable to specify that size of the log to be the same size as that
>> allowed in XML?
>>
>> David Oliva
>> ISSEP, CISSP, CAP, CISM, Security+, Network+
>>
>> Nov 23, 2010 04:45:52 PM, [hidden email]<

>> mailto:[hidden email] <[hidden email]>><

>> mailto:[hidden email] <[hidden email]>> wrote:
>> This may be a newb type of question, but are we constrained to IT audit
>> events or is there some wider applicability for CEE in the future? I can
>> conceive a physical sensor sending image and/or video data as part of a
>> record, for example. In that case, I could see the event record size
>> being large. If we are constrained to IT audit events, I can't see
>> anything larger than 1MB being realistic (if we see something like that,
>> it's anomalous in its own right).
>>
>> Also, it seems that the moment we truncate data, we're walking into the
>> realm of assumptions or at least guessing. If a packet is sent to 1000 IP
>> addresses, but we only report 128 or even 255 of them, we don't make the
>> job of containment/remediation for issues any easier. Of course, I could
>> be reading that question wrong.
>>
>> Not sure about the last question.
>>
>> Adam
>>
>> On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:
>>
>> >CEE is looking to recommend sizing constraints for event records to better
>> >enable compatibility.
>> >
>> >What are your thoughts on reasonable maximum limits for the following:
>> > + Event record size (1MB? 10MB? 100KB?)
>> > + Number of values for a single event field (e.g, a packet sent to 1000
>> >IP
>> >addresses) (128? 255?)
>> > + Value (string) size (1KB? 10KB?)
>> >
>> >
>> >Thank you,
>> >
>> >William Heinbockel
>> >Infosec Engineer, Sr.
>> >The MITRE Corporation
>> >202 Burlington Rd. MS S145
>> >Bedford, MA 01730

>> ><mailto:[hidden email] <[hidden email]>>[hidden email]<
>> mailto:[hidden email] <[hidden email]>><
>> mailto:[hidden email] <[hidden email]>>
>> >781-271-2615
>> >
>> >
>>
>

 

Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Chris Blask
In reply to this post by David Corlette
On 24 November 2010 11:41, David Corlette <[hidden email]> wrote:
FWIW, the XDAS v1 specification included a specific field that was supposed to be a URL pointer back to the original event source as a place to get more information (detail) about the described activity.  We expect XDAS v2 will do something similar.

- This obviously imposes a requirement on the event source to embed some sort of request engine that can respond to queries for that data - this could be HTTP, like RESTful interfaces, or it could be any other supported protocol (JDBC, whatever).

- I think in general it seems pretty silly to me to embed large data files in event data, since the point of the event to some extent is to summarize the activity that took place. If the event consumer wants more detail, it can and should go look it up.

- As examples of the types of data that could be referenced for more information, think images, raw packet captures, videos, live feeds....  This last one is most interesting, imagine you receive an event that says, essentially "someone is trying to break into your ATM" and you could click on a link to get a live video feed from the ATM's camera.

Was poking through the new PCI DSS the other day for other reasons and spot this, which seems something of a canary in a coal mine on the topic:

"9.1.1 Use video cameras and/or access control mechanisms to monitor individual physical access to sensitive areas. Review collected data and correlate with other entries."

That pretty well sums it up.

In other words, think of the event as a summary, and add metadata that will allow you to mash it up with data from other sources, rather than trying to shove everything under the sun into the event schema. This is what SIEMs do; they add contextual metadata to a baseline event schema (well, usually).

SIEM databases are already large and they need to get larger. Expecting them to actually process billions of datum that are each of gigabyte size is of course irrational, so there must be a different solution. 

Meta-data in the video context is a good example of what a SIEM would be interested in:

 - The fact that there is imagery. 
 - That it has certain basic attributes (length, resolution...).
 - That it is from a given device (with it's own known attributes: topological location, geographic location,orientation,type [IR,light-enhanced,still..]...).
 - Where the video is stored (another device with it's known attributes, including things like ability to support crypto/auth...).
 - Qualifying info (perhaps the source device includes pattern-recognition and has flagged the event as 'possible match with pattern Bob Smith').

The SIEM in question could correlate the event from a card-swipe with an event from the video system to create the context that someone claiming to be Bob Smith is standing at the southwest entrance (because it already knows where those devices are), and perhaps that the facial recognition system has given a nominal match with existing records. It already has all sorts of context about Bob and his environment.

Perhaps subsequent behavior associated with Bob causes an Incident to be brought to an analyst's attention. The analyst pulls up the video from   it's storage location and Bob's employee profile from another location.

Other examples show the same shape. Since it is unreasonable for an event processing system to handle all possible data the same way, storage and management of some data has to be federated. SIEM users (analysts, auditors, investigators) will be able to see whether federated data stores are themselves appropriately secured and draw conclusions about the suitability of the overall system's output for various purposes.

-chris
-- 
Chris Blask
VP Marketing
AlienVault

1901 S Bascom Avenue
Suite 220
Campbell, CA, 95008
[hidden email]
+1 408 465-9989
Skype: chrisblask


Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Peter Czanik
In reply to this post by heinbockel
Hello,

Bellow is the response from syslog-ng's lead developer.

Bye,
CzP

On 11/22/2010 11:08 PM, Heinbockel, Bill wrote:

> CEE is looking to recommend sizing constraints for event records to better
> > enable compatibility.
> >
> > What are your thoughts on reasonable maximum limits for the following:
> >  +  Event record size (1MB? 10MB? 100KB?)
>  
Right now, syslog-ng's internal representation limits all fields of an
event record to 256kB, but it uses some of this for book-keeping. It is
not unreasonable to increase this limit, but would increase the
book-keeping overhead.

Each additional event field takes up 10 bytes, plus the value field
(from the 256kb quoted above).


> >  +  Number of values for a single event field (e.g, a packet sent to 1000 IP
> > addresses) (128? 255?)
>  
Well, in the last incarnation of the spec, there was no arrays of values
for a single event field. This is not supported by syslog-ng right now,
unless some kind of serialization is performed in the value.


> >  +  Value (string) size (1KB? 10KB?)
>  
This is 64k right now.

But everything adds up into the same 256kB limit.

-- Bazsi
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

heinbockel
In reply to this post by Adam Montville
The scoping of CEE is currently for "events observed and generated by an
IT/electronic system"

This is not saying that CEE cannot be used elsewhere, but our focus is on IT
events


William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Adam Montville [mailto:[hidden email]]
>Sent: Tuesday, 23 November 2010 17:45
>To: Heinbockel, Bill; cee-discussion-list CEE-Related Discussion
>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>This may be a newb type of question, but are we constrained to IT audit
>events or is there some wider applicability for CEE in the future?  I can
>conceive a physical sensor sending image and/or video data as part of a
>record, for example.  In that case, I could see the event record size
>being large.  If we are constrained to IT audit events, I can't see
>anything larger than 1MB being realistic (if we see something like that,
>it's anomalous in its own right).
>
>Also, it seems that the moment we truncate data, we're walking into the
>realm of assumptions or at least guessing.  If a packet is sent to 1000 IP
>addresses, but we only report 128 or even 255 of them, we don't make the
>job of containment/remediation for issues any easier.  Of course, I could
>be reading that question wrong.
>
>Not sure about the last question.
>
>Adam
>
>On 11/22/10 2:08 PM, "Heinbockel, Bill" <[hidden email]> wrote:
>
>>CEE is looking to recommend sizing constraints for event records to better
>>enable compatibility.
>>
>>What are your thoughts on reasonable maximum limits for the following:
>> +  Event record size (1MB? 10MB? 100KB?)
>> +  Number of values for a single event field (e.g, a packet sent to 1000
>>IP
>>addresses) (128? 255?)
>> +  Value (string) size (1KB? 10KB?)
>>
>>
>>Thank you,
>>
>>William Heinbockel
>>Infosec Engineer, Sr.
>>The MITRE Corporation
>>202 Burlington Rd. MS S145
>>Bedford, MA 01730
>>[hidden email]
>>781-271-2615
>>
>>
>


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

heinbockel
In reply to this post by Mark Poepping
Yes, IT logs are currently the main focus.
While we expect that there will be no issue with 99% of the events, there are
events with may exceed traditional event message limits.

For example, what if an event pertains to the creation of 5000 user accounts?
Putting all of these into a single event is easy, but may exceed the processing
capabilities of the transport (e.g., Syslog) or the consuming system. Another
way is to send 5000 different events -- this is safe, but duplicates a lot of
data.

The final way is to allow a single event "record" to span multiple event
"messages"
To do this reliably across the audit management space, there has to be
reasonable limits set so both the event producer and consumer do no lose data.


William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Mark Poepping [mailto:[hidden email]]
>Sent: Wednesday, 24 November 2010 00:25
>To: cee-discussion-list CEE-Related Discussion
>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>Seems to me that CEE (with the OAS approach) is strongly biased to current
>IT-based logs, and I think it's worth that focus given the problems there.
>
>I would generalize your question to: is this about "IT log events" or
>"situational awareness" (*way more general*)?  I talked with Chuvakin about
>this three years ago, and my take at the time was that this effort was
>mostly about normalizing firewall and syslog information, since that's 95%
>of the market and more importantly, it's 98% of the *understanding of
>current practitioners*.  In that, I think there is good value in continuing
>to support this effort, and it's probably important to *not* make it too
>general, because that might jeopardize the possibility of improving stuff
>we've already decided we need to do (my opinion).
>
>Of course, I find the general question of situational awareness far more
>interesting.  I think it will be increasingly important to capture and
>incorporate video, network traffic flows, component health, physical sensor
>information, application/service transaction logs, "you-name-it" into the
>diagnostic management framework, but that's the stuff we're not so good at
>and we generally don't understand the value or approach to bringing it
>together.  The only thing I can tell you is that we're certainly *not* going
>to mine it all from a database:-).
>
>At present, many business sectors are experimenting with data from their own
>sensors and building analytics for the information they have, that's better
>than nothing, but they rarely share analytics and almost never cross that
>data with information from related infrastructures (e.g. when my motion
>sensor is unreachable, did it die, did the network connectivity to it go
>away, or did the service that's supposed to ping it go postal?)..  I think
>you should understand a problem before you standardize and that's why we try
>to assist local researchers with experimental analysis into this kind of
>thing.  At present however, as I said, 98% of the understanding (even here)
>doesn't yet *see* this level of correlation in their situational awareness
>picture - they're still trying to get data from their own stuff...
>
>Mark.
>
>
>>-----Original Message-----
>>From: Adam Montville [mailto:[hidden email]]
>>Sent: Tuesday, November 23, 2010 10:40 PM
>>To: [hidden email]
>>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>If the architecture of a system is to in fact store data of that size in
>the
>>DB, then yes that's a concern.  It seems that a better way to go would be
>to
>>store off the event unscathed as it were and then split the large data out
>to
>>somewhere other than the DB for analysis purposes.  But, yes, that seems
>like
>>something to consider for sure.
>>
>>But, I'm not sure that my original question was explicitly answered.  Is
>CEE
>>really intended to capture any kind of event or simply "traditional" IT log
>>events?  If the latter, then this discussion is probably academic anyway.
>>
>>Adam
>>
>>From: "Forbes, R R" <[hidden email]<mailto:[hidden email]>>
>>Date: Tue, 23 Nov 2010 19:34:20 -0800
>>To: Adam Montville
><[hidden email]<mailto:[hidden email]>>,
>>"[hidden email]<mailto:CEE-DISCUSSION-
>>[hidden email]>" <[hidden email]<mailto:CEE-
>>[hidden email]>>
>>Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>Would the problem be at that point how do you properly store those items in
>a
>>DB. If you start allowing fields to be unlimited size then you are into
>using
>>CLOB and NCLOB datatypes for storing these large objects and that isn't
>always
>>conducive to DB design or determing database sizing requirements
>>
>>________________________________
>>From: Adam Montville [mailto:[hidden email]]
>>Sent: Tue 11/23/2010 9:07 PM
>>To: [hidden email]<mailto:CEE-DISCUSSION-
>>[hidden email]>
>>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>
>>That's an interesting way to look at the issue.  Assuming that the
>requirement
>>would be limitless size and the technology used to express the requirement
>(for
>>now XML, but conceivably something else in the future), that could be OK.
>>
>>What issues would this raise for potential implementations?  Parser attacks
>or
>>anything along those lines? Is that something with which we should be
>concerned
>>at this level (I would argue not).
>>
>>
>>-Adam
>>
>>On Nov 23, 2010, at 3:57 PM,
>>"[hidden email]<mailto:[hidden email]><mailto:david.oliva
>@ver
>>izon.net>"
>><[hidden email]<mailto:[hidden email]><mailto:david.oliva
>@ver
>>izon.net>> wrote:
>>
>>
>>Hello to all:
>>
>>My role in the board is that of promoting the cause.  But I figure to throw
>in
>>a cent to this issue.
>>
>>If XML does not place a limit to the size of any data, would it be
>reasonable
>>to specify that size of the log to be the same size as that allowed in XML?
>>
>>David Oliva
>>ISSEP, CISSP, CAP, CISM, Security+, Network+
>>
>>Nov 23, 2010 04:45:52 PM,
>>[hidden email]<mailto:[hidden email]><mailto:amontville@T
>RIPW
>>IRE.COM> wrote:
>>This may be a newb type of question, but are we constrained to IT audit
>>events or is there some wider applicability for CEE in the future? I can
>>conceive a physical sensor sending image and/or video data as part of a
>>record, for example. In that case, I could see the event record size
>>being large. If we are constrained to IT audit events, I can't see
>>anything larger than 1MB being realistic (if we see something like that,
>>it's anomalous in its own right).
>>
>>Also, it seems that the moment we truncate data, we're walking into the
>>realm of assumptions or at least guessing. If a packet is sent to 1000 IP
>>addresses, but we only report 128 or even 255 of them, we don't make the
>>job of containment/remediation for issues any easier. Of course, I could
>>be reading that question wrong.
>>
>>Not sure about the last question.
>>
>>Adam
>>
>>On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:
>>
>>>CEE is looking to recommend sizing constraints for event records to better
>>>enable compatibility.
>>>
>>>What are your thoughts on reasonable maximum limits for the following:
>>> + Event record size (1MB? 10MB? 100KB?)
>>> + Number of values for a single event field (e.g, a packet sent to 1000
>>>IP
>>>addresses) (128? 255?)
>>> + Value (string) size (1KB? 10KB?)
>>>
>>>
>>>Thank you,
>>>
>>>William Heinbockel
>>>Infosec Engineer, Sr.
>>>The MITRE Corporation
>>>202 Burlington Rd. MS S145
>>>Bedford, MA 01730
>>><mailto:[hidden email]>[hidden email]<mailto:heinbockel@mitre.
>org>
>><mailto:[hidden email]>
>>>781-271-2615
>>>
>>>

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Lawrence Smith
In reply to this post by heinbockel
Agreed, but I would definitely include network-enabled physical sensors (motion detectors, video cameras, and other physical security systems) within the definition of an IT/electronic system.
Sent via BlackBerry by AT&T

-----Original Message-----
From: "Heinbockel, Bill" <[hidden email]>
Date:         Mon, 29 Nov 2010 10:37:48
To: <[hidden email]>
Reply-To: "Heinbockel, Bill" <[hidden email]>
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

The scoping of CEE is currently for "events observed and generated by an
IT/electronic system"

This is not saying that CEE cannot be used elsewhere, but our focus is on IT
events


William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Adam Montville [mailto:[hidden email]]
>Sent: Tuesday, 23 November 2010 17:45
>To: Heinbockel, Bill; cee-discussion-list CEE-Related Discussion
>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>This may be a newb type of question, but are we constrained to IT audit
>events or is there some wider applicability for CEE in the future?  I can
>conceive a physical sensor sending image and/or video data as part of a
>record, for example.  In that case, I could see the event record size
>being large.  If we are constrained to IT audit events, I can't see
>anything larger than 1MB being realistic (if we see something like that,
>it's anomalous in its own right).
>
>Also, it seems that the moment we truncate data, we're walking into the
>realm of assumptions or at least guessing.  If a packet is sent to 1000 IP
>addresses, but we only report 128 or even 255 of them, we don't make the
>job of containment/remediation for issues any easier.  Of course, I could
>be reading that question wrong.
>
>Not sure about the last question.
>
>Adam
>
>On 11/22/10 2:08 PM, "Heinbockel, Bill" <[hidden email]> wrote:
>
>>CEE is looking to recommend sizing constraints for event records to better
>>enable compatibility.
>>
>>What are your thoughts on reasonable maximum limits for the following:
>> +  Event record size (1MB? 10MB? 100KB?)
>> +  Number of values for a single event field (e.g, a packet sent to 1000
>>IP
>>addresses) (128? 255?)
>> +  Value (string) size (1KB? 10KB?)
>>
>>
>>Thank you,
>>
>>William Heinbockel
>>Infosec Engineer, Sr.
>>The MITRE Corporation
>>202 Burlington Rd. MS S145
>>Bedford, MA 01730
>>[hidden email]
>>781-271-2615
>>
>>
>


12