Event Record Size Limits

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

heinbockel
Yes, there needs to be the concept of an identifier that can be used to relate
events with data.
The big issue here is that things like video, packet capture, large data are
handled separate from the audit/log infrastructure. These can be later meshed by
using timestamps, packet ids, src/dst addresses, etc.

Ideally what we want is a better way to correlate the event with the data

To me, this is the next logical step after standardizing the event records with
CEE


William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Chris Blask [mailto:[hidden email]]
>Sent: Wednesday, 24 November 2010 00:35
>To: cee-discussion-list CEE-Related Discussion
>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>Hi folks,
>
>I second that emotion.
>
>Future size forecasts are notoriously incorrect.  Rich content of god-knows
>what size is almost inevitable and should probably be encouraged. We should
>want to be able to say: "this card-swipe related to this video coverage
>related to this workstation login related to this anomalous network behavior
>are worth knowing about".
>
>As I've thought about the collision of IT and CIP the shape of the solution
>has always seemed to include a "pointer to (video content, etc)" in
>messaging streams. Obviously you can't have a 100MB video capture in every
>(or any) syslog message, so the future solution has to include some sort of
>pointer-to for related content. This would seem to be a critical part of any
>messaging standard.
>
>-cheers
>
>-chris
>
>
>
>
>On 23 November 2010 19:59, Forbes, R R <[hidden email]> wrote:
>
>
> That's a good thought on dealing with large events. It would be easy
>enough to generate logic to store events that were larger than a certain
>size in a seperate tablespace thus elminating the performance impact when
>dealing with more traditionally sized "IT Events"
>
> Robert
>
>________________________________
>
>
> From: Adam Montville [mailto:[hidden email]]
>
> Sent: Tue 11/23/2010 10:40 PM
>
> To: [hidden email]
>
> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>
>
> If the architecture of a system is to in fact store data of that size
>in the DB, then yes that's a concern.  It seems that a better way to go
>would be to store off the event unscathed as it were and then split the
>large data out to somewhere other than the DB for analysis purposes.  But,
>yes, that seems like something to consider for sure.
>
> But, I'm not sure that my original question was explicitly answered.
>Is CEE really intended to capture any kind of event or simply "traditional"
>IT log events?  If the latter, then this discussion is probably academic
>anyway.
>
> Adam
>
> From: "Forbes, R R" <[hidden email]<mailto:[hidden email]>>
> Date: Tue, 23 Nov 2010 19:34:20 -0800
> To: Adam Montville
><[hidden email]<mailto:[hidden email]>>, "CEE-DISCUSSION-
>[hidden email]<mailto:[hidden email]>" <CEE-
>[hidden email]<mailto:[hidden email]>>
> Subject: RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
> Would the problem be at that point how do you properly store those
>items in a DB. If you start allowing fields to be unlimited size then you
>are into using CLOB and NCLOB datatypes for storing these large objects and
>that isn't always conducive to DB design or determing database sizing
>requirements
>
> ________________________________
> From: Adam Montville [mailto:[hidden email]]
> Sent: Tue 11/23/2010 9:07 PM
> To: [hidden email]<mailto:CEE-DISCUSSION-
>[hidden email]>
> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
>
> That's an interesting way to look at the issue.  Assuming that the
>requirement would be limitless size and the technology used to express the
>requirement (for now XML, but conceivably something else in the future),
>that could be OK.
>
> What issues would this raise for potential implementations?  Parser
>attacks or anything along those lines? Is that something with which we
>should be concerned at this level (I would argue not).
>
>
> -Adam
>
> On Nov 23, 2010, at 3:57 PM,
>"[hidden email]<mailto:[hidden email]><mailto:david.oliva@
>verizon.net>"
><[hidden email]<mailto:[hidden email]><mailto:david.oliva@
>verizon.net>> wrote:
>
>
> Hello to all:
>
> My role in the board is that of promoting the cause.  But I figure to
>throw in a cent to this issue.
>
> If XML does not place a limit to the size of any data, would it be
>reasonable to specify that size of the log to be the same size as that
>allowed in XML?
>
> David Oliva
> ISSEP, CISSP, CAP, CISM, Security+, Network+
>
> Nov 23, 2010 04:45:52 PM,
>[hidden email]<mailto:[hidden email]><mailto:amontville@TR
>IPWIRE.COM> wrote:
> This may be a newb type of question, but are we constrained to IT
>audit
> events or is there some wider applicability for CEE in the future? I
>can
> conceive a physical sensor sending image and/or video data as part of
>a
> record, for example. In that case, I could see the event record size
> being large. If we are constrained to IT audit events, I can't see
> anything larger than 1MB being realistic (if we see something like
>that,
> it's anomalous in its own right).
>
> Also, it seems that the moment we truncate data, we're walking into
>the
> realm of assumptions or at least guessing. If a packet is sent to 1000
>IP
> addresses, but we only report 128 or even 255 of them, we don't make
>the
> job of containment/remediation for issues any easier. Of course, I
>could
> be reading that question wrong.
>
> Not sure about the last question.
>
> Adam
>
> On 11/22/10 2:08 PM, "Heinbockel, Bill" wrote:
>
> >CEE is looking to recommend sizing constraints for event records to
>better
> >enable compatibility.
> >
> >What are your thoughts on reasonable maximum limits for the
>following:
> > + Event record size (1MB? 10MB? 100KB?)
> > + Number of values for a single event field (e.g, a packet sent to
>1000
> >IP
> >addresses) (128? 255?)
> > + Value (string) size (1KB? 10KB?)
> >
> >
> >Thank you,
> >
> >William Heinbockel
> >Infosec Engineer, Sr.
> >The MITRE Corporation
> >202 Burlington Rd. MS S145
> >Bedford, MA 01730
> ><mailto:[hidden email]>[hidden email]<mailto:heinbockel@m
>itre.org><mailto:[hidden email]>
> >781-271-2615
> >
> >
>
>
>
>
>
>--
>Chris Blask
>VP Marketing
>AlienVault
>
>1901 S Bascom Avenue
>Suite 220
>Campbell, CA, 95008
>[hidden email]
>+1 408 465-9989
>Skype: chrisblask
>
>Author: Security Information and Event Management (SIEM) Implementation
>McGraw Hill  ISBN-10: 0071701095 ISBN-13: 978-0071701099


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Eric Fitzgerald-2
In reply to this post by Peter Czanik
This is great information.  Here's the comparable data for Windows:

Windows Eventlog, prior to Windows Vista, has a 64kb per event record size limitation, and log file size is limited by paged pool (effectively a few hundred meg on x86, more on x64).  Each individual string provided to the eventlog is limited to 32kb in size.  As with syslog-ng, the 64kb is partially consumed by event record overhead of a couple dozen bytes per event record.
http://msdn.microsoft.com/en-us/library/aa363679(VS.85).aspx 
http://support.microsoft.com/kb/183097

The security event log actually is highly normalized and reserves 9kb for overhead for field normalization, so the maximum size of record that can be written to the security event log is 55kb.  Note that each record doesn't take 9kb of overhead, this is just an upper limit on the data provided to the logging API so that no internal normalization operations will fail due to event record size constraints.
http://msdn.microsoft.com/en-us/library/aa363679(VS.85).aspx
http://msdn.microsoft.com/en-us/library/aa376317(VS.85).aspx

Windows eventlog, starting with Windows Vista, there is still a 64kb per event record size limitation, but no limit on log size beyond the limits of the underlying file system.  I am not sure if there is a limitation on field size; I have to check.
http://msdn.microsoft.com/en-us/library/aa363753(v=VS.85).aspx




-----Original Message-----
From: Peter Czanik [mailto:[hidden email]]
Sent: Monday, November 29, 2010 1:48 AM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

Hello,

Bellow is the response from syslog-ng's lead developer.

Bye,
CzP

On 11/22/2010 11:08 PM, Heinbockel, Bill wrote:

> CEE is looking to recommend sizing constraints for event records to
> better
> > enable compatibility.
> >
> > What are your thoughts on reasonable maximum limits for the following:
> >  +  Event record size (1MB? 10MB? 100KB?)
>  
Right now, syslog-ng's internal representation limits all fields of an event record to 256kB, but it uses some of this for book-keeping. It is not unreasonable to increase this limit, but would increase the book-keeping overhead.

Each additional event field takes up 10 bytes, plus the value field (from the 256kb quoted above).


> >  +  Number of values for a single event field (e.g, a packet sent to
> > 1000 IP
> > addresses) (128? 255?)
>  
Well, in the last incarnation of the spec, there was no arrays of values for a single event field. This is not supported by syslog-ng right now, unless some kind of serialization is performed in the value.


> >  +  Value (string) size (1KB? 10KB?)
>  
This is 64k right now.

But everything adds up into the same 256kB limit.

-- Bazsi
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Rainer Gerhards
So on to some information on rsyslog: In rsyslog, I do not have any limit
except for the default message size limit, which is the RFC-recommended value
of 8K. But this can be configured to any value the operator chooses. There
are no limits on individual fields. Of course, performance degrades somewhat
with very large fields.

I have written libee[1] and liblognorm[2], the new normalizing libs based on
CEE, in a way that they also do not have any limits. Performance degrades a
little bit for fields over 128K.

Of course, there are many physical limits of the underlying HW and OS.

Rainer

[1] http://www.libee.org
[2] http://www.liblognorm.com

> -----Original Message-----
> From: Eric Fitzgerald [mailto:[hidden email]]
> Sent: Tuesday, November 30, 2010 12:41 AM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
> This is great information.  Here's the comparable data for Windows:
>
> Windows Eventlog, prior to Windows Vista, has a 64kb per event record
> size limitation, and log file size is limited by paged pool
> (effectively a few hundred meg on x86, more on x64).  Each individual
> string provided to the eventlog is limited to 32kb in size.  As with
> syslog-ng, the 64kb is partially consumed by event record overhead of a
> couple dozen bytes per event record.
> http://msdn.microsoft.com/en-us/library/aa363679(VS.85).aspx
> http://support.microsoft.com/kb/183097
>
> The security event log actually is highly normalized and reserves 9kb
> for overhead for field normalization, so the maximum size of record
> that can be written to the security event log is 55kb.  Note that each
> record doesn't take 9kb of overhead, this is just an upper limit on the
> data provided to the logging API so that no internal normalization
> operations will fail due to event record size constraints.
> http://msdn.microsoft.com/en-us/library/aa363679(VS.85).aspx
> http://msdn.microsoft.com/en-us/library/aa376317(VS.85).aspx
>
> Windows eventlog, starting with Windows Vista, there is still a 64kb
> per event record size limitation, but no limit on log size beyond the
> limits of the underlying file system.  I am not sure if there is a
> limitation on field size; I have to check.
> http://msdn.microsoft.com/en-us/library/aa363753(v=VS.85).aspx
>
>
>
>
> -----Original Message-----
> From: Peter Czanik [mailto:[hidden email]]
> Sent: Monday, November 29, 2010 1:48 AM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
> Hello,
>
> Bellow is the response from syslog-ng's lead developer.
>
> Bye,
> CzP
>
> On 11/22/2010 11:08 PM, Heinbockel, Bill wrote:
>
> > CEE is looking to recommend sizing constraints for event records to
> > better
> > > enable compatibility.
> > >
> > > What are your thoughts on reasonable maximum limits for the
> following:
> > >  +  Event record size (1MB? 10MB? 100KB?)
> >
> Right now, syslog-ng's internal representation limits all fields of an
> event record to 256kB, but it uses some of this for book-keeping. It is
> not unreasonable to increase this limit, but would increase the book-
> keeping overhead.
>
> Each additional event field takes up 10 bytes, plus the value field
> (from the 256kb quoted above).
>
>
> > >  +  Number of values for a single event field (e.g, a packet sent
> to
> > > 1000 IP
> > > addresses) (128? 255?)
> >
> Well, in the last incarnation of the spec, there was no arrays of
> values for a single event field. This is not supported by syslog-ng
> right now, unless some kind of serialization is performed in the value.
>
>
> > >  +  Value (string) size (1KB? 10KB?)
> >
> This is 64k right now.
>
> But everything adds up into the same 256kB limit.
>
> -- Bazsi
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Ruben Oliva
In reply to this post by heinbockel
To all:
 
I think the issue of the event size log was not fully addressed.

We may be able to come up with a standard scheme to identify log file sizes instead of a standard size.

 

For instance, using byte-width as a mechanism.

For log files which are less than 1 Megabyte (2**20 bytes), no size to be specified.

For log files larger than 1 Megabyte specify size as follows:

Event log File size (bytes)

Byte width

Standard size

 

1 Megabyte

2**20

20

 

2 Megabytes

2**21

21

 

4 Megabytes

2**22

22

 

8 Megabytes

2**23

23

 

16 Megabytes

2**24

24

 

 

The size of files would not have a limit, but there would be an orderly mechanism for compliant platforms to communicate.

As a use-case example, in an XML log management application a log file with a size of 8 Megabytes would look like this:

<LogFileSize>23</LogFileSize>.

 

A Log file of size 16 Megabytes would like <LogFileSize>24</LogFileSize>, and so on.

 

Advantages:

1.       No proprietary scheme is favored.

2.       Indefinitely extensible.

3.       Supports present and future size logs.

4.       Some file sizes could theoretically be optimized with hardware.

 
David Oliva


Nov 29, 2010 03:48:56 AM, [hidden email] wrote:
Hello,

Bellow is the response from syslog-ng's lead developer.

Bye,
CzP

On 11/22/2010 11:08 PM, Heinbockel, Bill wrote:

> CEE is looking to recommend sizing constraints for event records to better
> > enable compatibility.
> >
> > What are your thoughts on reasonable maximum limits for the following:
> > + Event record size (1MB? 10MB? 100KB?)
>
Right now, syslog-ng's internal representation limits all fields of an
event record to 256kB, but it uses some of this for book-keeping. It is
not unreasonable to increase this limit, but would increase the
book-keeping overhead.

Each additional event field takes up 10 bytes, plus the value field
(from the 256kb quoted above).


> > + Number of values for a single event field (e.g, a packet sent to 1000 IP
> > addresses) (128? 255?)
>
Well, in the last incarnation of the spec, there was no arrays of values
for a single event field. This is not supported by syslog-ng right now,
unless some kind of serialization is performed in the value.


> > + Value (string) size (1KB? 10KB?)
>
This is 64k right now.

But everything adds up into the same 256kB limit.

-- Bazsi
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Bill Scherr IV
Folks...

   Forgive me for jumping in the middle here.  Given the rediculous size and power of modern computers streaming a file through
a small, but flexible filtering mechanism would seem most valuable.  

   In this case, I see time as more important than size.  It is more easily matched to human events and queries!

B.

Circa 18:54, 6 Dec 2010, a note, claiming source [hidden email] <[hidden email]>, was sent to me:

From:           [hidden email]
Subject:         Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
To:             [hidden email]

> To all:
>
> I think the issue of the event size log was not fully addressed.
> We may be able to come up with a standard scheme to identify log file sizes instead of a standard size.
>
> For instance, using byte-width as a mechanism.
> For log files which are less than 1 Megabyte (2**20 bytes), no size to be specified.
> For log files larger than 1 Megabyte specify size as follows:

{snipped}


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies
[hidden email]
[hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Eric Fitzgerald
The question was not about the size of event LOGS, it was about the size of event RECORDS.

Raising an event requires resources- it's not just about log size.

For example, let's say that I want to raise an event that contains 10 strings, and I want to put 1k of data in each, except for one string where I want to put in 1MB of data.  This means that, at runtime, when I have a piece of code that is doing something to support my business but needs logging, I must stop doing work long enough to allocate a buffer, fill it with the 1MB of data, pass along over 1MB of data via some mechanism, allocate memory in the logging process for it, format the data, and deliver it to the file system.  There is a nontrivial likelihood that I will fail to allocate memory.  Then I have all sorts of bad decisions to make, like whether to fail/roll back the operation that was being logged or proceed without logging, and in the case of security logs on common criteria compliant systems, whether to halt the system (e.g. CrashOnAuditFail in Windows).

Currently, Windows and Syslog put upper boundaries on event record size.

Also, remember that human beings are the ultimate consumers of practically all log data (otherwise we generally don't use a log but rather a messaging protocol).  There is no realistic way for a logging utility to display a 1MB event- you just have to dump the 1MB of data to screen.  You have to use non-log utilities to actually do anything useful with the data, e.g. grep or some text editor with a find command, etc.

What the board is asking is whether we can put an upper boundary on the size of an individual event record.

We also would like to put an upper boundary on:
1. the number of fields in a particular event record
2. the number of data items in a particular field if it is formatted as a list
3. the total size of a particular field

We are thinking about large numbers, in the tens of kilobytes or higher for record size, and in the hundreds of fields per record and items per field.

But we want to know if there is some use case that such limits would break.

Thanks!
Eric





________________________________________
From: Bill Scherr IV [[hidden email]]
Sent: Monday, December 06, 2010 5:58 PM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

Folks...

   Forgive me for jumping in the middle here.  Given the rediculous size and power of modern computers streaming a file through
a small, but flexible filtering mechanism would seem most valuable.

   In this case, I see time as more important than size.  It is more easily matched to human events and queries!

B.

Circa 18:54, 6 Dec 2010, a note, claiming source [hidden email] <[hidden email]>, was sent to me:

From:                   [hidden email]
Subject:                Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
To:                     [hidden email]

> To all:
>
> I think the issue of the event size log was not fully addressed.
> We may be able to come up with a standard scheme to identify log file sizes instead of a standard size.
>
> For instance, using byte-width as a mechanism.
> For log files which are less than 1 Megabyte (2**20 bytes), no size to be specified.
> For log files larger than 1 Megabyte specify size as follows:

{snipped}


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies
[hidden email]
[hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Bill Scherr IV
Eric...

   I got that the subject matter was records (or entries), after I hit the send button!  

   A log entry should not require 1MB of data, for more reasons than the complexity required to post it.  As implied in my
post, matching to human events and queries is key.  So we agree that humans are the primary log consumers.  

   Overly complex log entries indicate systemic (i.e. task organization) problems.  Programmers are aware of the 256K
truncation in syslog, and most keep entries well shorter than that.  Sometimes the best way to assure effectiveness is to
impose a limit.  I think I sense some agreement here as well.

   Windows logging has always been overly flexible in possible entry (record) triggers, and overly restrictive in the available
reading utilities.  Common utilities (like pagers; more or less) are necessary in summarizing and analyzing logs, and therefore
should be included in the basic set of expected functions.  Building "security" in requires some intelligence be applied to the
administrative output (logs) by the system architects.  One element of intelligence is ASCII.

    Simple logs are vulnerable on the generating host.  One of syslog's strengths is the simple method of getting logs onto a
machine that does not participate in the same user management domain as the generators, near the time of the event.  If the
programmers get too complex, the end users cannot build and/or operate to achieve the security for which they are rightly
responsible.  This state highlights the fact that end user understanding of their fancy calculators (or typewriters) is key.  

   The bottom line is the human reasons for truncating log entries, and keeping them in text format.  

HTH

B.

Circa 5:17, 7 Dec 2010, a note, claiming source Eric Fitzgerald <[hidden email]>, was sent to me:

From:           Eric Fitzgerald <[hidden email]>
To:             "[hidden email]" <[hidden email]>,
        "[hidden email]" <[hidden email]>
Subject:         RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
Date sent:       Tue, 7 Dec 2010 05:17:17 +0000

> The question was not about the size of event LOGS, it was about the size of event RECORDS.
>
> Raising an event requires resources- it's not just about log size.
>
> For example, let's say that I want to raise an event that contains 10
> strings, and I want to put 1k of data in each, except for one string where
> I want to put in 1MB of data.  This means that, at runtime, when I have a
> piece of code that is doing something to support my business but needs
> logging, I must stop doing work long enough to allocate a buffer, fill it
> with the 1MB of data, pass along over 1MB of data via some mechanism,
> allocate memory in the logging process for it, format the data, and
> deliver it to the file system.  There is a nontrivial likelihood that I
> will fail to allocate memory.  Then I have all sorts of bad decisions to
> make, like whether to fail/roll back the operation that was being logged
> or proceed without logging, and in the case of security logs on common
> criteria compliant systems, whether to halt the system (e.g.
> CrashOnAuditFail in Windows).
>
> Currently, Windows and Syslog put upper boundaries on event record size.
>
> Also, remember that human beings are the ultimate consumers of practically
> all log data (otherwise we generally don't use a log but rather a
> messaging protocol).  There is no realistic way for a logging utility to
> display a 1MB event- you just have to dump the 1MB of data to screen.  You
> have to use non-log utilities to actually do anything useful with the
> data, e.g. grep or some text editor with a find command, etc.
>
> What the board is asking is whether we can put an upper boundary on the size of an individual event record.
>
> We also would like to put an upper boundary on:
> 1. the number of fields in a particular event record
> 2. the number of data items in a particular field if it is formatted as a list
> 3. the total size of a particular field
>
> We are thinking about large numbers, in the tens of kilobytes or higher
> for record size, and in the hundreds of fields per record and items per
> field.
>
> But we want to know if there is some use case that such limits would break.
>
> Thanks!
> Eric
>
>
>
>
>
> ________________________________________
> From: Bill Scherr IV [[hidden email]]
> Sent: Monday, December 06, 2010 5:58 PM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
> Folks...
>
>    Forgive me for jumping in the middle here.  Given the rediculous size and power of modern computers streaming a file through
> a small, but flexible filtering mechanism would seem most valuable.
>
>    In this case, I see time as more important than size.  It is more easily matched to human events and queries!
>
> B.
>
> Circa 18:54, 6 Dec 2010, a note, claiming source [hidden email] <[hidden email]>, was sent to me:
>
> From:                   [hidden email]
> Subject:                Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
> To:                     [hidden email]
>
> > To all:
> >
> > I think the issue of the event size log was not fully addressed.
> > We may be able to come up with a standard scheme to identify log file sizes instead of a standard size.
> >
> > For instance, using byte-width as a mechanism.
> > For log files which are less than 1 Megabyte (2**20 bytes), no size to be specified.
> > For log files larger than 1 Megabyte specify size as follows:
>
> {snipped}
{sig snipped}


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies
[hidden email]
[hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Eric Fitzgerald
Thanks Bill, very insightful!

-----Original Message-----
From: Bill Scherr IV [mailto:[hidden email]]
Sent: Tuesday, December 07, 2010 10:03 AM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits

Eric...

   I got that the subject matter was records (or entries), after I hit the send button!  

   A log entry should not require 1MB of data, for more reasons than the complexity required to post it.  As implied in my post, matching to human events and queries is key.  So we agree that humans are the primary log consumers.  

   Overly complex log entries indicate systemic (i.e. task organization) problems.  Programmers are aware of the 256K truncation in syslog, and most keep entries well shorter than that.  Sometimes the best way to assure effectiveness is to impose a limit.  I think I sense some agreement here as well.

   Windows logging has always been overly flexible in possible entry (record) triggers, and overly restrictive in the available reading utilities.  Common utilities (like pagers; more or less) are necessary in summarizing and analyzing logs, and therefore should be included in the basic set of expected functions.  Building "security" in requires some intelligence be applied to the administrative output (logs) by the system architects.  One element of intelligence is ASCII.

    Simple logs are vulnerable on the generating host.  One of syslog's strengths is the simple method of getting logs onto a machine that does not participate in the same user management domain as the generators, near the time of the event.  If the programmers get too complex, the end users cannot build and/or operate to achieve the security for which they are rightly responsible.  This state highlights the fact that end user understanding of their fancy calculators (or typewriters) is key.  

   The bottom line is the human reasons for truncating log entries, and keeping them in text format.  

HTH

B.

Circa 5:17, 7 Dec 2010, a note, claiming source Eric Fitzgerald <[hidden email]>, was sent to me:

From:           Eric Fitzgerald <[hidden email]>
To:             "[hidden email]" <[hidden email]>,
        "[hidden email]" <[hidden email]>
Subject:         RE: [CEE-DISCUSSION-LIST] Event Record Size Limits
Date sent:       Tue, 7 Dec 2010 05:17:17 +0000

> The question was not about the size of event LOGS, it was about the size of event RECORDS.
>
> Raising an event requires resources- it's not just about log size.
>
> For example, let's say that I want to raise an event that contains 10
> strings, and I want to put 1k of data in each, except for one string
> where I want to put in 1MB of data.  This means that, at runtime, when
> I have a piece of code that is doing something to support my business
> but needs logging, I must stop doing work long enough to allocate a
> buffer, fill it with the 1MB of data, pass along over 1MB of data via
> some mechanism, allocate memory in the logging process for it, format
> the data, and deliver it to the file system.  There is a nontrivial
> likelihood that I will fail to allocate memory.  Then I have all sorts
> of bad decisions to make, like whether to fail/roll back the operation
> that was being logged or proceed without logging, and in the case of
> security logs on common criteria compliant systems, whether to halt the system (e.g.
> CrashOnAuditFail in Windows).
>
> Currently, Windows and Syslog put upper boundaries on event record size.
>
> Also, remember that human beings are the ultimate consumers of
> practically all log data (otherwise we generally don't use a log but
> rather a messaging protocol).  There is no realistic way for a logging
> utility to display a 1MB event- you just have to dump the 1MB of data
> to screen.  You have to use non-log utilities to actually do anything
> useful with the data, e.g. grep or some text editor with a find command, etc.
>
> What the board is asking is whether we can put an upper boundary on the size of an individual event record.
>
> We also would like to put an upper boundary on:
> 1. the number of fields in a particular event record 2. the number of
> data items in a particular field if it is formatted as a list 3. the
> total size of a particular field
>
> We are thinking about large numbers, in the tens of kilobytes or
> higher for record size, and in the hundreds of fields per record and
> items per field.
>
> But we want to know if there is some use case that such limits would break.
>
> Thanks!
> Eric
>
>
>
>
>
> ________________________________________
> From: Bill Scherr IV [[hidden email]]
> Sent: Monday, December 06, 2010 5:58 PM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>
> Folks...
>
>    Forgive me for jumping in the middle here.  Given the rediculous
> size and power of modern computers streaming a file through a small, but flexible filtering mechanism would seem most valuable.
>
>    In this case, I see time as more important than size.  It is more easily matched to human events and queries!
>
> B.
>
> Circa 18:54, 6 Dec 2010, a note, claiming source [hidden email] <[hidden email]>, was sent to me:
>
> From:                   [hidden email]
> Subject:                Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
> To:                     [hidden email]
>
> > To all:
> >
> > I think the issue of the event size log was not fully addressed.
> > We may be able to come up with a standard scheme to identify log file sizes instead of a standard size.
> >
> > For instance, using byte-width as a mechanism.
> > For log files which are less than 1 Megabyte (2**20 bytes), no size to be specified.
> > For log files larger than 1 Megabyte specify size as follows:
>
> {snipped}
{sig snipped}


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies [hidden email] [hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|

Re: Event Record Size Limits

Adam Montville
In reply to this post by heinbockel
Sorry to nit pick, and a bit late at that, but I read "IT/electronic
system" as "IT and/or electronic" system.  Presumably, electronic (I.e.
Video) surveillance systems are often tied to IT systems.  It's fine that
we limit to IT events (I.e. Hardware, OS, application that would be
identified by CPE) if that's the scope, but the way the scoping is worded
seems misleading.

Adam

On 11/29/10 7:37 AM, "Heinbockel, Bill" <[hidden email]> wrote:

>The scoping of CEE is currently for "events observed and generated by an
>IT/electronic system"
>
>This is not saying that CEE cannot be used elsewhere, but our focus is on
>IT
>events
>
>
>William Heinbockel
>The MITRE Corporation
>
>
>>-----Original Message-----
>>From: Adam Montville [mailto:[hidden email]]
>>Sent: Tuesday, 23 November 2010 17:45
>>To: Heinbockel, Bill; cee-discussion-list CEE-Related Discussion
>>Subject: Re: [CEE-DISCUSSION-LIST] Event Record Size Limits
>>
>>This may be a newb type of question, but are we constrained to IT audit
>>events or is there some wider applicability for CEE in the future?  I can
>>conceive a physical sensor sending image and/or video data as part of a
>>record, for example.  In that case, I could see the event record size
>>being large.  If we are constrained to IT audit events, I can't see
>>anything larger than 1MB being realistic (if we see something like that,
>>it's anomalous in its own right).
>>
>>Also, it seems that the moment we truncate data, we're walking into the
>>realm of assumptions or at least guessing.  If a packet is sent to 1000
>>IP
>>addresses, but we only report 128 or even 255 of them, we don't make the
>>job of containment/remediation for issues any easier.  Of course, I could
>>be reading that question wrong.
>>
>>Not sure about the last question.
>>
>>Adam
>>
>>On 11/22/10 2:08 PM, "Heinbockel, Bill" <[hidden email]> wrote:
>>
>>>CEE is looking to recommend sizing constraints for event records to
>>>better
>>>enable compatibility.
>>>
>>>What are your thoughts on reasonable maximum limits for the following:
>>> +  Event record size (1MB? 10MB? 100KB?)
>>> +  Number of values for a single event field (e.g, a packet sent to
>>>1000
>>>IP
>>>addresses) (128? 255?)
>>> +  Value (string) size (1KB? 10KB?)
>>>
>>>
>>>Thank you,
>>>
>>>William Heinbockel
>>>Infosec Engineer, Sr.
>>>The MITRE Corporation
>>>202 Burlington Rd. MS S145
>>>Bedford, MA 01730
>>>[hidden email]
>>>781-271-2615
>>>
>>>
>>
>
12