Quantcast

Structured vs. Free-form Event Messages

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Structured vs. Free-form Event Messages

Bill Scherr IV
There are three tools that will work on either (freeform or structured)!.  They are grep, awk and sed.  They can form any
report you want.  If you put the output through less, you get immediate basic summarization and optimal viewing, providing your
"terminal" is wide enough.  Binary data requires decoding in any case!  That should be provided by the generators.

Obtaining Enterprise coverage is more of a human and network establishment task in the first place.  The big problems are
trusting the network you attach to, and allowing administrative support.  These are human issues, both with authority and skill
limits.  Structure of the logs will not address these.

Adding additional "structure" to the entry only makes them harder to read by hand.  When you can get to them, you should be
able to consolidate them.  The best database in the world for that is the ext4 file system.  Reiser may be better, but ext4 is
close to the basic ext2, making it the default in kernels in lightweight machines.  In any case, one will have to engineer a
filter.  Those filters will be close to POSIX regular expressions.  An order may need to be imposed.  Awk can stream any size
file.  Sed also carries the same memory limits (none).

If you need to get real fancy, there is PERL.  In any case, none of these tools reduce the work that needs to be done.  They
just add complexity!

my 2¢


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies
[hidden email]
[hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Structured vs. Free-form Event Messages

Eric Fitzgerald
The ability to use text parsing tools such as those below involves significant human effort and is very failure prone.  For instance, it's hard to determine from a log sample whether you have every format variation of every event that the software can output, so even during regex development there is a completeness problem.  Then of course, minor changes to event message strings or addition of new messages breaks the automation.

I disagree that structure reduces human readability.  This is a tools problem.

For instance, nobody uses a hex editor to manipulate SQL databases.  However we think that using a text editor is the best solution for log analysis?

Windows provides one possible model.  I am not asserting that it is the best model but it addresses a number of problems.

Event storage only includes instance-specific metadata.  The static text messages used to put the events into context, are stored separately and the metadata and messages are combined at view time by Event Viewer and underlying APIs.  This reduces storage requirements, allows structured indexing and searching, and as an additional bonus, allows very cheap localization that won't break automation.

I have advocated for a similar solution in CEE - the current CEE spec doesn't have very good support for static text in messages, much less localization - but the board hasn't seemed to want to build those extensions into v1.

Eric Fitzgerald
Microsoft Corporation

-----Original Message-----
From: Bill Scherr IV [mailto:[hidden email]]
Sent: Thursday, January 26, 2012 4:33 PM
To: [hidden email]
Subject: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages

There are three tools that will work on either (freeform or structured)!.  They are grep, awk and sed.  They can form any report you want.  If you put the output through less, you get immediate basic summarization and optimal viewing, providing your "terminal" is wide enough.  Binary data requires decoding in any case!  That should be provided by the generators.

Obtaining Enterprise coverage is more of a human and network establishment task in the first place.  The big problems are trusting the network you attach to, and allowing administrative support.  These are human issues, both with authority and skill limits.  Structure of the logs will not address these.

Adding additional "structure" to the entry only makes them harder to read by hand.  When you can get to them, you should be able to consolidate them.  The best database in the world for that is the ext4 file system.  Reiser may be better, but ext4 is close to the basic ext2, making it the default in kernels in lightweight machines.  In any case, one will have to engineer a filter.  Those filters will be close to POSIX regular expressions.  An order may need to be imposed.  Awk can stream any size file.  Sed also carries the same memory limits (none).

If you need to get real fancy, there is PERL.  In any case, none of these tools reduce the work that needs to be done.  They just add complexity!

my 2¢


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies [hidden email] [hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Structured vs. Free-form Event Messages

Bill Scherr IV
All...

One must have the ability to see the context of the system to get anything out of a subset of log entries.  Nothing happens in
a vacuum.  Therefore, understanding any log will require the same amount of effort.  Which system is immaterial.

The "regex" development MUST be ongoing to keep up with changes in the system.  In order to fully answer that "requirement" a
team of engineers is necessary.  This is true of any system.  One of the driving forces is product versions, but threat pokes
in, as well as business imperatives.  The log parser will break, eventually, and require attention of one sort or another.  The
fancier the interface, the more to translate/understand!  A million dollar system without the flexibility of grep is... [].

Running a system without an experienced system administrator is a risk in and of itself, and should not be encouraged, to say
nothing of other responses.  The processes I listed work on standard in and standard out!  The limits of that interface should
not need discussion here!  Standard out is one of the things any system admin should understand!

Human readability is but one of the issues with formatted logs.  Every event will have a format.  That is the nature of a
process.  Matching a format will increase complexity at design, development, integration, deployment, operations,  management
and others.  IMHO, the biggest vector of this effort is a barrier to entry.

I am not against introducing complexity for securities AND/OR functionalities sake.  We still have not addressed the core
impediment to enterprise coverage.  That is poor deployment planning, human authority and human access distribution.  I think
we can all agree that a log format will not do that!  In any case, the wall that is not watched by humans is the easier wall to
break.

Bill H broached the question.  I assumed that meant the subject was open again!  I think this clarifies by original 2¢, but
will offer to cover any extra!

B.

Circa 0:36, 27 Jan 2012, a note, claiming source Eric Fitzgerald <[hidden email]>, was sent to me:

Date sent:       Fri, 27 Jan 2012 00:36:15 +0000
Send reply to:   Eric Fitzgerald <[hidden email]>
From:           Eric Fitzgerald <[hidden email]>
Subject:         Re: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages
To:             <[hidden email]>

> The ability to use text parsing tools such as those below involves
> significant human effort and is very failure prone.  For instance, it's
> hard to determine from a log sample whether you have every format
> variation of every event that the software can output, so even during
> regex development there is a completeness problem.  Then of course, minor
> changes to event message strings or addition of new messages breaks the
> automation.
>
> I disagree that structure reduces human readability.  This is a tools
> problem.
>
> For instance, nobody uses a hex editor to manipulate SQL databases.
> However we think that using a text editor is the best solution for log
> analysis?
>
> Windows provides one possible model.  I am not asserting that it is the
> best model but it addresses a number of problems.
>
> Event storage only includes instance-specific metadata.  The static text
> messages used to put the events into context, are stored separately and
> the metadata and messages are combined at view time by Event Viewer and
> underlying APIs.  This reduces storage requirements, allows structured
> indexing and searching, and as an additional bonus, allows very cheap
> localization that won't break automation.
>
> I have advocated for a similar solution in CEE - the current CEE spec
> doesn't have very good support for static text in messages, much less
> localization - but the board hasn't seemed to want to build those
> extensions into v1.
>
> Eric Fitzgerald
> Microsoft Corporation
>
> -----Original Message-----
> From: Bill Scherr IV [mailto:[hidden email]]
> Sent: Thursday, January 26, 2012 4:33 PM
> To: [hidden email]
> Subject: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages
>
> There are three tools that will work on either (freeform or structured)!.  They are grep, awk and sed.  They can form any report you want.  If you put the output through less, you get immediate basic summarization and optimal viewing, providing your "terminal" is wide enough.  Binary data requires decoding in any case!  That should be provided by the generators.
>
> Obtaining Enterprise coverage is more of a human and network establishment task in the first place.  The big problems are trusting the network you attach to, and allowing administrative support.  These are human issues, both with authority and skill limits.  Structure of the logs will not address these.
>
> Adding additional "structure" to the entry only makes them harder to read by hand.  When you can get to them, you should be able to consolidate them.  The best database in the world for that is the ext4 file system.  Reiser may be better, but ext4 is close to the basic ext2, making it the default in kernels in lightweight machines.  In any case, one will have to engineer a filter.  Those filters will be close to POSIX regular expressions.  An order may need to be imposed.  Awk can stream any size file.  Sed also carries the same memory limits (none).
>
> If you need to get real fancy, there is PERL.  In any case, none of these tools reduce the work that needs to be done.  They just add complexity!
>
> my 2¢
>
>
> Bill Scherr IV, GSEC, GCIA
> Principal Security Engineer
> EWA Information and Infrastructure Technologies [hidden email] [hidden email]
> 703-478-7608
>
>


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies
[hidden email]
[hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Structured vs. Free-form Event Messages

Clayton Dukes (cdukes)
In reply to this post by Bill Scherr IV
I've been trying to read all of the comments, but forgive me if I passed over any that mentioned this.
It seems that some folks are not considering scale very much here. I work with customers that have a *lot* of log data to parse through - hundreds of millions or, in some cases, trillions of messages per day. If we grow the size of each message, how much will the customer need to store? In PCI compliant environments, that number is a min of one year (and I have a few customers that require 7).
I love sed, awk, grep, perl etc. as all nerds do, but they aren't the best tools for the job that level. In my "night job", I also write/maintain a tool known as LogZilla. I hit all of these boundaries many years ago (the tool is about 11 or so years old). I couldn't store much more than 1 or 2 mil rows of data without it taking too long to search through, build charts, etc. Thankfully, these days, I can handle about 200-300m per day - but if the standards change to having a single log message 10x the current size (just throwing a number out), then please consider the impact this will have on the admins, analyzers, storage, etc.

FWIW, in my "day job", I interact with the largest customers in the world - the number 1 problem I see is that there is just so much of it. They stick their heads in the sand, ignore their logs, and think it won't affect them - until it does. The next question I always get is "can you tell us what messages to look for?" - my answer is always the same. Yes, all of them.

P.S. If any of you are attending Cisco Live in London next week, stop by and see me at session BRKNMS-2031 - it's always nice to meet fellow logging nerds :-)


-----Original Message-----
From: Bill Scherr IV [mailto:[hidden email]]
Sent: Thursday, January 26, 2012 7:33 PM
To: [hidden email]
Subject: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages

There are three tools that will work on either (freeform or structured)!.  They are grep, awk and sed.  They can form any report you want.  If you put the output through less, you get immediate basic summarization and optimal viewing, providing your "terminal" is wide enough.  Binary data requires decoding in any case!  That should be provided by the generators.

Obtaining Enterprise coverage is more of a human and network establishment task in the first place.  The big problems are trusting the network you attach to, and allowing administrative support.  These are human issues, both with authority and skill limits.  Structure of the logs will not address these.

Adding additional "structure" to the entry only makes them harder to read by hand.  When you can get to them, you should be able to consolidate them.  The best database in the world for that is the ext4 file system.  Reiser may be better, but ext4 is close to the basic ext2, making it the default in kernels in lightweight machines.  In any case, one will have to engineer a filter.  Those filters will be close to POSIX regular expressions.  An order may need to be imposed.  Awk can stream any size file.  Sed also carries the same memory limits (none).

If you need to get real fancy, there is PERL.  In any case, none of these tools reduce the work that needs to be done.  They just add complexity!

my 2¢


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies [hidden email] [hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Structured vs. Free-form Event Messages

Natale, Bob
Yes, increasing scale has consequences for capacity, performance, reliability, human intervention, and data integrity management (which itself carries a very large set of requirements, irrespective of scale,  that are often overlooked up front).  Those consequences can be dealt with effectively, but that does not come easily or cheaply.

Wish-list view: I generally favor more structure, more intelligent instrumentation, and more capable tools ... natural language inputs only when comprehensively supported with appropriate semantic technologies.

Cheers,
BobN

-----Original Message-----
From: Clayton Dukes (cdukes) [mailto:[hidden email]]
Sent: Thursday, January 26, 2012 10:00 PM
To: cee-discussion-list CEE-Related Discussion
Subject: Re: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages

I've been trying to read all of the comments, but forgive me if I passed over any that mentioned this.
It seems that some folks are not considering scale very much here. I work with customers that have a *lot* of log data to parse through - hundreds of millions or, in some cases, trillions of messages per day. If we grow the size of each message, how much will the customer need to store? In PCI compliant environments, that number is a min of one year (and I have a few customers that require 7).
I love sed, awk, grep, perl etc. as all nerds do, but they aren't the best tools for the job that level. In my "night job", I also write/maintain a tool known as LogZilla. I hit all of these boundaries many years ago (the tool is about 11 or so years old). I couldn't store much more than 1 or 2 mil rows of data without it taking too long to search through, build charts, etc. Thankfully, these days, I can handle about 200-300m per day - but if the standards change to having a single log message 10x the current size (just throwing a number out), then please consider the impact this will have on the admins, analyzers, storage, etc.

FWIW, in my "day job", I interact with the largest customers in the world - the number 1 problem I see is that there is just so much of it. They stick their heads in the sand, ignore their logs, and think it won't affect them - until it does. The next question I always get is "can you tell us what messages to look for?" - my answer is always the same. Yes, all of them.

P.S. If any of you are attending Cisco Live in London next week, stop by and see me at session BRKNMS-2031 - it's always nice to meet fellow logging nerds :-)


-----Original Message-----
From: Bill Scherr IV [mailto:[hidden email]]
Sent: Thursday, January 26, 2012 7:33 PM
To: [hidden email]
Subject: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages

There are three tools that will work on either (freeform or structured)!.  They are grep, awk and sed.  They can form any report you want.  If you put the output through less, you get immediate basic summarization and optimal viewing, providing your "terminal" is wide enough.  Binary data requires decoding in any case!  That should be provided by the generators.

Obtaining Enterprise coverage is more of a human and network establishment task in the first place.  The big problems are trusting the network you attach to, and allowing administrative support.  These are human issues, both with authority and skill limits.  Structure of the logs will not address these.

Adding additional "structure" to the entry only makes them harder to read by hand.  When you can get to them, you should be able to consolidate them.  The best database in the world for that is the ext4 file system.  Reiser may be better, but ext4 is close to the basic ext2, making it the default in kernels in lightweight machines.  In any case, one will have to engineer a filter.  Those filters will be close to POSIX regular expressions.  An order may need to be imposed.  Awk can stream any size file.  Sed also carries the same memory limits (none).

If you need to get real fancy, there is PERL.  In any case, none of these tools reduce the work that needs to be done.  They just add complexity!

my 2¢


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies [hidden email] [hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Structured vs. Free-form Event Messages

Bill Scherr IV
In reply to this post by Clayton Dukes (cdukes)
I'm wondering what would happen if they added another "Clayton" to the system below.  Of course that person could realize some
economies of scale, as the rest of the network would not have to be duplicated, just the log analysis equipment and tools!  I
know it is hard to tell what the trends or threat will be next year, but giving a "Clayton" a sanity check would certainly NOT
slow down or reduce that vision.  One or two million lines translates to a ~200 MB log file.  Every day?  Well the filter I
wrote yesterday will be a starting point.  It will probably change today.  What will be the vectors causing log line growth?  
Growth of a system can be handled on a day to day basis.  It is on a month to month or year to year that additional resources
(brains) need to be considered.  This will occur regardless of the tool!

Can you tell me what messages to look for next month?  Have you tried Cron?

Can you give me a tool that will tell managers what they are giving access to, and convince them to give it?  How about next
month when they get that killer app installed?  The scale that does not work is depending on outside vendors.  There is
currently little continuity in system knowledge.

The trick is not to summarize or make a chart.  The trick is to know that you are wiping away the loud covering scan, and see
the actual threats.  In any case, we are only confirming that our actions worked with system logs.   Hows that "all threat"
wirespeed signature engine coming?

I think we all interact with the "biggest" customers in the world here.  Some of us are just independent.

B.

Circa 20:59, 26 Jan 2012, a note, claiming source Clayton Dukes (cdukes) <[hidden email]>, was sent to me:

Subject:         RE: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages
Date sent:       Thu, 26 Jan 2012 20:59:57 -0600
From:           "Clayton Dukes (cdukes)" <[hidden email]>
To:             <[hidden email]>, <[hidden email]>

> I've been trying to read all of the comments, but forgive me if I passed over any that mentioned this.
> It seems that some folks are not considering scale very much here. I work with customers that have a *lot* of log data to parse
> through - hundreds of millions or, in some cases, trillions of messages per day. If we grow the size of each message, how much
> will the customer need to store? In PCI compliant environments, that number is a min of one year (and I have a few customers
> that require 7). I love sed, awk, grep, perl etc. as all nerds do, but they aren't the best tools for the job that level. In my
> "night job", I also write/maintain a tool known as LogZilla. I hit all of these boundaries many years ago (the tool is about 11
> or so years old). I couldn't store much more than 1 or 2 mil rows of data without it taking too long to search through, build
> charts, etc. Thankfully, these days, I can handle about 200-300m per day - but if the standards change to having a single log
> message 10x the current size (just throwing a number out), then please consider the impact this will have on the admins,
> analyzers, storage, etc.
>
> FWIW, in my "day job", I interact with the largest customers in the world - the number 1 problem I see is that there is just so
> much of it. They stick their heads in the sand, ignore their logs, and think it won't affect them - until it does. The next
> question I always get is "can you tell us what messages to look for?" - my answer is always the same. Yes, all of them.
>
> P.S. If any of you are attending Cisco Live in London next week, stop by and see me at session BRKNMS-2031 - it's always nice to
> meet fellow logging nerds :-)
>


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies
[hidden email]
[hidden email]
703-478-7608
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Structured vs. Free-form Event Messages

Keith Robertson
In reply to this post by Eric Fitzgerald
On 01/26/2012 07:36 PM, Eric Fitzgerald wrote:
> The ability to use text parsing tools such as those below involves significant human effort and is very failure prone.  For instance, it's hard to determine from a log sample whether you have every format variation of every event that the software can output, so even during regex development there is a completeness problem.  Then of course, minor changes to event message strings or addition of new messages breaks the automation.
>
> I disagree that structure reduces human readability.  This is a tools problem.
Completely agree and I'm not arguing for completely unstructured
logging.  Rather, I think that we need to strictly set the requirement
for the structures that must be in every good CEE <Event/>.  These
"required" structures should be those that facilitate automated parsing
(e.g. id, timestamp, status, hostname, etc.).  There should also be
optional structures for programs that want to pass message text,
function arguments, etc.

> For instance, nobody uses a hex editor to manipulate SQL databases.  However we think that using a text editor is the best solution for log analysis?
>
> Windows provides one possible model.  I am not asserting that it is the best model but it addresses a number of problems.
>
> Event storage only includes instance-specific metadata.
Partially agree with this.  This model works really well when there is a
centralized dictionary that can correlate an event ID to a canned
message but, it presumes that there is a centralized dictionary with a
mapping of IDs to canned messages.  Some small projects may never have
the capability to do this; hence, the <Event/> needs to have enough
information that it can be somewhat self describing.

>    The static text messages used to put the events into context, are stored separately and the metadata and messages are combined at view time by Event Viewer and underlying APIs.
This is an excellent design paradigm and one that CEE should certainly
facilitate.  However, I also think that CEE should be flexible enough to
facilitate programs that wish to place the message directly in the event.
>   This reduces storage requirements, allows structured indexing and searching, and as an additional bonus, allows very cheap localization that won't break automation.
You can still have messages in a structured event that won't break
automation.  Look at the <LogText/> element in IBM's TMS standard:

Example: <LogText><![CDATA[HPDCO1054E Could not connect to the server
acld2, on port 7137. ]]]</LogText>

> I have advocated for a similar solution in CEE - the current CEE spec doesn't have very good support for static text in messages, much less localization - but the board hasn't seemed to want to build those extensions into v1.
I think you'll get a resounding ACK from RH.

> Eric Fitzgerald
> Microsoft Corporation
>
> -----Original Message-----
> From: Bill Scherr IV [mailto:[hidden email]]
> Sent: Thursday, January 26, 2012 4:33 PM
> To:[hidden email]
> Subject: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages
>
> There are three tools that will work on either (freeform or structured)!.  They are grep, awk and sed.  They can form any report you want.  If you put the output through less, you get immediate basic summarization and optimal viewing, providing your "terminal" is wide enough.  Binary data requires decoding in any case!  That should be provided by the generators.
>
> Obtaining Enterprise coverage is more of a human and network establishment task in the first place.  The big problems are trusting the network you attach to, and allowing administrative support.  These are human issues, both with authority and skill limits.  Structure of the logs will not address these.
>
> Adding additional "structure" to the entry only makes them harder to read by hand.  When you can get to them, you should be able to consolidate them.  The best database in the world for that is the ext4 file system.  Reiser may be better, but ext4 is close to the basic ext2, making it the default in kernels in lightweight machines.  In any case, one will have to engineer a filter.  Those filters will be close to POSIX regular expressions.  An order may need to be imposed.  Awk can stream any size file.  Sed also carries the same memory limits (none).
>
> If you need to get real fancy, there is PERL.  In any case, none of these tools reduce the work that needs to be done.  They just add complexity!
>
> my 2¢
>
>
> Bill Scherr IV, GSEC, GCIA
> Principal Security Engineer
> EWA Information and Infrastructure [hidden email]  [hidden email]
> 703-478-7608
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Structured vs. Free-form Event Messages

Burnes, James - NRCS, Fort Collins, CO
In reply to this post by Eric Fitzgerald
Eric.... agreed.

Thought I think the specific implementation of the event database is beyond the scope of the CEE taxonomy, dictionary and serialization formats (the "standard").

Jim Burnes

________________________________________
From: Eric Fitzgerald [[hidden email]]
Sent: Thursday, January 26, 2012 5:36 PM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages

The ability to use text parsing tools such as those below involves significant human effort and is very failure prone.  For instance, it's hard to determine from a log sample whether you have every format variation of every event that the software can output, so even during regex development there is a completeness problem.  Then of course, minor changes to event message strings or addition of new messages breaks the automation.

I disagree that structure reduces human readability.  This is a tools problem.

For instance, nobody uses a hex editor to manipulate SQL databases.  However we think that using a text editor is the best solution for log analysis?

Windows provides one possible model.  I am not asserting that it is the best model but it addresses a number of problems.

Event storage only includes instance-specific metadata.  The static text messages used to put the events into context, are stored separately and the metadata and messages are combined at view time by Event Viewer and underlying APIs.  This reduces storage requirements, allows structured indexing and searching, and as an additional bonus, allows very cheap localization that won't break automation.

I have advocated for a similar solution in CEE - the current CEE spec doesn't have very good support for static text in messages, much less localization - but the board hasn't seemed to want to build those extensions into v1.

Eric Fitzgerald
Microsoft Corporation

-----Original Message-----
From: Bill Scherr IV [mailto:[hidden email]]
Sent: Thursday, January 26, 2012 4:33 PM
To: [hidden email]
Subject: [CEE-DISCUSSION-LIST] Structured vs. Free-form Event Messages

There are three tools that will work on either (freeform or structured)!.  They are grep, awk and sed.  They can form any report you want.  If you put the output through less, you get immediate basic summarization and optimal viewing, providing your "terminal" is wide enough.  Binary data requires decoding in any case!  That should be provided by the generators.

Obtaining Enterprise coverage is more of a human and network establishment task in the first place.  The big problems are trusting the network you attach to, and allowing administrative support.  These are human issues, both with authority and skill limits.  Structure of the logs will not address these.

Adding additional "structure" to the entry only makes them harder to read by hand.  When you can get to them, you should be able to consolidate them.  The best database in the world for that is the ext4 file system.  Reiser may be better, but ext4 is close to the basic ext2, making it the default in kernels in lightweight machines.  In any case, one will have to engineer a filter.  Those filters will be close to POSIX regular expressions.  An order may need to be imposed.  Awk can stream any size file.  Sed also carries the same memory limits (none).

If you need to get real fancy, there is PERL.  In any case, none of these tools reduce the work that needs to be done.  They just add complexity!

my 2¢


Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies [hidden email] [hidden email]
703-478-7608
Loading...