Log Protocols

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Log Protocols

heinbockel
As we all know, event records are most useful if we can collect them in a
central location for searching, correlating, and archiving. However, it is my
opinion that there are no good protocols out there to support this.

Syslog has been the de facto log transport for years. While it does have its
benefits to being small and simple, it does not provide the security and
reliability protections that some environments need. Syslog over TLS/DTLS and
other tunneled Syslog protocols only meet some of the requirements.

Other protocols, such as SOAP, are overkill for the limited amount of data most
event records contain. BEEP has gotten some renewed interest as of late, but it
seems to be complex and uses synchronous messaging channels.

RELP (http://www.librelp.com/relp.html) seems to be one of the few standouts in
this area, but I don't think it is supported beyond rsyslog.

What if you were given the opportunity to design your own log protocol?
Something that was royalty-free, non-exclusive license that was made available
with commercial-friendly licensing. What would your requirements be? Are there
protocols out there that are already "good enough"?

Here is a brief listing of some requirements to start:
  +  reliable :: message-level ACKs
  +  parallel, asynchronous batch sending :: why should I have to wait for ACKs
before sending messages
  +  batch ACKs
  +  optional encryption
  +  integrity validation
  +  anti-replay protection
  +  anti-spoof protection
  +  non-repudiation
  +  compression
  +  event record affinity :: group related event records (by type, producer,
etc.)
  +  support for structured event records  (e.g., CEE)
  +  support for semi- or un-structured event records (e.g., debug logs, Syslog)
  +  support for transmitting related data :: send the packet/PCAP with the IDS
alert; the malicious file with the Anti-Virus alert; video evidence with the
tripped alarm...



William Heinbockel
Infosec Engineer, Sr.
The MITRE Corporation
202 Burlington Rd. MS S145
Bedford, MA 01730
[hidden email]
781-271-2615



smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

Bill Frank
What about SDEE?

Most people think of it as a Cisco proprietary protocol, but the
specification is out there for anyone to implement. At my previous company
(a SIEM vendor), we had to support it. But we liked it so much, we used it
internally among the product components.  

Bill Frank
Principal
Cymbel Corporation | Next Generation Defense-in-Depth
154 Wells Avenue, Newton, MA 02459, www.cymbel.com
[hidden email]| 508-878-4943 | www.twitter.com/riskpundit


-----Original Message-----
From: Heinbockel, Bill [mailto:[hidden email]]
Sent: Thursday, January 20, 2011 5:36 PM
To: [hidden email]
Subject: [CEE-DISCUSSION-LIST] Log Protocols

As we all know, event records are most useful if we can collect them in a
central location for searching, correlating, and archiving. However, it is
my
opinion that there are no good protocols out there to support this.

Syslog has been the de facto log transport for years. While it does have its
benefits to being small and simple, it does not provide the security and
reliability protections that some environments need. Syslog over TLS/DTLS
and
other tunneled Syslog protocols only meet some of the requirements.

Other protocols, such as SOAP, are overkill for the limited amount of data
most
event records contain. BEEP has gotten some renewed interest as of late, but
it
seems to be complex and uses synchronous messaging channels.

RELP (http://www.librelp.com/relp.html) seems to be one of the few standouts
in
this area, but I don't think it is supported beyond rsyslog.

What if you were given the opportunity to design your own log protocol?
Something that was royalty-free, non-exclusive license that was made
available
with commercial-friendly licensing. What would your requirements be? Are
there
protocols out there that are already "good enough"?

Here is a brief listing of some requirements to start:
  +  reliable :: message-level ACKs
  +  parallel, asynchronous batch sending :: why should I have to wait for
ACKs
before sending messages
  +  batch ACKs
  +  optional encryption
  +  integrity validation
  +  anti-replay protection
  +  anti-spoof protection
  +  non-repudiation
  +  compression
  +  event record affinity :: group related event records (by type,
producer,
etc.)
  +  support for structured event records  (e.g., CEE)
  +  support for semi- or un-structured event records (e.g., debug logs,
Syslog)
  +  support for transmitting related data :: send the packet/PCAP with the
IDS
alert; the malicious file with the Anti-Virus alert; video evidence with the
tripped alarm...



William Heinbockel
Infosec Engineer, Sr.
The MITRE Corporation
202 Burlington Rd. MS S145
Bedford, MA 01730
[hidden email]
781-271-2615
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

Eric Fitzgerald-2
I think that we should be wary of any protocol that MITRE doesn't own the spec for.


-----Original Message-----
From: Bill Frank [mailto:[hidden email]]
Sent: Thursday, January 20, 2011 4:12 PM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols

What about SDEE?

Most people think of it as a Cisco proprietary protocol, but the specification is out there for anyone to implement. At my previous company (a SIEM vendor), we had to support it. But we liked it so much, we used it internally among the product components.  

Bill Frank
Principal
Cymbel Corporation | Next Generation Defense-in-Depth
154 Wells Avenue, Newton, MA 02459, www.cymbel.com [hidden email]| 508-878-4943 | www.twitter.com/riskpundit


-----Original Message-----
From: Heinbockel, Bill [mailto:[hidden email]]
Sent: Thursday, January 20, 2011 5:36 PM
To: [hidden email]
Subject: [CEE-DISCUSSION-LIST] Log Protocols

As we all know, event records are most useful if we can collect them in a
central location for searching, correlating, and archiving. However, it is
my
opinion that there are no good protocols out there to support this.

Syslog has been the de facto log transport for years. While it does have its
benefits to being small and simple, it does not provide the security and
reliability protections that some environments need. Syslog over TLS/DTLS
and
other tunneled Syslog protocols only meet some of the requirements.

Other protocols, such as SOAP, are overkill for the limited amount of data
most
event records contain. BEEP has gotten some renewed interest as of late, but
it
seems to be complex and uses synchronous messaging channels.

RELP (http://www.librelp.com/relp.html) seems to be one of the few standouts
in
this area, but I don't think it is supported beyond rsyslog.

What if you were given the opportunity to design your own log protocol?
Something that was royalty-free, non-exclusive license that was made
available
with commercial-friendly licensing. What would your requirements be? Are
there
protocols out there that are already "good enough"?

Here is a brief listing of some requirements to start:
  +  reliable :: message-level ACKs
  +  parallel, asynchronous batch sending :: why should I have to wait for
ACKs
before sending messages
  +  batch ACKs
  +  optional encryption
  +  integrity validation
  +  anti-replay protection
  +  anti-spoof protection
  +  non-repudiation
  +  compression
  +  event record affinity :: group related event records (by type,
producer,
etc.)
  +  support for structured event records  (e.g., CEE)
  +  support for semi- or un-structured event records (e.g., debug logs,
Syslog)
  +  support for transmitting related data :: send the packet/PCAP with the
IDS
alert; the malicious file with the Anti-Virus alert; video evidence with the
tripped alarm...



William Heinbockel
Infosec Engineer, Sr.
The MITRE Corporation
202 Burlington Rd. MS S145
Bedford, MA 01730
[hidden email]
781-271-2615
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

Eric Fitzgerald-2
In reply to this post by heinbockel
What do you want the protocol to look like?

From the list of requirements below it looks like a "publish" or "push" protocol, e.g. syslog.

Do you want a "subscribe" model?

Do you want a "get" model (give me events [matching /foo/]), e.g. server-initiated (works better through firewalls)?

If client-initiated, how do clients know what server to talk to?  How do they failover/load-balance/affinity to the appropriate servers?  How do you proxy?  Do you want to support routing (above the IP layer, e.g. SOAP)?

Do you want connectionless, standing connections, or intermittent connections?

Do you want to support scheduling and bandwidth limits?

Spec'ing a protocol is a huge undertaking.

-----Original Message-----
From: Heinbockel, Bill [mailto:[hidden email]]
Sent: Thursday, January 20, 2011 2:36 PM
To: [hidden email]
Subject: [CEE-DISCUSSION-LIST] Log Protocols

As we all know, event records are most useful if we can collect them in a
central location for searching, correlating, and archiving. However, it is my
opinion that there are no good protocols out there to support this.

Syslog has been the de facto log transport for years. While it does have its
benefits to being small and simple, it does not provide the security and
reliability protections that some environments need. Syslog over TLS/DTLS and
other tunneled Syslog protocols only meet some of the requirements.

Other protocols, such as SOAP, are overkill for the limited amount of data most
event records contain. BEEP has gotten some renewed interest as of late, but it
seems to be complex and uses synchronous messaging channels.

RELP (http://www.librelp.com/relp.html) seems to be one of the few standouts in
this area, but I don't think it is supported beyond rsyslog.

What if you were given the opportunity to design your own log protocol?
Something that was royalty-free, non-exclusive license that was made available
with commercial-friendly licensing. What would your requirements be? Are there
protocols out there that are already "good enough"?

Here is a brief listing of some requirements to start:
  +  reliable :: message-level ACKs
  +  parallel, asynchronous batch sending :: why should I have to wait for ACKs
before sending messages
  +  batch ACKs
  +  optional encryption
  +  integrity validation
  +  anti-replay protection
  +  anti-spoof protection
  +  non-repudiation
  +  compression
  +  event record affinity :: group related event records (by type, producer,
etc.)
  +  support for structured event records  (e.g., CEE)
  +  support for semi- or un-structured event records (e.g., debug logs, Syslog)
  +  support for transmitting related data :: send the packet/PCAP with the IDS
alert; the malicious file with the Anti-Virus alert; video evidence with the
tripped alarm...



William Heinbockel
Infosec Engineer, Sr.
The MITRE Corporation
202 Burlington Rd. MS S145
Bedford, MA 01730
[hidden email]
781-271-2615
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

Bill Frank
In reply to this post by Eric Fitzgerald-2
I understand that just because Cisco said they want to make SDEE an industry
standard does not necessarily mean they would be willing to turn over
control of it to a third party like Mitre. But maybe they would.

Also all of the Log Management companies have experience with it because
they had to implement at least the "client" side to collect events from
Cisco IDS/IPS devices.

If you do a Google search on SDEE there are some good links including an
open source Perl implementation.

Bill Frank
Cymbel
508-878-4943

-----Original Message-----
From: Eric Fitzgerald [mailto:[hidden email]]
Sent: Thursday, January 20, 2011 7:39 PM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols

I think that we should be wary of any protocol that MITRE doesn't own the
spec for.


-----Original Message-----
From: Bill Frank [mailto:[hidden email]]
Sent: Thursday, January 20, 2011 4:12 PM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols

What about SDEE?

Most people think of it as a Cisco proprietary protocol, but the
specification is out there for anyone to implement. At my previous company
(a SIEM vendor), we had to support it. But we liked it so much, we used it
internally among the product components.  

Bill Frank
Principal
Cymbel Corporation | Next Generation Defense-in-Depth
154 Wells Avenue, Newton, MA 02459, www.cymbel.com [hidden email]|
508-878-4943 | www.twitter.com/riskpundit


-----Original Message-----
From: Heinbockel, Bill [mailto:[hidden email]]
Sent: Thursday, January 20, 2011 5:36 PM
To: [hidden email]
Subject: [CEE-DISCUSSION-LIST] Log Protocols

As we all know, event records are most useful if we can collect them in a
central location for searching, correlating, and archiving. However, it is
my opinion that there are no good protocols out there to support this.

Syslog has been the de facto log transport for years. While it does have its
benefits to being small and simple, it does not provide the security and
reliability protections that some environments need. Syslog over TLS/DTLS
and other tunneled Syslog protocols only meet some of the requirements.

Other protocols, such as SOAP, are overkill for the limited amount of data
most event records contain. BEEP has gotten some renewed interest as of
late, but it seems to be complex and uses synchronous messaging channels.

RELP (http://www.librelp.com/relp.html) seems to be one of the few standouts
in this area, but I don't think it is supported beyond rsyslog.

What if you were given the opportunity to design your own log protocol?
Something that was royalty-free, non-exclusive license that was made
available with commercial-friendly licensing. What would your requirements
be? Are there protocols out there that are already "good enough"?

Here is a brief listing of some requirements to start:
  +  reliable :: message-level ACKs
  +  parallel, asynchronous batch sending :: why should I have to wait for
ACKs before sending messages
  +  batch ACKs
  +  optional encryption
  +  integrity validation
  +  anti-replay protection
  +  anti-spoof protection
  +  non-repudiation
  +  compression
  +  event record affinity :: group related event records (by type,
producer,
etc.)
  +  support for structured event records  (e.g., CEE)
  +  support for semi- or un-structured event records (e.g., debug logs,
Syslog)
  +  support for transmitting related data :: send the packet/PCAP with the
IDS alert; the malicious file with the Anti-Virus alert; video evidence with
the tripped alarm...



William Heinbockel
Infosec Engineer, Sr.
The MITRE Corporation
202 Burlington Rd. MS S145
Bedford, MA 01730
[hidden email]
781-271-2615
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

heinbockel
In reply to this post by Eric Fitzgerald-2
Overall I'm thinking something with a simple core, something that can provide
the ease of Syslog but can grow to provide support for SASL/TLS and other
features.

Generally I am thinking more of something like RELP or ESMTP with support for
pipelining, multi-plexing, and feature extensions.


>-----Original Message-----
>From: Eric Fitzgerald [mailto:[hidden email]]
>Sent: Thursday, 20 January 2011 19:45
>To: Heinbockel, Bill; cee-discussion-list CEE-Related Discussion
>Subject: RE: Log Protocols
>
>What do you want the protocol to look like?
>
>From the list of requirements below it looks like a "publish" or "push"
>protocol, e.g. syslog.
>
From the research I've done, it sounds like the simpler the better.
As far as "push" v. "pull" v. "pub-sub", I think that they all have their
advantages when it comes to logs.
Ideally, I think of the protocol being able to support the standard "push" but
maybe having standardized extensions to provide "pull" and/or "pub-sub" support

>Do you want a "subscribe" model?
>
>Do you want a "get" model (give me events [matching /foo/]), e.g. server-
>initiated (works better through firewalls)?
>
>If client-initiated, how do clients know what server to talk to?  How do
>they failover/load-balance/affinity to the appropriate servers?  How do you
>proxy?  Do you want to support routing (above the IP layer, e.g. SOAP)?

The simplest here is just standardizing a protocol and port, and connecting via
IP addresses.
I would imagine that this could be combined with some sort of publication
service.
Failover/load-balance/affinity could be provided via log proxies or similar. I'm
not sure that this needs to be built into a spec, just that the spec should
allow this to occur

My initial thoughts is that SOAP is too much overhead and complexity for logs

>
>Do you want connectionless, standing connections, or intermittent
>connections?

Whichever makes the most sense: connectionless would appear to hamper the
possibilities for SASL/TLS support. Standing connections are troublesome over
high-latency or poor otherwise poor connections.

>
>Do you want to support scheduling and bandwidth limits?

Yes.

>
>Spec'ing a protocol is a huge undertaking.
>
>-----Original Message-----
>From: Heinbockel, Bill [mailto:[hidden email]]
>Sent: Thursday, January 20, 2011 2:36 PM
>To: [hidden email]
>Subject: [CEE-DISCUSSION-LIST] Log Protocols
>
>As we all know, event records are most useful if we can collect them in a
>central location for searching, correlating, and archiving. However, it is
>my
>opinion that there are no good protocols out there to support this.
>
>Syslog has been the de facto log transport for years. While it does have its
>benefits to being small and simple, it does not provide the security and
>reliability protections that some environments need. Syslog over TLS/DTLS
>and
>other tunneled Syslog protocols only meet some of the requirements.
>
>Other protocols, such as SOAP, are overkill for the limited amount of data
>most
>event records contain. BEEP has gotten some renewed interest as of late, but
>it
>seems to be complex and uses synchronous messaging channels.
>
>RELP (http://www.librelp.com/relp.html) seems to be one of the few standouts
>in
>this area, but I don't think it is supported beyond rsyslog.
>
>What if you were given the opportunity to design your own log protocol?
>Something that was royalty-free, non-exclusive license that was made
>available
>with commercial-friendly licensing. What would your requirements be? Are
>there
>protocols out there that are already "good enough"?
>
>Here is a brief listing of some requirements to start:
>  +  reliable :: message-level ACKs
>  +  parallel, asynchronous batch sending :: why should I have to wait for
>ACKs
>before sending messages
>  +  batch ACKs
>  +  optional encryption
>  +  integrity validation
>  +  anti-replay protection
>  +  anti-spoof protection
>  +  non-repudiation
>  +  compression
>  +  event record affinity :: group related event records (by type,
>producer,
>etc.)
>  +  support for structured event records  (e.g., CEE)
>  +  support for semi- or un-structured event records (e.g., debug logs,
>Syslog)
>  +  support for transmitting related data :: send the packet/PCAP with the
>IDS
>alert; the malicious file with the Anti-Virus alert; video evidence with the
>tripped alarm...
>
>
>
>William Heinbockel
>Infosec Engineer, Sr.
>The MITRE Corporation
>202 Burlington Rd. MS S145
>Bedford, MA 01730
>[hidden email]
>781-271-2615
>


smime.p7s (4K) Download Attachment
YP
Reply | Threaded
Open this post in threaded view
|

Out of Office AutoReply: Log Protocols

YP
This post has NOT been accepted by the mailing list yet.
I am on PTO today.
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

David Corlette
In reply to this post by Eric Fitzgerald-2
Here at Novell we implemented REST to do batch event delivery between instances of our SIEM. In our case the connection goes one way (from the event collector to the backend), but we've certainly discussed other options such as server-initiated fetches and subscription models. It also supports compression, encryption, SSL, scheduled delivery, etc.... none of which are in REST per se but have simply been implemented as services on top of REST.

The point being, don't get hung up on all the possible features you might need to add at some point. Choose something simple like REST where the two endpoints can negotiate what features each endpoint supports, and pick the highest common denominator between them.




>>> On 20.01.2011 at 19:45, in message
<[hidden email]
.microsoft.com>, Eric Fitzgerald <[hidden email]> wrote:

> What do you want the protocol to look like?
>
> From the list of requirements below it looks like a "publish" or "push"
> protocol, e.g. syslog.
>
> Do you want a "subscribe" model?
>
> Do you want a "get" model (give me events [matching /foo/]), e.g.
> server-initiated (works better through firewalls)?
>
> If client-initiated, how do clients know what server to talk to?  How do they
> failover/load-balance/affinity to the appropriate servers?  How do you proxy?  
> Do you want to support routing (above the IP layer, e.g. SOAP)?
>
> Do you want connectionless, standing connections, or intermittent
> connections?
>
> Do you want to support scheduling and bandwidth limits?
>
> Spec'ing a protocol is a huge undertaking.
>
> -----Original Message-----
> From: Heinbockel, Bill [mailto:[hidden email]]
> Sent: Thursday, January 20, 2011 2:36 PM
> To: [hidden email]
> Subject: [CEE-DISCUSSION-LIST] Log Protocols
>
> As we all know, event records are most useful if we can collect them in a
> central location for searching, correlating, and archiving. However, it is
> my
> opinion that there are no good protocols out there to support this.
>
> Syslog has been the de facto log transport for years. While it does have its
> benefits to being small and simple, it does not provide the security and
> reliability protections that some environments need. Syslog over TLS/DTLS
> and
> other tunneled Syslog protocols only meet some of the requirements.
>
> Other protocols, such as SOAP, are overkill for the limited amount of data
> most
> event records contain. BEEP has gotten some renewed interest as of late, but
> it
> seems to be complex and uses synchronous messaging channels.
>
> RELP (http://www.librelp.com/relp.html) seems to be one of the few standouts
> in
> this area, but I don't think it is supported beyond rsyslog.
>
> What if you were given the opportunity to design your own log protocol?
> Something that was royalty-free, non-exclusive license that was made available
> with commercial-friendly licensing. What would your requirements be? Are
> there
> protocols out there that are already "good enough"?
>
> Here is a brief listing of some requirements to start:
>   +  reliable :: message-level ACKs
>   +  parallel, asynchronous batch sending :: why should I have to wait for
> ACKs
> before sending messages
>   +  batch ACKs
>   +  optional encryption
>   +  integrity validation
>   +  anti-replay protection
>   +  anti-spoof protection
>   +  non-repudiation
>   +  compression
>   +  event record affinity :: group related event records (by type,
> producer,
> etc.)
>   +  support for structured event records  (e.g., CEE)
>   +  support for semi- or un-structured event records (e.g., debug logs,
> Syslog)
>   +  support for transmitting related data :: send the packet/PCAP with the
> IDS
> alert; the malicious file with the Anti-Virus alert; video evidence with the
> tripped alarm...
>
>
>
> William Heinbockel
> Infosec Engineer, Sr.
> The MITRE Corporation
> 202 Burlington Rd. MS S145
> Bedford, MA 01730
> [hidden email]
> 781-271-2615
YP
Reply | Threaded
Open this post in threaded view
|

Out of Office AutoReply: Log Protocols

YP
This post has NOT been accepted by the mailing list yet.
I am on PTO today.
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

Eric Fitzgerald-2
In reply to this post by David Corlette
My point is, that designing a protocol is a lot of work and involves a lot of decisions.  REST might have been fine for your use case but might be inappropriate for other use cases.  Same thing for syslog or <insert any other event transport protocol here>.

As I have stated from the beginning of the project, I think that we should stay out of the transport protocol business.

It's unlikely that we would see wide adoption of any protocol that we proposed, and I am concerned that if the standard became equated with a protocol then it might be the case that people would opt out simply because they don't want to adopt the protocol.

-----Original Message-----
From: David Corlette [mailto:[hidden email]]
Sent: Monday, January 24, 2011 8:07 AM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols

Here at Novell we implemented REST to do batch event delivery between instances of our SIEM. In our case the connection goes one way (from the event collector to the backend), but we've certainly discussed other options such as server-initiated fetches and subscription models. It also supports compression, encryption, SSL, scheduled delivery, etc.... none of which are in REST per se but have simply been implemented as services on top of REST.

The point being, don't get hung up on all the possible features you might need to add at some point. Choose something simple like REST where the two endpoints can negotiate what features each endpoint supports, and pick the highest common denominator between them.




>>> On 20.01.2011 at 19:45, in message
<[hidden email]
.microsoft.com>, Eric Fitzgerald <[hidden email]> wrote:

> What do you want the protocol to look like?
>
> From the list of requirements below it looks like a "publish" or "push"
> protocol, e.g. syslog.
>
> Do you want a "subscribe" model?
>
> Do you want a "get" model (give me events [matching /foo/]), e.g.
> server-initiated (works better through firewalls)?
>
> If client-initiated, how do clients know what server to talk to?  How
> do they failover/load-balance/affinity to the appropriate servers?  How do you proxy?
> Do you want to support routing (above the IP layer, e.g. SOAP)?
>
> Do you want connectionless, standing connections, or intermittent
> connections?
>
> Do you want to support scheduling and bandwidth limits?
>
> Spec'ing a protocol is a huge undertaking.
>
> -----Original Message-----
> From: Heinbockel, Bill [mailto:[hidden email]]
> Sent: Thursday, January 20, 2011 2:36 PM
> To: [hidden email]
> Subject: [CEE-DISCUSSION-LIST] Log Protocols
>
> As we all know, event records are most useful if we can collect them
> in a central location for searching, correlating, and archiving.
> However, it is my opinion that there are no good protocols out there
> to support this.
>
> Syslog has been the de facto log transport for years. While it does
> have its benefits to being small and simple, it does not provide the
> security and reliability protections that some environments need.
> Syslog over TLS/DTLS and other tunneled Syslog protocols only meet
> some of the requirements.
>
> Other protocols, such as SOAP, are overkill for the limited amount of
> data most event records contain. BEEP has gotten some renewed interest
> as of late, but it seems to be complex and uses synchronous messaging
> channels.
>
> RELP (http://www.librelp.com/relp.html) seems to be one of the few
> standouts in this area, but I don't think it is supported beyond
> rsyslog.
>
> What if you were given the opportunity to design your own log protocol?
> Something that was royalty-free, non-exclusive license that was made
> available with commercial-friendly licensing. What would your
> requirements be? Are there protocols out there that are already "good
> enough"?
>
> Here is a brief listing of some requirements to start:
>   +  reliable :: message-level ACKs
>   +  parallel, asynchronous batch sending :: why should I have to wait
> for ACKs before sending messages
>   +  batch ACKs
>   +  optional encryption
>   +  integrity validation
>   +  anti-replay protection
>   +  anti-spoof protection
>   +  non-repudiation
>   +  compression
>   +  event record affinity :: group related event records (by type,
> producer,
> etc.)
>   +  support for structured event records  (e.g., CEE)
>   +  support for semi- or un-structured event records (e.g., debug
> logs,
> Syslog)
>   +  support for transmitting related data :: send the packet/PCAP
> with the IDS alert; the malicious file with the Anti-Virus alert;
> video evidence with the tripped alarm...
>
>
>
> William Heinbockel
> Infosec Engineer, Sr.
> The MITRE Corporation
> 202 Burlington Rd. MS S145
> Bedford, MA 01730
> [hidden email]
> 781-271-2615
YP
Reply | Threaded
Open this post in threaded view
|

Out of Office AutoReply: Log Protocols

YP
This post has NOT been accepted by the mailing list yet.
I am on PTO today.
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

Ruben Oliva
In reply to this post by heinbockel

To all:

Please ignore me if I am out of place.

I think we are looking at perception issues.

1.        Any currently available propriety protocol will appear to exceed the basic requirements that we are seeking to make standard.  Your perception will probably be that the standard will be “too little”, “inadequate”, etc.

2.       A standard protocol will address issues that current proprietary protocols do not address and will appear as unnecessary for your particular application.  Your perception will probably be that the standard asks for “too much” or “not really needed”.

To make CEE useful and profitable for your companies we need to work on a common denominator that appears to offer less than current products can offer (on the one hand) but maximizes the ability to talk logs among all future products (on the other hand). 

 

R. David Oliva



Jan 24, 2011 12:25:26 PM, [hidden email] wrote:
My point is, that designing a protocol is a lot of work and involves a lot of decisions. REST might have been fine for your use case but might be inappropriate for other use cases. Same thing for syslog or .

As I have stated from the beginning of the project, I think that we should stay out of the transport protocol business.

It's unlikely that we would see wide adoption of any protocol that we proposed, and I am concerned that if the standard became equated with a protocol then it might be the case that people would opt out simply because they don't want to adopt the protocol.

-----Original Message-----
From: David Corlette [mailto:[hidden email]]
Sent: Monday, January 24, 2011 8:07 AM
To: [hidden email]
Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols

Here at Novell we implemented REST to do batch event delivery between instances of our SIEM. In our case the connection goes one way (from the event collector to the backend), but we've certainly discussed other options such as server-initiated fetches and subscription models. It also supports compression, encryption, SSL, scheduled delivery, etc.... none of which are in REST per se but have simply been implemented as services on top of REST.

The point being, don't get hung up on all the possible features you might need to add at some point. Choose something simple like REST where the two endpoints can negotiate what features each endpoint supports, and pick the highest common denominator between them.




>>> On 20.01.2011 at 19:45, in message
.microsoft.com>, Eric Fitzgerald wrote:

> What do you want the protocol to look like?
>
> From the list of requirements below it looks like a "publish" or "push"
> protocol, e.g. syslog.
>
> Do you want a "subscribe" model?
>
> Do you want a "get" model (give me events [matching /foo/]), e.g.
> server-initiated (works better through firewalls)?
>
> If client-initiated, how do clients know what server to talk to? How
> do they failover/load-balance/affinity to the appropriate servers? How do you proxy?
> Do you want to support routing (above the IP layer, e.g. SOAP)?
>
> Do you want connectionless, standing connections, or intermittent
> connections?
>
> Do you want to support scheduling and bandwidth limits?
>
> Spec'ing a protocol is a huge undertaking.
>
> -----Original Message-----
> From: Heinbockel, Bill [mailto:[hidden email]]
> Sent: Thursday, January 20, 2011 2:36 PM
> To: [hidden email]
> Subject: [CEE-DISCUSSION-LIST] Log Protocols
>
> As we all know, event records are most useful if we can collect them
> in a central location for searching, correlating, and archiving.
> However, it is my opinion that there are no good protocols out there
> to support this.
>
> Syslog has been the de facto log transport for years. While it does
> have its benefits to being small and simple, it does not provide the
> security and reliability protections that some environments need.
> Syslog over TLS/DTLS and other tunneled Syslog protocols only meet
> some of the requirements.
>
> Other protocols, such as SOAP, are overkill for the limited amount of
> data most event records contain. BEEP has gotten some renewed interest
> as of late, but it seems to be complex and uses synchronous messaging
> channels.
>
> RELP (http://www.librelp.com/relp.html) seems to be one of the few
> standouts in this area, but I don't think it is supported beyond
> rsyslog.
>
> What if you were given the opportunity to design your own log protocol?
> Something that was royalty-free, non-exclusive license that was made
> available with commercial-friendly licensing. What would your
> requirements be? Are there protocols out there that are already "good
> enough"?
>
> Here is a brief listing of some requirements to start:
> + reliable :: message-level ACKs
> + parallel, asynchronous batch sending :: why should I have to wait
> for ACKs before sending messages
> + batch ACKs
> + optional encryption
> + integrity validation
> + anti-replay protection
> + anti-spoof protection
> + non-repudiation
> + compression
> + event record affinity :: group related event records (by type,
> producer,
> etc.)
> + support for structured event records (e.g., CEE)
> + support for semi- or un-structured event records (e.g., debug
> logs,
> Syslog)
> + support for transmitting related data :: send the packet/PCAP
> with the IDS alert; the malicious file with the Anti-Virus alert;
> video evidence with the tripped alarm...
>
>
>
> William Heinbockel
> Infosec Engineer, Sr.
> The MITRE Corporation
> 202 Burlington Rd. MS S145
> Bedford, MA 01730
> [hidden email]
> 781-271-2615
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

Rainer Gerhards
In reply to this post by heinbockel
Replying to this after having read the full discussion thread.

I am a bit on Eric's side: designing a protocol is a hard technical task.
Getting multi-vendor agreement is a very hard political task (but Mitre
obviously has a bonus). Getting Implementations is even harder. Entangling
CEE with a new protocol will probably mean it takes another 5 to 10 years
until CEE can be finished.

From Bill's first mail I got the impression that his "protocol question" is
not necessarily driven by CEE alone. If not, it would probably be helpful to
know the driving force (maybe classified info?). If there is a good reason,
it can make sense to try to create a new protocol. But I'd strongly suggest
to do this outside of the core CEE effort.

In order to see if we actually need a new protocol, we need to know a more
precise requirement spec. IMHO Bill's bullet point requirement list is by far
not precise enough to help us make this decision.

On BEEP: I was one of a handful of folks who implemented a minimal BEEP stack
for RFC3195. BEEP is a pretty capable and elegant protocol and comes close to
what Bill projects. It is a bit hard to understand at first, and it has some
overhead. But it is very flexible. A problem is the lack of good libraries. I
am not sure if BEEP is already dead or can be revived. It has lots of good
concepts.

On RELP: This is rsyslog-only (at least as far as I know). I developed it to
provide reliability when I really needed it. But it is in its infancy at
best.

Rainer
 

> -----Original Message-----
> From: Heinbockel, Bill [mailto:[hidden email]]
> Sent: Monday, January 24, 2011 4:28 PM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols
>
> Overall I'm thinking something with a simple core, something that can
> provide
> the ease of Syslog but can grow to provide support for SASL/TLS and
> other
> features.
>
> Generally I am thinking more of something like RELP or ESMTP with
> support for
> pipelining, multi-plexing, and feature extensions.
>
>
> >-----Original Message-----
> >From: Eric Fitzgerald [mailto:[hidden email]]
> >Sent: Thursday, 20 January 2011 19:45
> >To: Heinbockel, Bill; cee-discussion-list CEE-Related Discussion
> >Subject: RE: Log Protocols
> >
> >What do you want the protocol to look like?
> >
> >From the list of requirements below it looks like a "publish" or
> "push"
> >protocol, e.g. syslog.
> >
>
> From the research I've done, it sounds like the simpler the better.
> As far as "push" v. "pull" v. "pub-sub", I think that they all have
> their
> advantages when it comes to logs.
> Ideally, I think of the protocol being able to support the standard
> "push" but
> maybe having standardized extensions to provide "pull" and/or "pub-sub"
> support
>
> >Do you want a "subscribe" model?
> >
> >Do you want a "get" model (give me events [matching /foo/]), e.g.
> server-
> >initiated (works better through firewalls)?
> >
> >If client-initiated, how do clients know what server to talk to?  How
> do
> >they failover/load-balance/affinity to the appropriate servers?  How
> do you
> >proxy?  Do you want to support routing (above the IP layer, e.g.
> SOAP)?
>
> The simplest here is just standardizing a protocol and port, and
> connecting via
> IP addresses.
> I would imagine that this could be combined with some sort of
> publication
> service.
> Failover/load-balance/affinity could be provided via log proxies or
> similar. I'm
> not sure that this needs to be built into a spec, just that the spec
> should
> allow this to occur
>
> My initial thoughts is that SOAP is too much overhead and complexity
> for logs
>
> >
> >Do you want connectionless, standing connections, or intermittent
> >connections?
>
> Whichever makes the most sense: connectionless would appear to hamper
> the
> possibilities for SASL/TLS support. Standing connections are
> troublesome over
> high-latency or poor otherwise poor connections.
>
> >
> >Do you want to support scheduling and bandwidth limits?
>
> Yes.
>
> >
> >Spec'ing a protocol is a huge undertaking.
> >
> >-----Original Message-----
> >From: Heinbockel, Bill [mailto:[hidden email]]
> >Sent: Thursday, January 20, 2011 2:36 PM
> >To: [hidden email]
> >Subject: [CEE-DISCUSSION-LIST] Log Protocols
> >
> >As we all know, event records are most useful if we can collect them
> in a
> >central location for searching, correlating, and archiving. However,
> it is
> >my
> >opinion that there are no good protocols out there to support this.
> >
> >Syslog has been the de facto log transport for years. While it does
> have its
> >benefits to being small and simple, it does not provide the security
> and
> >reliability protections that some environments need. Syslog over
> TLS/DTLS
> >and
> >other tunneled Syslog protocols only meet some of the requirements.
> >
> >Other protocols, such as SOAP, are overkill for the limited amount of
> data
> >most
> >event records contain. BEEP has gotten some renewed interest as of
> late, but
> >it
> >seems to be complex and uses synchronous messaging channels.
> >
> >RELP (http://www.librelp.com/relp.html) seems to be one of the few
> standouts
> >in
> >this area, but I don't think it is supported beyond rsyslog.
> >
> >What if you were given the opportunity to design your own log
> protocol?
> >Something that was royalty-free, non-exclusive license that was made
> >available
> >with commercial-friendly licensing. What would your requirements be?
> Are
> >there
> >protocols out there that are already "good enough"?
> >
> >Here is a brief listing of some requirements to start:
> >  +  reliable :: message-level ACKs
> >  +  parallel, asynchronous batch sending :: why should I have to wait
> for
> >ACKs
> >before sending messages
> >  +  batch ACKs
> >  +  optional encryption
> >  +  integrity validation
> >  +  anti-replay protection
> >  +  anti-spoof protection
> >  +  non-repudiation
> >  +  compression
> >  +  event record affinity :: group related event records (by type,
> >producer,
> >etc.)
> >  +  support for structured event records  (e.g., CEE)
> >  +  support for semi- or un-structured event records (e.g., debug
> logs,
> >Syslog)
> >  +  support for transmitting related data :: send the packet/PCAP
> with the
> >IDS
> >alert; the malicious file with the Anti-Virus alert; video evidence
> with the
> >tripped alarm...
> >
> >
> >
> >William Heinbockel
> >Infosec Engineer, Sr.
> >The MITRE Corporation
> >202 Burlington Rd. MS S145
> >Bedford, MA 01730
> >[hidden email]
> >781-271-2615
> >
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

Chris Lonvick
In reply to this post by Eric Fitzgerald-2
Hi,

Like Rainer, I'm responding after reading the entire thread.

I agree with Eric.  To do a proper job, the group would have to agree upon
very specific characteristics that meet well defined requirements.  While
doing that, you'd need to create a threat model and specify how the new
protocol will thwart attacks.

While there may be good reasons to define a new protocol, I'd suggest that
the group keep a focus on CEE and use transports that are "good enough for
now".

Thanks,
Chris

On Mon, 24 Jan 2011, Eric Fitzgerald wrote:

> My point is, that designing a protocol is a lot of work and involves a lot of decisions.  REST might have been fine for your use case but might be inappropriate for other use cases.  Same thing for syslog or <insert any other event transport protocol here>.
>
> As I have stated from the beginning of the project, I think that we should stay out of the transport protocol business.
>
> It's unlikely that we would see wide adoption of any protocol that we proposed, and I am concerned that if the standard became equated with a protocol then it might be the case that people would opt out simply because they don't want to adopt the protocol.
>
> -----Original Message-----
> From: David Corlette [mailto:[hidden email]]
> Sent: Monday, January 24, 2011 8:07 AM
> To: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols
>
> Here at Novell we implemented REST to do batch event delivery between instances of our SIEM. In our case the connection goes one way (from the event collector to the backend), but we've certainly discussed other options such as server-initiated fetches and subscription models. It also supports compression, encryption, SSL, scheduled delivery, etc.... none of which are in REST per se but have simply been implemented as services on top of REST.
>
> The point being, don't get hung up on all the possible features you might need to add at some point. Choose something simple like REST where the two endpoints can negotiate what features each endpoint supports, and pick the highest common denominator between them.
>
>
>
>
>>>> On 20.01.2011 at 19:45, in message
> <[hidden email]
> .microsoft.com>, Eric Fitzgerald <[hidden email]> wrote:
>> What do you want the protocol to look like?
>>
>> From the list of requirements below it looks like a "publish" or "push"
>> protocol, e.g. syslog.
>>
>> Do you want a "subscribe" model?
>>
>> Do you want a "get" model (give me events [matching /foo/]), e.g.
>> server-initiated (works better through firewalls)?
>>
>> If client-initiated, how do clients know what server to talk to?  How
>> do they failover/load-balance/affinity to the appropriate servers?  How do you proxy?
>> Do you want to support routing (above the IP layer, e.g. SOAP)?
>>
>> Do you want connectionless, standing connections, or intermittent
>> connections?
>>
>> Do you want to support scheduling and bandwidth limits?
>>
>> Spec'ing a protocol is a huge undertaking.
>>
>> -----Original Message-----
>> From: Heinbockel, Bill [mailto:[hidden email]]
>> Sent: Thursday, January 20, 2011 2:36 PM
>> To: [hidden email]
>> Subject: [CEE-DISCUSSION-LIST] Log Protocols
>>
>> As we all know, event records are most useful if we can collect them
>> in a central location for searching, correlating, and archiving.
>> However, it is my opinion that there are no good protocols out there
>> to support this.
>>
>> Syslog has been the de facto log transport for years. While it does
>> have its benefits to being small and simple, it does not provide the
>> security and reliability protections that some environments need.
>> Syslog over TLS/DTLS and other tunneled Syslog protocols only meet
>> some of the requirements.
>>
>> Other protocols, such as SOAP, are overkill for the limited amount of
>> data most event records contain. BEEP has gotten some renewed interest
>> as of late, but it seems to be complex and uses synchronous messaging
>> channels.
>>
>> RELP (http://www.librelp.com/relp.html) seems to be one of the few
>> standouts in this area, but I don't think it is supported beyond
>> rsyslog.
>>
>> What if you were given the opportunity to design your own log protocol?
>> Something that was royalty-free, non-exclusive license that was made
>> available with commercial-friendly licensing. What would your
>> requirements be? Are there protocols out there that are already "good
>> enough"?
>>
>> Here is a brief listing of some requirements to start:
>>   +  reliable :: message-level ACKs
>>   +  parallel, asynchronous batch sending :: why should I have to wait
>> for ACKs before sending messages
>>   +  batch ACKs
>>   +  optional encryption
>>   +  integrity validation
>>   +  anti-replay protection
>>   +  anti-spoof protection
>>   +  non-repudiation
>>   +  compression
>>   +  event record affinity :: group related event records (by type,
>> producer,
>> etc.)
>>   +  support for structured event records  (e.g., CEE)
>>   +  support for semi- or un-structured event records (e.g., debug
>> logs,
>> Syslog)
>>   +  support for transmitting related data :: send the packet/PCAP
>> with the IDS alert; the malicious file with the Anti-Virus alert;
>> video evidence with the tripped alarm...
>>
>>
>>
>> William Heinbockel
>> Infosec Engineer, Sr.
>> The MITRE Corporation
>> 202 Burlington Rd. MS S145
>> Bedford, MA 01730
>> [hidden email]
>> 781-271-2615
>
Reply | Threaded
Open this post in threaded view
|

Re: Log Protocols

heinbockel
The initial question had two intents:
1.  As far as CEE is concerned, I think we should recommend features that should
be present in a good log protocol and highlight some of those that we recommend

2.  I have been requested to help develop a log protocol able to perform under
certain requirements. Has the group has already said, protocol design is
difficult. Right now I've investigated several protocols such as RELP and BEEP.
Neither meet all of the requirements. So, I'm trying to determine if it is best
to use something like REST or BEEP and add the components we need, or to design
a new protocol (maybe based on something simpler like RELP)


Here is a mostly complete listing of current requirements:

1. Log over High-Latency Networks
---------------------------------
Networks such as those operating over SATCOM (satellite) links have a
notoriously high latency. For some networks, the round-trip time (rtt) of a
packet is 10 seconds. On most enterprise networks, the rtt is usually under
50ms. With a large rtt, transport protocols such as TCP are problematic, as they
require additional tuning, including increasing the TCP transmission window size
and using optional TCP extensions (e.g., selective acknowledgments (SACK).
When dealing with small messages, such as event records, high-latency networks
can cause havoc. This is especially the case when using protocols that do not
support pipelining-many messages may be in transmission at any time. Instead of
being able to send many small records before waiting for replies, each record
must be ACKed before the next record is sent. The server then ACKs the packets
as they are received. Any additional delays, dropped packets, misconfigurations,
or a lost host means that a packet or ACK may not be received in time and will
have to be retransmitted. This may result in even further processing delays and
cause a large queue of messages to be built on the client.
Not only is this an issue with sending data, but it can cause issues in
multiplexed protocols that have separate control channels or support in-band
reconfiguration, such as BEEP. When negotiating tuning profiles, this can cause
a large number of retransmissions and complexities.

2. Log with Limited Bandwidth
-----------------------------
Not all networks have the available bandwidth to support sending many event
records. Some networks may have bandwidth allotted by time, where event records
can only be sent during certain hours. Other networks may not have much capacity
or segment a limited bandwidth cap for non-essential functions such as logging.
In order to maximize the amount of data that can be sent over the limited
capacity, the log protocol must be able to reorder the event transmission queue
so that the most important messages get priority. Additionally, the client must
support batch transmissions; that is it must be able to include multiple event
records per message. Similarly, the server must be able to bulk acknowledge
receipt of the individual event records in a single message.
The protocol should also provide a method so that the client can compress the
message contents before transmitting, in order to reduce bandwidth usage.

3. Log in Environments with Troublesome Hosts and Networks
----------------------------------------------------------
Depending on the network and environment, some TCP connections are not as
reliable as we might want. There may be a non-trivial chance of packet loss or
data corruption, or one endpoint may become unresponsive. When this occurs, it
is possible that protocol messages are sent but may never be received or may
cause the TCP connection to hang for extended periods.
In order to counteract these possibilities, the log protocol must allow the
server to be able to acknowledge the proper reception of each transmitted event
record. Only after acknowledgment that the event record was received can the
sender clear the event record from its transmission queue. If no acknowledgment
is received after a certain period, the client should retransmit the event
record.
To support environments would rather have speed or reduced network utilization
over the additional integrity, the protocol may support the option to mark the
message or event records as not requiring a receipt of acknowledgment. If this
is the case, the event records can be cleared out of the client's transmission
queue once the event record has been transmitted and ignore any related ACKs.

4. Security Log within the Presence of Attackers
------------------------------------------------
Traditional log protocols offer minimal protection against attackers. An
attacker can easily snoop on Syslog's clear text transmissions, alter or spoof
messages with man-in-the-middle attacks, or replay previously sent messages. Any
log protocol must offer protection against these potential attack vectors.
Log protocols should at least support transmission encryption (e.g., TLS) to
provide some protections. Preferably, the client should be allowed to
authenticate to the server and negotiate the encryption options before being
allowed to transmit data over an encrypted channel. For example, the protocol
may include SASL support.
To protect against message replay, the protocol may decide to utilize unique
connection and message identifiers. It is recommended that the server and client
report idiosyncrasies that may occur with repeated or unexpected message
identifiers. TLS provides some protection by requiring a nonce connection
identifier to be used.

5. Log Structured and Unstructured Event Records
------------------------------------------------
The log protocol must be able to support various types of event records and
related data. Different products may choose to use different event record
formats. Some may use a structured CEE, XML, or JSON encoding syntax, while
others might use Syslog or plain text data. Others may transmit binary data.
Text may be encoded with UTF-8, ISO-8859, or some other method.
Regardless, the log protocol must not unnecessarily restrict the data encoding
formats and provide a mechanism for the client and server to know how the data
was encoded. For example, HTML and SMTP use MIME. BEEP uses MIME, but can also
define the data encoding within a profile.
Additionally, the log protocol must provide a way for the client to associate
related event records within the transmitted message. This could be done by
assigning the same identifier to each record or by providing a message-based
grouping of the data.

6. Track Logs through NATs and Log Relays
-----------------------------------------
Some environments require event records to be passed through NATs or Log Relays
that can alter the event record. Any log protocol must require that all devices
that retransmit event records be able to identify the network path the event
record has traversed.

William Heinbockel
The MITRE Corporation


>-----Original Message-----
>From: Chris Lonvick [mailto:[hidden email]]
>Sent: Tuesday, 25 January 2011 09:47
>To: cee-discussion-list CEE-Related Discussion
>Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols
>
>Hi,
>
>Like Rainer, I'm responding after reading the entire thread.
>
>I agree with Eric.  To do a proper job, the group would have to agree upon
>very specific characteristics that meet well defined requirements.  While
>doing that, you'd need to create a threat model and specify how the new
>protocol will thwart attacks.
>
>While there may be good reasons to define a new protocol, I'd suggest that
>the group keep a focus on CEE and use transports that are "good enough for
>now".
>
>Thanks,
>Chris
>
>On Mon, 24 Jan 2011, Eric Fitzgerald wrote:
>
>> My point is, that designing a protocol is a lot of work and involves a lot
>of decisions.  REST might have been fine for your use case but might be
>inappropriate for other use cases.  Same thing for syslog or <insert any
>other event transport protocol here>.
>>
>> As I have stated from the beginning of the project, I think that we should
>stay out of the transport protocol business.
>>
>> It's unlikely that we would see wide adoption of any protocol that we
>proposed, and I am concerned that if the standard became equated with a
>protocol then it might be the case that people would opt out simply because
>they don't want to adopt the protocol.
>>
>> -----Original Message-----
>> From: David Corlette [mailto:[hidden email]]
>> Sent: Monday, January 24, 2011 8:07 AM
>> To: [hidden email]
>> Subject: Re: [CEE-DISCUSSION-LIST] Log Protocols
>>
>> Here at Novell we implemented REST to do batch event delivery between
>instances of our SIEM. In our case the connection goes one way (from the
>event collector to the backend), but we've certainly discussed other options
>such as server-initiated fetches and subscription models. It also supports
>compression, encryption, SSL, scheduled delivery, etc.... none of which are
>in REST per se but have simply been implemented as services on top of REST.
>>
>> The point being, don't get hung up on all the possible features you might
>need to add at some point. Choose something simple like REST where the two
>endpoints can negotiate what features each endpoint supports, and pick the
>highest common denominator between them.
>>
>>
>>
>>
>>>>> On 20.01.2011 at 19:45, in message
>>
><[hidden email].
>ntde
>> .microsoft.com>, Eric Fitzgerald <[hidden email]> wrote:
>>> What do you want the protocol to look like?
>>>
>>> From the list of requirements below it looks like a "publish" or "push"
>>> protocol, e.g. syslog.
>>>
>>> Do you want a "subscribe" model?
>>>
>>> Do you want a "get" model (give me events [matching /foo/]), e.g.
>>> server-initiated (works better through firewalls)?
>>>
>>> If client-initiated, how do clients know what server to talk to?  How
>>> do they failover/load-balance/affinity to the appropriate servers?  How
>do you proxy?
>>> Do you want to support routing (above the IP layer, e.g. SOAP)?
>>>
>>> Do you want connectionless, standing connections, or intermittent
>>> connections?
>>>
>>> Do you want to support scheduling and bandwidth limits?
>>>
>>> Spec'ing a protocol is a huge undertaking.
>>>
>>> -----Original Message-----
>>> From: Heinbockel, Bill [mailto:[hidden email]]
>>> Sent: Thursday, January 20, 2011 2:36 PM
>>> To: [hidden email]
>>> Subject: [CEE-DISCUSSION-LIST] Log Protocols
>>>
>>> As we all know, event records are most useful if we can collect them
>>> in a central location for searching, correlating, and archiving.
>>> However, it is my opinion that there are no good protocols out there
>>> to support this.
>>>
>>> Syslog has been the de facto log transport for years. While it does
>>> have its benefits to being small and simple, it does not provide the
>>> security and reliability protections that some environments need.
>>> Syslog over TLS/DTLS and other tunneled Syslog protocols only meet
>>> some of the requirements.
>>>
>>> Other protocols, such as SOAP, are overkill for the limited amount of
>>> data most event records contain. BEEP has gotten some renewed interest
>>> as of late, but it seems to be complex and uses synchronous messaging
>>> channels.
>>>
>>> RELP (http://www.librelp.com/relp.html) seems to be one of the few
>>> standouts in this area, but I don't think it is supported beyond
>>> rsyslog.
>>>
>>> What if you were given the opportunity to design your own log protocol?
>>> Something that was royalty-free, non-exclusive license that was made
>>> available with commercial-friendly licensing. What would your
>>> requirements be? Are there protocols out there that are already "good
>>> enough"?
>>>
>>> Here is a brief listing of some requirements to start:
>>>   +  reliable :: message-level ACKs
>>>   +  parallel, asynchronous batch sending :: why should I have to wait
>>> for ACKs before sending messages
>>>   +  batch ACKs
>>>   +  optional encryption
>>>   +  integrity validation
>>>   +  anti-replay protection
>>>   +  anti-spoof protection
>>>   +  non-repudiation
>>>   +  compression
>>>   +  event record affinity :: group related event records (by type,
>>> producer,
>>> etc.)
>>>   +  support for structured event records  (e.g., CEE)
>>>   +  support for semi- or un-structured event records (e.g., debug
>>> logs,
>>> Syslog)
>>>   +  support for transmitting related data :: send the packet/PCAP
>>> with the IDS alert; the malicious file with the Anti-Virus alert;
>>> video evidence with the tripped alarm...
>>>
>>>
>>>
>>> William Heinbockel
>>> Infosec Engineer, Sr.
>>> The MITRE Corporation
>>> 202 Burlington Rd. MS S145
>>> Bedford, MA 01730
>>> [hidden email]
>>> 781-271-2615
>>

smime.p7s (4K) Download Attachment