Quantcast

@cee and MSG content

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

@cee and MSG content

Rainer Gerhards
Hi all,

the CEE spec from early this year says that a syslog message with @cee MUST contain JSON content only, and not a message. Take for example this syslog payload:

@cee:{"f":"1"} some text

As of the spec, this is not CEE-enhanced syslog and as such NO JSON object is to be generated. Instead, this would need to be encoded as

@cee:{"f:":"1", "msg":"some text"}

To create a proper CEE-enhanced message.

I wonder what our position in lumberjack is in regard to this. My impression is that the "invalid" format (the first sample) will be seen in practice.

Comments?

Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Keith Robertson
On 09/12/2012 05:41 AM, Rainer Gerhards wrote:
> Hi all,
>
> the CEE spec from early this year says that a syslog message with @cee MUST contain JSON content only, and not a message. Take for example this syslog payload:
>
> @cee:{"f":"1"} some text
This is bad idea.  Data is being sent outside the JSON formatted string
and will be difficult or (less easy) to parse.
>
> As of the spec, this is not CEE-enhanced syslog and as such NO JSON object is to be generated. Instead, this would need to be encoded as
>
> @cee:{"f:":"1", "msg":"some text"}
Agree this is the proper structured way.  Everything is neatly contained
withing a JSON string. Zero special processing required.
>
> To create a proper CEE-enhanced message.
>
> I wonder what our position in lumberjack is in regard to this. My impression is that the "invalid" format (the first sample) will be seen in practice.
I hope not.  What is the point of having structured messages when we
allow for unstructured content.
>
> Comments?
>
> Rainer
I
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Rainer Gerhards
> -----Original Message-----
> From: Keith Robertson [mailto:[hidden email]]
> Sent: Wednesday, September 12, 2012 2:32 PM
> To: Rainer Gerhards
> Cc: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] @cee and MSG content
>
> On 09/12/2012 05:41 AM, Rainer Gerhards wrote:
> > Hi all,
> >
> > the CEE spec from early this year says that a syslog message with
> @cee MUST contain JSON content only, and not a message. Take for
> example this syslog payload:
> >
> > @cee:{"f":"1"} some text
> This is bad idea.  Data is being sent outside the JSON formatted string
> and will be difficult or (less easy) to parse.
> >
> > As of the spec, this is not CEE-enhanced syslog and as such NO JSON
> object is to be generated. Instead, this would need to be encoded as
> >
> > @cee:{"f:":"1", "msg":"some text"}
> Agree this is the proper structured way.  Everything is neatly
> contained
> withing a JSON string. Zero special processing required.
> >
> > To create a proper CEE-enhanced message.
> >
> > I wonder what our position in lumberjack is in regard to this. My
> impression is that the "invalid" format (the first sample) will be seen
> in practice.
> I hope not.  What is the point of having structured messages when we
> allow for unstructured content.

I am actually glad to read these comments. I thought that things would move down that road (there have been a couple of proposals in the past). So all in all, I can rsyslog keep the way that when it sees data after the JSON content, it does not detect the JSON as CEE-enhanced message (aka the first sample would be non-CEE, the second CEE). Right?
Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Keith Robertson
On 09/12/2012 08:34 AM, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: Keith Robertson [mailto:[hidden email]]
>> Sent: Wednesday, September 12, 2012 2:32 PM
>> To: Rainer Gerhards
>> Cc: [hidden email]
>> Subject: Re: [CEE-DISCUSSION-LIST] @cee and MSG content
>>
>> On 09/12/2012 05:41 AM, Rainer Gerhards wrote:
>>> Hi all,
>>>
>>> the CEE spec from early this year says that a syslog message with
>> @cee MUST contain JSON content only, and not a message. Take for
>> example this syslog payload:
>>> @cee:{"f":"1"} some text
>> This is bad idea.  Data is being sent outside the JSON formatted string
>> and will be difficult or (less easy) to parse.
>>> As of the spec, this is not CEE-enhanced syslog and as such NO JSON
>> object is to be generated. Instead, this would need to be encoded as
>>> @cee:{"f:":"1", "msg":"some text"}
>> Agree this is the proper structured way.  Everything is neatly
>> contained
>> withing a JSON string. Zero special processing required.
>>> To create a proper CEE-enhanced message.
>>>
>>> I wonder what our position in lumberjack is in regard to this. My
>> impression is that the "invalid" format (the first sample) will be seen
>> in practice.
>> I hope not.  What is the point of having structured messages when we
>> allow for unstructured content.
> I am actually glad to read these comments. I thought that things would move down that road (there have been a couple of proposals in the past). So all in all, I can rsyslog keep the way that when it sees data after the JSON content
I would think so.  I'm not sure what Syslog's behavior is or should be
if it receives data after a block of JSON that has been identified to be
CEE via the '@cee:' tag.  I would think you could even discard it.
> , it does not detect the JSON as CEE-enhanced message (aka the first sample would be non-CEE, the second CEE). Right?
IMO: The second example is valid CEE and the first is non-standard.
You've got plain text outside of the structure (i.e. JSON).
> Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Rainer Gerhards
> > I am actually glad to read these comments. I thought that things
> would move down that road (there have been a couple of proposals in the
> past). So all in all, I can rsyslog keep the way that when it sees data
> after the JSON content
> I would think so.  I'm not sure what Syslog's behavior is or should be
> if it receives data after a block of JSON that has been identified to
> be
> CEE via the '@cee:' tag.  I would think you could even discard it.

That is the core of my question ;) The current published CEE spec says that if data follows to the JSON, the whole message is NOT a CEE message (and as such to be treated like any other syslog message). I have had a request to change that behavior and so I wanted to check what the right thing to do is ;)

Rainer

> > , it does not detect the JSON as CEE-enhanced message (aka the first
> sample would be non-CEE, the second CEE). Right?
> IMO: The second example is valid CEE and the first is non-standard.
> You've got plain text outside of the structure (i.e. JSON).
> > Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Keith Robertson
On 09/12/2012 08:45 AM, Rainer Gerhards wrote:
>>> I am actually glad to read these comments. I thought that things
>> would move down that road (there have been a couple of proposals in the
>> past). So all in all, I can rsyslog keep the way that when it sees data
>> after the JSON content
>> I would think so.  I'm not sure what Syslog's behavior is or should be
>> if it receives data after a block of JSON that has been identified to
>> be
>> CEE via the '@cee:' tag.  I would think you could even discard it.
> That is the core of my question ;) The current published CEE spec says that if data follows to the JSON, the whole message is NOT a CEE message (and as such to be treated like any other syslog message). I have had a request to change that behavior and so I wanted to check what the right thing to do is ;)
IMHO, if you announce that you're sending structured data via a 'CEE:'
tag, then the entire message needs to be *structured*.  We're just
making it harder for parsers/processors if we say that it is acceptable
to send both in a single message.  If a source wants to send
unstructured data, then fallback to the old way of doing things.  My 2c.

Cheers,
Keith


>
> Rainer
>
>>> , it does not detect the JSON as CEE-enhanced message (aka the first
>> sample would be non-CEE, the second CEE). Right?
>> IMO: The second example is valid CEE and the first is non-standard.
>> You've got plain text outside of the structure (i.e. JSON).
>>> Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Rainer Gerhards
> -----Original Message-----
> From: Keith Robertson [mailto:[hidden email]]
> Sent: Wednesday, September 12, 2012 4:59 PM
> To: Rainer Gerhards
> Cc: [hidden email]
> Subject: Re: [CEE-DISCUSSION-LIST] @cee and MSG content
>
> On 09/12/2012 08:45 AM, Rainer Gerhards wrote:
> >>> I am actually glad to read these comments. I thought that things
> >> would move down that road (there have been a couple of proposals in
> the
> >> past). So all in all, I can rsyslog keep the way that when it sees
> data
> >> after the JSON content
> >> I would think so.  I'm not sure what Syslog's behavior is or should
> be
> >> if it receives data after a block of JSON that has been identified
> to
> >> be
> >> CEE via the '@cee:' tag.  I would think you could even discard it.
> > That is the core of my question ;) The current published CEE spec
> says that if data follows to the JSON, the whole message is NOT a CEE
> message (and as such to be treated like any other syslog message). I
> have had a request to change that behavior and so I wanted to check
> what the right thing to do is ;)
> IMHO, if you announce that you're sending structured data via a 'CEE:'
> tag, then the entire message needs to be *structured*.  We're just
> making it harder for parsers/processors if we say that it is acceptable
> to send both in a single message.  If a source wants to send
> unstructured data, then fallback to the old way of doing things.  My
> 2c.

That's my PoV as well. But it has been challenged more than once in the past. So if nobody speaks up, I think our position still holds. I'll not permit any changes to rsyslog that changes that.

Thanks for the feedback,
Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Keith Robertson
On 09/12/2012 11:24 AM, Rainer Gerhards wrote:

>
>> -----Original Message-----
>> From: Keith Robertson [mailto:[hidden email]]
>> Sent: Wednesday, September 12, 2012 4:59 PM
>> To: Rainer Gerhards
>> Cc: [hidden email]
>> Subject: Re: [CEE-DISCUSSION-LIST] @cee and MSG content
>>
>> On 09/12/2012 08:45 AM, Rainer Gerhards wrote:
>>>>> I am actually glad to read these comments. I thought that things
>>>> would move down that road (there have been a couple of proposals in
>> the
>>>> past). So all in all, I can rsyslog keep the way that when it sees
>> data
>>>> after the JSON content
>>>> I would think so.  I'm not sure what Syslog's behavior is or should
>> be
>>>> if it receives data after a block of JSON that has been identified
>> to
>>>> be
>>>> CEE via the '@cee:' tag.  I would think you could even discard it.
>>> That is the core of my question ;) The current published CEE spec
>> says that if data follows to the JSON, the whole message is NOT a CEE
>> message (and as such to be treated like any other syslog message). I
>> have had a request to change that behavior and so I wanted to check
>> what the right thing to do is ;)
>> IMHO, if you announce that you're sending structured data via a 'CEE:'
>> tag, then the entire message needs to be *structured*.  We're just
>> making it harder for parsers/processors if we say that it is acceptable
>> to send both in a single message.  If a source wants to send
>> unstructured data, then fallback to the old way of doing things.  My
>> 2c.
> That's my PoV as well. But it has been challenged more than once in the past. So if nobody speaks up, I think our position still holds. I'll not permit any changes to rsyslog that changes that.
Which line in the standard actually permits this condition, BTW?  It is
it simply not addressed?
>
> Thanks for the feedback,
> Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Rainer Gerhards
> > That's my PoV as well. But it has been challenged more than once in
> > the past. So if nobody speaks up, I think our position still holds.
> > I'll not permit any changes to rsyslog that changes that.
> Which line in the standard actually permits this condition, BTW?  

It's NOT permitted, it's forbidden (but this has been challenged several times in the past, because putting a plain text message AFTER the JSON encoding looks so "natural" ;))

> It is
> it simply not addressed?

See
http://cee.mitre.org/language/1.0-beta1/clt.html#appendix-1-cee-over-syslog-transport-mapping

search for "syslog body". It says the body MUST be valid CLS (CEE) JSON. Obviously, having text after the JSON makes the body invalid.

Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: @cee and MSG content

Keith Robertson
On 09/12/2012 11:34 AM, Rainer Gerhards wrote:
>>> That's my PoV as well. But it has been challenged more than once in
>>> the past. So if nobody speaks up, I think our position still holds.
>>> I'll not permit any changes to rsyslog that changes that.
>> Which line in the standard actually permits this condition, BTW?
> It's NOT permitted, it's forbidden (but this has been challenged several times in the past, because putting a plain text message AFTER the JSON encoding looks so "natural" ;))
That's like saying it's acceptable to add an additional M bytes of data
to a TCP datagram when the length is N and the checksum is computed on
that.  Seriously doubt any TCP stacks would accept that condition.  Why
should a CEE stack be different?
>
>> It is
>> it simply not addressed?
> See
> http://cee.mitre.org/language/1.0-beta1/clt.html#appendix-1-cee-over-syslog-transport-mapping
>
> search for "syslog body". It says the body MUST be valid CLS (CEE) JSON. Obviously, having text after the JSON makes the body invalid.
Pretty clear that example 1 is non-compliant.
>
> Rainer
Loading...