Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

mdavidson

As mentioned by Jason and hinted at by Bret, I think this all really comes back down to the idea that we need a set of concrete use cases. I had a variety of lengthy (and probably mundane) responses typed out, and I’ve attempted to boil them down to the key takeaways:

 

The current design had these core requirements and assumptions:

- [Assumption] Extended Headers wouldn’t exist beyond a single sharing group

- [Assumption] Anyone who implements specific Extended Headers (but not necessarily extended headers in general) will also be able to do any necessary format translations while preserving semantics

- [Requirement – sort of] The current solution enables XML schema validation of Extended Headers, which is a nice to have

 

Against these considerations, I’ll assert that the current design is reasonable. However, we should challenge and attempt to improve these considerations (which is what I think we’re doing here). For instance, I agree with the “gut feeling” that a set of key/value pairs (value meaning a scalar, not an object) should be sufficient for nearly every use case. We should dig deeper in an attempt to validate this.

 

There’s also a vision that has come to the forefront since the publication of TAXII 1.1: Federated information sharing. TAXII 1.1 was designed for individual information sharing groups (note the “we can do all these architectures” statements in the scope section of the TAXII Overview). At the time, attempting to solve for federated information sharing seemed like an attempt to “boil the ocean”, and was therefore determined to be future consideration. It’s possible that the TAXII community has evolved to the point where there’s a solid foundation for solving federated information sharing.

                This means that Bret’s scenario of preserving general extended headers across multiple sharing groups – including across intermediary groups that don’t understand the extended headers – was not explicitly solved for. Within the current design, all participant groups would have to coordinate semantics and representations ahead of time.

 

My opinion is that the actionable item is this: Validate that key/value pairs, when value is restricted to a scalar/primitive (string, int, etc), can solve most Extended Header use cases. If this can be validated, amend the TAXII Specification as such. I think this will have the effect of addressing a number of the challenges mentioned so far, simplifying TAXII in general (always a positive), and (though the aforementioned simplification) enabling support for federated information sharing use cases in the future.

 

Thoughts on this? If the actionable item is agreed upon by enough people, I can add it to the TAXII-Specifications tracker.

 

Thank you.

-Mark

 

From: Jason Keirstead [mailto:[hidden email]]
Sent: Friday, March 27, 2015 5:42 PM
To: taxii-discussion-list Trusted Automated Exchange of Indicator In
Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5

 

I have a hard time coming up with a possible headers use case that a simple key/value pair would not cover. 

 

Maybe extended headers and Status do not need to allow such complex structures? Is there actually a projected need/use case for this?
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown



-----"Jordan, Bret" <[hidden email]> wrote: -----

To: <[hidden email]>
From: "Jordan, Bret" <[hidden email]>
Date: 03/27/2015 04:54PM
Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5

Very good points and I had those are same questions while writing the specification. The response I got from the community and others was, any organization that wants to use extended headers will need to know how to use them and thus they will have the internal code logic to process them. The same problem exists with the “Content” object where the payload of data, per the specification, can contain mixed data.  Meaning that XML-TAXII today can transport JSON, YAML, CSV, or other payload data.. I do not see someone taking raw TAXII messages, storying them and then, and later trying to convert them.  They will consume them and process the data that is within them. 

 

The way I see it…..

 

Groups 1 sends JSON based TAXII message that contains extended headers to Group 2

If Group 2 knows how to use that data and understands everything in it, it can then pull all of the data out and use it.  It is now in a variables / structs in memory.

 

Group 2 can now create a new TAXII message that contains the same extended header data but in XML and send to Group 3, assuming Group 2 and Group 3 have defined a way to communicate extended headers, just like Group 1 and Group 2 did.  

 

But that means if Group1, Group2, and Group3 want to use the same extended headers, then they all need to know how to use them and they will all need to make sure their code can support what ever they are trying to do, that is the problem with open, and non-defined fields.

 

Another option for Group 1, Group 2, and Group 3, is that Group 1 could transmit an XML based extended header in his JSON TAXII message. 

 

Thanks,

 

Bret

 

 

 

Bret Jordan CISSP

Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

 

On Mar 27, 2015, at 12:59 PM, Jacobsen, Jasen W. <jasenj1@...> wrote:

 

I believe there is a fundamental problem with Extended Headers and Status Details. I don't think there is any way to preserve the arbitrary structured content from one representation to another without the translating algorithms having knowledge of every other format. Read on for details.

 

2.2.2. Extended Headers and Status Details 

Values of an Extended Header or Status Detail can be any content, including other JSON objects, and appear in the body of their respective sub-elements. In this binding, the value undergoes lax processing - if the provider of the third party value includes JSON elements that conform to some other JSON schema then JSON validation can check for schema conformance but lack of a schema does not cause validation to fail. 

 

How would this map to & from XML?

For example, you might have a JSON extended header such as:

"extended_headers":{
    "myJSONHeader"
:{"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]
    
}     
 
}

 

Which would map to XML as:

<Extended_Header name="myJSONHeader">
     {"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]}
</Extended_Header>

 

And then back to JSON as ???. Is the value contained in the element a string or is it JSON?

"extended_headers" : {

  "myJSONHeader": "{\"some_things\":[{\"index\":1,\"name\":\"Foo\"},{\"index\":2,\"name\":\"Bar\"},{\"index\":3,\"name\":\"Baz\"}]}"

}

----

Conversely, if you have a TAXII XML extended header it could look like this:

<Extended_Headers>
  <Extended_Header
name="myXMLHeader">
    <list_of_things
xmlns="http://myschema.com">
        <foo
item="1">Ack.</foo>
        <foo
item="2">Ick.</foo>
        <foo
item="3">Oop.</foo>       
    </list_of_things>
  </Extended_Header>
</Extended_Headers>    

 

I believe this would map to JSON as:

"extended_headers":{
     "myXmlHeader":"<list_of_things xmlns=\"http://myschema.com\">
        <foo item=\"1\">Ack.</foo>
        <foo item=\"2\">Ick.</foo>
        <foo item=\"3\">Oop.</foo>       
    </list_of_things>"

 }

 

Now let's round-trip it back to XML:

<Extended_Headers>
  <Extended_Header name="myXMLHeader">

???? What goes here?

  </Extended_Header>

</Extended_Headers>

  

When the XML was converted to JSON, the extended header value changed from a block of XML to a string. When going from JSON back to XML, there really is no way to know that the value is supposed to be a block of XML. Even parsing it, we don't know if the value really was a string that happened to look like XML. I think the most likely thing to happen would be the XML string would be escaped (">" to "&gt;" etc.)

 

The problem here is not with the JSON mapping, but with the ability of the Extended Headers & Status Details to contain complex structured values. I don't think there is any way to preserve the structured content from one representation to another without the translating algorithms having knowledge of every other format. (Imagine if we threw a YAML representation into the mix. If an extended header was allowed to contain YAML structures, how would XML & JSON handle that?)

 

- Jasen.

 



[attachment "signature.asc" removed by Jason Keirstead/CanEast/IBM]

Reply | Threaded
Open this post in threaded view
|

Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

Jordan, Bret
That action item seems very reasonable. 

Bret 

Sent from my Commodore 64

On Mar 30, 2015, at 7:03 AM, Davidson II, Mark S <[hidden email]> wrote:

As mentioned by Jason and hinted at by Bret, I think this all really comes back down to the idea that we need a set of concrete use cases. I had a variety of lengthy (and probably mundane) responses typed out, and I’ve attempted to boil them down to the key takeaways:

 

The current design had these core requirements and assumptions:

- [Assumption] Extended Headers wouldn’t exist beyond a single sharing group

- [Assumption] Anyone who implements specific Extended Headers (but not necessarily extended headers in general) will also be able to do any necessary format translations while preserving semantics

- [Requirement – sort of] The current solution enables XML schema validation of Extended Headers, which is a nice to have

 

Against these considerations, I’ll assert that the current design is reasonable. However, we should challenge and attempt to improve these considerations (which is what I think we’re doing here). For instance, I agree with the “gut feeling” that a set of key/value pairs (value meaning a scalar, not an object) should be sufficient for nearly every use case. We should dig deeper in an attempt to validate this.

 

There’s also a vision that has come to the forefront since the publication of TAXII 1.1: Federated information sharing. TAXII 1.1 was designed for individual information sharing groups (note the “we can do all these architectures” statements in the scope section of the TAXII Overview). At the time, attempting to solve for federated information sharing seemed like an attempt to “boil the ocean”, and was therefore determined to be future consideration. It’s possible that the TAXII community has evolved to the point where there’s a solid foundation for solving federated information sharing.

                This means that Bret’s scenario of preserving general extended headers across multiple sharing groups – including across intermediary groups that don’t understand the extended headers – was not explicitly solved for. Within the current design, all participant groups would have to coordinate semantics and representations ahead of time.

 

My opinion is that the actionable item is this: Validate that key/value pairs, when value is restricted to a scalar/primitive (string, int, etc), can solve most Extended Header use cases. If this can be validated, amend the TAXII Specification as such. I think this will have the effect of addressing a number of the challenges mentioned so far, simplifying TAXII in general (always a positive), and (though the aforementioned simplification) enabling support for federated information sharing use cases in the future.

 

Thoughts on this? If the actionable item is agreed upon by enough people, I can add it to the TAXII-Specifications tracker.

 

Thank you.

-Mark

 

From: Jason Keirstead [[hidden email]]
Sent: Friday, March 27, 2015 5:42 PM
To: taxii-discussion-list Trusted Automated Exchange of Indicator In
Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5

 

I have a hard time coming up with a possible headers use case that a simple key/value pair would not cover. 

 

Maybe extended headers and Status do not need to allow such complex structures? Is there actually a projected need/use case for this?
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown



-----"Jordan, Bret" <[hidden email]> wrote: -----

To: <[hidden email]>
From: "Jordan, Bret" <[hidden email]>
Date: 03/27/2015 04:54PM
Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5

Very good points and I had those are same questions while writing the specification. The response I got from the community and others was, any organization that wants to use extended headers will need to know how to use them and thus they will have the internal code logic to process them. The same problem exists with the “Content” object where the payload of data, per the specification, can contain mixed data.  Meaning that XML-TAXII today can transport JSON, YAML, CSV, or other payload data.. I do not see someone taking raw TAXII messages, storying them and then, and later trying to convert them.  They will consume them and process the data that is within them. 

 

The way I see it…..

 

Groups 1 sends JSON based TAXII message that contains extended headers to Group 2

If Group 2 knows how to use that data and understands everything in it, it can then pull all of the data out and use it.  It is now in a variables / structs in memory.

 

Group 2 can now create a new TAXII message that contains the same extended header data but in XML and send to Group 3, assuming Group 2 and Group 3 have defined a way to communicate extended headers, just like Group 1 and Group 2 did.  

 

But that means if Group1, Group2, and Group3 want to use the same extended headers, then they all need to know how to use them and they will all need to make sure their code can support what ever they are trying to do, that is the problem with open, and non-defined fields.

 

Another option for Group 1, Group 2, and Group 3, is that Group 1 could transmit an XML based extended header in his JSON TAXII message. 

 

Thanks,

 

Bret

 

 

 

Bret Jordan CISSP

Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

 

On Mar 27, 2015, at 12:59 PM, Jacobsen, Jasen W. <jasenj1@...> wrote:

 

I believe there is a fundamental problem with Extended Headers and Status Details. I don't think there is any way to preserve the arbitrary structured content from one representation to another without the translating algorithms having knowledge of every other format. Read on for details.

 

2.2.2. Extended Headers and Status Details 

Values of an Extended Header or Status Detail can be any content, including other JSON objects, and appear in the body of their respective sub-elements. In this binding, the value undergoes lax processing - if the provider of the third party value includes JSON elements that conform to some other JSON schema then JSON validation can check for schema conformance but lack of a schema does not cause validation to fail. 

 

How would this map to & from XML?

For example, you might have a JSON extended header such as:

"extended_headers":{
    "myJSONHeader"
:{"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]
    
}     
 
}

 

Which would map to XML as:

<Extended_Header name="myJSONHeader">
     {"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]}
</Extended_Header>

 

And then back to JSON as ???. Is the value contained in the element a string or is it JSON?

"extended_headers" : {

  "myJSONHeader": "{\"some_things\":[{\"index\":1,\"name\":\"Foo\"},{\"index\":2,\"name\":\"Bar\"},{\"index\":3,\"name\":\"Baz\"}]}"

}

----

Conversely, if you have a TAXII XML extended header it could look like this:

<Extended_Headers>
  <Extended_Header
name="myXMLHeader">
    <list_of_things
xmlns="http://myschema.com">
        <foo
item="1">Ack.</foo>
        <foo
item="2">Ick.</foo>
        <foo
item="3">Oop.</foo>       
    </list_of_things>
  </Extended_Header>
</Extended_Headers>    

 

I believe this would map to JSON as:

"extended_headers":{
     "myXmlHeader":"<list_of_things xmlns=\"http://myschema.com\">
        <foo item=\"1\">Ack.</foo>
        <foo item=\"2\">Ick.</foo>
        <foo item=\"3\">Oop.</foo>       
    </list_of_things>"

 }

 

Now let's round-trip it back to XML:

<Extended_Headers>
  <Extended_Header name="myXMLHeader">

???? What goes here?

  </Extended_Header>

</Extended_Headers>

  

When the XML was converted to JSON, the extended header value changed from a block of XML to a string. When going from JSON back to XML, there really is no way to know that the value is supposed to be a block of XML. Even parsing it, we don't know if the value really was a string that happened to look like XML. I think the most likely thing to happen would be the XML string would be escaped (">" to "&gt;" etc.)

 

The problem here is not with the JSON mapping, but with the ability of the Extended Headers & Status Details to contain complex structured values. I don't think there is any way to preserve the structured content from one representation to another without the translating algorithms having knowledge of every other format. (Imagine if we threw a YAML representation into the mix. If an extended header was allowed to contain YAML structures, how would XML & JSON handle that?)

 

- Jasen.

 



[attachment "signature.asc" removed by Jason Keirstead/CanEast/IBM]

Reply | Threaded
Open this post in threaded view
|

Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

Sergey Polzunov
Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/primitive can solve most  use cases.

Sergey Polzunov
Intelworks




> On 30 Mar 2015, at 16:19, Jordan, Bret <[hidden email]> wrote:
>
> That action item seems very reasonable.
>
> Bret
>
> Sent from my Commodore 64
>
> On Mar 30, 2015, at 7:03 AM, Davidson II, Mark S <[hidden email]> wrote:
>
>> As mentioned by Jason and hinted at by Bret, I think this all really comes back down to the idea that we need a set of concrete use cases. I had a variety of lengthy (and probably mundane) responses typed out, and I’ve attempted to boil them down to the key takeaways:
>>  
>> The current design had these core requirements and assumptions:
>> - [Assumption] Extended Headers wouldn’t exist beyond a single sharing group
>> - [Assumption] Anyone who implements specific Extended Headers (but not necessarily extended headers in general) will also be able to do any necessary format translations while preserving semantics
>> - [Requirement – sort of] The current solution enables XML schema validation of Extended Headers, which is a nice to have
>>  
>> Against these considerations, I’ll assert that the current design is reasonable. However, we should challenge and attempt to improve these considerations (which is what I think we’re doing here). For instance, I agree with the “gut feeling” that a set of key/value pairs (value meaning a scalar, not an object) should be sufficient for nearly every use case. We should dig deeper in an attempt to validate this.
>>  
>> There’s also a vision that has come to the forefront since the publication of TAXII 1.1: Federated information sharing. TAXII 1.1 was designed for individual information sharing groups (note the “we can do all these architectures” statements in the scope section of the TAXII Overview). At the time, attempting to solve for federated information sharing seemed like an attempt to “boil the ocean”, and was therefore determined to be future consideration. It’s possible that the TAXII community has evolved to the point where there’s a solid foundation for solving federated information sharing.
>>                 This means that Bret’s scenario of preserving general extended headers across multiple sharing groups – including across intermediary groups that don’t understand the extended headers – was not explicitly solved for. Within the current design, all participant groups would have to coordinate semantics and representations ahead of time.
>>  
>> My opinion is that the actionable item is this: Validate that key/value pairs, when value is restricted to a scalar/primitive (string, int, etc), can solve most Extended Header use cases. If this can be validated, amend the TAXII Specification as such. I think this will have the effect of addressing a number of the challenges mentioned so far, simplifying TAXII in general (always a positive), and (though the aforementioned simplification) enabling support for federated information sharing use cases in the future.
>>  
>> Thoughts on this? If the actionable item is agreed upon by enough people, I can add it to the TAXII-Specifications tracker.
>>  
>> Thank you.
>> -Mark
>>  
>> From: Jason Keirstead [mailto:[hidden email]]
>> Sent: Friday, March 27, 2015 5:42 PM
>> To: taxii-discussion-list Trusted Automated Exchange of Indicator In
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>  
>> I have a hard time coming up with a possible headers use case that a simple key/value pair would not cover.
>>  
>> Maybe extended headers and Status do not need to allow such complex structures? Is there actually a projected need/use case for this?
>> -
>> Jason Keirstead
>> Product Architect, Security Intelligence, IBM Security Systems
>> www.ibm.com/security | www.securityintelligence.com
>>
>> Without data, all you are is just another person with an opinion - Unknown
>>
>>
>> -----"Jordan, Bret" <[hidden email]> wrote: -----
>> To: <[hidden email]>
>> From: "Jordan, Bret" <[hidden email]>
>> Date: 03/27/2015 04:54PM
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>
>> Very good points and I had those are same questions while writing the specification. The response I got from the community and others was, any organization that wants to use extended headers will need to know how to use them and thus they will have the internal code logic to process them. The same problem exists with the “Content” object where the payload of data, per the specification, can contain mixed data.  Meaning that XML-TAXII today can transport JSON, YAML, CSV, or other payload data.. I do not see someone taking raw TAXII messages, storying them and then, and later trying to convert them.  They will consume them and process the data that is within them.
>>  
>> The way I see it…..
>>  
>> Groups 1 sends JSON based TAXII message that contains extended headers to Group 2
>> If Group 2 knows how to use that data and understands everything in it, it can then pull all of the data out and use it.  It is now in a variables / structs in memory.
>>  
>> Group 2 can now create a new TAXII message that contains the same extended header data but in XML and send to Group 3, assuming Group 2 and Group 3 have defined a way to communicate extended headers, just like Group 1 and Group 2 did.  
>>  
>> But that means if Group1, Group2, and Group3 want to use the same extended headers, then they all need to know how to use them and they will all need to make sure their code can support what ever they are trying to do, that is the problem with open, and non-defined fields.
>>  
>> Another option for Group 1, Group 2, and Group 3, is that Group 1 could transmit an XML based extended header in his JSON TAXII message.
>>  
>> Thanks,
>>  
>> Bret
>>  
>>  
>>  
>> Bret Jordan CISSP
>> Director of Security Architecture and Standards | Office of the CTO
>> Blue Coat Systems
>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
>>  
>> On Mar 27, 2015, at 12:59 PM, Jacobsen, Jasen W. <[hidden email]> wrote:
>>  
>> I believe there is a fundamental problem with Extended Headers and Status Details. I don't think there is any way to preserve the arbitrary structured content from one representation to another without the translating algorithms having knowledge of every other format. Read on for details.
>>  
>> 2.2.2. Extended Headers and Status Details
>> Values of an Extended Header or Status Detail can be any content, including other JSON objects, and appear in the body of their respective sub-elements. In this binding, the value undergoes lax processing - if the provider of the third party value includes JSON elements that conform to some other JSON schema then JSON validation can check for schema conformance but lack of a schema does not cause validation to fail.
>>  
>> How would this map to & from XML?
>> For example, you might have a JSON extended header such as:
>> "extended_headers":{
>>     "myJSONHeader":{"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]
>>     }    
>>  }
>>  
>> Which would map to XML as:
>> <Extended_Header name="myJSONHeader">
>>      {"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]}
>> </Extended_Header>
>>  
>> And then back to JSON as ???. Is the value contained in the element a string or is it JSON?
>> "extended_headers" : {
>>   "myJSONHeader": "{\"some_things\":[{\"index\":1,\"name\":\"Foo\"},{\"index\":2,\"name\":\"Bar\"},{\"index\":3,\"name\":\"Baz\"}]}"
>> }
>> ----
>> Conversely, if you have a TAXII XML extended header it could look like this:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>>     <list_of_things xmlns="http://myschema.com">
>>         <foo item="1">Ack.</foo>
>>         <foo item="2">Ick.</foo>
>>         <foo item="3">Oop.</foo>        
>>     </list_of_things>
>>   </Extended_Header>
>> </Extended_Headers>    
>>  
>> I believe this would map to JSON as:
>> "extended_headers":{
>>      "myXmlHeader":"<list_of_things xmlns=\"http://myschema.com\">
>>         <foo item=\"1\">Ack.</foo>
>>         <foo item=\"2\">Ick.</foo>
>>         <foo item=\"3\">Oop.</foo>        
>>     </list_of_things>"
>>  }
>>  
>> Now let's round-trip it back to XML:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>> ???? What goes here?
>>   </Extended_Header>
>> </Extended_Headers>
>>  
>> When the XML was converted to JSON, the extended header value changed from a block of XML to a string. When going from JSON back to XML, there really is no way to know that the value is supposed to be a block of XML. Even parsing it, we don't know if the value really was a string that happened to look like XML. I think the most likely thing to happen would be the XML string would be escaped (">" to "&gt;" etc.)
>>  
>> The problem here is not with the JSON mapping, but with the ability of the Extended Headers & Status Details to contain complex structured values. I don't think there is any way to preserve the structured content from one representation to another without the translating algorithms having knowledge of every other format. (Imagine if we threw a YAML representation into the mix. If an extended header was allowed to contain YAML structures, how would XML & JSON handle that?)
>>  
>> - Jasen.
>>  
>>
>>
>> [attachment "signature.asc" removed by Jason Keirstead/CanEast/IBM]
Reply | Threaded
Open this post in threaded view
|

Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

Jason Keirstead

The one part of this I don't really agree with is "[Assumption] Extended Headers wouldn’t exist beyond a single sharing group"

When I envision how extended headers could be used, they are most likely going to often be used by pairs of STIX/TAXII applications that want to exchange some type of information not covered by one of the standards... either because it is too early (still in draft stage), or it's proprietary to the application developer. One only needs to look at HTTP headers themselves for examples of the wide variety of use cases they can fulfil. I don't feel like we should be making the assumption that headers would be isolated to a sharing group... the use of them would have more to do with the tools implementing TAXII than the groups using the tools, I think...

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Inactive hide details for Sergey Polzunov ---2015/03/30 11:44:02 AM---Good point. I agree that Extended Headers as the key/valuSergey Polzunov ---2015/03/30 11:44:02 AM---Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/

From: Sergey Polzunov <[hidden email]>
To: <[hidden email]>
Date: 2015/03/30 11:44 AM
Subject: Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)





Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/primitive can solve most  use cases.

Sergey Polzunov
Intelworks




> On 30 Mar 2015, at 16:19, Jordan, Bret <[hidden email]> wrote:
>
> That action item seems very reasonable.
>
> Bret
>
> Sent from my Commodore 64
>
> On Mar 30, 2015, at 7:03 AM, Davidson II, Mark S <[hidden email]> wrote:
>
>> As mentioned by Jason and hinted at by Bret, I think this all really comes back down to the idea that we need a set of concrete use cases. I had a variety of lengthy (and probably mundane) responses typed out, and I’ve attempted to boil them down to the key takeaways:
>>  
>> The current design had these core requirements and assumptions:
>> - [Assumption] Extended Headers wouldn’t exist beyond a single sharing group
>> - [Assumption] Anyone who implements specific Extended Headers (but not necessarily extended headers in general) will also be able to do any necessary format translations while preserving semantics
>> - [Requirement – sort of] The current solution enables XML schema validation of Extended Headers, which is a nice to have
>>  
>> Against these considerations, I’ll assert that the current design is reasonable. However, we should challenge and attempt to improve these considerations (which is what I think we’re doing here). For instance, I agree with the “gut feeling” that a set of key/value pairs (value meaning a scalar, not an object) should be sufficient for nearly every use case. We should dig deeper in an attempt to validate this.
>>  
>> There’s also a vision that has come to the forefront since the publication of TAXII 1.1: Federated information sharing. TAXII 1.1 was designed for individual information sharing groups (note the “we can do all these architectures” statements in the scope section of the TAXII Overview). At the time, attempting to solve for federated information sharing seemed like an attempt to “boil the ocean”, and was therefore determined to be future consideration. It’s possible that the TAXII community has evolved to the point where there’s a solid foundation for solving federated information sharing.
>>                 This means that Bret’s scenario of preserving general extended headers across multiple sharing groups – including across intermediary groups that don’t understand the extended headers – was not explicitly solved for. Within the current design, all participant groups would have to coordinate semantics and representations ahead of time.
>>  
>> My opinion is that the actionable item is this: Validate that key/value pairs, when value is restricted to a scalar/primitive (string, int, etc), can solve most Extended Header use cases. If this can be validated, amend the TAXII Specification as such. I think this will have the effect of addressing a number of the challenges mentioned so far, simplifying TAXII in general (always a positive), and (though the aforementioned simplification) enabling support for federated information sharing use cases in the future.
>>  
>> Thoughts on this? If the actionable item is agreed upon by enough people, I can add it to the TAXII-Specifications tracker.
>>  
>> Thank you.
>> -Mark
>>  
>> From: Jason Keirstead [
[hidden email]]
>> Sent: Friday, March 27, 2015 5:42 PM
>> To: taxii-discussion-list Trusted Automated Exchange of Indicator In
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>  
>> I have a hard time coming up with a possible headers use case that a simple key/value pair would not cover.
>>  
>> Maybe extended headers and Status do not need to allow such complex structures? Is there actually a projected need/use case for this?
>> -
>> Jason Keirstead
>> Product Architect, Security Intelligence, IBM Security Systems
>>
www.ibm.com/security | www.securityintelligence.com
>>
>> Without data, all you are is just another person with an opinion - Unknown
>>
>>
>> -----"Jordan, Bret" <[hidden email]> wrote: -----
>> To: <[hidden email]>
>> From: "Jordan, Bret" <[hidden email]>
>> Date: 03/27/2015 04:54PM
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>
>> Very good points and I had those are same questions while writing the specification. The response I got from the community and others was, any organization that wants to use extended headers will need to know how to use them and thus they will have the internal code logic to process them. The same problem exists with the “Content” object where the payload of data, per the specification, can contain mixed data.  Meaning that XML-TAXII today can transport JSON, YAML, CSV, or other payload data.. I do not see someone taking raw TAXII messages, storying them and then, and later trying to convert them.  They will consume them and process the data that is within them.
>>  
>> The way I see it…..
>>  
>> Groups 1 sends JSON based TAXII message that contains extended headers to Group 2
>> If Group 2 knows how to use that data and understands everything in it, it can then pull all of the data out and use it.  It is now in a variables / structs in memory.
>>  
>> Group 2 can now create a new TAXII message that contains the same extended header data but in XML and send to Group 3, assuming Group 2 and Group 3 have defined a way to communicate extended headers, just like Group 1 and Group 2 did.  
>>  
>> But that means if Group1, Group2, and Group3 want to use the same extended headers, then they all need to know how to use them and they will all need to make sure their code can support what ever they are trying to do, that is the problem with open, and non-defined fields.
>>  
>> Another option for Group 1, Group 2, and Group 3, is that Group 1 could transmit an XML based extended header in his JSON TAXII message.
>>  
>> Thanks,
>>  
>> Bret
>>  
>>  
>>  
>> Bret Jordan CISSP
>> Director of Security Architecture and Standards | Office of the CTO
>> Blue Coat Systems
>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
>>  
>> On Mar 27, 2015, at 12:59 PM, Jacobsen, Jasen W. <[hidden email]> wrote:
>>  
>> I believe there is a fundamental problem with Extended Headers and Status Details. I don't think there is any way to preserve the arbitrary structured content from one representation to another without the translating algorithms having knowledge of every other format. Read on for details.
>>  
>> 2.2.2. Extended Headers and Status Details
>> Values of an Extended Header or Status Detail can be any content, including other JSON objects, and appear in the body of their respective sub-elements. In this binding, the value undergoes lax processing - if the provider of the third party value includes JSON elements that conform to some other JSON schema then JSON validation can check for schema conformance but lack of a schema does not cause validation to fail.
>>  
>> How would this map to & from XML?
>> For example, you might have a JSON extended header such as:
>> "extended_headers":{
>>     "myJSONHeader":{"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]
>>     }    
>>  }
>>  
>> Which would map to XML as:
>> <Extended_Header name="myJSONHeader">
>>      {"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]}
>> </Extended_Header>
>>  
>> And then back to JSON as ???. Is the value contained in the element a string or is it JSON?
>> "extended_headers" : {
>>   "myJSONHeader": "{\"some_things\":[{\"index\":1,\"name\":\"Foo\"},{\"index\":2,\"name\":\"Bar\"},{\"index\":3,\"name\":\"Baz\"}]}"
>> }
>> ----
>> Conversely, if you have a TAXII XML extended header it could look like this:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>>     <list_of_things xmlns="
http://myschema.com">
>>         <foo item="1">Ack.</foo>
>>         <foo item="2">Ick.</foo>
>>         <foo item="3">Oop.</foo>        
>>     </list_of_things>
>>   </Extended_Header>
>> </Extended_Headers>    
>>  
>> I believe this would map to JSON as:
>> "extended_headers":{
>>      "myXmlHeader":"<list_of_things xmlns=\"
<a href="http://myschema.com\">http://myschema.com\">
>>         <foo item=\"1\">Ack.</foo>
>>         <foo item=\"2\">Ick.</foo>
>>         <foo item=\"3\">Oop.</foo>        
>>     </list_of_things>"
>>  }
>>  
>> Now let's round-trip it back to XML:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>> ???? What goes here?
>>   </Extended_Header>
>> </Extended_Headers>
>>  
>> When the XML was converted to JSON, the extended header value changed from a block of XML to a string. When going from JSON back to XML, there really is no way to know that the value is supposed to be a block of XML. Even parsing it, we don't know if the value really was a string that happened to look like XML. I think the most likely thing to happen would be the XML string would be escaped (">" to "&gt;" etc.)
>>  
>> The problem here is not with the JSON mapping, but with the ability of the Extended Headers & Status Details to contain complex structured values. I don't think there is any way to preserve the structured content from one representation to another without the translating algorithms having knowledge of every other format. (Imagine if we threw a YAML representation into the mix. If an extended header was allowed to contain YAML structures, how would XML & JSON handle that?)
>>  
>> - Jasen.
>>  
>>
>>
>> [attachment "signature.asc" removed by Jason Keirstead/CanEast/IBM]



Reply | Threaded
Open this post in threaded view
|

Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

Jordan, Bret
Yes, I see them being more eco-system centric and by definition tool specific.  I also think that a standard object with values restricted to strings or similar should solve the use case needs.

Bret 

Sent from my Commodore 64

On Mar 30, 2015, at 8:59 AM, Jason Keirstead <[hidden email]> wrote:

The one part of this I don't really agree with is "[Assumption] Extended Headers wouldn’t exist beyond a single sharing group"

When I envision how extended headers could be used, they are most likely going to often be used by pairs of STIX/TAXII applications that want to exchange some type of information not covered by one of the standards... either because it is too early (still in draft stage), or it's proprietary to the application developer. One only needs to look at HTTP headers themselves for examples of the wide variety of use cases they can fulfil. I don't feel like we should be making the assumption that headers would be isolated to a sharing group... the use of them would have more to do with the tools implementing TAXII than the groups using the tools, I think...

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif>Sergey Polzunov ---2015/03/30 11:44:02 AM---Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/

From: Sergey Polzunov <[hidden email]>
To: <[hidden email]>
Date: 2015/03/30 11:44 AM
Subject: Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)





Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/primitive can solve most  use cases.

Sergey Polzunov
Intelworks




> On 30 Mar 2015, at 16:19, Jordan, Bret <[hidden email]> wrote:
>
> That action item seems very reasonable.
>
> Bret
>
> Sent from my Commodore 64
>
> On Mar 30, 2015, at 7:03 AM, Davidson II, Mark S <[hidden email]> wrote:
>
>> As mentioned by Jason and hinted at by Bret, I think this all really comes back down to the idea that we need a set of concrete use cases. I had a variety of lengthy (and probably mundane) responses typed out, and I’ve attempted to boil them down to the key takeaways:
>>  
>> The current design had these core requirements and assumptions:
>> - [Assumption] Extended Headers wouldn’t exist beyond a single sharing group
>> - [Assumption] Anyone who implements specific Extended Headers (but not necessarily extended headers in general) will also be able to do any necessary format translations while preserving semantics
>> - [Requirement – sort of] The current solution enables XML schema validation of Extended Headers, which is a nice to have
>>  
>> Against these considerations, I’ll assert that the current design is reasonable. However, we should challenge and attempt to improve these considerations (which is what I think we’re doing here). For instance, I agree with the “gut feeling” that a set of key/value pairs (value meaning a scalar, not an object) should be sufficient for nearly every use case. We should dig deeper in an attempt to validate this.
>>  
>> There’s also a vision that has come to the forefront since the publication of TAXII 1.1: Federated information sharing. TAXII 1.1 was designed for individual information sharing groups (note the “we can do all these architectures” statements in the scope section of the TAXII Overview). At the time, attempting to solve for federated information sharing seemed like an attempt to “boil the ocean”, and was therefore determined to be future consideration. It’s possible that the TAXII community has evolved to the point where there’s a solid foundation for solving federated information sharing.
>>                 This means that Bret’s scenario of preserving general extended headers across multiple sharing groups – including across intermediary groups that don’t understand the extended headers – was not explicitly solved for. Within the current design, all participant groups would have to coordinate semantics and representations ahead of time.
>>  
>> My opinion is that the actionable item is this: Validate that key/value pairs, when value is restricted to a scalar/primitive (string, int, etc), can solve most Extended Header use cases. If this can be validated, amend the TAXII Specification as such. I think this will have the effect of addressing a number of the challenges mentioned so far, simplifying TAXII in general (always a positive), and (though the aforementioned simplification) enabling support for federated information sharing use cases in the future.
>>  
>> Thoughts on this? If the actionable item is agreed upon by enough people, I can add it to the TAXII-Specifications tracker.
>>  
>> Thank you.
>> -Mark
>>  
>> From: Jason Keirstead [
[hidden email]]
>> Sent: Friday, March 27, 2015 5:42 PM
>> To: taxii-discussion-list Trusted Automated Exchange of Indicator In
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>  
>> I have a hard time coming up with a possible headers use case that a simple key/value pair would not cover.
>>  
>> Maybe extended headers and Status do not need to allow such complex structures? Is there actually a projected need/use case for this?
>> -
>> Jason Keirstead
>> Product Architect, Security Intelligence, IBM Security Systems
>>
www.ibm.com/security | www.securityintelligence.com
>>
>> Without data, all you are is just another person with an opinion - Unknown
>>
>>
>> -----"Jordan, Bret" <[hidden email]> wrote: -----
>> To: <[hidden email]>
>> From: "Jordan, Bret" <[hidden email]>
>> Date: 03/27/2015 04:54PM
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>
>> Very good points and I had those are same questions while writing the specification. The response I got from the community and others was, any organization that wants to use extended headers will need to know how to use them and thus they will have the internal code logic to process them. The same problem exists with the “Content” object where the payload of data, per the specification, can contain mixed data.  Meaning that XML-TAXII today can transport JSON, YAML, CSV, or other payload data.. I do not see someone taking raw TAXII messages, storying them and then, and later trying to convert them.  They will consume them and process the data that is within them.
>>  
>> The way I see it…..
>>  
>> Groups 1 sends JSON based TAXII message that contains extended headers to Group 2
>> If Group 2 knows how to use that data and understands everything in it, it can then pull all of the data out and use it.  It is now in a variables / structs in memory.
>>  
>> Group 2 can now create a new TAXII message that contains the same extended header data but in XML and send to Group 3, assuming Group 2 and Group 3 have defined a way to communicate extended headers, just like Group 1 and Group 2 did.  
>>  
>> But that means if Group1, Group2, and Group3 want to use the same extended headers, then they all need to know how to use them and they will all need to make sure their code can support what ever they are trying to do, that is the problem with open, and non-defined fields.
>>  
>> Another option for Group 1, Group 2, and Group 3, is that Group 1 could transmit an XML based extended header in his JSON TAXII message.
>>  
>> Thanks,
>>  
>> Bret
>>  
>>  
>>  
>> Bret Jordan CISSP
>> Director of Security Architecture and Standards | Office of the CTO
>> Blue Coat Systems
>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
>>  
>> On Mar 27, 2015, at 12:59 PM, Jacobsen, Jasen W. <[hidden email]> wrote:
>>  
>> I believe there is a fundamental problem with Extended Headers and Status Details. I don't think there is any way to preserve the arbitrary structured content from one representation to another without the translating algorithms having knowledge of every other format. Read on for details.
>>  
>> 2.2.2. Extended Headers and Status Details
>> Values of an Extended Header or Status Detail can be any content, including other JSON objects, and appear in the body of their respective sub-elements. In this binding, the value undergoes lax processing - if the provider of the third party value includes JSON elements that conform to some other JSON schema then JSON validation can check for schema conformance but lack of a schema does not cause validation to fail.
>>  
>> How would this map to & from XML?
>> For example, you might have a JSON extended header such as:
>> "extended_headers":{
>>     "myJSONHeader":{"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]
>>     }    
>>  }
>>  
>> Which would map to XML as:
>> <Extended_Header name="myJSONHeader">
>>      {"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]}
>> </Extended_Header>
>>  
>> And then back to JSON as ???. Is the value contained in the element a string or is it JSON?
>> "extended_headers" : {
>>   "myJSONHeader": "{\"some_things\":[{\"index\":1,\"name\":\"Foo\"},{\"index\":2,\"name\":\"Bar\"},{\"index\":3,\"name\":\"Baz\"}]}"
>> }
>> ----
>> Conversely, if you have a TAXII XML extended header it could look like this:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>>     <list_of_things xmlns="
http://myschema.com">
>>         <foo item="1">Ack.</foo>
>>         <foo item="2">Ick.</foo>
>>         <foo item="3">Oop.</foo>        
>>     </list_of_things>
>>   </Extended_Header>
>> </Extended_Headers>    
>>  
>> I believe this would map to JSON as:
>> "extended_headers":{
>>      "myXmlHeader":"<list_of_things xmlns=\"
<a href="http://myschema.com\">http://myschema.com\">
>>         <foo item=\"1\">Ack.</foo>
>>         <foo item=\"2\">Ick.</foo>
>>         <foo item=\"3\">Oop.</foo>        
>>     </list_of_things>"
>>  }
>>  
>> Now let's round-trip it back to XML:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>> ???? What goes here?
>>   </Extended_Header>
>> </Extended_Headers>
>>  
>> When the XML was converted to JSON, the extended header value changed from a block of XML to a string. When going from JSON back to XML, there really is no way to know that the value is supposed to be a block of XML. Even parsing it, we don't know if the value really was a string that happened to look like XML. I think the most likely thing to happen would be the XML string would be escaped (">" to "&gt;" etc.)
>>  
>> The problem here is not with the JSON mapping, but with the ability of the Extended Headers & Status Details to contain complex structured values. I don't think there is any way to preserve the structured content from one representation to another without the translating algorithms having knowledge of every other format. (Imagine if we threw a YAML representation into the mix. If an extended header was allowed to contain YAML structures, how would XML & JSON handle that?)
>>  
>> - Jasen.
>>  
>>
>>
>> [attachment "signature.asc" removed by Jason Keirstead/CanEast/IBM]




graycol.gif (144 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

Terry MacDonald
+1 to Mark's list of action items. And I agree with Jason's statement that the extended headers use would most likely be tool-centric. 

Cheers
Terry MacDonald

Cheers
Terry MacDonald


On 31 March 2015 at 02:30, Jordan, Bret <[hidden email]> wrote:
Yes, I see them being more eco-system centric and by definition tool specific.  I also think that a standard object with values restricted to strings or similar should solve the use case needs.

Bret 

Sent from my Commodore 64

On Mar 30, 2015, at 8:59 AM, Jason Keirstead <[hidden email]> wrote:

The one part of this I don't really agree with is "[Assumption] Extended Headers wouldn’t exist beyond a single sharing group"

When I envision how extended headers could be used, they are most likely going to often be used by pairs of STIX/TAXII applications that want to exchange some type of information not covered by one of the standards... either because it is too early (still in draft stage), or it's proprietary to the application developer. One only needs to look at HTTP headers themselves for examples of the wide variety of use cases they can fulfil. I don't feel like we should be making the assumption that headers would be isolated to a sharing group... the use of them would have more to do with the tools implementing TAXII than the groups using the tools, I think...

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif>Sergey Polzunov ---2015/03/30 11:44:02 AM---Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/



From: Sergey Polzunov <[hidden email]>
To: <[hidden email]>
Date: 2015/03/30 11:44 AM
Subject: Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)





Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/primitive can solve most  use cases.

Sergey Polzunov
Intelworks




> On 30 Mar 2015, at 16:19, Jordan, Bret <[hidden email]> wrote:
>
> That action item seems very reasonable.
>
> Bret
>
> Sent from my Commodore 64
>
> On Mar 30, 2015, at 7:03 AM, Davidson II, Mark S <[hidden email]> wrote:
>
>> As mentioned by Jason and hinted at by Bret, I think this all really comes back down to the idea that we need a set of concrete use cases. I had a variety of lengthy (and probably mundane) responses typed out, and I’ve attempted to boil them down to the key takeaways:
>>  
>> The current design had these core requirements and assumptions:
>> - [Assumption] Extended Headers wouldn’t exist beyond a single sharing group
>> - [Assumption] Anyone who implements specific Extended Headers (but not necessarily extended headers in general) will also be able to do any necessary format translations while preserving semantics
>> - [Requirement – sort of] The current solution enables XML schema validation of Extended Headers, which is a nice to have
>>  
>> Against these considerations, I’ll assert that the current design is reasonable. However, we should challenge and attempt to improve these considerations (which is what I think we’re doing here). For instance, I agree with the “gut feeling” that a set of key/value pairs (value meaning a scalar, not an object) should be sufficient for nearly every use case. We should dig deeper in an attempt to validate this.
>>  
>> There’s also a vision that has come to the forefront since the publication of TAXII 1.1: Federated information sharing. TAXII 1.1 was designed for individual information sharing groups (note the “we can do all these architectures” statements in the scope section of the TAXII Overview). At the time, attempting to solve for federated information sharing seemed like an attempt to “boil the ocean”, and was therefore determined to be future consideration. It’s possible that the TAXII community has evolved to the point where there’s a solid foundation for solving federated information sharing.
>>                 This means that Bret’s scenario of preserving general extended headers across multiple sharing groups – including across intermediary groups that don’t understand the extended headers – was not explicitly solved for. Within the current design, all participant groups would have to coordinate semantics and representations ahead of time.
>>  
>> My opinion is that the actionable item is this: Validate that key/value pairs, when value is restricted to a scalar/primitive (string, int, etc), can solve most Extended Header use cases. If this can be validated, amend the TAXII Specification as such. I think this will have the effect of addressing a number of the challenges mentioned so far, simplifying TAXII in general (always a positive), and (though the aforementioned simplification) enabling support for federated information sharing use cases in the future.
>>  
>> Thoughts on this? If the actionable item is agreed upon by enough people, I can add it to the TAXII-Specifications tracker.
>>  
>> Thank you.
>> -Mark
>>  
>> From: Jason Keirstead [
[hidden email]]
>> Sent: Friday, March 27, 2015 5:42 PM
>> To: taxii-discussion-list Trusted Automated Exchange of Indicator In
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>  
>> I have a hard time coming up with a possible headers use case that a simple key/value pair would not cover.
>>  
>> Maybe extended headers and Status do not need to allow such complex structures? Is there actually a projected need/use case for this?
>> -
>> Jason Keirstead
>> Product Architect, Security Intelligence, IBM Security Systems
>>
www.ibm.com/security | www.securityintelligence.com
>>
>> Without data, all you are is just another person with an opinion - Unknown
>>
>>
>> -----"Jordan, Bret" <[hidden email]> wrote: -----
>> To: <[hidden email]>
>> From: "Jordan, Bret" <[hidden email]>
>> Date: 03/27/2015 04:54PM
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>
>> Very good points and I had those are same questions while writing the specification. The response I got from the community and others was, any organization that wants to use extended headers will need to know how to use them and thus they will have the internal code logic to process them. The same problem exists with the “Content” object where the payload of data, per the specification, can contain mixed data.  Meaning that XML-TAXII today can transport JSON, YAML, CSV, or other payload data.. I do not see someone taking raw TAXII messages, storying them and then, and later trying to convert them.  They will consume them and process the data that is within them.
>>  
>> The way I see it…..
>>  
>> Groups 1 sends JSON based TAXII message that contains extended headers to Group 2
>> If Group 2 knows how to use that data and understands everything in it, it can then pull all of the data out and use it.  It is now in a variables / structs in memory.
>>  
>> Group 2 can now create a new TAXII message that contains the same extended header data but in XML and send to Group 3, assuming Group 2 and Group 3 have defined a way to communicate extended headers, just like Group 1 and Group 2 did.  
>>  
>> But that means if Group1, Group2, and Group3 want to use the same extended headers, then they all need to know how to use them and they will all need to make sure their code can support what ever they are trying to do, that is the problem with open, and non-defined fields.
>>  
>> Another option for Group 1, Group 2, and Group 3, is that Group 1 could transmit an XML based extended header in his JSON TAXII message.
>>  
>> Thanks,
>>  
>> Bret
>>  
>>  
>>  
>> Bret Jordan CISSP
>> Director of Security Architecture and Standards | Office of the CTO
>> Blue Coat Systems
>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
>>  
>> On Mar 27, 2015, at 12:59 PM, Jacobsen, Jasen W. <[hidden email]> wrote:
>>  
>> I believe there is a fundamental problem with Extended Headers and Status Details. I don't think there is any way to preserve the arbitrary structured content from one representation to another without the translating algorithms having knowledge of every other format. Read on for details.
>>  
>> 2.2.2. Extended Headers and Status Details
>> Values of an Extended Header or Status Detail can be any content, including other JSON objects, and appear in the body of their respective sub-elements. In this binding, the value undergoes lax processing - if the provider of the third party value includes JSON elements that conform to some other JSON schema then JSON validation can check for schema conformance but lack of a schema does not cause validation to fail.
>>  
>> How would this map to & from XML?
>> For example, you might have a JSON extended header such as:
>> "extended_headers":{
>>     "myJSONHeader":{"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]
>>     }    
>>  }
>>  
>> Which would map to XML as:
>> <Extended_Header name="myJSONHeader">
>>      {"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]}
>> </Extended_Header>
>>  
>> And then back to JSON as ???. Is the value contained in the element a string or is it JSON?
>> "extended_headers" : {
>>   "myJSONHeader": "{\"some_things\":[{\"index\":1,\"name\":\"Foo\"},{\"index\":2,\"name\":\"Bar\"},{\"index\":3,\"name\":\"Baz\"}]}"
>> }
>> ----
>> Conversely, if you have a TAXII XML extended header it could look like this:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>>     <list_of_things xmlns="
http://myschema.com">
>>         <foo item="1">Ack.</foo>
>>         <foo item="2">Ick.</foo>
>>         <foo item="3">Oop.</foo>        
>>     </list_of_things>
>>   </Extended_Header>
>> </Extended_Headers>    
>>  
>> I believe this would map to JSON as:
>> "extended_headers":{
>>      "myXmlHeader":"<list_of_things xmlns=\"
http://myschema.com\">
>>         <foo item=\"1\">Ack.</foo>
>>         <foo item=\"2\">Ick.</foo>
>>         <foo item=\"3\">Oop.</foo>        
>>     </list_of_things>"
>>  }
>>  
>> Now let's round-trip it back to XML:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>> ???? What goes here?
>>   </Extended_Header>
>> </Extended_Headers>
>>  
>> When the XML was converted to JSON, the extended header value changed from a block of XML to a string. When going from JSON back to XML, there really is no way to know that the value is supposed to be a block of XML. Even parsing it, we don't know if the value really was a string that happened to look like XML. I think the most likely thing to happen would be the XML string would be escaped (">" to "&gt;" etc.)
>>  
>> The problem here is not with the JSON mapping, but with the ability of the Extended Headers & Status Details to contain complex structured values. I don't think there is any way to preserve the structured content from one representation to another without the translating algorithms having knowledge of every other format. (Imagine if we threw a YAML representation into the mix. If an extended header was allowed to contain YAML structures, how would XML & JSON handle that?)
>>  
>> - Jasen.
>>  
>>
>>
>> [attachment "signature.asc" removed by Jason Keirstead/CanEast/IBM]




Reply | Threaded
Open this post in threaded view
|

Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

mdavidson

I’ve added a tracker item for modifying (aka simplifying) extended headers [1].

 

Jason, Bret, Terry, thank you for the different opinion on extended headers. You may have changed my opinion.

 

Thank you.

-Mark

[1] https://github.com/TAXIIProject/TAXII-Specifications/issues/65

 

From: Terry MacDonald [mailto:[hidden email]]
Sent: Monday, March 30, 2015 9:30 PM
To: taxii-discussion-list Trusted Automated Exchange of Indicator In
Subject: Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)

 

+1 to Mark's list of action items. And I agree with Jason's statement that the extended headers use would most likely be tool-centric. 

 

Cheers

Terry MacDonald


Cheers
Terry MacDonald

 

 

On 31 March 2015 at 02:30, Jordan, Bret <[hidden email]> wrote:

Yes, I see them being more eco-system centric and by definition tool specific.  I also think that a standard object with values restricted to strings or similar should solve the use case needs.

 

Bret 

Sent from my Commodore 64


On Mar 30, 2015, at 8:59 AM, Jason Keirstead <[hidden email]> wrote:

The one part of this I don't really agree with is "[Assumption] Extended Headers wouldn’t exist beyond a single sharing group"

When I envision how extended headers could be used, they are most likely going to often be used by pairs of STIX/TAXII applications that want to exchange some type of information not covered by one of the standards... either because it is too early (still in draft stage), or it's proprietary to the application developer. One only needs to look at HTTP headers themselves for examples of the wide variety of use cases they can fulfil. I don't feel like we should be making the assumption that headers would be isolated to a sharing group... the use of them would have more to do with the tools implementing TAXII than the groups using the tools, I think...

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif>Sergey Polzunov ---2015/03/30 11:44:02 AM---Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/



From: Sergey Polzunov <[hidden email]>
To: <[hidden email]>
Date: 2015/03/30 11:44 AM
Subject: Re: [TAXII] Extended Header design (was: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5)





Good point. I agree that Extended Headers as the key/value pairs with values restricted to a scalar/primitive can solve most  use cases.

Sergey Polzunov
Intelworks




> On 30 Mar 2015, at 16:19, Jordan, Bret <[hidden email]> wrote:
>
> That action item seems very reasonable.
>
> Bret
>
> Sent from my Commodore 64
>
> On Mar 30, 2015, at 7:03 AM, Davidson II, Mark S <[hidden email]> wrote:
>
>> As mentioned by Jason and hinted at by Bret, I think this all really comes back down to the idea that we need a set of concrete use cases. I had a variety of lengthy (and probably mundane) responses typed out, and I’ve attempted to boil them down to the key takeaways:
>>  
>> The current design had these core requirements and assumptions:
>> - [Assumption] Extended Headers wouldn’t exist beyond a single sharing group
>> - [Assumption] Anyone who implements specific Extended Headers (but not necessarily extended headers in general) will also be able to do any necessary format translations while preserving semantics
>> - [Requirement – sort of] The current solution enables XML schema validation of Extended Headers, which is a nice to have
>>  
>> Against these considerations, I’ll assert that the current design is reasonable. However, we should challenge and attempt to improve these considerations (which is what I think we’re doing here). For instance, I agree with the “gut feeling” that a set of key/value pairs (value meaning a scalar, not an object) should be sufficient for nearly every use case. We should dig deeper in an attempt to validate this.
>>  
>> There’s also a vision that has come to the forefront since the publication of TAXII 1.1: Federated information sharing. TAXII 1.1 was designed for individual information sharing groups (note the “we can do all these architectures” statements in the scope section of the TAXII Overview). At the time, attempting to solve for federated information sharing seemed like an attempt to “boil the ocean”, and was therefore determined to be future consideration. It’s possible that the TAXII community has evolved to the point where there’s a solid foundation for solving federated information sharing.
>>                 This means that Bret’s scenario of preserving general extended headers across multiple sharing groups – including across intermediary groups that don’t understand the extended headers – was not explicitly solved for. Within the current design, all participant groups would have to coordinate semantics and representations ahead of time.
>>  
>> My opinion is that the actionable item is this: Validate that key/value pairs, when value is restricted to a scalar/primitive (string, int, etc), can solve most Extended Header use cases. If this can be validated, amend the TAXII Specification as such. I think this will have the effect of addressing a number of the challenges mentioned so far, simplifying TAXII in general (always a positive), and (though the aforementioned simplification) enabling support for federated information sharing use cases in the future.
>>  
>> Thoughts on this? If the actionable item is agreed upon by enough people, I can add it to the TAXII-Specifications tracker.
>>  
>> Thank you.
>> -Mark
>>  
>> From: Jason Keirstead [[hidden email]]
>> Sent: Friday, March 27, 2015 5:42 PM
>> To: taxii-discussion-list Trusted Automated Exchange of Indicator In
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>  
>> I have a hard time coming up with a possible headers use case that a simple key/value pair would not cover.
>>  
>> Maybe extended headers and Status do not need to allow such complex structures? Is there actually a projected need/use case for this?
>> -
>> Jason Keirstead
>> Product Architect, Security Intelligence, IBM Security Systems
>> www.ibm.com/security | www.securityintelligence.com
>>
>> Without data, all you are is just another person with an opinion - Unknown
>>
>>
>> -----"Jordan, Bret" <[hidden email]> wrote: -----
>> To: <[hidden email]>
>> From: "Jordan, Bret" <[hidden email]>
>> Date: 03/27/2015 04:54PM
>> Subject: Re: [TAXII] [TAXII] RFC and formal submission of JSON Message Binding Spec v0.5
>>
>> Very good points and I had those are same questions while writing the specification. The response I got from the community and others was, any organization that wants to use extended headers will need to know how to use them and thus they will have the internal code logic to process them. The same problem exists with the “Content” object where the payload of data, per the specification, can contain mixed data.  Meaning that XML-TAXII today can transport JSON, YAML, CSV, or other payload data.. I do not see someone taking raw TAXII messages, storying them and then, and later trying to convert them.  They will consume them and process the data that is within them.
>>  
>> The way I see it…..
>>  
>> Groups 1 sends JSON based TAXII message that contains extended headers to Group 2
>> If Group 2 knows how to use that data and understands everything in it, it can then pull all of the data out and use it.  It is now in a variables / structs in memory.
>>  
>> Group 2 can now create a new TAXII message that contains the same extended header data but in XML and send to Group 3, assuming Group 2 and Group 3 have defined a way to communicate extended headers, just like Group 1 and Group 2 did.  
>>  
>> But that means if Group1, Group2, and Group3 want to use the same extended headers, then they all need to know how to use them and they will all need to make sure their code can support what ever they are trying to do, that is the problem with open, and non-defined fields.
>>  
>> Another option for Group 1, Group 2, and Group 3, is that Group 1 could transmit an XML based extended header in his JSON TAXII message.
>>  
>> Thanks,
>>  
>> Bret
>>  
>>  
>>  
>> Bret Jordan CISSP
>> Director of Security Architecture and Standards | Office of the CTO
>> Blue Coat Systems
>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
>>  
>> On Mar 27, 2015, at 12:59 PM, Jacobsen, Jasen W. <[hidden email]> wrote:
>>  
>> I believe there is a fundamental problem with Extended Headers and Status Details. I don't think there is any way to preserve the arbitrary structured content from one representation to another without the translating algorithms having knowledge of every other format. Read on for details.
>>  
>> 2.2.2. Extended Headers and Status Details
>> Values of an Extended Header or Status Detail can be any content, including other JSON objects, and appear in the body of their respective sub-elements. In this binding, the value undergoes lax processing - if the provider of the third party value includes JSON elements that conform to some other JSON schema then JSON validation can check for schema conformance but lack of a schema does not cause validation to fail.
>>  
>> How would this map to & from XML?
>> For example, you might have a JSON extended header such as:
>> "extended_headers":{
>>     "myJSONHeader":{"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]
>>     }    
>>  }
>>  
>> Which would map to XML as:
>> <Extended_Header name="myJSONHeader">
>>      {"some_things":[{"index":1,"name":"Foo"},{"index":2,"name":"Bar"},{"index":3,"name":"Baz"}]}
>> </Extended_Header>
>>  
>> And then back to JSON as ???. Is the value contained in the element a string or is it JSON?
>> "extended_headers" : {
>>   "myJSONHeader": "{\"some_things\":[{\"index\":1,\"name\":\"Foo\"},{\"index\":2,\"name\":\"Bar\"},{\"index\":3,\"name\":\"Baz\"}]}"
>> }
>> ----
>> Conversely, if you have a TAXII XML extended header it could look like this:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>>     <list_of_things xmlns="http://myschema.com">
>>         <foo item="1">Ack.</foo>
>>         <foo item="2">Ick.</foo>
>>         <foo item="3">Oop.</foo>        
>>     </list_of_things>
>>   </Extended_Header>
>> </Extended_Headers>    
>>  
>> I believe this would map to JSON as:
>> "extended_headers":{
>>      "myXmlHeader":"<list_of_things xmlns=\"http://myschema.com\">
>>         <foo item=\"1\">Ack.</foo>
>>         <foo item=\"2\">Ick.</foo>
>>         <foo item=\"3\">Oop.</foo>        
>>     </list_of_things>"
>>  }
>>  
>> Now let's round-trip it back to XML:
>> <Extended_Headers>
>>   <Extended_Header name="myXMLHeader">
>> ???? What goes here?
>>   </Extended_Header>
>> </Extended_Headers>
>>  
>> When the XML was converted to JSON, the extended header value changed from a block of XML to a string. When going from JSON back to XML, there really is no way to know that the value is supposed to be a block of XML. Even parsing it, we don't know if the value really was a string that happened to look like XML. I think the most likely thing to happen would be the XML string would be escaped (">" to "&gt;" etc.)
>>  
>> The problem here is not with the JSON mapping, but with the ability of the Extended Headers & Status Details to contain complex structured values. I don't think there is any way to preserve the structured content from one representation to another without the translating algorithms having knowledge of every other format. (Imagine if we threw a YAML representation into the mix. If an extended header was allowed to contain YAML structures, how would XML & JSON handle that?)
>>  
>> - Jasen.
>>  
>>
>>
>> [attachment "signature.asc" removed by Jason Keirstead/CanEast/IBM]