[STIX] STIX Community Meeting - November

classic Classic list List threaded Threaded
44 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[STIX] STIX Community Meeting - November

Wunder, John A.

All,

 

This month’s STIX community meeting will be held on Thursday, November 13th at 10am EST.

 

Rather than focus on one particular theme, we’d like to use this month’s meeting to go over a collection of important, albeit unrelated, topics that we haven’t had adequate time to cover during earlier meetings.  Of course, we’re also happy to cover anything you feel the community should talk about – send us an email before Thursday if you have any suggestions.

 

Right now our tentative schedule is:

 

1.       Status Update – where we are, where we’re going (15 minutes)

2.       Documentation – helping you find all the great new content we’ve added (30 minutes)

3.       Data Markings – including a demo of a marking parser (30 minutes)

4.       Hailataxii – Aharon Chernon will present this public TAXII server (15 minutes)

5.       Community Profiles Features & Capabilities (30 minutes)

6.       Others, per your input

I sent an update to the recurring meeting earlier today so that everyone will have it on your calendars. I also attached an iCal invite for just this week’s call. Finally, dial-in and web-share information are below.

 

Thanks,
STIX Project Team

 

.........................................................................................................................................

à Join Lync Meeting      

 

Join by phone

<a href="tel:&#43;1%20(781)%20271-2020">+1 (781) 271-2020 (US)                      English (United States)

<a href="tel:&#43;1%20(703)%20983-2020">+1 (703) 983-2020 (US)                      English (United States)  

Find a local number

 

Conference ID: 81357784

 

Forgot your dial-in PIN? |Help    

 

[!OC([1033])!]

.........................................................................................................................................

 

Quick Tips: Join by Phone for Meeting Audio
  • PRIOR to the meeting make sure to create your PIN (MITRE only, FJ:UCPIN) to use with your MITRE work number. Non-MITRE participants do not utilize a PIN.
  • Select Do not join audio in the Join Meeting Audio window after clicking the Join online meeting link. This indicates you will dial in by phone.
  • To go quickly to the MITRE work number + PIN entry prompts: Press #, # after entering the Conference ID. (External partners, press #,#,#).
More information: How to join a MITRE meeting

Monthly STIX Community Meeting.ics (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

[STIX] Mitre examples not matching all Mitre suggested practices

Eric Bleher
Hello,
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX.
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/.

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!

Regards,
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Wunder, John A.

Hi Eric,

 

Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 

In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

 

That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

 

Hope that helps!

John

 

From: Eric Bleher [mailto:[hidden email]]
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices

 

Hello,
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX.
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/.

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!

Regards,
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Terry MacDonald
Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald

On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,

 

Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 

In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

 

That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

 

Hope that helps!

John

 

From: Eric Bleher [mailto:[hidden email]]
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices

 

Hello,
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX.
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/.

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!

Regards,
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.


Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Collie, Byron S.

Custom Report object that is mapped to a Collection and Source. This is one of the major problems with the current STIX instantiation from our perspective.

If you are going to consume data from multiple sources you need to be able to cross reference individual report objects with other report objectives from different sources. We maintain the report object in its original state and have various related STIX entities created from it. That way for example, you could have one human produced report that contains multiple actors, campaigns, incidents and ttps and it provides a hierarchy for them rather than a matrix from which you have no root. We needed this to allow human intelligence analysts to manipulate data effectively, rate reports overall and tie those ratings back to source to confidence score the source. Dealing with everything at an atomic field within each STIX entity is a technical implementation that is not conducive to human analysis or data manipulation from our perspective.  

 

All reasons why FSISAC and we individual members proposed the report object…..

 

From: Terry MacDonald [mailto:[hidden email]]
Sent: Monday, November 24, 2014 6:47 PM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

 

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?


Cheers
Terry MacDonald

 

On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,

 

Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 

In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

 

That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

 

Hope that helps!

John

 

From: Eric Bleher [mailto:[hidden email]]
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices

 

Hello,
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX.
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/.

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!

Regards,
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Wunder, John A.
In reply to this post by Terry MacDonald
Terry,

That's been suggested a few times before, we're currently tracking it in Github: https://github.com/STIXProject/schemas/issues/221. So you're in good company...

John


From: Terry MacDonald [[hidden email]]
Sent: Monday, November 24, 2014 6:47 PM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald

On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,

 

Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 

In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

 

That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

 

Hope that helps!

John

 

From: Eric Bleher [mailto:[hidden email]]
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices

 

Hello,
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX.
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/.

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!

Regards,
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.


Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Michael P. Stone
In reply to this post by Terry MacDonald

Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)

 

From: Terry MacDonald [mailto:[hidden email]]
Sent: Monday, November 24, 2014 6:47 PM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

 

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?


Cheers
Terry MacDonald

 

On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,

 

Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 

In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

 

That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

 

Hope that helps!

John

 

From: Eric Bleher [mailto:[hidden email]]
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices

 

Hello,
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX.
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/.

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!

Regards,
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Foley, Alexander - GIS

+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.

 

1.      We need them to build complex STIX packages so things can reference each other

a.      thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier

2.      Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.      If we make them mandatory, we get a phenomenal primary key

3.      However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX

a.      It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs

4.      So, IDs need to be untrustworthy between distributed systems

a.      If our IDs are untrustworthy, why make them required?

5.      No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.      thank you UUID… python-stix shows how easy it is to implement this

6.      Let’s namespace this ID field and hope for the best

 

Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.

 

On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.

 

Thanks,

 

Alex

 

From: Michael P. Stone [mailto:[hidden email]]
Sent: Tuesday, November 25, 2014 9:19 AM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)

 

From: Terry MacDonald [[hidden email]]
Sent: Monday, November 24, 2014 6:47 PM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

 

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?


Cheers
Terry MacDonald

 

On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,

 

Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 

In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

 

That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

 

Hope that helps!

John

 

From: Eric Bleher [mailto:[hidden email]]
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices

 

Hello,
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX.
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/.

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!

Regards,
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 


This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Eric Burger
I would offer that when people ask me to compare STIX versus IODEF, one of the strongest concepts in favor of STIX is id/idref. Replicating the same set of data elements over and over again is a recipe for disaster. It is much better to have a single reference and refer to the reference.

I would also offer that mandating id/idref, as Alex points out, will not cure world hunger or incapacitate all of the hackers.

My suggestion: strongly, strongly hint that id/idref really, really should be present. Alex’s list of how to make it workable should be in the How To guide for implementors.

Can anyone come up with a reason not to make it mandatory? The GitHub arguments seem to focus on the difficulty of specifying it in XML Schema, not whether or not it should be mandatory.

Remember, every ‘thing’ that can be made mandatory in CybOX / STIX reduces the complexity of implementation, more especially for servers, which have to deal with seeing things / not seeing things. It’s a lot easier to always expect something and simply reject the message if it’s not there than to try to guess what the client meant if it is not there.

On Nov 25, 2014, at 9:23 AM, Foley, Alexander <[hidden email]> wrote:

+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 
1.      We need them to build complex STIX packages so things can reference each other
a.      thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier
2.      Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
a.      If we make them mandatory, we get a phenomenal primary key
3.      However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX
a.      It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs
4.      So, IDs need to be untrustworthy between distributed systems
a.      If our IDs are untrustworthy, why make them required?
5.      No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”
a.      thank you UUID… python-stix shows how easy it is to implement this
6.      Let’s namespace this ID field and hope for the best
 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.
 
On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
 
Thanks,
 
Alex
 
From: Michael P. Stone [[hidden email]] 
Sent: Tuesday, November 25, 2014 9:19 AM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)
 
From: Terry MacDonald [[hidden email]] 
Sent: Monday, November 24, 2014 6:47 PM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices
 

Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.
 
Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald
 
On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:
Hi Eric,
 
Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):
 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 
In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.
 
That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.
 
Hope that helps!
John
 
From: Eric Bleher [mailto:[hidden email]] 
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices 
 
Hello, 
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards. 
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/. 

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks! 

Regards, 
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

-- 

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.



smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [STIX] EXTERNAL: Re: [STIX] Mitre examples not matching all Mitre suggested practices

Otto Fowler
Are we on track to have the duplication discussion for the next community meeting?  Maybe we can float some ideas or problems ahead of time?


Otto Fowler

Principal Software Engineer

Industrial Defender Solutions
Lockheed Martin Corporation
 
16 Chestnut Street, Suite 300, Foxborough, MA 02035
O  508-718-6726  | E  [hidden email]

On Nov 25, 2014, at 10:11 AM, Eric Burger <[hidden email]> wrote:

I would offer that when people ask me to compare STIX versus IODEF, one of the strongest concepts in favor of STIX is id/idref. Replicating the same set of data elements over and over again is a recipe for disaster. It is much better to have a single reference and refer to the reference.

I would also offer that mandating id/idref, as Alex points out, will not cure world hunger or incapacitate all of the hackers.

My suggestion: strongly, strongly hint that id/idref really, really should be present. Alex’s list of how to make it workable should be in the How To guide for implementors.

Can anyone come up with a reason not to make it mandatory? The GitHub arguments seem to focus on the difficulty of specifying it in XML Schema, not whether or not it should be mandatory.

Remember, every ‘thing’ that can be made mandatory in CybOX / STIX reduces the complexity of implementation, more especially for servers, which have to deal with seeing things / not seeing things. It’s a lot easier to always expect something and simply reject the message if it’s not there than to try to guess what the client meant if it is not there.

On Nov 25, 2014, at 9:23 AM, Foley, Alexander <[hidden email]> wrote:

+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 
1.      We need them to build complex STIX packages so things can reference each other
a.      thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier
2.      Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
a.      If we make them mandatory, we get a phenomenal primary key
3.      However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX
a.      It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs
4.      So, IDs need to be untrustworthy between distributed systems
a.      If our IDs are untrustworthy, why make them required?
5.      No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”
a.      thank you UUID… python-stix shows how easy it is to implement this
6.      Let’s namespace this ID field and hope for the best
 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.
 
On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
 
Thanks,
 
Alex
 
From: Michael P. Stone [[hidden email]] 
Sent: Tuesday, November 25, 2014 9:19 AM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)
 
From: Terry MacDonald [[hidden email]] 
Sent: Monday, November 24, 2014 6:47 PM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices
 

Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.
 
Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald
 
On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:
Hi Eric,
 
Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):
 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 
In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.
 
That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.
 
Hope that helps!
John
 
From: Eric Bleher [mailto:[hidden email]] 
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices 
 
Hello, 
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards. 
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/. 

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks! 

Regards, 
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

-- 

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.



Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Sergey Polzunov
In reply to this post by Foley, Alexander - GIS
In our current thinking, the IDs are a great way to understand intent of
a producer rather then use it to solve any particular problem as a
consumer. After all, even though they might be mandatory and even though
they might match, it says nothing of the validity of content itself.
What mandatory IDs would possible solve is guaranteeing (well.. At least
encouraging) the same content from the same producer always have the
same ID.

Even creating keys based on hashing of content will be tricky
considering some fields might be mandatory and others not, still
allowing for diverging content with the same ID.

We’re going from the assumption that the only road right now, granted
that we have use-cases of correlation, re-assambly, analysis,
visualization, etc., is to at least derive secondary storage / indexes /
etc from STIX packages. Requiring internal IDs of the object levels of
STIX, CyBox, etc. and possibly using the externally provided IDs as
secondary keys/indexes or correlation mechanisms. Depending on what pace
STIX will evolve in, I might argue a whole secondary data-model could be
in order.

This obviously is interesting in the context of versions, multiple
collaborators, authorization, revocation, etc.

We (Intelworks engineering team building our STIX based platform) will
be happy to partake and/or share its experiences / research with the
group in any future STIX community meeting.

Thanks,
Sergey Polzunov
Intelworks


On 25/11/14 15:23, Foley, Alexander wrote:

> +1, my guess is we can have quite the debate on object IDs and settle
> right where we are now… TL/DR; I’m down to make it required but it’s not
> going to save the world.
>
> 1.We need them to build complex STIX packages so things can reference
> each other
>
> a.thank you id element in HTML for proving this concept works, without
> you my JavaScript would be even uglier
>
> 2.Now that we agree an ID of some kind is necessary, do we make it
> mandatory or optional?
>
> a.If we make them mandatory, we get a phenomenal primary key
>
> 3.However, we’re not all making sausage in one factory here… many
> distributed content creators need to make STIX
>
> a.It’s impractical to host a centralized service to coordinate our
> distributed systems to create “guaranteed unique” IDs
>
> 4.So, IDs need to be untrustworthy between distributed systems
>
> a.If our IDs are untrustworthy, why make them required?
>
> 5.No need to give up all hope, we don’t need to end up with foolish IDs
> like 1, 2, 3, 4 everywhere… let’s just make them long enough and random
> enough we can ease collision – “practically unique”
>
> a.thank you UUID… python-stix shows how easy it is
> <https://github.com/STIXProject/python-stix/blob/master/stix/utils/idgen.py>
> to implement this
>
> 6.Let’s namespace this ID field and hope for the best
>
> Re: what you should use as a primary key, I fear that if you rely on
> this ID field at all you may run into situations where you’re writing
> many times a second into your table with identical version / ID pairs
> where someone hasn’t implemented a “practically unique” identifier on
> their end.  I’ve been monkeying around with storing the STIX packages
> with a hash of their contents as the primary key in our key / value
> store to try to go with something more unique, but have yet to build out
> a data model for counting and referencing the hash collisions.
>
> On the other hand, I’m still going to be left with the challenge of
> searching, joining and filtering on multiple parts of these packages no
> matter how I store them… I’m hoping one of the Soltra guys will rescue
> me with a defense of why this fact (and others) make it essential to
> store STIX as STIX and not try to unpack it into a relational database
> that you’re keying on.
>
> Thanks,
>
> Alex
>
> *From:*Michael P. Stone [mailto:[hidden email]]
> *Sent:* Tuesday, November 25, 2014 9:19 AM
> *To:* [hidden email]
> *Subject:* Re: [STIX] Mitre examples not matching all Mitre suggested
> practices
>
> Depending on the implementation there may not be a particularly sane
> thing to use as the ID. Is the ID important enough that you’d rather
> just not have the data? I also think it’s safe to say that there is zero
> chance that anyone can receive data from multiple organizations and make
> correlations entirely via IDs, without having to implement logic to
> compare the values within the objects. (So the problem of finding
> duplicates is going to have to be solved, regardless of the presence or
> absence of indicators.)
>
> *From:*Terry MacDonald [mailto:[hidden email]]
> *Sent:* Monday, November 24, 2014 6:47 PM
> *To:* [hidden email]
> <mailto:[hidden email]>
> *Subject:* Re: [STIX] Mitre examples not matching all Mitre suggested
> practices
>
> Hi John/Eric,
>
> This does bring up an interesting point. Not putting an ID on the
> encapsulated object effectively prevents it from being referred to by
> other future objects. For example if we have an Incident with an
> encapsulated Indicator with an Observable in it, and the Indicator
> doesn't have an ID, then if we subsequently see another similar sighting
> and we are unable to refer to the original Indicator to update it. Or
> maybe we see another incident in another business unit that uses the
> same indicator. We can't effectively communicate that without
> duplicating the indicator - causing problems for consumers who now need
> to identify the similarities between Incidents by diff'ing the two
> Indicator objects. Not having an ID also seems to effectively prevent
> versioning of that object, as we don't have any way to refer to that
> object for versioning purposes.
>
> I would rather see it that all STIX objects MUST have an ID associated
> with them. It would mean that producers and consumers could use the ID,
> timestamp and version fields combined together as a 'primary key' for
> the database for storage (setting timestamp and version to 0 if they
> don't exist). It would simplify programming as the object ID would
> always be available. And it would also allow the object to be referred
> to (and reused) in subsequent objects/relationships.
>
> Also - I'm curious what others use as primary key for their STIX data
> stores? Anyone free to comment?
>
>
> Cheers
> Terry MacDonald
>
> On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Eric,
>
>     Thanks for the note! In this case I think that example does conform
>     to our suggested practices. From the page that you linked, in the
>     “Assigning IDs” section
>     (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):
>
>
>     “As a simple general rule *specifying IDs is not suggested for
>     constructs embedded within other constructs […] where the embedded
>     constructs are really only relevant/valid/important within the
>     context of the enclosing construct*. In other words they provide
>     contextual characterization for the enclosing construct but would
>     not be of interest on their own. The upside of this is slightly less
>     complexity of IDs on everything. The downside is that it would not
>     be possible to reference or pivot on the embedded constructs.”
>
>     In this case, the observable used for the indicator is just that:
>     only used for the indicator pattern. It isn’t referenced anywhere
>     else and does not have any meaningful information used anywhere
>     else. For that reason it isn’t given an ID.
>
>     That said, note that not all STIX content will always conform to the
>     suggested practices. They’re only suggested, not mandatory, because
>     in some cases it can make sense to not follow them. MITRE’s sample
>     content SHOULD always follow them when appropriate (so along those
>     lines please let us know if you find any other cases of this) but
>     that doesn’t apply to either cases when the suggested practices
>     aren’t appropriate or to producers who for whatever reason do not
>     follow them.
>
>     Hope that helps!
>
>     John
>
>     *From:*Eric Bleher [mailto:[hidden email]
>     <mailto:[hidden email]>]
>     *Sent:* Monday, November 24, 2014 11:05 AM
>     *To:* Wunder, John A.
>     *Cc:* stix-discussion-list Structured Threat Information Expression/ST
>     *Subject:* Mitre examples not matching all Mitre suggested practices
>
>     Hello,
>     As probably many, I’ve used the samples available at
>     http://stix.mitre.org/language/version1.1.1/samples.html to grab
>     some ideas about how to export different information into STIX.
>     Especially, concerning the Cybox-MAEC-Stix integration, the STIX
>     file flow_example(3of3)-stix_indicator_combined.xml (found on the
>     "all samples" Zip available on that MITRE page) get an interesting
>     view about the complexity of the different standards.
>     However, arguing with that file with an other STIX system
>     (specifically SoltraEdge), it comes out the file does not have an
>     object ID for every object (some do and some do not). This is
>     against MITRE's own suggested practices
>     http://stixproject.github.io/documentation/suggested-practices/.
>
>     @MITRE team, you might want to review all the samples available on
>     that page, and either rework or remove them. Or at least put a big
>     warning on them! Thanks!
>
>     Regards,
>     ---
>     Eric Bleher
>     Cyber Security - Malware Response & Research
>     DeutscheBank AG, Germany
>
>     --
>
>     Informationen (einschließlich Pflichtangaben) zu einzelnen,
>     innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des
>     Konzerns Deutsche Bank finden Sie unter
>     http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese
>     E-Mail enthält vertrauliche und/ oder rechtlich geschützte
>     Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
>     E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den
>     Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren
>     sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.
>
>     Please refer to http://www.db.com/en/content/eu_disclosures.htmfor
>     information (including mandatory corporate particulars) on selected
>     Deutsche Bank branches and group companies registered or
>     incorporated in the European Union. This e-mail may contain
>     confidential and/or privileged information. If you are not the
>     intended recipient (or have received this e-mail in error) please
>     notify the sender immediately and delete this e-mail. Any
>     unauthorized copying, disclosure or distribution of the material in
>     this e-mail is strictly forbidden.
>
> ------------------------------------------------------------------------
> This message, and any attachments, is for the intended recipient(s)
> only, may contain information that is privileged, confidential and/or
> proprietary and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the
> intended recipient, please delete this message.
Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Aharon
In reply to this post by Foley, Alexander - GIS

Good morning Mr Foley!

 

1.       I’m down to make it required but it’s not going to save the world.

a.      No single STIX change is going to make a huge impact. However, the combination of small changes proposed to STIX will make a large impact.

2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.      I am all for making ID mandatory. I do not see any consumer demand for ID optional either.

3.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.      We should standardize on what a GUID should look like. Agree with the UUID approach.

 

I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.

               

                I will nudge Ben and see if I can get him to respond to this thread. We don’t store the STIX as XML, we use the Mitre STIX API to_dict/from_dict to convert the XML into JSON for storage. Dealing with JSON also happens to be python and developer friendly. However, we had to make decisions on what we wanted to support. Supporting reading native STIX JSON made more sense to us than supporting a system that mapped out all the native STIX elements and attributes into a columnized relational database. We tried this in the past and I didn’t feel like we were in a good place. I know this doesn’t fix your joining challenges, but that is what is Edge is for J Have you tore apart how we lay out the data in Mongo?

 

Aharon

 

From: Foley, Alexander [mailto:[hidden email]]
Sent: Tuesday, November 25, 2014 9:23 AM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.

 

1.       We need them to build complex STIX packages so things can reference each other

a.       thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier

2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.       If we make them mandatory, we get a phenomenal primary key

3.       However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX

a.       It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs

4.       So, IDs need to be untrustworthy between distributed systems

a.       If our IDs are untrustworthy, why make them required?

5.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.       thank you UUID… python-stix shows how easy it is to implement this

6.       Let’s namespace this ID field and hope for the best

 

Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.

 

On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.

 

Thanks,

 

Alex

 

From: Michael P. Stone [[hidden email]]
Sent: Tuesday, November 25, 2014 9:19 AM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)

 

From: Terry MacDonald [[hidden email]]
Sent: Monday, November 24, 2014 6:47 PM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

 

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?


Cheers
Terry MacDonald

 

On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,

 

Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 

In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

 

That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

 

Hope that helps!

John

 

From: Eric Bleher [mailto:[hidden email]]
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices

 

Hello,
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX.
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/.

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!

Regards,
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 


This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.
Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Jason Keirstead

I struggle with the concept of using a UUID as a mandatory ID for content.

If UUID is used, then content A and content B created by two different organizations will have two different iDs for the same identical content. As such it seems to negate all the benefits of having distinguishable IDs to me because you won't be able to use them for searching or for identifying if content already exists, and they won't help with having duplicate content everywhere.

I would think that an ID that was algorithmically defined by the unique content inside the document would be much more useful. But you would have to come up with a standard algorithm for implementers to follow to create and validate these IDs.

The main thing I think is that if it is decided that ID should be mandatory, there should be a specification for the ID and I don't think UUID will cut it, it doesn't solve the use cases for a unique derivable identifier that spans organizations.
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Inactive hide details for "Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make i"Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make it required but it’s not going to save the world.

From: "Chernin, Michael A." <[hidden email]>
To: <[hidden email]>
Date: 2014/11/25 12:14 PM
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices





Good morning Mr Foley!
 
    1.       I’m down to make it required but it’s not going to save the world.
      a.      No single STIX change is going to make a huge impact. However, the combination of small changes proposed to STIX will make a large impact.
    2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
      a.      I am all for making ID mandatory. I do not see any consumer demand for ID optional either.
    3.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”
      a.      We should standardize on what a GUID should look like. Agree with the UUID approach.
 
I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
               
                I will nudge Ben and see if I can get him to respond to this thread. We don’t store the STIX as XML, we use the Mitre STIX API to_dict/from_dict to convert the XML into JSON for storage. Dealing with JSON also happens to be python and developer friendly. However, we had to make decisions on what we wanted to support. Supporting reading native STIX JSON made more sense to us than supporting a system that mapped out all the native STIX elements and attributes into a columnized relational database. We tried this in the past and I didn’t feel like we were in a good place. I know this doesn’t fix your joining challenges, but that is what is Edge is for J Have you tore apart how we lay out the data in Mongo?
 
Aharon
 
From: Foley, Alexander [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:23 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 
    1.       We need them to build complex STIX packages so things can reference each other
      a.       thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier
    2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
      a.       If we make them mandatory, we get a phenomenal primary key
    3.       However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX
      a.       It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs
    4.       So, IDs need to be untrustworthy between distributed systems
      a.       If our IDs are untrustworthy, why make them required?
    5.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique” 6.       Let’s namespace this ID field and hope for the best
 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.
 
On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
 
Thanks,
 
Alex
 
From: Michael P. Stone [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:19 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)
 
From: Terry MacDonald [[hidden email]]
Sent:
 Monday, November 24, 2014 6:47 PM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Hi John/Eric,
This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.
I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.
 
Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald

 
On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:
    Hi Eric,
     
    Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):
     
    “As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”
     
    In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.
     
    That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.
     
    Hope that helps!
    John
     
    From: Eric Bleher [mailto:[hidden email]]
    Sent:
     Monday, November 24, 2014 11:05 AM
    To:
     Wunder, John A.
    Cc:
     stix-discussion-list Structured Threat Information Expression/ST
    Subject:
     Mitre examples not matching all Mitre suggested practices
     
    Hello, 
    As probably many, I’ve used the samples available at
    http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
    Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
     
    However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices
    http://stixproject.github.io/documentation/suggested-practices/. 

    @MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!
     

    Regards,
     
    ---
    Eric Bleher
    Cyber Security - Malware Response & Research
    DeutscheBank AG, Germany
     

    --

    Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

    Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Jason Keirstead
In reply to this post by Wunder, John A.

Another thought - one way you could look at it is the ID should be defined by the system providing the content, not the system sharing the content.

IE, the TAXII service or whatever threat repository is providing the STIX document to you should be assigning IDs to it, and those IDs should only be considered to be unique *within that repository*.  

For example, it should be up to the Soltra system to put IDs on the content, not the system who puts the content into Soltra, because only Soltra knows if that content already exists and can be resolved into other existing relationships.. the person sharing the content doesn't know this and can't figure it out until it gets into the system.

This would kind of solve all cases and removes the burden of creating IDs from the people who make content.


-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


__________________

I struggle with the concept of using a UUID as a mandatory ID for content.

If UUID is used, then content A and content B created by two different organizations will have two different iDs for the same identical content. As such it seems to negate all the benefits of having distinguishable IDs to me because you won't be able to use them for searching or for identifying if content already exists, and they won't help with having duplicate content everywhere.

I would think that an ID that was algorithmically defined by the unique content inside the document would be much more useful. But you would have to come up with a standard algorithm for implementers to follow to create and validate these IDs.

The main thing I think is that if it is decided that ID should be mandatory, there should be a specification for the ID and I don't think UUID will cut it, it doesn't solve the use cases for a unique derivable identifier that spans organizations.
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Inactive hide details for "Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make i"Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make it required but it’s not going to save the world.

From: "Chernin, Michael A." <[hidden email]>
To: <[hidden email]>
Date: 2014/11/25 12:14 PM
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices





Good morning Mr Foley!
 
    1.       I’m down to make it required but it’s not going to save the world.
      a.      No single STIX change is going to make a huge impact. However, the combination of small changes proposed to STIX will make a large impact.
    2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
      a.      I am all for making ID mandatory. I do not see any consumer demand for ID optional either.
    3.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”
      a.      We should standardize on what a GUID should look like. Agree with the UUID approach.
 
I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
               
                I will nudge Ben and see if I can get him to respond to this thread. We don’t store the STIX as XML, we use the Mitre STIX API to_dict/from_dict to convert the XML into JSON for storage. Dealing with JSON also happens to be python and developer friendly. However, we had to make decisions on what we wanted to support. Supporting reading native STIX JSON made more sense to us than supporting a system that mapped out all the native STIX elements and attributes into a columnized relational database. We tried this in the past and I didn’t feel like we were in a good place. I know this doesn’t fix your joining challenges, but that is what is Edge is for J Have you tore apart how we lay out the data in Mongo?
 
Aharon
 
From: Foley, Alexander [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:23 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 
    1.       We need them to build complex STIX packages so things can reference each other
      a.       thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier
    2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
      a.       If we make them mandatory, we get a phenomenal primary key
    3.       However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX
      a.       It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs
    4.       So, IDs need to be untrustworthy between distributed systems
      a.       If our IDs are untrustworthy, why make them required?
    5.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique” 6.       Let’s namespace this ID field and hope for the best
 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.
 
On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
 
Thanks,
 
Alex
 
From: Michael P. Stone [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:19 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)
 
From: Terry MacDonald [[hidden email]]
Sent:
 Monday, November 24, 2014 6:47 PM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Hi John/Eric,
This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.
I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.
 
Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald

 
On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:
    Hi Eric,
     
    Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):
     
    “As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”
     
    In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.
     
    That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.
     
    Hope that helps!
    John
     
    From: Eric Bleher [mailto:[hidden email]]
    Sent:
     Monday, November 24, 2014 11:05 AM
    To:
     Wunder, John A.
    Cc:
     stix-discussion-list Structured Threat Information Expression/ST
    Subject:
     Mitre examples not matching all Mitre suggested practices
     
    Hello, 
    As probably many, I’ve used the samples available at
    http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
    Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
     
    However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices
    http://stixproject.github.io/documentation/suggested-practices/. 

    @MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!
     

    Regards,
     
    ---
    Eric Bleher
    Cyber Security - Malware Response & Research
    DeutscheBank AG, Germany
     

    --

    Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

    Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Wunder, John A.
In reply to this post by Jason Keirstead

I’m going to throw a scenario out and see what happens (sorry, it’s what I do).

 

Org A discovers a piece of malware and doesn’t know a ton about it. They characterize it via a TTP w/ a name based on the filename.

Org B discovers the same piece of malware and doesn’t know a ton about it. They characterize it via a TTP w/ a name based on the filename.

 

Are those pieces of content identical? Ignoring practical concerns about whether it’s possible to get them assigned the same ID, is it ideal that they be assigned the same ID? Or, because they were created by two different organizations, should they be assigned different IDs?

 

Is the answer to that question different if the content is an observable rather than a TTP?

 

I guess my point is to try to clarify whether it’s always the case that content that appears identical should be “merged”. Or in some cases does it make sense to allow producers to version their own content and evolve it over time without consumers merging it with other stuff because it looks identical at that point in time?

 

Re-reading that it sounds like it was a leading question, but I really didn’t mean it to be. I think a case can be made for either approach: consumers do with content what they will (including merging it) and track incoming content per their own processes or consumers adhere to whatever versioning/IDing approach that producers give them and add their own analysis on top of that. It seems to me though that the approach to IDs is affected by what we think the more common consumption model is going to be.

 

John

 

From: Jason Keirstead [mailto:[hidden email]]
Sent: Tuesday, November 25, 2014 11:58 AM
To: stix-discussion-list Structured Threat Information Expression/ST
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

I struggle with the concept of using a UUID as a mandatory ID for content.

If UUID is used, then content A and content B created by two different organizations will have two different iDs for the same identical content. As such it seems to negate all the benefits of having distinguishable IDs to me because you won't be able to use them for searching or for identifying if content already exists, and they won't help with having duplicate content everywhere.

I would think that an ID that was algorithmically defined by the unique content inside the document would be much more useful. But you would have to come up with a standard algorithm for implementers to follow to create and validate these IDs.

The main thing I think is that if it is decided that ID should be mandatory, there should be a specification for the ID and I don't think UUID will cut it, it doesn't solve the use cases for a unique derivable identifier that spans organizations.
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Inactive hide details for "Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make i"Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make it required but it’s not going to save the world.

From: "Chernin, Michael A." <[hidden email]>
To: <[hidden email]>
Date: 2014/11/25 12:14 PM
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices





Good morning Mr Foley!
 

1.       I’m down to make it required but it’s not going to save the world.

a.      No single STIX change is going to make a huge impact. However, the combination of small changes proposed to STIX will make a large impact.

2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.      I am all for making ID mandatory. I do not see any consumer demand for ID optional either.

3.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.      We should standardize on what a GUID should look like. Agree with the UUID approach.

 
I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
               
                I will nudge Ben and see if I can get him to respond to this thread. We don’t store the STIX as XML, we use the Mitre STIX API to_dict/from_dict to convert the XML into JSON for storage. Dealing with JSON also happens to be python and developer friendly. However, we had to make decisions on what we wanted to support. Supporting reading native STIX JSON made more sense to us than supporting a system that mapped out all the native STIX elements and attributes into a columnized relational database. We tried this in the past and I didn’t feel like we were in a good place. I know this doesn’t fix your joining challenges, but that is what is Edge is for J Have you tore apart how we lay out the data in Mongo?
 
Aharon
 
From: Foley, Alexander [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:23 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

 
+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 

1.       We need them to build complex STIX packages so things can reference each other

a.       thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier

2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.       If we make them mandatory, we get a phenomenal primary key

3.       However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX

a.       It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs

4.       So, IDs need to be untrustworthy between distributed systems

a.       If our IDs are untrustworthy, why make them required?

5.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.       thank you UUID… python-stix shows how easy it is to implement this

6.       Let’s namespace this ID field and hope for the best

 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.
 
On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
 
Thanks,
 
Alex
 
From: Michael P. Stone [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:19 AM
To:
 
[hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)
 
From: Terry MacDonald [[hidden email]]
Sent:
 Monday, November 24, 2014 6:47 PM
To:
 
[hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Hi John/Eric,
This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.
I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.
 
Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald
 
On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,
 
Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):
 
“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”
 
In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.
 
That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.
 
Hope that helps!
John
 
From: Eric Bleher [mailto:[hidden email]]
Sent:
 Monday, November 24, 2014 11:05 AM
To:
 Wunder, John A.
Cc:
 stix-discussion-list Structured Threat Information Expression/ST
Subject:
 Mitre examples not matching all Mitre suggested practices

 
Hello, 
As probably many, I’ve used the samples available at
http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
 
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices
http://stixproject.github.io/documentation/suggested-practices/. 

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!
 

Regards,
 
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany
 

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 


This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] EXTERNAL: Re: [STIX] Mitre examples not matching all Mitre suggested practices

Wunder, John A.
In reply to this post by Otto Fowler

Seems like a good topic to me, this kind of conversation is always better over the phone than via 100 e-mails. I’ll go ahead and add it to the agenda.

 

In preparation I agree that it would be nice to float some ideas, problems, and use cases ahead of time.

 

John

 

From: Otto Fowler [mailto:[hidden email]]
Sent: Tuesday, November 25, 2014 10:32 AM
To: stix-discussion-list Structured Threat Information Expression/ST
Subject: Re: [STIX] EXTERNAL: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Are we on track to have the duplication discussion for the next community meeting?  Maybe we can float some ideas or problems ahead of time?

 

 

Otto Fowler

Principal Software Engineer

Industrial Defender Solutions
Lockheed Martin Corporation
 
16 Chestnut Street, Suite 300, Foxborough, MA 02035
O  508-718-6726  | E  [hidden email]

 

On Nov 25, 2014, at 10:11 AM, Eric Burger <[hidden email]> wrote:

 

I would offer that when people ask me to compare STIX versus IODEF, one of the strongest concepts in favor of STIX is id/idref. Replicating the same set of data elements over and over again is a recipe for disaster. It is much better to have a single reference and refer to the reference.

 

I would also offer that mandating id/idref, as Alex points out, will not cure world hunger or incapacitate all of the hackers.

 

My suggestion: strongly, strongly hint that id/idref really, really should be present. Alex’s list of how to make it workable should be in the How To guide for implementors.

 

Can anyone come up with a reason not to make it mandatory? The GitHub arguments seem to focus on the difficulty of specifying it in XML Schema, not whether or not it should be mandatory.

 

Remember, every ‘thing’ that can be made mandatory in CybOX / STIX reduces the complexity of implementation, more especially for servers, which have to deal with seeing things / not seeing things. It’s a lot easier to always expect something and simply reject the message if it’s not there than to try to guess what the client meant if it is not there.

 

On Nov 25, 2014, at 9:23 AM, Foley, Alexander <[hidden email]> wrote:

 

+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.

 

1.      We need them to build complex STIX packages so things can reference each other

a.      thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier

2.      Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.      If we make them mandatory, we get a phenomenal primary key

3.      However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX

a.      It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs

4.      So, IDs need to be untrustworthy between distributed systems

a.      If our IDs are untrustworthy, why make them required?

5.      No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.      thank you UUID… python-stix shows how easy it is to implement this

6.      Let’s namespace this ID field and hope for the best

 

Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.

 

On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.

 

Thanks,

 

Alex

 

From: Michael P. Stone [[hidden email]] 
Sent: Tuesday, November 25, 2014 9:19 AM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)

 

From: Terry MacDonald [[hidden email]] 
Sent: Monday, November 24, 2014 6:47 PM
To: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Hi John/Eric,

This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.

I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

 

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?


Cheers
Terry MacDonald

 

On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,

 

Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

 

“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

 

In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

 

That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

 

Hope that helps!

John

 

From: Eric Bleher [mailto:[hidden email]] 
Sent: Monday, November 24, 2014 11:05 AM
To: Wunder, John A.
Cc: stix-discussion-list Structured Threat Information Expression/ST
Subject: Mitre examples not matching all Mitre suggested practices 

 

Hello, 
As probably many, I’ve used the samples available at http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards. 
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices http://stixproject.github.io/documentation/suggested-practices/. 

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks! 

Regards, 
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany

-- 

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 


This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Jason Keirstead
In reply to this post by Wunder, John A.

I would think that you would want to merge it under one piece of content with multiple references to organizations who saw it.

Regardless of if this is how it is presented to the user, any *system* consuming content will rely on eliminating duplicates for optimization purposes.

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Inactive hide details for "Wunder, John A." ---2014/11/25 01:17:33 PM---I'm going to throw a scenario out and see what happens "Wunder, John A." ---2014/11/25 01:17:33 PM---I'm going to throw a scenario out and see what happens (sorry, it's what I do). Org A discovers a pi

From: "Wunder, John A." <[hidden email]>
To: Jason Keirstead/CanEast/IBM@IBMCA, "stix-discussion-list Structured Threat Information Expression/ST" <[hidden email]>
Date: 2014/11/25 01:17 PM
Subject: RE: [STIX] Mitre examples not matching all Mitre suggested practices





I’m going to throw a scenario out and see what happens (sorry, it’s what I do).
 
Org A discovers a piece of malware and doesn’t know a ton about it. They characterize it via a TTP w/ a name based on the filename.
Org B discovers the same piece of malware and doesn’t know a ton about it. They characterize it via a TTP w/ a name based on the filename.
 
Are those pieces of content identical? Ignoring practical concerns about whether it’s possible to get them assigned the same ID, is it ideal that they be assigned the same ID? Or, because they were created by two different organizations, should they be assigned different IDs?
 
Is the answer to that question different if the content is an observable rather than a TTP?
 
I guess my point is to try to clarify whether it’s always the case that content that appears identical should be “merged”. Or in some cases does it make sense to allow producers to version their own content and evolve it over time without consumers merging it with other stuff because it looks identical at that point in time?
 
Re-reading that it sounds like it was a leading question, but I really didn’t mean it to be. I think a case can be made for either approach: consumers do with content what they will (including merging it) and track incoming content per their own processes or consumers adhere to whatever versioning/IDing approach that producers give them and add their own analysis on top of that. It seems to me though that the approach to IDs is affected by what we think the more common consumption model is going to be.
 
John
 
From: Jason Keirstead [[hidden email]]
Sent:
 Tuesday, November 25, 2014 11:58 AM
To:
 stix-discussion-list Structured Threat Information Expression/ST
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 

I struggle with the concept of using a UUID as a mandatory ID for content.

If UUID is used, then content A and content B created by two different organizations will have two different iDs for the same identical content. As such it seems to negate all the benefits of having distinguishable IDs to me because you won't be able to use them for searching or for identifying if content already exists, and they won't help with having duplicate content everywhere.


I would think that an ID that was algorithmically defined by the unique content inside the document would be much more useful. But you would have to come up with a standard algorithm for implementers to follow to create and validate these IDs.


The main thing I think is that if it is decided that ID should be mandatory, there should be a specification for the ID and I don't think UUID will cut it, it doesn't solve the use cases for a unique derivable identifier that spans organizations.
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems

www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown



Inactive hide details for "Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make i"Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make it required but it’s not going to save the world.

From:
"Chernin, Michael A." <[hidden email]>
To:
<[hidden email]>
Date:
2014/11/25 12:14 PM
Subject:
Re: [STIX] Mitre examples not matching all Mitre suggested practices






Good morning Mr Foley!
 
    1.       I’m down to make it required but it’s not going to save the world. 
      a.      No single STIX change is going to make a huge impact. However, the combination of small changes proposed to STIX will make a large impact.
    2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
      a.      I am all for making ID mandatory. I do not see any consumer demand for ID optional either.
    3.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique” 
      a.      We should standardize on what a GUID should look like. Agree with the UUID approach.
 
I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.

               
               I will nudge Ben and see if I can get him to respond to this thread. We don’t store the STIX as XML, we use the Mitre STIX API to_dict/from_dict to convert the XML into JSON for storage. Dealing with JSON also happens to be python and developer friendly. However, we had to make decisions on what we wanted to support. Supporting reading native STIX JSON made more sense to us than supporting a system that mapped out all the native STIX elements and attributes into a columnized relational database. We tried this in the past and I didn’t feel like we were in a good place. I know this doesn’t fix your joining challenges, but that is what is Edge is for
J Have you tore apart how we lay out the data in Mongo?

Aharon

From:
 Foley, Alexander [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:23 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 
    1.       We need them to build complex STIX packages so things can reference each other 
      a.       thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier
    2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
      a.       If we make them mandatory, we get a phenomenal primary key
    3.       However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX 
      a.       It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs
    4.       So, IDs need to be untrustworthy between distributed systems 
      a.       If our IDs are untrustworthy, why make them required?
    5.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”  6.       Let’s namespace this ID field and hope for the best
 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.

On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.

Thanks,

Alex

From:
 Michael P. Stone [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:19 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)

From:
 Terry MacDonald [[hidden email]]
Sent:
 Monday, November 24, 2014 6:47 PM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

Hi John/Eric,
This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.
I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald

On 25 November 2014 at 07:14, Wunder, John A. <
[hidden email]> wrote:
    Hi Eric,

    Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (
    http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

    “As a simple general rule
    specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

    In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

    That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

    Hope that helps!
    John

    From:
     Eric Bleher [mailto:[hidden email]]
    Sent:
     Monday, November 24, 2014 11:05 AM
    To:
     Wunder, John A.
    Cc:
     stix-discussion-list Structured Threat Information Expression/ST
    Subject:
     Mitre examples not matching all Mitre suggested practices

    Hello,
     
    As probably many, I’ve used the samples available at
    http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
    Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
     
    However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices
    http://stixproject.github.io/documentation/suggested-practices/. 

    @MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!
     

    Regards,
     
    ---
    Eric Bleher
    Cyber Security - Malware Response & Research
    DeutscheBank AG, Germany
     

    --

    Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. 

    Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Joep Gommers
I would argue this would entirely depend on the role you have and the plane of abstraction you’re thinking in. As an intelligence analyst, I would see both instances are two observations of different sources where its meta (name, timing, characterization, other associated content, TLP, etc.) are highly relevant. Depending on the context of my analysis and point in time, I would see both as the same, similar or not.

Unless the intent is creating a huge “wikipedia” of “current” knowledge, and not a peer-peer network of current intelligence updates distributed to each other – I think any collapsing is in the eye of the consumer.

Best regards,
Joep
 

From: Jason Keirstead <[hidden email]>
Reply-To: Jason Keirstead <[hidden email]>
Date: Tuesday, November 25, 2014 at 6:19 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

I would think that you would want to merge it under one piece of content with multiple references to organizations who saw it.

Regardless of if this is how it is presented to the user, any *system* consuming content will rely on eliminating duplicates for optimization purposes.

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Inactive hide details for "Wunder, John A." ---2014/11/25 01:17:33 PM---I'm going to throw a scenario out and see what happens "Wunder, John A." ---2014/11/25 01:17:33 PM---I'm going to throw a scenario out and see what happens (sorry, it's what I do). Org A discovers a pi

From: "Wunder, John A." <[hidden email]>
To: Jason Keirstead/CanEast/IBM@IBMCA, "stix-discussion-list Structured Threat Information Expression/ST" <[hidden email]>
Date: 2014/11/25 01:17 PM
Subject: RE: [STIX] Mitre examples not matching all Mitre suggested practices





I’m going to throw a scenario out and see what happens (sorry, it’s what I do).
 
Org A discovers a piece of malware and doesn’t know a ton about it. They characterize it via a TTP w/ a name based on the filename.
Org B discovers the same piece of malware and doesn’t know a ton about it. They characterize it via a TTP w/ a name based on the filename.
 
Are those pieces of content identical? Ignoring practical concerns about whether it’s possible to get them assigned the same ID, is it ideal that they be assigned the same ID? Or, because they were created by two different organizations, should they be assigned different IDs?
 
Is the answer to that question different if the content is an observable rather than a TTP?
 
I guess my point is to try to clarify whether it’s always the case that content that appears identical should be “merged”. Or in some cases does it make sense to allow producers to version their own content and evolve it over time without consumers merging it with other stuff because it looks identical at that point in time?
 
Re-reading that it sounds like it was a leading question, but I really didn’t mean it to be. I think a case can be made for either approach: consumers do with content what they will (including merging it) and track incoming content per their own processes or consumers adhere to whatever versioning/IDing approach that producers give them and add their own analysis on top of that. It seems to me though that the approach to IDs is affected by what we think the more common consumption model is going to be.
 
John
 
From: Jason Keirstead [[hidden email]]
Sent:
 Tuesday, November 25, 2014 11:58 AM
To:
 stix-discussion-list Structured Threat Information Expression/ST
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 

I struggle with the concept of using a UUID as a mandatory ID for content.

If UUID is used, then content A and content B created by two different organizations will have two different iDs for the same identical content. As such it seems to negate all the benefits of having distinguishable IDs to me because you won't be able to use them for searching or for identifying if content already exists, and they won't help with having duplicate content everywhere.


I would think that an ID that was algorithmically defined by the unique content inside the document would be much more useful. But you would have to come up with a standard algorithm for implementers to follow to create and validate these IDs.


The main thing I think is that if it is decided that ID should be mandatory, there should be a specification for the ID and I don't think UUID will cut it, it doesn't solve the use cases for a unique derivable identifier that spans organizations.
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems

www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown



Inactive hide details for "Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make i"Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make it required but it’s not going to save the world.

From:
"Chernin, Michael A." <[hidden email]>
To:
<[hidden email]>
Date:
2014/11/25 12:14 PM
Subject:
Re: [STIX] Mitre examples not matching all Mitre suggested practices






Good morning Mr Foley!
 
    1.       I’m down to make it required but it’s not going to save the world. 
      a.      No single STIX change is going to make a huge impact. However, the combination of small changes proposed to STIX will make a large impact.
    2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
      a.      I am all for making ID mandatory. I do not see any consumer demand for ID optional either.
    3.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique” 
      a.      We should standardize on what a GUID should look like. Agree with the UUID approach.
 
I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.

               
               I will nudge Ben and see if I can get him to respond to this thread. We don’t store the STIX as XML, we use the Mitre STIX API to_dict/from_dict to convert the XML into JSON for storage. Dealing with JSON also happens to be python and developer friendly. However, we had to make decisions on what we wanted to support. Supporting reading native STIX JSON made more sense to us than supporting a system that mapped out all the native STIX elements and attributes into a columnized relational database. We tried this in the past and I didn’t feel like we were in a good place. I know this doesn’t fix your joining challenges, but that is what is Edge is for
J Have you tore apart how we lay out the data in Mongo?

Aharon

From:
 Foley, Alexander [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:23 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 
    1.       We need them to build complex STIX packages so things can reference each other 
      a.       thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier
    2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?
      a.       If we make them mandatory, we get a phenomenal primary key
    3.       However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX 
      a.       It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs
    4.       So, IDs need to be untrustworthy between distributed systems 
      a.       If our IDs are untrustworthy, why make them required?
    5.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”  6.       Let’s namespace this ID field and hope for the best
 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.

On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.

Thanks,

Alex

From:
 Michael P. Stone [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:19 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)

From:
 Terry MacDonald [[hidden email]]
Sent:
 Monday, November 24, 2014 6:47 PM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

Hi John/Eric,
This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.
I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.

Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald

On 25 November 2014 at 07:14, Wunder, John A. <
[hidden email]> wrote:
    Hi Eric,

    Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (
    http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):

    “As a simple general rule
    specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”

    In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.

    That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.

    Hope that helps!
    John

    From:
     Eric Bleher [mailto:[hidden email]]
    Sent:
     Monday, November 24, 2014 11:05 AM
    To:
     Wunder, John A.
    Cc:
     stix-discussion-list Structured Threat Information Expression/ST
    Subject:
     Mitre examples not matching all Mitre suggested practices

    Hello,
     
    As probably many, I’ve used the samples available at
    http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
    Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
     
    However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices
    http://stixproject.github.io/documentation/suggested-practices/. 

    @MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!
     

    Regards,
     
    ---
    Eric Bleher
    Cyber Security - Malware Response & Research
    DeutscheBank AG, Germany
     

    --

    Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. 

    Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.

Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Aharon
In reply to this post by Jason Keirstead

There is a discussion we need to have as a community around detecting duplicates. Duplicate detection is something that we have had to deal with inside our software and outside of the ID. Observables are relatively easy to detect duplicates in, other objects are not so straight forward. Objects like indicators and incidents contain a lot of subjective content. We can’t simply look at the referenced objects to determine a dupe, as two different analysts could submit the same incident (with the same objects) and include an extra comma in a description somewhere that changed the entire context of the incident (Lets’ eat Dan vs Lets eat, Dan). YUM. Lets not forget that unique IDs also make it easier to retain duplicate content provided to you (if you happen to use ID as a primary key).

 

To me it is only important that the IDs remain unique. I don’t necessarily care if the content is unique, we can use our “machines” to get a good idea of whether or not the content is unique. We could even come together as a community and define common dupe check algos that could be included in the Mitre STIX libraries.

 

Aharon

 

From: Jason Keirstead [mailto:[hidden email]]
Sent: Tuesday, November 25, 2014 11:58 AM
To: Chernin, Michael A.
Cc: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

I struggle with the concept of using a UUID as a mandatory ID for content.

If UUID is used, then content A and content B created by two different organizations will have two different iDs for the same identical content. As such it seems to negate all the benefits of having distinguishable IDs to me because you won't be able to use them for searching or for identifying if content already exists, and they won't help with having duplicate content everywhere.

I would think that an ID that was algorithmically defined by the unique content inside the document would be much more useful. But you would have to come up with a standard algorithm for implementers to follow to create and validate these IDs.

The main thing I think is that if it is decided that ID should be mandatory, there should be a specification for the ID and I don't think UUID will cut it, it doesn't solve the use cases for a unique derivable identifier that spans organizations.
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Inactive hide details for "Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make i"Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make it required but it’s not going to save the world.

From: "Chernin, Michael A." <[hidden email]>
To: <[hidden email]>
Date: 2014/11/25 12:14 PM
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices





Good morning Mr Foley!
 

1.       I’m down to make it required but it’s not going to save the world.

a.      No single STIX change is going to make a huge impact. However, the combination of small changes proposed to STIX will make a large impact.

2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.      I am all for making ID mandatory. I do not see any consumer demand for ID optional either.

3.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.      We should standardize on what a GUID should look like. Agree with the UUID approach.

 
I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
               
                I will nudge Ben and see if I can get him to respond to this thread. We don’t store the STIX as XML, we use the Mitre STIX API to_dict/from_dict to convert the XML into JSON for storage. Dealing with JSON also happens to be python and developer friendly. However, we had to make decisions on what we wanted to support. Supporting reading native STIX JSON made more sense to us than supporting a system that mapped out all the native STIX elements and attributes into a columnized relational database. We tried this in the past and I didn’t feel like we were in a good place. I know this doesn’t fix your joining challenges, but that is what is Edge is for J Have you tore apart how we lay out the data in Mongo?
 
Aharon
 
From: Foley, Alexander [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:23 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

 
+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 

1.       We need them to build complex STIX packages so things can reference each other

a.       thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier

2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.       If we make them mandatory, we get a phenomenal primary key

3.       However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX

a.       It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs

4.       So, IDs need to be untrustworthy between distributed systems

a.       If our IDs are untrustworthy, why make them required?

5.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.       thank you UUID… python-stix shows how easy it is to implement this

6.       Let’s namespace this ID field and hope for the best

 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.
 
On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
 
Thanks,
 
Alex
 
From: Michael P. Stone [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:19 AM
To:
 
[hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)
 
From: Terry MacDonald [[hidden email]]
Sent:
 Monday, November 24, 2014 6:47 PM
To:
 
[hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Hi John/Eric,
This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.
I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.
 
Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald
 
On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,
 
Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):
 
“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”
 
In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.
 
That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.
 
Hope that helps!
John
 
From: Eric Bleher [mailto:[hidden email]]
Sent:
 Monday, November 24, 2014 11:05 AM
To:
 Wunder, John A.
Cc:
 stix-discussion-list Structured Threat Information Expression/ST
Subject:
 Mitre examples not matching all Mitre suggested practices

 
Hello, 
As probably many, I’ve used the samples available at
http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
 
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices
http://stixproject.github.io/documentation/suggested-practices/. 

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!
 

Regards,
 
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany
 

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 


This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.
Reply | Threaded
Open this post in threaded view
|

Re: [STIX] Mitre examples not matching all Mitre suggested practices

Aharon
In reply to this post by Jason Keirstead

Jason,

 

First off, thanks for the plug. In Soltra went down almost the same exact path you describe in earlier releases of Avalanche/Edge. I actually prefer having the repository issue new IDs when receiving new content and forcing the local repository to manage all IDs.

 

There are some places where having the repository issue all IDs falls apart (it actually made me sad), this is just a few:

 

·         Fidelity – The repository is making a statement that producer X said statement Y. We are changing the original document the producer sent us if we modify it in anyway (including the ID). We try to modify the documents as little as possible.

·         STIX object reuse – For example, we could reuse objects from the Poison Ivy report. However, if we change the IDs every time we import a report, everyone will have a different ID for the same objects in a report. Making it difficult for me to do reuse.

·         Peer 2 peer TAXII ping pong – Keeping the original ID helps to keep the repositories from TAXII ping ponging the same data in an efficient manner (new term: indicator storm)

 

Hope this helps,

 

Aharon

 

From: Jason Keirstead [mailto:[hidden email]]
Sent: Tuesday, November 25, 2014 12:03 PM
To: Chernin, Michael A.
Cc: [hidden email]
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices

 

Another thought - one way you could look at it is the ID should be defined by the system providing the content, not the system sharing the content.

IE, the TAXII service or whatever threat repository is providing the STIX document to you should be assigning IDs to it, and those IDs should only be considered to be unique *within that repository*.  

For example, it should be up to the Soltra system to put IDs on the content, not the system who puts the content into Soltra, because only Soltra knows if that content already exists and can be resolved into other existing relationships.. the person sharing the content doesn't know this and can't figure it out until it gets into the system.

This would kind of solve all cases and removes the burden of creating IDs from the people who make content.


-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


__________________

I struggle with the concept of using a UUID as a mandatory ID for content.

If UUID is used, then content A and content B created by two different organizations will have two different iDs for the same identical content. As such it seems to negate all the benefits of having distinguishable IDs to me because you won't be able to use them for searching or for identifying if content already exists, and they won't help with having duplicate content everywhere.

I would think that an ID that was algorithmically defined by the unique content inside the document would be much more useful. But you would have to come up with a standard algorithm for implementers to follow to create and validate these IDs.

The main thing I think is that if it is decided that ID should be mandatory, there should be a specification for the ID and I don't think UUID will cut it, it doesn't solve the use cases for a unique derivable identifier that spans organizations.
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Inactive hide details for "Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make i"Chernin, Michael A." ---2014/11/25 12:14:57 PM---Good morning Mr Foley! 1.       I’m down to make it required but it’s not going to save the world.

From: "Chernin, Michael A." <[hidden email]>
To: <[hidden email]>
Date: 2014/11/25 12:14 PM
Subject: Re: [STIX] Mitre examples not matching all Mitre suggested practices





Good morning Mr Foley!
 

1.       I’m down to make it required but it’s not going to save the world.

a.      No single STIX change is going to make a huge impact. However, the combination of small changes proposed to STIX will make a large impact.

2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.      I am all for making ID mandatory. I do not see any consumer demand for ID optional either.

3.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.      We should standardize on what a GUID should look like. Agree with the UUID approach.

 
I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
               
                I will nudge Ben and see if I can get him to respond to this thread. We don’t store the STIX as XML, we use the Mitre STIX API to_dict/from_dict to convert the XML into JSON for storage. Dealing with JSON also happens to be python and developer friendly. However, we had to make decisions on what we wanted to support. Supporting reading native STIX JSON made more sense to us than supporting a system that mapped out all the native STIX elements and attributes into a columnized relational database. We tried this in the past and I didn’t feel like we were in a good place. I know this doesn’t fix your joining challenges, but that is what is Edge is for J Have you tore apart how we lay out the data in Mongo?
 
Aharon
 
From: Foley, Alexander [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:23 AM
To:
 [hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices

 
+1, my guess is we can have quite the debate on object IDs and settle right where we are now… TL/DR; I’m down to make it required but it’s not going to save the world.
 

1.       We need them to build complex STIX packages so things can reference each other

a.       thank you id element in HTML for proving this concept works, without you my JavaScript would be even uglier

2.       Now that we agree an ID of some kind is necessary, do we make it mandatory or optional?

a.       If we make them mandatory, we get a phenomenal primary key

3.       However, we’re not all making sausage in one factory here… many distributed content creators need to make STIX

a.       It’s impractical to host a centralized service to coordinate our distributed systems to create “guaranteed unique” IDs

4.       So, IDs need to be untrustworthy between distributed systems

a.       If our IDs are untrustworthy, why make them required?

5.       No need to give up all hope, we don’t need to end up with foolish IDs like 1, 2, 3, 4 everywhere… let’s just make them long enough and random enough we can ease collision – “practically unique”

a.       thank you UUID… python-stix shows how easy it is to implement this

6.       Let’s namespace this ID field and hope for the best

 
Re: what you should use as a primary key, I fear that if you rely on this ID field at all you may run into situations where you’re writing many times a second into your table with identical version / ID pairs where someone hasn’t implemented a “practically unique” identifier on their end.  I’ve been monkeying around with storing the STIX packages with a hash of their contents as the primary key in our key / value store to try to go with something more unique, but have yet to build out a data model for counting and referencing the hash collisions.
 
On the other hand, I’m still going to be left with the challenge of searching, joining and filtering on multiple parts of these packages no matter how I store them… I’m hoping one of the Soltra guys will rescue me with a defense of why this fact (and others) make it essential to store STIX as STIX and not try to unpack it into a relational database that you’re keying on.
 
Thanks,
 
Alex
 
From: Michael P. Stone [[hidden email]]
Sent:
 Tuesday, November 25, 2014 9:19 AM
To:
 
[hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Depending on the implementation there may not be a particularly sane thing to use as the ID. Is the ID important enough that you’d rather just not have the data? I also think it’s safe to say that there is zero chance that anyone can receive data from multiple organizations and make correlations entirely via IDs, without having to implement logic to compare the values within the objects. (So the problem of finding duplicates is going to have to be solved, regardless of the presence or absence of indicators.)
 
From: Terry MacDonald [[hidden email]]
Sent:
 Monday, November 24, 2014 6:47 PM
To:
 
[hidden email]
Subject:
 Re: [STIX] Mitre examples not matching all Mitre suggested practices
 
Hi John/Eric,
This does bring up an interesting point. Not putting an ID on the encapsulated object effectively prevents it from being referred to by other future objects. For example if we have an Incident with an encapsulated Indicator with an Observable in it, and the Indicator doesn't have an ID, then if we subsequently see another similar sighting and we are unable to refer to the original Indicator to update it. Or maybe we see another incident in another business unit that uses the same indicator. We can't effectively communicate that without duplicating the indicator - causing problems for consumers who now need to identify the similarities between Incidents by diff'ing the two Indicator objects. Not having an ID also seems to effectively prevent versioning of that object, as we don't have any way to refer to that object for versioning purposes.
I would rather see it that all STIX objects MUST have an ID associated with them. It would mean that producers and consumers could use the ID, timestamp and version fields combined together as a 'primary key' for the database for storage (setting timestamp and version to 0 if they don't exist). It would simplify programming as the object ID would always be available. And it would also allow the object to be referred to (and reused) in subsequent objects/relationships.
 
Also - I'm curious what others use as primary key for their STIX data stores? Anyone free to comment?

Cheers
Terry MacDonald
 
On 25 November 2014 at 07:14, Wunder, John A. <[hidden email]> wrote:

Hi Eric,
 
Thanks for the note! In this case I think that example does conform to our suggested practices. From the page that you linked, in the “Assigning IDs” section (http://stixproject.github.io/documentation/suggested-practices/#assigning-ids):
 
“As a simple general rule specifying IDs is not suggested for constructs embedded within other constructs […] where the embedded constructs are really only relevant/valid/important within the context of the enclosing construct. In other words they provide contextual characterization for the enclosing construct but would not be of interest on their own. The upside of this is slightly less complexity of IDs on everything. The downside is that it would not be possible to reference or pivot on the embedded constructs.”
 
In this case, the observable used for the indicator is just that: only used for the indicator pattern. It isn’t referenced anywhere else and does not have any meaningful information used anywhere else. For that reason it isn’t given an ID.
 
That said, note that not all STIX content will always conform to the suggested practices. They’re only suggested, not mandatory, because in some cases it can make sense to not follow them. MITRE’s sample content SHOULD always follow them when appropriate (so along those lines please let us know if you find any other cases of this) but that doesn’t apply to either cases when the suggested practices aren’t appropriate or to producers who for whatever reason do not follow them.
 
Hope that helps!
John
 
From: Eric Bleher [mailto:[hidden email]]
Sent:
 Monday, November 24, 2014 11:05 AM
To:
 Wunder, John A.
Cc:
 stix-discussion-list Structured Threat Information Expression/ST
Subject:
 Mitre examples not matching all Mitre suggested practices

 
Hello, 
As probably many, I’ve used the samples available at
http://stix.mitre.org/language/version1.1.1/samples.html  to grab some ideas about how to export different information into STIX. 
Especially, concerning the Cybox-MAEC-Stix integration, the STIX file flow_example(3of3)-stix_indicator_combined.xml (found on the "all samples" Zip available on that MITRE page) get an interesting view about the complexity of the different standards.
 
However, arguing with that file with an other STIX system (specifically SoltraEdge), it comes out the file does not have an object ID for every object (some do and some do not). This is against MITRE's own suggested practices
http://stixproject.github.io/documentation/suggested-practices/. 

@MITRE team, you might want to review all the samples available on that page, and either rework or remove them. Or at least put a big warning on them! Thanks!
 

Regards,
 
---
Eric Bleher
Cyber Security - Malware Response & Research
DeutscheBank AG, Germany
 

--

Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

 


This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.


DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses.  The company accepts no liability for any damage caused by any virus transmitted by this email.
123