CWE Compatibility Program

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

CWE Compatibility Program

Andrew Buttner
Administrator
CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.  
 
The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program.  

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys.  

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


CWE Compatibility Program.pdf (142K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CWE Compatibility Program

Kurt Seifried
The biggest problem I see is how do I get data into CWE?

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

So my ask would be: 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

2) How to submit data to CWE for inclusion

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 


On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:
CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515



--
Kurt Seifried
[hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: CWE Compatibility Program

Chris Eng

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: CWE Compatibility Program

Kurt Seifried
That's a start but every step is "MITRE does X" with no timeline, any many have no clear direction, e.g.:

Inclusion DecisionMITRE determines what content should be included in CWE. MITRE may decide to (1) create a new entry, (2) integrate the submitted information into an existing entry, and/or (3) create additional supporting entries that facilitate use of CWE's major views (Development, Researcher, and Architecture).

What is the criteria? E.g. from my list (please note it needs more SPLIT/MERGE and cleaning/work, but it gives an idea of stuff I want to add to CWE):


Are there any which would definitely not be included, or definitely would be included? E.g. the Smart Contract stuff that sort of maps to existing CWEs, can they be broken out like the .NET stuff sometimes is, or would the update a CWE, or some other option?



On Thu, Oct 29, 2020 at 11:11 AM Chris Eng <[hidden email]> wrote:

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]



--
Kurt Seifried
[hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: CWE Compatibility Program

Fung, Jason M

Agreed with Kurt that having an SLA and a timeline target for in-flight CWE submissions would encourage more participation.   

 

  • E.g., Status can include Received à Under Review à Release Targeting CWE 4.4 à Released (with URL) 
  • E.g., 2 weeks SLA from Under Review to the next stage

 

Concerning CWE Compatibility Program, the “Compatibility” portion seems to be a mean to the end.  The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.  Today, decision makers have to pick up the burden to assemble their own benchmarks to assess tool effectiveness.  Such evaluations are often limited without serious investments in identifying a robust and comprehensive benchmark that help to reflect the false positive and false negative rates of the tools under tests.  Organizations can benefit from an industry effort, and feedback from the certification process can also help tool vendors to create better products.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 10:19 AM
To: Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

That's a start but every step is "MITRE does X" with no timeline, any many have no clear direction, e.g.:

 

Inclusion Decision

MITRE determines what content should be included in CWE. MITRE may decide to (1) create a new entry, (2) integrate the submitted information into an existing entry, and/or (3) create additional supporting entries that facilitate use of CWE's major views (Development, Researcher, and Architecture).

 

What is the criteria? E.g. from my list (please note it needs more SPLIT/MERGE and cleaning/work, but it gives an idea of stuff I want to add to CWE):

 

 

Are there any which would definitely not be included, or definitely would be included? E.g. the Smart Contract stuff that sort of maps to existing CWEs, can they be broken out like the .NET stuff sometimes is, or would the update a CWE, or some other option?

 

 

 

On Thu, Oct 29, 2020 at 11:11 AM Chris Eng <[hidden email]> wrote:

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]


 

--

Kurt Seifried
[hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: CWE Compatibility Program

Chris Eng

Agree on all of this. It’s one of the reasons a CWE roadmap and release schedule are at the top of my wish list. Assuming that submission entries are sufficiently thorough we would be able to define an SLA, not in terms of a maximum number of weeks/months between each stage but rather something like a feature freeze date (last day for submissions) and a code freeze date (review/approval due) prior to the release date.  With a well-defined release schedule you could also pin larger submission sets such as the blockchain items in Kurt’s list to a specific future release.  If we anticipated a longer review process, we could target it for 2 releases out instead of 1.  Of course you can’t do any of this without a predictable release schedule.  Eventually it would be great if CWE were managed like any other shipping product, with JIRA-style (or equivalent) tracking and visibility. I think it would benefit submitters as well as the rest of us in terms of being able to plan better for CWE updates in our own CWE-compatible products.

 

 

From: Fung, Jason M <[hidden email]>
Sent: Thursday, October 29, 2020 8:53 PM
To: Kurt Seifried <[hidden email]>; Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


Agreed with Kurt that having an SLA and a timeline target for in-flight CWE submissions would encourage more participation.   

 

  • E.g., Status can include Received à Under Review à Release Targeting CWE 4.4 à Released (with URL) 
  • E.g., 2 weeks SLA from Under Review to the next stage

 

Concerning CWE Compatibility Program, the “Compatibility” portion seems to be a mean to the end.  The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.  Today, decision makers have to pick up the burden to assemble their own benchmarks to assess tool effectiveness.  Such evaluations are often limited without serious investments in identifying a robust and comprehensive benchmark that help to reflect the false positive and false negative rates of the tools under tests.  Organizations can benefit from an industry effort, and feedback from the certification process can also help tool vendors to create better products.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 10:19 AM
To: Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

That's a start but every step is "MITRE does X" with no timeline, any many have no clear direction, e.g.:

 

Inclusion Decision

MITRE determines what content should be included in CWE. MITRE may decide to (1) create a new entry, (2) integrate the submitted information into an existing entry, and/or (3) create additional supporting entries that facilitate use of CWE's major views (Development, Researcher, and Architecture).

 

What is the criteria? E.g. from my list (please note it needs more SPLIT/MERGE and cleaning/work, but it gives an idea of stuff I want to add to CWE):

 

 

Are there any which would definitely not be included, or definitely would be included? E.g. the Smart Contract stuff that sort of maps to existing CWEs, can they be broken out like the .NET stuff sometimes is, or would the update a CWE, or some other option?

 

 

 

On Thu, Oct 29, 2020 at 11:11 AM Chris Eng <[hidden email]> wrote:

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]


 

--

Kurt Seifried
[hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [External] - CWE Compatibility Program

Paul Anderson
In reply to this post by Andrew Buttner
Drew:

It's a little hard to judge whether this will be effective without seeing the cover letter. I was expecting a "call to action" paragraph in the document, but if that's in the cover letter, that's probably fine.

-Paul

--
Paul Anderson, VP of Engineering, GrammaTech, Inc.
531 Esty St., Ithaca, NY 14850
Tel: +1 607 273-7340 x118; https://www.grammatech.com

-----Original Message-----
From: Drew Buttner <[hidden email]>
Sent: Thursday, October 29, 2020 7:08 AM
To: CWE CAPEC Board <[hidden email]>
Subject: [External] - CWE Compatibility Program

CAUTION: External Email


CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program.

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys.

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515

________________________________
The information contained in this e-mail and any attachments from GrammaTech, Inc may contain confidential and/or proprietary information, and is intended only for the named recipient to whom it was originally addressed. If you are not the intended recipient, any disclosure, distribution, or copying of this e-mail or its attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by return e-mail and permanently delete the e-mail and any attachments.
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: CWE Compatibility Program

Jason Oberg
In reply to this post by Chris Eng
The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.

Completely agree with Jason F. here. I think what is really needed (both in hardware and software tools) is transparency about what effective really means. Coverage of CWEs in lists like Top 25 provide a good starting basis for measuring effectiveness but I think transparency to the user about effectiveness as it relates to risk reduction is really important. This can be done by providing clear guidance on what CWEs should be covered to reduce risk for a given application. Otherwise, as Jason said, it's up to each organization/user to determine whether a tool is effective based on their own internal metrics. Unfortunately only the biggest organizations have the resources to make that type of judgement call. 

On Fri, Oct 30, 2020 at 5:01 AM Chris Eng <[hidden email]> wrote:

Agree on all of this. It’s one of the reasons a CWE roadmap and release schedule are at the top of my wish list. Assuming that submission entries are sufficiently thorough we would be able to define an SLA, not in terms of a maximum number of weeks/months between each stage but rather something like a feature freeze date (last day for submissions) and a code freeze date (review/approval due) prior to the release date.  With a well-defined release schedule you could also pin larger submission sets such as the blockchain items in Kurt’s list to a specific future release.  If we anticipated a longer review process, we could target it for 2 releases out instead of 1.  Of course you can’t do any of this without a predictable release schedule.  Eventually it would be great if CWE were managed like any other shipping product, with JIRA-style (or equivalent) tracking and visibility. I think it would benefit submitters as well as the rest of us in terms of being able to plan better for CWE updates in our own CWE-compatible products.

 

 

From: Fung, Jason M <[hidden email]>
Sent: Thursday, October 29, 2020 8:53 PM
To: Kurt Seifried <[hidden email]>; Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


Agreed with Kurt that having an SLA and a timeline target for in-flight CWE submissions would encourage more participation.   

 

  • E.g., Status can include Received à Under Review à Release Targeting CWE 4.4 à Released (with URL) 
  • E.g., 2 weeks SLA from Under Review to the next stage

 

Concerning CWE Compatibility Program, the “Compatibility” portion seems to be a mean to the end.  The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.  Today, decision makers have to pick up the burden to assemble their own benchmarks to assess tool effectiveness.  Such evaluations are often limited without serious investments in identifying a robust and comprehensive benchmark that help to reflect the false positive and false negative rates of the tools under tests.  Organizations can benefit from an industry effort, and feedback from the certification process can also help tool vendors to create better products.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 10:19 AM
To: Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

That's a start but every step is "MITRE does X" with no timeline, any many have no clear direction, e.g.:

 

Inclusion Decision

MITRE determines what content should be included in CWE. MITRE may decide to (1) create a new entry, (2) integrate the submitted information into an existing entry, and/or (3) create additional supporting entries that facilitate use of CWE's major views (Development, Researcher, and Architecture).

 

What is the criteria? E.g. from my list (please note it needs more SPLIT/MERGE and cleaning/work, but it gives an idea of stuff I want to add to CWE):

 

 

Are there any which would definitely not be included, or definitely would be included? E.g. the Smart Contract stuff that sort of maps to existing CWEs, can they be broken out like the .NET stuff sometimes is, or would the update a CWE, or some other option?

 

 

 

On Thu, Oct 29, 2020 at 11:11 AM Chris Eng <[hidden email]> wrote:

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]


 

--

Kurt Seifried
[hidden email]



--

Tortuga Logic

Dr. Jason Oberg | Co-Founder and CTO | +1 (808) 635-7604

Tortuga Logic  |  75 E Santa Clara Street, San Jose, CA 95113


NOTICE TO RECIPIENT | This email and any attachments may contain private, confidential and privileged material for the sole use of the intended recipient. If you are not the intended recipient, please immediately notify the sender of the error by return email and delete this email and any attachments.

Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: CWE Compatibility Program

Kurt Seifried
This is also target specific, e.g. the top 25 CWE list for Web vs IOT Hardware vs Blockchai/DLT vs Cloud...

One thing that helps here is increasing usage of ATT&CK which includes real world attack data (and thus which classes of CWE's are being exploited). It's like a rainforest I guess, everything is touching everything else. 

On Mon, Nov 2, 2020 at 11:36 AM Jason Oberg <[hidden email]> wrote:
The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.

Completely agree with Jason F. here. I think what is really needed (both in hardware and software tools) is transparency about what effective really means. Coverage of CWEs in lists like Top 25 provide a good starting basis for measuring effectiveness but I think transparency to the user about effectiveness as it relates to risk reduction is really important. This can be done by providing clear guidance on what CWEs should be covered to reduce risk for a given application. Otherwise, as Jason said, it's up to each organization/user to determine whether a tool is effective based on their own internal metrics. Unfortunately only the biggest organizations have the resources to make that type of judgement call. 

On Fri, Oct 30, 2020 at 5:01 AM Chris Eng <[hidden email]> wrote:

Agree on all of this. It’s one of the reasons a CWE roadmap and release schedule are at the top of my wish list. Assuming that submission entries are sufficiently thorough we would be able to define an SLA, not in terms of a maximum number of weeks/months between each stage but rather something like a feature freeze date (last day for submissions) and a code freeze date (review/approval due) prior to the release date.  With a well-defined release schedule you could also pin larger submission sets such as the blockchain items in Kurt’s list to a specific future release.  If we anticipated a longer review process, we could target it for 2 releases out instead of 1.  Of course you can’t do any of this without a predictable release schedule.  Eventually it would be great if CWE were managed like any other shipping product, with JIRA-style (or equivalent) tracking and visibility. I think it would benefit submitters as well as the rest of us in terms of being able to plan better for CWE updates in our own CWE-compatible products.

 

 

From: Fung, Jason M <[hidden email]>
Sent: Thursday, October 29, 2020 8:53 PM
To: Kurt Seifried <[hidden email]>; Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


Agreed with Kurt that having an SLA and a timeline target for in-flight CWE submissions would encourage more participation.   

 

  • E.g., Status can include Received à Under Review à Release Targeting CWE 4.4 à Released (with URL) 
  • E.g., 2 weeks SLA from Under Review to the next stage

 

Concerning CWE Compatibility Program, the “Compatibility” portion seems to be a mean to the end.  The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.  Today, decision makers have to pick up the burden to assemble their own benchmarks to assess tool effectiveness.  Such evaluations are often limited without serious investments in identifying a robust and comprehensive benchmark that help to reflect the false positive and false negative rates of the tools under tests.  Organizations can benefit from an industry effort, and feedback from the certification process can also help tool vendors to create better products.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 10:19 AM
To: Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

That's a start but every step is "MITRE does X" with no timeline, any many have no clear direction, e.g.:

 

Inclusion Decision

MITRE determines what content should be included in CWE. MITRE may decide to (1) create a new entry, (2) integrate the submitted information into an existing entry, and/or (3) create additional supporting entries that facilitate use of CWE's major views (Development, Researcher, and Architecture).

 

What is the criteria? E.g. from my list (please note it needs more SPLIT/MERGE and cleaning/work, but it gives an idea of stuff I want to add to CWE):

 

 

Are there any which would definitely not be included, or definitely would be included? E.g. the Smart Contract stuff that sort of maps to existing CWEs, can they be broken out like the .NET stuff sometimes is, or would the update a CWE, or some other option?

 

 

 

On Thu, Oct 29, 2020 at 11:11 AM Chris Eng <[hidden email]> wrote:

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]


 

--

Kurt Seifried
[hidden email]



--

Tortuga Logic

Dr. Jason Oberg | Co-Founder and CTO | +1 (808) 635-7604

Tortuga Logic  |  75 E Santa Clara Street, San Jose, CA 95113


NOTICE TO RECIPIENT | This email and any attachments may contain private, confidential and privileged material for the sole use of the intended recipient. If you are not the intended recipient, please immediately notify the sender of the error by return email and delete this email and any attachments.



--
Kurt Seifried
[hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: CWE Compatibility Program

Fung, Jason M

The ideal scenario would be the availability of a comprehensive benchmark that covers virtually all CWEs.  Tool vendors going through the certification process will suggest the subset of CWEs their tools can reliably detect, and the certification process will assess and report how many the tool actually finds/misses, along with other useful data such as false +ve rates, running time, etc.

 

To help it scale, and to be unbiased, the benchmark can be a community-driven effort with contributions coming from researchers, product developers, tool vendors, etc.  The better the benchmark reflects real world code complexity, the better it helps to measure capability effectiveness of tools.  There are already a lot of vulnerable code and fixes available (to be tagged) from open source databases.  I expect it is an initiative to be well received by the academic research community as they encounter the same challenge to showcase their algorithms on real world benchmarks.  It is also something tool vendors may want to contribute, for example, to put out tough examples that their tools (but not their competitors) can detect.

 

There are already some corpus in place that people can build upon.  For SW CWEs, DAPRA launched their Grand Cybersecurity Challenge to share a pretty good set of tagged vulnerabilities.  For HW CWEs, we attempt to demonstrate common HW vulnerabilities in an open SoC via our Hack@DAC and Hack@Sec HW CTF effort.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Monday, November 2, 2020 10:42 AM
To: Jason Oberg <[hidden email]>
Cc: Chris Eng <[hidden email]>; Fung, Jason M <[hidden email]>; Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

This is also target specific, e.g. the top 25 CWE list for Web vs IOT Hardware vs Blockchai/DLT vs Cloud...

 

One thing that helps here is increasing usage of ATT&CK which includes real world attack data (and thus which classes of CWE's are being exploited). It's like a rainforest I guess, everything is touching everything else. 

 

On Mon, Nov 2, 2020 at 11:36 AM Jason Oberg <[hidden email]> wrote:

The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.

 

Completely agree with Jason F. here. I think what is really needed (both in hardware and software tools) is transparency about what effective really means. Coverage of CWEs in lists like Top 25 provide a good starting basis for measuring effectiveness but I think transparency to the user about effectiveness as it relates to risk reduction is really important. This can be done by providing clear guidance on what CWEs should be covered to reduce risk for a given application. Otherwise, as Jason said, it's up to each organization/user to determine whether a tool is effective based on their own internal metrics. Unfortunately only the biggest organizations have the resources to make that type of judgement call. 

 

On Fri, Oct 30, 2020 at 5:01 AM Chris Eng <[hidden email]> wrote:

Agree on all of this. It’s one of the reasons a CWE roadmap and release schedule are at the top of my wish list. Assuming that submission entries are sufficiently thorough we would be able to define an SLA, not in terms of a maximum number of weeks/months between each stage but rather something like a feature freeze date (last day for submissions) and a code freeze date (review/approval due) prior to the release date.  With a well-defined release schedule you could also pin larger submission sets such as the blockchain items in Kurt’s list to a specific future release.  If we anticipated a longer review process, we could target it for 2 releases out instead of 1.  Of course you can’t do any of this without a predictable release schedule.  Eventually it would be great if CWE were managed like any other shipping product, with JIRA-style (or equivalent) tracking and visibility. I think it would benefit submitters as well as the rest of us in terms of being able to plan better for CWE updates in our own CWE-compatible products.

 

 

From: Fung, Jason M <[hidden email]>
Sent: Thursday, October 29, 2020 8:53 PM
To: Kurt Seifried <[hidden email]>; Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


Agreed with Kurt that having an SLA and a timeline target for in-flight CWE submissions would encourage more participation.   

 

  • E.g., Status can include Received à Under Review à Release Targeting CWE 4.4 à Released (with URL) 
  • E.g., 2 weeks SLA from Under Review to the next stage

 

Concerning CWE Compatibility Program, the “Compatibility” portion seems to be a mean to the end.  The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.  Today, decision makers have to pick up the burden to assemble their own benchmarks to assess tool effectiveness.  Such evaluations are often limited without serious investments in identifying a robust and comprehensive benchmark that help to reflect the false positive and false negative rates of the tools under tests.  Organizations can benefit from an industry effort, and feedback from the certification process can also help tool vendors to create better products.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 10:19 AM
To: Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

That's a start but every step is "MITRE does X" with no timeline, any many have no clear direction, e.g.:

 

Inclusion Decision

MITRE determines what content should be included in CWE. MITRE may decide to (1) create a new entry, (2) integrate the submitted information into an existing entry, and/or (3) create additional supporting entries that facilitate use of CWE's major views (Development, Researcher, and Architecture).

 

What is the criteria? E.g. from my list (please note it needs more SPLIT/MERGE and cleaning/work, but it gives an idea of stuff I want to add to CWE):

 

 

Are there any which would definitely not be included, or definitely would be included? E.g. the Smart Contract stuff that sort of maps to existing CWEs, can they be broken out like the .NET stuff sometimes is, or would the update a CWE, or some other option?

 

 

 

On Thu, Oct 29, 2020 at 11:11 AM Chris Eng <[hidden email]> wrote:

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]


 

--

Kurt Seifried
[hidden email]


 

--

Dr. Jason Oberg | Co-Founder and CTO | +1 (808) 635-7604

Tortuga Logic  |  75 E Santa Clara Street, San Jose, CA 95113

 

NOTICE TO RECIPIENT | This email and any attachments may contain private, confidential and privileged material for the sole use of the intended recipient. If you are not the intended recipient, please immediately notify the sender of the error by return email and delete this email and any attachments.


 

--

Kurt Seifried
[hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: CWE Compatibility Program

Andrew Buttner
Administrator

Board,

 

Really great conversation.  Please keep it coming.  I will work to unpack this a bit more as we lead up to the next board meeting.  I will be leveraging the board to set the agenda, and will have these topics as some of the options.  So far, I think I am hearing the following:

 

  1. Desire for a service level agreement / timelines / better defined “acceptance procedures” relate to content submissions.

 

  1. Taking a crack at solving the "effectiveness" piece related to compatibility, including creating a useful / effective benchmark to measure tools against.

 

  1. CWE/CAPEC release roadmap and defined schedule.

 

Please keep this discussion going, and I will work on coordinating an agenda with this board for the upcoming meeting.

 

Thanks

Drew

 

 

From: Fung, Jason M <[hidden email]>
Sent: Monday, November 2, 2020 2:44 PM
To: Seifried, Kurt <[hidden email]>; Oberg, Jason <[hidden email]>
Cc: Eng, Chris <[hidden email]>; Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

The ideal scenario would be the availability of a comprehensive benchmark that covers virtually all CWEs.  Tool vendors going through the certification process will suggest the subset of CWEs their tools can reliably detect, and the certification process will assess and report how many the tool actually finds/misses, along with other useful data such as false +ve rates, running time, etc.

 

To help it scale, and to be unbiased, the benchmark can be a community-driven effort with contributions coming from researchers, product developers, tool vendors, etc.  The better the benchmark reflects real world code complexity, the better it helps to measure capability effectiveness of tools.  There are already a lot of vulnerable code and fixes available (to be tagged) from open source databases.  I expect it is an initiative to be well received by the academic research community as they encounter the same challenge to showcase their algorithms on real world benchmarks.  It is also something tool vendors may want to contribute, for example, to put out tough examples that their tools (but not their competitors) can detect.

 

There are already some corpus in place that people can build upon.  For SW CWEs, DAPRA launched their Grand Cybersecurity Challenge to share a pretty good set of tagged vulnerabilities.  For HW CWEs, we attempt to demonstrate common HW vulnerabilities in an open SoC via our Hack@DAC and Hack@Sec HW CTF effort.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Monday, November 2, 2020 10:42 AM
To: Jason Oberg <[hidden email]>
Cc: Chris Eng <[hidden email]>; Fung, Jason M <[hidden email]>; Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

This is also target specific, e.g. the top 25 CWE list for Web vs IOT Hardware vs Blockchai/DLT vs Cloud...

 

One thing that helps here is increasing usage of ATT&CK which includes real world attack data (and thus which classes of CWE's are being exploited). It's like a rainforest I guess, everything is touching everything else. 

 

On Mon, Nov 2, 2020 at 11:36 AM Jason Oberg <[hidden email]> wrote:

The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.

 

Completely agree with Jason F. here. I think what is really needed (both in hardware and software tools) is transparency about what effective really means. Coverage of CWEs in lists like Top 25 provide a good starting basis for measuring effectiveness but I think transparency to the user about effectiveness as it relates to risk reduction is really important. This can be done by providing clear guidance on what CWEs should be covered to reduce risk for a given application. Otherwise, as Jason said, it's up to each organization/user to determine whether a tool is effective based on their own internal metrics. Unfortunately only the biggest organizations have the resources to make that type of judgement call. 

 

On Fri, Oct 30, 2020 at 5:01 AM Chris Eng <[hidden email]> wrote:

Agree on all of this. It’s one of the reasons a CWE roadmap and release schedule are at the top of my wish list. Assuming that submission entries are sufficiently thorough we would be able to define an SLA, not in terms of a maximum number of weeks/months between each stage but rather something like a feature freeze date (last day for submissions) and a code freeze date (review/approval due) prior to the release date.  With a well-defined release schedule you could also pin larger submission sets such as the blockchain items in Kurt’s list to a specific future release.  If we anticipated a longer review process, we could target it for 2 releases out instead of 1.  Of course you can’t do any of this without a predictable release schedule.  Eventually it would be great if CWE were managed like any other shipping product, with JIRA-style (or equivalent) tracking and visibility. I think it would benefit submitters as well as the rest of us in terms of being able to plan better for CWE updates in our own CWE-compatible products.

 

 

From: Fung, Jason M <[hidden email]>
Sent: Thursday, October 29, 2020 8:53 PM
To: Kurt Seifried <[hidden email]>; Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


Agreed with Kurt that having an SLA and a timeline target for in-flight CWE submissions would encourage more participation.   

 

  • E.g., Status can include Received à Under Review à Release Targeting CWE 4.4 à Released (with URL) 
  • E.g., 2 weeks SLA from Under Review to the next stage

 

Concerning CWE Compatibility Program, the “Compatibility” portion seems to be a mean to the end.  The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.  Today, decision makers have to pick up the burden to assemble their own benchmarks to assess tool effectiveness.  Such evaluations are often limited without serious investments in identifying a robust and comprehensive benchmark that help to reflect the false positive and false negative rates of the tools under tests.  Organizations can benefit from an industry effort, and feedback from the certification process can also help tool vendors to create better products.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 10:19 AM
To: Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

That's a start but every step is "MITRE does X" with no timeline, any many have no clear direction, e.g.:

 

Inclusion Decision

MITRE determines what content should be included in CWE. MITRE may decide to (1) create a new entry, (2) integrate the submitted information into an existing entry, and/or (3) create additional supporting entries that facilitate use of CWE's major views (Development, Researcher, and Architecture).

 

What is the criteria? E.g. from my list (please note it needs more SPLIT/MERGE and cleaning/work, but it gives an idea of stuff I want to add to CWE):

 

 

Are there any which would definitely not be included, or definitely would be included? E.g. the Smart Contract stuff that sort of maps to existing CWEs, can they be broken out like the .NET stuff sometimes is, or would the update a CWE, or some other option?

 

 

 

On Thu, Oct 29, 2020 at 11:11 AM Chris Eng <[hidden email]> wrote:

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]


 

--

Kurt Seifried
[hidden email]


 

--

 

Dr. Jason Oberg | Co-Founder and CTO | +1 (808) 635-7604

Tortuga Logic  |  75 E Santa Clara Street, San Jose, CA 95113

 

NOTICE TO RECIPIENT | This email and any attachments may contain private, confidential and privileged material for the sole use of the intended recipient. If you are not the intended recipient, please immediately notify the sender of the error by return email and delete this email and any attachments.


 

--

Kurt Seifried
[hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: CWE Compatibility Program

Fung, Jason M

A topic raised in our last 2 meetings was the opportunity to partner with CVE Board to introduce a HW vs. SW root cause field into CVE, so as to help create HW vulnerability indicators and trends.

 

From: Drew Buttner <[hidden email]>
Sent: Monday, November 2, 2020 4:27 PM
To: CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

Board,

 

Really great conversation.  Please keep it coming.  I will work to unpack this a bit more as we lead up to the next board meeting.  I will be leveraging the board to set the agenda, and will have these topics as some of the options.  So far, I think I am hearing the following:

 

  1. Desire for a service level agreement / timelines / better defined “acceptance procedures” relate to content submissions.

 

  1. Taking a crack at solving the "effectiveness" piece related to compatibility, including creating a useful / effective benchmark to measure tools against.

 

  1. CWE/CAPEC release roadmap and defined schedule.

 

Please keep this discussion going, and I will work on coordinating an agenda with this board for the upcoming meeting.

 

Thanks

Drew

 

 

From: Fung, Jason M <[hidden email]>
Sent: Monday, November 2, 2020 2:44 PM
To: Seifried, Kurt <[hidden email]>; Oberg, Jason <[hidden email]>
Cc: Eng, Chris <[hidden email]>; Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

The ideal scenario would be the availability of a comprehensive benchmark that covers virtually all CWEs.  Tool vendors going through the certification process will suggest the subset of CWEs their tools can reliably detect, and the certification process will assess and report how many the tool actually finds/misses, along with other useful data such as false +ve rates, running time, etc.

 

To help it scale, and to be unbiased, the benchmark can be a community-driven effort with contributions coming from researchers, product developers, tool vendors, etc.  The better the benchmark reflects real world code complexity, the better it helps to measure capability effectiveness of tools.  There are already a lot of vulnerable code and fixes available (to be tagged) from open source databases.  I expect it is an initiative to be well received by the academic research community as they encounter the same challenge to showcase their algorithms on real world benchmarks.  It is also something tool vendors may want to contribute, for example, to put out tough examples that their tools (but not their competitors) can detect.

 

There are already some corpus in place that people can build upon.  For SW CWEs, DAPRA launched their Grand Cybersecurity Challenge to share a pretty good set of tagged vulnerabilities.  For HW CWEs, we attempt to demonstrate common HW vulnerabilities in an open SoC via our Hack@DAC and Hack@Sec HW CTF effort.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Monday, November 2, 2020 10:42 AM
To: Jason Oberg <[hidden email]>
Cc: Chris Eng <[hidden email]>; Fung, Jason M <[hidden email]>; Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

This is also target specific, e.g. the top 25 CWE list for Web vs IOT Hardware vs Blockchai/DLT vs Cloud...

 

One thing that helps here is increasing usage of ATT&CK which includes real world attack data (and thus which classes of CWE's are being exploited). It's like a rainforest I guess, everything is touching everything else. 

 

On Mon, Nov 2, 2020 at 11:36 AM Jason Oberg <[hidden email]> wrote:

The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.

 

Completely agree with Jason F. here. I think what is really needed (both in hardware and software tools) is transparency about what effective really means. Coverage of CWEs in lists like Top 25 provide a good starting basis for measuring effectiveness but I think transparency to the user about effectiveness as it relates to risk reduction is really important. This can be done by providing clear guidance on what CWEs should be covered to reduce risk for a given application. Otherwise, as Jason said, it's up to each organization/user to determine whether a tool is effective based on their own internal metrics. Unfortunately only the biggest organizations have the resources to make that type of judgement call. 

 

On Fri, Oct 30, 2020 at 5:01 AM Chris Eng <[hidden email]> wrote:

Agree on all of this. It’s one of the reasons a CWE roadmap and release schedule are at the top of my wish list. Assuming that submission entries are sufficiently thorough we would be able to define an SLA, not in terms of a maximum number of weeks/months between each stage but rather something like a feature freeze date (last day for submissions) and a code freeze date (review/approval due) prior to the release date.  With a well-defined release schedule you could also pin larger submission sets such as the blockchain items in Kurt’s list to a specific future release.  If we anticipated a longer review process, we could target it for 2 releases out instead of 1.  Of course you can’t do any of this without a predictable release schedule.  Eventually it would be great if CWE were managed like any other shipping product, with JIRA-style (or equivalent) tracking and visibility. I think it would benefit submitters as well as the rest of us in terms of being able to plan better for CWE updates in our own CWE-compatible products.

 

 

From: Fung, Jason M <[hidden email]>
Sent: Thursday, October 29, 2020 8:53 PM
To: Kurt Seifried <[hidden email]>; Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: RE: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


Agreed with Kurt that having an SLA and a timeline target for in-flight CWE submissions would encourage more participation.   

 

  • E.g., Status can include Received à Under Review à Release Targeting CWE 4.4 à Released (with URL) 
  • E.g., 2 weeks SLA from Under Review to the next stage

 

Concerning CWE Compatibility Program, the “Compatibility” portion seems to be a mean to the end.  The real value for organizations to differentiate tools and services in the marketplace can really benefit from a robust definition of the “Effectiveness” piece.  Today, decision makers have to pick up the burden to assemble their own benchmarks to assess tool effectiveness.  Such evaluations are often limited without serious investments in identifying a robust and comprehensive benchmark that help to reflect the false positive and false negative rates of the tools under tests.  Organizations can benefit from an industry effort, and feedback from the certification process can also help tool vendors to create better products.

 

- Jason

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 10:19 AM
To: Chris Eng <[hidden email]>
Cc: Drew Buttner <[hidden email]>; CWE CAPEC Board <[hidden email]>
Subject: Re: [EXTERNAL] Re: CWE Compatibility Program

 

That's a start but every step is "MITRE does X" with no timeline, any many have no clear direction, e.g.:

 

Inclusion Decision

MITRE determines what content should be included in CWE. MITRE may decide to (1) create a new entry, (2) integrate the submitted information into an existing entry, and/or (3) create additional supporting entries that facilitate use of CWE's major views (Development, Researcher, and Architecture).

 

What is the criteria? E.g. from my list (please note it needs more SPLIT/MERGE and cleaning/work, but it gives an idea of stuff I want to add to CWE):

 

 

Are there any which would definitely not be included, or definitely would be included? E.g. the Smart Contract stuff that sort of maps to existing CWEs, can they be broken out like the .NET stuff sometimes is, or would the update a CWE, or some other option?

 

 

 

On Thu, Oct 29, 2020 at 11:11 AM Chris Eng <[hidden email]> wrote:

You mean this submission template?  😊

https://cwe.mitre.org/community/submissions/guidelines.html

 

 

From: Kurt Seifried <[hidden email]>
Sent: Thursday, October 29, 2020 11:28 AM
To: Drew Buttner <[hidden email]>
Cc: CWE CAPEC Board <[hidden email]>
Subject: [EXTERNAL] Re: CWE Compatibility Program

 

This email originated from outside of Veracode.

 


The biggest problem I see is how do I get data into CWE?

 

I've asked in the past for templates/submission guides, AFAIK none exist, so I just copied what's on the main site. 

 

So my ask would be: 

 

1) How to make my data as CWE compatible and easy to consume by existing CWE users as possible (also why reinvent a standard when we have one?)

 

2) How to submit data to CWE for inclusion

 

3) How to help shepard the new entries along. Like I have no idea. Does a committee vote on them? a single person? 

 

which speaks to transparency and the need to document how CWE works (cause I'm on the board and I have no idea, if any of you do, please let me know and I'll shut up). 

 

 

On Thu, Oct 29, 2020 at 5:09 AM Drew Buttner <[hidden email]> wrote:

CWE/CAPEC Board,

No action required with this. I want to inform you about an effort to engage organizations participating in the CWE Compatibility Program about plans to revitalize/improve the program.   

The plan is to reach out to a couple of vendors to discuss current state and possible ways to improve the program.  In addition, we would like to gauge their interest in participating in a virtual vendor summit sometime in early 2021 that would include all participating members of the CWE Compatibility Program. 

Attached is a one-pager that will be sent to interested parties. This initial inquiry will be sent to several of whom are represented on the CWE/CAPEC Board -- Veracode, Micro Focus, GrammaTech, and Synopsys. 

Please let me know if you have any questions/comments/ideas, or if you have any objections. The future of the CWE Compatibility will most certainly be an agenda item at an upcoming board meeting.

Thanks
Drew

---------

Andrew Buttner
The MITRE Corporation
[hidden email]
781-271-3515


 

--

Kurt Seifried
[hidden email]


 

--

Kurt Seifried
[hidden email]


 

--

 

Dr. Jason Oberg | Co-Founder and CTO | +1 (808) 635-7604

Tortuga Logic  |  75 E Santa Clara Street, San Jose, CA 95113

 

NOTICE TO RECIPIENT | This email and any attachments may contain private, confidential and privileged material for the sole use of the intended recipient. If you are not the intended recipient, please immediately notify the sender of the error by return email and delete this email and any attachments.


 

--

Kurt Seifried
[hidden email]