Quantcast

CVE / CPE Gap

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

CVE / CPE Gap

Jan-Oliver Wagner-3
Hello,

this question has been raised already in early September
but I'd like to emphasize it again as IMHO it is essential.

We are currently working on a tight SCAP integration for
OpenVAS (www.openvas.org). The official CPE and CVE dictionaries
are integrated and nicely crosslinked etc.

What occurs to a user quickly is that CPEs of CVEs are not
present in the CPE dictionary.

On Thursday 01 September 2011 19:28:36 Cheikes, Brant A. wrote:

> This is a known issue.  NIST's NVD is populated by a team of analysts who
> review incoming CVEs from MITRE and attempt to link those CVEs
> authoritatively to the affected CPEs.  This process causes new CPEs to be
> generated by the NVD analysts and submitted for inclusion in the CPE
> Dictionary-but the pipe is filled far faster than the CPE review team is
> able to drain it (i.e., vet the CPEs and enter them into the official
> dictionary).  This results in a situation in which some CVEs in the NVD
> refer to CPEs that aren't yet in the dictionary; rather, they're submitted
> and under review.  NIST is aware and working to improve synchronization
> between the two data feeds.

We did a quick analysis on missing CPEs and in fact it seems that not
just some are missing, but quite number even with a growing gap (as of today):

88794 CPEs occuring in CVE but missing in CPE dictionary.

The gap as such is huge, but the _increase_ of the gap is what I am
most concerned with.

I am sure the NIST/MITRE teams do the best they can, but perhaps they
are simply outnumbered?

If you think the work load can be handled better in a distributed approach,
just tell us what we can do and what procedure to follow. The Greenbone
team already submitted a number of CPEs but we lack a clear procedure to
do so for existing CVEs. The key issue here is that the procedure should
avoid same CPEs being prepare for submission twice concurrently by different
parties.

Any other idea how we can assist in speeding up the process and avoid
the gap?

The main reason for for huge number is that it all depends on exact
version matches. While CPE dictionaries knows most products in general,
it has only few versions stored. Perhaps a solution to lower work load
could start here?

All the best

        Jan

--
Dr. Jan-Oliver Wagner |  ++49-541-335084-0  |  http://www.greenbone.net/
Greenbone Networks GmbH, Neuer Graben 17, 49074 Osnabrück | AG Osnabrück, HR B 202460
Geschäftsführer: Lukas Grunwald, Dr. Jan-Oliver Wagner
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CVE / CPE Gap

Booth, Harold
Hi Jan,

  I would first like to thank you for your interest and for all of your comments. I would agree that it is essential that we are able to stop the gap from increasing and as time permits shrink the backlog of CPEs. To provide some background the vast majority of the current backlog predates NVD's use of CPE when the NVD used its own method of identifying products.  NVD's product identifiers were machine converted to CPE format, but we are of the belief that most if not all of those CPEs require at least some human attention to insure that those names conform to the existing CPE specification.  The current process which migrates CPEs generated as part of the CVE analysis process is a very manual one which is why you see the gap continuing to increase.  The ability to keep the backlog from growing has been an identified need for quite some time and to that end, over the last few months we have undertaken the process of reworking the software which supports the analysis process for CVEs and the moderation process for CPEs. We are planning on deploying the new software into our production environment in the near future and we hope the increased automation will enable us to keep up with the new CPEs generated through CVE analysis.
  Additionally, we are hoping to make available a public interface to allow public users to submit CPE suggestions that would then be input into the moderation process.  The interface will initially be a human interface, but we hope to augment that with a machine interface (either REST or SOAP-based) not too long after.
  I am uncertain as to whether the entire backlog of old product identifiers mapped into CPE will ever be worked through, but priority for CPEs associated with CVEs recently created will be given a higher priority to those which are older.
  If you have identified CPEs that you would like to see included in the dictionary, associated to a CVE or not, we are always willing to accept external contributions.  I would ask if you are working on CPEs associated with a CVE that you note the CVE when providing your submission, and if you find that a CPE name is incorrect and associated with a CVE, a deprecation entry be provided with the submission, including the CPE which should be used instead.

Regards,

-Harold

-----Original Message-----
From: Jan-Oliver Wagner [mailto:[hidden email]]
Sent: Tuesday, November 08, 2011 4:20 AM
To: [hidden email]
Subject: [CPE-DISCUSSION-LIST] CVE / CPE Gap

Hello,

this question has been raised already in early September
but I'd like to emphasize it again as IMHO it is essential.

We are currently working on a tight SCAP integration for
OpenVAS (www.openvas.org). The official CPE and CVE dictionaries
are integrated and nicely crosslinked etc.

What occurs to a user quickly is that CPEs of CVEs are not
present in the CPE dictionary.

On Thursday 01 September 2011 19:28:36 Cheikes, Brant A. wrote:

> This is a known issue.  NIST's NVD is populated by a team of analysts who
> review incoming CVEs from MITRE and attempt to link those CVEs
> authoritatively to the affected CPEs.  This process causes new CPEs to be
> generated by the NVD analysts and submitted for inclusion in the CPE
> Dictionary-but the pipe is filled far faster than the CPE review team is
> able to drain it (i.e., vet the CPEs and enter them into the official
> dictionary).  This results in a situation in which some CVEs in the NVD
> refer to CPEs that aren't yet in the dictionary; rather, they're submitted
> and under review.  NIST is aware and working to improve synchronization
> between the two data feeds.

We did a quick analysis on missing CPEs and in fact it seems that not
just some are missing, but quite number even with a growing gap (as of today):

88794 CPEs occuring in CVE but missing in CPE dictionary.

The gap as such is huge, but the _increase_ of the gap is what I am
most concerned with.

I am sure the NIST/MITRE teams do the best they can, but perhaps they
are simply outnumbered?

If you think the work load can be handled better in a distributed approach,
just tell us what we can do and what procedure to follow. The Greenbone
team already submitted a number of CPEs but we lack a clear procedure to
do so for existing CVEs. The key issue here is that the procedure should
avoid same CPEs being prepare for submission twice concurrently by different
parties.

Any other idea how we can assist in speeding up the process and avoid
the gap?

The main reason for for huge number is that it all depends on exact
version matches. While CPE dictionaries knows most products in general,
it has only few versions stored. Perhaps a solution to lower work load
could start here?

All the best

        Jan

--
Dr. Jan-Oliver Wagner |  ++49-541-335084-0  |  http://www.greenbone.net/
Greenbone Networks GmbH, Neuer Graben 17, 49074 Osnabrück | AG Osnabrück, HR B 202460
Geschäftsführer: Lukas Grunwald, Dr. Jan-Oliver Wagner
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CVE / CPE Gap

Jan-Oliver Wagner-3
Harold,

indeed the old backlog will cause a problem. Not only because of the number.
Its also because the proof of a certain version is harder (web pages might have
gone meanwhile).

I appreciate there will be a better interface for reporting the CPEs.
Our CPE detection system is automatized anyway, so it could easily provided
automated content and even submit automated.
The human verification is of course not avoidable here.

It should be fairly easy to build some scripts that do prepare CPE content
from a verified RPM or DEB package database. Any such produced CPE content
could be checked against the CVE CPEs and where match is found, it could be added
to the CPE dictionary. Should catch quite a number of CPEs.

Best

        Jan

On Thursday, 10. November 2011, Booth, Harold wrote:
>   I would first like to thank you for your interest and for all of your comments. I would agree that it is essential that we are able to stop the gap from increasing and as time permits shrink the backlog of CPEs. To provide some background the vast majority of the current backlog predates NVD's use of CPE when the NVD used its own method of identifying products.  NVD's product identifiers were machine converted to CPE format, but we are of the belief that most if not all of those CPEs require at least some human attention to insure that those names conform to the existing CPE specification.  The current process which migrates CPEs generated as part of the CVE analysis process is a very manual one which is why you see the gap continuing to increase.  The ability to keep the backlog from growing has been an identified need for quite some time and to that end, over the last few months we have undertaken the process of reworking the software which supports the analysis process for CVEs and the moderation process for CPEs. We are planning on deploying the new software into our production environment in the near future and we hope the increased automation will enable us to keep up with the new CPEs generated through CVE analysis.
>   Additionally, we are hoping to make available a public interface to allow public users to submit CPE suggestions that would then be input into the moderation process.  The interface will initially be a human interface, but we hope to augment that with a machine interface (either REST or SOAP-based) not too long after.
>   I am uncertain as to whether the entire backlog of old product identifiers mapped into CPE will ever be worked through, but priority for CPEs associated with CVEs recently created will be given a higher priority to those which are older.
>   If you have identified CPEs that you would like to see included in the dictionary, associated to a CVE or not, we are always willing to accept external contributions.  I would ask if you are working on CPEs associated with a CVE that you note the CVE when providing your submission, and if you find that a CPE name is incorrect and associated with a CVE, a deprecation entry be provided with the submission, including the CPE which should be used instead.



--
Dr. Jan-Oliver Wagner |  ++49-541-335084-0  |  http://www.greenbone.net/
Greenbone Networks GmbH, Neuer Graben 17, 49074 Osnabrück | AG Osnabrück, HR B 202460
Geschäftsführer: Lukas Grunwald, Dr. Jan-Oliver Wagner
Loading...