Pattern matching for paths

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Pattern matching for paths

Harrison, Timothy [USA]

The need to use pattern matching for paths has been a long standing issue which based on my experience has had varied acceptance.  It is understandable for a tool vendor to simply ignore such uses of pattern matching as it can lead to scans which run for days on end and could possibly result in a DoS.  It is also understood that there may be a justifiable need to support such implementations however undesirable the result.

 

In many cases I have found that the need is simply for a single level of a path and rarely is intent upon checking multiple levels; however, the current approach used by OVALDI, possibly other tools, can result in a matching that tries all paths.  This is done regardless of the fact that a significant portion of the regular expression may actually be entirely escaped which could quickly reduce the set of paths to attempt.  In these instances it may be possible for tools processing the OVAL to short-circuit the matching knowing that, for instance, in Windows pairs of backslashes are separators and stepping down through the different levels of the path based on such separators could provide a shorter run time.  If content authors make use of the escape_regex to separate out the fixed portions even better as tools could take such liberties with a lesser chance of incorrectly evaluating a regular expression.

 

While I recognize using pattern matches in this fashion is less than ideal, at best, I also recognize at least for the interim we are stuck with them.  In my solutions above, which attempt to work within OVAL, are there any short-comings which I may have overlooked?  Should OVAL be changed to facilitate shorter run times for these pattern matches?  Besides using the escape_regex item and being very specific in regular expressions, are there ways content authors could better construct pattern matches which may have a similar effect?

 

This really goes beyond just paths so please make mention of other situations where pattern matching is used and may not be evaluated in the most efficient manner either due to the implementation in the content, support in OVAL or a tool. 

 

V/r,

Timothy S. Harrison

Booz|Allen|Hamilton

Cell: 717-372-5768

[hidden email]

 

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

Jon Baker
Administrator

One quick clarification… The OVAL Interpreter does have some built in optimization for processing regular expressions. The optimization is essentially what Tim described in his message below. Essentially, if you have anchored your path or registry key regular expressions and properly escaped them we can parse the regular expression and potentially save a lot of searching.

 

For example, take a look at oval:org.mitre.oval:obj:1436

<registry_object id="oval:org.mitre.oval:obj:1436" version="2" xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5#windows">
 
<hive>HKEY_LOCAL_MACHINE</hive>
 
<key operation="pattern match">^SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*</key>
 
<name>DisplayName</name>
</registry_object>

 

The OVAL Interpreter will parse “^SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*” and jump directly to “SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\”. This is possible because the expression is anchored and escaped. If the expression were unanchored (SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*) we would have to assume that any cahracters could appear before the initial SOFTWARE key. The interpreter first look to see if the pattern is anchored and then will look for the first regular expression special character that is not escaped. It then backs up from that position to the previous escaped path separator ‘\\’ and considers everything before it to be constant. This this results in a constant portion of the regex and a dynamic portion. The above regex is parsed into the following pieces:

 

Constant Portion = “SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\

Dynamic Portion = “.*”

 

I would love to hear what others think about this?

 

Jon

 

============================================

Jonathan O. Baker

G022 - IA Industry Collaboration

The MITRE Corporation

Email: [hidden email]

 

From: Harrison, Timothy [USA] [mailto:[hidden email]]
Sent: Tuesday, August 23, 2011 11:30 PM
To: oval-developer-list OVAL Developer List/Closed Public Discussion
Subject: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

The need to use pattern matching for paths has been a long standing issue which based on my experience has had varied acceptance.  It is understandable for a tool vendor to simply ignore such uses of pattern matching as it can lead to scans which run for days on end and could possibly result in a DoS.  It is also understood that there may be a justifiable need to support such implementations however undesirable the result.

 

In many cases I have found that the need is simply for a single level of a path and rarely is intent upon checking multiple levels; however, the current approach used by OVALDI, possibly other tools, can result in a matching that tries all paths.  This is done regardless of the fact that a significant portion of the regular expression may actually be entirely escaped which could quickly reduce the set of paths to attempt.  In these instances it may be possible for tools processing the OVAL to short-circuit the matching knowing that, for instance, in Windows pairs of backslashes are separators and stepping down through the different levels of the path based on such separators could provide a shorter run time.  If content authors make use of the escape_regex to separate out the fixed portions even better as tools could take such liberties with a lesser chance of incorrectly evaluating a regular expression.

 

While I recognize using pattern matches in this fashion is less than ideal, at best, I also recognize at least for the interim we are stuck with them.  In my solutions above, which attempt to work within OVAL, are there any short-comings which I may have overlooked?  Should OVAL be changed to facilitate shorter run times for these pattern matches?  Besides using the escape_regex item and being very specific in regular expressions, are there ways content authors could better construct pattern matches which may have a similar effect?

 

This really goes beyond just paths so please make mention of other situations where pattern matching is used and may not be evaluated in the most efficient manner either due to the implementation in the content, support in OVAL or a tool. 

 

V/r,

Timothy S. Harrison

Booz|Allen|Hamilton

Cell: 717-372-5768

[hidden email]

 

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

KenSimone

The McAfee OVAL interpreter has essentially the same optimization for pattern matching files and registry keys. It’s “not equals” that’s brutal  :-)

 

Ken

 

From: Baker, Jon [mailto:[hidden email]]
Sent: Wednesday, August 24, 2011 6:32 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

One quick clarification… The OVAL Interpreter does have some built in optimization for processing regular expressions. The optimization is essentially what Tim described in his message below. Essentially, if you have anchored your path or registry key regular expressions and properly escaped them we can parse the regular expression and potentially save a lot of searching.

 

For example, take a look at oval:org.mitre.oval:obj:1436

<registry_object id="oval:org.mitre.oval:obj:1436" version="2" xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5#windows">
 
<hive>HKEY_LOCAL_MACHINE</hive>
 
<key operation="pattern match">^SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*</key>
 
<name>DisplayName</name>
</registry_object>

 

The OVAL Interpreter will parse “^SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*” and jump directly to “SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\”. This is possible because the expression is anchored and escaped. If the expression were unanchored (SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*) we would have to assume that any cahracters could appear before the initial SOFTWARE key. The interpreter first look to see if the pattern is anchored and then will look for the first regular expression special character that is not escaped. It then backs up from that position to the previous escaped path separator ‘\\’ and considers everything before it to be constant. This this results in a constant portion of the regex and a dynamic portion. The above regex is parsed into the following pieces:

 

Constant Portion = “SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\

Dynamic Portion = “.*”

 

I would love to hear what others think about this?

 

Jon

 

============================================

Jonathan O. Baker

G022 - IA Industry Collaboration

The MITRE Corporation

Email: [hidden email]

 

From: Harrison, Timothy [USA] [mailto:[hidden email]]
Sent: Tuesday, August 23, 2011 11:30 PM
To: oval-developer-list OVAL Developer List/Closed Public Discussion
Subject: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

The need to use pattern matching for paths has been a long standing issue which based on my experience has had varied acceptance.  It is understandable for a tool vendor to simply ignore such uses of pattern matching as it can lead to scans which run for days on end and could possibly result in a DoS.  It is also understood that there may be a justifiable need to support such implementations however undesirable the result.

 

In many cases I have found that the need is simply for a single level of a path and rarely is intent upon checking multiple levels; however, the current approach used by OVALDI, possibly other tools, can result in a matching that tries all paths.  This is done regardless of the fact that a significant portion of the regular expression may actually be entirely escaped which could quickly reduce the set of paths to attempt.  In these instances it may be possible for tools processing the OVAL to short-circuit the matching knowing that, for instance, in Windows pairs of backslashes are separators and stepping down through the different levels of the path based on such separators could provide a shorter run time.  If content authors make use of the escape_regex to separate out the fixed portions even better as tools could take such liberties with a lesser chance of incorrectly evaluating a regular expression.

 

While I recognize using pattern matches in this fashion is less than ideal, at best, I also recognize at least for the interim we are stuck with them.  In my solutions above, which attempt to work within OVAL, are there any short-comings which I may have overlooked?  Should OVAL be changed to facilitate shorter run times for these pattern matches?  Besides using the escape_regex item and being very specific in regular expressions, are there ways content authors could better construct pattern matches which may have a similar effect?

 

This really goes beyond just paths so please make mention of other situations where pattern matching is used and may not be evaluated in the most efficient manner either due to the implementation in the content, support in OVAL or a tool. 

 

V/r,

Timothy S. Harrison

Booz|Allen|Hamilton

Cell: 717-372-5768

[hidden email]

 

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

joval
We should drop "not equals" from the language -- and force authors to use "equals" in conjunction with the "negate" attribute.

(I'm only half kidding)

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download



On 8/24/2011 9:11 AM, [hidden email] wrote:

The McAfee OVAL interpreter has essentially the same optimization for pattern matching files and registry keys. It’s “not equals” that’s brutal  :-)

 

Ken

 

From: Baker, Jon [[hidden email]]
Sent: Wednesday, August 24, 2011 6:32 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

One quick clarification… The OVAL Interpreter does have some built in optimization for processing regular expressions. The optimization is essentially what Tim described in his message below. Essentially, if you have anchored your path or registry key regular expressions and properly escaped them we can parse the regular expression and potentially save a lot of searching.

 

For example, take a look at oval:org.mitre.oval:obj:1436

<registry_object id="oval:org.mitre.oval:obj:1436" version="2" xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5#windows">
 
<hive>HKEY_LOCAL_MACHINE</hive>
 
<key operation="pattern match">^SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*</key>
 
<name>DisplayName</name>
</registry_object>

 

The OVAL Interpreter will parse “^SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*” and jump directly to “SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\”. This is possible because the expression is anchored and escaped. If the expression were unanchored (SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\.*) we would have to assume that any cahracters could appear before the initial SOFTWARE key. The interpreter first look to see if the pattern is anchored and then will look for the first regular expression special character that is not escaped. It then backs up from that position to the previous escaped path separator ‘\\’ and considers everything before it to be constant. This this results in a constant portion of the regex and a dynamic portion. The above regex is parsed into the following pieces:

 

Constant Portion = “SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\

Dynamic Portion = “.*”

 

I would love to hear what others think about this?

 

Jon

 

============================================

Jonathan O. Baker

G022 - IA Industry Collaboration

The MITRE Corporation

Email: [hidden email]

 

From: Harrison, Timothy [USA] [[hidden email]]
Sent: Tuesday, August 23, 2011 11:30 PM
To: oval-developer-list OVAL Developer List/Closed Public Discussion
Subject: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

The need to use pattern matching for paths has been a long standing issue which based on my experience has had varied acceptance.  It is understandable for a tool vendor to simply ignore such uses of pattern matching as it can lead to scans which run for days on end and could possibly result in a DoS.  It is also understood that there may be a justifiable need to support such implementations however undesirable the result.

 

In many cases I have found that the need is simply for a single level of a path and rarely is intent upon checking multiple levels; however, the current approach used by OVALDI, possibly other tools, can result in a matching that tries all paths.  This is done regardless of the fact that a significant portion of the regular expression may actually be entirely escaped which could quickly reduce the set of paths to attempt.  In these instances it may be possible for tools processing the OVAL to short-circuit the matching knowing that, for instance, in Windows pairs of backslashes are separators and stepping down through the different levels of the path based on such separators could provide a shorter run time.  If content authors make use of the escape_regex to separate out the fixed portions even better as tools could take such liberties with a lesser chance of incorrectly evaluating a regular expression.

 

While I recognize using pattern matches in this fashion is less than ideal, at best, I also recognize at least for the interim we are stuck with them.  In my solutions above, which attempt to work within OVAL, are there any short-comings which I may have overlooked?  Should OVAL be changed to facilitate shorter run times for these pattern matches?  Besides using the escape_regex item and being very specific in regular expressions, are there ways content authors could better construct pattern matches which may have a similar effect?

 

This really goes beyond just paths so please make mention of other situations where pattern matching is used and may not be evaluated in the most efficient manner either due to the implementation in the content, support in OVAL or a tool. 

 

V/r,

Timothy S. Harrison

Booz|Allen|Hamilton


Cell: 717-372-5768

[hidden email]

 

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].
To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download

Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

Eric Walker-2
In reply to this post by Jon Baker
I agree with Tim Harrison -- I think it's a mistake to get clever with
anchored regular expressions and try to extract optimizations from them.
In this case you have two code paths looking at the regular expression;
first you have the OVAL interpreter, which tries to pull out a parent
directory from the expression, and then you have the regular expression
library, which handles the hopefully cleanly truncated last part of the
pattern.  I would prefer that a single library interpret the regular
expression and that content authors take on part of the task of ensuring
that OVAL constructs will handle hard cases somewhat gracefully.

Eric


"Baker, Jon" <[hidden email]> wrote on 08/24/2011 04:32:21 AM:

> From: "Baker, Jon" <[hidden email]>
> To: <[hidden email]>
> Date: 08/24/2011 04:32 AM
> Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
>
> One quick clarification… The OVAL Interpreter does have some built
> in optimization for processing regular expressions. The optimization
> is essentially what Tim described in his message below. Essentially,
> if you have anchored your path or registry key regular expressions
> and properly escaped them we can parse the regular expression and
> potentially save a lot of searching.
>

[snip...]

>
> I would love to hear what others think about this?
>
> Jon
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

KenSimone
I don't want to dive too deep into this, since this is just an implementation detail. Implementers can do whatever they want as long as results are correct. I will add, though, that without any optimizations, "pattern match" on file system objects would be unusable. Every file on every drive would have to be considered. The registry wouldn't be much better.

Ken

-----Original Message-----
From: Eric Walker [mailto:[hidden email]]
Sent: Wednesday, August 24, 2011 10:12 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

I agree with Tim Harrison -- I think it's a mistake to get clever with anchored regular expressions and try to extract optimizations from them.
In this case you have two code paths looking at the regular expression; first you have the OVAL interpreter, which tries to pull out a parent directory from the expression, and then you have the regular expression library, which handles the hopefully cleanly truncated last part of the pattern.  I would prefer that a single library interpret the regular expression and that content authors take on part of the task of ensuring that OVAL constructs will handle hard cases somewhat gracefully.

Eric


"Baker, Jon" <[hidden email]> wrote on 08/24/2011 04:32:21 AM:

> From: "Baker, Jon" <[hidden email]>
> To: <[hidden email]>
> Date: 08/24/2011 04:32 AM
> Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
>
> One quick clarification… The OVAL Interpreter does have some built in
> optimization for processing regular expressions. The optimization is
> essentially what Tim described in his message below. Essentially, if
> you have anchored your path or registry key regular expressions and
> properly escaped them we can parse the regular expression and
> potentially save a lot of searching.
>

[snip...]

>
> I would love to hear what others think about this?
>
> Jon
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

Harrison, Timothy [USA]
In my initial email I failed to make mention of the anchor approach used for the OVAL Interpreter.  This approach seems to work and if content authors implement with this in mind it appears to work well.  My intent was to get community members to share any short-comings of this or other approaches as well as to stimulate conversation regarding alternate approaches and the differing support between tools which process OVAL.  It seems as though it would be beneficial if some standardized approach were defined.

I base this on the fact that in using SCAP tools I have found some tools simply do not process these pattern matches, some appear to use an inefficient approach, and others require special insights in order to create content which is processed efficiently.  The later may or may not be well handled by other tools.

Since SCAP is the primary use for OVAL this could be addressed in either; however, since the problem arises due to a lack of specific guidance/requirements in OVAL it should be addressed in OVAL.

~Tim

________________________________________
From: [hidden email] [[hidden email]]
Sent: Wednesday, August 24, 2011 2:30 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

I don't want to dive too deep into this, since this is just an implementation detail. Implementers can do whatever they want as long as results are correct. I will add, though, that without any optimizations, "pattern match" on file system objects would be unusable. Every file on every drive would have to be considered. The registry wouldn't be much better.

Ken

-----Original Message-----
From: Eric Walker [mailto:[hidden email]]
Sent: Wednesday, August 24, 2011 10:12 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

I agree with Tim Harrison -- I think it's a mistake to get clever with anchored regular expressions and try to extract optimizations from them.
In this case you have two code paths looking at the regular expression; first you have the OVAL interpreter, which tries to pull out a parent directory from the expression, and then you have the regular expression library, which handles the hopefully cleanly truncated last part of the pattern.  I would prefer that a single library interpret the regular expression and that content authors take on part of the task of ensuring that OVAL constructs will handle hard cases somewhat gracefully.

Eric


"Baker, Jon" <[hidden email]> wrote on 08/24/2011 04:32:21 AM:

> From: "Baker, Jon" <[hidden email]>
> To: <[hidden email]>
> Date: 08/24/2011 04:32 AM
> Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
>
> One quick clarification… The OVAL Interpreter does have some built in
> optimization for processing regular expressions. The optimization is
> essentially what Tim described in his message below. Essentially, if
> you have anchored your path or registry key regular expressions and
> properly escaped them we can parse the regular expression and
> potentially save a lot of searching.
>

[snip...]

>
> I would love to hear what others think about this?
>
> Jon

To unsubscribe, send an email message to [hidden email] with
SIGNOFF OVAL-DEVELOPER-LIST
in the BODY of the message.  If you have difficulties, write to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

joval
I think it would be useful if the language defined the concept of a search root -- a concrete path from which a pattern-match search should begin -- instead of relying on interpreters to out-think regular expression syntax.

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download



On 8/24/2011 2:33 PM, Harrison, Timothy [USA] wrote:
In my initial email I failed to make mention of the anchor approach used for the OVAL Interpreter.  This approach seems to work and if content authors implement with this in mind it appears to work well.  My intent was to get community members to share any short-comings of this or other approaches as well as to stimulate conversation regarding alternate approaches and the differing support between tools which process OVAL.  It seems as though it would be beneficial if some standardized approach were defined.

I base this on the fact that in using SCAP tools I have found some tools simply do not process these pattern matches, some appear to use an inefficient approach, and others require special insights in order to create content which is processed efficiently.  The later may or may not be well handled by other tools.

Since SCAP is the primary use for OVAL this could be addressed in either; however, since the problem arises due to a lack of specific guidance/requirements in OVAL it should be addressed in OVAL.

~Tim

________________________________________
From: [hidden email] [[hidden email]]
Sent: Wednesday, August 24, 2011 2:30 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

I don't want to dive too deep into this, since this is just an implementation detail. Implementers can do whatever they want as long as results are correct. I will add, though, that without any optimizations, "pattern match" on file system objects would be unusable. Every file on every drive would have to be considered. The registry wouldn't be much better.

Ken

-----Original Message-----
From: Eric Walker [[hidden email]]
Sent: Wednesday, August 24, 2011 10:12 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

I agree with Tim Harrison -- I think it's a mistake to get clever with anchored regular expressions and try to extract optimizations from them.
In this case you have two code paths looking at the regular expression; first you have the OVAL interpreter, which tries to pull out a parent directory from the expression, and then you have the regular expression library, which handles the hopefully cleanly truncated last part of the pattern.  I would prefer that a single library interpret the regular expression and that content authors take on part of the task of ensuring that OVAL constructs will handle hard cases somewhat gracefully.

Eric


"Baker, Jon" [hidden email] wrote on 08/24/2011 04:32:21 AM:

From: "Baker, Jon" [hidden email]
To: [hidden email]
Date: 08/24/2011 04:32 AM
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

One quick clarification… The OVAL Interpreter does have some built in
optimization for processing regular expressions. The optimization is
essentially what Tim described in his message below. Essentially, if
you have anchored your path or registry key regular expressions and
properly escaped them we can parse the regular expression and
potentially save a lot of searching.

[snip...]

I would love to hear what others think about this?

Jon
To unsubscribe, send an email message to [hidden email] with
SIGNOFF OVAL-DEVELOPER-LIST
in the BODY of the message.  If you have difficulties, write to [hidden email].
To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download

Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

Jon Baker
Administrator
In reply to this post by Harrison, Timothy [USA]
>In my initial email I failed to make mention of the anchor approach used for the
>OVAL Interpreter.  This approach seems to work and if content authors
>implement with this in mind it appears to work well.  My intent was to get
>community members to share any short-comings of this or other approaches as
>well as to stimulate conversation regarding alternate approaches and the
>differing support between tools which process OVAL.  It seems as though it
>would be beneficial if some standardized approach were defined.
>

We have encouraged content authors to properly anchor and end their regular expressions for years. Doing so is more precise, sort of. A regex with an anchor "^a.+" is very different then a regex "a.+" without an anchor. Try them out against a text file with perl or some free regular expression test app.

In most cases in the oval content we believe that content authors actually do want the anchor in their regular expressions. In the oval repository authoring style schematron rule we report a warning for all regular expressions that are not anchored or ended properly. This forces us to look at them and make sure the regex is correct.

>I base this on the fact that in using SCAP tools I have found some tools simply do
>not process these pattern matches, some appear to use an inefficient approach,
>and others require special insights in order to create content which is processed
>efficiently.  The later may or may not be well handled by other tools.
>

Testing of the tools with the OVAL test content should show which tools will evaluate regular expressions and which won't. This is a core capability that is widely used and really must be supported.

>Since SCAP is the primary use for OVAL this could be addressed in either;
>however, since the problem arises due to a lack of specific
>guidance/requirements in OVAL it should be addressed in OVAL.
>

I will let others comment on what additional guidance may be needed.

Thanks,

Jon

To unsubscribe, send an email message to [hidden email] with
SIGNOFF OVAL-DEVELOPER-LIST
in the BODY of the message.  If you have difficulties, write to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

Jon Baker
Administrator
In reply to this post by joval

Ironically, we were talking about just this sort of thing with in the team here at MITRE as we were looking to clean up some of the schema documentation around the used of behaviors for file searching. We will add a tracker item to further explore this concept in the next release.

 

Thanks,

 

Jon

 

============================================

Jonathan O. Baker

G022 - IA Industry Collaboration

The MITRE Corporation

Email: [hidden email]

 

From: David Solin [mailto:[hidden email]]
Sent: Wednesday, August 24, 2011 4:04 PM
To: oval-developer-list OVAL Developer List/Closed Public Discussion
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

I think it would be useful if the language defined the concept of a search root -- a concrete path from which a pattern-match search should begin -- instead of relying on interpreters to out-think regular expression syntax.

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download



On 8/24/2011 2:33 PM, Harrison, Timothy [USA] wrote:

In my initial email I failed to make mention of the anchor approach used for the OVAL Interpreter.  This approach seems to work and if content authors implement with this in mind it appears to work well.  My intent was to get community members to share any short-comings of this or other approaches as well as to stimulate conversation regarding alternate approaches and the differing support between tools which process OVAL.  It seems as though it would be beneficial if some standardized approach were defined.
 
I base this on the fact that in using SCAP tools I have found some tools simply do not process these pattern matches, some appear to use an inefficient approach, and others require special insights in order to create content which is processed efficiently.  The later may or may not be well handled by other tools.
 
Since SCAP is the primary use for OVAL this could be addressed in either; however, since the problem arises due to a lack of specific guidance/requirements in OVAL it should be addressed in OVAL.
 
~Tim
 
________________________________________
From: [hidden email] [[hidden email]]
Sent: Wednesday, August 24, 2011 2:30 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
I don't want to dive too deep into this, since this is just an implementation detail. Implementers can do whatever they want as long as results are correct. I will add, though, that without any optimizations, "pattern match" on file system objects would be unusable. Every file on every drive would have to be considered. The registry wouldn't be much better.
 
Ken
 
-----Original Message-----
From: Eric Walker [[hidden email]]
Sent: Wednesday, August 24, 2011 10:12 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
I agree with Tim Harrison -- I think it's a mistake to get clever with anchored regular expressions and try to extract optimizations from them.
In this case you have two code paths looking at the regular expression; first you have the OVAL interpreter, which tries to pull out a parent directory from the expression, and then you have the regular expression library, which handles the hopefully cleanly truncated last part of the pattern.  I would prefer that a single library interpret the regular expression and that content authors take on part of the task of ensuring that OVAL constructs will handle hard cases somewhat gracefully.
 
Eric
 
 
"Baker, Jon" [hidden email] wrote on 08/24/2011 04:32:21 AM:
 
From: "Baker, Jon" [hidden email]
To: [hidden email]
Date: 08/24/2011 04:32 AM
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
One quick clarification… The OVAL Interpreter does have some built in
optimization for processing regular expressions. The optimization is
essentially what Tim described in his message below. Essentially, if
you have anchored your path or registry key regular expressions and
properly escaped them we can parse the regular expression and
potentially save a lot of searching.
 
 
[snip...]
 
 
I would love to hear what others think about this?
 
Jon
 
To unsubscribe, send an email message to [hidden email] with
SIGNOFF OVAL-DEVELOPER-LIST
in the BODY of the message.  If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

KenSimone

I’m pretty sure OVAL already has this capability. I haven’t written content and tried it, but I think with object filters, and the recurse behavior, you can do this.

 

1)      Set the object’s path to be the concrete path.

2)      Set recurse_direction to “down” (or “up” if you want I guess).

3)      Set max_depth to -1 (or whatever you want).

4)      Add a filter to the object with the desired regex for the path.

 

Ken

 

From: Baker, Jon [mailto:[hidden email]]
Sent: Wednesday, August 24, 2011 8:22 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

Ironically, we were talking about just this sort of thing with in the team here at MITRE as we were looking to clean up some of the schema documentation around the used of behaviors for file searching. We will add a tracker item to further explore this concept in the next release.

 

Thanks,

 

Jon

 

============================================

Jonathan O. Baker

G022 - IA Industry Collaboration

The MITRE Corporation

Email: [hidden email]

 

From: David Solin [mailto:[hidden email]]
Sent: Wednesday, August 24, 2011 4:04 PM
To: oval-developer-list OVAL Developer List/Closed Public Discussion
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

I think it would be useful if the language defined the concept of a search root -- a concrete path from which a pattern-match search should begin -- instead of relying on interpreters to out-think regular expression syntax.

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download



On 8/24/2011 2:33 PM, Harrison, Timothy [USA] wrote:

In my initial email I failed to make mention of the anchor approach used for the OVAL Interpreter.  This approach seems to work and if content authors implement with this in mind it appears to work well.  My intent was to get community members to share any short-comings of this or other approaches as well as to stimulate conversation regarding alternate approaches and the differing support between tools which process OVAL.  It seems as though it would be beneficial if some standardized approach were defined.
 
I base this on the fact that in using SCAP tools I have found some tools simply do not process these pattern matches, some appear to use an inefficient approach, and others require special insights in order to create content which is processed efficiently.  The later may or may not be well handled by other tools.
 
Since SCAP is the primary use for OVAL this could be addressed in either; however, since the problem arises due to a lack of specific guidance/requirements in OVAL it should be addressed in OVAL.
 
~Tim
 
________________________________________
From: [hidden email] [[hidden email]]
Sent: Wednesday, August 24, 2011 2:30 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
I don't want to dive too deep into this, since this is just an implementation detail. Implementers can do whatever they want as long as results are correct. I will add, though, that without any optimizations, "pattern match" on file system objects would be unusable. Every file on every drive would have to be considered. The registry wouldn't be much better.
 
Ken
 
-----Original Message-----
From: Eric Walker [[hidden email]]
Sent: Wednesday, August 24, 2011 10:12 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
I agree with Tim Harrison -- I think it's a mistake to get clever with anchored regular expressions and try to extract optimizations from them.
In this case you have two code paths looking at the regular expression; first you have the OVAL interpreter, which tries to pull out a parent directory from the expression, and then you have the regular expression library, which handles the hopefully cleanly truncated last part of the pattern.  I would prefer that a single library interpret the regular expression and that content authors take on part of the task of ensuring that OVAL constructs will handle hard cases somewhat gracefully.
 
Eric
 
 
"Baker, Jon" [hidden email] wrote on 08/24/2011 04:32:21 AM:
 
From: "Baker, Jon" [hidden email]
To: [hidden email]
Date: 08/24/2011 04:32 AM
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
One quick clarification… The OVAL Interpreter does have some built in
optimization for processing regular expressions. The optimization is
essentially what Tim described in his message below. Essentially, if
you have anchored your path or registry key regular expressions and
properly escaped them we can parse the regular expression and
potentially save a lot of searching.
 
 
[snip...]
 
 
I would love to hear what others think about this?
 
Jon
 
To unsubscribe, send an email message to [hidden email] with
SIGNOFF OVAL-DEVELOPER-LIST
in the BODY of the message.  If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

joval
This is really more of a work-around, as in the general case it requires collection of the items so that the filter can then be applied (since a filter can apply to more than just a path).  Alternatively, one could pepper non-generic filtering code at various places in one's scanner, which in my view is as sub-optimal as it sounds.

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download



On 8/25/2011 8:24 AM, [hidden email] wrote:

I’m pretty sure OVAL already has this capability. I haven’t written content and tried it, but I think with object filters, and the recurse behavior, you can do this.

 

1)      Set the object’s path to be the concrete path.

2)      Set recurse_direction to “down” (or “up” if you want I guess).

3)      Set max_depth to -1 (or whatever you want).

4)      Add a filter to the object with the desired regex for the path.

 

Ken

 

From: Baker, Jon [[hidden email]]
Sent: Wednesday, August 24, 2011 8:22 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

Ironically, we were talking about just this sort of thing with in the team here at MITRE as we were looking to clean up some of the schema documentation around the used of behaviors for file searching. We will add a tracker item to further explore this concept in the next release.

 

Thanks,

 

Jon

 

============================================

Jonathan O. Baker

G022 - IA Industry Collaboration

The MITRE Corporation

Email: [hidden email]

 

From: David Solin [[hidden email]]
Sent: Wednesday, August 24, 2011 4:04 PM
To: oval-developer-list OVAL Developer List/Closed Public Discussion
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

I think it would be useful if the language defined the concept of a search root -- a concrete path from which a pattern-match search should begin -- instead of relying on interpreters to out-think regular expression syntax.

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download



On 8/24/2011 2:33 PM, Harrison, Timothy [USA] wrote:

In my initial email I failed to make mention of the anchor approach used for the OVAL Interpreter.  This approach seems to work and if content authors implement with this in mind it appears to work well.  My intent was to get community members to share any short-comings of this or other approaches as well as to stimulate conversation regarding alternate approaches and the differing support between tools which process OVAL.  It seems as though it would be beneficial if some standardized approach were defined.
 
I base this on the fact that in using SCAP tools I have found some tools simply do not process these pattern matches, some appear to use an inefficient approach, and others require special insights in order to create content which is processed efficiently.  The later may or may not be well handled by other tools.
 
Since SCAP is the primary use for OVAL this could be addressed in either; however, since the problem arises due to a lack of specific guidance/requirements in OVAL it should be addressed in OVAL.
 
~Tim
 
________________________________________
From: [hidden email] [[hidden email]]
Sent: Wednesday, August 24, 2011 2:30 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
I don't want to dive too deep into this, since this is just an implementation detail. Implementers can do whatever they want as long as results are correct. I will add, though, that without any optimizations, "pattern match" on file system objects would be unusable. Every file on every drive would have to be considered. The registry wouldn't be much better.
 
Ken
 
-----Original Message-----
From: Eric Walker [[hidden email]]
Sent: Wednesday, August 24, 2011 10:12 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
I agree with Tim Harrison -- I think it's a mistake to get clever with anchored regular expressions and try to extract optimizations from them.
In this case you have two code paths looking at the regular expression; first you have the OVAL interpreter, which tries to pull out a parent directory from the expression, and then you have the regular expression library, which handles the hopefully cleanly truncated last part of the pattern.  I would prefer that a single library interpret the regular expression and that content authors take on part of the task of ensuring that OVAL constructs will handle hard cases somewhat gracefully.
 
Eric
 
 
"Baker, Jon" [hidden email] wrote on 08/24/2011 04:32:21 AM:
 
From: "Baker, Jon" [hidden email]
To: [hidden email]
Date: 08/24/2011 04:32 AM
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
One quick clarification… The OVAL Interpreter does have some built in
optimization for processing regular expressions. The optimization is
essentially what Tim described in his message below. Essentially, if
you have anchored your path or registry key regular expressions and
properly escaped them we can parse the regular expression and
potentially save a lot of searching.
 
 
[snip...]
 
 
I would love to hear what others think about this?
 
Jon
 
To unsubscribe, send an email message to [hidden email] with
SIGNOFF OVAL-DEVELOPER-LIST
in the BODY of the message.  If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].
To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download

Reply | Threaded
Open this post in threaded view
|

Re: Pattern matching for paths

Jon Baker
Administrator
In reply to this post by KenSimone

In our discussion, we came to a similar solution using the recursion related behaviors to control searching. There is really a lot of power there if you get creative.

 

Jon

 

============================================

Jonathan O. Baker

G022 - IA Industry Collaboration

The MITRE Corporation

Email: [hidden email]

 

From: [hidden email] [mailto:[hidden email]]
Sent: Thursday, August 25, 2011 9:24 AM
To: oval-developer-list OVAL Developer List/Closed Public Discussion
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

I’m pretty sure OVAL already has this capability. I haven’t written content and tried it, but I think with object filters, and the recurse behavior, you can do this.

 

1)      Set the object’s path to be the concrete path.

2)      Set recurse_direction to “down” (or “up” if you want I guess).

3)      Set max_depth to -1 (or whatever you want).

4)      Add a filter to the object with the desired regex for the path.

 

Ken

 

From: Baker, Jon [hidden email]
Sent: Wednesday, August 24, 2011 8:22 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

Ironically, we were talking about just this sort of thing with in the team here at MITRE as we were looking to clean up some of the schema documentation around the used of behaviors for file searching. We will add a tracker item to further explore this concept in the next release.

 

Thanks,

 

Jon

 

============================================

Jonathan O. Baker

G022 - IA Industry Collaboration

The MITRE Corporation

Email: [hidden email]

 

From: David Solin [hidden email]
Sent: Wednesday, August 24, 2011 4:04 PM
To: oval-developer-list OVAL Developer List/Closed Public Discussion
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths

 

I think it would be useful if the language defined the concept of a search root -- a concrete path from which a pattern-match search should begin -- instead of relying on interpreters to out-think regular expression syntax.

jOVAL.org: OVAL implemented in Java.
Scan any machine from any machine. For free!
Learn More | Features | Download



On 8/24/2011 2:33 PM, Harrison, Timothy [USA] wrote:

In my initial email I failed to make mention of the anchor approach used for the OVAL Interpreter.  This approach seems to work and if content authors implement with this in mind it appears to work well.  My intent was to get community members to share any short-comings of this or other approaches as well as to stimulate conversation regarding alternate approaches and the differing support between tools which process OVAL.  It seems as though it would be beneficial if some standardized approach were defined.
 
I base this on the fact that in using SCAP tools I have found some tools simply do not process these pattern matches, some appear to use an inefficient approach, and others require special insights in order to create content which is processed efficiently.  The later may or may not be well handled by other tools.
 
Since SCAP is the primary use for OVAL this could be addressed in either; however, since the problem arises due to a lack of specific guidance/requirements in OVAL it should be addressed in OVAL.
 
~Tim
 
________________________________________
From: [hidden email] [[hidden email]]
Sent: Wednesday, August 24, 2011 2:30 PM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
I don't want to dive too deep into this, since this is just an implementation detail. Implementers can do whatever they want as long as results are correct. I will add, though, that without any optimizations, "pattern match" on file system objects would be unusable. Every file on every drive would have to be considered. The registry wouldn't be much better.
 
Ken
 
-----Original Message-----
From: Eric Walker [[hidden email]]
Sent: Wednesday, August 24, 2011 10:12 AM
To: [hidden email]
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
I agree with Tim Harrison -- I think it's a mistake to get clever with anchored regular expressions and try to extract optimizations from them.
In this case you have two code paths looking at the regular expression; first you have the OVAL interpreter, which tries to pull out a parent directory from the expression, and then you have the regular expression library, which handles the hopefully cleanly truncated last part of the pattern.  I would prefer that a single library interpret the regular expression and that content authors take on part of the task of ensuring that OVAL constructs will handle hard cases somewhat gracefully.
 
Eric
 
 
"Baker, Jon" [hidden email] wrote on 08/24/2011 04:32:21 AM:
 
From: "Baker, Jon" [hidden email]
To: [hidden email]
Date: 08/24/2011 04:32 AM
Subject: Re: [OVAL-DEVELOPER-LIST] Pattern matching for paths
 
One quick clarification… The OVAL Interpreter does have some built in
optimization for processing regular expressions. The optimization is
essentially what Tim described in his message below. Essentially, if
you have anchored your path or registry key regular expressions and
properly escaped them we can parse the regular expression and
potentially save a lot of searching.
 
 
[snip...]
 
 
I would love to hear what others think about this?
 
Jon
 
To unsubscribe, send an email message to [hidden email] with
SIGNOFF OVAL-DEVELOPER-LIST
in the BODY of the message.  If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].

To unsubscribe, send an email message to [hidden email] with SIGNOFF OVAL-DEVELOPER-LIST in the BODY of the message. If you have difficulties, write to [hidden email].