Add Red Hat Oval v2 vulnerability plugin (#1024)

* Vulnsrc: return flag map instead of single value

Updates can return only one value using FlagKey and FlagValue. This is
limited in some cases when updaters want to store more than one value.

This commit enables to return map with more than one key + value.

Signed-off-by: Ales Raszka <[email protected]>

* Clair use public oval v2 data (#21)

* CLAIR-262: add oval v2 class for parsing oval manifest, tests

* removed unused import

* CLAIR-263: updated oval v2 class to check oval manifest for only new advisories, added tests

* CLAIR-263: updated struct names, cleanup in oval v2 class for checking oval manifest for new advisories, added comments, additional tests

* CLAIR-264: added parse utilities for cpe names and rpm names, tests

* CLAIR-264: added module namespace parsing, tests, cleanup

* CLAIR-269: added package filtering by arch, tests

* WIP: refactored oval2 plugin and tests, based on changes required for rpm parsing and efficient ordering of assessments for whether a given advisory is already processed

* WIP: additional refactor for oval2 plugin and tests, fix nvra parsing and db lookups for already-processed, refactored manifest entry processing to iterate processing by document instead of all at once

* fix refactor issue: restore rewire for vuln namespace

* fix refactor issue: add check for empty lookup date, update test to cover that case

* WIP: refactor, cleanup, and fixes related to most recent PR review notes

* WIP: additional refactor/cleanup from PR review

* ensure redhat package is enabled by default for make deploy-local

* updated logging for redhat package

* cleanup comments, clarify logging in redhat package

* updated logging in redhat package, cleanup, moved db key/val date write to post-gather loop, fixed struct xml attribute ref

* added check to prevent advisories with severity "none" from being stored to database, updated tests

* cleanup: removed trailing whitespace

* use all cpe entries from affected_cpe_list (previously was intentionally excluding the first entry)

* updated supported arch check to support pattern-based arch lists

* removed no longer used function (ParseCpeStructFromAffectedCpeList)

* removed no longer used function (ParseCpeName)

* cleanup - removed redundant variable usage in GatherUnprocessedAdvisories

* cleanup - removed redundant second parse for already-parsed package list

* updated feature creation for module namespaces, to create a feature for each namespace

* removed no longer used function (ConstructVulnerabilityNames)

* removed no longer used function (IsRmpArchSupported)

* updated supported arch check to use regexp matcher

* refactored updater to use map of flags instead of just one flag (FlagName+FlagValue)

- redhat package uses ovalv2, needs multiple flags to update multiple key/value processing status markers
- updated existing vulnsrc impls to use the flag map

* cleanup - removed no longer used functions, related tests

* updated dependencies in go.sum to point to public repo

* add separate flag for last advisory date

* added support for checking definition class, test; only process patch definitions

* removed redundant entries from supported definition types

* cleanup, add check for non-empty parsed nvra data

* cleanup, lint-related formatting/comments

* fixed errors in go.mod, go.sum from rebase conflicts

Signed-off-by: Ales Raszka <[email protected]>

* CWF-129 (fixed rebase #2) related to changes for updating integration tests (#23)

* CLOUDWF-160: externalize base url for oval v2 data, to support fake api provider in integration tests

* cwf129: add check to skip duplicate packages, corresponding test, additional logging

* cwf129: additional checks to skip duplicate vulnerability data (filter out duplicate vuln data which appears across multiple oval docs), updated logging levels

* cwf129: additional checks to skip duplicate vulnerability data (filter out duplicate vuln data which appears across multiple oval docs), organized logging into logical groups, added tests

* cwf129: cleanup (formatting)

* cwf129: updated check for relevant criteria to include module criteria, removed unnecessary len() check from cpe name parsing

* cwf129: updated cpe parse to exclude empty names, added test, test data

* cwf129: cleanup (formatting)

* cwf129: cleanup - removed plugin-local log level assignment

* cwf129: updated dup vulnerability identification strategy to ensure any non-duplicate features of same-named vulnerabilities are merged rather than skipped, added tests

* CWF-129: fixed scoping for vulnerabilities accumulator so that vulns and packages are properly assessed across advisories, added test case and test data

* CWF-129: refactored vulnerabilities accumulator and features merge strategy to avoid using global variables, updated tests

Signed-off-by: Ales Raszka <[email protected]>

* Remove old rhel oval v1 scanner

Rhel vulnerability scanner was replaced by Red Hat's oval v2 scanner.
Oval 2 covers Red Hat's portfolio including RHEL.

The commit also fixes bugs in ubuntu scanner.

Signed-off-by: Ales Raszka <[email protected]>

Co-authored-by: John Bell <[email protected]>

cmd/clair/main.go

@@ -58,7 +58,7 @@ import (
 	_ "github.com/quay/clair/v3/ext/vulnsrc/amzn"
 	_ "github.com/quay/clair/v3/ext/vulnsrc/debian"
 	_ "github.com/quay/clair/v3/ext/vulnsrc/oracle"
-	_ "github.com/quay/clair/v3/ext/vulnsrc/rhel"
+	_ "github.com/quay/clair/v3/ext/vulnsrc/redhat"
 	_ "github.com/quay/clair/v3/ext/vulnsrc/suse"
 	_ "github.com/quay/clair/v3/ext/vulnsrc/ubuntu"
 )

config.yaml.sample

@@ -62,7 +62,7 @@ clair:
     enabledupdaters:
       - debian
       - ubuntu
-      - rhel
+      - redhat
       - oracle
       - alpine
       - suse

ext/vulnsrc/alpine/alpine.go

@@ -196,8 +196,8 @@ func (u *updater) Update(db database.Datastore) (resp vulnsrc.UpdateResponse, er
 	}
 
 	// Set the updaterFlag to equal the commit processed.
-	resp.FlagName = updaterFlag
-	resp.FlagValue = commit
+	resp.Flags = make(map[string]string)
+	resp.Flags[updaterFlag] = commit
 	if existingCommit, foundCommit, err = database.FindKeyValueAndRollback(db, updaterFlag); err != nil {
 		return
 	}

ext/vulnsrc/amzn/amzn.go

@@ -121,8 +121,8 @@ func (u *updater) Update(datastore database.Datastore) (vulnsrc.UpdateResponse,
 
 	// Set the flag value.
 	if timestamp != "" {
-		response.FlagName = u.UpdaterFlag
-		response.FlagValue = timestamp
+		response.Flags = make(map[string]string)
+		response.Flags[u.UpdaterFlag] = timestamp
 	} else {
 		log.WithField("package", u.Name).Debug("no update")
 	}

ext/vulnsrc/debian/debian.go

@@ -102,8 +102,8 @@ func buildResponse(jsonReader io.Reader, latestKnownHash string) (resp vulnsrc.U
 	// Defer the addition of flag information to the response.
 	defer func() {
 		if err == nil {
-			resp.FlagName = updaterFlag
-			resp.FlagValue = hash
+			resp.Flags = make(map[string]string)
+			resp.Flags[updaterFlag] = hash
 		}
 	}()
 

ext/vulnsrc/driver.go

@@ -33,8 +33,7 @@ var (
 
 // UpdateResponse represents the sum of results of an update.
 type UpdateResponse struct {
-	FlagName        string
-	FlagValue       string
+	Flags           map[string]string
 	Notes           []string
 	Vulnerabilities []database.VulnerabilityWithAffected
 }

ext/vulnsrc/oracle/oracle.go

@@ -196,8 +196,8 @@ func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateRespo
 
 	// Set the flag if we found anything.
 	if len(elsaList) > 0 {
-		resp.FlagName = updaterFlag
-		resp.FlagValue = strconv.Itoa(largest(elsaList))
+		resp.Flags = make(map[string]string)
+		resp.Flags[updaterFlag] = strconv.Itoa(largest(elsaList))
 	} else {
 		log.WithField("package", "Oracle Linux").Debug("no update")
 	}