Merge branch 'master' into release-2.8

.github/pull_request_template.md

@@ -3,6 +3,9 @@ The purpose of this document is to establish a process and checklist for new SCR
 all code changes have been completed and is ready for testing and subsequent release
 
 ## Checklist
+- [ ] Rev the version information
+    - [ ] Local VERSION file
+    - [ ] PIP setup.cfg file   
 - [ ] Review [CodeQL results](https://github.com/nasa/scrub/security/code-scanning)
     - [ ] Check for commented out/debugging code
     - [ ] Make necessary code changes based on findings

.github/workflows/codeql.yml

@@ -18,7 +18,7 @@ on:
     # The branches below must be a subset of the branches above
     branches: [ master ]
   schedule:
-    - cron: '0 5 * * *'
+    - cron: '0 5 * * 0'
 
 jobs:
   analyze:
@@ -65,14 +65,28 @@ jobs:
 
     - name: Perform CodeQL Analysis
       uses: github/codeql-action/analyze@v2
-
-    - run: |
-        pip install nasa-scrub
-        python3 -m scrub.tools.parsers.translate_results /home/runner/work/scrub/results/*.sarif /home/runner/work/scrub/results/codeql.scrub ${{ github.workspace }} scrub
-        python3 -m scrub.tools.parsers.csv_parser /home/runner/work/scrub/results
+
+    - name: Post-Process Output
+      run: |
+        python3 -m pip install nasa-scrub
+
+        results_dir=`realpath ${{ github.workspace }}/../results`
+        sarif_files=`find $results_dir -name '*.sarif'`
+
+        for sarif_file in $sarif_files
+        do
+          output_file="$results_dir/$(basename $sarif_file .sarif).scrub"
+
+          python3 -m scrub.tools.parsers.translate_results $sarif_file $output_file ${{ github.workspace }} scrub
+        done
+
+        python3 -m scrub.tools.parsers.csv_parser $results_dir
+
+        echo "RESULTS_DIR=$results_dir" >> $GITHUB_ENV
+        
       
     - name: Upload CodeQL Artifacts
       uses: actions/upload-artifact@v3
       with:
         name: codeql-artifacts
-        path: /home/runner/work/scrub/results/*
+        path: ${{ env.RESULTS_DIR }}

docs/output.md

@@ -52,6 +52,7 @@ An example of a set of two warnings that adhere to this format:
 The following section provides a description of the structure of the .scrub output directory located at `SOURCE_DIR` as specified in the `scrub.cfg` configuration file:
 
     .scrub
+    |  VERSION                          (Version of SCRUB that generated results)
     |  scrub.cfg                        (Copy of user-provided configuration file)
     |  SCRUBAnalysisFilteringList       (List of source files that will be included in analysis)
     |  SCRUBCollaboratorFiltering List  (List of source files that will be uploaded to Collaborator)

docs/reviewing.md

@@ -32,3 +32,12 @@ Please refer to the [Detailed Configuration](configuration.md) page for more inf
 ### Automated Invocation
 
 There is a section of the `scrub.cfg` file that can be used for pushing results to Collaborator automatically. For more information about the expected inputs for this process, please refer to the [Detailed Configuration](configuration.md) page. SCRUB analysis will be performed normally, but before execution is completed SCRUB will create a new Collaborator review that is initialized with all of the SCRUB results.
+
+### Collaborator Review Templates
+The Collaborator section of the `scrub.cfg` includes a section where users may define a review template to be used for Collaborator uploads. This can be any template on the Collaborator server, so long as it has the custom fields SCRUB expects. These fields are listed below, along with information to configure them on the Collaborator instance. Collaborator uploads may fail if these are not configured appropriately.
+
+| Collaborator Template Section | Field Name | Type                                                                | Default Value |
+| ----------------------------- | ---------- | ------------------------------------------------------------------- | ------------- |
+| Review Custom Fields          | Overview   | String (Multi-line)                                                 | None          |
+| Defect Custom Fields          | Severity   | Drop-down List (Blocker, Critical, Major, Moderate, Minor, Trivial) | Moderate      |
+

docs/usage.md

@@ -42,6 +42,13 @@ After SCRUB has been installed, it can be run using the command line interface.
 
         Inputs:
             - output: Path to desired output location [string] [optional]
+            
+    scrub version
+
+        This function prints the current SCRUB version information to the console
+        
+        Inputs:
+            - None
 
 **Note**: `scrub run-tool` is a legacy command and only included for backwards compatability. Users are incouraged to use the `scrub run` command and `--tools` flag to run individual tools.
 
@@ -56,9 +63,11 @@ Running SCRUB is a relatively straightforward process after it has been configur
 
 During execution SCRUB will print various status messages to the console. Additionally, log information and results will be stored in a hidden directory named `.scrub` located at `SOURCE_DIR` as defined in the scrub.cfg file used during execution.
 
-Tools can also be run individuall by using using the `run --tools` command. An example is provided below for CodeQL execution:
+A subset of tools can also be run by using using the `run --tools` command. The tools flag expects a space-separated list of tools to be provided.
 
     scrub run --tools codeql --config scrub.cfg --quiet
+
+    scrub run --tools coverity codesonar --debug
 
 
 ## Dependencies

scrub/__init__.py

@@ -4,6 +4,7 @@
 __email__ = '[email protected]'
 __version__ = '2.8'
 
+
 # Check the Python version for compatibility
 import sys
 if not sys.version_info >= (3, 6):

scrub/scrubme.py

@@ -26,7 +26,6 @@ def parse_arguments():
     parser.add_argument('--quiet', action='store_true')
     parser.add_argument('--tools', nargs='+', default=[])
     parser.add_argument('--targets', nargs='+', default=None)
-    parser.add_argument('-d', '--define', action="append")
 
     # Parse the arguments
     args = vars(parser.parse_args(sys.argv[2:]))
@@ -40,11 +39,11 @@ def parse_arguments():
         logging_level = logging.INFO
 
     # Run analysis
-    main(pathlib.Path(args['config']).resolve(), args['clean'], logging_level, args['tools'], args['targets'], args['define'])
+    main(pathlib.Path(args['config']).resolve(), args['clean'], logging_level, args['tools'], args['targets'])
 
 
 def main(conf_file=pathlib.Path('./scrub.cfg').resolve(), clean=False, console_logging=logging.INFO, tools=None,
-         targets=None, override_values=None):
+         targets=None):
     """
     This function runs all applicable tools present within the configuration file.
 
@@ -59,13 +58,11 @@ def main(conf_file=pathlib.Path('./scrub.cfg').resolve(), clean=False, console_l
             Default value: None
         - targets: List of output targets for exporting the analysis results [list of strings] [optional]
             Default value: None
-        - define: List of override values for SCRUB analysis [list of strings] [optional]
-            Default value: None
     """
 
     # Read in the configuration data
     if conf_file.exists():
-        scrub_conf_data = scrub_utilities.parse_common_configs(conf_file, override_values)
+        scrub_conf_data = scrub_utilities.parse_common_configs(conf_file)
     else:
         print('ERROR: Configuration file ' + str(conf_file) + ' does not exist.')
         sys.exit(10)

scrub/targets/collaborator/do_collaborator.py

@@ -117,7 +117,7 @@ def get_defects(scrub_file):
     defect_list = []
 
     # Import the defects from the file of interest
-    with open(scrub_file, 'r') as fh:
+    with open(scrub_file, 'r', encoding='UTF-8') as fh:
         results = fh.read()
 
     # Split the results into defects

scrub/tools/parsers/csv_parser.py

@@ -66,4 +66,4 @@ def parse_warnings(input_dir, output_format='legacy'):
     if len(sys.argv) == 2:
         parse_warnings(pathlib.Path(sys.argv[1]).resolve())
     else:
-        parse_warnings(pathlib.Path(sys.argv[1]).resolve(), sys.argv[2])
+        parse_warnings(pathlib.Path(sys.argv[1]).resolve(), sys.argv[2])

scrub/utils/scrub_utilities.py

@@ -305,13 +305,12 @@ def create_conf_file(output_path=None):
     shutil.copyfile(default_config_file, output_path)
 
 
-def parse_common_configs(user_conf_file, raw_override_values, scrub_keys=[]):
+def parse_common_configs(user_conf_file, scrub_keys=[]):
     """This function parses a SCRUB configuration file and adds default values.
 
     Inputs:
         - conf_file: Absolute path to the SCRUB configuration file [string]
         - scrub_keys: List of configuration file sections to be retrieved [list of strings]
-        - override_values: List of values that should override the config file values [list of strings]
 
     Outputs:
         - scrub_conf_data: Dictionary of values read from configuration file [dict]
@@ -321,22 +320,16 @@ def parse_common_configs(user_conf_file, raw_override_values, scrub_keys=[]):
     scrub_conf_data = {}
     scrub_init_path = pathlib.Path(__file__).parent.joinpath('scrub_defaults.cfg')
 
-    # Convert override values into dictionary values
-    override_values = {}
-    if raw_override_values:
-        for value in raw_override_values:
-            override_values[value.split('=', 1)[0]] = value.split('=', 1)[1]
-
     # Read in the default config data
     scrub_init_data = configparser.ConfigParser()
     scrub_init_data.read(scrub_init_path)
 
     # Set the keys, if necessary
-    # if not scrub_keys:
-    #     scrub_keys = scrub_init_data.sections()
+    if not scrub_keys:
+        scrub_keys = scrub_init_data.sections()
 
     # Convert to a dictionary
-    for key in scrub_init_data.sections():
+    for key in scrub_keys:
         scrub_conf_data.update(dict(scrub_init_data.items(key)))
 
     # Read in the values from the conf file
@@ -347,9 +340,7 @@ def parse_common_configs(user_conf_file, raw_override_values, scrub_keys=[]):
     for user_section in user_conf_data.sections():
         for section_key in user_conf_data.options(user_section):
             # Update the value if the user conf has something
-            if override_values.get(section_key):
-                scrub_conf_data.update({section_key: override_values.get(section_key)})
-            elif user_conf_data.get(user_section, section_key):
+            if user_conf_data.get(user_section, section_key):
                 scrub_conf_data.update({section_key: user_conf_data.get(user_section, section_key)})
             elif section_key not in scrub_conf_data.keys():
                 # Add the key if it doesn't exist

setup.cfg

@@ -21,6 +21,7 @@ project_urls =
 url = https://github.com/nasa/scrub
 version = 2.8
 
+
 [options]
 include_package_data = True
 install_requires =