Skip to content

Commit

Permalink
simple pattern matching added mercedes-benz#1912
Browse files Browse the repository at this point in the history
- *.xyz now works
- tests updated
- client exclude docs updated
- docs corrected from sechub to  `SecHub`
  • Loading branch information
sven-dmlr committed Feb 3, 2023
1 parent b2435f3 commit 395a3c2
Show file tree
Hide file tree
Showing 26 changed files with 292 additions and 283 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ build/
*.log
*.bak
*.zip
go.sum
sechub_report*.json
sechub_report*.html
sechub-false-positives*.json
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ func Test_prepareScan_binary_upload_respects_exclude_patterns(t *testing.T) {
"files": [ "sechub-cli-tmptest/test.bin" ],
"folders": [ "sechub-cli-tmptest/binaries/" ]
},
"excludes": [ "**/ignore/**", "**/*.bin" ]
"excludes": [ "**/ignore/**", "*.bin" ]
}
]
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@
package util

import (
"github.com/bmatcuk/doublestar/v4"
"path/filepath"
"strings"

"github.com/bmatcuk/doublestar/v4"
)

// FilePathMatch - This method provides ANT like selectors.
Expand All @@ -15,6 +16,11 @@ import (
// - "a1b.txt"
//
func FilePathMatch(path string, pattern string) (result bool) {
// Make simple patterns like `*.java` also work
if !strings.Contains(pattern, "/") {
pattern = "**/" + pattern
}

match, _ := doublestar.PathMatch(pattern, path)
return match
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ func TestXYZ_When_NO_double_asterisk_and_path_is_same_it_matches(t *testing.T) {
/* exact match*/
AssertTrue(FilePathMatch("/home/gargamel/schlumpfine/testfolder/a.txt", "/home/gargamel/schlumpfine/testfolder/a.txt"), t)

/* no match, no wildcards */
AssertFalse(FilePathMatch("/x/y/z/V/a.txt", "a.txt"), t)
/* simple pattern match (pattern without path separators) */
AssertTrue(FilePathMatch("/x/y/z/V/a.txt", "*.txt"), t)
}

func Test_When_double_asterisk_on_start_any_path_is_accepted_when_filename_without_asterisk_matches(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion sechub-cli/src/mercedes-benz.com/sechub/util/tar_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ func TestTarFileCanBeCreated_with_exclude_patterns_applied(t *testing.T) {
TarWriter: tarWriter,
PrefixInTar: "",
Folders: []string{dirname1, dirname2},
Excludes: []string{"**/sub3/**", "**/*.txt"},
Excludes: []string{"**/sub3/**", "*.txt"},
}

/* execute */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@ based upon your problem statement, the quality goals and key constraints.
| Development | We use AsciiDoc and checkin documenteation together inside one GIT repository
| see https://asciidoctor.org/docs/asciidoc-syntax-quick-reference/
| We want to have a way to provide special scan configurations in a dynamic way
| Deployment | We provide a JSON based scan configuration where different provider meta information can be defined. Dedicated providers can
use this information and handle dynamic configuration - e.g. based on name of sechub project.
| see https://github.com/mercedes-benz/sechub/issues/127 ,`ScanMappingConfigurationService` class or description inside `technical documentation`.
| Deployment | We provide a JSON based scan configuration where different provider meta information can be defined. Dedicated providers can
use this information and handle dynamic configuration - e.g. based on name of {secHub} project.
| see https://github.com/mercedes-benz/sechub/issues/127 ,`ScanMappingConfigurationService` class or description inside `technical documentation`.
|===
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ image::adapter-big-picture.svg[title="Adapter Big picture" opts="interactive,inl

By the *mock variant* it is possible to
- test the complete szenario in a very fast way...
- test configuration values from sechub are passed through adapter
- test configuration values from {secHub} are passed through adapter

===== Implementation of new adapters
*Only green highlighted classes* in former image above *needs to be*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,17 @@ Your Jenkins server should be at minimum: `Jenkins ver. 2.176.1`.

[NOTE]
====
The described Jenkins auto deployment is now *relying on the GitHub actions build* - see
The described Jenkins auto deployment is now *relying on the GitHub actions build* - see
https://github.com/mercedes-benz/sechub/tree/develop/.github/workflows[GitHub workflows folder].
Deployment artifacts are available at after gradle workflow has been done.
Deployment artifacts are available at after gradle workflow has been done.
In future the artifacts will be automatically available in
github packages so you can use those for your autodeployment process and just use it as a maven
repository. In the meantime you must deploy manually. Why still a jenkins file then?
repository. In the meantime you must deploy manually. Why still a jenkins file then?
- You get informed about new available releases.
- Also the jenkins will check additionally for problems
- You can do a checksum test for outputs - if you want to check sechub build is not corrupted and
- You get informed about new available releases.
- Also the jenkins will check additionally for problems
- You can do a checksum test for outputs - if you want to check that the {secHub} build is not corrupted and
every version can be rebuild at any time
====
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ sechub -jobUUID ${jobUUID} getReport
[[section-client-false-positives-mark]]
===== markFalsePositives
Read your report carefully and check which of the findings are really "false positives" (FP).
After this, create a json file containing jobUUID and ID of finding you decided to be a FP. If
After this, create a json file containing jobUUID and ID of finding you decided to be a FP. If
you like you can add an optional comment, describing your decision to mark this as FP.
Select this file by the `-file` argument and start action `markFalsePositives`.

Expand Down Expand Up @@ -201,12 +201,12 @@ Your choices are:

- `y` Yes - Mark as false positive. You are prompted for an optional comment line which can help to understand later why this has been marked.
- `n` No - Do nothing and go to next item.
- `s` Skip following findings - End the asking here and add the marked items to the false positives list on the sechub server.
- `s` Skip following findings - End the asking here and add the marked items to the false positives list on the {secHub} server.
- `c` Cancel - Quit and upload nothing

You can download and view your current false positives list with <<section-client-getFalsePositives,getFalsePositives>>

See also <<section-client-interactiveMarkFalsePositives-mark,example for interactiveMarkFalsePositives>>
See also <<section-client-interactiveMarkFalsePositives-mark,example for interactiveMarkFalsePositives>>

===== interactiveUnmarkFalsePositives
Instead of creating a json file to unmark false positives, you can also do this interactively:
Expand All @@ -217,10 +217,10 @@ Your choices are:

- `y` Yes - Unmark this false positive.
- `n` No - Do nothing and go to next item.
- `s` Skip the rest - End the asking here and remove the marked items from the false positives list on the sechub server.
- `s` Skip the rest - End the asking here and remove the marked items from the false positives list on the {secHub} server.
- `c` Cancel - Quit and change nothing

See also <<section-client-interactiveMarkFalsePositives-mark,example for interactiveMarkFalsePositives>>
See also <<section-client-interactiveMarkFalsePositives-mark,example for interactiveMarkFalsePositives>>


==== Configuration overview
Expand Down Expand Up @@ -256,15 +256,15 @@ In below table, there is an overview of what can be defined where.
Here is an overview of mandatory parameter for each `action`:
|===
| *action* | *mandatory parameters*
| scan |server + user + api token + projectID + configFile
| scanAsync |server + user + api token + projectID + configFile
| getStatus |server + user + api token + projectID + secHubJobUUID
| getReport |server + user + api token + projectID + secHubJobUUID
| scan |server + user + api token + projectID + configFile
| scanAsync |server + user + api token + projectID + configFile
| getStatus |server + user + api token + projectID + secHubJobUUID
| getReport |server + user + api token + projectID + secHubJobUUID
| getFalsePositives |server + user + api token + projectID
| markFalsePositives |server + user + api token + projectID + file
| unmarkFalsePositives |server + user + api token + projectID + file
| markFalsePositives |server + user + api token + projectID + file
| unmarkFalsePositives |server + user + api token + projectID + file
| interactiveMarkFalsePositives |server + user + api token + projectID + (file only if no report file is in current dir)
| interactiveUnmarkFalsePositives |server + user + apiToken + projectID
| interactiveUnmarkFalsePositives |server + user + apiToken + projectID
| help |
| version |
|===
Expand Down Expand Up @@ -304,8 +304,8 @@ Here is an overview of mandatory parameter for each `action`:
Username - mandatory, but can also be defined in config file or via environment variable.
- `-version` +
Shows version info and terminates
- `-wait <seconds>` +
Maximum wait time in seconds (default 60).
- `-wait <seconds>` +
Maximum wait time in seconds (default 60).
Will be used for periodic status checks when action=`scan` and for retries of HTTP calls.

==== Environment variables
Expand Down Expand Up @@ -335,7 +335,7 @@ Settings for debugging/client development:
- `SECHUB_DEBUG_HTTP` +
When set to `true` then log HTTP request contents.
- `SECHUB_IGNORE_DEFAULT_EXCLUDES` +
When set to `true` then default exclude folders `+{"**/test/**", "**/.git/**"}+` will be included for code scan. In this case, you should declare your own exclude list in the config json.
When set to `true` then default exclude folders `+{"**/test/**", "**/.git/**", "**/node_modules/**"}+` will be included for code scan. In this case, you should declare your own <<sechub-config-example-sourcescan,exclude list>> in the config json.
- `SECHUB_INITIAL_WAIT_INTERVAL` +
Initial wait time (floating point number) until SecHub client polls the first time if the report is ready (`sechub scan`). Meant for local integration tests.
- `SECHUB_KEEP_TEMPFILES` +
Expand All @@ -361,8 +361,8 @@ Examples (Linux/Mac):
[[section-client-configuration-file]]

=== Client configuration file
If not defined by option `-configfile`, secHub client will attempt to load `sechub.json` file from current directory
and use it as sechub configuration.
If not defined by option `-configfile`, {secHub} client will attempt to load `sechub.json` file from current directory
and use it as configuration.

For scan actions, a valid configuration file is needed. Other actions like getReport or false-positives handling need no configuration file.

Expand Down Expand Up @@ -393,7 +393,7 @@ The client will per default only include wellknown source files.
This is done by inspecting the filename ending. The accepted defaults are shown in next table.

*All files having file endings not listed here, will be ignored* and not contained in source upload to
{sechub} server!
{sechub} server!

NOTE: But of course you can also define additional source file endings - see <<sechub-config-example-sourcescan,config file example>> .

Expand Down Expand Up @@ -443,15 +443,15 @@ with corresponding `sechub.json` config file: <<sechub-config-example-infrascan-
SECHUB_APITOKEN=7536a8c4aa82407da7e06bdbEXAMPLE
sechub -file mark-fp.json markFalsePositives
----
with a corresponding `mark-fp.json` - see <<section-client-false-positives-mark,Mark false positives JSON format>>
with a corresponding `mark-fp.json` - see <<section-client-false-positives-mark,Mark false positives JSON format>>

==== Unmark false positives
[source, bash]
----
SECHUB_APITOKEN=7536a8c4aa82407da7e06bdbEXAMPLE
sechub -file mark-fp.json unmarkFalsePositives
----
with a corresponding `mark-fp.json` - see <<section-client-false-positives-unmark,Unmark false positives JSON format>>
with a corresponding `mark-fp.json` - see <<section-client-false-positives-unmark,Unmark false positives JSON format>>

[[section-client-interactiveMarkFalsePositives-mark]]
===== interactiveMarkFalsePositives
Expand All @@ -460,9 +460,9 @@ with a corresponding `mark-fp.json` - see <<section-client-false-positives-unmar
SECHUB_APITOKEN=7536a8c4aa82407da7e06bdbEXAMPLE
sechub interactiveMarkFalsePositives

/ ___| | | | | | |
\ `--. ___ ___| |_| |_ _| |__
`--. \/ _ \/ __| _ | | | | '_ \
/ ___| | | | | | |
\ `--. ___ ___| |_| |_ _| |__
`--. \/ _ \/ __| _ | | | | '_ \
/\__/ / __/ (__| | | | |_| | |_) |
\____/ \___|\___\_| |_/\__,_|_.__/ Client Version 0.29.0-20220401111823

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,48 +6,48 @@
== Security Products

TIP: You must only setup those products you have listed inside your
profiles. See <<section-initial-profile-and-executors,execution profiles and executor configurations>>
profiles. See <<section-initial-profile-and-executors,execution profiles and executor configurations>>

=== {pds}
==== General
The {pds} adapters can communicate with the {pds} instances which delegates to the underlying
security products.
The {pds} adapters can communicate with the {pds} instances which delegates to the underlying
security products.

Via {pds} every security product, even simple CLI tools, can be easily integrated
into {sechub}.

Please look into `Product delegation server` documentation for more information about the {pds} in common.

==== PDS solutions
There are already ready to use {pds} implementations available - we call them "{pds} solutions".

==== PDS solutions
There are already ready to use {pds} implementations available - we call them "{pds} solutions".

Every {pds} solution does work out of the box and provide the possibility to start in following three flavors:

- docker (single instance)
- docker compose (cluster)
- docker (single instance)
- docker compose (cluster)
- helm charts (k8s cluster)

The next sub chapters list the current implementations:

===== GoSec
The PDS integration of https://securego.io[GoSec] (SAST) can be found at
https://github.com/mercedes-benz/sechub/tree/develop/sechub-pds-solutions/gosec

===== Multi
This solution does integrate multiple SAST tools:

- https://github.com/PyCQA/bandit[Bandit]
- https://dwheeler.com/flawfinder/[FlawFinder]
- https://github.com/MobSF/mobsfscan[mobsfscan]
- https://dwheeler.com/flawfinder/[FlawFinder]
- https://github.com/MobSF/mobsfscan[mobsfscan]
- https://github.com/ajinabraham/njsscan[njsscan]

It can be found at https://github.com/mercedes-benz/sechub/tree/develop/sechub-pds-solutions/multi

===== Owasp-Zap
This solution does integrate the https://www.zaproxy.org/ (web application scanner) - please look at
This solution does integrate the https://www.zaproxy.org/ (web application scanner) - please look at
https://github.com/mercedes-benz/sechub/blob/develop/sechub-pds-solutions/owasp-zap/

===== PMD
https://pmd.github.io/ stand normally for quality checks, but it does also provide some security check mechanism. The integration can be found at
https://github.com/mercedes-benz/sechub/tree/develop/sechub-pds-solutions/pmd
Expand All @@ -72,24 +72,24 @@ include::../shared/adapter/checkmarx_adapter_flow.adoc[]
Please look at <<section-gen-config-scope-checkmarx,checkmarx config>> section.

====== Runtime
WARNING: Standard properties (like base URL, user id etc.) are currently NOT used from
existing product executor configurations but only the environment configuration is used at the moment.
WARNING: Standard properties (like base URL, user id etc.) are currently NOT used from
existing product executor configurations but only the environment configuration is used at the moment.

[options="header",cols="1,1,1"]
|===
|Parameter |Type |Description
|Parameter |Type |Description
//--------------------------------------------------
|checkmarx.fullscan.always |boolean |When 'true' every scan will be done as a full scan and delta scan is not used.
Use this only where a delta scan is not possible/ always rejected by checkmarx.
When 'false' or any other value, delta scan feature is used.
_If a scan fails because checkmarx has detected too many changes there will be a
retry with fullscan. If you have permanent problems with delta scanning for a
project you should set this option to 'true' in your checkmarx executor
configuration._
|checkmarx.fullscan.always |boolean |When 'true' every scan will be done as a full scan and delta scan is not used.
Use this only where a delta scan is not possible/ always rejected by checkmarx.
When 'false' or any other value, delta scan feature is used.
_If a scan fails because checkmarx has detected too many changes there will be a
retry with fullscan. If you have permanent problems with delta scanning for a
project you should set this option to 'true' in your checkmarx executor
configuration._
|===
==== Summary
Expand Down Expand Up @@ -137,5 +137,5 @@ include::../shared/adapter/nessus_adapter_flow.adoc[]
==== Summary
. At the moment only one {nessus} policy UUID is used
. Every scan has got its own name which starts with sechub job UUID
. Every scan has got its own name which starts with {secHub} job UUID
. Old scans are *NOT* automatically deleted
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ define a network policy, services, endpoints etc. etc.

[IMPORTANT]
====
Be aware to mount *only ONE pod* - the database is *shared by all sechub server
Be aware to mount *only ONE pod* - the database is *shared by all {secHub} server
PODs* _(means this is a shared kernel / bottle neck but necessary)_
====

Expand Down Expand Up @@ -46,7 +46,7 @@ define a network policy, services, endpoints etc.

[IMPORTANT]
====
Be aware to mount *only ONE pod* - the database is *shared by all sechub server
Be aware to mount *only ONE pod* - the database is *shared by all {secHub} server
PODs* _(means this is a shared kernel / bottle neck but necessary)_
====

Expand Down Expand Up @@ -158,7 +158,7 @@ accidently add the credentials to your repository.

*Logstash support*

When you want to use `logstash` (e.g. when using `ElasticSearch` or `OpenSearch`) you just have to set the environment variable
`LOGGING_TYPE` to `LOGSTASH_JSON` before server starts.
When you want to use `logstash` (e.g. when using `ElasticSearch` or `OpenSearch`) you just have to set the environment variable
`LOGGING_TYPE` to `LOGSTASH_JSON` before server starts.


Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
. All communication is done by REST API
. Login
. Ensure project exists (reuses existing project or creates new one)
.. projectname is sechub project name
.. projectname is {secHub} project name
.. teamId Is shared and always CHECKMARX installsetup team id for new projects...
. Upload source code from SecHub upload folder per REST to {checkmarx} server
. Start scan
Expand Down
Loading

0 comments on commit 395a3c2

Please sign in to comment.