feat(secret): support enable rules (aquasecurity#2035)

Co-authored-by: VaismanLior <[email protected]>

.github/workflows/mkdocs-dev.yaml

@@ -22,8 +22,7 @@ jobs:
       - name: Install dependencies
         run: |
           pip install git+https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git
-          pip install mike
-          pip install mkdocs-macros-plugin
+          pip install -r docs/build/requirements.txt
         env:
           GH_TOKEN: ${{ secrets.MKDOCS_AQUA_BOT }}
       - name: Configure the git user

.github/workflows/mkdocs-latest.yaml

@@ -24,8 +24,7 @@ jobs:
       - name: Install dependencies
         run: |
           pip install git+https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git
-          pip install mike
-          pip install mkdocs-macros-plugin
+          pip install -r docs/build/requirements.txt
         env:
           GH_TOKEN: ${{ secrets.MKDOCS_AQUA_BOT }}
       - name: Configure the git user

.github/workflows/test.yaml

@@ -92,9 +92,7 @@ jobs:
         python-version: 3.x
     - name: Install dependencies
       run: |
-        pip install mkdocs-material
-        pip install mike
-        pip install mkdocs-macros-plugin
+        pip install -r docs/build/requirements.txt
     - name: Configure the git user
       run: |
         git config user.name "knqyf263"

README.md

@@ -22,24 +22,31 @@ Scanner for vulnerabilities in container images, file systems, and Git repositor
 `Trivy` (`tri` pronounced like **tri**gger, `vy` pronounced like en**vy**) is a simple and comprehensive scanner for vulnerabilities in container images, file systems, and Git repositories, as well as for configuration issues.
 `Trivy` detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and language-specific packages (Bundler, Composer, npm, yarn, etc.).
 In addition, `Trivy` scans Infrastructure as Code (IaC) files such as Terraform, Dockerfile and Kubernetes, to detect potential configuration issues that expose your deployments to the risk of attack.
+`Trivy` also scans hardcoded secrets like passwords, API keys and tokens.
 `Trivy` is easy to use. Just install the binary and you're ready to scan.
 
-<p align="center">
+<figure style="text-aligh: center">
   <img src="docs/imgs/overview.png" width="800" alt="Trivy Overview">
-</p>
+</figure>
 
 ### Demo: Vulnerability Detection (Container Image)
-<p align="center">
+<figure style="text-aligh: center">
   <img src="docs/imgs/vuln-demo.gif" width="1000" alt="Vulnerability Detection">
-</p>
+</figure>
 
 ### Demo: Misconfiguration Detection (IaC Files)
-<p align="center">
+<figure style="text-aligh: center">
   <img src="docs/imgs/misconf-demo.gif" width="1000" alt="Misconfiguration Detection">
-</p>
+</figure>
+
+### Demo: Secret Detection
+<figure style="text-aligh: center">
+  <img src="docs/imgs/secret-demo.gif" width="1000">
+</figure>
+
 
 # Quick Start
-## Scan Image for Vulnerabilities
+## Scan Image for Vulnerabilities and Secrets
 Simply specify an image name (and a tag).
 
 ```
@@ -72,17 +79,17 @@ Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)
 ```
 </details>
 
-## Scan Filesystem for Vulnerabilities and Misconfigurations
+## Scan Filesystem for Vulnerabilities, Secrets and Misconfigurations
 Simply specify a directory to scan.
 
 ```bash
-$ trivy fs --security-checks vuln,config [YOUR_PROJECT_DIR]
+$ trivy fs --security-checks vuln,secret,config [YOUR_PROJECT_DIR]
 ```
 
 For example:
 
 ```bash
-$ trivy fs --security-checks vuln,config myproject/
+$ trivy fs --security-checks vuln,secret,config myproject/
 ```
 
 <details>
@@ -171,6 +178,10 @@ Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)
   - A wide variety of built-in policies are provided **out of the box**
     - Kubernetes, Docker, Terraform, and more coming soon
   - Support custom policies
+- Secret detection
+  - A wide variety of built-in rules are provided **out of the box**
+  - Support custom rules
+  - Scan container images at high speed
 - Simple
   - Specify only an image name, a path to config files, or an artifact name
 - Fast

docs/build/Dockerfile

@@ -1,9 +1,10 @@
-FROM squidfunk/mkdocs-material:7.0.6
+FROM squidfunk/mkdocs-material:8.2.10
 
 ## If you want to see exactly the same version as is published to GitHub pages
 ## use a private image for insiders, which requires authentication.
 
 # docker login -u ${GITHUB_USERNAME} -p ${GITHUB_TOKEN} ghcr.io
 # FROM ghcr.io/squidfunk/mkdocs-material-insiders
 
-RUN pip install mike mkdocs-macros-plugin
+COPY requirements.txt .
+RUN pip install -r requirements.txt

docs/build/requirements.txt

@@ -0,0 +1,30 @@
+click==8.1.2
+csscompressor==0.9.5
+ghp-import==2.0.2
+htmlmin==0.1.12
+importlib-metadata==4.11.3
+Jinja2==3.1.1
+jsmin==3.0.1
+Markdown==3.3.6
+MarkupSafe==2.1.1
+mergedeep==1.3.4
+mike==1.1.2
+mkdocs==1.3.0
+mkdocs-macros-plugin==0.7.0
+mkdocs-material==8.2.10
+mkdocs-material-extensions==1.0.3
+mkdocs-minify-plugin==0.5.0
+mkdocs-redirects==1.0.4
+packaging==21.3
+Pygments==2.11.2
+pymdown-extensions==9.3
+pyparsing==3.0.8
+python-dateutil==2.8.2
+PyYAML==6.0
+pyyaml-env-tag==0.1
+six==1.16.0
+termcolor==1.1.0
+verspec==0.1.0
+watchdog==2.1.7
+zipp==3.8.0
+

docs/docs/secret/configuration.md

@@ -1,7 +1,9 @@
 # Configuration
 Trivy tries to load `trivy-secret.yaml` in the current directory by default.
-If the file doesn't exist, only builtin rules are used.
-You can customize the config file name via the `--secret-config` flag.
+If the file doesn't exist, only built-in rules are used.
+You can customize the config file path via the `--secret-config` flag.
+
+You can see the example [here][examples].
 
 ## Custom Rules
 Trivy allows defining custom rules. You can see an example.
@@ -95,15 +97,37 @@ allow-rules:
 :   - Golang regular expression used to allow matched paths.
     - `regex` or `path` must be specified.
 
+## Enable Rules
+Trivy provides plenty of out-of-box rules and allow rules, but you may not need all of them.
+In that case, `enable-builin-rules` will be helpful.
+If you just need AWS secret detection, you can enable only relevant rules as shown below.
+It specifies AWS-related rule IDs in `enable-builin-rules`.
+All other rules are disabled, so the scanning will be much faster.
+We would strongly recommend using this option if you don't need all rules.
+
+You can see a full list of [built-in rule IDs][builtin] and [built-in allow rule IDs][builtin-allow].
+
+``` yaml
+enable-builtin-rules:
+  - aws-access-key-id
+  - aws-account-id
+  - aws-secret-access-key
+```
+
 ## Disable Rules
-Trivy offers builtin rules and allow rules, but you may want to disable some of them.
+Trivy offers built-in rules and allow rules, but you may want to disable some of them.
 For example, you don't use Slack, so Slack doesn't have to be scanned.
-You can specify `slack-access-token` and `slack-web-hook` in `disable-rules` so that those rules will be disabled for less false positives.
+You can specify the Slack rule IDs, `slack-access-token` and `slack-web-hook` in `disable-rules` so that those rules will be disabled for less false positives.
 
+You should specify either `enable-builin-rules` or `disable-rules`.
+If they both are specified, `disable-rules` takes precedence.
+In case `github-pat` is specified in `enable-builin-rules` and `disable-rules`, it will be disabled.
+
+In addition, there are some allow rules.
 Markdown files are ignored by default, but you may want to scan markdown files as well.
-You can disable the allow list by adding `markdown` to `disable-allow-rules`.
+You can disable the allow rule by adding `markdown` to `disable-allow-rules`.
 
-You can see a full list of rule IDs [here][builtin]. Allow rule IDs are below in the file.
+You can see a full list of [built-in rule IDs][builtin] and [built-in allow rule IDs][builtin-allow].
 
 ``` yaml
 disable-rules:
@@ -113,29 +137,7 @@ disable-allow-rules:
   - markdown
 ```
 
-## Example
-
-``` yaml
-$ cat trivy-secret.yaml
-rules:
-  - id: rule1
-    category: general
-    title: Generic Rule
-    severity: HIGH
-    regex: (?i)(?P<key>(secret))(=|:).{0,5}['"](?P<secret>[0-9a-zA-Z\-_=]{8,64})['"]
-    allow-rules:
-      - id: skip-text
-        description: skip text files
-        path: .*\.txt
-allow-rules:
-  - id: social-security-number
-    description: skip social security number
-    regex: 219-09-9999
-disable-rules:
-  - slack-access-token
-  - slack-web-hook
-disable-allow-rules:
-  - markdown
-```
 
-[builtin]: https://github.com/aquasecurity/fanal/blob/main/secret/builtin.go
+[builtin]: https://github.com/aquasecurity/fanal/blob/main/secret/builtin-rules.go
+[builtin]: https://github.com/aquasecurity/fanal/blob/main/secret/builtin-allow-rules.go
+[examples]: ./examples.md

docs/docs/secret/examples.md

@@ -0,0 +1,98 @@
+# Examples
+Also see [quick start][quick-start].
+
+## Skip Directories
+Trivy traversals directories and scans all files except those matching the built-in allow rules by default.
+If your have a lot of files in your container image or project, the scanning takes time.
+To make it faster, you can skip traversal in the specific directory.
+Also, it would be helpful if your project contains secrets and certificates for testing.
+
+``` shell
+$ trivy image --skip-dirs /var/lib --skip-dirs /var/log YOUR_IMAGE
+```
+
+``` shell
+$ trivy fs --skip-dirs ./my-test-dir --skip-dirs ./my-testing-cert/ /path/to/your_project
+```
+
+`--skip-fles` also works similarly.
+
+## Filter by severity
+
+Use `--severity` option.
+
+``` shell
+$ trivy fs --severity CRITICAL ./
+
+app/secret.sh (secrets)
+=======================
+Total: 1 (CRITICAL: 1)
+
++----------+-------------------+----------+---------+--------------------------------+
+| CATEGORY |    DESCRIPTION    | SEVERITY | LINE NO |             MATCH              |
++----------+-------------------+----------+---------+--------------------------------+
+|   AWS    | AWS Access Key ID | CRITICAL |   10    | export AWS_ACCESS_KEY_ID=***** |
++----------+-------------------+----------+---------+--------------------------------+
+```
+
+## Disable secret scanning
+If you need vulnerability scanning only, you can disable secret scanning via the `--security-checks` flag.
+
+``` shell
+$ trivy image --security-checks vuln alpine:3.15
+```
+
+## With configuration file
+`trivy-secret.yaml` in the working directory is loaded by default.
+
+``` yaml
+$ cat trivy-secret.yaml
+rules:
+  - id: rule1
+    category: general
+    title: Generic Rule
+    severity: HIGH
+    regex: (?i)(?P<key>(secret))(=|:).{0,5}['"](?P<secret>[0-9a-zA-Z\-_=]{8,64})['"]
+allow-rules:
+  - id: social-security-number
+    description: skip social security number
+    regex: 219-09-9999
+  - id: log-dir
+    description: skip log directory
+    path: ^\/var\/log\/
+disable-rules:
+  - slack-access-token
+  - slack-web-hook
+disable-allow-rules:
+  - markdown
+
+# The following command automatically loads the above configuration.
+$ trivy image YOUR_IMAGE
+```
+
+Also, you can customize the config file path via `--secret-config`.
+
+``` yaml
+$ cat ./secret-config/trivy.yaml
+rules:
+  - id: rule1
+    category: general
+    title: Generic Rule
+    severity: HIGH
+    regex: (?i)(?P<key>(secret))(=|:).{0,5}['"](?P<secret>[0-9a-zA-Z\-_=]{8,64})['"]
+    allow-rules:
+      - id: skip-text
+        description: skip text files
+        path: .*\.txt
+enable-builtin-rules:
+  - aws-access-key-id
+  - aws-account-id
+  - aws-secret-access-key
+disable-allow-rules:
+  - usr-dirs
+
+# Pass the above config with `--secret-config`.
+$ trivy fs --secret-config ./secret-config/trivy.yaml /path/to/your_project
+```
+
+[quick-start]: ./scanning.md#quick-start

docs/docs/secret/scanning.md

@@ -12,8 +12,11 @@ Trivy will scan every plaintext file, according to builtin rules or configuratio
 - Slack access token
 - etc.
 
-You can see a full list of builtin rules [here][builtin].
+You can see a full list of [built-in rules][builtin] and [built-in allow rules][builtin-allow].
 
+!!! tip
+    If your secret is not detected properly, please make sure that your file including the secret is not in [the allowed paths][builtin-allow].
+    You can disable allow rules via [disable-allow-rules][disable-rules].
 
 ## Quick start
 This section shows how to scan secrets in container image and filesystem. Other subcommands should be the same.
@@ -92,9 +95,30 @@ If you don't need secret scanning, you can disable it via the `--security-checks
 $ trivy image --security-checks vuln alpine:3.15
 ```
 
+## Recommendation
+We would recommend specifying `--skip-dirs` for faster secret scanning.
+In container image scanning, Trivy walks the file tree rooted  `/` and scans all the files other than [built-in allowed paths][builtin-allow].
+It will take a while if your image contains a lot of files even though Trivy tries to avoid scanning layers from a base image.
+If you want to make scanning faster, `--skip-dirs` and `--skip-files` helps so that Trivy will skip scanning those files and directories.
+The usage examples are [here][examples].
+
+`allow-rules` is also helpful. See the [allow-rules][allow-rules] section.
+
+In addition, all the built-in rules are enabled by default, so it takes some time to scan all of them.
+If you don't need all those rules, you can use `enable-builtin-rules` or `disable-rules` in the configuration file.
+You should use `enable-builin-rules` if you need only AWS secret detection, for example.
+All rules are disabled except for the ones you specify, so it runs very fast.
+On the other hand, you should use `disable-rules` if you just want to disable some built-in rules.
+See the [enable-rules][enable-rules] and [disable-rules][disable-rules] sections for the detail.
+
 ## Credit
 This feature is inspired by [gitleaks][gitleaks]. 
 
-[builtin]: https://github.com/aquasecurity/fanal/blob/main/secret/builtin.go
+[builtin]: https://github.com/aquasecurity/fanal/blob/main/secret/builtin-rules.go
+[builtin-allow]: https://github.com/aquasecurity/fanal/blob/main/secret/builtin-allow-rules.go
 [configuration]: ./configuration.md
+[allow-rules]: ./configuration.md#allow-rules
+[enable-rules]: ./configuration.md#enable-rules
+[disable-rules]: ./configuration.md#disable-rules
+[examples]: ./examples.md
 [gitleaks]: https://github.com/zricethezav/gitleaks

docs/index.md

@@ -9,7 +9,7 @@ hide:
 `Trivy` (`tri` pronounced like **tri**gger, `vy` pronounced like en**vy**) is a simple and comprehensive [vulnerability][vulnerability]/[misconfiguration][misconf]/[secret][secret] scanner for containers and other artifacts.
 `Trivy` detects vulnerabilities of [OS packages][os] (Alpine, RHEL, CentOS, etc.) and [language-specific packages][lang] (Bundler, Composer, npm, yarn, etc.).
 In addition, `Trivy` scans [Infrastructure as Code (IaC) files][iac] such as Terraform and Kubernetes, to detect potential configuration issues that expose your deployments to the risk of attack.
-`Trivy` also scans [hardcoded secrets][secret] like passwords, api keys, and tokens.
+`Trivy` also scans [hardcoded secrets][secret] like passwords, API keys and tokens.
 `Trivy` is easy to use. Just install the binary and you're ready to scan.
 All you need to do for scanning is to specify a target such as an image name of the container.