Skip to content

Commit

Permalink
Add quality gate (anchore#88)
Browse files Browse the repository at this point in the history
* add plugins + various fixes

Signed-off-by: Alex Goodman <[email protected]>

* final adjustments while testing the integration with enterprise

Signed-off-by: Alex Goodman <[email protected]>

* add quality gate

Signed-off-by: Alex Goodman <[email protected]>

* update docs + deps

Signed-off-by: Alex Goodman <[email protected]>

* add quality gate workflows

Signed-off-by: Alex Goodman <[email protected]>

* fix linting

Signed-off-by: Alex Goodman <[email protected]>

* fix composite action shell steps

Signed-off-by: Alex Goodman <[email protected]>

* fix python version

Signed-off-by: Alex Goodman <[email protected]>

* register pr quality gate

Signed-off-by: Alex Goodman <[email protected]>

* remove description

Signed-off-by: Alex Goodman <[email protected]>

* add importlib-metadata dep

Signed-off-by: Alex Goodman <[email protected]>

* re-enable on push event

Signed-off-by: Alex Goodman <[email protected]>

* show fileset changes

Signed-off-by: Alex Goodman <[email protected]>

* bump yardstick to v0.3.1

Signed-off-by: Alex Goodman <[email protected]>

* change the base ref lookup for fileset changes

Signed-off-by: Alex Goodman <[email protected]>

* ensure diff is always relative to origin

Signed-off-by: Alex Goodman <[email protected]>

* remove bin dep for select-providers step

Signed-off-by: Alex Goodman <[email protected]>

* add fan in evaluation

Signed-off-by: Alex Goodman <[email protected]>

* max retries should result in failure

Signed-off-by: Alex Goodman <[email protected]>

* merge rhel and centos testing

Signed-off-by: Alex Goodman <[email protected]>

* tweak exp backoff settings

Signed-off-by: Alex Goodman <[email protected]>

* disable quality gate on push and move grype-db install to script

Signed-off-by: Alex Goodman <[email protected]>

* enable nightly quality gate check on release

Signed-off-by: Alex Goodman <[email protected]>

* add github, rhel, and other quality gate images

Signed-off-by: Alex Goodman <[email protected]>

* move oracle testing image to oracle section

Signed-off-by: Alex Goodman <[email protected]>

* add validation for quality gate test tool versions

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: Alex Goodman <[email protected]>
  • Loading branch information
wagoodman authored Feb 28, 2023
1 parent 96ee3f4 commit 7938f40
Showing 43 changed files with 2,503 additions and 538 deletions.
78 changes: 78 additions & 0 deletions .github/actions/bootstrap/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
name: "Bootstrap"
description: "Bootstrap all tools and dependencies"
inputs:
python-version:
description: "Python version to install"
required: true
default: "3.9"
poetry-version:
description: "Poetry version to install"
required: true
default: "1.3.2"
use-poetry-cache:
description: "Restore poetry cache"
required: true
default: "true"
tools:
description: "Bootstrap tooling"
required: true
default: "true"
cache-key-prefix:
description: "Prefix all cache keys with this value"
required: true
default: "831180ac25"
bootstrap-apt-packages:
description: "Space delimited list of tools to install via apt"
default: ""

runs:
using: "composite"
steps:

- uses: actions/setup-python@v4
with:
python-version: ${{ inputs.python-version }}

- name: Install poetry
uses: abatilo/[email protected]
with:
poetry-version: ${{ inputs.poetry-version }}

- name: Cache Poetry virtualenv
uses: actions/cache@v3
if: inputs.use-poetry-cache == 'true'
id: cache
with:
path: ~/.virtualenvs
key: ${{ inputs.cache-key-prefix }}-python-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}-${{ hashFiles('poetry.lock') }}
restore-keys: |
${{ inputs.cache-key-prefix }}-python-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}
- name: Setup Poetry config
shell: bash
run: |
poetry config virtualenvs.in-project false
poetry config virtualenvs.path ~/.virtualenvs
- name: Restore tool cache
id: tool-cache
if: inputs.tools == 'true'
uses: actions/cache@v3
with:
path: ${{ github.workspace }}/.tmp
key: ${{ inputs.cache-key-prefix }}-${{ runner.os }}-tool-${{ hashFiles('Makefile') }}

- name: (cache-miss) Bootstrap tools
if: steps.tool-cache.outputs.cache-hit != 'true' && inputs.tools == 'true'
shell: bash
run: make bootstrap

- name: Install apt packages
if: inputs.bootstrap-apt-packages != ''
shell: bash
run: |
DEBIAN_FRONTEND=noninteractive sudo apt update && sudo -E apt install -y ${{ inputs.bootstrap-apt-packages }}
- name: Install dependencies and package
shell: bash
run: poetry install
21 changes: 21 additions & 0 deletions .github/actions/quality-gate/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: "Quality Gate"
description: "Run quality gate for a given provider"
inputs:
provider:
description: "Provider to check"
required: true

runs:
using: "composite"
steps:
# assume we have python and poetry installed

- name: Capture vulnerability results
shell: bash
working-directory: tests/quality
run: poetry run make capture provider=${{ inputs.provider }}

- name: Validate provider results
shell: bash
working-directory: tests/quality
run: poetry run make validate
31 changes: 23 additions & 8 deletions .github/scripts/trigger-release.sh
Original file line number Diff line number Diff line change
@@ -4,30 +4,45 @@ set -eu
bold=$(tput bold)
normal=$(tput sgr0)

TEMP_DIR=.tmp
chronicle=$TEMP_DIR/chronicle
if ! [ -x "$(command -v gh)" ]; then
echo "The GitHub CLI could not be found. To continue follow the instructions at https://github.com/cli/cli#installation"
exit 1
fi

gh auth status

# we need all of the git state to determine the next version. Since tagging is done by
# the release pipeline it is possible to not have all of the tags from previous releases.
git fetch --tags

NEXT_VERSION=$($chronicle next-version)
# populates the CHANGELOG.md and VERSION files
echo "${bold}Generating changelog...${normal}"
make changelog 2> /dev/null

NEXT_VERSION=$(cat VERSION)

echo "${bold}Proposed version:${normal} $NEXT_VERSION"
make changelog
if [[ "$NEXT_VERSION" == "" || "${NEXT_VERSION}" == "(Unreleased)" ]]; then
echo "Could not determine the next version to release. Exiting..."
exit 1
fi

while true; do
read -p "${bold}Do you want to trigger a release with this version?${normal} [y/n] " yn
read -p "${bold}Do you want to trigger a release for version '${NEXT_VERSION}'?${normal} [y/n] " yn
case $yn in
[Yy]* ) echo; break;;
[Nn]* ) echo; echo "Cancelling release..."; exit;;
* ) echo "Please answer yes or no.";;
esac
done

echo "${bold}Kicking off release for $NEXT_VERSION${normal}..."
# TODO: kick off nightly quality gate if:... (could we do this in the release workflow instead?)
# ... it's not already running
# ... it's not already failed (use the check name)
# ... it's not already succeeded (use the check name)

echo "${bold}Kicking off release for ${NEXT_VERSION}${normal}..."
echo
gh workflow run release.yaml -f version=$NEXT_VERSION
gh workflow run release.yaml -f version=${NEXT_VERSION}

echo
echo "${bold}Waiting for release to start...${normal}"
71 changes: 71 additions & 0 deletions .github/workflows/nightly-quality-gate.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: "Nightly Quality Gate"
on:
# allow for kicking off quality gate check manually
workflow_dispatch:

# run 2 AM (UTC) daily
schedule:
- cron: '0 2 * * *'

jobs:

select-providers:
runs-on: ubuntu-20.04
outputs:
providers: ${{ steps.determine-providers.outputs.providers }}
steps:
- uses: actions/checkout@v3
with:
# in order to properly resolve the version from git
fetch-depth: 0

- name: Bootstrap environment
uses: ./.github/actions/bootstrap
with:
tools: false

- name: Determine providers
id: determine-providers
run: |
# select all providers as test subjects (this populates the matrix downstream)
content=`cd tests/quality && poetry run make all-providers`
echo $content
echo "providers=$content" >> $GITHUB_OUTPUT
validate-provider:
runs-on: ubuntu-20.04
needs: select-providers
strategy:
matrix:
provider: ${{fromJson(needs.select-providers.outputs.providers)}}
fail-fast: false
permissions:
contents: read
packages: read
steps:
- uses: actions/checkout@v3
with:
# in order to properly resolve the version from git
fetch-depth: 0

- name: Bootstrap environment
uses: ./.github/actions/bootstrap

- name: Run quality gate
uses: ./.github/actions/quality-gate
with:
provider: ${{ matrix.provider }}

# note: the name for this check is referenced in release.yaml, do not change here without changing there
Nightly-Quality-Gate:
runs-on: ubuntu-20.04
needs: validate-provider
if: ${{ always() && !cancelled() }}
steps:
- run: |
echo "Validations Status: ${{ needs.run-provider-validation.result }}"
if [ "${{ needs.run-provider-validation.result }}" == "failure" ]; then
echo "Quality gate failed!"
exit 1
fi
echo "Quality gate passed!"
75 changes: 75 additions & 0 deletions .github/workflows/pr-quality-gate.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
name: "PR Quality Gate"
on:
issue_comment:
types: [created]

jobs:

select-providers:
runs-on: ubuntu-20.04
if: github.event.issue.pull_request && contains(github.event.comment.body, '/run-gate')
outputs:
providers: ${{ steps.determine-providers.outputs.providers }}
steps:
- uses: actions/checkout@v3
with:
# in order to properly resolve the version from git
fetch-depth: 0
# checkout relative to the PR head
ref: refs/pull/${{ github.event.issue.number }}/head

- name: Bootstrap environment
uses: ./.github/actions/bootstrap
with:
tools: false

- name: Determine providers
id: determine-providers
run: |
# be nice to folks troubleshooting in CI...
cd tests/quality
poetry run make show-changes
# determine which providers to run (to later populate the matrix)
content=`poetry run make select-providers`
echo $content
echo "providers=$content" >> $GITHUB_OUTPUT
validate-provider:
runs-on: ubuntu-20.04
needs: select-providers
strategy:
matrix:
provider: ${{fromJson(needs.select-providers.outputs.providers)}}
fail-fast: false
permissions:
contents: read
packages: read
steps:
- uses: actions/checkout@v3
with:
# in order to properly resolve the version from git
fetch-depth: 0
# checkout relative to the PR head
ref: refs/pull/${{ github.event.issue.number }}/head

- name: Bootstrap environment
uses: ./.github/actions/bootstrap

- name: Run quality gate
uses: ./.github/actions/quality-gate
with:
provider: ${{ matrix.provider }}

evaluate-quality-gate:
runs-on: ubuntu-20.04
needs: validate-provider
if: ${{ always() && !cancelled() }}
steps:
- run: |
echo "Validations Status: ${{ needs.run-provider-validation.result }}"
if [ "${{ needs.run-provider-validation.result }}" == "failure" ]; then
echo "Quality gate failed!"
exit 1
fi
echo "Quality gate passed!"
64 changes: 20 additions & 44 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
@@ -6,11 +6,8 @@ on:
description: tag the latest commit on main with the given version (prefixed with v)
required: true

env:
PYTHON_VERSION: "3.9"
POETRY_VERSION: "1.3.2"

jobs:

quality-gate:
runs-on: ubuntu-20.04
steps:
@@ -37,15 +34,28 @@ jobs:
checkName: "Validations"
ref: ${{ github.event.pull_request.head.sha || github.sha }}

- name: Quality gate
if: steps.validations.conclusion != 'success'
- name: Check nightly quality gate results
uses: fountainhead/[email protected]
id: nightly-quality-gate
with:
token: ${{ secrets.GITHUB_TOKEN }}
# This check name is defined as the github action job name (in .github/workflows/nightly-quality-gate.yaml)
checkName: "Nightly-Quality-Gate"
ref: ${{ github.event.pull_request.head.sha || github.sha }}
# If there is no result in 10 seconds, assume it hasn't run yet
timeoutSeconds: 10
intervalSeconds: 3

- name: Release quality gate
if: steps.validations.conclusion != 'success' || steps.nightly-quality-gate.conclusion != 'success'
run: |
echo "Validations Status: ${{ steps.validations.conclusion }}"
echo "Nightly Quality Gate Status: ${{ steps.nightly-quality-gate.conclusion }}"
false
release:
needs: [quality-gate]
needs:
- quality-gate
runs-on: ubuntu-20.04
environment: release
permissions:
@@ -59,42 +69,8 @@ jobs:
# in order to properly resolve the version from git
fetch-depth: 0

- uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install poetry
uses: abatilo/[email protected]
with:
poetry-version: ${{ env.POETRY_VERSION }}

- name: Cache Poetry virtualenv
uses: actions/cache@v3
id: cache
with:
path: ~/.virtualenvs
key: poetry-${{ hashFiles('poetry.lock') }}
restore-keys: |
poetry-${{ hashFiles('poetry.lock') }}
- name: Setup Poetry config
run: |
poetry config virtualenvs.in-project false
poetry config virtualenvs.path ~/.virtualenvs
- name: Restore tool cache
id: tool-cache
uses: actions/cache@v3
with:
path: ${{ github.workspace }}/.tmp
key: ${{ runner.os }}-tool-${{ hashFiles('Makefile') }}

- name: (cache-miss) Bootstrap tools
if: steps.tool-cache.outputs.cache-hit != 'true'
run: make bootstrap

- name: Install dependencies and package
run: poetry install
- name: Bootstrap environment
uses: ./.github/actions/bootstrap

- name: Login to ghcr.io
run: |
Loading

0 comments on commit 7938f40

Please sign in to comment.