forked from desmos-labs/desmos
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
build(ci): add mutation testing (desmos-labs#1061)
## Description This PR adds mutation testing ci. Closes: #XXXX --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/desmos-labs/desmos/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://docs.cosmos.network/v0.44/building-modules/intro.html) - [ ] included the necessary unit and integration [tests](https://github.com/desmos-labs/desmos/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
- Loading branch information
Showing
7 changed files
with
133 additions
and
0 deletions.
There are no files selected for viewing
6 changes: 6 additions & 0 deletions
6
.changeset/entries/43407cab0ad8588c2ad460e3263a18402e65f441fdd5abe33d7c1be740faf0df.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| type: ci | ||
| module: other | ||
| pull_request: 1061 | ||
| description: Added mutation testing | ||
| backward_compatible: true | ||
| date: 2023-01-04T05:13:54.9256337Z |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| name: Generate Mutation Test Errors | ||
|
|
||
| on: | ||
| tags: | ||
| - "v[0-9]+.[0-9]+.[0-9]+" # Push events to matching v*, i.e. v1.0, v20.15.10 | ||
| - "v[0-9]+.[0-9]+.[0-9]+-rc*" # Push events to matching v*, i.e. v1.0-rc1, v20.15.10-rc5 | ||
|
|
||
| jobs: | ||
| Test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout 🛎️ | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Setup Go 🧰 | ||
| uses: actions/setup-go@v3 | ||
| with: | ||
| go-version: 1.18 | ||
|
|
||
| - name: Run mutation tests 🧪 | ||
| continue-on-error: true | ||
| run: make test-mutation $MODULES | ||
| env: | ||
| MODULES: fees,profiles,subspaces,relationships,posts,reports,reactions | ||
|
|
||
| - name: Format output 🔍 | ||
| id: mutest-formatted | ||
| run: | | ||
| cat mutation_test_result.txt | grep -Ev "PASS" | grep -Ev "SKIP" | tee mutation_test_result.txt | ||
| - name: Generate code blocks 🧑💻 | ||
| id: gen-code-blocks | ||
| run: | | ||
| cat mutation_test_result.txt | sed "s# @@# @@\n\`\`\`go\n#g " | sed "s#FAIL#\`\`\`\nFAIL\n\n\n#g " > go_mutation_test_result.txt | ||
| - name: Read output file 👀 | ||
| id: result | ||
| uses: juliangruber/read-file-action@v1 | ||
| with: | ||
| path: go_mutation_test_result.txt | ||
|
|
||
| - name: Create issue 🚩 | ||
| uses: dacbd/create-issue-action@main | ||
| with: | ||
| token: ${{ secrets.GITHUB_TOKEN }} | ||
| title: Mutation test ${{ steps.vars.outputs.tag }} | ||
| body: ${{ steps.result.outputs.content }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -eo pipefail | ||
| oIFS="$IFS"; IFS=, ; set -- $1 ; IFS="$oIFS" | ||
|
|
||
| DISABLED_MUTATORS='branch/*' | ||
|
|
||
| # Only consider the following: | ||
| # * go files in types, keeper, or module root directories | ||
| # * ignore test and Protobuf files | ||
| # * ignore simulation test files | ||
| # * ignore legacy files | ||
| go_file_exclusions="-type f ! -path */client/* ! -path */simulation/* ! -path */legacy/* -name *.go -and -not -name *_test.go -and -not -name *pb* -and -not -name module.go -and -not -name ibc_module.go" | ||
| MUTATION_SOURCES=$(find ./x $go_file_exclusions ) | ||
|
|
||
| # Filter on a module-by-module basis as provided by input | ||
| arg_len=$# | ||
|
|
||
| for i in "$@"; do | ||
| if [ $arg_len -gt 1 ]; then | ||
| MODULE_FORMAT+="./x/$i\|" | ||
| MODULE_NAMES+="${i} " | ||
| let "arg_len--" | ||
| else | ||
| MODULE_FORMAT+="./x/$i" | ||
| MODULE_NAMES+="${i}" | ||
| fi | ||
| done | ||
|
|
||
| MUTATION_SOURCES=$(echo "$MUTATION_SOURCES" | grep "$MODULE_FORMAT") | ||
|
|
||
| # Collect multiple lines into a single line to be fed into go-mutesting | ||
| MUTATION_SOURCES=$(echo "$MUTATION_SOURCES" | tr '\n' ' ') | ||
|
|
||
| echo "################################################################################" | ||
| echo "### WARNING! This test will take hours to complete!" | ||
| echo "################################################################################" | ||
|
|
||
| echo "running mutation tests for the following module(s): $MODULE_NAMES" | ||
| OUTPUT=$(go run github.com/osmosis-labs/go-mutesting/cmd/go-mutesting --disable=$DISABLED_MUTATORS $MUTATION_SOURCES | tee /dev/tty) | ||
|
|
||
| # Fetch the final result output and the overall mutation testing score | ||
| RESULT=$(echo "$OUTPUT" | grep 'The mutation score') | ||
| SCORE=$(echo "$RESULT" | grep -Eo '[[:digit:]]\.[[:digit:]]+') | ||
|
|
||
| echo "writing mutation test result to mutation_test_result.txt" | ||
| echo "$OUTPUT" > mutation_test_result.txt | ||
|
|
||
| # Print the mutation score breakdown | ||
| echo $RESULT | ||
|
|
||
| # Return a non-zero exit code if the score is below 75% | ||
| if (( $(echo "$SCORE < 0.75" |bc -l) )); then | ||
| echo "Mutation testing score below desired level ($SCORE < 0.75)" | ||
| exit 1 | ||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters