Skip to content

Commit

Permalink
Fix benchmark PR check (risc0#1711)
Browse files Browse the repository at this point in the history
* Save results to S3 bucket, but only if pushing to main or a release
branch.
* Compare results from S3 bucket, based on current target branch.
  • Loading branch information
flaub authored Apr 24, 2024
1 parent dcf25f8 commit cbe7b8a
Showing 1 changed file with 13 additions and 12 deletions.
25 changes: 13 additions & 12 deletions .github/workflows/bench_pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ jobs:
env:
FEATURE: ${{ matrix.feature }}
DISABLE_S3: ${{ matrix.device == 'nvidia_rtx_3090_ti' }}
S3_BENCH_PATH: s3://risc0-ci-cache/public/bench/${{ github.base_ref || github.ref_name }}/fib/${{ matrix.os }}-${{ matrix.device }}

steps:
- uses: actions/checkout@v4
Expand All @@ -55,33 +56,33 @@ jobs:
key: ${{ matrix.os }}-${{ matrix.feature }}
disable_s3: ${{ env.DISABLE_S3 }}

- id: aws-creds
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: us-west-2
role-to-assume: arn:aws:iam::083632199359:role/gha_oidc_risc0_cache_public_access

- name: Download previous benchmark data
run: aws s3 cp $S3_BENCH_PATH/external.json ./cache/external.json

- run: cargo run --bin cargo-risczero --no-default-features -- risczero install --version $RISC0_TOOLCHAIN_VERSION

- name: Run benchmarks
run: cargo bench --bench fib -F $FEATURE

- name: Download previous benchmark data
uses: actions/cache/restore@v4
with:
path: ./cache
key: fib-${{ runner.os }}-${{ matrix.device }}

- name: Analyze benchmark results
uses: risc0/[email protected]
with:
name: "${{ matrix.os }}-${{ matrix.device }}"
tool: 'customBiggerIsBetter'
output-file-path: target/hotbench/fib/benchmark.json
external-data-json-path: ./cache/benchmark.json
external-data-json-path: ./cache/external.json
github-token: ${{ secrets.GITHUB_TOKEN }}
alert-threshold: '120%'
comment-on-alert: true
fail-on-alert: true
summary-always: true

- name: Save results
- name: Save current benchmark data
if: github.event_name == 'push'
uses: actions/cache/save@v4
with:
path: ./cache
key: fib-${{ runner.os }}-${{ matrix.device }}
run: aws s3 cp ./cache/external.json $S3_BENCH_PATH/external.json

0 comments on commit cbe7b8a

Please sign in to comment.