Add PR benchmarking #12
Workflow file for this run
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| --- | |
| name: Benchmarks on AMD64 | |
| permissions: read-all | |
| on: [pull_request] | |
| jobs: | |
| benchmark-pull-request: | |
| runs-on: ubuntu-latest-8-cores | |
| steps: | |
| - uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4.1.4 | |
| with: | |
| fetch-depth: 0 | |
| - id: goversion | |
| run: echo "goversion=$(cat .go-version)" >> "$GITHUB_OUTPUT" | |
| - uses: actions/setup-go@0c52d547c9bc32b1aa3301fd7a9cb496313a4491 # v5.0.0 | |
| with: | |
| go-version: ${{ steps.goversion.outputs.goversion }} | |
| - name: Run Benchmarks | |
| run: | | |
| BENCHSTAT_OUTPUT_FILE=result.txt make test-benchmark-compare REF=${{ github.event.pull_request.base.sha }} | |
| - run: | | |
| echo "\`\`\`" >> "$GITHUB_STEP_SUMMARY" | |
| cat result.txt >> "$GITHUB_STEP_SUMMARY" | |
| echo "\`\`\`" >> "$GITHUB_STEP_SUMMARY" | |
| cat <<EOL >> "$GITHUB_STEP_SUMMARY" | |
| <hr /> | |
| The table shows the median and 90% confidence interval (CI) summaries for each benchmark comparing the HEAD and the BASE of the pull request, and an A/B comparison under "vs base". The last column shows the statistical p-value with ten runs (n=10). | |
| The last row has the Geometric Mean (geomean) for the given rows in the table. | |
| Refer to [benchstat's documentation](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat) for more help. | |
| EOL | |
| - name: Validate results under acceptable limit | |
| run: | | |
| export MAX_ACCEPTABLE_DIFFERENCE=5 | |
| while IFS= read -r line; do | |
| # Get fourth value, which is the comparison with the base. | |
| value="$(echo "$line" | awk '{print $4}')" | |
| if [[ "$value" = +* ]] || [[ "$value" = -* ]]; then | |
| if (( $(echo "${value//[^0-9.]/}"'>'"$MAX_ACCEPTABLE_DIFFERENCE" | bc -l) )); then | |
| echo "::error::$value is above the maximum acceptable difference ($MAX_ACCEPTABLE_DIFFERENCE)" | |
| exit 1 | |
| fi | |
| fi | |
| done < <(grep geomean result.txt) |