Cache benchmark results by base commit

[Bill Schnurr (bschnurr)]

Summary

• Compare each PR synthetic merge commit against the exact main commit used to create that merge.
• Cache validated base benchmark results by commit and hosted measurement profile.
• Keep dated, checked-in main benchmark results with commit metadata for historical charts.
• Publish results through artifacts, the job summary, and an updated bot comment.
• Report benchmark corpus or profile changes as successful but not comparable.

What is compared

The candidate is GitHub's synthetic merge commit, which combines the PR's current head with its current main base. The baseline is that same exact main base commit:

main commit B ----------------------------> baseline benchmark
           \
            + PR head H -> synthetic M ---> candidate benchmark

comparison: M (B + PR changes) versus B

This measures the performance impact the PR would have after merging. It does not compare the PR with its initial commit, benchmark the raw PR head alone, or compare against a moving newer main. If the PR head or base changes before trusted reporting validates the revisions, the report stops and /benchmark must be requested again.

Security and behavior

• /benchmark is restricted to maintainers and reruns an existing unprivileged pull_request workflow.
• The candidate workflow has read-only permissions and no cache access.
• Base benchmarking, cache restore/save, and reporting run separately under trusted default-branch workflow_run code.
• Trusted jobs check out only ${{ github.sha }}; candidate output crosses the boundary only as bounded, provenance-validated JSON.
• Cache restores require an exact key and validated revision/profile metadata.
• Newly measured base results can update the dated and latest baseline files only for same-repository PRs whose head has not changed.

Benchmark history

Checked-in results under build/benchmark/baselines/ include the source commit SHA, subject, and timestamp. build/benchmark/benchmark_history.ipynb graphs execution-time and peak-memory changes across those main commits and can export a static dashboard under docs/benchmark-results/.

Dashboard preview

The checked-in history currently contains one hosted main baseline, so these previews show the initial package measurements. Lines and commit-over-commit changes will appear as dated runs accumulate.

Execution time

Peak memory

After merge, the full dashboard will be available at https://microsoft.github.io/pyright/benchmark-results/.

Validation

• python -m unittest discover -s build/benchmark -p "test_*.py" (52 tests)
• Workflow YAML parsing and Prettier checks
• CodeQL, including Actions analysis