Skip to content

bench: judge each ratio by the upper end of its 95% interval - #196

Merged
tamnd merged 1 commit into
mainfrom
bench-interval
Oct 27, 2026
Merged

tamnd merged 1 commit into
mainfrom
bench-interval

Conversation

@tamnd

@tamnd tamnd commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Plan 12.2 says a benchmark is within 5% when the upper end of the 95% interval of its rucc/gcc median ratio is at most 1.05, and K8's exit criterion says the same. rk bench only looked at the ratio of the two medians, so one lucky run could pass a benchmark that is really slower.

Bench::interval resamples the runs of each kernel with replacement 2000 times, takes the ratio of the medians each time and keeps the 2.5th and 97.5th percentiles. The seed is fixed, so the same bench.json always gives the same interval. The gate now checks the upper end of each interval instead of the ratio. The geometric mean check stays on the medians, as the exit criterion has it. The table gets a "95% interval" column.

New test: equal runs give an interval of just the ratio, a spread of a few percent widens it past 1.05, and an empty bench has none. cargo test -p rk bench and clippy pass on server3.

Part of #9.

@tamnd tamnd added kind/feature New capability in the harness area/rk The rk command line labels Oct 27, 2026
@tamnd
tamnd merged commit 0d16a85 into main Oct 27, 2026
2 checks passed
@tamnd
tamnd deleted the bench-interval branch October 27, 2026 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/rk The rk command line kind/feature New capability in the harness

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant