Skip to content

bench: one-pass release evidence with lower-noise engine comparisons - #72

Closed
chh-ay wants to merge 3 commits into
feat/array-arithmeticfrom
bench/evidence-protocol
Closed

chh-ay wants to merge 3 commits into
feat/array-arithmeticfrom
bench/evidence-protocol

Conversation

@chh-ay

@chh-ay chh-ay commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Scope

Make the performance evidence trustworthy and capturable for 0.5.0. Bench tooling and the evidence renderer only; no published package changes. Part of #59. Stacked on #71 (base feat/array-arithmetic).

Problems

  1. The page mixed capture vintages: render and data from July (47f1643), formula and XLSX from October. Each artifact's provenance stamp required a clean tree, but every capture writes a tracked artifact, so only the first capture of a session could stamp clean.
  2. "20 runs did not finish" gave no cause.
  3. The default-vs-full "paired median ratio" for identical work spanned 0.38–3.51 (p10–p90): each sample was one cold 1–3 ms fixture run.
  4. 0.5.0 work had no evidence: packed bulk loads, CSV import, large undo, paged residency.
  5. The documented regression check could not pass: the committed baseline pins harness file hashes and the runner (browser, Bun, Node, kernel), which have all changed.

Change

  • bench:release captures every artifact the page needs in one pass on a clean tree; capture outputs no longer count as dirty, so the whole set shares one commit. It ends by re-recording the machine- and harness-pinned regression baseline. It refuses a dirty tree and stops at the first failure.
  • One capture header for the page; mixed commits are flagged with a per-artifact table instead of hidden.
  • Render failures name engine, size, step, and error. The 20 failures are upstream: Handsontable 18 remove_row at 1,000,000 rows overflows the call stack in its own DataMap.filterData.
  • Engine comparison samples now warm up and aggregate ≥50 ms of repeated runs. Identical-work spread drops from 8.99× to 1.59× (p90/p10); rows above 1.5× are marked unsupported.
  • New "Core write path" section: packed 1M-cell setBlock, detailed-capture block, 200k×10 CSV import, 500k-cell clear + undo, paged revisit residency, 10k-row bridge insert, each with correctness checkpoints.
  • New "XLSX codec" section comparing the current codec with the pre-0.5.0 codec checkout.
  • The analysis-engine capture renders on its own when the paired capture is missing.
  • README and hints use machine-neutral guidance (idle machine, one pinned CPU).

Verification

  • bun test bench/test: 101 pass. bench:release:smoke writes 5 stamp-verified artifacts from one clean commit.
  • Layer gates: build, docs:check, api:check, typecheck, lint pass; packages 869 pass / 6 skip / 0 fail; Rust 147 / 223 pass.
  • The committed evidence page still shows the old artifacts as pending or mixed; the release capture on the final tree fills them in.

chh-ay added 2 commits October 6, 2026 15:14
…o their step

- `bench:release` captures every artifact the evidence page publishes, in
  order, from a clean tree; it stops at the first failure and rejects an
  artifact that does not stamp that commit. `bench:release:smoke` rehearses
  into `results/smoke/`, which is never published.
- Capture outputs under `bench/results` no longer count against provenance:
  one clean tree can stamp a whole evidence set with a single commit instead
  of one commit per artifact. The page now opens with that capture header and
  lists every vintage when the artifacts disagree.
- The matched engine protocol aggregates each sample over untimed warm-up
  fixtures until at least 50 ms of measured time and records the iteration
  count, so a paired ratio is no longer decided by one cold 2 ms run; the
  render evidence takes five samples with two warm-ups per round, and a
  recorded failure names the action, counter, prepare, or cleanup step that
  failed.
- New page sections: the core write path (packed block writes, CSV import,
  large restore, paged residency) and the XLSX codec capture; render failures
  are listed with their stage, recorded error, and cell count.
- The release capture re-records the regression baseline as its last step,
  because a harness, sampling, protocol, or runner change invalidates the
  committed fingerprint.
…gine capture alone

- The full XLSX capture compares against a prepared checkout of the
  pre-0.5.0 codec, so `bench:release` now requires
  `--xlsx-baseline-root <path>` up front and fails with the exact worktree
  and build commands instead of after the browser capture has already run.
- The default/full engine section renders the measured analysis capture
  (shared matrix, WASM sizes, Node cold init) even when the paired-ratio
  capture is absent, and names the capture that is missing instead of
  withholding the evidence that does exist.
@chh-ay
chh-ay added this pull request to stack #69 October 6, 2026 08:41
@chh-ay

chh-ay commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

Landed on develop in the 0.5.0 release squash: #112 (55cfa82).

@chh-ay chh-ay closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant