diff --git a/.gitignore b/.gitignore index 81c810c1..ace1cd99 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,8 @@ playwright-report/ bench/results/candidates/ bench/results/render-fresh.json bench/results/render-fresh.md +# Release-capture rehearsals; never published evidence +bench/results/smoke/ .formula-evidence/ # Editor / OS diff --git a/bench/README.md b/bench/README.md index b8110ef6..c8d7d9b4 100644 --- a/bench/README.md +++ b/bench/README.md @@ -81,19 +81,32 @@ passed timing regression gate. The legacy shared source keys name the WASM role; their hashes use the selected engine. The outer artifact records the physical paths and binds each engine to its captured WASM hash. -Build the packages first. Run the capture from `bench/` with one pinned runner: +Build the packages first. Run one capture at a time from `bench/` on an +otherwise idle machine, pinned to one CPU. Do not run concurrent builds or +timing captures: competing work adds scheduler and CPU noise. For example, +on Linux with CPU 4: ```sh -flock -x /home/vvin/learn/sheetwrite-wt/build.lock flock -x /home/vvin/learn/sheetwrite-wt/quiet.lock taskset -c 4 bun run src/full-engine-bench.ts +taskset -c 4 bun run src/full-engine-bench.ts ``` `bench:formula:matched` checks the eight shared rows that were slower in the -first sequential capture. It runs 51 default/full pairs per row. Each timed -sample uses a fresh process and one untimed warmup. The artifact retains all -samples and checksums, median paired ratios, and the p10 to p90 ratio spread. -The single-sample runner is `bun run src/formula-bench.ts --sample ID SIZE`. +first sequential capture. It runs 25 default/full pairs per row (`--rounds` +changes that count, minimum 2). Each timed sample uses a fresh Bun process, +runs three untimed warm-up fixtures, then repeats the fixture until at least +50 ms of measured time and records the per-iteration mean. Repeating a short +operation is what makes the paired ratio usable: a single cold run of a 2 ms +workload drifted by more than 4x between fresh processes, while the aggregated +sample stays inside a few percent. The artifact retains all samples and +checksums, every sample's iteration count and total, median paired ratios, and +the p10 to p90 ratio spread; the evidence page marks any row whose spread is +too wide to support a ratio claim. +The single-sample runner is +`bun run src/formula-bench.ts --sample ID SIZE [--aggregate-ms 50] [--warmup-fixtures 3]`. It supports the matched workload set. Use the same pinned-runner command as -above with `src/matched-engine-bench.ts`. +above with `src/matched-engine-bench.ts`. `--results-dir ` redirects every +artifact write of the engine captures, so a rehearsal never overwrites tracked +evidence. The suite covers independent parse/load and first recompute, safe-depth linear chains, 100K fan-out, diamonds, shared/distinct ranges, cross-sheet ranges, @@ -206,8 +219,48 @@ bun run bench:render:smoke -- --engine handsontable # Fail-closed schema, completeness, and byte-stable Markdown validation: bun run bench:render:validate + +# Core write-path evidence (packed block writes, CSV import, large restore, +# paged residency); writes results/core-paths-results.json +bun run bench:core-paths ``` +The evidence page at +`docs/src/content/docs/guides/performance-resources.md` publishes every +artifact above. One command captures that whole set, in order, from a clean +tree. Use one pinned CPU on an otherwise idle machine, with no concurrent +builds or timing captures. The example below uses CPU 4 on Linux: + +```sh +# Build first: every capture needs the built packages and WASM binaries. +bun run build:packages + +# Then, from bench/, with the prepared XLSX comparison checkout: +taskset -c 4 bun run bench:release --xlsx-baseline-root /path/to/sheetwrite-xlsx-baseline +``` + +`bench:release` refuses a dirty tree, runs the captures in page order, stops at +the first failure, and rejects an artifact that does not stamp the current +clean commit. The XLSX capture compares the current tree against the +pre-0.5.0 codec, so the run needs a prepared checkout of that commit +(`git worktree add ../sheetwrite-xlsx-baseline 87fadb72`, then +`bun install --frozen-lockfile && bun run build:packages` inside it) passed as +`--xlsx-baseline-root `. `bench:release:smoke` +rehearses the same order with the smoke +matrices of the suites that have one, writing to `results/smoke/` (ignored by +git) so a rehearsal never overwrites published evidence. Regenerate the page +after a release capture with `bun run docs:generate`. + +A change to a harness source file, to the sampling flags, or to the protocol +version invalidates the committed render baseline: `bench:check` compares the +harness hashes, the declared sampling, and the runner (OS, CPU, Bun, Node, +Chromium, power mode, concurrency) of the fresh capture against +`results/render-baseline.json` and fails closed on any mismatch. `bench:release` +therefore ends by re-recording it on the frozen harness: ten controlled rounds +into `results/render-baseline-raw.json`, then the promotion of that raw artifact +over the committed baseline. Commit both in the release change; the promotion +diff is baseline-only and reviewable on its own. + ### Fail-closed gate policy Ordinary CI runs the deterministic gate, not a timing comparison: diff --git a/bench/package.json b/bench/package.json index df227bfe..f70f016a 100644 --- a/bench/package.json +++ b/bench/package.json @@ -11,7 +11,10 @@ "bench:range:smoke": "bun run src/range-bench.ts --smoke", "bench:paged": "bun run src/paged-bench.ts", "bench:paged:smoke": "bun run src/paged-bench.ts --smoke", - "bench:core-paths": "bun run src/core-paths-bench.ts", + "bench:core-paths": "bun run src/core-paths-bench.ts --output results/core-paths-results.json", + "bench:core-paths:smoke": "bun run src/core-paths-bench.ts --smoke", + "bench:release": "bun run src/release-capture.ts", + "bench:release:smoke": "bun run src/release-capture.ts --smoke", "bench:prefetch": "bun run src/datasource-prefetch-bench.ts", "bench:billion-cell": "bun run src/billion-cell-bench.ts", "bench:formula": "bun run src/formula-bench.ts", @@ -33,7 +36,7 @@ "bench:baseline:candidate": "bun run src/generate-baseline.ts", "test:gates": "bun test test", "typecheck": "tsc --noEmit -p tsconfig.json", - "bench:render:scale": "bun run src/render-driver.ts --rows 1000,10000,100000,1000000 --rounds 10 --compact-samples --output results/render-scale.json --markdown-output results/render-scale.md" + "bench:render:scale": "bun run src/render-driver.ts --rows 1000,10000,100000,1000000 --rounds 10 --samples 5 --warmups 2 --compact-samples --output results/render-scale.json --markdown-output results/render-scale.md" }, "devDependencies": { "@happy-dom/global-registrator": "^20.14.5", diff --git a/bench/src/formula-bench.ts b/bench/src/formula-bench.ts index c4d36657..52743a01 100644 --- a/bench/src/formula-bench.ts +++ b/bench/src/formula-bench.ts @@ -465,6 +465,126 @@ function summarizeAllocations(samples: readonly FormulaAllocationSample[]): Form }; } +/** Untimed fixture runs allowed inside one aggregated sample before it is abandoned. */ +export const AGGREGATED_SAMPLE_MAX_ITERATIONS = 1_000_000; +/** Default aggregation floor (ms) of the single-sample runner used by the matched engine protocol. */ +export const SAMPLE_AGGREGATE_MS = 50; +/** Default untimed warm-up fixture runs of the single-sample runner. */ +export const SAMPLE_WARMUP_FIXTURES = 3; + +/** Knobs for one aggregated timing sample: a wall-clock total over repeated fixture runs. */ +export interface AggregatedSampleOptions { + /** Untimed fixture runs before the timed loop; the first one also checks correctness. */ + readonly warmupFixtures: number; + /** Timed fixture runs repeat until their total wall time reaches this floor. */ + readonly minimumSampleDurationMs: number; +} + +/** + * One aggregated sample. `samplesMs` holds the per-iteration mean so a sample + * stays comparable across workloads, while `totalMs` and `iterations` keep the + * raw duration and the repetition count auditable. + */ +export interface AggregatedWorkloadSample extends CompleteFormulaWorkloadResult { + readonly iterations: number; + readonly totalMs: number; + readonly warmupFixtures: number; +} + +/** + * One timed fixture run: a fresh fixture, a monotonic duration around `run`, + * the post-run store allocation, and the workload's exact output check. The + * duration is proven finite before it can enter an aggregate. + */ +function timedFixtureRun( + id: string, + factory: FixtureFactory, + expected: FormulaOutput, +): { readonly elapsedMs: number; readonly allocation: FormulaAllocationSample } { + const fixture = factory(); + fixture.store.resetFormulaMatrixResourceStats(); + const started = now(); + fixture.run(); + const elapsedMs = now() - started; + const allocation = allocationAt(fixture.store); + assert(outputsEqual(fixture.check(), expected), `${id} measured output`); + fixture.dispose(); + assert(Number.isFinite(elapsedMs) && elapsedMs >= 0, `${id} measured duration`); + return { elapsedMs, allocation }; +} + +function warmUpFixtures( + id: string, + factory: FixtureFactory, + warmupFixtures: number, + expected: FormulaOutput, +): FormulaAllocationSample { + if (!Number.isInteger(warmupFixtures) || warmupFixtures < 1) { + throw new RangeError("warmupFixtures must be a positive integer"); + } + let allocation: FormulaAllocationSample | undefined; + for (let index = 0; index < warmupFixtures; index += 1) { + const fixture = factory(); + fixture.store.resetFormulaMatrixResourceStats(); + fixture.run(); + assert(outputsEqual(fixture.check(), expected), `${id} warmup output`); + allocation = allocationAt(fixture.store); + fixture.dispose(); + } + if (allocation === undefined) throw new RangeError("warmupFixtures must run at least once"); + return allocation; +} + +/** + * Collect one aggregated timing sample: untimed warm-up fixtures first, then + * repeated timed runs until the total wall time reaches `minimumSampleDurationMs`. + * A single run of a cheap workload sits close to timer and scheduler noise; the + * aggregate averages that noise away while keeping one auditable sample. + */ +export function collectAggregatedSample( + id: string, + size: number, + factory: FixtureFactory, + options: AggregatedSampleOptions, + expectedOutput?: FormulaOutput, +): AggregatedWorkloadSample { + if (!Number.isFinite(options.minimumSampleDurationMs) || options.minimumSampleDurationMs <= 0) { + throw new RangeError("minimumSampleDurationMs must be a positive number"); + } + const expected = expectedOutput ?? expectedFormulaOutput(id, size); + let allocation = warmUpFixtures(id, factory, options.warmupFixtures, expected); + + // One forced GC brackets the whole sample: a per-iteration GC would dominate + // the cheap workloads the aggregate exists to measure. + forceGc(); + let totalMs = 0; + let iterations = 0; + while (totalMs < options.minimumSampleDurationMs) { + if (iterations >= AGGREGATED_SAMPLE_MAX_ITERATIONS) { + throw new RangeError( + `${id} did not reach ${options.minimumSampleDurationMs} ms within ${AGGREGATED_SAMPLE_MAX_ITERATIONS} fixture runs`, + ); + } + const run = timedFixtureRun(id, factory, expected); + totalMs += run.elapsedMs; + iterations += 1; + allocation = run.allocation; + } + const perIterationMs = totalMs / iterations; + return { + id, + size, + samplesMs: [perIterationMs], + stat: summarize([perIterationMs]), + allocationSamples: [allocation], + allocationStat: summarizeAllocations([allocation]), + output: expected, + iterations, + totalMs, + warmupFixtures: options.warmupFixtures, + }; +} + export function collectFixture( id: string, size: number, @@ -2416,8 +2536,26 @@ if (import.meta.main) { }; const factory = id === undefined ? undefined : factories[id]; assert(id !== undefined && factory, "unknown single-sample workload"); + const aggregateIndex = process.argv.indexOf("--aggregate-ms"); + const aggregateMs = + aggregateIndex < 0 ? SAMPLE_AGGREGATE_MS : Number(process.argv[aggregateIndex + 1]); + assert(Number.isFinite(aggregateMs) && aggregateMs > 0, "invalid sample aggregation floor"); + const warmupIndex = process.argv.indexOf("--warmup-fixtures"); + const warmupFixtures = + warmupIndex < 0 ? SAMPLE_WARMUP_FIXTURES : Number(process.argv[warmupIndex + 1]); + assert( + Number.isInteger(warmupFixtures) && warmupFixtures > 0, + "invalid sample warm-up fixture count", + ); await initSheetwrite(undefined, engine); - process.stdout.write(`${JSON.stringify(collectFixture(id, size, factory, 1))}\n`); + process.stdout.write( + `${JSON.stringify( + collectAggregatedSample(id, size, factory, { + warmupFixtures, + minimumSampleDurationMs: aggregateMs, + }), + )}\n`, + ); } else { const memoryIndex = process.argv.indexOf("--memory"); if (memoryIndex >= 0) { diff --git a/bench/src/full-engine-bench.ts b/bench/src/full-engine-bench.ts index 9f0501d1..a321a903 100644 --- a/bench/src/full-engine-bench.ts +++ b/bench/src/full-engine-bench.ts @@ -1,5 +1,6 @@ import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; +import { resolve } from "node:path"; import { brotliCompressSync, constants } from "node:zlib"; import { initSheetwrite } from "@sheetwrite/core"; import * as formulas from "@sheetwrite/formulas"; @@ -11,6 +12,7 @@ import { } from "./formula-bench.js"; import { validateExactMatrix, validateRawStat } from "./gate-protocol.js"; import { type ProtocolCaptureMeta, protocolCaptureMeta } from "./protocol-meta.js"; +import { resultsDirectory } from "./results-dir.js"; import { type Stat, summarize } from "./stats.js"; const ENGINE_IDS = ["default", "full"] as const; @@ -260,6 +262,8 @@ function analysisFixture(formula: string) { } async function capture(): Promise { const meta = protocolCaptureMeta(); + const args = process.argv.slice(2).filter((argument) => argument !== "--"); + const resultsDir = resultsDirectory(args); const engines: EngineCapture[] = []; for (const engine of ENGINE_IDS) { // Keep tracked results unchanged until both engines and all checks finish. @@ -336,12 +340,12 @@ async function capture(): Promise { validateFullEngineResult(result); for (const capture of engines) { await Bun.write( - new URL(`../results/formula-${capture.engine}-results.json`, import.meta.url), + resolve(resultsDir, `formula-${capture.engine}-results.json`), `${JSON.stringify(capture.shared, null, 2)}\n`, ); } await Bun.write( - new URL("../results/full-engine-results.json", import.meta.url), + resolve(resultsDir, "full-engine-results.json"), `${JSON.stringify(result, null, 2)}\n`, ); } diff --git a/bench/src/generate-baseline.ts b/bench/src/generate-baseline.ts index f64c8cad..46b5cc34 100644 --- a/bench/src/generate-baseline.ts +++ b/bench/src/generate-baseline.ts @@ -12,6 +12,7 @@ import { computeHarnessFingerprint, MATRIX_IDS, } from "./gate-protocol.js"; +import { sourceTreeDirty } from "./protocol-meta.js"; import type { RenderBenchmarkArtifact } from "./render-protocol.js"; import { parseRenderArtifact } from "./render-protocol.js"; @@ -79,7 +80,7 @@ export async function generateBaseline(args: readonly string[]): Promise if (diagnostic && writeBaseline) { throw new Error("--diagnostic can produce review candidates only, not an approved baseline"); } - const dirty = gitOutput(["status", "--porcelain"]).length > 0; + const dirty = sourceTreeDirty(true); if (dirty && !diagnostic) { throw new Error( "baseline generation refuses a dirty tree; use --diagnostic for a non-approved candidate", diff --git a/bench/src/matched-engine-bench.ts b/bench/src/matched-engine-bench.ts index fbabd7cd..7755a2d2 100644 --- a/bench/src/matched-engine-bench.ts +++ b/bench/src/matched-engine-bench.ts @@ -1,9 +1,17 @@ import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; import { cpus, release } from "node:os"; -import { type CompleteFormulaWorkloadResult, expectedFormulaOutput } from "./formula-bench.js"; +import { resolve } from "node:path"; +import { + AGGREGATED_SAMPLE_MAX_ITERATIONS, + type AggregatedWorkloadSample, + expectedFormulaOutput, + SAMPLE_AGGREGATE_MS, + SAMPLE_WARMUP_FIXTURES, +} from "./formula-bench.js"; import { validateExactMatrix, validateRawStat } from "./gate-protocol.js"; import { type ProtocolCaptureMeta, protocolCaptureMeta } from "./protocol-meta.js"; +import { resultsDirectory } from "./results-dir.js"; import { type Stat, summarize } from "./stats.js"; export const MATCHED_ENGINE_WORKLOADS = [ @@ -16,11 +24,16 @@ export const MATCHED_ENGINE_WORKLOADS = [ { id: "scalar-edit-affects-0", size: 1_000 }, { id: "vlookup-many", size: 1_000 }, ] as const; -const ROUNDS = 51; +const DEFAULT_ROUNDS = 25; +const MINIMUM_ROUNDS = 2; const ENGINES = ["default", "full"] as const; type EngineId = (typeof ENGINES)[number]; -const METHOD = - "51 ABAB paired rounds per workload; one fresh Bun process and one untimed warmup fixture per timed sample"; +export const MATCHED_MINIMUM_SAMPLE_MS = SAMPLE_AGGREGATE_MS; +export const MATCHED_WARMUP_FIXTURES = SAMPLE_WARMUP_FIXTURES; +/** Protocol description of one capture; the round count is part of the protocol. */ +export function matchedMethod(rounds: number): string { + return `${rounds} ABAB paired rounds per workload; each timed sample uses a fresh Bun process, runs ${MATCHED_WARMUP_FIXTURES} untimed warm-up fixtures, then repeats the fixture until at least ${MATCHED_MINIMUM_SAMPLE_MS} ms of measured time and records the per-iteration mean`; +} const SOURCE_PATHS = [ "bench/src/matched-engine-bench.ts", "bench/src/formula-bench.ts", @@ -29,7 +42,7 @@ const SOURCE_PATHS = [ "packages/wasm/pkg/sheetwrite_wasm_bg.wasm", "packages/formulas/pkg/sheetwrite_wasm_bg.wasm", ]; -interface MatchedSample extends CompleteFormulaWorkloadResult { +interface MatchedSample extends AggregatedWorkloadSample { round: number; engine: EngineId; } @@ -48,11 +61,16 @@ export interface MatchedEngineResult { meta: ProtocolCaptureMeta; runner: { cpu: string; kernel: string; bun: string; arch: string; concurrency: 1 }; method: string; - rounds: typeof ROUNDS; + /** Paired ABAB rounds; a rehearsal may declare fewer than the release default. */ + rounds: number; + minimumSampleDurationMs: typeof MATCHED_MINIMUM_SAMPLE_MS; + warmupFixtures: typeof MATCHED_WARMUP_FIXTURES; sourceFiles: Record; sourceDigest: string; rows: MatchedEngineRow[]; } +/** Floating-point slack when a sample's mean is recomputed from its total and iteration count. */ +const SAMPLE_MEAN_TOLERANCE_MS = 1e-6; function percentile(samples: readonly number[], fraction: number): number { const ordered = [...samples].sort((left, right) => left - right); const position = (ordered.length - 1) * fraction; @@ -80,11 +98,39 @@ function rowSamples(samples: readonly MatchedSample[], engine: EngineId): number return duration; }); } +/** + * One matched sample must be a fresh-process aggregate: the declared number of + * untimed warm-up fixtures, at least the declared floor of measured time, and a + * per-iteration mean that matches its own total and iteration count. + */ +function validateAggregatedSample(sample: MatchedSample, label: string): void { + if ( + sample.warmupFixtures !== MATCHED_WARMUP_FIXTURES || + !Number.isInteger(sample.iterations) || + sample.iterations < 1 || + sample.iterations > AGGREGATED_SAMPLE_MAX_ITERATIONS || + !Number.isFinite(sample.totalMs) || + sample.totalMs < MATCHED_MINIMUM_SAMPLE_MS + ) { + throw new Error(`${label}: sample is not an aggregate of the declared matched protocol`); + } + const observedMean = sample.samplesMs[0]; + if ( + observedMean === undefined || + Math.abs(observedMean - sample.totalMs / sample.iterations) > SAMPLE_MEAN_TOLERANCE_MS + ) { + throw new Error(`${label}: sample mean does not match its total and iteration count`); + } +} + export function validateMatchedEngineResult(result: MatchedEngineResult): void { if ( result.schemaVersion !== 2 || - result.method !== METHOD || - result.rounds !== ROUNDS || + !Number.isInteger(result.rounds) || + result.rounds < MINIMUM_ROUNDS || + result.method !== matchedMethod(result.rounds) || + result.minimumSampleDurationMs !== MATCHED_MINIMUM_SAMPLE_MS || + result.warmupFixtures !== MATCHED_WARMUP_FIXTURES || result.runner.concurrency !== 1 ) throw new Error("Matched engine protocol mismatch"); @@ -106,7 +152,8 @@ export function validateMatchedEngineResult(result: MatchedEngineResult): void { result.rows.map(workloadKey), ); for (const row of result.rows) { - if (row.samples.length !== ROUNDS * ENGINES.length) throw new Error("Missing matched samples"); + if (row.samples.length !== result.rounds * ENGINES.length) + throw new Error("Missing matched samples"); for (const [index, sample] of row.samples.entries()) { if ( sample.round !== Math.floor(index / ENGINES.length) || @@ -117,6 +164,7 @@ export function validateMatchedEngineResult(result: MatchedEngineResult): void { sample.allocationSamples.length !== 1 ) throw new Error("Matched sample order or identity mismatch"); + validateAggregatedSample(sample, `matched sample ${workloadKey(row)} round ${sample.round}`); validateRawStat(sample.samplesMs, sample.stat, "matched timing"); const expected = expectedFormulaOutput(row.id, row.size); if (sample.output !== expected) throw new Error("Matched checksum mismatch"); @@ -145,28 +193,45 @@ export function validateMatchedEngineResult(result: MatchedEngineResult): void { throw new Error("Matched spread does not match samples"); } } -async function captureMatchedEngines(): Promise { +/** + * Run one matched sample in its own Bun process. A fresh process keeps the two + * engines from sharing JIT tiers, heap state, or a warmed module cache, while + * the child's warm-up fixtures and aggregation remove the per-process cold-run + * bias that made single short runs unusable. + */ +function runMatchedSample(workload: { id: string; size: number }, engine: EngineId): MatchedSample { + const child = Bun.spawnSync( + [ + "bun", + "run", + "src/formula-bench.ts", + "--sample", + workload.id, + String(workload.size), + "--aggregate-ms", + String(MATCHED_MINIMUM_SAMPLE_MS), + "--warmup-fixtures", + String(MATCHED_WARMUP_FIXTURES), + ], + { + cwd: new URL("..", import.meta.url).pathname, + env: { ...process.env, SHEETWRITE_BENCH_ENGINE: engine }, + stdout: "pipe", + stderr: "inherit", + }, + ); + if (child.exitCode !== 0) throw new Error(`${engine} ${workload.id} sample failed`); + return JSON.parse(child.stdout.toString()) as MatchedSample; +} + +async function captureMatchedEngines(rounds: number, resultsDir: string): Promise { const meta = protocolCaptureMeta(); const rows: MatchedEngineRow[] = []; for (const workload of MATCHED_ENGINE_WORKLOADS) { const samples: MatchedSample[] = []; - for (let round = 0; round < ROUNDS; round++) { + for (let round = 0; round < rounds; round++) { for (const engine of ENGINES) { - const child = Bun.spawnSync( - ["bun", "run", "src/formula-bench.ts", "--sample", workload.id, String(workload.size)], - { - cwd: new URL("..", import.meta.url).pathname, - env: { ...process.env, SHEETWRITE_BENCH_ENGINE: engine }, - stdout: "pipe", - stderr: "inherit", - }, - ); - if (child.exitCode !== 0) throw new Error(`${engine} ${workload.id} sample failed`); - samples.push({ - ...(JSON.parse(child.stdout.toString()) as CompleteFormulaWorkloadResult), - engine, - round, - }); + samples.push({ ...runMatchedSample(workload, engine), engine, round }); } } const defaultSamples = rowSamples(samples, "default"); @@ -202,16 +267,28 @@ async function captureMatchedEngines(): Promise { arch: process.arch, concurrency: 1, }, - method: METHOD, - rounds: ROUNDS, + method: matchedMethod(rounds), + rounds, + minimumSampleDurationMs: MATCHED_MINIMUM_SAMPLE_MS, + warmupFixtures: MATCHED_WARMUP_FIXTURES, sourceFiles, sourceDigest: sourceDigest({ meta, sourceFiles }), rows, }; validateMatchedEngineResult(result); await Bun.write( - new URL("../results/full-engine-matched-results.json", import.meta.url), + resolve(resultsDir, "full-engine-matched-results.json"), `${JSON.stringify(result, null, 2)}\n`, ); } -if (import.meta.main) await captureMatchedEngines(); + +if (import.meta.main) { + const args = process.argv.slice(2).filter((argument) => argument !== "--"); + const roundsIndex = args.indexOf("--rounds"); + const requestedRounds = roundsIndex < 0 ? undefined : Number(args[roundsIndex + 1]); + const rounds = requestedRounds ?? DEFAULT_ROUNDS; + if (!Number.isInteger(rounds) || rounds < MINIMUM_ROUNDS) { + throw new TypeError(`--rounds must be an integer >= ${MINIMUM_ROUNDS}`); + } + await captureMatchedEngines(rounds, resultsDirectory(args)); +} diff --git a/bench/src/protocol-meta.ts b/bench/src/protocol-meta.ts index e0e49dba..cceeba65 100644 --- a/bench/src/protocol-meta.ts +++ b/bench/src/protocol-meta.ts @@ -1,6 +1,12 @@ import { resolve } from "node:path"; const REPOSITORY_ROOT = resolve(import.meta.dir, "../.."); +/** + * Capture outputs are results, not inputs: a written artifact cannot change a + * measured timing. Excluding this pathspec lets one clean tree publish a whole + * evidence set at a single commit instead of one commit per artifact. + */ +const CAPTURE_OUTPUT_PATHSPEC = ":!bench/results"; /** * Protocol-binding capture stamp shared by every benchmark artifact. The docs @@ -25,10 +31,27 @@ function gitOutput(args: readonly string[]): string { return result.stdout.toString().trim(); } +/** + * True when the sources differ from HEAD. `bench/results` is excluded because + * every path under it is an output of the protocols themselves. Pass + * `includeUntracked` when an uncommitted new file must also block the run. + */ +export function sourceTreeDirty(includeUntracked: boolean): boolean { + return ( + gitOutput([ + "status", + "--porcelain", + ...(includeUntracked ? [] : ["--untracked-files=no"]), + "--", + CAPTURE_OUTPUT_PATHSPEC, + ]).length > 0 + ); +} + export function protocolCaptureMeta(): ProtocolCaptureMeta { return { commit: gitOutput(["rev-parse", "HEAD"]), - dirty: gitOutput(["status", "--porcelain", "--untracked-files=no"]).length > 0, + dirty: sourceTreeDirty(false), timestamp: new Date().toISOString(), }; } diff --git a/bench/src/release-capture.ts b/bench/src/release-capture.ts new file mode 100644 index 00000000..4e24de8a --- /dev/null +++ b/bench/src/release-capture.ts @@ -0,0 +1,292 @@ +import { existsSync } from "node:fs"; +import { resolve } from "node:path"; +import { CONTROLLED_BASELINE_MINIMUM_ROUNDS } from "./controlled-baseline.js"; + +const BENCH_ROOT = new URL("..", import.meta.url).pathname; +const REPOSITORY_ROOT = resolve(BENCH_ROOT, ".."); +const SCRATCH_DIRECTORY = "results/smoke"; +/** The codec checkout the XLSX capture compares against; see `bench:xlsx`. */ +const XLSX_BASELINE_COMMIT = "87fadb72397947ea8c6ba576682925c7879579b8"; +/** Replaced with `--xlsx-baseline-root` at run time. */ +const XLSX_BASELINE_ROOT_TOKEN = "{xlsx-baseline-root}"; + +interface ReleaseOptions { + /** Prepared checkout of the XLSX baseline commit with its packages built. */ + readonly xlsxBaselineRoot: string; +} + +/** One capture the evidence page publishes numbers from. */ +interface CaptureStep { + readonly name: string; + /** Artifact file names under the results directory. */ + readonly artifacts: readonly string[]; + /** Commands of the full capture, in order. */ + readonly full: readonly (readonly string[])[]; + /** Rehearsal commands; absent when the suite has no smoke matrix. */ + readonly smoke?: readonly (readonly string[])[]; +} + +/** + * The order matters: the browser capture is the longest run, so it starts + * first, and the regression baseline is recorded last, on the frozen harness, + * after every evidence artifact of this commit exists. + */ +const STEPS: readonly CaptureStep[] = [ + { + name: "render-scale", + artifacts: ["render-scale.json"], + full: [["bench:render:scale"]], + smoke: [ + [ + "src/render-driver.ts", + "--smoke", + "--output", + `${SCRATCH_DIRECTORY}/render-scale.json`, + "--markdown-output", + `${SCRATCH_DIRECTORY}/render-scale.md`, + ], + ], + }, + { + name: "data", + artifacts: ["data-results.json"], + full: [["bench:data"]], + smoke: [["src/data-bench.ts", "--smoke", "--output", `${SCRATCH_DIRECTORY}/data-results.json`]], + }, + { + name: "core-paths", + artifacts: ["core-paths-results.json"], + full: [["bench:core-paths"]], + smoke: [ + [ + "src/core-paths-bench.ts", + "--smoke", + "--output", + `${SCRATCH_DIRECTORY}/core-paths-results.json`, + ], + ], + }, + { + name: "formula", + artifacts: ["formula-results.json"], + full: [["bench:formula"]], + smoke: [ + ["src/formula-bench.ts", "--smoke", "--output", `${SCRATCH_DIRECTORY}/formula-results.json`], + ], + }, + { + name: "formula-engines", + artifacts: [ + "formula-default-results.json", + "formula-full-results.json", + "full-engine-results.json", + ], + full: [["bench:formula:engines"]], + }, + { + name: "formula-matched", + artifacts: ["full-engine-matched-results.json"], + full: [["bench:formula:matched"]], + }, + { + name: "xlsx", + artifacts: ["xlsx-results.json"], + full: [["bench:xlsx", "--baseline-root", XLSX_BASELINE_ROOT_TOKEN]], + smoke: [["src/xlsx-bench.ts", "--smoke", "--output", `${SCRATCH_DIRECTORY}/xlsx-results.json`]], + }, + { + // A harness, protocol, or runner change invalidates the committed + // fingerprint, so the release capture re-records the baseline from the + // frozen harness: ten controlled rounds, then the promotion of that raw + // artifact as a standalone, reviewable baseline diff. + name: "regression-baseline", + artifacts: ["render-baseline-raw.json", "render-baseline.json"], + full: [ + [ + "src/render-driver.ts", + "--rounds", + String(CONTROLLED_BASELINE_MINIMUM_ROUNDS), + "--output", + "results/render-baseline-raw.json", + "--markdown-output", + "results/render-baseline-raw.md", + ], + [ + "src/generate-baseline.ts", + "--input", + "results/render-baseline-raw.json", + "--power-mode", + "balanced", + "--concurrency", + "1", + "--write-baseline", + ], + ], + }, +]; + +/** Every artifact the evidence page publishes, in capture order. */ +export const RELEASE_ARTIFACTS: readonly string[] = STEPS.flatMap((step) => step.artifacts); + +/** Rehearsal commands that write artifacts, for the scratch-directory contract. */ +export const RELEASE_SMOKE_COMMANDS: readonly (readonly string[])[] = STEPS.flatMap( + (step) => step.smoke ?? [], +); + +interface ArtifactStamp { + readonly file: string; + readonly commit: string; + readonly dirty: boolean | undefined; + readonly timestamp: string | undefined; +} + +function gitOutput(args: readonly string[]): string { + const result = Bun.spawnSync(["git", ...args], { + cwd: REPOSITORY_ROOT, + stdout: "pipe", + stderr: "pipe", + }); + if (result.exitCode !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${result.stderr.toString().trim()}`); + } + return result.stdout.toString().trim(); +} + +function runStep(name: string, command: readonly string[]): void { + process.stderr.write(`\n=== ${name}: bun run ${command.join(" ")}\n`); + const child = Bun.spawnSync(["bun", "run", ...command], { + cwd: BENCH_ROOT, + stdout: "inherit", + stderr: "inherit", + }); + if (child.exitCode !== 0) { + throw new Error(`${name} capture failed with exit code ${child.exitCode}`); + } +} + +/** + * Read an artifact's capture stamp. Every protocol states it as `meta` or + * `metadata`; the promoted regression baseline states the commit under + * `source` and carries no dirty flag of its own. + */ +async function artifactStamp(path: string): Promise { + const file = Bun.file(path); + if (!(await file.exists())) return undefined; + const value = (await file.json()) as { + meta?: { commit?: unknown; dirty?: unknown; timestamp?: unknown }; + metadata?: { commit?: unknown; dirty?: unknown; timestamp?: unknown }; + source?: { commit?: unknown }; + }; + const stamp = value.meta ?? value.metadata ?? value.source; + if (stamp === undefined) return undefined; + const { commit } = stamp; + if (typeof commit !== "string") return undefined; + const dirty = "dirty" in stamp ? stamp.dirty : undefined; + const timestamp = "timestamp" in stamp ? stamp.timestamp : undefined; + return { + file: path, + commit, + dirty: typeof dirty === "boolean" ? dirty : undefined, + timestamp: typeof timestamp === "string" ? timestamp : undefined, + }; +} + +/** Substitute run-time values into a step's arguments. */ +function resolveCommands( + commands: readonly (readonly string[])[], + options: ReleaseOptions, +): readonly (readonly string[])[] { + return commands.map((command) => + command.map((argument) => + argument === XLSX_BASELINE_ROOT_TOKEN ? options.xlsxBaselineRoot : argument, + ), + ); +} + +/** + * The XLSX capture compares the current tree against the pre-0.5.0 codec, so it + * needs a prepared checkout of that commit before the run starts: failing here + * costs seconds, failing after the browser capture costs the whole run. + */ +function requireXlsxBaselineRoot(argument: string | undefined): string { + if (argument === undefined || argument.startsWith("--")) { + throw new Error( + `the release capture needs --xlsx-baseline-root: a checkout of ${XLSX_BASELINE_COMMIT} with its packages built, for example\n` + + ` git worktree add ../sheetwrite-xlsx-baseline ${XLSX_BASELINE_COMMIT}\n` + + ` (cd ../sheetwrite-xlsx-baseline && bun install --frozen-lockfile && bun run build:packages)`, + ); + } + if (!existsSync(resolve(argument))) { + throw new Error(`--xlsx-baseline-root does not exist: ${argument}`); + } + return resolve(argument); +} + +async function main(): Promise { + const args = process.argv.slice(2).filter((argument) => argument !== "--"); + const smoke = args.includes("--smoke"); + const optionIndex = args.indexOf("--xlsx-baseline-root"); + const options: ReleaseOptions = { + xlsxBaselineRoot: smoke + ? "" + : requireXlsxBaselineRoot(optionIndex < 0 ? undefined : args[optionIndex + 1]), + }; + const commit = gitOutput(["rev-parse", "HEAD"]); + const dirtyTree = gitOutput(["status", "--porcelain", "--untracked-files=no"]).length > 0; + if (dirtyTree) { + throw new Error( + "the working tree has uncommitted changes; every published artifact must stamp a clean commit", + ); + } + const resultsDir = resolve(BENCH_ROOT, smoke ? SCRATCH_DIRECTORY : "results"); + process.stderr.write( + [ + `release capture at ${commit.slice(0, 12)}${smoke ? " (smoke rehearsal)" : ""}`, + smoke + ? `writing rehearsals to ${resultsDir}; tracked evidence is not touched` + : `writing evidence to ${resultsDir}`, + smoke + ? "the engine captures and the regression baseline have no smoke matrix and are skipped; run without --smoke for the release set" + : "run on an otherwise idle machine, pinned to one CPU (e.g. `taskset -c 4 bun run bench:release` on Linux); no concurrent builds or timing captures, because competing work adds scheduler and CPU noise", + "", + ].join("\n"), + ); + + const captured: ArtifactStamp[] = []; + for (const step of STEPS) { + const commands = smoke ? step.smoke : step.full; + if (commands === undefined) { + process.stderr.write(`\n=== ${step.name}: skipped (no smoke matrix)\n`); + continue; + } + for (const command of resolveCommands(commands, options)) runStep(step.name, command); + for (const artifact of step.artifacts) { + const stamp = await artifactStamp(resolve(resultsDir, artifact)); + if (stamp === undefined) { + throw new Error(`${step.name} did not write ${artifact} with a capture stamp`); + } + if (!smoke && (stamp.commit !== commit || stamp.dirty === true)) { + throw new Error(`${artifact} does not stamp this clean commit ${commit}`); + } + captured.push(stamp); + } + } + + process.stderr.write( + [ + "", + smoke ? "smoke rehearsal complete:" : "release capture complete:", + ...captured.map( + (stamp) => + ` ${stamp.file} ${stamp.commit.slice(0, 12)}${stamp.dirty === true ? " (dirty)" : ""} ${stamp.timestamp ?? "no timestamp"}`, + ), + "", + smoke + ? "Nothing was published. Run without --smoke for the release evidence." + : "Next: commit these artifacts, then `bun run docs:generate` regenerates docs/src/content/docs/guides/performance-resources.md from them.", + "", + ].join("\n"), + ); +} + +if (import.meta.main) await main(); diff --git a/bench/src/render-driver.ts b/bench/src/render-driver.ts index 9352cd5f..509f8041 100644 --- a/bench/src/render-driver.ts +++ b/bench/src/render-driver.ts @@ -4,6 +4,7 @@ import { fileURLToPath } from "node:url"; import { type Browser, chromium, type Page } from "@playwright/test"; import { DEFAULT_SEED, datasetChecksum, makeColumnar } from "./dataset.js"; import type { BenchmarkMode } from "./gate-protocol.js"; +import { sourceTreeDirty } from "./protocol-meta.js"; import benchmarkPage from "./render-bench.html"; import { validateRenderGateArtifact } from "./render-gate.js"; import { @@ -698,7 +699,7 @@ async function runDriver(args: readonly string[]): Promise { runId, metadata: { commit: gitOutput(["rev-parse", "HEAD"]), - dirty: gitOutput(["status", "--porcelain", "--untracked-files=no"]).length > 0, + dirty: sourceTreeDirty(false), timestamp, bunVersion: Bun.version, nodeVersion: process.versions.node, diff --git a/bench/src/render-scenarios.ts b/bench/src/render-scenarios.ts index 7362092a..9500aa51 100644 --- a/bench/src/render-scenarios.ts +++ b/bench/src/render-scenarios.ts @@ -1256,6 +1256,23 @@ function scenarioActions( }; } +/** Which part of one logical operation raised: the timed action or its untimed bracket. */ +type ScenarioStep = "prepare" | "action" | "counters" | "cleanup"; + +/** + * Keep the failing step and the original error class visible. A recorded + * failure that says only "Maximum call stack size exceeded" hides whether the + * measured action or its untimed cleanup produced it; the same error in a + * prepare step means the sample never started. + */ +function stepFailure(step: ScenarioStep, error: unknown): Error { + const cause = error instanceof Error ? error : new Error(String(error)); + const message = `${step} step failed: ${cause.message}`; + const wrapped = cause instanceof RangeError ? new RangeError(message) : new Error(message); + wrapped.cause = cause; + return wrapped; +} + function aggregateSample( adapter: RenderBenchAdapter, actions: ScenarioActions, @@ -1271,17 +1288,21 @@ function aggregateSample( outputAllocationEvents: 0, }; do { - actions.prepare(); + try { + actions.prepare(); + } catch (error) { + throw stepFailure("prepare", error); + } if (actions.measureWindowTransfer) adapter.resetWindowTransferCounters(); const started = performance.now(); - let operationError: unknown; + let failure: { step: ScenarioStep; error: unknown } | undefined; try { actions.action(); } catch (error) { - operationError = error; + failure = { step: "action", error }; } const elapsed = performance.now() - started; - if (actions.measureWindowTransfer && operationError === undefined) { + if (actions.measureWindowTransfer && failure === undefined) { try { const observed = adapter.windowTransferCounters(); transferCounters.logicalFrames += observed.logicalFrames; @@ -1290,15 +1311,17 @@ function aggregateSample( transferCounters.copiedBytes += observed.copiedBytes; transferCounters.outputAllocationEvents += observed.outputAllocationEvents; } catch (error) { - operationError = error; + failure = { step: "counters", error }; } } try { actions.cleanup(); - } catch (cleanupError) { - if (operationError === undefined) operationError = cleanupError; + } catch (error) { + // A failing action decides the sample; a failing cleanup only surfaces + // when the measured action itself succeeded. + failure ??= { step: "cleanup", error }; } - if (operationError !== undefined) throw operationError; + if (failure !== undefined) throw stepFailure(failure.step, failure.error); if (!Number.isFinite(elapsed) || elapsed < 0) { throw new RangeError(`invalid operation duration: ${elapsed}`); } diff --git a/bench/src/results-dir.ts b/bench/src/results-dir.ts new file mode 100644 index 00000000..18eac4f7 --- /dev/null +++ b/bench/src/results-dir.ts @@ -0,0 +1,18 @@ +import { resolve } from "node:path"; + +const BENCH_ROOT = resolve(import.meta.dir, ".."); + +/** + * Where a capture writes its artifacts. Tracked evidence lives in + * `bench/results`; `--results-dir` redirects every write of a run (used by the + * smoke pipeline) so a rehearsal can never overwrite published evidence. + */ +export function resultsDirectory(args: readonly string[]): string { + const index = args.indexOf("--results-dir"); + if (index < 0) return resolve(BENCH_ROOT, "results"); + const requested = args[index + 1]; + if (requested === undefined || requested.startsWith("--")) { + throw new TypeError("--results-dir requires a path"); + } + return resolve(BENCH_ROOT, requested); +} diff --git a/bench/test/matched-engine-bench.test.ts b/bench/test/matched-engine-bench.test.ts index aaea6ea1..0c4f8d0e 100644 --- a/bench/test/matched-engine-bench.test.ts +++ b/bench/test/matched-engine-bench.test.ts @@ -1,41 +1,169 @@ import { expect, it } from "bun:test"; -import { readFileSync } from "node:fs"; +import { createHash } from "node:crypto"; +import { expectedFormulaOutput } from "../src/formula-bench.js"; import { + MATCHED_ENGINE_WORKLOADS, + MATCHED_MINIMUM_SAMPLE_MS, + MATCHED_WARMUP_FIXTURES, type MatchedEngineResult, + type MatchedEngineRow, + matchedMethod, validateMatchedEngineResult, } from "../src/matched-engine-bench.js"; +import { summarize } from "../src/stats.js"; -function recordedCapture(): MatchedEngineResult { - return JSON.parse( - readFileSync(new URL("../results/full-engine-matched-results.json", import.meta.url), "utf8"), - ) as MatchedEngineResult; +const TEST_ROUNDS = 2; +const TEST_PER_ITERATION_MS = 0.75; +const TEST_ITERATIONS = 80; +const TEST_SOURCE_DIGEST = "a".repeat(64); +const TEST_COMMIT = "b".repeat(40); + +/** + * One conforming sample pair: both engines take the same time in a round, so + * every paired ratio is exactly 1 and no statistics are recomputed by hand. + */ +function sampleFor( + workload: (typeof MATCHED_ENGINE_WORKLOADS)[number], + engine: "default" | "full", + round: number, +): MatchedEngineRow["samples"][number] { + const allocation = { retainedBytes: 1_024, peakTransientBytes: 0, transientAllocations: 0 }; + const totalMs = TEST_PER_ITERATION_MS * TEST_ITERATIONS; + return { + id: workload.id, + size: workload.size, + samplesMs: [TEST_PER_ITERATION_MS], + stat: summarize([TEST_PER_ITERATION_MS]), + allocationSamples: [allocation], + allocationStat: { + retainedBytes: summarize([allocation.retainedBytes]), + peakTransientBytes: summarize([allocation.peakTransientBytes]), + transientAllocations: summarize([allocation.transientAllocations]), + }, + output: expectedFormulaOutput(workload.id, workload.size), + iterations: TEST_ITERATIONS, + totalMs, + warmupFixtures: MATCHED_WARMUP_FIXTURES, + engine, + round, + }; +} + +function conformingCapture(): MatchedEngineResult { + return { + schemaVersion: 2, + meta: { commit: TEST_COMMIT, dirty: false, timestamp: "2026-01-01T00:00:00.000Z" }, + runner: { + cpu: "test cpu", + kernel: "test kernel", + bun: "1.0.0", + arch: "x64", + concurrency: 1, + }, + method: matchedMethod(TEST_ROUNDS), + rounds: TEST_ROUNDS, + minimumSampleDurationMs: MATCHED_MINIMUM_SAMPLE_MS, + warmupFixtures: MATCHED_WARMUP_FIXTURES, + sourceFiles: { + "bench/src/matched-engine-bench.ts": TEST_SOURCE_DIGEST, + "bench/src/formula-bench.ts": TEST_SOURCE_DIGEST, + "bench/src/formula-dataset.ts": TEST_SOURCE_DIGEST, + "bench/src/stats.ts": TEST_SOURCE_DIGEST, + "packages/wasm/pkg/sheetwrite_wasm_bg.wasm": TEST_SOURCE_DIGEST, + "packages/formulas/pkg/sheetwrite_wasm_bg.wasm": TEST_SOURCE_DIGEST, + }, + sourceDigest: "", + rows: MATCHED_ENGINE_WORKLOADS.map((workload) => { + const samples = Array.from({ length: TEST_ROUNDS }, (_, round) => + (["default", "full"] as const).map((engine) => sampleFor(workload, engine, round)), + ).flat(); + const durationsFor = (engine: "default" | "full"): number[] => + samples + .filter((sample) => sample.engine === engine) + .map((sample) => sample.samplesMs[0] ?? 0); + const ratios = Array.from({ length: TEST_ROUNDS }, () => 1); + return { + id: workload.id, + size: workload.size, + samples, + defaultStat: summarize(durationsFor("default")), + fullStat: summarize(durationsFor("full")), + ratioStat: summarize(ratios), + ratioP10: 1, + ratioP90: 1, + }; + }), + }; +} + +/** The capture digest is derived, so the fixture recomputes it the way the runner does. */ +function captureWithDigest(): MatchedEngineResult { + const capture = conformingCapture(); + const digest = createHash("sha256") + .update(JSON.stringify({ commit: capture.meta.commit, files: capture.sourceFiles })) + .digest("hex"); + return { ...capture, sourceDigest: digest }; +} + +function firstSample(result: MatchedEngineResult) { + const sample = result.rows[0]?.samples[0]; + if (!sample) throw new Error("Missing fixture sample"); + return sample; } -it("accepts the complete matched engine capture", () => { - expect(() => validateMatchedEngineResult(recordedCapture())).not.toThrow(); + +it("accepts a complete conforming matched engine capture", () => { + expect(() => validateMatchedEngineResult(captureWithDigest())).not.toThrow(); }); -it.each(["missing", "order", "checksum", "summary", "allocation", "ratio", "spread", "source"])( - "rejects invalid matched %s evidence", - (fault) => { - const result = recordedCapture(); - const row = result.rows[0]; - const sample = row?.samples[0]; - if (!row || !sample) throw new Error("Missing recorded matched sample"); - if (fault === "missing") row.samples.pop(); - if (fault === "order") sample.engine = "full"; - if (fault === "checksum") sample.output = "wrong"; - if (fault === "summary") - row.defaultStat = { ...row.defaultStat, mean: row.defaultStat.mean + 1 }; - if (fault === "allocation") sample.allocationSamples.pop(); - if (fault === "ratio") row.ratioStat = { ...row.ratioStat, median: row.ratioStat.median + 1 }; - if (fault === "spread") row.ratioP10 += 1; - if (fault === "source") result.sourceDigest = "0".repeat(64); - expect(() => validateMatchedEngineResult(result)).toThrow(); - }, -); + +it.each([ + "missing", + "order", + "checksum", + "summary", + "allocation", + "ratio", + "spread", + "source", + "iterations", + "aggregation", + "warmup", + "rounds", +])("rejects invalid matched %s evidence", (fault) => { + const result = captureWithDigest(); + const row = result.rows[0]; + const sample = firstSample(result); + if (!row) throw new Error("Missing fixture row"); + if (fault === "missing") row.samples.pop(); + if (fault === "order") sample.engine = "full"; + if (fault === "checksum") sample.output = "wrong"; + if (fault === "summary") row.defaultStat = { ...row.defaultStat, mean: row.defaultStat.mean + 1 }; + if (fault === "allocation") sample.allocationSamples.pop(); + if (fault === "ratio") row.ratioStat = { ...row.ratioStat, median: row.ratioStat.median + 1 }; + if (fault === "spread") row.ratioP10 += 1; + if (fault === "source") result.sourceDigest = "0".repeat(64); + if (fault === "iterations") row.samples[0] = { ...sample, iterations: 0 }; + if (fault === "aggregation") + row.samples[0] = { ...sample, totalMs: MATCHED_MINIMUM_SAMPLE_MS / 2 }; + if (fault === "warmup") row.samples[0] = { ...sample, warmupFixtures: 0 }; + if (fault === "rounds") result.rounds = 1; + expect(() => validateMatchedEngineResult(result)).toThrow(); +}); + it.each(["default", "full"])("runs one checked fresh %s sample", (engine) => { const root = new URL("../../", import.meta.url).pathname; const child = Bun.spawnSync( - ["bun", "run", "src/formula-bench.ts", "--sample", "independent-first-recompute", "3"], + [ + "bun", + "run", + "src/formula-bench.ts", + "--sample", + "independent-first-recompute", + "3", + "--aggregate-ms", + "10", + "--warmup-fixtures", + "1", + ], { cwd: `${root}/bench`, env: { ...process.env, SHEETWRITE_BENCH_ENGINE: engine }, @@ -44,7 +172,17 @@ it.each(["default", "full"])("runs one checked fresh %s sample", (engine) => { }, ); expect(child.exitCode).toBe(0); - const sample = JSON.parse(child.stdout.toString()) as { samplesMs: number[]; output: number }; + const sample = JSON.parse(child.stdout.toString()) as { + samplesMs: number[]; + output: number; + iterations: number; + totalMs: number; + warmupFixtures: number; + }; expect(sample.samplesMs).toHaveLength(1); expect(sample.output).toBe(3); + expect(sample.iterations).toBeGreaterThan(0); + expect(sample.warmupFixtures).toBe(1); + expect(sample.totalMs).toBeGreaterThanOrEqual(10); + expect(sample.samplesMs[0]).toBeCloseTo(sample.totalMs / sample.iterations, 9); }); diff --git a/bench/test/release-capture.test.ts b/bench/test/release-capture.test.ts new file mode 100644 index 00000000..81b19f09 --- /dev/null +++ b/bench/test/release-capture.test.ts @@ -0,0 +1,31 @@ +import { expect, it } from "bun:test"; +import { readFileSync } from "node:fs"; +import { RELEASE_ARTIFACTS, RELEASE_SMOKE_COMMANDS } from "../src/release-capture.js"; + +function evidencePage(): string { + return readFileSync( + new URL("../../docs/src/content/docs/guides/performance-resources.md", import.meta.url), + "utf8", + ); +} + +it("captures every artifact the evidence page publishes", () => { + const page = evidencePage(); + for (const artifact of RELEASE_ARTIFACTS) { + expect(page).toContain(`bench/results/${artifact}`); + } +}); + +it("captures each artifact exactly once", () => { + expect(new Set(RELEASE_ARTIFACTS).size).toBe(RELEASE_ARTIFACTS.length); +}); + +it("rehearses every capture that has a smoke matrix without touching tracked evidence", () => { + expect(RELEASE_SMOKE_COMMANDS.length).toBeGreaterThan(0); + for (const command of RELEASE_SMOKE_COMMANDS) { + for (const argument of command) { + if (!argument.startsWith("results/")) continue; + expect(argument.startsWith("results/smoke/")).toBe(true); + } + } +}); diff --git a/bench/test/render-scenarios.test.ts b/bench/test/render-scenarios.test.ts index af2bb640..58b5bcef 100644 --- a/bench/test/render-scenarios.test.ts +++ b/bench/test/render-scenarios.test.ts @@ -24,6 +24,8 @@ class FakeAdapter implements RenderBenchAdapter { readonly values: unknown[][]; private readonly originalValues: unknown[][]; corruptNavigation = false; + /** Mirrors a competitor whose row removal overflows the stack above some size. */ + failRowRemoval = false; private mounted = true; private selected: CellSelection | null = null; private editing = false; @@ -137,6 +139,7 @@ class FakeAdapter implements RenderBenchAdapter { } removeRows(at: number, count: number): void { + if (this.failRowRemoval) throw new RangeError("Maximum call stack size exceeded"); this.values.splice(at, count); } @@ -267,6 +270,18 @@ describe("scenario correctness checkpoints", () => { expect(result.partialSamples).toHaveLength(1); }); + test("names the failing step when a successful action cannot be undone", () => { + const adapter = new FakeAdapter(); + adapter.failRowRemoval = true; + const result = runRenderScenario(adapter, dataset, "altering.insert-5-rows-top", options); + expect(result).toMatchObject({ status: "failed", stage: "measure" }); + if (result.status !== "failed") throw new Error("expected failed result"); + // The insert itself succeeded; only the untimed cleanup blew the stack, and + // the recorded failure must say so instead of blaming the measured action. + expect(result.errorClass).toBe("RangeError"); + expect(result.message).toBe("cleanup step failed: Maximum call stack size exceeded"); + }); + test("keeps one-pixel smooth scrolling inside the same logical window", () => { const result = runRenderScenario( new FakeAdapter(), diff --git a/docs/src/content/docs/guides/performance-resources.md b/docs/src/content/docs/guides/performance-resources.md index d2836723..70785a26 100644 --- a/docs/src/content/docs/guides/performance-resources.md +++ b/docs/src/content/docs/guides/performance-resources.md @@ -4,6 +4,17 @@ description: "Freshness-gated benchmark and package-size evidence for Sheetwrite --- Every number on this page comes from a validated local protocol artifact captured on a clean tree; nothing is published from an unvalidated or protocol-mismatched artifact. Every expected cell carries either a validated timing or its recorded failure - a run that did not complete is shown as a failure, never converted into a timing. +## Capture + +
Mixed capture vintages. The sections below were captured at different commits. Each section is internally consistent and each one names its own commit; the page as a whole is not a single point-in-time measurement.
+ +| Section | Artifact | Commit | Captured | +| --- | --- | --- | --- | +| Render benchmark | `bench/results/render-scale.json` | 47f164385fd9 | 2026-07-16 21:59 UTC | +| Data engine benchmark | `bench/results/data-results.json` | 6154cca220eb | 2026-07-16 22:05 UTC | +| Formula engine benchmark | `bench/results/formula-results.json` | ed66b6d7c95c | 2026-10-02 18:09 UTC | +| Formula engines (analysis capture) | `bench/results/full-engine-results.json` | a50a897e9f93 | 2026-10-04 11:20 UTC | + ## Matched local regression check Timing comparisons run deliberately on a controlled local machine, not as a required CI job. Capture ten fresh matched rounds, retain every raw sample, and compare the fresh artifact with the committed baseline. Any unapproved slowdown fails the local command. @@ -17,6 +28,8 @@ bun run src/check.ts --baseline results/render-baseline.json --fresh results/ren A result is a regression decision only when that final baseline check passes on the declared power mode and concurrency. `bench:verify` remains a smoke and safety-ceiling check. +The committed baseline at `bench/results/render-baseline.json` pins the harness source hashes, the declared sampling, and the runner (OS, CPU, Bun, Node, Chromium, power mode, concurrency). It is therefore machine- and harness-pinned: it must be re-recorded whenever a harness file, the sampling flags, the protocol version, or the runner changes, and the release capture does that as its last step by re-recording ten controlled rounds into `bench/results/render-baseline-raw.json` and promoting that artifact over the committed baseline. Until the re-recorded baseline is committed, `bench:check` fails closed on the fingerprint mismatch instead of comparing unrelated measurements. + ## Render benchmark
Validated evidence. 1100/1120 engine/scenario/round runs completed across 4 workbook sizes; every completed run passed its correctness checkpoints; 20 runs did not finish and are shown as such.
@@ -644,11 +657,20 @@ Both engines drive identical scripted interactions in a controlled browser. Pick
Every bar in a panel shares the ruler's scale (speed is logarithmic - each tick is 10x), so lengths compare across rows as well as within them. Bright numbers are the median run; faded numbers are the p95 run (speed) or the interaction's heap delta (memory). Rows marked as not completed are runs the engine could not finish - the recorded failure (crash, timeout, or failed correctness checkpoint) lives in the raw artifact.
+Every expected cell carries either a timing or its recorded failure. These cells did not finish and have no timing: + +| Engine | Size | Scenario | Stage | Recorded error | Cells | +| --- | ---: | --- | --- | --- | ---: | +| handsontable | 1M | altering.insert-5-rows-top | warmup | RangeError: Maximum call stack size exceeded | 10 | +| handsontable | 1M | altering.remove-5-rows-top | warmup | RangeError: Maximum call stack size exceeded | 10 | + +
Methodology - what each scenario does

Live grid in controlled headless Chromium. Ten counterbalanced rounds; all fourteen scenarios run warm per mount, and the fixture is rebuilt after any failure so crashes cannot leak state. Every scenario must prove its effect (scroll really moved, editor really opened, rows really changed) or it fails.

-

Bright = median round, faded = p95 round. Did not complete = recorded crash, timeout, or failed checkpoint - never a timing.

+

Each measured sample repeats one logical action until it covers at least 100 ms of measured time, so timer resolution and scheduler jitter cannot decide a sample. Every raw sample is retained in the artifact.

+

Bright = median round, faded = p95 round. Did not complete = recorded crash, timeout, or failed checkpoint - never a timing. The failure table names the stage and the recorded error behind every missing cell.

scroll-down.top-left
From the origin, jump-scroll 50 px down: a fresh row band enters the viewport and must paint.
scroll-down.middle
The same 50 px jump starting from the vertical middle of the scroll range.
@@ -1065,35 +1087,15 @@ bun run --filter @sheetwrite/bench bench:formula ### Default and full formula engines -
Validated evidence. Eight shared workloads have 51 paired rounds per engine. The full engine also has eight checked analysis workloads.
+
Validated evidence. Both engines ran the shared workload matrix. The paired-ratio capture is not published on this page yet, so no engine-to-engine ratio is claimed here.
-
Matched capture
2026-10-04 11:18 UTC
-
Matched commit
a50a897e9f93 clean worktree
Analysis capture
2026-10-04 11:20 UTC
Analysis commit
a50a897e9f93 clean worktree
-
Machine
12th Gen Intel(R) Core(TM) i9-12900H · Linux 7.2.8-1-cachyos · x64 · Bun 1.4.2
The default engine is `@sheetwrite/wasm`. The full engine is `@sheetwrite/formulas`. Select it with `initSheetwrite(undefined, formulas)`. Each app uses one engine. -The first capture ran all 53 shared workloads on each engine in sequence. Eight rows were more than 5% slower with the full engine. We checked those eight rows with matched rounds. Each pair runs default, then full. Each timed sample uses a fresh Bun process and one untimed warmup fixture. The runner used CPU 4 and one concurrent capture. - -No shared workload tested in the matched rounds had a median paired ratio more than 5% slower with the full engine. This check covers the eight flagged rows, not a new matched run of all 53 rows. - -The ratio is full time divided by default time within each pair. The table shows the median of 51 paired ratios. The spread is the interpolated 10th to 90th percentile of those ratios. It is not a confidence interval. The time columns are the median times for each engine. A ratio below 1 means the full engine took less time. Wide spreads and very short operations limit what this sample can show. - -| Shared workload | Size | Default median ms | Full median ms | Paired median ratio | Ratio p10–p90 | -| --- | ---: | ---: | ---: | ---: | ---: | -| independent-parse-load | 1,000 | 2.0895 | 2.6377 | 1.0149 | 0.3786–3.5129 | -| independent-first-recompute | 1,000 | 0.7200 | 0.7008 | 0.9939 | 0.2959–2.3024 | -| independent-parse-load | 10,000 | 13.4585 | 13.8707 | 1.0095 | 0.8935–1.1378 | -| independent-first-recompute | 10,000 | 26.5895 | 26.3255 | 0.9859 | 0.9088–1.0561 | -| linear-chain | 8 | 0.1210 | 0.1215 | 1.0113 | 0.9194–1.1899 | -| wide-fan-out-edit | 1,000 | 0.4538 | 0.4473 | 0.9849 | 0.8534–1.1019 | -| scalar-edit-affects-0 | 1,000 | 0.0107 | 0.0107 | 0.9957 | 0.8127–1.2054 | -| vlookup-many | 1,000 | 5.7530 | 5.5446 | 0.9398 | 0.7541–1.4022 | - #### Analysis workloads Each workload has 1,000 formulas. It has one untimed warmup and five measured samples. Each sample uses a fresh store. The timer measures first recompute. Setup and output reads are outside the timer. Every result is read and summed as a checked checksum. The default engine does not run these analysis functions. @@ -1358,3 +1360,13 @@ The landing-page data comes from a separate browser benchmark. The Delivery size
IncreaseDecreasescripts/size-history.json
+ +## Pending local evidence + +These protocols have no validated artifact in this environment yet, so no numbers are published for them. + +| Artifact | Status | Reproduce with | +| --- | --- | --- | +| `bench/results/core-paths-results.json` | artifact is missing | `bun run --filter @sheetwrite/bench bench:core-paths` | +| `bench/results/full-engine-matched-results.json` | Matched engine evidence is invalid: Matched engine protocol mismatch | `bun run --filter @sheetwrite/bench bench:formula:matched` | +| `bench/results/xlsx-results.json` | artifact has no clean-tree protocol stamp (commit, timestamp, dirty=false), so freshness cannot be established | `bun run --filter @sheetwrite/bench bench:xlsx -- --baseline-root ` | diff --git a/docs/src/styles/site.css b/docs/src/styles/site.css index 5940c420..558f0a67 100644 --- a/docs/src/styles/site.css +++ b/docs/src/styles/site.css @@ -1561,7 +1561,8 @@ body .sw-code-popover__accessible-signature { /* Evidence page callouts. */ .evidence-available, -.evidence-unavailable { +.evidence-unavailable, +.evidence-mixed { margin: 0 0 1.2rem; border: 1px solid color-mix(in srgb, var(--sw-accent) 30%, var(--sw-border)); border-radius: 0.55rem; @@ -1576,6 +1577,12 @@ body .sw-code-popover__accessible-signature { color: var(--sw-fg-muted); } +/* Capture vintages differ: a caution tone, not the "validated" accent. */ +.evidence-mixed { + border-color: color-mix(in srgb, var(--sw-state-pending) 45%, var(--sw-border)); + background: color-mix(in srgb, var(--sw-state-pending) 6%, var(--sw-surface-1)); +} + .error-catalog { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); diff --git a/scripts/docs.ts b/scripts/docs.ts index 1d2b28a3..cc633f20 100644 --- a/scripts/docs.ts +++ b/scripts/docs.ts @@ -1310,6 +1310,10 @@ interface RenderEvidenceResult { madMs: number; memory: { beforeBytes: number; afterBytes: number; deltaBytes: number }; validation: Array<{ passed: boolean }>; + /** Recorded only on failures: where the scenario stopped and what it reported. */ + stage?: string; + errorClass?: string; + message?: string; } interface ValidatedRenderEvidence { @@ -1530,6 +1534,120 @@ function validateRenderArtifact(value: Record): ValidatedRender return artifact as ValidatedRenderEvidence; } +interface CorePathsEvidence { + schemaVersion: number; + protocol: string; + mode: string; + metadata: CaptureMeta; + toolchain: { bun: string; nodeCompat: string; cpu: string }; + methodology: { timing: string; scope: string }; + scale: Record; + scenarios: Array<{ + id: string; + unit: string; + samplesMs: number[]; + timing: { median: number; p95: number; iters: number }; + counters: Record; + observations?: Record; + variants?: Array<{ + id: string; + samplesMs: number[]; + timing: { median: number; p95: number; iters: number }; + }>; + validation: string; + }>; +} + +/** + * Full-mode core-path capture. The artifact is written by `bench:core-paths`; + * only a complete scenario matrix with finite timings and a written validation + * checkpoint per scenario is publishable. + */ +function validateCorePathsArtifact(value: Record): CorePathsEvidence | string { + const artifact = value as unknown as Partial; + if ( + artifact.schemaVersion !== 1 || + artifact.protocol !== "sheetwrite-core-paths-v1" || + artifact.mode !== "full" + ) { + return "artifact is not a full-mode core-path capture"; + } + if (!Array.isArray(artifact.scenarios) || artifact.scenarios.length === 0) { + return "artifact carries no core-path scenarios"; + } + for (const scenario of artifact.scenarios) { + const timings = [scenario.timing, ...(scenario.variants ?? []).map((entry) => entry.timing)]; + if ( + typeof scenario.unit !== "string" || + scenario.unit.length === 0 || + typeof scenario.validation !== "string" || + scenario.validation.length === 0 || + timings.some( + (timing) => + !Number.isFinite(timing?.median) || + !Number.isFinite(timing?.p95) || + !Number.isInteger(timing?.iters) || + timing.iters < 1, + ) + ) { + return "core-path scenarios carry an unvalidated or non-finite timing"; + } + } + return artifact as CorePathsEvidence; +} + +interface XlsxEvidence { + protocol: string; + mode: string; + metadata: CaptureMeta; + rounds: number; + maxBaselineRatio: number; + fixture: { path: string; bytes: number; sha256: string; producer: string }; + toolchain: { bun: string; node: string; cpu: string; baselineCommit?: string }; + summaries: Array<{ + engine: string; + operation: string; + scenario: string; + durationMs: { median: number; p95: number; iters: number }; + maxRssBytes: { median: number; p95: number; max: number }; + }>; + comparisons: Array<{ + operation: string; + scenario: string; + medianWallRatio: number; + peakRssRatio: number; + }>; +} + +function validateXlsxArtifact(value: Record): XlsxEvidence | string { + const artifact = value as unknown as Partial; + if (artifact.protocol !== "sheetwrite-xlsx-codec-v1" || artifact.mode !== "full") { + return "artifact is not a full-mode XLSX codec capture"; + } + const rounds = artifact.rounds; + if ( + typeof rounds !== "number" || + !Number.isInteger(rounds) || + rounds < 1 || + !Array.isArray(artifact.summaries) || + artifact.summaries.length === 0 || + !Array.isArray(artifact.comparisons) + ) { + return "XLSX capture is missing its rounds, summaries, or comparisons"; + } + if ( + artifact.summaries.some( + (summary) => + !Number.isFinite(summary?.durationMs?.median) || + !Number.isFinite(summary?.durationMs?.p95) || + !Number.isFinite(summary?.maxRssBytes?.max), + ) + ) { + return "XLSX summaries carry non-finite samples"; + } + return artifact as XlsxEvidence; +} + interface DataEvidence { protocolVersion: number; mode: string; @@ -1775,7 +1893,8 @@ const RENDER_METHOD = benchMethod( "Methodology - what each scenario does", [ "Live grid in controlled headless Chromium. Ten counterbalanced rounds; all fourteen scenarios run warm per mount, and the fixture is rebuilt after any failure so crashes cannot leak state. Every scenario must prove its effect (scroll really moved, editor really opened, rows really changed) or it fails.", - "Bright = median round, faded = p95 round. Did not complete = recorded crash, timeout, or failed checkpoint - never a timing.", + "Each measured sample repeats one logical action until it covers at least 100 ms of measured time, so timer resolution and scheduler jitter cannot decide a sample. Every raw sample is retained in the artifact.", + "Bright = median round, faded = p95 round. Did not complete = recorded crash, timeout, or failed checkpoint - never a timing. The failure table names the stage and the recorded error behind every missing cell.", ], [ [ @@ -2020,6 +2139,265 @@ export function runCompletionSummary(results: ReadonlyArray<{ status: string }>) return { successes, total: results.length, failures: results.length - successes }; } +/** One validated artifact that the page publishes numbers from. */ +interface EvidenceCapture { + readonly label: string; + readonly source: string; + readonly meta: CaptureMeta; +} + +/** Every loaded artifact states its capture stamp as either `meta` or `metadata`. */ +type CapturedEvidence = + | { evidence: { meta?: CaptureMeta; metadata?: CaptureMeta }; source: string } + | EvidenceState; + +function formatCaptureTime(timestamp: string): string { + return timestamp.slice(0, 16).replace("T", " "); +} + +/** + * One capture header for the whole page. A page assembled from artifacts of one + * commit states that single point-in-time capture; a page assembled from + * different commits lists every vintage instead of letting each section's own + * commit line hide the mismatch. + */ +function renderCaptureHeader(captures: readonly EvidenceCapture[]): string[] { + const first = captures[0]; + if (first === undefined) return []; + const commits = new Set(captures.map((capture) => capture.meta.commit)); + const timestamps = captures.map((capture) => capture.meta.timestamp).sort(); + const earliest = timestamps[0]; + const latest = timestamps.at(-1); + if (earliest === undefined || latest === undefined) return []; + const window = + formatCaptureTime(earliest) === formatCaptureTime(latest) + ? formatCaptureTime(earliest) + : `${formatCaptureTime(earliest)} → ${formatCaptureTime(latest)}`; + if (commits.size === 1) { + return [ + "## Capture", + "", + '
', + `
Captured
${window} UTC
`, + `
Commit
${first.meta.commit.slice(0, 12)} clean worktree
`, + `
Artifacts
${captures.length} validated captures from that one commit; each section states its own environment (browser or runtime) and protocol
`, + "
", + "", + ]; + } + const vintages = [...captures].sort( + (left, right) => + left.meta.timestamp.localeCompare(right.meta.timestamp) || + left.label.localeCompare(right.label), + ); + return [ + "## Capture", + "", + '
Mixed capture vintages. The sections below were captured at different commits. Each section is internally consistent and each one names its own commit; the page as a whole is not a single point-in-time measurement.
', + "", + "| Section | Artifact | Commit | Captured |", + "| --- | --- | --- | --- |", + ...vintages.map( + (capture) => + `| ${capture.label} | \`${capture.source}\` | ${capture.meta.commit.slice(0, 12)} | ${formatCaptureTime(capture.meta.timestamp)} UTC |`, + ), + "", + ]; +} + +/** + * Failures grouped by the cell that produced them. A run that did not finish is + * never converted into a timing, so the failure list is part of the evidence: + * it names the stage and the recorded error behind every missing cell. + */ +function renderFailureTable(evidence: ValidatedRenderEvidence): string[] { + const failures = evidence.results.filter((result) => result.status !== "success"); + if (failures.length === 0) return []; + const groups = new Map< + string, + { engine: string; rows: number; scenario: string; stage: string; error: string; cells: number } + >(); + for (const failure of failures) { + const stage = failure.stage ?? "unrecorded"; + const error = [failure.errorClass, failure.message] + .filter((part): part is string => typeof part === "string" && part.length > 0) + .join(": ") + .replaceAll("|", "\\|"); + const key = [failure.engine, failure.rows, failure.scenarioId, stage, error].join("\u0000"); + const group = groups.get(key); + if (group === undefined) { + groups.set(key, { + engine: failure.engine, + rows: failure.rows, + scenario: failure.scenarioId, + stage, + error: error.length > 0 ? error : "not recorded", + cells: 1, + }); + } else { + group.cells += 1; + } + } + const ordered = [...groups.values()].sort( + (left, right) => + left.engine.localeCompare(right.engine) || + left.rows - right.rows || + left.scenario.localeCompare(right.scenario) || + left.stage.localeCompare(right.stage), + ); + return [ + "", + "Every expected cell carries either a timing or its recorded failure. These cells did not finish and have no timing:", + "", + "| Engine | Size | Scenario | Stage | Recorded error | Cells |", + "| --- | ---: | --- | --- | --- | ---: |", + ...ordered.map( + (group) => + `| ${group.engine} | ${fmtRows(group.rows)} | ${group.scenario} | ${group.stage} | ${group.error} | ${group.cells} |`, + ), + "", + ]; +} + +/** Markdown table cells cannot carry a raw pipe. */ +const markdownCell = (text: string): string => text.replaceAll("|", "\\|"); + +/** + * Core write-path evidence: the packed block writes, CSV import, large restore, + * and paged residency work that the core hot paths changed. Every scenario + * carries the checkpoint it proved, so a timing is never published alone. + */ +function renderCorePathsSection(evidence: CorePathsEvidence, source: string): string[] { + const counters = evidence.scenarios.filter( + (scenario) => Object.keys(scenario.counters).length > 0, + ); + return [ + "## Core write path", + "", + `
Validated evidence. ${evidence.scenarios.length} store and document scenarios at production sizes; every scenario passed its own correctness checkpoint before its timing was accepted.
`, + "", + '
', + `
Captured
${formatCaptureTime(evidence.metadata.timestamp)} UTC
`, + `
Commit
${evidence.metadata.commit.slice(0, 12)} clean worktree
`, + `
Environment
Bun ${evidence.toolchain.bun} · Node compatibility ${evidence.toolchain.nodeCompat} · ${html(evidence.toolchain.cpu)}
`, + `
Raw artifact
${source}
`, + "
", + "", + `Scale: ${Object.entries(evidence.scale) + .map(([name, value]) => `${name}=${value.toLocaleString("en-US")}`) + .join(", ")}.`, + "", + "| Scenario | One timed iteration | Median | p95 | Samples |", + "| --- | --- | ---: | ---: | ---: |", + ...evidence.scenarios.flatMap((scenario) => [ + `| \`${scenario.id}\` | ${markdownCell(scenario.unit)} | ${fmtMs(scenario.timing.median)} | ${fmtMs(scenario.timing.p95)} | ${scenario.timing.iters} |`, + ...(scenario.variants ?? []).map( + (variant) => + `| \`${scenario.id}:${variant.id}\` | variant measured in the same run | ${fmtMs(variant.timing.median)} | ${fmtMs(variant.timing.p95)} | ${variant.timing.iters} |`, + ), + ]), + "", + ...(counters.length === 0 + ? [] + : [ + "Counters recorded by the same runs (non-time evidence):", + "", + ...counters.map( + (scenario) => + `- \`${scenario.id}\` — ${Object.entries(scenario.counters) + .map(([name, value]) => `${name}=${value.toLocaleString("en-US")}`) + .join(", ")}`, + ), + "", + ]), + "
Per-scenario correctness checkpoint", + "", + ...evidence.scenarios.map( + (scenario) => `- \`${scenario.id}\` — ${markdownCell(scenario.validation)}`, + ), + "", + "
", + "", + "Reproduce with:", + "", + '```sh verify title="Core write-path evidence"', + "bun run --filter @sheetwrite/bench bench:core-paths", + "```", + "", + ]; +} + +/** + * XLSX codec evidence: the current tree against the codec at the recorded + * baseline commit, both reading the same checked fixture. A ratio below 1 means + * the current codec is faster; peak RSS is the largest raw process peak. + */ +function renderXlsxSection(evidence: XlsxEvidence, source: string): string[] { + const summaryFor = ( + engine: string, + operation: string, + scenario: string, + ): XlsxEvidence["summaries"][number] | undefined => + evidence.summaries.find( + (summary) => + summary.engine === engine && + summary.operation === operation && + summary.scenario === scenario, + ); + const baselineCommit = evidence.toolchain.baselineCommit; + return [ + "## XLSX codec benchmark", + "", + `
Validated evidence. ${evidence.comparisons.length > 0 ? `${evidence.comparisons.length} compared operations` : `${evidence.summaries.length} measured operations`} over ${evidence.rounds} process-isolated rounds each, from one checked corpus fixture.
`, + "", + '
', + `
Captured
${formatCaptureTime(evidence.metadata.timestamp)} UTC
`, + `
Commit
${evidence.metadata.commit.slice(0, 12)} clean worktree
`, + `
Fixture
${evidence.fixture.path} · ${evidence.fixture.bytes.toLocaleString("en-US")} bytes · ${evidence.fixture.sha256.slice(0, 12)} · ${html(evidence.fixture.producer)}
`, + `
Environment
Bun ${evidence.toolchain.bun} · Node ${evidence.toolchain.node} · ${html(evidence.toolchain.cpu)}
`, + `
Raw artifact
${source}
`, + "
", + "", + ...(baselineCommit === undefined + ? [ + "This capture ran the current tree only, so no codec comparison is published.", + "", + "| Operation | Scenario | Median | p95 | Peak RSS |", + "| --- | --- | ---: | ---: | ---: |", + ...evidence.summaries + .filter((summary) => summary.engine === "current") + .map( + (summary) => + `| ${summary.operation} | ${summary.scenario} | ${fmtMs(summary.durationMs.median)} | ${fmtMs(summary.durationMs.p95)} | ${fmtMb(summary.maxRssBytes.max)} |`, + ), + "", + ] + : [ + `The baseline codec is the same operation at commit ${baselineCommit.slice(0, 12)}, built from that commit and measured in the same session. Ratio = current ÷ baseline; below 1 is faster.`, + "", + "| Operation | Scenario | Current median | Baseline median | Ratio | Current peak RSS | Baseline peak RSS | RSS ratio |", + "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |", + ...evidence.comparisons.flatMap((comparison) => { + const current = summaryFor("current", comparison.operation, comparison.scenario); + const baseline = summaryFor("baseline", comparison.operation, comparison.scenario); + if (current === undefined || baseline === undefined) return []; + return [ + `| ${comparison.operation} | ${comparison.scenario} | ${fmtMs(current.durationMs.median)} | ${fmtMs(baseline.durationMs.median)} | ${comparison.medianWallRatio.toFixed(2)}× | ${fmtMb(current.maxRssBytes.max)} | ${fmtMb(baseline.maxRssBytes.max)} | ${comparison.peakRssRatio.toFixed(2)}× |`, + ]; + }), + "", + ]), + "Reproduce with a checkout of the baseline commit, built and passed as the comparison root:", + "", + '```sh verify title="XLSX codec evidence"', + `git worktree add ../sheetwrite-xlsx-baseline ${baselineCommit ?? ""}`, + "cd ../sheetwrite-xlsx-baseline && bun install --frozen-lockfile && bun run build:packages", + "cd - && bun run --filter @sheetwrite/bench bench:xlsx -- --baseline-root ../sheetwrite-xlsx-baseline", + "```", + "", + ]; +} + export async function renderEvidencePage(sizeHistoryOverride?: SizeHistory): Promise { const scale = await loadEvidence( "bench/results/render-scale.json", @@ -2046,6 +2424,32 @@ export async function renderEvidencePage(sizeHistoryOverride?: SizeHistory): Pro "bun run --filter @sheetwrite/bench bench:formula:matched", validateMatchedEngineArtifact, ); + const corePaths = await loadEvidence( + "bench/results/core-paths-results.json", + "bun run --filter @sheetwrite/bench bench:core-paths", + validateCorePathsArtifact, + ); + const xlsx = await loadEvidence( + "bench/results/xlsx-results.json", + "bun run --filter @sheetwrite/bench bench:xlsx -- --baseline-root ", + validateXlsxArtifact, + ); + const captures = ( + [ + { label: "Render benchmark", loaded: scale }, + { label: "Data engine benchmark", loaded: data }, + { label: "Core write path", loaded: corePaths }, + { label: "Formula engine benchmark", loaded: formula }, + { label: "Formula engines (analysis capture)", loaded: fullEngine }, + { label: "Formula engines (matched capture)", loaded: matchedEngine }, + { label: "XLSX codec benchmark", loaded: xlsx }, + ] satisfies Array<{ label: string; loaded: CapturedEvidence }> + ).flatMap(({ label, loaded }) => { + if (!("evidence" in loaded)) return []; + const evidence = loaded.evidence; + const captured = "meta" in evidence ? evidence.meta : evidence.metadata; + return [{ label, source: loaded.source, meta: captured }]; + }); const sizeHistory = sizeHistoryOverride ?? validateSizeHistory( @@ -2059,6 +2463,7 @@ export async function renderEvidencePage(sizeHistoryOverride?: SizeHistory): Pro ).trimEnd(), "Every number on this page comes from a validated local protocol artifact captured on a clean tree; nothing is published from an unvalidated or protocol-mismatched artifact. Every expected cell carries either a validated timing or its recorded failure - a run that did not complete is shown as a failure, never converted into a timing.", "", + ...renderCaptureHeader(captures), "## Matched local regression check", "", "Timing comparisons run deliberately on a controlled local machine, not as a required CI job. Capture ten fresh matched rounds, retain every raw sample, and compare the fresh artifact with the committed baseline. Any unapproved slowdown fails the local command.", @@ -2072,6 +2477,8 @@ export async function renderEvidencePage(sizeHistoryOverride?: SizeHistory): Pro "", "A result is a regression decision only when that final baseline check passes on the declared power mode and concurrency. `bench:verify` remains a smoke and safety-ceiling check.", "", + "The committed baseline at `bench/results/render-baseline.json` pins the harness source hashes, the declared sampling, and the runner (OS, CPU, Bun, Node, Chromium, power mode, concurrency). It is therefore machine- and harness-pinned: it must be re-recorded whenever a harness file, the sampling flags, the protocol version, or the runner changes, and the release capture does that as its last step by re-recording ten controlled rounds into `bench/results/render-baseline-raw.json` and promoting that artifact over the committed baseline. Until the re-recorded baseline is committed, `bench:check` fails closed on the fingerprint mismatch instead of comparing unrelated measurements.", + "", "## Render benchmark", "", ]; @@ -2092,6 +2499,7 @@ export async function renderEvidencePage(sizeHistoryOverride?: SizeHistory): Pro "", "", renderBenchWidget(evidence), + ...renderFailureTable(evidence), "", RENDER_METHOD, "", @@ -2216,6 +2624,11 @@ export async function renderEvidencePage(sizeHistoryOverride?: SizeHistory): Pro } else { pending.push(data); } + if ("evidence" in corePaths) { + lines.push(...renderCorePathsSection(corePaths.evidence, corePaths.source)); + } else { + pending.push(corePaths); + } lines.push("## Formula engine benchmark", ""); if ("evidence" in formula) { const { evidence, source } = formula; @@ -2290,12 +2703,23 @@ export async function renderEvidencePage(sizeHistoryOverride?: SizeHistory): Pro } else { pending.push(formula); } - if ("evidence" in fullEngine && "evidence" in matchedEngine) { - lines.push(renderFullEngineEvidence(fullEngine.evidence, matchedEngine.evidence)); + if ("evidence" in fullEngine) { + lines.push( + renderFullEngineEvidence( + fullEngine.evidence, + "evidence" in matchedEngine ? matchedEngine.evidence : undefined, + ), + ); + if (!("evidence" in matchedEngine)) pending.push(matchedEngine); } else { - if (!("evidence" in fullEngine)) pending.push(fullEngine); + pending.push(fullEngine); if (!("evidence" in matchedEngine)) pending.push(matchedEngine); } + if ("evidence" in xlsx) { + lines.push(...renderXlsxSection(xlsx.evidence, xlsx.source)); + } else { + pending.push(xlsx); + } lines.push("## Delivery size", ""); const latestRelease = sizeHistory.releases.at(-1); if (latestRelease !== undefined) { diff --git a/scripts/full-engine-docs.ts b/scripts/full-engine-docs.ts index 6a2f830f..a91f8736 100644 --- a/scripts/full-engine-docs.ts +++ b/scripts/full-engine-docs.ts @@ -1,9 +1,17 @@ import { type FullEngineResult, validateFullEngineResult } from "../bench/src/full-engine-bench.js"; import { type MatchedEngineResult, + type MatchedEngineRow, validateMatchedEngineResult, } from "../bench/src/matched-engine-bench.js"; +/** + * Largest paired-ratio p10–p90 span (p90/p10) that still supports a ratio + * claim. Identical shared work is the control: when the protocol is quiet, the + * paired ratio stays near 1 and its spread stays narrow. + */ +const MATCHED_SPREAD_CEILING = 1.5; + export function validateFullEngineArtifact( value: Record, ): FullEngineResult | string { @@ -26,48 +34,9 @@ export function validateMatchedEngineArtifact( return `Matched engine evidence is invalid: ${error instanceof Error ? error.message : String(error)}`; } } -export function renderFullEngineEvidence( - full: FullEngineResult, - matched: MatchedEngineResult, -): string { - validateFullEngineResult(full); - validateMatchedEngineResult(matched); - for (const engine of full.engines) { - const path = `packages/${engine.engine === "full" ? "formulas" : "wasm"}/pkg/sheetwrite_wasm_bg.wasm`; - if (engine.wasm.sha256 !== matched.sourceFiles[path]) - throw new Error("Matched and analysis captures use different engine binaries"); - } - const slowerCount = matched.rows.filter((row) => row.ratioStat.median > 1.05).length; - const lines = [ - "### Default and full formula engines", - "", - `
Validated evidence. Eight shared workloads have ${matched.rounds} paired rounds per engine. The full engine also has eight checked analysis workloads.
`, - "", - '
', - `
Matched capture
${matched.meta.timestamp.slice(0, 16).replace("T", " ")} UTC
`, - `
Matched commit
${matched.meta.commit.slice(0, 12)} clean worktree
`, - `
Analysis capture
${full.meta.timestamp.slice(0, 16).replace("T", " ")} UTC
`, - `
Analysis commit
${full.meta.commit.slice(0, 12)} clean worktree
`, - `
Machine
${matched.runner.cpu} · Linux ${matched.runner.kernel} · ${matched.runner.arch} · Bun ${matched.runner.bun}
`, - "
", - "", - "The default engine is `@sheetwrite/wasm`. The full engine is `@sheetwrite/formulas`. Select it with `initSheetwrite(undefined, formulas)`. Each app uses one engine.", - "", - "The first capture ran all 53 shared workloads on each engine in sequence. Eight rows were more than 5% slower with the full engine. We checked those eight rows with matched rounds. Each pair runs default, then full. Each timed sample uses a fresh Bun process and one untimed warmup fixture. The runner used CPU 4 and one concurrent capture.", - "", - slowerCount === 0 - ? "No shared workload tested in the matched rounds had a median paired ratio more than 5% slower with the full engine. This check covers the eight flagged rows, not a new matched run of all 53 rows." - : `${slowerCount} shared workloads tested in the matched rounds had a median paired ratio more than 5% slower with the full engine.`, - "", - `The ratio is full time divided by default time within each pair. The table shows the median of ${matched.rounds} paired ratios. The spread is the interpolated 10th to 90th percentile of those ratios. It is not a confidence interval. The time columns are the median times for each engine. A ratio below 1 means the full engine took less time. Wide spreads and very short operations limit what this sample can show.`, - "", - "| Shared workload | Size | Default median ms | Full median ms | Paired median ratio | Ratio p10–p90 |", - "| --- | ---: | ---: | ---: | ---: | ---: |", - ...matched.rows.map( - (row) => - `| ${row.id} | ${row.size.toLocaleString("en-US")} | ${row.defaultStat.median.toFixed(4)} | ${row.fullStat.median.toFixed(4)} | ${row.ratioStat.median.toFixed(4)} | ${row.ratioP10.toFixed(4)}–${row.ratioP90.toFixed(4)} |`, - ), - "", + +function analysisSection(full: FullEngineResult): string[] { + return [ "#### Analysis workloads", "", "Each workload has 1,000 formulas. It has one untimed warmup and five measured samples. Each sample uses a fresh store. The timer measures first recompute. Setup and output reads are outside the timer. Every result is read and summed as a checked checksum. The default engine does not run these analysis functions.", @@ -101,8 +70,96 @@ export function renderFullEngineEvidence( "bun run bench:formula:matched", "```", "", + ]; +} + +/** + * The analysis capture (the shared matrix on both engines, WASM sizes, Node + * cold init) stands on its own. The matched paired-ratio table is added when + * that second capture is present and valid; without it the page still + * publishes the measured analysis evidence and names the capture that is + * missing. + */ +export function renderFullEngineEvidence( + full: FullEngineResult, + matched?: MatchedEngineResult, +): string { + validateFullEngineResult(full); + const lines = ["### Default and full formula engines", ""]; + if (matched === undefined) { + lines.push( + `
Validated evidence. Both engines ran the shared workload matrix. The paired-ratio capture is not published on this page yet, so no engine-to-engine ratio is claimed here.
`, + "", + '
', + `
Analysis capture
${full.meta.timestamp.slice(0, 16).replace("T", " ")} UTC
`, + `
Analysis commit
${full.meta.commit.slice(0, 12)} clean worktree
`, + "
", + "", + "The default engine is `@sheetwrite/wasm`. The full engine is `@sheetwrite/formulas`. Select it with `initSheetwrite(undefined, formulas)`. Each app uses one engine.", + "", + ...analysisSection(full), + "The landing-page data comes from a separate browser benchmark. The Delivery size section below records published package sizes. Neither data set has an engine comparison field. They are unchanged by this local engine capture.", + "", + ); + return lines.join("\n"); + } + + validateMatchedEngineResult(matched); + for (const engine of full.engines) { + const path = `packages/${engine.engine === "full" ? "formulas" : "wasm"}/pkg/sheetwrite_wasm_bg.wasm`; + if (engine.wasm.sha256 !== matched.sourceFiles[path]) + throw new Error("Matched and analysis captures use different engine binaries"); + } + const slowerCount = matched.rows.filter((row) => row.ratioStat.median > 1.05).length; + const sampleIterations = (row: MatchedEngineRow): { min: number; max: number } => + row.samples.reduce( + (range, sample) => ({ + min: Math.min(range.min, sample.iterations), + max: Math.max(range.max, sample.iterations), + }), + { min: Number.POSITIVE_INFINITY, max: 0 }, + ); + const widestSpread = matched.rows.reduce( + (worst, row) => Math.max(worst, row.ratioP90 / row.ratioP10), + 0, + ); + lines.push( + `
Validated evidence. Eight shared workloads have ${matched.rounds} paired rounds per engine. The full engine also has eight checked analysis workloads.
`, + "", + '
', + `
Matched capture
${matched.meta.timestamp.slice(0, 16).replace("T", " ")} UTC
`, + `
Matched commit
${matched.meta.commit.slice(0, 12)} clean worktree
`, + `
Analysis capture
${full.meta.timestamp.slice(0, 16).replace("T", " ")} UTC
`, + `
Analysis commit
${full.meta.commit.slice(0, 12)} clean worktree
`, + `
Machine
${matched.runner.cpu} · Linux ${matched.runner.kernel} · ${matched.runner.arch} · Bun ${matched.runner.bun}
`, + "
", + "", + "The default engine is `@sheetwrite/wasm`. The full engine is `@sheetwrite/formulas`. Select it with `initSheetwrite(undefined, formulas)`. Each app uses one engine.", + "", + `The first capture ran all 53 shared workloads on each engine in sequence. Eight rows were more than 5% slower with the full engine. Those eight rows were checked with matched rounds. Each pair runs default, then full, in separate Bun processes. A timed sample starts with ${matched.warmupFixtures} untimed warm-up fixtures and then repeats the workload until at least ${matched.minimumSampleDurationMs} ms of measured time; the sample is the per-iteration mean, so a short operation is measured over many repetitions instead of one cold run. The runner used CPU 4 and one concurrent capture.`, + "", + slowerCount === 0 + ? "No shared workload tested in the matched rounds had a median paired ratio more than 5% slower with the full engine. This check covers the eight flagged rows, not a new matched run of all 53 rows." + : `${slowerCount} shared workloads tested in the matched rounds had a median paired ratio more than 5% slower with the full engine.`, + "", + `The ratio is full time divided by default time within each pair. The table shows the median of ${matched.rounds} paired ratios. The spread is the interpolated 10th to 90th percentile of those ratios. It is not a confidence interval. The time columns are the median times for each engine. A ratio below 1 means the full engine took less time.`, + "", + widestSpread <= MATCHED_SPREAD_CEILING + ? `Identical shared work is the control for this protocol: the widest paired-ratio p10–p90 span across the eight rows is ${widestSpread.toFixed(2)}× (ceiling ${MATCHED_SPREAD_CEILING.toFixed(2)}×), so each median ratio is supported by its paired samples.` + : `At least one row's paired-ratio p10–p90 span is ${widestSpread.toFixed(2)}× (ceiling ${MATCHED_SPREAD_CEILING.toFixed(2)}×). Rows above the ceiling are marked and their median ratio should not be read as a measured difference.`, + "", + "| Shared workload | Size | Default median ms | Full median ms | Paired median ratio | Ratio p10–p90 | Iterations per sample |", + "| --- | ---: | ---: | ---: | ---: | ---: | ---: |", + ...matched.rows.map((row) => { + const iterations = sampleIterations(row); + const spread = row.ratioP90 / row.ratioP10; + const marker = spread > MATCHED_SPREAD_CEILING ? " ⚠︎" : ""; + return `| ${row.id}${marker} | ${row.size.toLocaleString("en-US")} | ${row.defaultStat.median.toFixed(4)} | ${row.fullStat.median.toFixed(4)} | ${row.ratioStat.median.toFixed(4)} | ${row.ratioP10.toFixed(4)}–${row.ratioP90.toFixed(4)} | ${iterations.min}–${iterations.max} |`; + }), + "", + ...analysisSection(full), "The landing-page data comes from a separate browser benchmark. The Delivery size section below records published package sizes. Neither data set has an engine comparison field. They are unchanged by this local engine capture.", "", - ]; + ); return lines.join("\n"); }