Skip to content

Capture heap visibility and reject incomparable benchmark trials - #38

Merged
bweis merged 2 commits into
mainfrom
bench/visibility-controls
Sep 19, 2026
Merged

bweis merged 2 commits into
mainfrom
bench/visibility-controls

Conversation

@bweis

@bweis bweis commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Benchmarks can otherwise attribute changed heap visibility to an optimization: identical query/settings pairs in our TIN experiments switched from a 16 ms heap scan to a sub-millisecond count path. Record actual visibility-map coverage around each local run and withhold paired ratios when initial coverage differs, evidence is missing, or read-only coverage drifts.

Capture untimed snapshots after setup VACUUM, before driver/warmup, and after the stopped container restarts. Preserve estimated live/dead tuples, maintenance/mutation counters, and table options. Mutation endpoints remain measured outcomes rather than required equal inputs. The report explicitly states that post-restart evidence is not an exact end-of-traffic snapshot or proof of cold-cache conditions.

Based on main after #31, retaining its resource reporting. Requires pg_visibility and fails setup if unavailable. Existing individual artifacts remain readable; historical paired runs need fresh visibility evidence to satisfy the stronger contract.

Validation: 124 Python benchmark tests, 171 source-header checks, and git diff checks pass. Native PostgreSQL18 smoke passed across insert, VACUUM, update, second VACUUM, and restart: coverage0/36 →36/36 →31/40 →40/40 →40/40. Read-only drift was rejected and mutation outcome accepted. This exercises actual capture SQL and comparison logic; it does not claim a full Docker-driver lifecycle run. Evidence is checked into docs/benchmarks/visibility-controls-smoke.json. Ready for review; merge remains gated on all updated-head CI checks passing.

@bweis
bweis force-pushed the bench/visibility-controls branch from 5afc502 to eb16245 Compare September 19, 2026 22:51
@bweis
bweis changed the base branch from bench/million-memory-pressure to main September 19, 2026 22:51
@bweis
bweis marked this pull request as ready for review September 19, 2026 22:57
@bweis
bweis merged commit 0e2c764 into main Sep 19, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant