Capture heap visibility and reject incomparable benchmark trials - #38
Merged
Merged
Conversation
bweis
force-pushed
the
bench/visibility-controls
branch
from
September 19, 2026 22:51
5afc502 to
eb16245
Compare
bweis
marked this pull request as ready for review
September 19, 2026 22:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmarks can otherwise attribute changed heap visibility to an optimization: identical query/settings pairs in our TIN experiments switched from a 16 ms heap scan to a sub-millisecond count path. Record actual visibility-map coverage around each local run and withhold paired ratios when initial coverage differs, evidence is missing, or read-only coverage drifts.
Capture untimed snapshots after setup VACUUM, before driver/warmup, and after the stopped container restarts. Preserve estimated live/dead tuples, maintenance/mutation counters, and table options. Mutation endpoints remain measured outcomes rather than required equal inputs. The report explicitly states that post-restart evidence is not an exact end-of-traffic snapshot or proof of cold-cache conditions.
Based on main after #31, retaining its resource reporting. Requires pg_visibility and fails setup if unavailable. Existing individual artifacts remain readable; historical paired runs need fresh visibility evidence to satisfy the stronger contract.
Validation: 124 Python benchmark tests, 171 source-header checks, and git diff checks pass. Native PostgreSQL18 smoke passed across insert, VACUUM, update, second VACUUM, and restart: coverage0/36 →36/36 →31/40 →40/40 →40/40. Read-only drift was rejected and mutation outcome accepted. This exercises actual capture SQL and comparison logic; it does not claim a full Docker-driver lifecycle run. Evidence is checked into docs/benchmarks/visibility-controls-smoke.json. Ready for review; merge remains gated on all updated-head CI checks passing.