Skip to content

Add published-query benchmarks for Stannum and stored-vector GIN - #27

Open
bweis wants to merge 1 commit into
mainfrom
bench/tin-methodology
Open

bweis wants to merge 1 commit into
mainfrom
bench/tin-methodology

Conversation

@bweis

@bweis bweis commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Our existing performance comparisons use a small fixed query set and expression-index GIN. This adds a local entry point around PlanetScale’s pinned ParadeDB Benchmarker fork, replaying the published Wikipedia trace against Stannum and stored-vector GIN with normal planner settings. The upstream driver owns timing and traffic; our wrapper owns setup, correctness, provenance, and reporting.

The integration exposed two issues and retains regressions for both: PostgreSQL position truncation makes it unsuitable as Stannum’s raw-text phrase oracle, and the upstream update picker can select empty heap pages. Per-engine oracles now record cross-engine count differences explicitly, while the picker falls forward/wraps to a live tuple without weakening exactly-one-row accounting. Update runs verify all 906 counts remain unchanged afterward.

Validation: 98 Python harness tests, 6 attribution tests, 8 JavaScript adapter tests, and 7 real-server Go driver tests passed. Native ARM64 PostgreSQL 18.6 runs passed 100k-document mixed counts in both engine orders, 1k-document concurrent updates, and exhaustive same-engine top-10 checks for all 906 forms. Results, limitations, pinned identities, and failure history are recorded in docs/benchmarks/published-trace-results.md.

This establishes the replacement path; it does not delete unique mutation/VACUUM coverage or claim the full benchmark matrix is finished. The guide lists retirement gates for 585 lines of older launchers plus the read-only runner. No extension runtime code changed. No AWS resources were used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant