#171: FTS Track 4 — substrate search path behind ?fts=v1#330
Merged
Conversation
…ed FNV-1a assets/js/search_substrate.js — the browser query pipeline as PURE functions (no DuckDB, no fetch, no DOM), so every piece of query logic is Node-testable and the explorer's ?fts=v1 wiring (Phase B) stays thin: - fnv1a32: 32-bit FNV-1a with a shift-decomposed multiply (no BigInt); parity with tools/build_search_index.py pinned by tests/search_fnv1a_regression.json (values generated from Python, fixtures include CJK/Cyrillic/Greek inputs). - planQuery: tokenize → query-time stopword drop (contract §3 list) → duplicate-term dedup → shard/hot-file resolution (hot_tokens.json manifest aware) with cross-token file dedup. Distinguishes 'all stopwords' (controlled empty state) from 'no tokens'. - bm25Contribution: k1=1.2 b=0.75, contract §5 field weights. - combineAndRank: AND semantics, per-pid score summation, stable ordering (score desc, pid asc), TOP_K=50. 12/12 unit tests, incl. the isamplesorg#172 hard-fail preconditions: stopword near-equivalence ('pottery from Cyprus' plans identically to 'pottery Cyprus'), duplicate-term identity, all-stopword controlled empty, hot-vs-base file resolution. Phase B (explorer.qmd ?fts=v1 flag wiring + benchmark harness) follows separately. Stacked on isamplesorg#329 (feat/170-search-index-builder). Refs isamplesorg#171, isamplesorg#170, isamplesorg#169. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
…hot policy Sync with isamplesorg#329's merged contract (§6 two-tier rule + round-5 df embedding): - planQuery returns a mode ('empty'|'allStopwords'|'normal'|'topk'): non-fetchable common terms are dropped from the AND with an ignoredCommon disclosure list (UI must show it); fetchable hot tokens join the AND via their hot/ sub-files; all-common queries plan a single hot_topk.parquet fetch. - expectedBytes computed up-front from shard_sizes.json + manifest total_bytes (§7 transfer accounting; feeds the benchmark metric). - bm25Contribution takes {totalDocs, avgDocLenByField} from build_stats.json (per-field corpus averages — matches the builder's hot_topk scoring exactly); df arrives embedded in rows. 15 unit tests incl. the two-tier cases (drop-with-disclosure, fetchable joins AND, all-common topk mode, expectedBytes accounting). 110/110 across the repo. Refs isamplesorg#171. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
Feature-flagged wiring of the published sharded index (data.isamples.org/isamples_202608_search_index_v1/, PR isamplesorg#329): - buildSearchFilterSubstrate(term): plans via the unit-tested JS module (tokenize → stopwords → two-tier hot policy → file resolution), then scores in DuckDB SQL (BM25 §5, identical constants to the builder's hot_topk and the module's bm25Contribution; df embedded in rows — the query path fetches ONLY the plan's files). Produces the SAME singleton search_pids table (pid, label, source, place_name, relevance_score) as the interim path, so the table/points/facet-count semi-joins and the side panel are shared unchanged. - topk mode: all-common queries intersect the precomputed per-token top-500 sidecar. - Controlled-empty states for empty/all-stopword queries (§3 copy). - Three KB-scale sidecars (hot_tokens/shard_sizes/build_stats) fetched once on FIRST substrate search — flag-off visitors pay zero. - Disclosure (§3/§6): the results heading lists exactly which very common words were ignored ('material', 'sample', …) — silent semantic change is the one thing the policy forbids. - Route: doSearch's single buildSearchFilter call site branches on ?fts=v1. Default path byte-identical in behavior. Refs isamplesorg#171. Benchmark harness (the isamplesorg#172 input) is the remaining piece. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
…arto resources Without these, the page-runtime import 404s and the OJS dependency cascade kills the whole search cell — boot green, search dead, smoke times out at the search step with no pageerror. (First page-runtime consumer of these modules; isamplesorg#170 only needed them for Node tests.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
Six checks against the published index over the real CDN: ILIKE ground-truth parity (284), keyword-concept recall (pottery Cyprus 1,305, expectedBytes < 6 MB), common-term drop with user-visible disclosure, all-common topk mode engagement, controlled all-stopword empty, and default-path non-regression (interim search still used without the flag — the one test needing a raised timeout, because the interim path cold-downloads 69 MB while every substrate test fits the 30 s default; that asymmetry is the point). 6/6 green against the fork preview. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
…py, lazy import
1. Superseded-search races: token guards added after the module-import
and meta-fetch awaits, and before the controlled-empty
clearSearchFilter (which bumps the token — a stale search calling it
would invalidate the newer one). doSearch's build-failure catch now
guards its clearSearchFilter on searchId === _searchSeq — an error
inside a SUPERSEDED build must not blow away the newer search's
filter (applies to both paths; strictly safer for the interim one).
2. All-stopword copy is now user-visible: the zero-results branch
renders the substrate's controlled-empty note ("All terms in your
query are common words…") in both the status line and the panel,
instead of a generic "No results". Spec asserts the VISIBLE copy,
not just internal state.
3. Flag-off isolation: the substrate module import moved from an OJS
top-level cell (which flag-off visitors downloaded and whose failure
could kill the whole search cell) to a lazy import inside
buildSearchFilterSubstrate — first flagged search pays it, nobody
else does, and a load failure surfaces as a caught search error.
110/110 unit tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
First substrate search pays boot + lazy import + 3 sidecars + shards; 22s observed passing, a cold CDN edge tips the 30s default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
rdhyee
added
the
CC+Codex+LGTM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Implements #171's browser query path against the index published by #329 (
data.isamples.org/isamples_202608_search_index_v1/), opt-in via?fts=v1— default behavior is untouched (e2e-asserted) until the #172 GO/NO-GO gate.assets/js/search_substrate.js— pure query engine (no DuckDB/fetch/DOM): FNV-1a-32 parity-pinned to the Python builder via fixture;planQueryimplements the contract's two-tier hot rule (non-fetchable common terms dropped with user-visible disclosure; fetchable hot joins the AND; all-common queries → precomputed top-K sidecar); BM25 per §5; up-frontexpectedBytesfromshard_sizes.json. 15 unit tests (110 repo-wide).explorer.qmd:buildSearchFilterSubstrateplans via the module, scores in DuckDB SQL (identical §5 constants to the builder'shot_topk), and emits the identicalsearch_pidssingleton as the interim path — table, points, facet counts, side panel all shared unchanged. Lazy module import (flag-off visitors download nothing; a load failure is a caught search error, not a dead search cell). Controlled-empty copy for all-stopword queries, rendered visibly.tests/playwright/fts-v1.spec.js— 6 e2e checks against the fork preview + live CDN, all green.Measured (fresh browser, live CDN index)
Zero console errors throughout. The interim path's cold cost (69 MB facets scan) vs the substrate's per-query 1.4–10 MB is the headline.
Codex review (GPT-5.6, high effort)
Round 1 blockers (all fixed in
9882429): superseded-search races around the new awaits + an unconditionalclearSearchFilterin doSearch's catch (guard now applies to both paths — strictly safer for the interim one too); all-stopword copy was internal-state-only (now user-visible + e2e-asserted); flag-off users were downloading the modules via a top-level OJS import whose failure could kill the search cell (now lazy). Round 1 also confirmed: SQL token interpolation safe by tokenizer construction, page-lifetime sidecar caching appropriate, area-scope composition equivalent downstream.Round 2: "LGTM — no blockers. Verified all three fixes in the diff… No regressions found."
Not in this PR
The benchmark harness producing #172's metrics JSON (next up), and any default-on decision (that IS #172).
🤖 Generated with Claude Code
https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9