Skip to content

fix(server): serve without BM25, route chunk-id tools to a notebook - #581

Merged
chris-dare-dev merged 2 commits into
mainfrom
fix/batch-o-retrieval-routing
Nov 21, 2026
Merged

chris-dare-dev merged 2 commits into
mainfrom
fix/batch-o-retrieval-routing

Conversation

@chris-dare-dev

Copy link
Copy Markdown
Owner

Two retrieval findings: BM25 could take the server down although no tool reads it, and the chunk-id tools ignored the notebook a search came from.

#335: serve without an unbuildable BM25 index

  • Before, a BM25 load or auto-build failure refused /readyz at startup and abandoned a corpus rebind. Yet no served tool reads the BM25 phase: search_papers runs the dense route and reports retrieval_mode="dense_only" whether or not BM25 loaded.
    • So a read-only dataset mount, or a corpus with empty body_tokens, took the server down or kept a fresh ingest invisible, over a component that changes no answer.
  • Both binding paths now go through one helper that returns None and logs why.
  • An artifact that fails the pickle file-safety check is still never loaded, and is logged at ERROR.

#334: route the chunk-id tools to a notebook

  • Before, search_papers(filters={"notebook": slug}) returned ids from that notebook's chunks table, but get_chunk, get_paper and cite_neighbors always read the shared corpus. The documented next step, get_chunk on a returned id, came back found=false for a chunk the server had just served.
  • Corpus selection moves out of search_papers into server/handlers/corpus_route.py:
    • parse_notebook validates the slug before any I/O;
    • resolve_corpus opens the notebook's table and names the version the envelope echoes.
  • search_papers keeps its error messages. get_chunk, get_paper and cite_neighbors take an optional notebook argument that goes through the same code. Without it they read the shared corpus as before.

Tool schema change. Three inputSchemas grow, so TOOL_SCHEMA_VERSION goes to 26, both result schemas move with it, and EXPECTED_TOOL_SCHEMA_SHA256 is re-pinned. Clients that cache tools/list will see the new version. No tool description changes.

Closes #335
Closes #334

Verification

  • ruff check . is clean.
  • Full suite on this branch: 5603 passed, 182 skipped, 1 xfailed (Python 3.12, Ubuntu).
  • This branch merged with the equations and package-layering batches on 7cc5620: 5610 passed, 182 skipped, 1 xfailed. make wheel-check passes on that merge.
  • The package-layering PR conflicts with this one in server/handlers/search.py and imports a moved module in corpus_route.py, so it lands after this one with both resolved.

The wiki's tool reference should list the new notebook argument on get_chunk, get_paper and cite_neighbors.

No served tool reads the BM25 phase: search_papers runs the dense
route and reports retrieval_mode="dense_only" whether or not BM25
loaded. Yet a BM25 load or auto-build failure refused /readyz at
startup and abandoned a corpus rebind, so a read-only dataset mount
or a corpus with empty body_tokens took the server down, or kept a
fresh ingest invisible, over a component that changes no answer.

Both binding paths now go through one helper that returns None and
logs why. An artifact that fails the pickle file-safety check is
still never loaded, and is logged at ERROR.
search_papers(filters={"notebook": slug}) returns ids from that
notebook's chunks table, but get_chunk, get_paper and cite_neighbors
always read the shared corpus. The documented next step, get_chunk on
a returned id, came back found=false under the shared corpus_version
for a chunk the server had just served.

Corpus selection moves out of search_papers into
server/handlers/corpus_route.py: parse_notebook validates the slug
before any I/O, and resolve_corpus opens the notebook's table and
names the version the envelope echoes. search_papers keeps its error
messages; get_chunk, get_paper and cite_neighbors take an optional
notebook argument that goes through the same seam. Without it they
read the shared corpus as before.

Three inputSchemas grow, so TOOL_SCHEMA_VERSION goes to 26 with both
result schemas in lockstep and EXPECTED_TOOL_SCHEMA_SHA256 re-pinned.
No tool description changes, so the BP1 prefix does not move.
@chris-dare-dev
chris-dare-dev merged commit 12561bd into main Nov 21, 2026
4 checks passed
@chris-dare-dev
chris-dare-dev deleted the fix/batch-o-retrieval-routing branch November 21, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant