Repository navigation
fix(server): serve without BM25, route chunk-id tools to a notebook - #581
Merged
Merged
Conversation
No served tool reads the BM25 phase: search_papers runs the dense route and reports retrieval_mode="dense_only" whether or not BM25 loaded. Yet a BM25 load or auto-build failure refused /readyz at startup and abandoned a corpus rebind, so a read-only dataset mount or a corpus with empty body_tokens took the server down, or kept a fresh ingest invisible, over a component that changes no answer. Both binding paths now go through one helper that returns None and logs why. An artifact that fails the pickle file-safety check is still never loaded, and is logged at ERROR.
search_papers(filters={"notebook": slug}) returns ids from that
notebook's chunks table, but get_chunk, get_paper and cite_neighbors
always read the shared corpus. The documented next step, get_chunk on
a returned id, came back found=false under the shared corpus_version
for a chunk the server had just served.
Corpus selection moves out of search_papers into
server/handlers/corpus_route.py: parse_notebook validates the slug
before any I/O, and resolve_corpus opens the notebook's table and
names the version the envelope echoes. search_papers keeps its error
messages; get_chunk, get_paper and cite_neighbors take an optional
notebook argument that goes through the same seam. Without it they
read the shared corpus as before.
Three inputSchemas grow, so TOOL_SCHEMA_VERSION goes to 26 with both
result schemas in lockstep and EXPECTED_TOOL_SCHEMA_SHA256 re-pinned.
No tool description changes, so the BP1 prefix does not move.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two retrieval findings: BM25 could take the server down although no tool reads it, and the chunk-id tools ignored the notebook a search came from.
#335: serve without an unbuildable BM25 index
/readyzat startup and abandoned a corpus rebind. Yet no served tool reads the BM25 phase:search_papersruns the dense route and reportsretrieval_mode="dense_only"whether or not BM25 loaded.body_tokens, took the server down or kept a fresh ingest invisible, over a component that changes no answer.Noneand logs why.#334: route the chunk-id tools to a notebook
search_papers(filters={"notebook": slug})returned ids from that notebook's chunks table, butget_chunk,get_paperandcite_neighborsalways read the shared corpus. The documented next step,get_chunkon a returned id, came backfound=falsefor a chunk the server had just served.search_papersintoserver/handlers/corpus_route.py:parse_notebookvalidates the slug before any I/O;resolve_corpusopens the notebook's table and names the version the envelope echoes.search_paperskeeps its error messages.get_chunk,get_paperandcite_neighborstake an optionalnotebookargument that goes through the same code. Without it they read the shared corpus as before.Tool schema change. Three
inputSchemas grow, soTOOL_SCHEMA_VERSIONgoes to 26, both result schemas move with it, andEXPECTED_TOOL_SCHEMA_SHA256is re-pinned. Clients that cachetools/listwill see the new version. No tool description changes.Closes #335
Closes #334
Verification
ruff check .is clean.7cc5620: 5610 passed, 182 skipped, 1 xfailed.make wheel-checkpasses on that merge.server/handlers/search.pyand imports a moved module incorpus_route.py, so it lands after this one with both resolved.The wiki's tool reference should list the new
notebookargument onget_chunk,get_paperandcite_neighbors.