Repository navigation
fix(ingest): one writer per dataset, data-dir paths - #577
Merged
Merged
Conversation
The single-writer rule the store docstring stated had no enforcement, so a console-started notebook ingest and a terminal run could write the same dataset, and the write-audit gate then blamed an arithmetic regression in correct code. Every ingest write entry point now holds a non-blocking flock on a lock file beside the dataset: bulk ingest (and so notebook ingest), the OAI delta, re-embed, ops.prune_versions, write_chunks, delete_paper and the equation, definition, theorem-name and derived-index builders. A second writer fails at once with DatasetWriteLockedError naming the holder, and the CLIs exit with WRITE_LOCKED_EXIT_CODE (4). Holding is reentrant within a thread so a driver's own writes pass. The file sits beside the dataset so a cutover rename cannot move a held lock. A writer that bypasses the lock is now named by the gate: when the marker read back names another version, or the table has moved past the version this call wrote, the gate reports concurrent_writer and says to stop that writer, instead of routing to S6. Closes #312
preamble, extract_equations, embed_equations, oai_delta, re_embed, the BM25 stats log, intra_paper_refs and the Kuzu ingest CLIs still built their paths from the checkout (REPO_ROOT / "var" / ..., or a path relative to the working directory), so ARXMCP_DATA_DIR moved the server but not these writers, and an installed package wrote under site-packages. Every ingest path now comes from ApplicationPaths. Notebook ingest now passes its own ops dir to run_bulk_ingest, so a notebook run writes ingest-summary.json in the notebook's ops/ instead of overwriting the shared corpus summary that /metrics reads. A notebook re-embed keeps its failures log with the notebook too. The per-paper intermediate trees stay shared across datasets on purpose, and the bulk-ingest docstring now says why. They are a cache: the same paper and code give the same files, each set is replaced atomically, and the embedder re-encodes any set from another version, so a leftover set costs a re-embed, never a wrong row. A tree per dataset would re-embed every paper once per notebook. Closes #316
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two ingest write-safety findings: a single-writer rule that nothing enforced, and corpus paths that ignored the data directory.
#312: lock each dataset to one writer
flockon a lock file beside the dataset:ops.prune_versions,write_chunksanddelete_paper;DatasetWriteLockedError, which names the holder, and the CLIs exit withWRITE_LOCKED_EXIT_CODE(4).concurrent_writerand says to stop that writer. It used to route to the arithmetic diagnosis instead.#316: route corpus paths through the data directory
Several ingest writers built their paths from the checkout (
REPO_ROOT / "var" / ...) or the working directory: preamble, equation extraction and embedding, the OAI delta, re-embed, the BM25 stats log, intra-paper refs and the Kùzu ingest CLIs.ARXMCP_DATA_DIRmoved the server's data but not theirs.site-packages.ApplicationPaths.Notebook ingest now passes its own ops directory to
run_bulk_ingest. A notebook run therefore writesingest-summary.jsonin the notebook'sops/, instead of overwriting the shared corpus summary that/metricsreads.A notebook re-embed keeps its failures log with the notebook too.
The per-paper intermediate trees stay shared across datasets on purpose, and the bulk-ingest docstring says why. They act as a cache:
A leftover set therefore costs a re-embed, never a wrong row. A tree per dataset would re-embed every paper once per notebook.
Closes #312
Closes #316
Verification
ruff check .is clean.5ac66b2: 5593 passed, 182 skipped, 1 xfailed.The wiki's ingest and notebook pages should cover the new exit code 4, the lock file beside each dataset, and the summary's move to the notebook's
ops/.