Skip to content

fix(ingest): one writer per dataset, data-dir paths - #577

Merged
chris-dare-dev merged 2 commits into
mainfrom
fix/batch-k-write-safety
Nov 21, 2026
Merged

chris-dare-dev merged 2 commits into
mainfrom
fix/batch-k-write-safety

Conversation

@chris-dare-dev

Copy link
Copy Markdown
Owner

Two ingest write-safety findings: a single-writer rule that nothing enforced, and corpus paths that ignored the data directory.

#312: lock each dataset to one writer

  • Before, the store's single-writer rule had no enforcement. A notebook ingest started from the console and a terminal run could write the same dataset at once, and the write-audit gate then blamed an arithmetic regression in correct code.
  • Every ingest write entry point now holds a non-blocking flock on a lock file beside the dataset:
    • bulk ingest (and so notebook ingest), the OAI delta and re-embed;
    • ops.prune_versions, write_chunks and delete_paper;
    • the equation, definition, theorem-name and derived-index builders.
  • A second writer fails at once with DatasetWriteLockedError, which names the holder, and the CLIs exit with WRITE_LOCKED_EXIT_CODE (4).
  • The lock is reentrant within a thread, so a driver's own writes pass.
  • The lock file sits beside the dataset, not inside it, so a cutover rename can't move a held lock.
  • When a writer bypasses the lock, the gate now reports concurrent_writer and says to stop that writer. It used to route to the arithmetic diagnosis instead.

#316: route corpus paths through the data directory

  • Several ingest writers built their paths from the checkout (REPO_ROOT / "var" / ...) or the working directory: preamble, equation extraction and embedding, the OAI delta, re-embed, the BM25 stats log, intra-paper refs and the Kùzu ingest CLIs.

    • ARXMCP_DATA_DIR moved the server's data but not theirs.
    • An installed package wrote under site-packages.
    • Every ingest path now comes from ApplicationPaths.
  • Notebook ingest now passes its own ops directory to run_bulk_ingest. A notebook run therefore writes ingest-summary.json in the notebook's ops/, instead of overwriting the shared corpus summary that /metrics reads.

  • A notebook re-embed keeps its failures log with the notebook too.

  • The per-paper intermediate trees stay shared across datasets on purpose, and the bulk-ingest docstring says why. They act as a cache:

    • the same paper and code give the same files;
    • each set is replaced atomically;
    • the embedder re-encodes any set written by another version.

    A leftover set therefore costs a re-embed, never a wrong row. A tree per dataset would re-embed every paper once per notebook.

Closes #312
Closes #316

Verification

  • ruff check . is clean.
  • Full suite on this branch: 5559 passed, 182 skipped, 1 xfailed (Python 3.12, Ubuntu).
  • This branch merged with the CA-pinning and server-runtime batches on 5ac66b2: 5593 passed, 182 skipped, 1 xfailed.

The wiki's ingest and notebook pages should cover the new exit code 4, the lock file beside each dataset, and the summary's move to the notebook's ops/.

The single-writer rule the store docstring stated had no enforcement,
so a console-started notebook ingest and a terminal run could write the
same dataset, and the write-audit gate then blamed an arithmetic
regression in correct code.

Every ingest write entry point now holds a non-blocking flock on a
lock file beside the dataset: bulk ingest (and so notebook ingest), the
OAI delta, re-embed, ops.prune_versions, write_chunks, delete_paper and
the equation, definition, theorem-name and derived-index builders. A
second writer fails at once with DatasetWriteLockedError naming the
holder, and the CLIs exit with WRITE_LOCKED_EXIT_CODE (4). Holding is
reentrant within a thread so a driver's own writes pass. The file sits
beside the dataset so a cutover rename cannot move a held lock.

A writer that bypasses the lock is now named by the gate: when the
marker read back names another version, or the table has moved past
the version this call wrote, the gate reports concurrent_writer and
says to stop that writer, instead of routing to S6.

Closes #312
preamble, extract_equations, embed_equations, oai_delta, re_embed, the
BM25 stats log, intra_paper_refs and the Kuzu ingest CLIs still built
their paths from the checkout (REPO_ROOT / "var" / ..., or a path
relative to the working directory), so ARXMCP_DATA_DIR moved the
server but not these writers, and an installed package wrote under
site-packages. Every ingest path now comes from ApplicationPaths.

Notebook ingest now passes its own ops dir to run_bulk_ingest, so a
notebook run writes ingest-summary.json in the notebook's ops/ instead
of overwriting the shared corpus summary that /metrics reads. A
notebook re-embed keeps its failures log with the notebook too.

The per-paper intermediate trees stay shared across datasets on
purpose, and the bulk-ingest docstring now says why. They are a cache:
the same paper and code give the same files, each set is replaced
atomically, and the embedder re-encodes any set from another version,
so a leftover set costs a re-embed, never a wrong row. A tree per
dataset would re-embed every paper once per notebook.

Closes #316
@chris-dare-dev
chris-dare-dev merged commit 7b79858 into main Nov 21, 2026
4 checks passed
@chris-dare-dev
chris-dare-dev deleted the fix/batch-k-write-safety branch November 21, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant