fix(ingest): #773 byte-identical re-ingest converges instead of accumulating - #783
Merged
Conversation
…ulating Extraction is non-deterministic, so re-running it on an unchanged source unioned every set the model ever produced (17 objects from one 81-line file) -- create-only dedup only catches verbatim-reproduced slugs. A byte-identical re-ingest of an already-extracted source now skips extraction entirely: no model call, nothing written, one stderr line naming --re-extract as the deliberate redo. Retryable debt still re-extracts without the flag (extraction_status: failed, judge-degrade extraction_notice -- the exact retry lint names), a legacy Source with no origin_key takes the full path once to backfill it, and a post-forget regenerate stays a fresh run. testing.md's 'idempotent' wording is now true as written; specs and cli.md updated to the real contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An identical re-ingest accumulated near-duplicate objects instead of replacing them — 17 objects from one 81-line source, including candidates the first run's judge had explicitly dropped — because extraction re-ran on every re-ingest and create-only dedup can only catch verbatim-reproduced slugs.
--re-extract.testing.md§3.1's word idempotent is now accurate as written.extraction_status: failed(feat: record and surface sources whose extraction was skipped #187) and the judge-degradeextraction_noticetokens (extraction: a judge that could not run keeps every candidate unfiltered, and nothing marks them for later review #772) — exactly the states whoselintfindings name a plain re-ingest as the remedy. Includes the degrade→re-ingest self-clear test the fix(extraction,lint): #772 quarantine unjudged extractions instead of admitting them silently #782 review's WARNING asked for.--re-extractis the deliberate redo on a healthy source; per-slug create-only reconciliation is unchanged when extraction runs.origin_key) takes the full path once, preserving the no-verb self-migration; a post-forgetregenerate stays a fresh run; an unparseable prior Source falls through to the full pipeline (new test).docs/cli.md, anddocs/testing.mdupdated to the real contract; 28 pre-existing re-ingest tests migrated to spell the redo explicitly.Verification
ruff,ruff format --check,mypy .clean.review-486f068b72495fcdapproved (4R; 0 blocking findings; one repair round fixed the two lens-corroborated defects — a self-contradicting cli.md paragraph and a stranded dataclass docstring). Gates pre-commit/pre-push/pre-pr allow.Closes #773
🤖 Generated with Claude Code
https://claude.ai/code/session_011rKLjyxsEZJtZWDCeozZhs