Repository navigation
Conversation
Chunk IDs come from file content, so byte-identical files share them. ChunkStore.remove matched chunks by doc_id, collected both files' chunks and passed duplicate IDs to vectors.delete. Chroma raised DuplicateIDError, nothing was written, and every later Update failed the same way. Match chunks on the source path instead. Delete only the vectors no remaining chunk uses, and write the surviving chunk back over the rest so a citation never names the removed file. Chunk IDs stay shared on purpose: search merges results by chunk ID, which stops the same passage filling two answer slots.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes a crash in
ChunkStore.removethat left a section unable to update.Chunk IDs come from file content, so two byte-identical files in a collection share IDs. When one of them was deleted or edited,
removematched chunks bydoc_id, picked up both files' chunks and passed duplicate IDs tovectors.delete. Chroma raisedDuplicateIDError, nothing was written, and every later Update failed the same way.removenow:SOURCE(the file path), so only the removed file's chunks goadd_documentsupserts, so this relabels the vector and a citation never names the removed fileaddandremovenow share a small_write_vectorshelper.Chunk IDs stay shared on purpose. Search merges results by chunk ID, which stops the same passage filling two answer slots, so this PR does not make them unique per file.
test_identical_files_take_one_place_in_search_resultspins that behaviour.Why
A section holding two identical files, for example a document saved twice, crashes on Update once either copy is deleted or edited, and stays broken from then on. There is no linked issue.
Sections already stuck need no migration. The old code crashed before writing anything and chunk IDs don't change, so the next Update with this fix goes through.
How it was tested
tests/test_ingest.py, run against real Chroma storage: deleting either copy, updating again afterwards, editing one copy, deleting both, and search slots.DuplicateIDError. The search slots case passes before and after, because it guards behaviour that must not change.pytest --covgives 122 passed with 95% coverage.ruff check src testsandruff format --check src testspass.One cost to know about: removing or editing a copy re-embeds the chunks it shared with a surviving file, which is one embedding call per shared chunk. It only happens for identical files.
Checklist
docs/, or a new ADR for architectural decisions): not needed, this is a bug fix with no documented behaviour to change