Skip to content

fix: keep Update working when one of two identical files goes - #8

Open
TWadi wants to merge 1 commit into
mainfrom
twadi/fix-identical-files
Open

TWadi wants to merge 1 commit into
mainfrom
twadi/fix-identical-files

Conversation

@TWadi

@TWadi TWadi commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

What

Fixes a crash in ChunkStore.remove that left a section unable to update.

Chunk IDs come from file content, so two byte-identical files in a collection share IDs. When one of them was deleted or edited, remove matched chunks by doc_id, picked up both files' chunks and passed duplicate IDs to vectors.delete. Chroma raised DuplicateIDError, nothing was written, and every later Update failed the same way.

remove now:

  • matches chunks on SOURCE (the file path), so only the removed file's chunks go
  • deletes only the vectors no remaining chunk uses, without duplicates
  • writes the surviving chunk back over any vector a remaining file still uses. add_documents upserts, so this relabels the vector and a citation never names the removed file

add and remove now share a small _write_vectors helper.

Chunk IDs stay shared on purpose. Search merges results by chunk ID, which stops the same passage filling two answer slots, so this PR does not make them unique per file. test_identical_files_take_one_place_in_search_results pins that behaviour.

Why

A section holding two identical files, for example a document saved twice, crashes on Update once either copy is deleted or edited, and stays broken from then on. There is no linked issue.

Sections already stuck need no migration. The old code crashed before writing anything and chunk IDs don't change, so the next Update with this fix goes through.

How it was tested

  • 5 new tests (6 cases) in tests/test_ingest.py, run against real Chroma storage: deleting either copy, updating again afterwards, editing one copy, deleting both, and search slots.
  • Before the fix, 5 of the 6 cases failed with DuplicateIDError. The search slots case passes before and after, because it guards behaviour that must not change.
  • After the fix, pytest --cov gives 122 passed with 95% coverage. ruff check src tests and ruff format --check src tests pass.
  • No evaluation numbers, because ranking and generation code are unchanged.
  • The desktop tests were not run locally. This PR changes no desktop code, and CI covers them.

One cost to know about: removing or editing a copy re-embeds the chunks it shared with a surviving file, which is one embedding call per shared chunk. It only happens for identical files.

Checklist

  • Tests cover the change
  • Docs updated (README, docs/, or a new ADR for architectural decisions): not needed, this is a bug fix with no documented behaviour to change
  • No secrets, keys or private data in the diff

Chunk IDs come from file content, so byte-identical files share them.
ChunkStore.remove matched chunks by doc_id, collected both files' chunks
and passed duplicate IDs to vectors.delete. Chroma raised DuplicateIDError,
nothing was written, and every later Update failed the same way.

Match chunks on the source path instead. Delete only the vectors no
remaining chunk uses, and write the surviving chunk back over the rest so
a citation never names the removed file.

Chunk IDs stay shared on purpose: search merges results by chunk ID, which
stops the same passage filling two answer slots.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant