Skip to content

fix(sqlite): serialize putBulk across shared-connection SqliteTabularStorage instances#655

Open
sroussey wants to merge 1 commit into
mainfrom
claude/dazzling-archimedes-2ni5lb
Open

fix(sqlite): serialize putBulk across shared-connection SqliteTabularStorage instances#655
sroussey wants to merge 1 commit into
mainfrom
claude/dazzling-archimedes-2ni5lb

Conversation

@sroussey

Copy link
Copy Markdown
Collaborator

Summary

HIGH-priority correctness fix from a scheduled review of the last 24h of main (surfaced by review of the putBulk engine refactor in chore: release 0.3.27).

The bug

SqliteTabularStorage._putBulkInternal guarded on better-sqlite3's driver-level db.inTransaction, which is scoped to the connection, not to the storage instance. When two SqliteTabularStorage instances share one Sqlite.Database (the normal case for repos on the same file), a second putBulk starting during another's inter-chunk await gap read inTransaction === true, skipped its own BEGIN/COMMIT, executed rows inside the OTHER instance's transaction, emitted put events, and returned success. If the first instance later rolled back on a unique-constraint violation, the second's rows silently disappeared even though its caller had received the returned entities and subscribers had already seen put events for phantom rows.

The fix

  • Module-scope WeakMap<Sqlite.Database, Promise<void>> plus a protected connectionMutex() helper — GC-safe, keyed by the DB handle.
  • _putBulkInternal keys off this.inTransaction (per-instance) and wraps its BEGIN/runBulkPut/COMMIT|ROLLBACK bracket in connectionMutex(...) when not already in a transaction.
  • withTransaction wraps its own bracket the same way, inside the existing per-instance mutex — instance-first / connection-second so two instances on one handle cannot deadlock.
  • createTxView reports inTransaction === true through the proxy so the tx-routed putBulk path takes the nested branch and skips both BEGIN and the connection lock (the outer withTransaction already holds both).
  • SqliteAiVectorStorage's overridden _putBulkInternal mirrors the parent — keys off this.inTransaction and takes connectionMutex on the owned-transaction path.

Verification

Regression tests added under packages/test/src/test/storage-tabular/SqliteTabularStorage.integration.test.ts:

  • Two SqliteTabularStorage instances on DIFFERENT tables sharing one Sqlite.Database.
  • Race Promise.allSettled([repoA.putBulk(large), repoB.putBulk(small)]) with runBulkPut spied to yield microtasks and then throw on A.
  • Asserts A rejects, B fulfills, and repoB's rows survive.
  • Second test asserts B's put event stream never includes rows that were rolled back with A.

Run:

bun x vitest run packages/test/src/test/storage-tabular/SqliteTabularStorage.integration.test.ts

Rollback

Revert the single file (providers/sqlite/src/storage/SqliteTabularStorage.ts, plus the mirror in SqliteAiVectorStorage.ts, plus the test). No schema/migration/API changes.

🤖 Generated with Claude Code


Generated by Claude Code

…Storage instances

_putBulkInternal guarded on better-sqlite3's driver-level db.inTransaction,
which is scoped to the CONNECTION, not to the storage instance. When two
SqliteTabularStorage instances shared one Sqlite.Database (repos on the
same file), a second putBulk starting during another's inter-chunk await
gap read inTransaction === true, skipped its own BEGIN/COMMIT, executed
its rows inside the OTHER instance's transaction, emitted put events,
and returned success. If the first instance later rolled back on a unique-
constraint violation, the second's rows silently disappeared even though
its caller had received the returned entities and subscribers had seen
put events for phantom rows.

Add a module-scope WeakMap<Sqlite.Database, Promise<void>> and a protected
connectionMutex() helper (GC-safe, keyed by the DB handle) and:

  - _putBulkInternal now keys off this.inTransaction (per-instance), and
    when not already in a transaction wraps the whole BEGIN/runBulkPut/
    COMMIT|ROLLBACK bracket in connectionMutex(...).
  - withTransaction wraps its BEGIN/fn/COMMIT|ROLLBACK bracket in the
    same connectionMutex, inside the existing per-instance mutex —
    instance-first / connection-second so two instances on one handle
    cannot deadlock.
  - createTxView reports inTransaction === true through the proxy so the
    tx-routed putBulk path takes the nested branch and skips both BEGIN
    and the connection lock (the outer withTransaction already holds
    both).
  - SqliteAiVectorStorage's overridden _putBulkInternal now also keys
    off this.inTransaction and takes the connectionMutex on the owned-
    transaction path, mirroring the parent.

Regression test in packages/test/src/test/storage-tabular:
  - Constructs two SqliteTabularStorage instances on DIFFERENT tables
    sharing one Sqlite.Database.
  - Races Promise.allSettled([repoA.putBulk(large), repoB.putBulk(small)])
    with runBulkPut spied to yield microtasks and then throw on A.
  - Asserts A rejects, B fulfills, and repoB's rows survive.
  - Second test asserts B's put event stream never includes rows that
    were rolled back with A.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 63.06% 27385 / 43423
🔵 Statements 62.92% 28375 / 45096
🔵 Functions 63.44% 5213 / 8216
🔵 Branches 51.73% 13479 / 26054
File CoverageNo changed files found.
Generated in workflow #2775 for commit ae68280 by the Vitest Coverage Report Action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants