Refactor SQL bulk-put to shared multi-row INSERT engine

[sroussey]

Summary

Consolidates per-row INSERT loops in SQLite, Postgres, and DuckDB putBulk implementations into a shared multi-row bulk-insert engine in BaseSqlTabularStorage. Each chunk now executes as a single INSERT … VALUES (…),(…),… RETURNING * statement instead of N separate statements, improving throughput while preserving atomicity, ordering, and event semantics.

Key Changes

• New BulkPutDialect interface in BaseSqlTabularStorage: Captures dialect-specific SQL generation (quote character, placeholder format, INSERT INTO clause, conflict handling, UUID minting policy) for the shared engine.

• Shared bulk-put engine (runBulkPut, buildBulkPutValues, buildBulkCells, bulkKeyOf, resolveBulkAutoKey):

  • Resolves auto-generated key policies (client-side UUID minting, server-side defaults, required keys)
  • Deduplicates duplicate primary keys within a batch (last-write-wins) when all rows are keyed
  • Chunks by parameter budget (maxBulkParams)
  • Re-aligns returned rows to input order via key matching (keyed batches) or response order (unkeyed)
  • Returns both aligned (per-input-position) and distinct (per-committed-row) results for correct event emission

• Per-backend dialect implementations:

  • SQLite: sqliteBulkDialect() — backtick quotes, ? placeholders, INSERT OR REPLACE or ON CONFLICT upsert, client-side UUID minting
  • Postgres: postgresBulkDialect() — double-quote quotes, $N placeholders, ON CONFLICT … DO UPDATE upsert, server-side UUID defaults
  • DuckDB: duckdbBulkDialect() — double-quote quotes, $N placeholders, ON CONFLICT … DO UPDATE upsert, client-side UUID minting

• Updated _putBulkInternal implementations: All three backends now call runBulkPut with their dialect, replacing per-row insert loops. Transaction handling (BEGIN/COMMIT/ROLLBACK) and event deferral remain unchanged.

• New test suite (genericSqlBulkPutTests): Validates duplicate-key deduplication, event emission (one per distinct row), and all-or-nothing rollback semantics across all SQL backends.

• Extended generic tests: Added coverage for batches larger than SQL parameter limits, last-write-wins deduplication, and composite key collision avoidance.

Notable Implementation Details

• Key matching: Uses bulkKeyOf() to stringify primary-key columns (after jsToSqlValue conversion) so returned rows can be matched back to input rows even when the server assigns keys. Keyed batches use a Map; unkeyed batches fall back to response order.

• Auto-key resolution: Respects clientProvidedKeys policy ("always", "if-missing", "never") and autoGeneratedKeyStrategy ("uuid", "integer"). Client-side UUID minting happens here; server-side defaults bind DEFAULT literals in the VALUES clause.

• Parameter chunking: Calculates perRowParams from the first row's column count, then chunks to Math.max(1, Math.floor(maxBulkParams / perRowParams)) rows per statement.

• Atomicity preserved: SQLite uses manual BEGIN/COMMIT (not db.transaction(), which requires sync body); Postgres and DuckDB wrap chunks in explicit transactions. Rollback emits rollback event with empty ids (batch fully reverted).

• Event deferral: put events fire only for distinct rows after all chunks commit, matching single-row put semantics.

Backward Compatibility

All changes are internal refactors. Public API (putBulk signature, return type, event semantics) is unchanged. Behavior is identical except:

• Duplicate primary keys within a

https://claude.ai/code/session_011jSSGKf1DQhuF1vK31Jb7u

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	63.05%	27367 / 43402
🔵	Statements	62.9%	28356 / 45074
🔵	Functions	63.42%	5207 / 8210
🔵	Branches	51.71%	13473 / 26052

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/storage/src/tabular/ITabularStorage.ts	20%	0%	0%	28.57%	226-230

Generated in workflow #2772 for commit e4febab by the Vitest Coverage Report Action