Skip to content

fix(sqlite): serialize management store writes through the engine write lock - #340

Merged
LeeroyHannigan merged 2 commits into
mainfrom
fix/sqlite-management-write-lock
Oct 13, 2026
Merged

LeeroyHannigan merged 2 commits into
mainfrom
fix/sqlite-management-write-lock

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

What

Routes every management and credential-plane write in the SQLite backend through the engine's existing write_lock, so the two connection pools that open the same database file never contend for the SQLite file write lock.

  • SqliteCatalogStore::with_encryption_key takes the engine's Arc<Mutex<()>>; the server wiring in lib.rs passes the one instance the engine already uses, so engine writers and management writers queue on a single lock.
  • A lock_writes() helper acquires it. Every write statement or transaction in the catalog store (set_setting, metrics insert and prune, failed-login bookkeeping), the admin store, and the management stores (users, access keys, policies, groups, roles, accounts, sessions) takes the lock before the statement or begin and releases it with the statement. Reads never take it.
  • The credential store has no write statements and is unchanged. The GSI propagation worker already held the lock.
  • The out-of-process CLI constructors (settings, doctor) keep a fresh lock, since no engine exists in that process.
  • The store.rs module doc now lists init-time bootstrap as the only deliberate exclusion from the lock.
  • Seven new tests in d1_write_lock_tests, one per store family, each proving the write blocks while the engine lock is held and completes after release, using the assert_serialized harness introduced by fix(sqlite): serialize all writers through the engine write lock #321.

Why

run-integration-sqlite fails intermittently with one test out of about 980 returning HTTP 500 where 204 was expected. Three occurrences this week, all the same assertion in tests/test_auth_integration.py::TestIBAC::test_readonly_user_denied_write on put_user_policy:

The server journal from the failed run names the cause at the same second:

ERROR extenddb_storage_sqlite::management_store::policies: put_policy: error returned from database: (code: 5) database is locked
ERROR extenddb_storage_sqlite::workers: GSI propagation worker: batch error: Internal error: error returned from database: (code: 5) database is locked

SQLite permits one writer per file. The engine serializes its writers with the in-process write_lock (design decision D1 in store.rs), and #321 moved every remaining engine writer inside it after measuring this same failure in August. It left the management stores outside and said so in the module doc: they write through the second sqlx pool in lib.rs, which for a file-backed database opens the same file, as plain autocommit statements with no lock. When such a write arrives while another connection has held the file write lock past the 5 second busy_timeout, SQLite returns SQLITE_BUSY and the management handler maps it to 500. The integration suite runs pytest with parallel workers against one server, so management writes from one worker overlap data-plane writes from another, which is the window.

Both upstream occurrences postdate #321, and neither PR they failed on touches the SQLite backend or the management store.

Testing done

Unit and lint gates on the branch:

cargo fmt --all -- --check                      clean
cargo clippy --all-targets -- -D warnings       clean
cargo test -p extenddb-storage-sqlite           80 passed, 0 failed, 1 ignored (13 d1_write_lock_tests, 7 new)
cargo test --workspace                          1183 passed, 0 failed

Failing-first check: with the lock call removed from put_policy, management_put_policy_waits_for_the_write_lock fails; restored, it passes.

Integration, reproducing the CI job locally (release build with --features sqlite, extenddb init --backend sqlite, serve --foreground --write-pid-file, then the CI invocation devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export"):

pytest suite            941 passed, 17 skipped, 1 xfailed  (plus 43 passed on a rerun of three files whose
                        first pass failed only from local CA-bundle and cwd differences; 984 distinct tests green)
comprehensive suite     331 passed
tests/test_auth_integration.py, three consecutive parallel runs   15 passed, 15 passed, 15 passed
server log             0 occurrences of "database is locked" across all runs

tests/test_gsi_async_queue_sqlite.py emits five "extenddb stop did not exit within 30s" warnings; the same file against a main-built binary emits the same five in the same 274 seconds, so that is existing harness behavior and not introduced here.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. Applies the existing design decision D1 recorded in the store.rs module doc (every SQLite writer holds the engine write lock) to the writers #321 left outside it. No wire, trait, auth, on-disk, or CLI change.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

ssbytedev pushed a commit to ssbytedev/extenddb that referenced this pull request Sep 13, 2026
…xtendDB#344)

* ci(integration): restore the indentation that stops the workflow loading

Commit 8d3d585 (ExtendDB#312) moved four lines in integration.yml one column left:
two comment lines and the two "run: sudo journalctl" lines in the syslog
dump steps of run-rust-integration and run-rust-integration-postgres-novector.
The run key no longer aligns with its sibling if and name keys, so the file
is not valid YAML. GitHub cannot load it, and since 10:41 UTC on 2026-09-11
every run of this workflow is a zero-job failure named by the file path
instead of "Integration Tests", for pull_request, merge_group and push alike.

Branch protection on main requires the integration status. A status that can
never be reported leaves every pull request opened since then BLOCKED with all
other checks green (ExtendDB#341, ExtendDB#340, ExtendDB#343 at the time of writing), and the merge
queue cannot advance.

This restores the four lines to their previous indentation. Nothing else
changes. Verified by parsing the file before and after: the main version fails
with "while parsing a block collection", the fixed version loads with the seven
expected jobs.

* chore(licenses): regenerate MongoDB notices for workspace version 0.1.11

SOFTWARE-LICENSE-NOTICES-MONGODB.html was generated on the ExtendDB#331 branch,
which was cut before ExtendDB#332 bumped the workspace to 0.1.11. The ten drifted
lines are the workspace's own crates at 0.1.10; no third-party dependency
or license changed. The Licenses workflow's "MongoDB notices must be
current" step fails on main since the ExtendDB#331 merge.

Signed-off-by: Lee Hannigan <lhnng@amazon.com>
(cherry picked from commit 5268480)

* ci(licenses): run the notices check on every pull request and merge group

The notices-current job becomes a required status check. A required check
whose workflow is path-filtered never reports on a pull request outside its
paths, and that pull request can never merge, so the pull_request and
merge_group triggers lose their path filters. The job takes under three
minutes. The push trigger keeps its filter since it gates nothing.

Signed-off-by: Lee Hannigan <lhnng@amazon.com>
(cherry picked from commit 35f42a2)

---------

Signed-off-by: Lee Hannigan <lhnng@amazon.com>
… the engine write lock

SQLite permits one writer per database file. The engine serializes its own
writers with an in-process write_lock (store.rs, design decision D1) so they
never contend at the file level, and #321 moved every remaining engine writer
inside it after measuring this failure class in August.

The management stores (users, access keys, policies, groups, roles,
accounts, admin, settings, metrics) were left outside the lock. They write
through the second sqlx pool that lib.rs opens on the same file, as plain
autocommit statements. When one of those writes arrives while another
connection holds the file write lock for longer than the 5 second
busy_timeout, SQLite returns SQLITE_BUSY ("database is locked") and the
management handler answers HTTP 500. The integration suite runs pytest with
parallel workers against one server, so management writes from one worker
overlap data-plane writes from another. The run-integration-sqlite job hit
this three times this week (runs 34507896936, 34518061214, and the private
mirror), each time as a 500 from PUT user policy in
test_auth_integration.py::TestIBAC::test_readonly_user_denied_write, with
"put_policy: ... (code: 5) database is locked" in the server journal.

This change shares the engine's write_lock into SqliteCatalogStore through
with_encryption_key, adds a lock_writes helper, and takes the lock around
every write statement or transaction in the catalog, admin, and management
stores (lock first, then begin, released with the statement; reads never
take it). The credential store has no write statements and is unchanged.
The GSI propagation worker already held the lock; it appeared in the journal
as a victim of the unlocked management write, not as a second offender.
The out-of-process CLI constructors (settings, doctor) keep a fresh lock
since no engine exists there. Init-time bootstrap is now the only deliberate
exclusion, and the store.rs module doc says so.

Tests: seven new cases in d1_write_lock_tests (policy put, user create,
access key create, account create, group create, admin create, set_setting)
each prove the write blocks while the engine lock is held and completes
after release, using the existing assert_serialized harness.

Signed-off-by: Lee Hannigan <lhnng@amazon.com>
@LeeroyHannigan
LeeroyHannigan force-pushed the fix/sqlite-management-write-lock branch from f82ce56 to f7f14d4 Compare September 14, 2026 13:28
@LeeroyHannigan
LeeroyHannigan added this pull request to the merge queue Oct 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 13, 2026
@LeeroyHannigan
LeeroyHannigan added this pull request to the merge queue Oct 13, 2026
Merged via the queue into main with commit 86a82b4 Oct 13, 2026
29 checks passed
@LeeroyHannigan
LeeroyHannigan deleted the fix/sqlite-management-write-lock branch October 13, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants