Skip to content

fix(sqlite): one extenddb process per database file - #391

Merged
robinnsc merged 1 commit into
mainfrom
fix/sqlite-serve-lock
Oct 7, 2026
Merged

robinnsc merged 1 commit into
mainfrom
fix/sqlite-serve-lock

Conversation

@robinnsc

@robinnsc robinnsc commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

What

A SQLite database file has one owner at a time. extenddb serve takes an exclusive flock(2) on <database>.lock before it opens the database and holds it for the life of the process; init and migrate take the same lock for as long as they run (through the bootstrapper's migration-lock hook), and destroy takes it before it unlinks the file. Each refuses, rather than waits, when another process holds it:

another extenddb process is already using /path/db.sqlite (lock held on /path/db.sqlite.lock);
stop it first, or point this command at a different database

Read-only commands (settings, manage, verify, status) take no lock and run alongside a server.

  • crates/storage-sqlite/src/serve_lock.rs: ServeLock, and database_file(), which resolves the configured location to the file SQLite actually opens by parsing it with sqlx's own SqliteConnectOptions (so sqlite: URL forms, percent-encoding such as a%20b.sqlite, and file: URIs resolve as the engine resolves them) and returns nothing for in-memory databases. The lock path is the canonicalized database path plus .lock, so two spellings of one file, a .. path, or a symlink and its target share one lock.
  • crates/storage-sqlite/src/lib.rs: the server factory takes the lock first, before it opens the database or runs startup recovery, and stores it on the engine (an Arc shared by everything in the server).
  • crates/storage-sqlite/src/bootstrapper.rs: acquire_migration_lock / release_migration_lock take and drop the same lock; drop_databases holds it while removing the file.
  • The kernel releases the lock when the process exits, including on kill -9, so a crash never leaves a stale lock. The lock file stays on disk; its presence means nothing.
  • In-memory databases (sqlite-memory, dev mode) take no lock. On non-Unix platforms no lock is taken and the server logs a warning. flock is advisory; on network filesystems its behaviour depends on the server and mount options, so the docs say to keep SQLite databases on local disk.
  • libc::flock because std's File::try_lock needs Rust 1.89 and the workspace MSRV is 1.88. libc is already a workspace dependency and listed in every license notices file.
  • Docs: docs/troubleshooting.md entry for the error with its scope; README, docs/manuals/11-deployment-guide.md, and docs/manuals/02-design-guide.md now say that one instance per catalog is the supported deployment, that SQLite enforces it, and that PostgreSQL and MongoDB do not. The deployment guide previously said multiple PostgreSQL instances were consistent because there was no in-process cache; there are per-instance credential and policy caches with no cross-instance invalidation, every worker runs on every instance, and /health is liveness only.

Why

No issue filed. Raised while reviewing #384, which removes restore targets left CREATING at SQLite startup and is only safe if one server at a time holds the file.

Nothing stopped two servers from opening the same SQLite file, and the backend is not safe that way: writers are serialized by a lock inside each server process, and startup recovery (pending control-plane transitions, GSI and vector index rebuilds, abandoned restores) assumes no other server is running. Nothing stopped destroy from unlinking the file under a running server either, which left that server serving an unlinked inode while a later init created a fresh database at the same path.

This is the SQLite half of the readiness review's P0-7 ("state that exactly one instance may run against a catalog and enforce it"). PostgreSQL and MongoDB are now documented as single-instance but nothing enforces it there; P0-7 stays open for them.

Testing done

Unit tests in extenddb-storage-sqlite:

  • database_file(): plain and relative paths, sqlite: and sqlite:// URLs with parameters, percent-encoded names, file: URIs (file:/abs, file:///abs, file://localhost/abs, file:rel; a foreign authority is an error), in-memory forms, and a disk file whose name contains mode=memory.
  • ServeLock: a second lock on the same database is refused; free again once dropped; independent across databases; shared across a .. spelling of one file and across a file symlink and a directory symlink.
  • The real server factory called twice on one file: the second call fails while the first server is up and succeeds once it is dropped.
  • The bootstrapper: destroy and migrate are refused while a server holds the file (and destroy leaves the file in place); a server is refused while migrate holds it; both succeed once it is released.

By hand (previous revision), against a release build: a second serve on the same file exits 1 with the message above and the first server stays healthy; after kill -9 of the first server a new one starts at once; two dev-mode in-memory servers both start.

cargo fmt --all -- --check
cargo +1.97.0 clippy --workspace --all-targets -- -D warnings   # the CI toolchain
cargo test -p extenddb-storage-sqlite                           # 91 passed, 1 ignored
cargo +1.88.0 check --workspace --locked

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. A startup check in the SQLite backend; no wire, trait, auth, on-disk format, or CLI flag change. The only new file on disk is the empty lock file next to the database.

Breaking changes

  • A second extenddb serve against a SQLite file another server is using, or init/migrate/destroy against a file a server is using, now fails instead of proceeding. Running those concurrently was never safe.
  • The directory holding a SQLite database must be writable so the lock file can be created; a database in a read-only directory now fails to start with cannot open lock file.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

@robinnsc
robinnsc force-pushed the fix/sqlite-serve-lock branch from aa43f5b to 539df81 Compare October 6, 2026 07:43
robinnsc added a commit that referenced this pull request Oct 6, 2026
…; recover abandoned restores

PostgreSQL and SQLite backup and restore now reproduce the table that was
backed up, with bounded memory and crash recovery. This closes the two
restore defects of the 1.0 readiness review's P0-2 (sort keys lost on
PostgreSQL, indexes dropped on both). It does not add point-in-time
recovery or off-host backup storage; those stay with #335, #336, and #337.

Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL
checked for a column named `sk`, which data tables do not have (`sk_s`,
`sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys
and restoring one left the target in CREATING for good. Backups now store
`pk` and `item_data` only; restore derives every key column from the item.
Backups written by earlier binaries carry the whole item in `item_data` and
restore correctly.

Restored, from a new per-backup table definition: global and local
secondary indexes with key schemas, projections, and GSI throughput, filled
during the copy; billing mode and provisioned throughput (was always 5/5);
table class, SSE specification, on-demand throughput. Not restored, as on
the service: streams, TTL, tags, deletion protection. A definition captured
from a PAY_PER_REQUEST table carries no provisioned throughput for the
table or its indexes, even when the catalog still holds values from before
a switch from PROVISIONED, so a restored on-demand table reports what a
freshly created one does.

Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add
`backup_definitions` (one row per backup, the definition as JSON behind a
version marker; `extenddb_storage::backup_definition`) and `table_restores`
(one row per restored table: source backup ARN and restore time, not a
foreign key to `backups` so the summary outlives the backup, as on the
service). Backups and restores from before the upgrade have no rows and
behave as before. The migration runners write the compiled catalog version
after every walk, so a replayed earlier migration cannot leave the version
behind the schema; they refuse to write it when the stored version is
newer, and `extenddb migrate` refuses a catalog newer than the binary
outright, so an older binary can no longer stamp its version onto a newer
schema and then pass its own startup gate. #335 also claims 0.0.4 and
migration 003; whichever merges second renumbers.

Consistency:
- PostgreSQL CreateBackup holds the source table row FOR SHARE from its
  first catalog read until its data snapshot (REPEATABLE READ) is taken, and
  locks the source data table ACCESS SHARE before releasing the row.
  UpdateTable and DeleteTable take the row FOR UPDATE, so the definition
  recorded is the one in force when the items were read. A GSI whose data
  table UpdateTable has not built yet is left out.
- SQLite CreateBackup reads the table row, its definition, and its items in
  one write transaction. Every other write on that server waits for it.
- A restore registers itself in the same transaction that creates its
  target (the `table_restores` row on SQLite, the provenance fields on the
  target's catalog document on MongoDB; PostgreSQL holds the backup row FOR
  SHARE while it records itself), so DeleteBackup can never observe the
  target without its source and returns BackupInUseException (HTTP 400)
  for the whole restore. A restore reads from a snapshot (PostgreSQL) or
  re-checks under the write lock for each batch (SQLite), so it copies all
  of a backup or fails with BackupNotFoundException.
- DeleteTable on a table being restored returns ResourceInUseException.
- DescribeTable reports RestoreSummary for a restored table, in progress
  while CREATING; the restore response and later DescribeTable calls carry
  the same RestoreDateTime. MongoDB gets the same two behaviours.

Memory and concurrency: backup and restore stream items and buffer at most
500 items or 4 MiB of stored JSON. SQLite restores commit per batch.
PostgreSQL runs up to four restores at once per process; a fifth waits up
to 30 s and is refused with LimitExceededException.

Failure and crash: a failed copy removes its target by table id, claiming
it CREATING -> DELETING in one statement so removal and activation cannot
both happen; MongoDB removes a partial target the same way, including when
target creation itself fails part-way. PostgreSQL restores hold a session
advisory lock keyed by account and target name on a dedicated connection;
the control-plane pass removes CREATING targets with no scheduled
transition, older than 60 s, whose lock is free. SQLite removes the same
targets at startup, which is safe only because #391 lets one server hold
the file; this change should merge after #391.

Refused rather than restored wrongly: backups of tables with vector indexes
(both backends now refuse; SQLite previously restored without them),
backups of multi-part-key tables, definitions with an unknown version, and
RestoreTableFromBackup requests carrying BillingModeOverride,
GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride,
ProvisionedThroughputOverride, or SSESpecificationOverride, which were
silently ignored before. RestoreTableFromBackup validates TargetTableName.

New DynamoDbError::BackupInUseException and StorageError::BackupInUse.

Docs: docs/dynamodb-limits.md no longer says backup and restore are
unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore
section (what is restored, the refusals, where backups live, the SQLite
write-lock cost) and its vector-index row matches the code; the upgrade
manual has a 0.0.4 section and notes the SQLite lock; the admin guide's
version literal and migrate guidance are current.

Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash
and sort keys compared item by item, a provisioned table with four GSIs
and an LSI including projections, throughput, and ordered index reads,
RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST
restoring with zero throughput on the table and its GSI);
crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL
storage-level CI step: old formats, failure cleanup, the consistency
barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the
abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11,
including the restore intent committed with its target and a 0.0.3
database in the old row format); unit tests for the definition
normalization, the override rejections, CatalogVersion ordering, and the
newer-catalog refusals in the migrate command and both schema runners.

Signed-off-by: Scott Robinson <robinnsc@amazon.com>
@robinnsc robinnsc changed the title fix(sqlite): allow one extenddb serve per database file fix(sqlite): one extenddb process per database file Oct 6, 2026
robinnsc added a commit that referenced this pull request Oct 6, 2026
…; recover abandoned restores

PostgreSQL and SQLite backup and restore now reproduce the table that was
backed up, with bounded memory and crash recovery. This closes the two
restore defects of the 1.0 readiness review's P0-2 (sort keys lost on
PostgreSQL, indexes dropped on both). It does not add point-in-time
recovery or off-host backup storage; those stay with #335, #336, and #337.

Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL
checked for a column named `sk`, which data tables do not have (`sk_s`,
`sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys
and restoring one left the target in CREATING for good. Backups now store
`pk` and `item_data` only; restore derives every key column from the item.
Backups written by earlier binaries carry the whole item in `item_data` and
restore correctly.

Restored, from a new per-backup table definition: global and local
secondary indexes with key schemas, projections, and GSI throughput, filled
during the copy; billing mode and provisioned throughput (was always 5/5);
table class, SSE specification, on-demand throughput. Not restored, as on
the service: streams, TTL, tags, deletion protection. A definition captured
from a PAY_PER_REQUEST table carries no provisioned throughput for the
table or its indexes, even when the catalog still holds values from before
a switch from PROVISIONED, so a restored on-demand table reports what a
freshly created one does.

Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add
`backup_definitions` (one row per backup, the definition as JSON behind a
version marker; `extenddb_storage::backup_definition`) and `table_restores`
(one row per restored table: source backup ARN and restore time, not a
foreign key to `backups` so the summary outlives the backup, as on the
service). Backups and restores from before the upgrade have no rows and
behave as before. The migration runners write the compiled catalog version
after every walk, so a replayed earlier migration cannot leave the version
behind the schema; they refuse to write it when the stored version is
newer, and `extenddb migrate` refuses a catalog newer than the binary
outright, so an older binary can no longer stamp its version onto a newer
schema and then pass its own startup gate. #335 also claims 0.0.4 and
migration 003; whichever merges second renumbers.

Consistency:
- PostgreSQL CreateBackup holds the source table row FOR SHARE from its
  first catalog read until its data snapshot (REPEATABLE READ) is taken, and
  locks the source data table ACCESS SHARE before releasing the row.
  UpdateTable and DeleteTable take the row FOR UPDATE, so the definition
  recorded is the one in force when the items were read. A GSI whose data
  table UpdateTable has not built yet is left out.
- SQLite CreateBackup reads the table row, its definition, and its items in
  one write transaction. Every other write on that server waits for it.
- A restore registers itself in the same transaction that creates its
  target (the `table_restores` row on SQLite, the provenance fields on the
  target's catalog document on MongoDB; PostgreSQL holds the backup row FOR
  SHARE while it records itself), so DeleteBackup can never observe the
  target without its source and returns BackupInUseException (HTTP 400)
  for the whole restore. A restore reads from a snapshot (PostgreSQL) or
  re-checks under the write lock for each batch (SQLite), so it copies all
  of a backup or fails with BackupNotFoundException.
- DeleteTable on a table being restored returns ResourceInUseException.
- DescribeTable reports RestoreSummary for a restored table, in progress
  while CREATING; the restore response and later DescribeTable calls carry
  the same RestoreDateTime. MongoDB gets the same two behaviours.

Memory and concurrency: backup and restore stream items and buffer at most
500 items or 4 MiB of stored JSON. SQLite restores commit per batch.
PostgreSQL runs up to four restores at once per process; a fifth waits up
to 30 s and is refused with LimitExceededException.

Failure and crash: a failed copy removes its target by table id, claiming
it CREATING -> DELETING in one statement so removal and activation cannot
both happen; MongoDB removes a partial target the same way, including when
target creation itself fails part-way. PostgreSQL restores hold a session
advisory lock keyed by account and target name on a dedicated connection;
the control-plane pass removes CREATING targets with no scheduled
transition, older than 60 s, whose lock is free. SQLite removes the same
targets at startup, which is safe only because #391 lets one server hold
the file; this change should merge after #391.

Refused rather than restored wrongly: backups of tables with vector indexes
(both backends now refuse; SQLite previously restored without them),
backups of multi-part-key tables, definitions with an unknown version, and
RestoreTableFromBackup requests carrying BillingModeOverride,
GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride,
ProvisionedThroughputOverride, or SSESpecificationOverride, which were
silently ignored before. RestoreTableFromBackup validates TargetTableName.

New DynamoDbError::BackupInUseException and StorageError::BackupInUse.

Docs: docs/dynamodb-limits.md no longer says backup and restore are
unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore
section (what is restored, the refusals, where backups live, the SQLite
write-lock cost) and its vector-index row matches the code; the upgrade
manual has a 0.0.4 section and notes the SQLite lock; the admin guide's
version literal and migrate guidance are current.

Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash
and sort keys compared item by item, a provisioned table with four GSIs
and an LSI including projections, throughput, and ordered index reads,
RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST
restoring with zero throughput on the table and its GSI);
crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL
storage-level CI step: old formats, failure cleanup, the consistency
barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the
abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11,
including the restore intent committed with its target and a 0.0.3
database in the old row format); unit tests for the definition
normalization, the override rejections, CatalogVersion ordering, and the
newer-catalog refusals in the migrate command and both schema runners.

Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc added a commit that referenced this pull request Oct 6, 2026
…; recover abandoned restores

PostgreSQL and SQLite backup and restore now reproduce the table that was
backed up, with bounded memory and crash recovery. This closes the two
restore defects of the 1.0 readiness review's P0-2 (sort keys lost on
PostgreSQL, indexes dropped on both). It does not add point-in-time
recovery or off-host backup storage; those stay with #335, #336, and #337.

Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL
checked for a column named `sk`, which data tables do not have (`sk_s`,
`sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys
and restoring one left the target in CREATING for good. Backups now store
`pk` and `item_data` only; restore derives every key column from the item.
Backups written by earlier binaries carry the whole item in `item_data` and
restore correctly.

Restored, from a new per-backup table definition: global and local
secondary indexes with key schemas, projections, and GSI throughput, filled
during the copy; billing mode and provisioned throughput (was always 5/5);
table class, SSE specification, on-demand throughput. Not restored, as on
the service: streams, TTL, tags, deletion protection. A definition captured
from a PAY_PER_REQUEST table carries no provisioned throughput for the
table or its indexes, even when the catalog still holds values from before
a switch from PROVISIONED, so a restored on-demand table reports what a
freshly created one does.

Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add
`backup_definitions` (one row per backup, the definition as JSON behind a
version marker; `extenddb_storage::backup_definition`) and `table_restores`
(one row per restored table: source backup ARN and restore time, not a
foreign key to `backups` so the summary outlives the backup, as on the
service). Backups and restores from before the upgrade have no rows and
behave as before. The migration runners write the compiled catalog version
after every walk, so a replayed earlier migration cannot leave the version
behind the schema; they refuse to write it when the stored version is
newer, and `extenddb migrate` refuses a catalog newer than the binary
outright, so an older binary can no longer stamp its version onto a newer
schema and then pass its own startup gate. #335 also claims 0.0.4 and
migration 003; whichever merges second renumbers.

Consistency:
- PostgreSQL CreateBackup holds the source table row FOR SHARE from its
  first catalog read until its data snapshot (REPEATABLE READ) is taken, and
  locks the source data table ACCESS SHARE before releasing the row.
  UpdateTable and DeleteTable take the row FOR UPDATE, so the definition
  recorded is the one in force when the items were read. A GSI whose data
  table UpdateTable has not built yet is left out.
- SQLite CreateBackup reads the table row, its definition, and its items in
  one write transaction. Every other write on that server waits for it.
- A restore registers itself in the same transaction that creates its
  target (the `table_restores` row on SQLite, the provenance fields on the
  target's catalog document on MongoDB; PostgreSQL holds the backup row FOR
  SHARE while it records itself), so DeleteBackup can never observe the
  target without its source and returns BackupInUseException (HTTP 400)
  for the whole restore. A restore reads from a snapshot (PostgreSQL) or
  re-checks under the write lock for each batch (SQLite), so it copies all
  of a backup or fails with BackupNotFoundException.
- DeleteTable on a table being restored returns ResourceInUseException.
- DescribeTable reports RestoreSummary for a restored table, in progress
  while CREATING; the restore response and later DescribeTable calls carry
  the same RestoreDateTime. MongoDB gets the same two behaviours.

Memory and concurrency: backup and restore stream items and buffer at most
500 items or 4 MiB of stored JSON. SQLite restores commit per batch.
PostgreSQL runs up to four restores at once per process; a fifth waits up
to 30 s and is refused with LimitExceededException.

Failure and crash: a failed copy removes its target by table id, claiming
it CREATING -> DELETING in one statement so removal and activation cannot
both happen; MongoDB removes a partial target the same way, including when
target creation itself fails part-way. PostgreSQL restores hold a session
advisory lock keyed by account and target name on a dedicated connection;
the control-plane pass removes CREATING targets with no scheduled
transition, older than 60 s, whose lock is free. SQLite removes the same
targets at startup, which is safe only because #391 lets one server hold
the file; this change should merge after #391.

Refused rather than restored wrongly: backups of tables with vector indexes
(both backends now refuse; SQLite previously restored without them),
backups of multi-part-key tables, definitions with an unknown version, and
RestoreTableFromBackup requests carrying BillingModeOverride,
GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride,
ProvisionedThroughputOverride, or SSESpecificationOverride, which were
silently ignored before. RestoreTableFromBackup validates TargetTableName.

New DynamoDbError::BackupInUseException and StorageError::BackupInUse.

Docs: docs/dynamodb-limits.md no longer says backup and restore are
unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore
section (what is restored, the refusals, where backups live, the SQLite
write-lock cost) and its vector-index row matches the code; the upgrade
manual has a 0.0.4 section and notes the SQLite lock; the admin guide's
version literal and migrate guidance are current.

Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash
and sort keys compared item by item, a provisioned table with four GSIs
and an LSI including projections, throughput, and ordered index reads,
RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST
restoring with zero throughput on the table and its GSI);
crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL
storage-level CI step: old formats, failure cleanup, the consistency
barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the
abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11,
including the restore intent committed with its target and a 0.0.3
database in the old row format); unit tests for the definition
normalization, the override rejections, CatalogVersion ordering, and the
newer-catalog refusals in the migrate command and both schema runners.

Signed-off-by: Scott Robinson <robinnsc@amazon.com>
jcshepherd
jcshepherd previously approved these changes Oct 6, 2026
Comment thread crates/storage-sqlite/src/serve_lock.rs Outdated
Comment on lines +144 to +150
let file = std::fs::OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(false)
.open(&path)
.map_err(|e| {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create the lock file as 0600. Right now it gets the default mode, so other users can read it. flock only needs a read handle, so any local user can hold the lock and stop the server from starting.

The database files are already 0600 for this reason, and the lock file should match. Please add a test for it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment on lines +116 to +117
pub(crate) fn lock_path(db_path: &Path) -> PathBuf {
let canonical = std::fs::canonicalize(db_path).unwrap_or_else(|_| {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A dangling symlink gets around the lock. Say the database path is a symlink and its target does not exist yet. The first server locks <link>.lock, the next one locks <target>.lock, and both run on the same file. Resolving the link when canonicalize fails would fix this.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, read_link loop now resolves the chain before the parent is canonicalized

Comment on lines +492 to +493
async fn release_migration_lock(&self) -> OpResult<()> {
self.held_lock

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

init releases the lock before it writes the encryption key, the default account, and the admin user. A server that starts in that window fails because the encryption key is missing. Holding the lock until init finishes would match what the description says.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Nothing stopped two `extenddb serve` processes from opening the same SQLite
file. The backend serializes writers with a lock inside the server process,
so two servers write concurrently and can hit SQLITE_BUSY mid-transaction;
and startup recovery (control-plane transitions, GSI and vector index
rebuilds, and, with #384, abandoned restores) assumes no other server is
running, so a second server starting up would undo the first one's work in
progress. Nothing stopped `destroy` from unlinking the file under a running
server either, which left that server serving an unlinked inode while a
later `init` created a fresh database at the same path.

`extenddb serve` on a file database now takes an exclusive flock(2) on
`<database>.lock` before opening the database and holds it for the life of
the process (the lock lives in the engine, so it is held while any clone
is). `init` and `migrate` take the same lock for their duration through the
bootstrapper's migration-lock hook, and `destroy` takes it before it
removes the file; each refuses, rather than waits, when a server holds it.
`init` holds it through the encryption key, default account, and admin
user, not only the schema, so a server cannot start in the window where
the schema exists but the key does not. Read-only commands (`settings`,
`manage`, `verify`, `status`) take no lock. A second holder fails with:

  another extenddb process is already using <db> (lock held on <db>.lock);
  stop it first, or point this command at a different database

The lock file is named after the file SQLite actually opens, not the
configured string: the location is parsed with sqlx's own
SqliteConnectOptions (so `sqlite:` URL forms, percent-encoding such as
`a%20b.sqlite`, and `file:` URIs resolve as the engine resolves them), and
the path is canonicalized, so two spellings of one file, a `..` path, or a
symlink and its target all share one lock. A dangling symlink (the target
not created yet) is followed by hand, since canonicalize refuses it. A disk
file whose name happens to contain `mode=memory` is a file. In-memory
databases take no lock.

The lock file is created 0600 and an existing one is tightened to 0600 on
open, matching the database and its sidecars: flock needs only a read
handle, so a world-readable lock file would let any local user hold the
lock and keep the server from starting.

The kernel releases the lock when the process exits, including on kill -9,
so there is no stale-lock cleanup. On non-Unix platforms no lock is taken
and a warning is logged. flock is advisory and its behaviour on network
filesystems depends on the server and mount; the docs say to keep SQLite
databases on local disk. The directory holding the database must be
writable so the lock file can be created.

flock through libc rather than std's File::try_lock, which needs Rust 1.89;
the workspace MSRV is 1.88. libc is already a dependency of the workspace
and listed in every license notices file.

Docs: troubleshooting entry for the error with its scope; README and the
deployment and design guides now state that one instance per catalog is
the supported deployment, that SQLite enforces it, and that PostgreSQL and
MongoDB do not (per-instance caches without cross-instance invalidation,
every worker on every instance, a liveness-only /health). The deployment
guide previously said multiple PostgreSQL instances were consistent because
there was no in-process cache, which was not true.

Tests: database_file resolution for plain paths, sqlite: URLs,
percent-encoding, file: URIs, in-memory forms, and a disk file named like
a memory parameter; ServeLock refused on the same file, independent across
files, shared across `..` spellings, across file and directory symlinks,
and across a dangling symlink and its future target (one and two links);
the lock file created 0600 and an existing 0644 one tightened; the server
factory refusing a second server on one file; destroy and migrate refused
while a server holds the file and a server refused while migrate holds it.

Signed-off-by: Scott Robinson <robinnsc@amazon.com>
@robinnsc
robinnsc added this pull request to the merge queue Oct 7, 2026
Merged via the queue into main with commit 9e3c986 Oct 7, 2026
29 checks passed
@robinnsc
robinnsc deleted the fix/sqlite-serve-lock branch October 7, 2026 18:42
robinnsc added a commit that referenced this pull request Oct 9, 2026
…; recover abandoned restores

PostgreSQL and SQLite backup and restore now reproduce the table that was
backed up, with bounded memory and crash recovery. This closes the two
restore defects of the 1.0 readiness review's P0-2 (sort keys lost on
PostgreSQL, indexes dropped on both). It does not add point-in-time
recovery or off-host backup storage; those stay with #335, #336, and #337.

Sort keys. create_backup and restore_table_from_backup on PostgreSQL
checked for a column named `sk`, which data tables do not have (sk_s,
sk_n, sk_b). Every backup of a table with a sort key lost its sort keys
and restoring one left the target in CREATING for good. Backups now store
pk and item_data only; restore derives every key column from the item.
Backups written by earlier binaries carry the whole item in item_data and
restore correctly.

Restored, from a new per-backup table definition: global and local
secondary indexes with key schemas, projections, and GSI throughput, filled
during the copy; billing mode and provisioned throughput (was always 5/5);
table class, SSE specification, on-demand throughput. Not restored, as on
the service: streams, TTL, tags, deletion protection. A definition captured
from a PAY_PER_REQUEST table carries no provisioned throughput for the
table or its indexes, even when the catalog still holds values from before
a switch from PROVISIONED, so a restored on-demand table reports what a
freshly created one does.

RestoreTableFromBackup overrides. BillingModeOverride and
ProvisionedThroughputOverride are implemented on all three backends and
passed to storage as an explicit `overrides: RestoreTableOverrides`
parameter on BackupEngine::restore_table_from_backup (a trait change).
BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST
normalization and refuses, before any target is created, a PROVISIONED
restore of a backup whose GSIs carry no throughput (the service needs
GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride,
LocalSecondaryIndexOverride, SSESpecificationOverride,
OnDemandThroughputOverride, and VectorIndexOverride are refused with
ValidationException before any storage read; they were silently ignored.
Throughput values reuse CreateTable's validator so the error text matches.

Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add
backup_definitions (one row per backup, the definition as JSON behind a
version marker; extenddb_storage::backup_definition) and table_restores
(one row per restored table: source backup ARN and restore time, not a
foreign key to backups so the summary outlives the backup, as on the
service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore
cursors on it rather than the implicit rowid, which a VACUUM between
batches could renumber. An existing backup_items is rebuilt in place
(copy in rowid order, idempotent). backup_items.pk now holds the real
partition key on SQLite as it does on PostgreSQL. Backups and restores from
before the upgrade have no definition rows and behave as before. The
migration runners write the compiled catalog version after every walk, so
a replayed earlier migration cannot leave the version behind the schema;
they refuse to write it when the stored version is newer, and `extenddb
migrate` refuses a catalog newer than the binary outright, so an older
binary can no longer stamp its version onto a newer schema and then pass
its own startup gate. #335 also claims 0.0.4 and migration 003; whichever
merges second renumbers.

Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE
from its first catalog read until its data snapshot is taken, and the
snapshot is pinned by a real read of the table (LOCK TABLE is a utility
statement and does not establish a REPEATABLE READ snapshot; the first
revision relied on it and the definition and the items could come from
different instants). UpdateTable and DeleteTable take the row FOR UPDATE,
so the definition recorded is the one in force when the items were read.
SQLite CreateBackup takes the engine write lock only to read the
definition, insert the backup row as CREATING, and open the item reader's
WAL snapshot; the items are then streamed without the lock and written in
batches under short lock holds, so other writers wait for a batch rather
than the whole backup (a 100,000-item backup previously stalled every write
for about twelve seconds). The copy runs detached from the request, so a
client that disconnects does not leave a half-written CREATING backup;
one left by a crash is removed at startup. A CREATING backup is visible to
ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses
it. The long reader pins WAL checkpointing for the backup's duration. A
file-backed SQLite pool is clamped to at least two connections so the
reader and the batch writer can coexist.

A restore registers itself in the same transaction that creates its target
on PostgreSQL (backup row FOR SHARE, table_restores inserted with the
tables row) and SQLite, so DeleteBackup orders strictly before (the restore
then sees no backup) or after (it sees the restore, BackupInUseException).
MongoDB has no cross-collection transaction and uses a claim marker on the
backup's metadata instead: a restore claims before reading and releases
once its CREATING target exists; DeleteBackup claims the same way before
dropping data; stale claims from a crashed process are cleared at startup.
DeleteTable on a table being restored returns ResourceInUseException on all
three backends. DescribeTable reports RestoreSummary for a restored table,
in progress while CREATING; the restore response and later DescribeTable
calls carry the same RestoreDateTime.

Memory and concurrency: backup and restore stream items and buffer at most
500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at
once per process; a fifth waits up to 30 s and is refused with
LimitExceededException.

Failure and crash: a failed copy removes its target by table id, claiming
it CREATING -> DELETING in one statement so removal and activation cannot
both happen; MongoDB removes a partial target the same way. PostgreSQL
restores hold a session advisory lock keyed by account and target name on
a dedicated connection (its idle_session_timeout SET is best-effort, for
managed instances that refuse it); the control-plane pass removes CREATING
targets with no scheduled transition, older than 60 s, whose lock is free,
which with the pass's 60 s idle timeout means between 60 and about 120 s.
SQLite removes the same targets at startup, which is safe because one
server at a time holds the file (#391). The control-plane worker's
DeleteTable drop runs under a 3 s lock_timeout and drops data tables before
deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE
lock fails fast and the row stays DELETING for the next pass (the previous
order deleted the row first, so a failed drop was never retried). The
stale-vector-index recovery skips indexes whose parent table is DELETING
and cleans up if the parent vanishes under it, so the reordered drop cannot
leave an orphan vector table.

Refused rather than restored wrongly: backups of tables with vector indexes
(both backends now refuse; SQLite previously restored without them),
backups of multi-part-key tables, definitions with an unknown version, and
the unsupported overrides above. RestoreTableFromBackup validates
TargetTableName. New DynamoDbError::BackupInUseException and
StorageError::BackupInUse.

Docs: docs/dynamodb-limits.md no longer says backup and restore are
unsupported and its summary row is recounted; docs/differences-from-
dynamodb.md gains a Backup and Restore section with every row qualified by
backend (what is restored, the seven overrides, the refusals, CREATING
visibility, the delete race, crash recovery, where backups live, the
consistency model and WAL cost); the upgrade manual has a 0.0.4 section
that warns PostgreSQL operators that targets the old bug left in CREATING
are removed on the first control-plane pass after the upgrade, describes
the SQLite lock, and notes WAL headroom; the admin guide's version literal
and migrate guidance are current.

Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash
and sort keys compared item by item; a provisioned table with four GSIs and
an LSI including projections, throughput, and ordered index reads;
RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring
with zero throughput; BillingModeOverride in both directions; a refused
PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no
target). crates/storage-postgres/tests/backup_restore.rs (21, in the
PostgreSQL storage-level CI step: old formats, failure cleanup, the
consistency barrier including a production-path test that fails if the
snapshot-pinning read is removed and the two transaction shapes that show
why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both
orders, the abandoned-restore sweep, the drop lock_timeout leaving a row
DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a
writer that keeps making progress during a backup, items written after the
snapshot excluded, a cancelled request finishing with every item, the
startup sweeps, a 0.0.3 database in the old row format, the backup_items
rebuild). Unit tests for the definition normalization and override
preflight (12), the engine's override parsing and refusals (15), Mongo's
claim predicates and GSI throughput handling, CatalogVersion ordering, the
newer-catalog refusals, and the vector-recovery predicate.

Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc added a commit that referenced this pull request Oct 12, 2026
…; recover abandoned restores

PostgreSQL and SQLite backup and restore now reproduce the table that was
backed up, with bounded memory and crash recovery. This closes the two
restore defects of the 1.0 readiness review's P0-2 (sort keys lost on
PostgreSQL, indexes dropped on both). It does not add point-in-time
recovery or off-host backup storage; those stay with #335, #336, and #337.

Sort keys. create_backup and restore_table_from_backup on PostgreSQL
checked for a column named `sk`, which data tables do not have (sk_s,
sk_n, sk_b). Every backup of a table with a sort key lost its sort keys
and restoring one left the target in CREATING for good. Backups now store
pk and item_data only; restore derives every key column from the item.
Backups written by earlier binaries carry the whole item in item_data and
restore correctly.

Restored, from a new per-backup table definition: global and local
secondary indexes with key schemas, projections, and GSI throughput, filled
during the copy; billing mode and provisioned throughput (was always 5/5);
table class, SSE specification, on-demand throughput. Not restored, as on
the service: streams, TTL, tags, deletion protection. A definition captured
from a PAY_PER_REQUEST table carries no provisioned throughput for the
table or its indexes, even when the catalog still holds values from before
a switch from PROVISIONED, so a restored on-demand table reports what a
freshly created one does.

RestoreTableFromBackup overrides. BillingModeOverride and
ProvisionedThroughputOverride are implemented on all three backends and
passed to storage as an explicit `overrides: RestoreTableOverrides`
parameter on BackupEngine::restore_table_from_backup (a trait change).
BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST
normalization and refuses, before any target is created, a PROVISIONED
restore of a backup whose GSIs carry no throughput (the service needs
GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride,
LocalSecondaryIndexOverride, SSESpecificationOverride,
OnDemandThroughputOverride, and VectorIndexOverride are refused with
ValidationException before any storage read; they were silently ignored.
Throughput values reuse CreateTable's validator so the error text matches.

Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add
backup_definitions (one row per backup, the definition as JSON behind a
version marker; extenddb_storage::backup_definition) and table_restores
(one row per restored table: source backup ARN and restore time, not a
foreign key to backups so the summary outlives the backup, as on the
service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore
cursors on it rather than the implicit rowid, which a VACUUM between
batches could renumber. An existing backup_items is rebuilt in place
(copy in rowid order, idempotent). backup_items.pk now holds the real
partition key on SQLite as it does on PostgreSQL. Backups and restores from
before the upgrade have no definition rows and behave as before. The
migration runners write the compiled catalog version after every walk, so
a replayed earlier migration cannot leave the version behind the schema;
they refuse to write it when the stored version is newer, and `extenddb
migrate` refuses a catalog newer than the binary outright, so an older
binary can no longer stamp its version onto a newer schema and then pass
its own startup gate. #335 also claims 0.0.4 and migration 003; whichever
merges second renumbers.

Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE
from its first catalog read until its data snapshot is taken, and the
snapshot is pinned by a real read of the table (LOCK TABLE is a utility
statement and does not establish a REPEATABLE READ snapshot; the first
revision relied on it and the definition and the items could come from
different instants). UpdateTable and DeleteTable take the row FOR UPDATE,
so the definition recorded is the one in force when the items were read.
SQLite CreateBackup takes the engine write lock only to read the
definition, insert the backup row as CREATING, and open the item reader's
WAL snapshot; the items are then streamed without the lock and written in
batches under short lock holds, so other writers wait for a batch rather
than the whole backup (a 100,000-item backup previously stalled every write
for about twelve seconds). The copy runs detached from the request, so a
client that disconnects does not leave a half-written CREATING backup;
one left by a crash is removed at startup. A CREATING backup is visible to
ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses
it. The long reader pins WAL checkpointing for the backup's duration. A
file-backed SQLite pool is clamped to at least two connections so the
reader and the batch writer can coexist.

A restore registers itself in the same transaction that creates its target
on PostgreSQL (backup row FOR SHARE, table_restores inserted with the
tables row) and SQLite, so DeleteBackup orders strictly before (the restore
then sees no backup) or after (it sees the restore, BackupInUseException).
MongoDB has no cross-collection transaction and uses a claim marker on the
backup's metadata instead: a restore claims before reading and releases
once its CREATING target exists; DeleteBackup claims the same way before
dropping data; stale claims from a crashed process are cleared at startup.
DeleteTable on a table being restored returns ResourceInUseException on all
three backends. DescribeTable reports RestoreSummary for a restored table,
in progress while CREATING; the restore response and later DescribeTable
calls carry the same RestoreDateTime.

Memory and concurrency: backup and restore stream items and buffer at most
500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at
once per process; a fifth waits up to 30 s and is refused with
LimitExceededException.

Failure and crash: a failed copy removes its target by table id, claiming
it CREATING -> DELETING in one statement so removal and activation cannot
both happen; MongoDB removes a partial target the same way. PostgreSQL
restores hold a session advisory lock keyed by account and target name on
a dedicated connection (its idle_session_timeout SET is best-effort, for
managed instances that refuse it); the control-plane pass removes CREATING
targets with no scheduled transition, older than 60 s, whose lock is free,
which with the pass's 60 s idle timeout means between 60 and about 120 s.
SQLite removes the same targets at startup, which is safe because one
server at a time holds the file (#391). The control-plane worker's
DeleteTable drop runs under a 3 s lock_timeout and drops data tables before
deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE
lock fails fast and the row stays DELETING for the next pass (the previous
order deleted the row first, so a failed drop was never retried). The
stale-vector-index recovery skips indexes whose parent table is DELETING
and cleans up if the parent vanishes under it, so the reordered drop cannot
leave an orphan vector table.

Refused rather than restored wrongly: backups of tables with vector indexes
(both backends now refuse; SQLite previously restored without them),
backups of multi-part-key tables, definitions with an unknown version, and
the unsupported overrides above. RestoreTableFromBackup validates
TargetTableName. New DynamoDbError::BackupInUseException and
StorageError::BackupInUse.

Docs: docs/dynamodb-limits.md no longer says backup and restore are
unsupported and its summary row is recounted; docs/differences-from-
dynamodb.md gains a Backup and Restore section with every row qualified by
backend (what is restored, the seven overrides, the refusals, CREATING
visibility, the delete race, crash recovery, where backups live, the
consistency model and WAL cost); the upgrade manual has a 0.0.4 section
that warns PostgreSQL operators that targets the old bug left in CREATING
are removed on the first control-plane pass after the upgrade, describes
the SQLite lock, and notes WAL headroom; the admin guide's version literal
and migrate guidance are current.

Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash
and sort keys compared item by item; a provisioned table with four GSIs and
an LSI including projections, throughput, and ordered index reads;
RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring
with zero throughput; BillingModeOverride in both directions; a refused
PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no
target). crates/storage-postgres/tests/backup_restore.rs (21, in the
PostgreSQL storage-level CI step: old formats, failure cleanup, the
consistency barrier including a production-path test that fails if the
snapshot-pinning read is removed and the two transaction shapes that show
why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both
orders, the abandoned-restore sweep, the drop lock_timeout leaving a row
DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a
writer that keeps making progress during a backup, items written after the
snapshot excluded, a cancelled request finishing with every item, the
startup sweeps, a 0.0.3 database in the old row format, the backup_items
rebuild). Unit tests for the definition normalization and override
preflight (12), the engine's override parsing and refusals (15), Mongo's
claim predicates and GSI throughput handling, CatalogVersion ordering, the
newer-catalog refusals, and the vector-recovery predicate.

Signed-off-by: Scott Robinson <robinnsc@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants