[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework · PR 6 the catalog index · this PR: configuration, the store identity file and backup purge · next: the CreateBackup job and worker (PR 8), the restore job (PR 9a).
Closes #670 (part of #624).

Stacked on PR 6 (feat/backup-catalog-index, head 7f79f87e), itself on #666's 4fa6769b (main through #395, #638, #412, #511, #471, #473, #680, #429): this PR adds nothing to the catalog schema (PR 6 created backup_maintenance, the store_* columns and catalog_id); it binds them. The diff shows this PR's single commit on top of PR 6's. Rebasing onto that head conflicted in one file, the design document, in two places: PR 6's interface block, where PR 6 now lists list_dead_restore_jobs and fail_restore_job and this PR changes check_maintenance_claim to take the stale window (both kept), and the revision log, where PR 6 extended item 27 and this PR adds item 28 (both kept). The rebase also meets #666 and #473 in code that merged without a conflict: PR 6's MongoDB factory now checks the catalog before it builds the data engine and starts #473's key repair, and this PR has the repair start only for a server, never for a command reading beside one; #666's PostgreSQL serving registration, which every backup command takes, is named after the command and its refusals end the command with serve's exit codes (below). Where the two meet: the reconciler's cycle carries PR 6's ReconcilerState (here with the shutdown token), worker metrics and FAILED-target duty, whose three-way Removal reports a target left to its owner (targets_left_to_owner, which backup purge's zero-window pass prints), beside this PR's store check and DELETING duty; the permit refusal is PR 6's limit_exceeded_message(JobBound) naming [backup] max_concurrent_jobs, while the maintenance refusal keeps its own text naming the command; CreateBackup's request checks run in PR 6's order (check_backup_arn_fits: the region, then the table name, before the ARN), then the maintenance claim, then the permit; RestoreTableFromBackup's run in one order too: PR 6's target-name check through TableEngine::describe_table (check_restore_target_name), the backup row, this PR's store-health refusal on a row of the configured store, the maintenance claim, the permit (the engine's restore doc and the design's 5.3 step 1 with revision item 28 state it); the SQLite snapshot-slot refusal names PR 6's [storage.sqlite] pool_size snapshot-slot bound, and the limits doc's bound-naming sentence carries this PR's exception for the maintenance claim; DeleteBackup over a row builds its description before the DELETING write (PR 6) and then deletes the objects (this PR); backup purge's claim heartbeats at PR 6's JOB_HEARTBEAT_INTERVAL; the SQLite beside_server path builds the engine through the one constructor, so the backup slots it shares with #395's snapshots and PR 6's synchronous permit are the server's; Cassandra's list_dead_restore_jobs lists PR 6's DELETING targets with their registration and reads the coordinator clock after its scans (this PR); the engine's test double carries PR 6's record_still_running and its TableEngine (tables) beside this PR's row and store knobs, and PR 6's other-store cell builds its runtime through this PR's ConfiguredStore; every Cassandra statement here names its keyspace, as #638 requires.

What

The [backup.store] configuration with its per-backend defaults and validation, the store identity file and the health component, put_if_absent on the store trait, the maintenance claim on every backend, DeleteBackup running against the configured store, and extenddb backup list and backup purge. No request or response shape changes; the API-visible changes are the refusals of a store-backed row's restore and delete while a configured store is unhealthy or its first check pending, the LimitExceededException of CreateBackup and RestoreTableFromBackup while a maintenance claim is held (its message says a backup maintenance command holds the claim, without naming one), and the /health body (reason is the problem's category alone; checking before an S3 store's first check). backup list and backup purge say why there is no store (the default root failed its check, the section absent, disabled, the root not yet created); a defaulted root with rows for it is reported and refused as a configured one; the refusal before the first check says the first check is pending. backup list and the dry run give up on a store that does not answer after the check's bound; serve and the commands exit within 5 s of their last work with a thread parked on a stalled mount; a FIFO at the identity key is reported as neither readable nor creatable; a root with edge whitespace is quoted in the banner and the startup error.

Store (crates/backup-store). BackupStore::put_if_absent(key, body) -> Option<ObjectMeta> creates the object only if none is under the key and returns None, leaving the existing object untouched, otherwise. S3 sends PutObject (or CompleteMultipartUpload for a body over one part) with If-None-Match: *; a 412 is "present", a 409 ConditionalRequestConflict is re-sent after a back-off of 100 ms doubling to 1.6 s, eight waits in all, and a conditional complete that finds the key present aborts its upload. The filesystem store writes and syncs its temporary as put does (created O_CREAT | O_EXCL) and links it to the key, which fails if anything is there, so the object appears complete or not at all; the temporary is unlinked on every path; the trait doc says a failed parent sync after link leaves the object. put's temporary write and ancestor sync are factored out and shared; put_impl's unreachable arm is gone. BackupStoreKind::Disabled and open_described (the store with its kind and root label: the canonical directory, or s3://<bucket>/<prefix> with ?endpoint=<url> for the endpoint the client uses, from the config or the SDK's environment and shared config, spelled one way: lowercase scheme and host, no default port, no / for an empty path, no user information, query or fragment) are new; validate keeps a transport failure's class (the bucket joins the error chain) so a caller can tell it from a configuration error. check_private_directory(dir, what) is exported for the spool. README and module docs updated; the README names the S3-compatible-server caveat (a server that ignores If-None-Match makes the create a plain put) and the hard-link requirement (CIFS and FUSE mounts that refuse link with EPERM cannot hold a root).

Config (crates/config/src/backup.rs). store: Option<BackupStoreConfig> under [backup] (the crate's type, nested under its own key so deny_unknown_fields holds on both tables). resolve_store(database_file) applies the per-backend rule: an absent section on a file-backed SQLite database is a filesystem store at backups/ beside the file, marked defaulted (created 0700 by the server; a failure of its open, validation or identity check is a warning that leaves the store unconfigured); on a server database or an in-memory one it is unconfigured; kind = "disabled" is unconfigured on purpose. The spool: required with an S3 store except beside a database file (backup-spool/); an explicit spool_path is refused inside or containing a filesystem root, and when it is or contains the database directory, compared on normalized and, where they exist, canonicalized paths; a filesystem root that is or contains the database directory is refused beside the spool rule; a placement refusal spells both paths as compared. A relative [backup.store.filesystem] path or spool_path is made absolute against the working directory at load (make_paths_absolute), as storage.path is; an empty path is compared with nothing in resolve_store and left for open to refuse. StorageConfig::database_file (default None; SQLite answers its file through the one database_file rule it already has) is how the server learns the backend's file. The init template and extenddb.sample.toml carry the commented block.

Storage (crates/storage/src/backup_catalog.rs). read_store_ids(store_kind, store_root): the distinct ids of a root's rows, every status. maintenance_claim(holder, stale_after) -> Box<dyn MaintenanceClaim> and check_maintenance_claim(stale_after): the claim is backup_maintenance's one record, born owned with a fresh generation, heartbeated and released through the handle; a holder whose heartbeat is older than the stale window is dead, its claim is not reported held and is taken over; a second live holder is AlreadyExists naming the first; on PostgreSQL the check tests the advisory lock first (a transaction-scoped shared try-lock in one autocommit statement, so a cancelled request leaves no lock on the pool), so a live session is held whatever its heartbeat says; on SQLite a server's startup under the serve lock removes the record outright, and a command under the serve lock takes over a record whatever its heartbeat says (the lock is the proof that its holder is dead); on PostgreSQL the take tries the exclusive lock up to six times 2 ms apart on its claim connection, so a job's one-statement shared lock does not refuse it with no holder to name. remove_dead_creating_backup_row(key, stale_after): a CREATING row whose heartbeat is older than the window is removed outright (the maintenance command's write). list_backup_rows_older_than(status, store, clock, age) with StaleClock::{Heartbeat, Completed}: the rows of a store with a status whose heartbeat_at, or completed_at (never older when unset), is older than the age, compared in the backend's own query on the database's clock (Cassandra: the coordinator's, read once before the scan); purge's test of a live owner, and the listing PR 8's reconciler duties read. ServerComponentsOptions::beside_server: a read beside a server takes no SQLite serve lock and runs no startup recovery (the catalog's status is still read, on a read-only connection); on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB the repair of documents stored without their key (#473) only for a server. ServerComponentsOptions::command names the command that opens the catalog (None is serve): on PostgreSQL the serving registration every caller takes carries it as its session's application_name (ServerRegistration::register_as, kept when the registration is taken again), so migrate --yes names the command while it waits for it, and a refusal while a migration runs names the migration's sessions without the server's advice to start the server. Per backend: PostgreSQL takes an advisory lock (backup_maintenance, the shared seed) on a dedicated connection as the proof of a live holder and upserts the row, and release deletes the row and closes the session; SQLite takes the row under the engine write lock; MongoDB takes over a stale record with a conditional update and inserts a missing one with an upsert whose filter a live record never matches ($expr is not allowed in an upsert's filter, so the two are two writes); Cassandra inserts IF NOT EXISTS and takes over a stale row IF owner_generation = <old>, its claim-row reads run at LOCAL_SERIAL, and an insert whose answer was lost is settled by re-reading the generation; its list_dead_restore_jobs reads the coordinator clock after its scans. MaintenanceClaim is a new trait (generation, heartbeat, release).

Engine (crates/engine/src/backup/). BackupRuntime.configured_store is a ConfiguredStore: kind, root label, the store handle, the spool (path, spool_host = <hostname>:<path>), the id once a check has read it, and the health as the last check left it: Checking until an S3 store's first check, Healthy, or Unhealthy; a transition between categories is logged once; a check that cannot read the catalog changes nothing. check_configured_store runs validate (until it has passed) then establish or recheck, under the store's bound (5 s for a filesystem root, 30 s for a bucket, under the SDK's per-attempt timeout), recording a timed-out check as unreachable; a filesystem root's calls run as tasks of their own, so an abandoned check's call stays parked on one thread until the mount answers and no check starts while it is (recorded unreachable again); an S3 store's abandoned request is dropped with the check; read_identity_bounded is the command's identity read under the same bound; the reconciler's cycle, which selects it against the shutdown token, and the backup command call it. A transport problem's text is the store error's whole cause chain; every other store error of a check is failed validation. store_identity runs the checks: establish reads extenddb-store.json, creates it with put_if_absent when the root has no file and the catalog no rows for the root, re-reads it, and compares it with catalog_id and the rows' ids; recheck re-reads and compares, rewriting a lost file with the known id when no rows exist; both end with the manifest probe (head of the newest AVAILABLE row's manifest). The identity file is read up to 64 KiB. The problems: another catalog's file, an id the rows do not carry, more than one id on the rows, a missing file with rows, a create race lost to a store that ignored the condition (the re-read holds another id), a missing manifest, the store unreachable, a failed validation, the file unreadable, the catalog unreadable; each has an operator line naming the ids and a /health reason that is the category alone. require_healthy_store is the ValidationException naming the kind and the category, or, before the first check, that the first check is pending. CreateBackup takes the in-catalog path whatever the store's health (the gate behind STORE_BACKED_CREATE is PR 8's) and logs that the store is configured; the in-catalog create and the restore refuse with LimitExceededException while a maintenance claim is held, after every other check and before the permit (refuse_while_maintenance_held; the check runs before the job's row, which the backends commit and detach from), the message saying that a backup maintenance command holds the claim (every maintenance command holds the one claim, so it names none). HeldClaim heartbeats a maintenance claim on a timer for its life (Drop stops the timer; release waits for a heartbeat in flight). DeleteBackup of a configured store's row runs finish_backup_delete after start_backup_delete: the manifest, the prefix, a listing that must be empty, set_backup_deleted; a failure leaves the row DELETING and answers InternalServerError. The reconciler's cycle runs the store's check each cycle, records the outcome, and while healthy finishes the configured store's DELETING rows; while unhealthy it touches no row of the store; while a maintenance claim is held it runs only the duties the command waits on or finishes itself (the branch the later duties go in is empty and says so); check_maintenance_claim takes the stale window. A caller that holds the claim says so through its state (ReconcilerState::for_maintenance_holder) and the cycle takes the held branch without reading the claim; a cycle run with a zero dead-owner window would otherwise read its caller's own claim through that window, and see it held only within one millisecond of its heartbeat. wait_for_quiet tells a live owner from a dead one through list_backup_rows_older_than on the heartbeat clock, never with this host's clock. maintenance also carries wait_for_quiet (WaitOutcome), purge_row and Interrupt, the purge's steps against the catalog trait, tested against the engine's double.

Server (crates/server/src/backup_store.rs, lib.rs). Startup resolves and opens the store, takes the spool (0700, .lock 0600 under flock; another live holder is a startup error naming the path; an existing spool must be owned by the server's user with mode 0700, parents included), and for a filesystem store runs validate and establish under a 5 s bound (LOCAL_CHECK_TIMEOUT, in the engine beside S3_CHECK_TIMEOUT, 30 s; past it, a warning and the store checking; a root that does not answer before open holds startup, as the database directory would); an S3 store's validate and establish run in the reconciler's first cycle, after the listener is up, so a stalled endpoint never holds the data plane (the store is checking until then; a validate refusal there makes it unhealthy). A configuration error that needs no request (open refusing the configuration, the spool rule, an S3 store without a spool on a backend with no default) stops startup with the path or bucket named; the defaulted SQLite root turns a validation or identity-check failure into a warning that leaves the store unconfigured while the catalog has no rows for it (read_store_ids); with rows for it the root is handled as a configured one, and configure_described carries the reason for an absent store out (NotConfigured) for backup list and backup purge to print. A filesystem root's I/O failure at validate's probe write degrades the store rather than stopping startup; the owner-only and existence refusals stay startup errors. A transport, identity or probe failure, or a check that does not finish within its bound, leaves the server running with the store unhealthy and logs it. /health is {"status": "healthy" | "degraded", "components": {...}}, HTTP 200 either way, with backup_store (status healthy, checking or unhealthy, kind, and when unhealthy reason, the category) when a store is configured; the console banner names the store beside the storage line, a filesystem root in the spelling the log line and the rows use (the canonical directory) with the configured spelling beside it only where the two differ (a root reached through a symlink or written relative), and the log names its id and the spool. catalog_refusal builds the message and exit code serve gives for a catalog the backend refuses, for serve and the backup commands alike.

App (crates/app/src/cmd_backup.rs). extenddb backup list [--failed] prints the catalog's rows for the configured store's root, every store id they carry, with each row's store_root and store_id; it is a read beside a server (no SQLite serve lock, no startup recovery), reads the identity file without creating one, under the store's check bound (5 s for a root, 30 s for a bucket), runs on an unhealthy store naming the problem in its header, and prints (none) with exit 0 when the defaulted root does not exist yet. extenddb backup purge --confirm opens the catalog as serve does (on SQLite under the serve lock), refuses an unhealthy store, prints the plan, takes the maintenance claim and heartbeats it for its whole run (HeldClaim at JOB_HEARTBEAT_INTERVAL; a lost claim stops the deletion loop before its next row), on SQLite runs the reconciler's duties with a zero window before the wait, as the claim's holder (a legacy restore the stopped server left RUNNING is failed and its target removed (a manifest one, PR 9a's, is left for its resumer and waited on), the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished, a target that is no longer the restore target its record names printed as left), waits up to 2 * stale_job_after_secs until no CREATING or FAILED row has a heartbeat within the window (on the catalog's clock, through list_backup_rows_older_than) and no restore job is RUNNING, refuses and releases if they have not gone (and releases on a catalog read failure), then deletes each row: a CREATING row with a dead owner through remove_dead_creating_backup_row first and its prefix after, every other row's prefix (manifest first) and then the row outright through the status machine and remove_backup_row, leaving extenddb-store.json, exports/ and prefixes with no row; each deletion is logged under extenddb::audit. A row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run right after the killed one takes it over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. Without --confirm it is a read like list: it prints the plan, changes nothing and exits 0, with no store configured as with one (an empty plan naming the reason). --config is accepted before or after the subcommand. On PostgreSQL list, the dry run and purge --confirm each hold a serving registration for their run whose session is named extenddb backup list or extenddb backup purge, so migrate --yes waits for them and names them; a catalog the backend refuses ends the command with serve's message and exit code (6 while a migration runs, naming its sessions; 3 for a catalog behind the binary, with the migrate --yes advice), where it exited 1 with a message telling the operator to start a server. On SQLite purge --confirm beside a running server is refused on the serve lock with exit 6, as a second serve is, where it exited 1. Every command and serve end their tokio runtime with a 5 s bounded shutdown, so a thread parked on a mount does not hold the process open past its drain. extenddb destroy's help and output say the store is left untouched and that a re-created catalog needs the old root's extenddb-store.json removed or a new root.

Docs. Admin guide ([backup.store] section: the rules, the identity file, the health component, backup list/purge (on PostgreSQL their serving registration and the refusals' exit codes, on SQLite the serve lock's exit 6), the shared-root rule, the re-init paragraph; /health examples and the reason list with catalog unreadable; the spool_path row), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; disabled as the opt-out; the SQLite default root rule and what the warning means; the re-init sentence; the rollback paragraph's store note), the deployment guide's multi-instance notes (/health degraded, one shared root, one spool per instance, the hard-link requirement, two host-local roots on an empty catalog), the sample config and init template ("will live, from a later release"), the store crate's README, the four backends' and the engine's module docs, design revision item 28. The four licence notice files are regenerated: the server binaries now link the store crate's AWS SDK tree (Apache-2.0, MIT, ISC), which cargo about adds.

Why

#396 sections 4(c), 4(f), 7, 8, 9 and the PR 7 section. The store-backed CreateBackup (PR 8) and the restore job (PR 9a) need a configured, validated store whose identity the catalog's rows carry, a health state that refuses work when the root is not the one the rows describe, and a maintenance claim that excludes jobs on every instance while an operator deletes, which this PR already applies to the in-catalog create and restore, before their row. Landing them first, with the identity file and the health component, lets PR 8 write rows under an id that is already checked at startup and each cycle.

Decisions taken while implementing, each stated in the module doc of the code it concerns: maintenance_claim takes the holder's label and the stale window and returns a MaintenanceClaim (heartbeat and release) rather than a JobClaim, and check_maintenance_claim takes the stale window (the design's signatures had no duration to measure a dead holder by, and no release); a configured store whose startup check failed keeps its id unknown, and a row of its kind and root is a row of this store for the refusals until a check reads the id; CreateBackup takes the in-catalog path whatever the store's health (no store-backed writer exists on this head); an S3 store's checks run after the listener so an endpoint cannot hold startup; purge's "live owner" for a CREATING or FAILED row is the backend's list_backup_rows_older_than on the database's clock (an earlier version of this PR compared heartbeat_at with this host's clock and said that could only lengthen the wait, which is false: a host running ahead of the database shortens it; the method is the listing PR 8's reconciler needs, brought into this PR so the wait is right on its own); the purge's own reconciler cycle is told that its caller holds the claim through a ReconcilerState constructor rather than a second duration, so every other caller keeps the one-window call; purge waits on FAILED rows with a live heartbeat and deletes CREATING rows with a dead owner, the window being zero on SQLite; the restore part of its wait set is the RUNNING records alone: the FAILED records the zero-window listing also returns are the reconciler's duty; purge does not wait on DELETING rows but finishes them itself (nothing heartbeats one, the sequence is idempotent, and on SQLite no reconciler runs while the command holds the file); the jobs' claim check runs before the row on this head; the MongoDB claim is two writes.

Testing done

Counts are of the whole tree on this head, rebased onto PR 6's head 7f79f87e; "new" is against that head.

• extenddb-backup-store: lib 37 (4 new: the conditional outcomes from synthetic replies, the root label in one spelling from four spellings of an endpoint, a validation transport failure keeping its class, disabled); conformance 123 (7 new: put_if_absent creates the object and answers None for a key that exists on both stores, the second call on the large key multipart so the aborted upload on S3 is exercised (on the filesystem store the existence check before the link already answers a second sequential call, so this test does not reach the conditional step there); eight concurrent callers on one key leave the winner's body, three times over, both stores, the test that catches a create that is last-writer-wins (with rename in place of link the filesystem store fails it); the filesystem refusals; an empty root path refused by name; a scripted HTTP/1.1 server answering 409 twice then 200, 409 then 412, and 409 forever, with the back-off measured), against moto at 127.0.0.1:9200; the SDK-environment cells also assert store_root() names the env-, profile- and services-sourced endpoint.
• extenddb-config: 23 (9 new, 1 rewritten: the section parses as the crate's shape and refuses unknown keys at every level, the per-backend default, the spool required with S3 except beside a database file, every cell of the spool placement rule, path comparison through a symlink, relative paths made absolute with the empty-path cell, an empty root compared with nothing and left for open, a root that is or contains the database directory refused and a child, a sibling and the default accepted, a placement refusal spelling both paths as compared).
• extenddb-engine: 243 (35 new: the identity file's shape and refusals, the public reason naming no root, compare's rules; an empty root creates the file and a second startup reads it; two instances on one empty root end with one id; another catalog's file; ids that disagree, a missing file with rows, rows with two ids; the manifest probe on the newest AVAILABLE row; a create race lost on a store ignoring the condition; an unreachable store unhealthy and recovering through the reconciler, with the id settled on recovery and both links of the transport cause chain in the reason; a stalled store's check abandoned at its bound (5 s for a filesystem root, 30 s for a bucket) with the dead-owner duties run and the store unreachable, the next cycle on a stalled root starting no check and waiting no bound while the abandoned call is parked, and on a stalled bucket each cycle sending a fresh request; a command's identity read returning at the same bound; a FIFO at the identity key neither readable nor creatable; a shutdown during the check ending the cycle without waiting the bound; a transition logged across categories only; a recheck rewriting a lost file; a store checked after startup checking until its first check, its refusal saying the first check is pending; the health reason the category only; an oversized identity file unreadable; a catalog read failure leaving the health unchanged; the refusals naming the kind on delete and the restore check with no slot taken, and the in-catalog create not refused; a healthy store leaving create on the in-catalog path; jobs refused while a maintenance claim is held, the window passed through; the heartbeat timer keeping a claim held through a long run and a dropped handle stopping it, on tokio's paused clock; delete of a configured store's row deleting manifest, prefix, then marking DELETED; a failed deletion left DELETING and finished by the reconciler once the store answers; another store's DELETING row untouched; a prefix that still lists an object left DELETING; purge finishing a row another deleter moved to DELETING or DELETED, or removed, between its read and its turn, and leaving a FAILED one; a young CREATING row left under a 300 s window and removed under the zero one; the wait's five outcomes (quiet on a FAILED record and a dead FAILED row, busy naming a live CREATING and FAILED row and a RUNNING restore), a signal ending its poll sleep at once; a DELETING row finished while the claim is held; the purge's own cycle, run with a zero window by the claim's holder, taking the held branch without reading the claim, where a default state reads the same claim through the window as free; the wait reading a live owner on the catalog's clock, with that clock 400 s behind this host's (a heartbeat this host reads as past the window is live) and 400 s ahead (one stamped with this host's now is dead)). The engine's MemStore double implements BackupStore in memory with a failure switch, a stall switch and a post-conditional-put overwrite hook; its catalog double implements the purge's row writes.
• extenddb-server: 255 (10 new: the spool lock refusing a second instance, narrowing a permissive lock file, and an existing spool refused unless owner-only, a permissive parent included; a defaulted root with rows for it configured and unhealthy rather than unconfigured, and a foreign identity file a warning without rows and owned by another catalog with them; a defaulted root whose open refuses a warning without rows and a startup error naming the rows with them; the startup checks returning at their 5 s bound on a stalled validate and an I/O failure of the probe degrading where a permission refusal stops startup; NotConfigured reading as the commands print it; an unreadable catalog keeping a failing defaulted root configured; a root with edge whitespace quoted in the banner; the banner showing a filesystem root in its canonical spelling, with the configured spelling beside it for a root reached through a symlink and for one not created yet under a symlinked parent).
• extenddb-app: 48 (1 new: the tokio runtime, multi-thread and current-thread, ending within its 5 s shutdown bound with a blocking thread parked forever).
• extenddb-storage-postgres (127.0.0.1:55432): lib 75; backup_restore 47 (10 new: read_store_ids; the maintenance claim excluding a second holder, released, held by a dead session's heartbeat until it ages, then taken over, kept by a live session whatever its heartbeat, reported Lost to a holder whose generation moved with its row left and still reported held through the heartbeat arm, and leaving nothing but the holder's lock in pg_locks after a check; a dead CREATING row removed and a live or moved one kept; two PostgresEngines on one catalog with two FilesystemStores: one empty root gives one id and both read it back; host-local roots under one label leave the second unhealthy, missing file first and then, with the file copied, the manifest probe; a root owned by another catalog; the purge wait quiet on a FAILED record whose target is CREATING with no duty run, busy naming a FAILED row with a live heartbeat and quiet once it has aged; the factory with beside_server leaving a gsi_pending row for the server and applying it as a server; list_backup_rows_older_than on the heartbeat and completed clocks, a row crossing the window by waiting on the database's clock; a maintenance take refused with no holder to name while a check's shared lock is held across its tries, and taking the lock on a later try once the shared lock ends); migrations_serving 6 (1 new: a command's registration named after it, the migration's wait naming it, the name kept when the registration is taken again, and a refusal while a migration runs not advising to start a server); table_snapshot 81 and the other 19 binaries green.
• extenddb-storage-sqlite: 247, 1 ignored (6 new: read_store_ids, the maintenance claim with the stale takeover, a dead CREATING row removed, a server start clearing a maintenance claim a stopped command left, list_backup_rows_older_than on both clocks, a command under the serve lock taking over a claim whose heartbeat is young, the holder it took it from finding its claim lost).
• extenddb-storage-mongodb (replica set at 127.0.0.1:27117): lib 152 (1 new: the older-than filter naming one clock column and requiring it set), interpreter_selftests 18, pushdown_parity 20; -- --ignored --test-threads=1 live_ 50 (5 new: read_store_ids; the maintenance claim with the stale takeover and a heartbeat-less record of an older release treated as stale; a dead CREATING row removed; list_backup_rows_older_than on both clocks; an open beside a server starting no key repair of a document stored without its key, where a server's open then repairs it).
• extenddb-storage-cassandra (127.0.0.1:9042, one binary at a time): lib 78; direct_integration 196 with --features test-hooks (4 new: read_store_ids over the status partitions, the maintenance claim with the LWT takeover, a dead CREATING row removed with its status partition checked, list_backup_rows_older_than on the coordinator's clock); ttl_integration 26 passed, 1 failed, 1 ignored, test_ttl_enable_rejects_asynchronous_gsi being the pre-existing failure PR 6 recorded (the file is not touched here).
• Wire: tests/test_backup_store_config.py (new; excluded from the backend-agnostic suite, devtools/run-tests ignores it): on the SQLite build 12 cells (the default root created 0700 with the identity file and reported healthy, the banner line in the log line's spelling, CreateBackup logging the configured store; disabled with no component and the deprecation warning; an in-memory database ([storage.sqlite] path = ":memory:", sqlite-memory's default) with [backup.store.filesystem] path set healthy with its identity file and a backup created there restoring with its item, and without [backup.store] no default root, no component and CreateBackup logging the warning; a group-writable parent making the default root a warning and the same root explicit a startup error; a missing explicit root and a spool in the database directory stopping startup; an S3 endpoint that refuses connections starting the server with the component unhealthy, reason unreachable, CreateBackup still served, the restore and delete refused naming s3 and not the bucket, recovering when a forwarder to moto starts listening, creating the identity file in the bucket, and degrading again when it stops; a stalled S3 endpoint holding neither the listener nor the reconciler: /health answered with the component checking, a seeded row's restore refused as pending, then after the 30 s bound unreachable, the next cycle's check opening a new connection within seconds and shutdown not waiting; one AVAILABLE row under the default root with no manifest making it a configured store, degraded with manifest probe failed, the row's restore and delete refused naming the kind, CreateBackup served, healthy again once the manifest is back, backup list listing the row and purge --confirm refusing on the unhealthy store; a relative root in daemon mode resolving against the shell's directory; a catalog re-created after destroy warning about the old default root; backup purge --confirm with stale_job_after_secs = 45 over seeded AVAILABLE, FAILED, DELETING and dead-owner CREATING rows, a FAILED and a CREATING row with heartbeats 5 s old, and a RUNNING legacy restore with a CREATING target, resolving the restore without waiting, deleting each prefix and row outright with no wait on the young rows, leaving a prefix with no row, exports/ and the identity file, logging six audit lines, releasing the claim, then the wait's limit and refusal on a FAILED row whose heartbeat is stamped 120 s ahead of the clock (under the zero window the only input still counted live): 90 s, refusing naming it and deleting nothing; a waiting purge sent SIGTERM exiting within 5 s with the claim released and nothing deleted, one SIGKILLed leaving its claim, a second purge run right after it taking the claim over and reaching its wait, and the next server start clearing it with CreateBackup accepted at once; and the server starting healthy against the purged root afterwards; backup list and the dry run before any server creating nothing, the dry run naming the missing default root and exiting 0, the dry run and list beside a running server, backup purge --confirm on an empty store and refused with exit 6 while a server holds the SQLite serve lock, --config after the subcommand, and destroy leaving the store); on the PostgreSQL build 4 cells (no store: no component and the warning; an S3 store without spool_path stopping startup; a filesystem store healthy with the identity file naming postgres, and an S3 Express bucket name stopping startup naming the bucket; backup list, the dry run and purge --confirm exiting 6 while a session holds the serving key as a migration does, naming that session without advising to start the server, then list and purge --confirm exiting 3 with the migrate --yes advice on a catalog migrate --forget 5 put behind the binary, and list exiting 0 once migrate --yes has run; with the command's refusal mapping removed it fails at the first exit code, 1 for 6). On the tree before its rebase onto 7f79f87e (not re-run since), tests/test_backup_restore_fidelity.py and tests/test_table_operations.py on three served legs: SQLite with the default store (63 passed, 1 skipped: the DELETING-table test needs EXTENDDB_TEST_BACKEND and a host SQLite newer than this one's), PostgreSQL without a store and PostgreSQL with an S3 store on moto (64 passed each with EXTENDDB_TEST_BACKEND=postgres EXTENDDB_CONFIG=<toml>); every server drained its workers at shutdown; the deprecation warning appears on every CreateBackup without a store and never with one; the S3 leg's store is checking at the listener line and passed its check 140 ms later.
• The CI jobs, run their way on this head: the SQLite integration job (devtools/check-sqlite-catalog-lock, all checks passed; devtools/check-sqlite-older-binary against extenddb/extenddb-dev:0.1.14, all checks passed; the schema check, no difference; devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" against a server initialised in the checkout's root, 1581 passed, 21 skipped, 1 xfailed, and comprehensive 340 passed); the PostgreSQL job's devtools/check-postgres-catalog-lock, all checks passed (its pass through a connection pooler skipped: no POOLER_URL); MongoDB's mongodb-rust (devtools/run-mongodb-tests --rust-live -- --rust-integration with the mongodb-test-hooks release build): tests/rust 512, the records' live tests 6, data_engine::live_tests:: 18; Cassandra's storage job as above.

cargo fmt --all -- --check
cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings
cargo +1.97.0 clippy --locked -p extenddb-storage-cassandra --all-targets --features test-hooks -- -D warnings
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_S3_REQUIRED=1 ... cargo test --locked --workspace --exclude extenddb-storage-cassandra --exclude extenddb-storage-mongodb
EXTENDDB_TEST_MONGODB_REQUIRED=1 cargo test --locked -p extenddb-storage-mongodb; ... --lib -- --ignored live_
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --lib | --features test-hooks --test direct_integration | --test ttl_integration -- --test-threads=1
devtools/generate-software-license-notices --check [--backend mongodb|cassandra]; devtools/generate-dev-license-notices --check

Not changed here

• The store-backed CreateBackup job and worker (PR 8): with a store configured, CreateBackup writes the in-catalog format and says so at info; the catalog has no CREATING or AVAILABLE rows of a store until then, so the manifest probe passes on an empty root and purge finds nothing to delete in a fresh deployment. The jobs' claim check after the row exists is PR 8's store-backed job; the in-catalog paths check before their row.
• The manifest restore (PR 9a): the restore check of a configured store's AVAILABLE row still answers "this release restores in-catalog backups only", after the health check.
• backup sync and its options, --adopt-store included (PR 10): a root whose file names another catalog stays unhealthy until an operator changes the file.
• sqlite-memory's startup sync (PR 10).
• The audit log target extenddb::audit is new with backup purge; CreateBackup, DeleteBackup and RestoreTableFromBackup are not yet logged under it.
• The reconciler's other store duties (resolving a dead CREATING row, deleting a FAILED row's prefix, tombstone retention) are PR 8's.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace), the pre-existing Cassandra TTL failure above excepted
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy --workspace --all-targets -- -D warnings)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, sections 4(c), 4(f), 7, 8, 9 and the PR 7 section, revision item 28. Storage trait: BackupCatalog gains read_store_ids, maintenance_claim, remove_dead_creating_backup_row and list_backup_rows_older_than (with StaleClock), check_maintenance_claim takes the stale window, MaintenanceClaim is new; ServerComponentsOptions gains beside_server and command; StorageConfig gains database_file with a default. BackupStore gains put_if_absent.

Breaking changes

• /health answers {"status": ..., "components": {...}} instead of {"status": "healthy"}; status is degraded while a component is unhealthy, with HTTP 200 kept; reason is a category; an S3 store's component is checking right after startup. A probe that parsed the body for "healthy" alone now sees degraded during a store outage.
• A configuration with [backup.store] is validated at startup: a bad root, bucket, part size, prefix, endpoint or spool placement, an S3 store without spool_path on PostgreSQL, MongoDB or Cassandra, and a spool another instance holds, stop the server with the path or bucket named. A relative filesystem.path or spool_path now resolves against the working directory at load; an existing spool with a mode other than 0700, or under a parent others can write to, stops startup.
• On file-backed SQLite a server without [backup.store] now creates backups/ beside the database file (0700) and extenddb-store.json in it; a root that fails the owner-only rule or whose identity file names another catalog is a logged warning, not an error.
• With a configured store that is unhealthy or not yet checked, the restore and delete of a store-backed backup answer ValidationException naming the store's kind; CreateBackup is not refused.
• backup purge holding the maintenance claim makes CreateBackup and RestoreTableFromBackup answer LimitExceededException on every instance.
• Out-of-tree storage backends must implement read_store_ids, maintenance_claim, remove_dead_creating_backup_row, list_backup_rows_older_than and the new check_maintenance_claim signature, and honour ServerComponentsOptions::beside_server (and may name their sessions after ServerComponentsOptions::command) (on PostgreSQL it also decides whether the GSI queue workers start, a note for backends with their own); out-of-tree BackupStore implementations must implement put_if_absent.
• New CLI surface: extenddb backup list [--failed] and extenddb backup purge [--confirm]. On PostgreSQL they hold a serving registration for their run, so migrate --yes waits for them, and they exit 6 while a migration runs and 3 on a catalog behind the binary, as serve does. On SQLite purge --confirm beside a running server exits 6 (it exited 1).

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.