Off-host backup and restore (tracking) #624

Tracking issue for off-host backup and restore: one portable backup format, a filesystem or S3 store, and crash-safe asynchronous backup and restore jobs. The design is docs/design/15-backup-restore.md (#396), which lists the full PR series and the acceptance criteria for each. PR 0 is #384 (restore fidelity on all four backends), merged.

Merged:

• Backup/restore: design document #625 PR 2, the design document (docs(backup): design revision for off-host backup and restore [backup/restore 1/4] #396)
• Backup/restore: backup store crate (filesystem and S3) #626 PR 3, backup store crate with filesystem and S3 implementations (feat(backup-store): store trait with filesystem and S3 implementations [backup/restore 2/4] #393)
• Backup/restore: backup format v1 #627 PR 4, version 1 of the backup file format (feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394)
• Backup/restore: SnapshotEngine on every backend #628 PR 5, SnapshotEngine on every backend (feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395)
• Backup/restore: PITR surface, point-in-time recovery reported as unsupported #745 PR 1, the PITR surface (fix(backup): report point-in-time recovery as unsupported #335)

Open, stacked in this order (each PR's base is the one before it; #671's base is main since #666 merged, and the stack needs a rebase onto it):

• Backup/restore: catalog index, BackupCatalog and JobClaim on every backend, the legacy reader #669 PR 6, catalog index, BackupCatalog, JobClaim, the legacy reader (feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671)
• Backup/restore: [backup.store] configuration, the store identity file, backup purge #670 PR 7, [backup.store], the store identity file, backup purge (feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672)
• Backup/restore: CreateBackup job, worker and reconciler duties #699 PR 8, CreateBackup job, worker and reconciler duties (feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700)
• Backup/restore: RestoreTableFromBackup job from the store and the cross-backend matrix #739 PR 9, RestoreTableFromBackup job and the cross-backend matrix (feat(backup): RestoreTableFromBackup job and cross-backend matrix [backup/restore 9] #740; the design's PR 9a and 9b as one PR)

Open, independent of the stack:

• Backup/restore: admin guide section for backing up SQLite and PostgreSQL with their own tools #750 admin guide section for backing up with the databases' own tools (docs: back up and restore SQLite and PostgreSQL with their own tools #737; outside the design numbering)

Planned:

• Backup/restore: backup sync and its options, migrate-legacy #746 PR 10, backup sync and its options, migrate-legacy (after PR 8)
• Backup/restore: CI filters, the two open measurement cells, the backup manual #747 PR 11, CI filters, the two open measurement cells, the backup manual (after PR 9)
• Backup/restore: export and import through the store #748 PR 13, export and import through the store (Phase 2, after PR 10)
• Backup/restore: legacy reader removal #749 PR 14, legacy reader removal (one minor release after PR 6, after PR 10)

Related issues found while implementing the series: #755 (gate the jobs and the reconciler on the PostgreSQL serving registration), #667 (restore heartbeat DDL precondition races Cassandra DDL speed), #742 (Cassandra index row deleted on an unchanged re-put; fixed in #740).