From ff392e0da626a778dcb7cc184fadc0dcf8daf388 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Thu, 16 Jul 2026 02:12:26 -0500 Subject: [PATCH 1/7] Document RocksDB backup & restore operations Covers the new backup/restore Operations API operations (create_backup, list_backups, verify_backup, delete_backup, purge_backups, restore_backup) and RocksDB support for get_backup (gzipped-by-default tar; gzip=false for plain tar; remote target support). Documents when a database can be restored online vs offline (system db and component-held dbs require the server stopped), the backupPath storage option, and the 5.2 release note. Companion to HarperFast/harper#1831. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/cli/commands.md | 67 +++++++++++++++ reference/configuration/options.md | 1 + reference/operations-api/operations.md | 108 ++++++++++++++++++++++++- release-notes/v5-lincoln/5.2.md | 6 ++ 4 files changed, 180 insertions(+), 2 deletions(-) diff --git a/reference/cli/commands.md b/reference/cli/commands.md index 24cfbc57..3032a89c 100644 --- a/reference/cli/commands.md +++ b/reference/cli/commands.md @@ -294,6 +294,73 @@ Database files are stored in the `hdb/database` directory. As long as the snapsh - Cloud provider volume snapshots (AWS EBS, Azure Disk, GCP Persistent Disk) - Enterprise backup solutions with snapshot capabilities +## Backup Commands + + + +These are the RocksDB backup [operations](../operations-api/operations.md#backup--restore) invoked from the CLI. Each uses its **operation name** — there is no hyphenated form: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, `restore_backup`, and `get_backup`. Managed backups are a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `/backup`), one subdirectory per database. + +Each command works whether or not Harper is running. When Harper is running, the operation is forwarded to the server; when the local server is **stopped**, the command operates directly on the database/backup files. RocksDB is single-writer, so the process holding the database open must do the work: an in-place `restore_backup` of the `system` database, or of any database a loaded component keeps open, must therefore be run with the server stopped. `get_backup` is the exception — it streams from a **running** server and has no offline form. + +All commands accept `database=` (defaults to `data`). Like any CLI operation, they also accept a remote `target=` to run against another instance instead of the local one (see [Remote Operations](#remote-operations)); with a remote target the operation is always forwarded (there is no offline path). + +### `harper create_backup` + +Create an incremental directory backup of a database. + +```bash +harper create_backup database=data +``` + +### `harper list_backups` + +List the managed backups for a database (id, timestamp, size, file count). + +```bash +harper list_backups database=data +``` + +### `harper verify_backup` + +Verify a managed backup's integrity. Pass `verify_checksum=true` to also verify checksums (slower). + +```bash +harper verify_backup database=data backup_id=1 verify_checksum=true +``` + +### `harper delete_backup` + +Delete a single managed backup. + +```bash +harper delete_backup database=data backup_id=1 +``` + +### `harper purge_backups` + +Delete all but the newest `keep_count` managed backups. + +```bash +harper purge_backups database=data keep_count=3 +``` + +### `harper get_backup` + +Download a full snapshot of a database to a local file, from the local instance or a remote `target=` (the server must be **running** — there is no offline form). For RocksDB this is a `tar` archive, **gzipped by default** (`data.tar.gz`); pass `gzip=false` for a plain `tar`. For LMDB it is the `.mdb` file. Pass `out=` to choose the output file (defaults to a name derived from the response). + +```bash +harper get_backup database=data out=./data.tar.gz +harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz +``` + +### `harper restore_backup` + +Restore a database from a managed backup. Pass `backup_id=` to choose a backup (defaults to the latest), or `target_database=` to restore into a new database rather than in place. Run it with the server **stopped** to restore the `system` database, any database a loaded component keeps open, or into a `target_database`; while Harper is running, those cases are rejected with a pointer to stop the server. + +```bash +harper restore_backup database=data backup_id=1 +``` + ## Remote Operations The CLI supports executing commands on remote Harper instances. For details, see [CLI Overview - Remote Operations](./overview.md#remote-operations). diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 482c7d9d..8b05ed59 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -257,6 +257,7 @@ storage: - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `path` — Database files directory; _Default_: `/database` +- `backupPath` — Directory for managed database backups (created by [`create_backup`](../operations-api/operations.md#backup--restore)), one subdirectory per database; _Default_: `/backup` (Added in: v5.2.0) - `blobPaths` — Blob storage directory or directories; _Default_: `/blobs` (Added in: v4.5.0) - `pageSize` — Database page size (bytes); _Default_: OS default - `reclamation.threshold` — Free-space ratio below which reclamation begins evicting from caching tables; _Default_: `0.4` (Added in: v4.5.0) diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index e958e881..bbacf078 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -44,7 +44,6 @@ Detailed documentation: [Database Overview](../database/overview.md) | `drop_table` | Drops a table and all its records | super_user | | `create_attribute` | Adds a new attribute to a table | super_user | | `drop_attribute` | Removes an attribute and all its values from a table | super_user | -| `get_backup` | Returns a binary snapshot of a database for backup purposes | super_user | ### `describe_all` @@ -133,9 +132,114 @@ Drops an attribute and all its values from the specified table. } ``` +--- + +## Backup & Restore + +Operations for backing up and restoring databases. A backup is always a whole-database copy (all tables plus the audit/transaction log); there is no per-table granularity. + +Two complementary mechanisms are exposed: + +- **Managed backups** — a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `/backup`), one subdirectory per database. Create, list, verify, delete, purge, and restore are all available. These operations require the RocksDB storage engine. +- **Download** — `get_backup` streams a full snapshot of the database in the HTTP response with no server-side artifact, for pulling a copy off-host. + +Every managed backup and every RocksDB `get_backup` includes the audit/transaction log, so a restored database keeps its `read_audit_log` history as of the backup point. A restore is a point-in-time rollback; in a replicated cluster, coordinate a restore with replication before bringing the node back. + +| Operation | Description | Role Required | +| ---------------- | ------------------------------------------------------------------- | ------------- | +| `create_backup` | Creates a managed, incremental directory backup of a database (job) | super_user | +| `list_backups` | Lists the managed backups for a database | super_user | +| `verify_backup` | Verifies a managed backup's integrity (job) | super_user | +| `delete_backup` | Deletes a single managed backup | super_user | +| `purge_backups` | Deletes all but the newest N managed backups | super_user | +| `restore_backup` | Restores a database from a managed backup (job) | super_user | +| `get_backup` | Streams a full snapshot of a database in the response for download | super_user | + +### `create_backup` + + + +Creates an incremental directory backup of the database (RocksDB only) under the configured backup root. Runs as a background [job](#jobs): the operation returns a `job_id` immediately, and [`get_job`](#get_job) reports the outcome including the new `backupId`, `size`, and `timestamp`. `database` defaults to `data`. + +Directory backups in the same location share unchanged files, so the second and subsequent backups of a database only copy what changed. Use `purge_backups` to manage retention. + +```json +{ "operation": "create_backup", "database": "dev" } +``` + +### `list_backups` + + + +Returns the managed backups for a database (RocksDB only), each with its `backupId`, `timestamp`, `size`, and file count. Returns an empty array if no backups have been created yet. + +```json +{ "operation": "list_backups", "database": "dev" } +``` + +### `verify_backup` + + + +Verifies a managed backup's file sizes, and optionally their checksums when `verify_checksum` is `true` (slower). Runs as a background [job](#jobs). `backup_id` is required. + +```json +{ "operation": "verify_backup", "database": "dev", "backup_id": 1, "verify_checksum": true } +``` + +### `delete_backup` + + + +Deletes a single managed backup. Files shared with other backups are reference-counted and removed only when no remaining backup references them. `backup_id` is required. + +```json +{ "operation": "delete_backup", "database": "dev", "backup_id": 1 } +``` + +### `purge_backups` + + + +Deletes all but the newest `keep_count` managed backups, returning the number `deleted` and the number `remaining`. + +```json +{ "operation": "purge_backups", "database": "dev", "keep_count": 3 } +``` + +### `restore_backup` + + + +Restores a database in place from a managed backup (RocksDB only), without stopping Harper. Runs as a background [job](#jobs): Harper closes the database across all worker threads, restores it, and reloads it. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data. + +```json +{ "operation": "restore_backup", "database": "dev", "backup_id": 1 } +``` + +#### When can a database be restored? + +An in-place restore purges and rewrites the database's files, which requires the database to be **fully closed** first. Harper closes it across its own worker threads, but it cannot close a handle held by a loaded component, nor stop the `system` database it depends on to run. So whether a restore can run online depends on what is holding the database open: + +| Database | Online `restore_backup` (server running) | Offline `harper restore_backup` (server stopped) | +| -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------ | +| A user database not opened by any loaded component | Yes — restored in place | Yes | +| A user database that a loaded component keeps open | No — the job fails with a `409` | Yes | +| The `system` database | No — rejected up front | Yes | + +- **Online** (this operation) works only when no loaded component is holding the database open. If a component is using it, the job fails fast with a `409` telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.) +- **Offline** — the same [`restore_backup`](../cli/commands.md#harper-restore_backup) command run from the CLI with the server **stopped** — works for **any** database, because no components are loaded and nothing holds it open. This is the required path for the `system` database and for any database a component keeps open. + +Online restore always restores in place. To restore into a new database instead of overwriting the source, run `restore_backup` with `target_database` from the CLI with the server stopped. + +A restore is a point-in-time rollback; in a replicated cluster, coordinate it with replication before bringing the node back (see the note at the top of this section). + ### `get_backup` -Returns a binary snapshot of the specified database (or individual table). Safe for backup while Harper is running. Specify `"table"` for a single table or `"tables"` for a set. +Streams a full snapshot of the specified database in the HTTP response for download - there is no server-side artifact and nothing to clean up. Behavior depends on the storage engine: + +- **RocksDB** — streams a `tar` archive of the current database state (all tables plus the transaction log), **gzipped by default** (`gzip` is RocksDB-only and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`. It is always the current state; downloading a specific historical `backup_id` is not supported - to move a retained backup off-host, copy the backup directory. +- **LMDB** — streams the `.mdb` file. Specify `"table"` for a single table or `"tables"` for a set, and `"include_audit": true` to include the audit store. These options are LMDB-only. ```json { "operation": "get_backup", "database": "dev" } diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 365ddf38..dabc12b4 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -8,6 +8,12 @@ title: '5.2' All patch release notes for 5.2.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.2&expanded=true). +## Backup & Restore + +New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `/backup`), and every backup captures the audit/transaction log alongside the data. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backup & Restore Operations](/reference/v5/operations-api/operations#backup--restore) and [Backup Commands](/reference/v5/cli/commands#backup-commands). + +`get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. + ## Configuration ### Replicated `set_configuration` From f7211a60fefbc6e45419272ff3a4e0bc17fb33ba Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Thu, 16 Jul 2026 02:56:25 -0500 Subject: [PATCH 2/7] Address docs review: keep_count wording, target_database CLI example Companion to HarperFast/harper#1831. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/cli/commands.md | 2 ++ reference/operations-api/operations.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/reference/cli/commands.md b/reference/cli/commands.md index 3032a89c..c9155598 100644 --- a/reference/cli/commands.md +++ b/reference/cli/commands.md @@ -359,6 +359,8 @@ Restore a database from a managed backup. Pass `backup_id=` to choose a back ```bash harper restore_backup database=data backup_id=1 +# restore into a new database instead of overwriting the source (server stopped) +harper restore_backup database=data backup_id=1 target_database=data_restored ``` ## Remote Operations diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index bbacf078..98553e32 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -151,7 +151,7 @@ Every managed backup and every RocksDB `get_backup` includes the audit/transacti | `list_backups` | Lists the managed backups for a database | super_user | | `verify_backup` | Verifies a managed backup's integrity (job) | super_user | | `delete_backup` | Deletes a single managed backup | super_user | -| `purge_backups` | Deletes all but the newest N managed backups | super_user | +| `purge_backups` | Deletes all but the newest `keep_count` managed backups | super_user | | `restore_backup` | Restores a database from a managed backup (job) | super_user | | `get_backup` | Streams a full snapshot of a database in the response for download | super_user | From 561160bc5be4bc5d3d5d072dce8c0d4c1690a0f8 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Thu, 16 Jul 2026 10:09:32 -0500 Subject: [PATCH 3/7] Use snake_case response fields for backup operations (backup_id, file_count) Matches HarperFast/harper#1831, which now returns snake_case. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/operations-api/operations.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 98553e32..20c8ec7a 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -159,7 +159,7 @@ Every managed backup and every RocksDB `get_backup` includes the audit/transacti -Creates an incremental directory backup of the database (RocksDB only) under the configured backup root. Runs as a background [job](#jobs): the operation returns a `job_id` immediately, and [`get_job`](#get_job) reports the outcome including the new `backupId`, `size`, and `timestamp`. `database` defaults to `data`. +Creates an incremental directory backup of the database (RocksDB only) under the configured backup root. Runs as a background [job](#jobs): the operation returns a `job_id` immediately, and [`get_job`](#get_job) reports the outcome including the new `backup_id`, `size`, and `timestamp`. `database` defaults to `data`. Directory backups in the same location share unchanged files, so the second and subsequent backups of a database only copy what changed. Use `purge_backups` to manage retention. @@ -171,7 +171,7 @@ Directory backups in the same location share unchanged files, so the second and -Returns the managed backups for a database (RocksDB only), each with its `backupId`, `timestamp`, `size`, and file count. Returns an empty array if no backups have been created yet. +Returns the managed backups for a database (RocksDB only), each with its `backup_id`, `timestamp`, `size`, and `file_count`. Returns an empty array if no backups have been created yet. ```json { "operation": "list_backups", "database": "dev" } From 752cdef804ef0674b80383ba3c9002dfb7ffd6b6 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Fri, 17 Jul 2026 11:23:00 -0500 Subject: [PATCH 4/7] Restructure backup docs into a dedicated Backups reference section Backups now has its own sidebar section with an Overview (how the system works, limitations, worked examples for incremental backup/restore and get_backup snapshot download + manual restore) and an Operations page (all seven operations with Operations API and CLI examples). - Add EngineBadge component (globally registered, like VersionBadge) to tag storage-engine support: managed ops are RocksDB-only, get_backup supports RocksDB and LMDB - Remove the Backup Commands section from the CLI commands page; leave a pointer next to the volume-snapshot guidance - Trim operations-api Backup & Restore to the brief index pattern used by Logs/Certificate Management, linking to the new section - Add the backup operations to the CLI operations table - Drop the hyphenated-alias mention entirely Cross-model review (Codex) fixes: scope the whole-database claim and super_user requirement, document the single-root-store restriction, and point LMDB manual restore at storage.path rather than the default path. Co-Authored-By: Claude Fable 5 --- AGENTS.md | 1 + reference/backups/operations.md | 140 +++++++++++++++++++++++ reference/backups/overview.md | 107 +++++++++++++++++ reference/cli/commands.md | 69 +---------- reference/cli/operations-api-commands.md | 7 ++ reference/configuration/options.md | 2 +- reference/index.md | 1 + reference/operations-api/operations.md | 57 ++------- release-notes/v5-lincoln/5.2.md | 2 +- sidebarsReference.ts | 18 +++ src/components/EngineBadge.module.css | 7 ++ src/components/EngineBadge.tsx | 24 ++++ src/theme/MDXComponents.tsx | 2 + 13 files changed, 319 insertions(+), 118 deletions(-) create mode 100644 reference/backups/operations.md create mode 100644 reference/backups/overview.md create mode 100644 src/components/EngineBadge.module.css create mode 100644 src/components/EngineBadge.tsx diff --git a/AGENTS.md b/AGENTS.md index f3fb859f..49b4feb8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,6 +47,7 @@ Prefer plain ASCII characters in Markdown unless a typographic character is genu - Reference is versioned; current version is v5 (`reference/`). Archived v4 content is in `reference_versioned_docs/version-v4/`. - Do not modify `reference_versioned_docs/` for current (v5) work; edit `reference/` instead. - The `` component is globally registered — no import needed in `.md`/`.mdx` files. +- The `` component (also global) tags storage-engine support inline: `` or ``. - See the complete repository organization in `CONTRIBUTING.md` ## Versioning Content diff --git a/reference/backups/operations.md b/reference/backups/operations.md new file mode 100644 index 00000000..7ac0bfdd --- /dev/null +++ b/reference/backups/operations.md @@ -0,0 +1,140 @@ +--- +id: operations +title: Backup Operations +--- + + + +Operations for backing up and restoring databases. For how the backup system works, its limitations, and worked examples, see the [Backups Overview](./overview.md). + +All backup operations require a `super_user` role when invoked through a running server; run offline (from the CLI with the server stopped), they are governed by filesystem permissions instead. All accept a `database` parameter that defaults to `data`. The engine badge on each operation indicates which storage engines support it: the managed-backup operations require RocksDB, while `get_backup` works with both RocksDB and LMDB. + +| Operation | Description | Role Required | +| ----------------------------------- | ------------------------------------------------------------------- | ------------- | +| [`create_backup`](#create_backup) | Creates a managed, incremental directory backup of a database (job) | super_user | +| [`list_backups`](#list_backups) | Lists the managed backups for a database | super_user | +| [`verify_backup`](#verify_backup) | Verifies a managed backup's integrity (job) | super_user | +| [`delete_backup`](#delete_backup) | Deletes a single managed backup | super_user | +| [`purge_backups`](#purge_backups) | Deletes all but the newest `keep_count` managed backups | super_user | +| [`restore_backup`](#restore_backup) | Restores a database from a managed backup (job) | super_user | +| [`get_backup`](#get_backup) | Streams a full snapshot of a database in the response for download | super_user | + +## Running backup operations from the CLI + +Every operation on this page can be run from the CLI under its operation name, for example `harper create_backup database=data`. When Harper is running, the CLI forwards the operation to the server; when it is stopped, the command operates directly on the database and backup files. `get_backup` is the exception — it streams from a running server and has no offline form. + +Like any [CLI operation](../cli/operations-api-commands.md), backup commands also accept a remote `target=` to run against another instance instead of the local one (see [Remote Operations](../cli/overview.md#remote-operations)). With a remote target the operation is always forwarded — there is no offline path. + +Offline invocation matters most for restore: an in-place `restore_backup` of the `system` database, or of any database a loaded component keeps open, must be run with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) + +## `create_backup` + + + +Creates an incremental directory backup of the database under the configured backup root ([`storage.backupPath`](../configuration/options.md#storage), default `/backup`). Through a running server this runs as a background [job](../operations-api/operations.md#jobs): the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome including the new `backup_id`, `size`, and `timestamp`. + +Backups of the same database share unchanged files, so the second and subsequent backups only copy what changed. Use [`purge_backups`](#purge_backups) to manage retention. + +```json +{ "operation": "create_backup", "database": "data" } +``` + +```bash +harper create_backup database=data +``` + +## `list_backups` + + + +Returns the managed backups for a database, each with its `backup_id`, `timestamp`, `size`, and `file_count`. Returns an empty array if no backups have been created yet. + +```json +{ "operation": "list_backups", "database": "data" } +``` + +```bash +harper list_backups database=data +``` + +## `verify_backup` + + + +Verifies a managed backup's file sizes, and optionally their checksums when `verify_checksum` is `true` (slower). Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. + +```json +{ "operation": "verify_backup", "database": "data", "backup_id": 1, "verify_checksum": true } +``` + +```bash +harper verify_backup database=data backup_id=1 verify_checksum=true +``` + +## `delete_backup` + + + +Deletes a single managed backup. Files shared with other backups are reference-counted and removed only when no remaining backup references them. `backup_id` is required. + +```json +{ "operation": "delete_backup", "database": "data", "backup_id": 1 } +``` + +```bash +harper delete_backup database=data backup_id=1 +``` + +## `purge_backups` + + + +Deletes all but the newest `keep_count` managed backups, returning the number `deleted` and the number `remaining`. + +```json +{ "operation": "purge_backups", "database": "data", "keep_count": 3 } +``` + +```bash +harper purge_backups database=data keep_count=3 +``` + +## `restore_backup` + + + +Restores a database in place from a managed backup. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data. + +Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — restoring the `system` database, or a database a component keeps open, requires running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) + +From the CLI with the server stopped, `target_database=` restores into a new database instead of overwriting the source. The target must not already exist; Harper picks the new database up on the next start. + +```json +{ "operation": "restore_backup", "database": "data", "backup_id": 1 } +``` + +```bash +harper restore_backup database=data backup_id=1 +# restore into a new database instead of overwriting the source (server stopped) +harper restore_backup database=data backup_id=1 target_database=data_restored +``` + +## `get_backup` + + + +Streams a full snapshot of the specified database in the HTTP response for download — there is no server-side artifact and nothing to clean up. The server must be running; this is the one backup operation with no offline form. Behavior depends on the storage engine: + +- **RocksDB** — streams a `tar` archive of the current database state (all tables plus the transaction log), gzipped by default (`gzip` is a RocksDB-only option and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`. The snapshot is always the current state; downloading a specific historical `backup_id` is not supported — to move a retained backup off-host, copy its backup directory. +- **LMDB** — streams the `.mdb` file. Specify `"table"` for a single table or `"tables"` for a set, and `"include_audit": true` to include the audit store. These options are LMDB-only. + +```json +{ "operation": "get_backup", "database": "data" } +``` + +From the CLI, pass `out=` to choose the output file (defaults to a name derived from the response): + +```bash +harper get_backup database=data out=./data.tar.gz +harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz +``` diff --git a/reference/backups/overview.md b/reference/backups/overview.md new file mode 100644 index 00000000..a0a94e39 --- /dev/null +++ b/reference/backups/overview.md @@ -0,0 +1,107 @@ +--- +id: overview +title: Backups +--- + + + +Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables plus the audit/transaction log — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) + +Two complementary mechanisms are available: + +- **Managed backups** — incremental, checksum-verified backups kept in a server-side repository. Harper creates, lists, verifies, deletes, purges, and restores them through the [backup operations](./operations.md). +- **Snapshot download** — [`get_backup`](./operations.md#get_backup) streams a full snapshot of a database over HTTP for storage off-host, with no server-side artifact. + +Every backup operation is available through the [Operations API](../operations-api/overview.md) and as a [CLI command](./operations.md#running-backup-operations-from-the-cli) under the same operation name. + +## How managed backups work + +Managed backups use the RocksDB backup engine. Each backup is a directory under the configured backup root ([`storage.backupPath`](../configuration/options.md#storage), default `/backup`), with one subdirectory per database. Because RocksDB data files are immutable once written, backups in the same location share unchanged files: the first backup copies the whole database, and each subsequent backup only copies what changed since the last one. Shared files are reference-counted, so deleting a backup only removes files no remaining backup references. + +Every managed backup includes the database's transaction log, so a restored database keeps its `read_audit_log` history as of the backup point. + +The long-running operations (`create_backup`, `verify_backup`, `restore_backup`) run as background [jobs](../operations-api/operations.md#jobs) when invoked through a running server: the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome. + +## Online and offline + +Backup operations work whether or not Harper is running. Invoked from the CLI while the server is running, the operation is forwarded to the server; while the server is stopped, the command operates directly on the database and backup files. `get_backup` is the exception — it streams from a running server and has no offline form. + +The offline path matters for restore. RocksDB is single-writer, so an in-place restore requires the database to be fully closed first. A running Harper server can close its own worker threads' handles, but it cannot close a handle held by a loaded component, nor stop the `system` database it depends on to run — those databases can only be restored with the server stopped. See [when can a database be restored?](#when-can-a-database-be-restored) below. + +## Limitations + +- **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#how-backups-work). +- **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.) +- **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. +- **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server. +- **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its backup directory. +- **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. + +### When can a database be restored? + +An in-place restore purges and rewrites the database's files, which requires the database to be fully closed first. Whether a restore can run online depends on what is holding the database open: + +| Database | Online `restore_backup` (server running) | Offline `harper restore_backup` (server stopped) | +| -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------ | +| A user database not opened by any loaded component | Yes — restored in place | Yes | +| A user database that a loaded component keeps open | No — the job fails with a `409` | Yes | +| The `system` database | No — rejected up front | Yes | + +- **Online** works only when no loaded component is holding the database open. If a component is using it, the job fails fast with a `409` telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.) +- **Offline** — the same [`restore_backup`](./operations.md#restore_backup) command run from the CLI with the server stopped — works for any database, because no components are loaded and nothing holds it open. This is the required path for the `system` database and for any database a component keeps open. + +Online restore always restores in place. To restore into a new database instead of overwriting the source, run `restore_backup` with `target_database` from the CLI with the server stopped. + +## Example: incremental backups and restore + +Create a managed backup of the `data` database (the default): + +```bash +harper create_backup database=data +``` + +The first backup copies the entire database; each one after that only copies files that changed, so frequent backups are cheap. Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: + +```bash +harper purge_backups database=data keep_count=7 +``` + +List the backups to find the one to restore: + +```bash +harper list_backups database=data +``` + +Restore the latest backup, or pass `backup_id=` for an earlier one: + +```bash +harper restore_backup database=data +``` + +With the server running, this restores the database in place — Harper closes the database across its worker threads, restores it, and reloads it — as long as nothing is holding the database open (see [when can a database be restored?](#when-can-a-database-be-restored)). With the server stopped, the same command restores the files directly and works for any database. + +## Example: download a snapshot and restore it manually + +`get_backup` streams a full snapshot of a database from a running server — local or remote — which makes it the simplest way to keep backups off-host. For a RocksDB database the stream is a `tar` archive, gzipped by default: + +```bash +harper get_backup database=data out=./data.tar.gz +# or pull from another instance +harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz +``` + +The archive contains the database's current state — all tables plus the transaction log — and extracts into a directory that opens directly as a RocksDB database. + +To restore it, stop Harper, replace the database's directory under the storage path ([`storage.path`](../configuration/options.md#storage), default `/database`) with the extracted archive, and start Harper again: + +```bash +harper stop +mv ~/hdb/database/data ~/hdb/database/data.old +mkdir -p ~/hdb/database/data +tar -xzf data.tar.gz -C ~/hdb/database/data +harper start +``` + +Keeping the previous directory as `data.old` makes the swap easy to undo; delete it once the restored database checks out. + +For an LMDB database, `get_backup` streams the database's `.mdb` file instead; restore it by replacing the database's `.mdb` file under the same storage path (`/.mdb`) with the downloaded file while Harper is stopped. diff --git a/reference/cli/commands.md b/reference/cli/commands.md index c9155598..7a108426 100644 --- a/reference/cli/commands.md +++ b/reference/cli/commands.md @@ -294,74 +294,7 @@ Database files are stored in the `hdb/database` directory. As long as the snapsh - Cloud provider volume snapshots (AWS EBS, Azure Disk, GCP Persistent Disk) - Enterprise backup solutions with snapshot capabilities -## Backup Commands - - - -These are the RocksDB backup [operations](../operations-api/operations.md#backup--restore) invoked from the CLI. Each uses its **operation name** — there is no hyphenated form: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, `restore_backup`, and `get_backup`. Managed backups are a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `/backup`), one subdirectory per database. - -Each command works whether or not Harper is running. When Harper is running, the operation is forwarded to the server; when the local server is **stopped**, the command operates directly on the database/backup files. RocksDB is single-writer, so the process holding the database open must do the work: an in-place `restore_backup` of the `system` database, or of any database a loaded component keeps open, must therefore be run with the server stopped. `get_backup` is the exception — it streams from a **running** server and has no offline form. - -All commands accept `database=` (defaults to `data`). Like any CLI operation, they also accept a remote `target=` to run against another instance instead of the local one (see [Remote Operations](#remote-operations)); with a remote target the operation is always forwarded (there is no offline path). - -### `harper create_backup` - -Create an incremental directory backup of a database. - -```bash -harper create_backup database=data -``` - -### `harper list_backups` - -List the managed backups for a database (id, timestamp, size, file count). - -```bash -harper list_backups database=data -``` - -### `harper verify_backup` - -Verify a managed backup's integrity. Pass `verify_checksum=true` to also verify checksums (slower). - -```bash -harper verify_backup database=data backup_id=1 verify_checksum=true -``` - -### `harper delete_backup` - -Delete a single managed backup. - -```bash -harper delete_backup database=data backup_id=1 -``` - -### `harper purge_backups` - -Delete all but the newest `keep_count` managed backups. - -```bash -harper purge_backups database=data keep_count=3 -``` - -### `harper get_backup` - -Download a full snapshot of a database to a local file, from the local instance or a remote `target=` (the server must be **running** — there is no offline form). For RocksDB this is a `tar` archive, **gzipped by default** (`data.tar.gz`); pass `gzip=false` for a plain `tar`. For LMDB it is the `.mdb` file. Pass `out=` to choose the output file (defaults to a name derived from the response). - -```bash -harper get_backup database=data out=./data.tar.gz -harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz -``` - -### `harper restore_backup` - -Restore a database from a managed backup. Pass `backup_id=` to choose a backup (defaults to the latest), or `target_database=` to restore into a new database rather than in place. Run it with the server **stopped** to restore the `system` database, any database a loaded component keeps open, or into a `target_database`; while Harper is running, those cases are rejected with a pointer to stop the server. - -```bash -harper restore_backup database=data backup_id=1 -# restore into a new database instead of overwriting the source (server stopped) -harper restore_backup database=data backup_id=1 target_database=data_restored -``` +Harper also has a built-in backup system — managed incremental backups and snapshot downloads, runnable from the CLI under their operation names (e.g. `harper create_backup`). See [Backups](../backups/overview.md). ## Remote Operations diff --git a/reference/cli/operations-api-commands.md b/reference/cli/operations-api-commands.md index e893f42e..23a7f5f7 100644 --- a/reference/cli/operations-api-commands.md +++ b/reference/cli/operations-api-commands.md @@ -97,6 +97,13 @@ This is just a brief overview of all operations available as CLI commands. Revie | `get_fingerprint` | Get instance fingerprint | [Licensing](../operations-api/operations.md#registration--licensing) | v4.3.0 | | `set_license` | Set license key | [Licensing](../operations-api/operations.md#registration--licensing) | v4.3.0 | | `get_usage_licenses` | Get usage and license info | [Licensing](../operations-api/operations.md#registration--licensing) | v4.7.3 | +| `create_backup` | Create a managed database backup | [Backups](../backups/operations.md) | v5.2.0 | +| `list_backups` | List managed backups | [Backups](../backups/operations.md) | v5.2.0 | +| `verify_backup` | Verify a managed backup's integrity | [Backups](../backups/operations.md) | v5.2.0 | +| `delete_backup` | Delete a managed backup | [Backups](../backups/operations.md) | v5.2.0 | +| `purge_backups` | Delete all but the newest backups | [Backups](../backups/operations.md) | v5.2.0 | +| `restore_backup` | Restore a database from a backup | [Backups](../backups/operations.md) | v5.2.0 | +| `get_backup` | Download a full database snapshot | [Backups](../backups/operations.md) | v5.2.0 | | `get_job` | Get job status | [Jobs](../operations-api/operations.md#jobs) | v4.3.0 | | `search_jobs_by_start_date` | Search jobs by start date | [Jobs](../operations-api/operations.md#jobs) | v4.3.0 | | `read_log` | Read application logs | [Logging](../operations-api/operations.md#logs) | v4.3.0 | diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 8b05ed59..48b80fc6 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -257,7 +257,7 @@ storage: - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `path` — Database files directory; _Default_: `/database` -- `backupPath` — Directory for managed database backups (created by [`create_backup`](../operations-api/operations.md#backup--restore)), one subdirectory per database; _Default_: `/backup` (Added in: v5.2.0) +- `backupPath` — Directory for managed database backups (created by [`create_backup`](../backups/operations.md#create_backup)), one subdirectory per database; _Default_: `/backup` (Added in: v5.2.0) - `blobPaths` — Blob storage directory or directories; _Default_: `/blobs` (Added in: v4.5.0) - `pageSize` — Database page size (bytes); _Default_: OS default - `reclamation.threshold` — Free-space ratio below which reclamation begins evicting from caching tables; _Default_: `0.4` (Added in: v4.5.0) diff --git a/reference/index.md b/reference/index.md index afb446d1..aff311c9 100644 --- a/reference/index.md +++ b/reference/index.md @@ -40,6 +40,7 @@ For concept introductions, tutorials, and guides, see the [Learn](/learn) sectio | Section | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------ | +| [Backups](./backups/overview.md) | Managed incremental backups, snapshot downloads, and restore | | [Logging](./logging/overview.md) | Log configuration, the `logger` API, and log management operations | | [Analytics](./analytics/overview.md) | Resource and storage analytics, system tables | | [MQTT](./mqtt/overview.md) | MQTT broker configuration and usage | diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 20c8ec7a..7f278ba1 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -136,14 +136,9 @@ Drops an attribute and all its values from the specified table. ## Backup & Restore -Operations for backing up and restoring databases. A backup is always a whole-database copy (all tables plus the audit/transaction log); there is no per-table granularity. +Operations for backing up and restoring databases. Managed backups require the RocksDB storage engine; `get_backup` works with both RocksDB and LMDB. -Two complementary mechanisms are exposed: - -- **Managed backups** — a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `/backup`), one subdirectory per database. Create, list, verify, delete, purge, and restore are all available. These operations require the RocksDB storage engine. -- **Download** — `get_backup` streams a full snapshot of the database in the HTTP response with no server-side artifact, for pulling a copy off-host. - -Every managed backup and every RocksDB `get_backup` includes the audit/transaction log, so a restored database keeps its `read_audit_log` history as of the backup point. A restore is a point-in-time rollback; in a replicated cluster, coordinate a restore with replication before bringing the node back. +Detailed documentation: [Backup Operations](../backups/operations.md) | Operation | Description | Role Required | | ---------------- | ------------------------------------------------------------------- | ------------- | @@ -157,11 +152,7 @@ Every managed backup and every RocksDB `get_backup` includes the audit/transacti ### `create_backup` - - -Creates an incremental directory backup of the database (RocksDB only) under the configured backup root. Runs as a background [job](#jobs): the operation returns a `job_id` immediately, and [`get_job`](#get_job) reports the outcome including the new `backup_id`, `size`, and `timestamp`. `database` defaults to `data`. - -Directory backups in the same location share unchanged files, so the second and subsequent backups of a database only copy what changed. Use `purge_backups` to manage retention. +Creates an incremental directory backup of the database under the configured backup root. Runs as a background [job](#jobs) that reports the new `backup_id`. ```json { "operation": "create_backup", "database": "dev" } @@ -169,9 +160,7 @@ Directory backups in the same location share unchanged files, so the second and ### `list_backups` - - -Returns the managed backups for a database (RocksDB only), each with its `backup_id`, `timestamp`, `size`, and `file_count`. Returns an empty array if no backups have been created yet. +Returns the managed backups for a database, each with its `backup_id`, `timestamp`, `size`, and `file_count`. ```json { "operation": "list_backups", "database": "dev" } @@ -179,9 +168,7 @@ Returns the managed backups for a database (RocksDB only), each with its `backup ### `verify_backup` - - -Verifies a managed backup's file sizes, and optionally their checksums when `verify_checksum` is `true` (slower). Runs as a background [job](#jobs). `backup_id` is required. +Verifies a managed backup's integrity, including checksums when `verify_checksum` is `true` (slower). Runs as a background [job](#jobs). ```json { "operation": "verify_backup", "database": "dev", "backup_id": 1, "verify_checksum": true } @@ -189,9 +176,7 @@ Verifies a managed backup's file sizes, and optionally their checksums when `ver ### `delete_backup` - - -Deletes a single managed backup. Files shared with other backups are reference-counted and removed only when no remaining backup references them. `backup_id` is required. +Deletes a single managed backup. ```json { "operation": "delete_backup", "database": "dev", "backup_id": 1 } @@ -199,9 +184,7 @@ Deletes a single managed backup. Files shared with other backups are reference-c ### `purge_backups` - - -Deletes all but the newest `keep_count` managed backups, returning the number `deleted` and the number `remaining`. +Deletes all but the newest `keep_count` managed backups. ```json { "operation": "purge_backups", "database": "dev", "keep_count": 3 } @@ -209,37 +192,15 @@ Deletes all but the newest `keep_count` managed backups, returning the number `d ### `restore_backup` - - -Restores a database in place from a managed backup (RocksDB only), without stopping Harper. Runs as a background [job](#jobs): Harper closes the database across all worker threads, restores it, and reloads it. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data. +Restores a database in place from a managed backup, as a background [job](#jobs). `backup_id` defaults to the latest backup. Restoring the `system` database, or a database a loaded component keeps open, requires the server to be stopped — see [when can a database be restored?](../backups/overview.md#when-can-a-database-be-restored) ```json { "operation": "restore_backup", "database": "dev", "backup_id": 1 } ``` -#### When can a database be restored? - -An in-place restore purges and rewrites the database's files, which requires the database to be **fully closed** first. Harper closes it across its own worker threads, but it cannot close a handle held by a loaded component, nor stop the `system` database it depends on to run. So whether a restore can run online depends on what is holding the database open: - -| Database | Online `restore_backup` (server running) | Offline `harper restore_backup` (server stopped) | -| -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------ | -| A user database not opened by any loaded component | Yes — restored in place | Yes | -| A user database that a loaded component keeps open | No — the job fails with a `409` | Yes | -| The `system` database | No — rejected up front | Yes | - -- **Online** (this operation) works only when no loaded component is holding the database open. If a component is using it, the job fails fast with a `409` telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.) -- **Offline** — the same [`restore_backup`](../cli/commands.md#harper-restore_backup) command run from the CLI with the server **stopped** — works for **any** database, because no components are loaded and nothing holds it open. This is the required path for the `system` database and for any database a component keeps open. - -Online restore always restores in place. To restore into a new database instead of overwriting the source, run `restore_backup` with `target_database` from the CLI with the server stopped. - -A restore is a point-in-time rollback; in a replicated cluster, coordinate it with replication before bringing the node back (see the note at the top of this section). - ### `get_backup` -Streams a full snapshot of the specified database in the HTTP response for download - there is no server-side artifact and nothing to clean up. Behavior depends on the storage engine: - -- **RocksDB** — streams a `tar` archive of the current database state (all tables plus the transaction log), **gzipped by default** (`gzip` is RocksDB-only and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`. It is always the current state; downloading a specific historical `backup_id` is not supported - to move a retained backup off-host, copy the backup directory. -- **LMDB** — streams the `.mdb` file. Specify `"table"` for a single table or `"tables"` for a set, and `"include_audit": true` to include the audit store. These options are LMDB-only. +Streams a full snapshot of the specified database in the HTTP response for download. For RocksDB , a `tar` archive of the current state, gzipped by default; for LMDB, the `.mdb` file. ```json { "operation": "get_backup", "database": "dev" } diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index a52a42cd..dd89cb81 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -10,7 +10,7 @@ All patch release notes for 5.2.x are available on the [releases page](https://g ## Backup & Restore -New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `/backup`), and every backup captures the audit/transaction log alongside the data. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backup & Restore Operations](/reference/v5/operations-api/operations#backup--restore) and [Backup Commands](/reference/v5/cli/commands#backup-commands). +New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `/backup`), and every backup captures the audit/transaction log alongside the data. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backups](/reference/v5/backups/overview) and [Backup Operations](/reference/v5/backups/operations). `get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. diff --git a/sidebarsReference.ts b/sidebarsReference.ts index 63957800..c0cbba40 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -429,6 +429,24 @@ const sidebars: SidebarsConfig = { }, ], }, + { + type: 'category', + label: 'Backups', + collapsible: false, + className: 'reference-category-header', + items: [ + { + type: 'doc', + id: 'backups/overview', + label: 'Overview', + }, + { + type: 'doc', + id: 'backups/operations', + label: 'Operations', + }, + ], + }, { type: 'category', label: 'Logging', diff --git a/src/components/EngineBadge.module.css b/src/components/EngineBadge.module.css new file mode 100644 index 00000000..6f748ecf --- /dev/null +++ b/src/components/EngineBadge.module.css @@ -0,0 +1,7 @@ +.badge { + display: inline-block; + font-size: 0.8rem; + color: var(--ifm-color-content-secondary); + font-style: italic; + margin-bottom: 0.75rem; +} diff --git a/src/components/EngineBadge.tsx b/src/components/EngineBadge.tsx new file mode 100644 index 00000000..cf7e37cf --- /dev/null +++ b/src/components/EngineBadge.tsx @@ -0,0 +1,24 @@ +import React from 'react'; +import styles from './EngineBadge.module.css'; + +const CANONICAL_NAMES: Record = { + rocksdb: 'RocksDB', + lmdb: 'LMDB', +}; + +interface EngineBadgeProps { + /** Comma-separated storage engines, e.g. "RocksDB" or "RocksDB, LMDB" */ + engines: string; +} + +export default function EngineBadge({ engines }: EngineBadgeProps) { + const names = engines.split(',').map((engine) => { + const trimmed = engine.trim(); + return CANONICAL_NAMES[trimmed.toLowerCase()] ?? trimmed; + }); + return ( + + {names.length > 1 ? 'Engines' : 'Engine'}: {names.join(', ')} + + ); +} diff --git a/src/theme/MDXComponents.tsx b/src/theme/MDXComponents.tsx index 7c722ec4..ff8e3b6f 100644 --- a/src/theme/MDXComponents.tsx +++ b/src/theme/MDXComponents.tsx @@ -1,7 +1,9 @@ import MDXComponents from '@theme-original/MDXComponents'; import VersionBadge from '@site/src/components/VersionBadge'; +import EngineBadge from '@site/src/components/EngineBadge'; export default { ...MDXComponents, VersionBadge, + EngineBadge, }; From 94a3d5af3444c58f7ea466ef4de03205741e3f98 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Mon, 27 Jul 2026 18:20:36 -0500 Subject: [PATCH 5/7] docs(backups): clarify engine-specific backup paths and reorder ops - overview: split manual snapshot restore into RocksDB (tar) and LMDB (.mdb) examples; mark the RocksDB walkthrough as engine-specific - cli/commands: frame "How Backups Work" as a per-engine choice (volume snapshots for LMDB, RocksDB backup engine for RocksDB) and scope the atomic-snapshot requirement to LMDB - operations-api: move Backup & Restore section down into the operational cluster (after System, before Jobs) Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/backups/operations.md | 2 +- reference/backups/overview.md | 33 ++++-- reference/cli/commands.md | 6 +- reference/operations-api/operations.md | 148 ++++++++++++------------- 4 files changed, 104 insertions(+), 85 deletions(-) diff --git a/reference/backups/operations.md b/reference/backups/operations.md index 7ac0bfdd..5ec60cab 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -25,7 +25,7 @@ Every operation on this page can be run from the CLI under its operation name, f Like any [CLI operation](../cli/operations-api-commands.md), backup commands also accept a remote `target=` to run against another instance instead of the local one (see [Remote Operations](../cli/overview.md#remote-operations)). With a remote target the operation is always forwarded — there is no offline path. -Offline invocation matters most for restore: an in-place `restore_backup` of the `system` database, or of any database a loaded component keeps open, must be run with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) +Offline invocation matters most for restore: an in-place `restore_backup` of the `system` database, or of any database a loaded component keeps open, must be run with the server stopped. See ["when can a database be restored"](./overview.md#when-can-a-database-be-restored). ## `create_backup` diff --git a/reference/backups/overview.md b/reference/backups/overview.md index a0a94e39..5140b8a2 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -26,11 +26,11 @@ The long-running operations (`create_backup`, `verify_backup`, `restore_backup`) Backup operations work whether or not Harper is running. Invoked from the CLI while the server is running, the operation is forwarded to the server; while the server is stopped, the command operates directly on the database and backup files. `get_backup` is the exception — it streams from a running server and has no offline form. -The offline path matters for restore. RocksDB is single-writer, so an in-place restore requires the database to be fully closed first. A running Harper server can close its own worker threads' handles, but it cannot close a handle held by a loaded component, nor stop the `system` database it depends on to run — those databases can only be restored with the server stopped. See [when can a database be restored?](#when-can-a-database-be-restored) below. +The offline path matters for restore. RocksDB is single-writer, so an in-place restore requires the database to be fully closed first. A running Harper server can close its own worker threads' handles, but it cannot close a handle held by a loaded component, nor stop the `system` database it depends on to run — those databases can only be restored with the server stopped. See ["when can a database be restored"](#when-can-a-database-be-restored) below. ## Limitations -- **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#how-backups-work). +- **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#backing-up-with-volume-snapshots). - **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.) - **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. - **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server. @@ -82,7 +82,11 @@ With the server running, this restores the database in place — Harper closes t ## Example: download a snapshot and restore it manually -`get_backup` streams a full snapshot of a database from a running server — local or remote — which makes it the simplest way to keep backups off-host. For a RocksDB database the stream is a `tar` archive, gzipped by default: +`get_backup` streams a full snapshot of a database from a running server — local or remote — which makes it the simplest way to keep backups off-host. What it streams depends on the storage engine, so the manual restore differs too. The examples below use the `data` database and the default storage path ([`storage.path`](../configuration/options.md#storage), default `/database`). + +### RocksDB + +For a RocksDB database the stream is a `tar` archive, gzipped by default: ```bash harper get_backup database=data out=./data.tar.gz @@ -92,16 +96,31 @@ harper get_backup database=data target=https://node-2.example.com:9925 out=./dat The archive contains the database's current state — all tables plus the transaction log — and extracts into a directory that opens directly as a RocksDB database. -To restore it, stop Harper, replace the database's directory under the storage path ([`storage.path`](../configuration/options.md#storage), default `/database`) with the extracted archive, and start Harper again: +To restore it, stop Harper, replace the database's directory under the storage path with the extracted archive, and start Harper again: ```bash harper stop -mv ~/hdb/database/data ~/hdb/database/data.old +mv ~/hdb/database/data ~/hdb/database/data.old # optional: keep a copy of the previous database mkdir -p ~/hdb/database/data tar -xzf data.tar.gz -C ~/hdb/database/data harper start ``` -Keeping the previous directory as `data.old` makes the swap easy to undo; delete it once the restored database checks out. +### LMDB + +For an LMDB database the stream is the database's single `.mdb` file: + +```bash +harper get_backup database=data out=./data.mdb +# or pull from another instance +harper get_backup database=data target=https://node-2.example.com:9925 out=./data.mdb +``` + +To restore it, stop Harper, replace the database's `.mdb` file under the storage path (`/.mdb`) with the downloaded file, and start Harper again: -For an LMDB database, `get_backup` streams the database's `.mdb` file instead; restore it by replacing the database's `.mdb` file under the same storage path (`/.mdb`) with the downloaded file while Harper is stopped. +```bash +harper stop +mv ~/hdb/database/data.mdb ~/hdb/database/data.mdb.old # optional: keep a copy of the previous database +cp data.mdb ~/hdb/database/data.mdb +harper start +``` diff --git a/reference/cli/commands.md b/reference/cli/commands.md index 7a108426..549ef2cb 100644 --- a/reference/cli/commands.md +++ b/reference/cli/commands.md @@ -272,6 +272,8 @@ See also: [Database Compaction](../database/compaction.md) for more information. #### How Backups Work +Which backup approach to use depends on the storage engine: use **volume snapshots** for **LMDB** databases, and the [**RocksDB backup engine**](../backups/overview.md) (`harper create_backup`) for **RocksDB** databases. The RocksDB backup engine produces incremental, checksum-verified backups directly, with no need for atomic volume snapshots. The rest of this section covers the volume-snapshot approach. + Harper uses a transactional commit process that ensures data on disk is always transactionally consistent with storage. This means Harper maintains database integrity in the event of a crash and allows you to use standard volume snapshot tools to make backups. **Backup Process**: @@ -280,7 +282,7 @@ Database files are stored in the `hdb/database` directory. As long as the snapsh **Important Notes**: -- **Atomic Snapshots**: Use volume snapshot tools that create atomic snapshots +- **Atomic Snapshots**: Use volume snapshot tools for LMDB databases and the [RocksDB backup engine](../backups/overview.md) for RocksDB databases. - **Not Safe**: Simply copying an in-use database file using `cp` is **not reliable** - Progressive reads occur at different points in time - Results in an unreliable copy that likely won't be usable @@ -294,8 +296,6 @@ Database files are stored in the `hdb/database` directory. As long as the snapsh - Cloud provider volume snapshots (AWS EBS, Azure Disk, GCP Persistent Disk) - Enterprise backup solutions with snapshot capabilities -Harper also has a built-in backup system — managed incremental backups and snapshot downloads, runnable from the CLI under their operation names (e.g. `harper create_backup`). See [Backups](../backups/overview.md). - ## Remote Operations The CLI supports executing commands on remote Harper instances. For details, see [CLI Overview - Remote Operations](./overview.md#remote-operations). diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index fd116587..cb906186 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -134,80 +134,6 @@ Drops an attribute and all its values from the specified table. --- -## Backup & Restore - -Operations for backing up and restoring databases. Managed backups require the RocksDB storage engine; `get_backup` works with both RocksDB and LMDB. - -Detailed documentation: [Backup Operations](../backups/operations.md) - -| Operation | Description | Role Required | -| ---------------- | ------------------------------------------------------------------- | ------------- | -| `create_backup` | Creates a managed, incremental directory backup of a database (job) | super_user | -| `list_backups` | Lists the managed backups for a database | super_user | -| `verify_backup` | Verifies a managed backup's integrity (job) | super_user | -| `delete_backup` | Deletes a single managed backup | super_user | -| `purge_backups` | Deletes all but the newest `keep_count` managed backups | super_user | -| `restore_backup` | Restores a database from a managed backup (job) | super_user | -| `get_backup` | Streams a full snapshot of a database in the response for download | super_user | - -### `create_backup` - -Creates an incremental directory backup of the database under the configured backup root. Runs as a background [job](#jobs) that reports the new `backup_id`. - -```json -{ "operation": "create_backup", "database": "dev" } -``` - -### `list_backups` - -Returns the managed backups for a database, each with its `backup_id`, `timestamp`, `size`, and `file_count`. - -```json -{ "operation": "list_backups", "database": "dev" } -``` - -### `verify_backup` - -Verifies a managed backup's integrity, including checksums when `verify_checksum` is `true` (slower). Runs as a background [job](#jobs). - -```json -{ "operation": "verify_backup", "database": "dev", "backup_id": 1, "verify_checksum": true } -``` - -### `delete_backup` - -Deletes a single managed backup. - -```json -{ "operation": "delete_backup", "database": "dev", "backup_id": 1 } -``` - -### `purge_backups` - -Deletes all but the newest `keep_count` managed backups. - -```json -{ "operation": "purge_backups", "database": "dev", "keep_count": 3 } -``` - -### `restore_backup` - -Restores a database in place from a managed backup, as a background [job](#jobs). `backup_id` defaults to the latest backup. Restoring the `system` database, or a database a loaded component keeps open, requires the server to be stopped — see [when can a database be restored?](../backups/overview.md#when-can-a-database-be-restored) - -```json -{ "operation": "restore_backup", "database": "dev", "backup_id": 1 } -``` - -### `get_backup` - -Streams a full snapshot of the specified database in the HTTP response for download. For RocksDB , a `tar` archive of the current state, gzipped by default; for LMDB, the `.mdb` file. - -```json -{ "operation": "get_backup", "database": "dev" } -``` - ---- - ## NoSQL Operations Operations for inserting, updating, deleting, and querying records using NoSQL. @@ -1103,6 +1029,80 @@ Manage in-memory application status values. Status types: `primary`, `maintenanc --- +## Backup & Restore + +Operations for backing up and restoring databases. Managed backups require the RocksDB storage engine; `get_backup` works with both RocksDB and LMDB. + +Detailed documentation: [Backup Operations](../backups/operations.md) + +| Operation | Description | Role Required | +| ---------------- | ------------------------------------------------------------------- | ------------- | +| `create_backup` | Creates a managed, incremental directory backup of a database (job) | super_user | +| `list_backups` | Lists the managed backups for a database | super_user | +| `verify_backup` | Verifies a managed backup's integrity (job) | super_user | +| `delete_backup` | Deletes a single managed backup | super_user | +| `purge_backups` | Deletes all but the newest `keep_count` managed backups | super_user | +| `restore_backup` | Restores a database from a managed backup (job) | super_user | +| `get_backup` | Streams a full snapshot of a database in the response for download | super_user | + +### `create_backup` + +Creates an incremental directory backup of the database under the configured backup root. Runs as a background [job](#jobs) that reports the new `backup_id`. + +```json +{ "operation": "create_backup", "database": "dev" } +``` + +### `list_backups` + +Returns the managed backups for a database, each with its `backup_id`, `timestamp`, `size`, and `file_count`. + +```json +{ "operation": "list_backups", "database": "dev" } +``` + +### `verify_backup` + +Verifies a managed backup's integrity, including checksums when `verify_checksum` is `true` (slower). Runs as a background [job](#jobs). + +```json +{ "operation": "verify_backup", "database": "dev", "backup_id": 1, "verify_checksum": true } +``` + +### `delete_backup` + +Deletes a single managed backup. + +```json +{ "operation": "delete_backup", "database": "dev", "backup_id": 1 } +``` + +### `purge_backups` + +Deletes all but the newest `keep_count` managed backups. + +```json +{ "operation": "purge_backups", "database": "dev", "keep_count": 3 } +``` + +### `restore_backup` + +Restores a database in place from a managed backup, as a background [job](#jobs). `backup_id` defaults to the latest backup. Restoring the `system` database, or a database a loaded component keeps open, requires the server to be stopped — see [when can a database be restored?](../backups/overview.md#when-can-a-database-be-restored) + +```json +{ "operation": "restore_backup", "database": "dev", "backup_id": 1 } +``` + +### `get_backup` + +Streams a full snapshot of the specified database in the HTTP response for download. For RocksDB , a `tar` archive of the current state, gzipped by default; for LMDB, the `.mdb` file. + +```json +{ "operation": "get_backup", "database": "dev" } +``` + +--- + ## Jobs Operations for querying background job status. From b2d60531cc006bbc9565b4b350dc58ea0e0ed236 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Fri, 31 Jul 2026 11:54:04 -0500 Subject: [PATCH 6/7] docs(backups): align with harper#1831 (blobs, verify scope, restore semantics) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address PR #590 review feedback (kriszyp, Ethan-Arrowood), verified against harper#1831 head: - Blobs (blocker): document that get_backup's tar and managed backups include file-backed blobs; add a correct manual snapshot-restore procedure that restores blob roots separately (the naive single-extract recipe silently lost blob data); cover exclude_blobs and restore's blob purge-and-rewrite; carve blobs into the scope line and 5.2 notes - DR copy: backups share files across IDs — copy the whole per-database repository while quiesced, not a single backup dir - verify_backup: scope to RocksDB files + tx-log framing; blobs unverified - Incremental cost: tx-log and blob snapshots are copied in full each backup (not incremental) - restore_backup: online open-DB failure is job status ERROR (not HTTP 409); system/target_database rejected synchronously; target may be an empty dir; document interrupted-restore recovery (rerun restore_backup) - list_backups: define timestamp/size/file_count and note they exclude tx-log/blob snapshots - get_backup: note it can clone a database from another node via target Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/backups/operations.md | 23 +++++++++---- reference/backups/overview.md | 45 ++++++++++++++++++-------- reference/operations-api/operations.md | 2 +- release-notes/v5-lincoln/5.2.md | 4 +-- 4 files changed, 50 insertions(+), 24 deletions(-) diff --git a/reference/backups/operations.md b/reference/backups/operations.md index 5ec60cab..2e04b33d 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -33,7 +33,7 @@ Offline invocation matters most for restore: an in-place `restore_backup` of the Creates an incremental directory backup of the database under the configured backup root ([`storage.backupPath`](../configuration/options.md#storage), default `/backup`). Through a running server this runs as a background [job](../operations-api/operations.md#jobs): the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome including the new `backup_id`, `size`, and `timestamp`. -Backups of the same database share unchanged files, so the second and subsequent backups only copy what changed. Use [`purge_backups`](#purge_backups) to manage retention. +Backups of the same database share unchanged RocksDB data files, so the second and subsequent backups only copy the data that changed. The transaction-log snapshot — and the blob snapshot, if the database has file-backed blobs — is copied in full on every backup, so those portions are not incremental; with a large audit-retention window, frequent backups are not free for them. Pass `exclude_blobs: true` to skip the blob snapshot. Use [`purge_backups`](#purge_backups) to manage retention. ```json { "operation": "create_backup", "database": "data" } @@ -47,7 +47,14 @@ harper create_backup database=data -Returns the managed backups for a database, each with its `backup_id`, `timestamp`, `size`, and `file_count`. Returns an empty array if no backups have been created yet. +Returns the managed backups for a database, each with: + +- `backup_id` — the monotonic integer identifier. +- `timestamp` — creation time in seconds since the epoch. +- `size` — bytes of the backup's RocksDB file payloads. +- `file_count` — number of RocksDB files (some are shared across backups). + +`size` and `file_count` come from the RocksDB backup engine and exclude the transaction-log and blob snapshots, so they undercount the repository's true on-disk footprint. Returns an empty array if no backups have been created yet. ```json { "operation": "list_backups", "database": "data" } @@ -61,7 +68,7 @@ harper list_backups database=data -Verifies a managed backup's file sizes, and optionally their checksums when `verify_checksum` is `true` (slower). Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. +Verifies a managed backup's RocksDB file sizes — and their checksums when `verify_checksum` is `true` (slower) — together with the framing of its transaction-log snapshot (always checked). The blob snapshot is not verified. Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. ```json { "operation": "verify_backup", "database": "data", "backup_id": 1, "verify_checksum": true } @@ -103,11 +110,13 @@ harper purge_backups database=data keep_count=3 -Restores a database in place from a managed backup. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data. +Restores a database in place from a managed backup. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data, and — for a database with file-backed blobs — the blob roots are purged and rewritten from the backup's blob snapshot. + +Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — if one does, the job ends in `ERROR` (surfaced by [`get_job`](../operations-api/operations.md#get_job)) telling you to restore offline. Restoring the `system` database is rejected up front, before a job is created, as is `target_database` while the server is running. These cases require running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) -Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — restoring the `system` database, or a database a component keeps open, requires running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) +From the CLI with the server stopped, `target_database=` restores into a separate database instead of overwriting the source. The target must not already exist, or must be an empty directory; Harper picks the new database up on the next start. -From the CLI with the server stopped, `target_database=` restores into a new database instead of overwriting the source. The target must not already exist; Harper picks the new database up on the next start. +If a restore is interrupted before it completes (crash, power loss), Harper marks the database as incompletely restored and refuses to load it on the next start, logging an incomplete-restore error. Recover by rerunning `restore_backup` for the same database and `backup_id` — do not try to load or hand-repair the directory. ```json { "operation": "restore_backup", "database": "data", "backup_id": 1 } @@ -125,7 +134,7 @@ harper restore_backup database=data backup_id=1 target_database=data_restored Streams a full snapshot of the specified database in the HTTP response for download — there is no server-side artifact and nothing to clean up. The server must be running; this is the one backup operation with no offline form. Behavior depends on the storage engine: -- **RocksDB** — streams a `tar` archive of the current database state (all tables plus the transaction log), gzipped by default (`gzip` is a RocksDB-only option and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`. The snapshot is always the current state; downloading a specific historical `backup_id` is not supported — to move a retained backup off-host, copy its backup directory. +- **RocksDB** — streams a `tar` archive of the current database state — all tables, the transaction log, and any file-backed blobs — gzipped by default (`gzip` is a RocksDB-only option and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`, or `"exclude_blobs": true` to leave blobs out. The snapshot is always the current state; downloading a specific historical `backup_id` is not supported — to move a retained managed backup off-host, copy its whole per-database backup repository (see [Limitations](./overview.md#limitations)). Restoring a RocksDB snapshot by hand takes an extra step when the database has blobs — see [download a snapshot and restore it manually](./overview.md#example-download-a-snapshot-and-restore-it-manually). - **LMDB** — streams the `.mdb` file. Specify `"table"` for a single table or `"tables"` for a set, and `"include_audit": true` to include the audit store. These options are LMDB-only. ```json diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 5140b8a2..fc157b7b 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -5,11 +5,11 @@ title: Backups -Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables plus the audit/transaction log — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) +Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) Two complementary mechanisms are available: -- **Managed backups** — incremental, checksum-verified backups kept in a server-side repository. Harper creates, lists, verifies, deletes, purges, and restores them through the [backup operations](./operations.md). +- **Managed backups** — incremental, verifiable backups kept in a server-side repository. Harper creates, lists, verifies, deletes, purges, and restores them through the [backup operations](./operations.md). - **Snapshot download** — [`get_backup`](./operations.md#get_backup) streams a full snapshot of a database over HTTP for storage off-host, with no server-side artifact. Every backup operation is available through the [Operations API](../operations-api/overview.md) and as a [CLI command](./operations.md#running-backup-operations-from-the-cli) under the same operation name. @@ -18,7 +18,9 @@ Every backup operation is available through the [Operations API](../operations-a Managed backups use the RocksDB backup engine. Each backup is a directory under the configured backup root ([`storage.backupPath`](../configuration/options.md#storage), default `/backup`), with one subdirectory per database. Because RocksDB data files are immutable once written, backups in the same location share unchanged files: the first backup copies the whole database, and each subsequent backup only copies what changed since the last one. Shared files are reference-counted, so deleting a backup only removes files no remaining backup references. -Every managed backup includes the database's transaction log, so a restored database keeps its `read_audit_log` history as of the backup point. +Every managed backup includes the database's transaction log — so a restored database keeps its `read_audit_log` history as of the backup point — and, for a database with file-backed blobs, a snapshot of those blobs (skip it with `exclude_blobs`). + +Only the RocksDB data files are shared and incremental. The transaction log and the blob snapshot are each copied in full on every backup, so with a large audit-retention window or many blobs, frequent backups cost more than the data-only view suggests. The long-running operations (`create_backup`, `verify_backup`, `restore_backup`) run as background [jobs](../operations-api/operations.md#jobs) when invoked through a running server: the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome. @@ -33,9 +35,10 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r - **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#backing-up-with-volume-snapshots). - **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.) - **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. -- **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server. -- **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its backup directory. +- **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. +- **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. +- **An interrupted restore leaves the database unloadable.** If `restore_backup` is interrupted before completing (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory. ### When can a database be restored? @@ -44,10 +47,10 @@ An in-place restore purges and rewrites the database's files, which requires the | Database | Online `restore_backup` (server running) | Offline `harper restore_backup` (server stopped) | | -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------ | | A user database not opened by any loaded component | Yes — restored in place | Yes | -| A user database that a loaded component keeps open | No — the job fails with a `409` | Yes | +| A user database that a loaded component keeps open | No — the job ends in `ERROR` | Yes | | The `system` database | No — rejected up front | Yes | -- **Online** works only when no loaded component is holding the database open. If a component is using it, the job fails fast with a `409` telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.) +- **Online** works only when no loaded component is holding the database open. Because `restore_backup` runs as a background job, the request returns a `job_id` first; if a component is holding the database open, the check fails inside the job, so the job ends in `ERROR` (surfaced by [`get_job`](../operations-api/operations.md#get_job)) with a message telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.) The `system` database is rejected up front, before a job is created. - **Offline** — the same [`restore_backup`](./operations.md#restore_backup) command run from the CLI with the server stopped — works for any database, because no components are loaded and nothing holds it open. This is the required path for the `system` database and for any database a component keeps open. Online restore always restores in place. To restore into a new database instead of overwriting the source, run `restore_backup` with `target_database` from the CLI with the server stopped. @@ -60,7 +63,7 @@ Create a managed backup of the `data` database (the default): harper create_backup database=data ``` -The first backup copies the entire database; each one after that only copies files that changed, so frequent backups are cheap. Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: +The first backup copies the entire database; each one after that only copies the RocksDB data files that changed, so re-backing-up the data is cheap. (The transaction-log snapshot, and the blob snapshot for databases with file-backed blobs, are copied in full each time — see [how managed backups work](#how-managed-backups-work).) Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: ```bash harper purge_backups database=data keep_count=7 @@ -82,21 +85,19 @@ With the server running, this restores the database in place — Harper closes t ## Example: download a snapshot and restore it manually -`get_backup` streams a full snapshot of a database from a running server — local or remote — which makes it the simplest way to keep backups off-host. What it streams depends on the storage engine, so the manual restore differs too. The examples below use the `data` database and the default storage path ([`storage.path`](../configuration/options.md#storage), default `/database`). +`get_backup` streams a full snapshot of a database from a running server — local or remote — which makes it the simplest way to keep backups off-host. Because it can stream from a remote `target`, it also doubles as a way to **clone** a database from another node onto this one: pull the snapshot with `target=`, then restore it locally as shown below. What it streams depends on the storage engine, so the manual restore differs too. The examples below use the `data` database and the default storage path ([`storage.path`](../configuration/options.md#storage), default `/database`). ### RocksDB -For a RocksDB database the stream is a `tar` archive, gzipped by default: +For a RocksDB database the stream is a `tar` archive, gzipped by default. It contains the database's current state — all tables, the transaction log, and, for a database with file-backed blobs, a `blobs//…` tree for each blob root (pass `exclude_blobs=true` to omit them): ```bash harper get_backup database=data out=./data.tar.gz -# or pull from another instance +# or pull from another node — this clones that node's database onto the current one harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz ``` -The archive contains the database's current state — all tables plus the transaction log — and extracts into a directory that opens directly as a RocksDB database. - -To restore it, stop Harper, replace the database's directory under the storage path with the extracted archive, and start Harper again: +**If the database has no file-backed blobs,** the archive extracts into a directory that opens directly as a RocksDB database. Stop Harper, replace the database's directory under the storage path with the extracted archive, and start Harper again: ```bash harper stop @@ -106,6 +107,22 @@ tar -xzf data.tar.gz -C ~/hdb/database/data harper start ``` +**If the database has file-backed blobs,** the RocksDB files and the `blobs/` tree restore to different locations: the RocksDB files into the database directory, and each `blobs//…` tree into its blob root (by default `/blobs/`, or the paths configured in [`storage.blobPaths`](../configuration/options.md#storage); the archive's `blobs/README.md` lists the index-to-path mapping). Extracting the whole archive into the database directory would bury the blobs where the engine can't find them, silently losing blob data. Restore the two parts separately: + +```bash +harper stop +# 1. RocksDB files → database directory (exclude the blobs tree) +mv ~/hdb/database/data ~/hdb/database/data.old # optional: keep a copy of the previous database +mkdir -p ~/hdb/database/data +tar -xzf data.tar.gz -C ~/hdb/database/data --exclude='blobs' --exclude='blobs/*' +# 2. each blobs// → its blob root (index 0 shown; repeat per root) +mkdir -p ~/hdb/blobs/data +tar -xzf data.tar.gz -C ~/hdb/blobs/data --strip-components=2 blobs/0 +harper start +``` + +The target node must have at least as many blob roots configured as the snapshot captured — blob references store their root index, so a missing root leaves those blobs unresolved. For anything beyond the simplest single-root case, prefer [managed backups](#example-incremental-backups-and-restore): `restore_backup` restores the RocksDB files and every blob root automatically. + ### LMDB For an LMDB database the stream is the database's single `.mdb` file: diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index cb906186..e99674af 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1095,7 +1095,7 @@ Restores a database in place from a managed backup, as a background [job](#jobs) ### `get_backup` -Streams a full snapshot of the specified database in the HTTP response for download. For RocksDB , a `tar` archive of the current state, gzipped by default; for LMDB, the `.mdb` file. +Streams a full snapshot of the specified database in the HTTP response for download. For RocksDB , a `tar` archive of the current state (including file-backed blobs unless `exclude_blobs` is set), gzipped by default; for LMDB, the `.mdb` file. ```json { "operation": "get_backup", "database": "dev" } diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 3e52aeca..32606d31 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -10,9 +10,9 @@ All patch release notes for 5.2.x are available on the [releases page](https://g ## Backup & Restore -New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `/backup`), and every backup captures the audit/transaction log alongside the data. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backups](/reference/v5/backups/overview) and [Backup Operations](/reference/v5/backups/operations). +New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side repository under the configured backup root (`storage.backupPath`, default `/backup`); the RocksDB data files are shared and incremental across backups, while the audit/transaction log — and, for databases with file-backed blobs, the blobs (skip with `exclude_blobs`) — are captured in full on every backup. `verify_backup` checks the RocksDB files (checksums with `verify_checksum`) and the transaction-log snapshot; it does not verify blobs. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backups](/reference/v5/backups/overview) and [Backup Operations](/reference/v5/backups/operations). -`get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. +`get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — including any file-backed blobs (pass `exclude_blobs: true` to omit them), gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. ## Querying From 7ddbe0e8676109bd3acbcc4b8512f383019d9321 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Mon, 3 Aug 2026 09:40:25 -0500 Subject: [PATCH 7/7] docs(backups): use plain ASCII in blob path notation and shell comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address kriszyp review nit on #590: replace non-ASCII ellipsis (…) with "..." in blobs// path notation and the "→" arrows with "->" in the manual-restore shell comments, per repo content style. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/backups/overview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/backups/overview.md b/reference/backups/overview.md index fc157b7b..e9fdf9ff 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -89,7 +89,7 @@ With the server running, this restores the database in place — Harper closes t ### RocksDB -For a RocksDB database the stream is a `tar` archive, gzipped by default. It contains the database's current state — all tables, the transaction log, and, for a database with file-backed blobs, a `blobs//…` tree for each blob root (pass `exclude_blobs=true` to omit them): +For a RocksDB database the stream is a `tar` archive, gzipped by default. It contains the database's current state — all tables, the transaction log, and, for a database with file-backed blobs, a `blobs//...` tree for each blob root (pass `exclude_blobs=true` to omit them): ```bash harper get_backup database=data out=./data.tar.gz @@ -107,15 +107,15 @@ tar -xzf data.tar.gz -C ~/hdb/database/data harper start ``` -**If the database has file-backed blobs,** the RocksDB files and the `blobs/` tree restore to different locations: the RocksDB files into the database directory, and each `blobs//…` tree into its blob root (by default `/blobs/`, or the paths configured in [`storage.blobPaths`](../configuration/options.md#storage); the archive's `blobs/README.md` lists the index-to-path mapping). Extracting the whole archive into the database directory would bury the blobs where the engine can't find them, silently losing blob data. Restore the two parts separately: +**If the database has file-backed blobs,** the RocksDB files and the `blobs/` tree restore to different locations: the RocksDB files into the database directory, and each `blobs//...` tree into its blob root (by default `/blobs/`, or the paths configured in [`storage.blobPaths`](../configuration/options.md#storage); the archive's `blobs/README.md` lists the index-to-path mapping). Extracting the whole archive into the database directory would bury the blobs where the engine can't find them, silently losing blob data. Restore the two parts separately: ```bash harper stop -# 1. RocksDB files → database directory (exclude the blobs tree) +# 1. RocksDB files -> database directory (exclude the blobs tree) mv ~/hdb/database/data ~/hdb/database/data.old # optional: keep a copy of the previous database mkdir -p ~/hdb/database/data tar -xzf data.tar.gz -C ~/hdb/database/data --exclude='blobs' --exclude='blobs/*' -# 2. each blobs// → its blob root (index 0 shown; repeat per root) +# 2. each blobs// -> its blob root (index 0 shown; repeat per root) mkdir -p ~/hdb/blobs/data tar -xzf data.tar.gz -C ~/hdb/blobs/data --strip-components=2 blobs/0 harper start