diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md new file mode 100644 index 000000000..80867cbef --- /dev/null +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -0,0 +1,417 @@ +--- +title: Migrate QuestDB onto the Kubernetes Operator +description: + Migrate an existing QuestDB Enterprise deployment onto the Kubernetes + Operator with a replica-first cutover. +--- + +# Migrate QuestDB onto the Kubernetes Operator + +This guide creates an operator-managed, replica-only follower of an external +QuestDB Enterprise deployment, lets it restore the source backup and consume +replication WAL, then promotes it after a controlled source drain. + +The workflow is cloud-neutral. The provider-specific bucket or container, +credentials, and pod identity are kept in an existing `QuestDBObjectStore`. + +:::warning +Before running a command, replace every `` value. An unreplaced +placeholder can be interpreted as shell redirection. +::: + +## Before you start + +This guide assumes that: + +- the QuestDB Enterprise Kubernetes Operator is installed; +- the tenant namespace exists; +- the namespace has access to the QuestDB Enterprise image, either through an + `imagePullSecret` or ambient node credentials; +- a `ReadWriteOnce` StorageClass with `fsGroup` support is available; +- a same-namespace `QuestDBObjectStore` and any referenced credential Secret + already provide access to the source object store; and +- you know the source backup and replication WAL prefixes. + +See +[Configuration](/docs/enterprise-kubernetes-operator/configuration/#object-storage) +if object-store access is not ready. The operator does not test, list, read, or +write the store. QuestDB pods perform the object-store I/O, and the consuming +cluster's conditions are the readiness signal. + +Confirm the APIs, source store, and StorageClass before continuing: + +```sh +kubectl get crd questdbclusters.questdb.io \ + questdbobjectstores.questdb.io questdbpromotions.questdb.io +kubectl get questdbobjectstore -n +kubectl get storageclass +``` + +Copy the exact QuestDB Enterprise image and `imagePullSecrets` from a working +cluster when possible. Remove the `imagePullSecrets` block from the examples +only when every destination node has ambient pull access. + +## Migration workflow + +This path keeps the external source writable while an operator-managed replica +restores its backup and consumes its replication WAL. Cutover downtime is +limited to stopping and draining the source, consuming the final WAL, and +promoting the follower. + +The operator does not connect to, configure, stop, or fence the external source. +Those steps remain your responsibility. + +### 1. Prepare the source + +The source must run a QuestDB Enterprise version compatible with the destination +image. Replication only carries changes to WAL-enabled tables. Inventory the +source before enabling replication: + +```questdb-sql title="Check source tables" +SELECT table_name, walEnabled +FROM tables() +ORDER BY table_name; +``` + +:::warning[Non-WAL tables do not replicate] +Data already present in a non-WAL table is included in the seed backup, but +later changes to that table are not replicated. Stop writes to non-WAL tables +before the seed backup and keep them stopped through cutover, or arrange to +synchronize them separately. +::: + +Choose a backup prefix and a replication WAL prefix in the **same bucket**. The +follower has one `QuestDBObjectStore` for both uses, so separate source buckets +cannot be represented by the follower specification. The roots must be distinct, +stable prefixes with a trailing `/`. No other live primary may write to the WAL +prefix. + +The following S3 example uses ambient AWS credentials from an EC2 instance +profile. Add it to the source's `server.conf`, replacing every placeholder: + +```ini title="server.conf on the external source" +backup.enabled=true +backup.object.store=s3::bucket=;root=;region=; +backup.schedule.cron=0 * * * * +backup.schedule.tz=UTC +backup.cleanup.keep.latest.n=5 + +replication.role=primary +replication.object.store=s3::bucket=;root=;region=; +replication.primary.cleaner.enabled=false +``` + +The example disables WAL cleanup on the source for the rest of the migration. +WAL objects will accumulate in object storage, but QuestDB will not delete them +before the follower consumes them. + +Separately, `backup.cleanup.keep.latest.n=5` retains the five most recent +completed backups—not five hours of backups. Before creating the follower, +confirm that at least one completed backup is still available. + +:::note Other object-store providers +The backup and replication mechanics are provider-independent. For another +provider supported by your Operator release, replace the two S3 connection +strings with that provider's +[object-store connection strings](/docs/high-availability/setup/#1-configure-object-storage). +Keep the backup and WAL roots distinct, and configure the destination +`QuestDBObjectStore` for the same underlying store. +::: + +Every `server.conf` setting can instead be supplied as an environment variable. +See [Environment variables](/docs/configuration/overview/#environment-variables) +for the naming convention. If an object-store string contains static +credentials, load it from a protected file as described in +[Secrets from files](/docs/configuration/overview/#secrets-from-files). + +The object-store and replication-role settings are not reloadable. Restart the +source with its normal service manager or container runtime, then trigger the +seed backup: + +```questdb-sql title="Take the seed backup" +BACKUP DATABASE; +``` + +`BACKUP DATABASE` starts the backup asynchronously. Poll its latest record until +it reports `backup complete`: + +```questdb-sql title="Check the seed backup" +SELECT status, progress_percent, start_ts, end_ts, backup_error +FROM backups() +ORDER BY start_ts DESC +LIMIT 1; +``` + +After the backup completes, record the source's exact backup instance name: + +```questdb-sql title="Get the source backup instance name" +SELECT backup_instance_name(); +``` + +QuestDB generates the name as three random lowercase words separated by hyphens, +for example `happy-green-turtle`. Copy the entire value exactly. Then confirm +that both prefixes contain objects: + +```bash title="Check the S3 source prefixes" +aws s3api list-objects-v2 \ + --region '' \ + --bucket '' \ + --prefix '' \ + --max-keys 5 +aws s3api list-objects-v2 \ + --region '' \ + --bucket '' \ + --prefix '' \ + --max-keys 5 +``` + +Do not create the follower unless the seed backup is complete, both commands +return objects, the source WAL cleaner remains disabled, all source selectors +match the intended store, and the source can later be stopped and restarted once +with `replication.role=primary-catchup-uploads`. Copy the backup prefix, WAL +prefix, and backup instance name exactly into the follower specification; those +source selectors are immutable. + +### 2. Create a replica-only follower + +Save the following as `follower.yaml`: + +```yaml +apiVersion: questdb.io/v1alpha1 +kind: QuestDBCluster +metadata: + name: + namespace: +spec: + image: + imagePullSecrets: + - name: + instances: 1 + storage: + storageClassName: + size: 100Gi + resources: + requests: + memory: 4Gi + limits: + memory: 4Gi + objectStoreRef: + name: + backup: + enabled: true + schedule: "0 * * * *" + timezone: UTC + retention: 5 + root: + replication: + root: + bootstrap: + follow: + sourceInstanceName: +``` + +While the cluster is following, every instance is a replica and the backup +scheduler is paused. After promotion, this cluster adopts the source prefixes +and begins taking its own backups there. + +Review all immutable source selectors, then apply the file: + +```sh +kubectl apply -f follower.yaml +``` + +### 3. Wait for the follower to serve reads + +Wait until the replica has restored its baseline and reconciliation is settled: + +```sh +kubectl wait questdbcluster/ -n \ + --for=jsonpath='{.status.phase}'=Following --timeout=30m +``` + +A healthy follower deliberately has no current primary and no RW endpoint. +Confirm both properties: + +```sh +kubectl get questdbcluster -n \ + -o jsonpath='following={.status.replication.following}{" primary="}{.status.currentPrimary}{"\n"}' +kubectl get endpointslice -n \ + -l kubernetes.io/service-name=-rw \ + -o jsonpath='{range .items[*].endpoints[*]}{.addresses}{"\n"}{end}' +``` + +The second command must print no endpoint addresses. + +### 4. Confirm replication catch-up + +Inspect the live follower position: + +```sh +kubectl get questdbcluster -n \ + -o jsonpath='{range .status.replication.replicas[*]}{.instance}{" caughtUpNow="}{.caughtUpNow}{" lagTxns="}{.lagTxns}{" suspended="}{.suspendedTables}{"\n"}{end}{range .status.conditions[?(@.type=="ReplicationHealthy")]}ReplicationHealthy={.status}{"/"}{.reason}{" "}{.message}{"\n"}{end}{.status.replication.stream}{"\n"}' +``` + +Prefer to begin cutover with `caughtUpNow=true` and `lagTxns=0`. A busy source +may briefly move away from zero. A quiet source may report `StreamNotDetermined` +because the engine omits already-caught-up tables from its poll; that is not +proof of success. In that case, reconfirm the immutable source identity and +roots, then query `-ro` and verify a recent, known source +record. + +Do not proceed with `ReplicationHealthy=False`, suspended tables, a known +backlog that is not advancing, or unverified source selectors. The planned +promotion performs a final fail-closed check after the source is drained. + +### 5. Stop writes and drain the source + +Record the UTC cutover start time: + +```sh +date -u +%FT%TZ +``` + +Keep this timestamp so you can confirm that the first completed backup happened +after cutover began. + +Then perform these steps with the external source's service manager or container +runtime: + +1. Stop all application writes to the external source. +2. Stop the source QuestDB process. +3. Configure the source to start once with + `replication.role=primary-catchup-uploads`. +4. Start the source and watch its logs. +5. Wait for the source process to exit with code `0`. A non-zero exit means the + final WAL upload did not complete; do not promote. +6. Disable automatic restarts. + +The final upload has no safe fixed timeout. Supervise it at the source until it +succeeds. + +:::danger +Do not promote while the external source may still be running as a primary. The +operator cannot fence an unmanaged process. Keep the old data available for +rollback investigation, but ensure the process and its supervisor cannot restart +it. +::: + +### 6. Promote the follower + +Create a one-shot planned promotion targeting the follower's instance serial +`1`: + +```sh +kubectl apply -f - < + namespace: +spec: + clusterRef: + name: + target: 1 + mode: Planned + catchUpTimeoutSeconds: 900 + primaryGracePeriodSeconds: 120 +EOF +``` + +The promotion waits for the source stream to remain quiet for at least 60 +seconds and for the target to consume the published WAL. It fails closed rather +than silently accepting a lossy cutover. + +Watch until the promotion completes or fails: + +```bash +PHASE="" +for _ in $(seq 1 180); do + PHASE="$(kubectl get questdbpromotion -n \ + -o jsonpath='{.status.phase}')" + printf '%s %s\n' "$(date -u +%FT%TZ)" "$PHASE" + case "$PHASE" in + Completed|Failed) break ;; + esac + sleep 10 +done +kubectl get questdbpromotion -n \ + -o jsonpath='{.status.phase}{" "}{.status.reason}{": "}{.status.message}{"\n"}{range .status.conditions[*]}{.type}{"="}{.status}{"/"}{.reason}{" "}{.message}{"\n"}{end}' +[ "$PHASE" = "Completed" ] +``` + +If it fails, leave the source stopped and read the reported reason before taking +another action. A failed promotion is terminal; correct the cause and create a +new promotion object. Do not remove the promotion finalizer. See +[If promotion stalls or fails](/docs/enterprise-kubernetes-operator/high-availability/#if-promotion-stalls-or-fails). + +### 7. Verify the new primary + +Wait for the promoted cluster to report `Running`: + +```sh +kubectl wait questdbcluster/ -n \ + --for=jsonpath='{.status.phase}'=Running --timeout=20m +kubectl get questdbcluster -n -o wide +``` + +Confirm that `-rw` now has an endpoint, then connect through +that Service and validate recent data and application writes: + +```sh +kubectl get endpointslice -n \ + -l kubernetes.io/service-name=-rw +``` + +Permanently decommission the old source so that it cannot restart and contend +for the adopted WAL stream. + +### 8. Verify the first post-cutover backup + +The new primary inherits the source's backup and WAL history but initially has +no backup under its own backup-instance name. The operator therefore keeps the +WAL cleaner disabled until the new primary completes its first backup and +establishes its own retention point. + +After that backup, the operator restores the configured WAL-cleaner setting. +Because `replication.primary.cleaner.enabled` is not reloadable, applying the +setting recreates the primary pod once. The replacement reuses the same PVC, but +expect a brief interruption to writes. With the hourly schedule in this guide, +allow one schedule interval plus the operator's roughly two-minute observation +delay. + +First, record the primary pod's current UID. The replacement keeps the same pod +name, so the UID is how you distinguish it from the original pod: + +```sh +kubectl get pod -1 -n \ + -o custom-columns='NAME:.metadata.name,UID:.metadata.uid' +``` + +Then wait for the first backup to complete and print its completion time: + +```sh +kubectl wait questdbcluster/ -n \ + --for=jsonpath='{.status.backup.lastBackup.status}'=completed --timeout=75m +kubectl get questdbcluster -n \ + -o jsonpath='{.status.backup.lastBackup.endTime}{" completed\n"}' +``` + +Confirm that the completion time is later than the cutover start time recorded +in step 5. + +After the backup completes, watch the pod until it has a new UID and is ready, +then press Control-C: + +```sh +kubectl get pod -1 -n --watch \ + -o custom-columns='NAME:.metadata.name,UID:.metadata.uid,READY:.status.containerStatuses[0].ready,PHASE:.status.phase' +``` + +After the new pod appears, repeat the writer-health check from the previous step +and require it to settle before declaring the migration complete. + +With multiple follower instances, promote the healthy instance you verified in +step 4. For source loss, see +[Emergency promotion](/docs/enterprise-kubernetes-operator/high-availability/#emergency-promotion). +For cutover problems, see +[If promotion stalls or fails](/docs/enterprise-kubernetes-operator/high-availability/#if-promotion-stalls-or-fails). diff --git a/documentation/enterprise-kubernetes-operator/high-availability.md b/documentation/enterprise-kubernetes-operator/high-availability.md index 48da9305a..325e9fd07 100644 --- a/documentation/enterprise-kubernetes-operator/high-availability.md +++ b/documentation/enterprise-kubernetes-operator/high-availability.md @@ -233,148 +233,9 @@ cutover. Contact support before considering it. ## Migrate an existing QuestDB onto the operator -A follower reduces migration downtime by restoring the source's backup and -consuming its WAL while the external source continues serving. +Use the [migration guide](/docs/enterprise-kubernetes-operator/getting-started/migrate/) +for the canonical replica-first migration procedure. -### Source prerequisites - -The source must: - -- run a compatible QuestDB Enterprise version; -- create completed backups in a known object-store backup root; -- upload replication WAL to a known root in the same store; -- retain WAL back to the seed backup; -- expose its exact `SELECT backup_instance_name();` value, matching - `^[a-z0-9]+(-[a-z0-9]+)*$`; and -- support a controlled stop and one final `primary-catchup-uploads` run. - -The operator never connects to, configures, stops, fences, or lists storage for -the source. - -### Create and verify the follower - -All source selectors are immutable. Confirm `sourceInstanceName`, backup root, -and WAL root before creating the cluster. The source instance name must be -copied exactly and match the lowercase hyphen-separated engine identity format. -Copy the working tenant cluster's `spec.image` and `spec.imagePullSecrets`: set -`` to that private image and -`` to the pull Secret in this namespace. Remove the -entire `imagePullSecrets` block only when every node has ambient pull access, -such as an authorized EKS worker-node role. - -```yaml -apiVersion: questdb.io/v1alpha1 -kind: QuestDBCluster -metadata: - name: - namespace: -spec: - image: - imagePullSecrets: - - name: - instances: 2 - storage: - storageClassName: - size: 100Gi - objectStoreRef: - name: - backup: - enabled: true - schedule: "0 * * * *" - root: - replication: - root: - bootstrap: - follow: - sourceInstanceName: -``` - -A healthy follower has `.status.replication.following=true`, no current primary, -an empty `-rw`, and reads through `-ro`. It intentionally omits -`WriteHealthy` because it has no primary; do not wait for that condition. Verify -all instances and `ReplicationHealthy`; a quiet source can make progress -indeterminate, so also confirm the immutable source identity and roots directly -against the source configuration. - -```sh -kubectl get questdbcluster -n \ - -o jsonpath='following={.status.replication.following}{" primary="}{.status.currentPrimary}{"\n"}{range .status.conditions[?(@.type=="ReplicationHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{.status.replication.stream}{"\n"}' -kubectl get endpointslice -n \ - -l kubernetes.io/service-name=-rw -``` - -### Cut over - -Immediately before stopping the source, record when cutover preparation began in -the same Bash shell you will use for the post-cutover check: - -```bash -CUTOVER_TIME_CAPTURED=false -CUTOVER_STARTED_AT="" -if CUTOVER_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" && \ - [ -n "$CUTOVER_STARTED_AT" ]; then - CUTOVER_TIME_CAPTURED=true -fi -[ "$CUTOVER_TIME_CAPTURED" = true ] && \ - printf 'Cutover preparation started at: %s\n' "$CUTOVER_STARTED_AT" -``` - -1. Stop application writes to the source, then stop the source database. -2. Restart the source once with `replication.role=primary-catchup-uploads`. -3. Wait for `CLOSE_REASON_UPLOADS_COMPLETE_SUCCESS` in the **source** logs, then - stop it again. This final upload may itself be unbounded; supervise it at the - source. A normal shutdown does not prove the tail reached object storage. -4. Create a `Planned` `QuestDBPromotion` for a healthy follower instance and use - the bounded watcher above. - -The planned follower gate requires the source stream to remain quiet for at -least 60 seconds and the target to consume the published WAL before promotion. -It still cannot prove an idle source process is stopped. If the engine reports -`SourceStillOwnsStore`, stop the source fully and retry with a new promotion -object. - -:::danger Emergency follower promotion accepts losing source WAL that was not -uploaded or not consumed. Use it only when the source cannot be drained and that -loss is explicitly accepted. ::: - -5. After a completed cutover, verify `-rw`, data, writes, and - `currentPrimary`. -6. Permanently decommission the source so it cannot restart and contend for the - adopted WAL root. -7. Wait for this cluster's own first post-cutover backup. The hourly deadline - must exceed one schedule interval plus the roughly two-minute observation - delay, so this check allows about 75 minutes. Do not accept an old - `completed` status: require a non-empty `endTime` later than the captured - cutover-start time. Both values are RFC3339 UTC timestamps, so the Bash - string comparison proves the observed backup completed after cutover - preparation began. The WAL cleaner remains off until this backup completes; - its release rolls the primary once. Verify the roll and writer readiness - afterward. - -```bash -BACKUP_VERIFIED=false -STATUS="" -END_TIME="" -for _ in $(seq 1 450); do - STATUS="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.backup.lastBackup.status}')" - END_TIME="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.backup.lastBackup.endTime}')" - if [ "${CUTOVER_TIME_CAPTURED:-false}" = true ] && \ - [ "$STATUS" = "completed" ] && [ -n "$END_TIME" ] && \ - [[ "$END_TIME" > "$CUTOVER_STARTED_AT" ]]; then - BACKUP_VERIFIED=true - break - fi - [ "$STATUS" = "failed" ] && break - sleep 10 -done -[ "$BACKUP_VERIFIED" = true ] && \ - kubectl get questdbcluster -n \ - -o jsonpath='{range .status.conditions[?(@.type=="Available")]}Available={.status}{"/"}{.reason}{"\n"}{end}{range .status.conditions[?(@.type=="Progressing")]}Progressing={.status}{"/"}{.reason}{"\n"}{end}{range .status.conditions[?(@.type=="WriteHealthy")]}WriteHealthy={.status}{"/"}{.reason}{"\n"}{end}' -``` - -After the roll, require `Available=True/PrimaryReady`, -`Progressing=False/Settled`, and `WriteHealthy=True/Healthy` at the current -generation before declaring the cluster writer-ready and the WAL cleaner -released. +If the source is lost before it can be drained, see +[Emergency promotion](#emergency-promotion). For diagnosis and recovery from a +cutover problem, see [If promotion stalls or fails](#if-promotion-stalls-or-fails). diff --git a/documentation/enterprise-kubernetes-operator/index.md b/documentation/enterprise-kubernetes-operator/index.md index 2951361d8..9f53d9920 100644 --- a/documentation/enterprise-kubernetes-operator/index.md +++ b/documentation/enterprise-kubernetes-operator/index.md @@ -37,6 +37,8 @@ static Secret. QuestDB database pods perform all object-store I/O. [AKS onboarding guide](/docs/enterprise-kubernetes-operator/getting-started/azure/). - **Shared install requirements:** see [Installation](/docs/enterprise-kubernetes-operator/installation/). +- **Migrate an existing QuestDB:** follow the cloud-neutral + [migration guide](/docs/enterprise-kubernetes-operator/getting-started/migrate/). - **PGWire TLS and network isolation:** plan them before creation with [Configuration](/docs/enterprise-kubernetes-operator/configuration/#pgwire-tls). - **Operate the operator:** use the diff --git a/documentation/sidebars.js b/documentation/sidebars.js index e0a4b8957..f0b4f587f 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -835,6 +835,11 @@ module.exports = { id: "enterprise-kubernetes-operator/getting-started/azure", label: "Azure AKS", }, + { + type: "doc", + id: "enterprise-kubernetes-operator/getting-started/migrate", + label: "Migrate existing QuestDB", + }, ], }, "enterprise-kubernetes-operator/configuration",