From fc7faedfb4df6141121afbcebff9273295532644 Mon Sep 17 00:00:00 2001 From: Pranay Sanghvi Date: Mon, 7 Sep 2026 11:14:41 +0530 Subject: [PATCH 1/2] docs: add guide for removing leftover PVCs without Epinio Document kubectl Job-based cleanup for orphaned app-data claims (EPINIO-727) and link it from storage and app-delete docs. --- .../concepts/applications/applications.mdx | 4 +- .../operations/remove_pvcs_without_epinio.md | 246 ++++++++++++++++++ docs/reference/concepts/storage.md | 6 +- 3 files changed, 253 insertions(+), 3 deletions(-) create mode 100644 docs/how-to/operator/operations/remove_pvcs_without_epinio.md diff --git a/docs/how-to/developer/concepts/applications/applications.mdx b/docs/how-to/developer/concepts/applications/applications.mdx index 87a769d3fc..09fa73b158 100644 --- a/docs/how-to/developer/concepts/applications/applications.mdx +++ b/docs/how-to/developer/concepts/applications/applications.mdx @@ -258,7 +258,9 @@ Epinio selects an application's data volumes by the `app.kubernetes.io/name` lab the application's namespace. A volume your chart provisions without that label is not deleted, and has to be removed with `kubectl`. See [storage lifecycle](../../../../reference/concepts/storage.md#storage-lifecycle) for how -this affects capacity planning. +this affects capacity planning, and +[Removing leftover PVCs without Epinio](../../../operator/operations/remove_pvcs_without_epinio.md) +for a kubectl Job-based cleanup when the application is already gone or Epinio is unavailable. ## Additional Actions diff --git a/docs/how-to/operator/operations/remove_pvcs_without_epinio.md b/docs/how-to/operator/operations/remove_pvcs_without_epinio.md new file mode 100644 index 0000000000..5e1ea82e8b --- /dev/null +++ b/docs/how-to/operator/operations/remove_pvcs_without_epinio.md @@ -0,0 +1,246 @@ +--- +sidebar_label: Removing leftover PVCs without Epinio +sidebar_position: 26 +title: Removing leftover PVCs without Epinio +description: How to find, inspect, and delete leftover Epinio application PVCs with kubectl Jobs when you cannot (or choose not to) use Epinio +keywords: [epinio, kubernetes, pvc, storage, cleanup, statefulset, job] +doc-type: [how-to] +doc-topic: [epinio, how-to, operations, remove-pvcs] +doc-persona: [epinio-operator] +--- + +Epinio preserves **application data** PersistentVolumeClaims by default when an application is deleted. +Staging volumes (build cache and source-blob PVCs) are removed automatically on delete; data volumes from a StatefulSet `volumeClaimTemplates` chart are not, unless you pass `--delete-pvc`. + +This guide shows how to reclaim those leftover claims with `kubectl` and one-off Jobs. +It does **not** add or rely on Epinio maintenance API endpoints. + +:::tip Prefer Epinio when you can + +On Epinio **1.14.2+**, delete the application with PVC cleanup instead of this procedure: + +```bash +epinio app delete --delete-pvc +``` + +Use this document when the application is already gone, Epinio is unavailable, or the claim was never labeled for Epinio to discover. + +::: + +## Who needs to follow this guide + +| Situation | Action required | +|---|---| +| App deleted **without** `--delete-pvc`, and StatefulSet data PVCs remain | Follow all steps below. | +| App scaled down; older ordinal PVCs remain | Follow all steps below. | +| Staging cache / source-blob PVC left behind (failed delete, manual edits, older install) | [Identify](#step-1-identify-target-pvcs) and [delete](#step-4-delete-the-pvc) those claims in the Epinio namespace. | +| You still have the app and run Epinio 1.14.2+ | Prefer `epinio app delete --delete-pvc` (or the UI checkbox). | +| Platform storage (SeaweedFS, registry, `image-export-pvc`) | **Do not** use this guide. Those claims belong to the Epinio install. | + +## Overview of Epinio PVCs + +| Kind | Namespace | Typical name | Created when | Removed by `epinio app delete` | +|---|---|---|---|---| +| Build cache | Epinio install namespace (default `epinio`) | `-cache--` | Staging with `server.stagingWorkload.storage.cache.emptyDir: false` (the default) | Always | +| Source blobs (staging workspace) | Epinio install namespace | `-sourceblobs--` | Staging with `sourceBlobs.emptyDir: false` (default is `true`, so usually absent) | Always | +| Application data | Application namespace | e.g. `stateful-r--0` | App chart with `volumeClaimTemplates` (bundled `application-stateful` chart) | Only with `--delete-pvc` | +| Platform | Epinio install namespace | SeaweedFS / registry / `image-export-pvc` | Helm install | Never by app delete — leave alone | + +Default staging storage (current chart): + +```yaml +server: + stagingWorkload: + storage: + cache: + emptyDir: false + size: 1Gi + sourceBlobs: + emptyDir: true +``` + +The bundled stateful app chart mounts application data at `/mnt/state` from a `volumeClaimTemplates` entry named `stateful`. +Kubernetes names each claim `{volumeClaimTemplate}-{statefulSetName}-{ordinal}` (for one replica, typically `stateful-r--0`). + +Epinio discovers application data PVCs with label `app.kubernetes.io/name=` in the application namespace. +Claims without that label are never deleted by Epinio and must be cleaned up with the steps below. + +## Step 1: Identify target PVCs + +List claims in the application namespace: + +```bash +kubectl get pvc -n +``` + +Narrow by app label when present (same selector Epinio uses): + +```bash +kubectl get pvc -n -l app.kubernetes.io/name= +``` + +List staging claims in the Epinio namespace (replace `epinio` if you installed elsewhere): + +```bash +kubectl get pvc -n epinio | grep -E 'cache|sourceblobs' || true +``` + +Confirm nothing important still uses a claim before you delete it: + +```bash +kubectl describe pvc -n +kubectl get pods -n -o json \ + | jq -r --arg pvc '' ' + .items[] + | select([.spec.volumes[]? | select(.persistentVolumeClaim.claimName == $pvc)] | length > 0) + | .metadata.name' +``` + +:::caution + +A PVC that is still mounted stays in `Terminating` until every Pod using it is gone. +Scale the workload to zero or delete the leftover Pods before Step 4. + +::: + +## Step 2: (Optional) Inspect or back up with a one-off Job + +Mount the claim in a short-lived Job, the same pattern as the [MinIO → SeaweedFS migration](../networking/migrate_minio_to_seaweedfs.md) Jobs. +No Epinio API is involved. + +```yaml title="inspect-pvc-job.yaml" +apiVersion: batch/v1 +kind: Job +metadata: + name: epinio-pvc-inspect + namespace: # same namespace as the PVC +spec: + ttlSecondsAfterFinished: 600 + template: + spec: + restartPolicy: Never + volumes: + - name: data + persistentVolumeClaim: + claimName: # e.g. stateful-r-....-0 + containers: + - name: inspect + image: busybox:1.36 + command: ["/bin/sh", "-c"] + args: + - | + set -e + echo "=== listing /data ===" + ls -la /data + du -sh /data 2>/dev/null || true + # Optional: copy out anything you must keep, e.g. to another mounted volume + # tar -czf /tmp/backup.tgz -C /data . + volumeMounts: + - name: data + mountPath: /data +``` + +```bash +kubectl apply -f inspect-pvc-job.yaml +kubectl wait --for=condition=complete job/epinio-pvc-inspect -n --timeout=120s +kubectl logs job/epinio-pvc-inspect -n +``` + +:::note Sidecar alternative for a still-running Pod + +If the application Pod is still running and you only need a quick look at the mounted volume, use an ephemeral debug container instead of a Job: + +```bash +kubectl debug -n -it --image=busybox:1.36 --target= -- sh +``` + +Then inspect the same mount path the app uses (`/mnt/state` for the bundled stateful chart). +Scale the app down before deleting the PVC. + +::: + +## Step 3: Ensure the claim is unused + +If the application still exists, scale it to zero (or delete it **without** expecting Epinio to remove data PVCs): + +```bash +# Example for a leftover StatefulSet from the stateful chart +kubectl scale statefulset -n --replicas=0 +kubectl wait --for=delete pod -n -l app.kubernetes.io/name= --timeout=120s +``` + +If only orphan Pods remain: + +```bash +kubectl delete pod -n --grace-period=0 --force +``` + +Re-check that no Pod mounts the claim (command in Step 1). + +## Step 4: Delete the PVC + +```bash +kubectl delete pvc -n +``` + +Delete several labeled app-data claims at once: + +```bash +kubectl delete pvc -n -l app.kubernetes.io/name= +``` + +Delete a known staging cache claim in the Epinio namespace: + +```bash +kubectl delete pvc -n epinio --ignore-not-found +kubectl delete pvc -n epinio --ignore-not-found +``` + +:::caution Deletion is irreversible for the claim + +Whether the underlying PersistentVolume and its data are removed depends on the StorageClass `reclaimPolicy`. +See Step 5 for `Retain`. + +::: + +## Step 5: Clean up `Retain` PersistentVolumes (if needed) + +If the StorageClass uses `Retain`, deleting the PVC leaves a Released PV and the disk data: + +```bash +kubectl get pv +kubectl describe pv +``` + +Only after you are sure the data can go: + +```bash +kubectl delete pv +``` + +Your storage provider may still need a separate volume delete; follow that provider’s process if the PV does not free capacity on its own. + +## Verification + +Confirm the claims are gone: + +```bash +kubectl get pvc -n -l app.kubernetes.io/name= +kubectl get pvc -n epinio | grep -E '|cache|sourceblobs' || true +``` + +If you deleted data for an app you still intend to run, push or restage it and confirm a **new** empty claim is provisioned (StatefulSet charts create a fresh PVC on the next deploy). + +## Cleanup + +Remove the inspect Job after you are done: + +```bash +kubectl delete job epinio-pvc-inspect -n --ignore-not-found +``` + +## Related + +- [What deletion removes](../../developer/concepts/applications/applications.mdx#what-deletion-removes) — Epinio’s built-in `--delete-pvc` / `--delete-image` behavior +- [Storage lifecycle](../../../reference/concepts/storage.md#storage-lifecycle) — capacity planning for leftover data volumes +- [Migrating from MinIO to SeaweedFS](../networking/migrate_minio_to_seaweedfs.md) — same Job-based operator pattern for storage work diff --git a/docs/reference/concepts/storage.md b/docs/reference/concepts/storage.md index e2a957eda3..61a0df5ee4 100644 --- a/docs/reference/concepts/storage.md +++ b/docs/reference/concepts/storage.md @@ -141,7 +141,9 @@ Epinio selects an application's data volumes by the `app.kubernetes.io/name` lab the application's namespace. A volume provisioned by a chart without that label is never deleted by Epinio. See [deleting an application](../../how-to/developer/concepts/applications/applications.mdx#what-deletion-removes) -for how to request each cleanup. +for how to request each cleanup, and +[Removing leftover PVCs without Epinio](../../how-to/operator/operations/remove_pvcs_without_epinio.md) +to reclaim claims with `kubectl` when Epinio cannot delete them. ## Storage Calculation Formulas @@ -307,7 +309,7 @@ Total Storage (GB) ≈ N apps × (0.05 + 1 + 1 + 0.5 × 3) = N apps × 3.55 GB 1. **Old Source Blobs**: Implement lifecycle policies to delete old S3 objects 2. **Old Images**: Configure registry garbage collection for unused images -3. **Unused PVCs**: Clean up PVCs left behind by deleted applications, see [Storage lifecycle](#storage-lifecycle) +3. **Unused PVCs**: Clean up PVCs left behind by deleted applications — see [Storage lifecycle](#storage-lifecycle) and [Removing leftover PVCs without Epinio](../../how-to/operator/operations/remove_pvcs_without_epinio.md) 4. **Build Cache**: Periodically clear build caches for applications that haven't been rebuilt recently ## Example Scenarios From 99dfff6843dfdfdd1a36890a64836d6140eb795d Mon Sep 17 00:00:00 2001 From: David Johnson Date: Wed, 9 Sep 2026 17:29:03 -0400 Subject: [PATCH 2/2] docs(Updates): Consolidated doc into a few steps, fixed some issues --- .../operations/remove_pvcs_without_epinio.md | 334 +++++++++--------- 1 file changed, 175 insertions(+), 159 deletions(-) diff --git a/docs/how-to/operator/operations/remove_pvcs_without_epinio.md b/docs/how-to/operator/operations/remove_pvcs_without_epinio.md index 5e1ea82e8b..dd0a8104de 100644 --- a/docs/how-to/operator/operations/remove_pvcs_without_epinio.md +++ b/docs/how-to/operator/operations/remove_pvcs_without_epinio.md @@ -2,245 +2,261 @@ sidebar_label: Removing leftover PVCs without Epinio sidebar_position: 26 title: Removing leftover PVCs without Epinio -description: How to find, inspect, and delete leftover Epinio application PVCs with kubectl Jobs when you cannot (or choose not to) use Epinio -keywords: [epinio, kubernetes, pvc, storage, cleanup, statefulset, job] +description: Find, back up, and delete Epinio PersistentVolumeClaims that no longer belong to an application, using kubectl alone +keywords: [epinio, kubernetes, pvc, storage, cleanup, statefulset, staging] doc-type: [how-to] doc-topic: [epinio, how-to, operations, remove-pvcs] doc-persona: [epinio-operator] --- -Epinio preserves **application data** PersistentVolumeClaims by default when an application is deleted. -Staging volumes (build cache and source-blob PVCs) are removed automatically on delete; data volumes from a StatefulSet `volumeClaimTemplates` chart are not, unless you pass `--delete-pvc`. - -This guide shows how to reclaim those leftover claims with `kubectl` and one-off Jobs. -It does **not** add or rely on Epinio maintenance API endpoints. +Epinio leaves two kinds of PersistentVolumeClaim behind, for two different reasons. +This guide finds both with one command, backs up the ones worth keeping, and deletes them using only `kubectl`. :::tip Prefer Epinio when you can -On Epinio **1.14.2+**, delete the application with PVC cleanup instead of this procedure: +If the application still exists and you are on Epinio **1.14.2+**, delete it with its data in one step: ```bash epinio app delete --delete-pvc ``` -Use this document when the application is already gone, Epinio is unavailable, or the claim was never labeled for Epinio to discover. +Use this guide when the application is already gone, when Epinio is unavailable, or when a claim was left behind by a delete that did not finish. ::: -## Who needs to follow this guide - -| Situation | Action required | -|---|---| -| App deleted **without** `--delete-pvc`, and StatefulSet data PVCs remain | Follow all steps below. | -| App scaled down; older ordinal PVCs remain | Follow all steps below. | -| Staging cache / source-blob PVC left behind (failed delete, manual edits, older install) | [Identify](#step-1-identify-target-pvcs) and [delete](#step-4-delete-the-pvc) those claims in the Epinio namespace. | -| You still have the app and run Epinio 1.14.2+ | Prefer `epinio app delete --delete-pvc` (or the UI checkbox). | -| Platform storage (SeaweedFS, registry, `image-export-pvc`) | **Do not** use this guide. Those claims belong to the Epinio install. | - -## Overview of Epinio PVCs - -| Kind | Namespace | Typical name | Created when | Removed by `epinio app delete` | -|---|---|---|---|---| -| Build cache | Epinio install namespace (default `epinio`) | `-cache--` | Staging with `server.stagingWorkload.storage.cache.emptyDir: false` (the default) | Always | -| Source blobs (staging workspace) | Epinio install namespace | `-sourceblobs--` | Staging with `sourceBlobs.emptyDir: false` (default is `true`, so usually absent) | Always | -| Application data | Application namespace | e.g. `stateful-r--0` | App chart with `volumeClaimTemplates` (bundled `application-stateful` chart) | Only with `--delete-pvc` | -| Platform | Epinio install namespace | SeaweedFS / registry / `image-export-pvc` | Helm install | Never by app delete — leave alone | - -Default staging storage (current chart): - -```yaml -server: - stagingWorkload: - storage: - cache: - emptyDir: false - size: 1Gi - sourceBlobs: - emptyDir: true -``` +## The two kinds of leftover claim -The bundled stateful app chart mounts application data at `/mnt/state` from a `volumeClaimTemplates` entry named `stateful`. -Kubernetes names each claim `{volumeClaimTemplate}-{statefulSetName}-{ordinal}` (for one replica, typically `stateful-r--0`). +Almost every question about leftover claims comes from treating these as one thing. They are not. -Epinio discovers application data PVCs with label `app.kubernetes.io/name=` in the application namespace. -Claims without that label are never deleted by Epinio and must be cleaned up with the steps below. +| | Application data | Staging cache | +|---|---|---| +| Lives in | the application namespace | the Epinio install namespace (default `epinio`) | +| Named | `stateful--` | `---` | +| Labeled | `app.kubernetes.io/name=` | **nothing**: no labels, no owner references | +| Holds | data your application wrote | buildpack layers from the last build | +| Left behind because | you deleted the app without `--delete-pvc` (**by design**) | a delete did not finish (**a fault**) | +| Worth backing up | yes | no, the next build rebuilds it | +| Created by | the chart's `volumeClaimTemplates` | every staging run | -## Step 1: Identify target PVCs +The consequences of that table are what make the two cases feel inconsistent: -List claims in the application namespace: +- **Application data claims are labeled**, because Kubernetes copies the StatefulSet's selector onto every claim its template creates. That label is how `--delete-pvc` finds them. +- **Staging claims are not labeled at all.** Epinio finds them by recomputing their name from the application, so a staging claim whose application is gone can no longer be found by anything. It is invisible, and it keeps its disk. -```bash -kubectl get pvc -n -``` +:::caution Platform storage is not in scope -Narrow by app label when present (same selector Epinio uses): +SeaweedFS, the registry, and `image-export-pvc` belong to the Epinio installation itself. +They are never leftovers. Do not delete them. -```bash -kubectl get pvc -n -l app.kubernetes.io/name= -``` +::: -List staging claims in the Epinio namespace (replace `epinio` if you installed elsewhere): +## Step 1: Identify + +This function lists every claim that no longer has an application behind it, of either kind. +Paste it into your shell once; the later steps reuse it. ```bash -kubectl get pvc -n epinio | grep -E 'cache|sourceblobs' || true +# Lists Epinio PersistentVolumeClaims that no longer have an application. +# Usage: epinio-dangling-pvcs [--names] +epinio-dangling-pvcs() { + local install_ns="${EPINIO_NAMESPACE:-epinio}" expected apps + + # Every staging claim name the surviving applications can account for. + expected=$(kubectl get apps.application.epinio.io -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace} {.metadata.name}{"\n"}{end}' | + while read -r ns app; do + [ -n "$app" ] || continue + for kind in cache sourceblobs; do + n="$ns-$kind-$app" + printf '%s-%s\n' "${n:0:22}" "$(printf '%s' "$n" | sha1sum | cut -d' ' -f1)" + done + done) + + apps=$(kubectl get apps.application.epinio.io -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}') + + kubectl get pvc -A -o json | jq -r \ + --arg install_ns "$install_ns" --arg expected "$expected" \ + --arg apps "$apps" --arg names "${1:-}" ' + ($expected | split("\n") | map(select(length > 0))) as $ok | + ($apps | split("\n") | map(select(length > 0))) as $live | + [ .items[] + | . as $p + | $p.metadata.namespace as $ns + | $p.metadata.name as $n + | ($p.metadata.labels // {}) as $l + | (if ($ns == $install_ns and ($n | test("-(cache|sourceblobs)-"))) + then (if ($ok | index($n)) then empty else {kind:"staging", app:"-"} end) + elif ($l["app.kubernetes.io/name"] and ($l["app.kubernetes.io/instance"] | not)) + then (if ($live | index($ns + "/" + $l["app.kubernetes.io/name"])) + then empty + else {kind:"app-data", app:$l["app.kubernetes.io/name"]} end) + else empty end) as $hit + | {ns:$ns, name:$n, kind:$hit.kind, app:$hit.app, + size:($p.status.capacity.storage // "-"), phase:$p.status.phase} + ] as $rows | + if $names == "--names" then + $rows[] | "\(.ns) \(.name)" + else + (["KIND","NAMESPACE","CLAIM","SIZE","APP"] | @tsv), + ($rows[] | [.kind, .ns, .name, .size, .app] | @tsv) + end' +} ``` -Confirm nothing important still uses a claim before you delete it: +Run it: ```bash -kubectl describe pvc -n -kubectl get pods -n -o json \ - | jq -r --arg pvc '' ' - .items[] - | select([.spec.volumes[]? | select(.persistentVolumeClaim.claimName == $pvc)] | length > 0) - | .metadata.name' +epinio-dangling-pvcs | column -t -s$'\t' ``` -:::caution +``` +KIND NAMESPACE CLAIM SIZE APP +staging epinio workspace-cache-exampl-1f9956f9... 1Gi - +app-data workspace stateful-r-09b90aa6...-0 1Gi my-stateful-app +``` + +Anything not listed still has an application that owns it. Claims belonging to Epinio **services** are skipped, because a service chart also sets `app.kubernetes.io/instance` and this function treats that as a sign the claim is owned by something else. -A PVC that is still mounted stays in `Terminating` until every Pod using it is gone. -Scale the workload to zero or delete the leftover Pods before Step 4. +:::note Custom application charts + +The `app-data` check assumes the claim carries `app.kubernetes.io/name` and no `app.kubernetes.io/instance`, which is what the bundled charts produce. +A custom chart that sets extra labels is skipped rather than listed. That errs toward leaving claims alone, so review a custom chart's claims by hand. ::: -## Step 2: (Optional) Inspect or back up with a one-off Job - -Mount the claim in a short-lived Job, the same pattern as the [MinIO → SeaweedFS migration](../networking/migrate_minio_to_seaweedfs.md) Jobs. -No Epinio API is involved. - -```yaml title="inspect-pvc-job.yaml" -apiVersion: batch/v1 -kind: Job -metadata: - name: epinio-pvc-inspect - namespace: # same namespace as the PVC -spec: - ttlSecondsAfterFinished: 600 - template: - spec: - restartPolicy: Never - volumes: - - name: data - persistentVolumeClaim: - claimName: # e.g. stateful-r-....-0 - containers: - - name: inspect - image: busybox:1.36 - command: ["/bin/sh", "-c"] - args: - - | - set -e - echo "=== listing /data ===" - ls -la /data - du -sh /data 2>/dev/null || true - # Optional: copy out anything you must keep, e.g. to another mounted volume - # tar -czf /tmp/backup.tgz -C /data . - volumeMounts: - - name: data - mountPath: /data -``` +## Step 2: Back up + +Only worth doing for `app-data` claims. A staging cache is rebuilt by the next `epinio push`, so there is nothing in it to lose. ```bash -kubectl apply -f inspect-pvc-job.yaml -kubectl wait --for=condition=complete job/epinio-pvc-inspect -n --timeout=120s -kubectl logs job/epinio-pvc-inspect -n +# Copies the contents of a claim into a local tarball. +# Usage: epinio-pvc-backup +epinio-pvc-backup() { + local ns="$1" claim="$2" out="$3" pod rc elsewhere + + if [ -z "$ns" ] || [ -z "$claim" ] || [ -z "$out" ]; then + echo "usage: epinio-pvc-backup " >&2 + return 2 + fi + + if ! kubectl get pvc -n "$ns" "$claim" >/dev/null 2>&1; then + echo "No claim '$claim' in namespace '$ns'." >&2 + elsewhere=$(kubectl get pvc -A \ + -o jsonpath="{range .items[?(@.metadata.name=='$claim')]}{.metadata.namespace}{'\n'}{end}" 2>/dev/null) + if [ -n "$elsewhere" ]; then + echo "It is in namespace '$elsewhere'. A staging claim is named after the" >&2 + echo "application's namespace but lives in the Epinio install namespace." >&2 + fi + return 1 + fi + + pod="pvc-backup-$RANDOM$RANDOM" + kubectl run "$pod" -n "$ns" --restart=Never --quiet --image=busybox:1.36 \ + --overrides="$(printf '%s' '{"spec":{"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"CLAIM"}}], + "containers":[{"name":"b","image":"busybox:1.36","command":["sleep","3600"], + "volumeMounts":[{"name":"d","mountPath":"/d","readOnly":true}]}]}}' | sed "s/CLAIM/$claim/")" >/dev/null || return 1 + + if kubectl wait --for=condition=Ready "pod/$pod" -n "$ns" --timeout=5m >&2; then + kubectl exec -n "$ns" "$pod" -- tar -czf - -C /d . > "$out"; rc=$? + else + echo "the backup Pod never became Ready. Inspect it with:" >&2 + echo " kubectl describe pod -n $ns $pod" >&2 + rc=1 + fi + + kubectl delete pod -n "$ns" "$pod" --wait=false >/dev/null 2>&1 + [ "$rc" -eq 0 ] && echo "wrote $out ($(wc -c < "$out") bytes)" >&2 + return "$rc" +} ``` -:::note Sidecar alternative for a still-running Pod - -If the application Pod is still running and you only need a quick look at the mounted volume, use an ephemeral debug container instead of a Job: +Pass the namespace from the `NAMESPACE` column of Step 1, not the one embedded in the claim's name: ```bash -kubectl debug -n -it --image=busybox:1.36 --target= -- sh +epinio-pvc-backup workspace stateful-r-09b90aa6...-0 my-app-data.tgz +tar -tzf my-app-data.tgz ``` -Then inspect the same mount path the app uses (`/mnt/state` for the bundled stateful chart). -Scale the app down before deleting the PVC. +The claim is mounted read-only, and the Pod is removed whether the copy succeeds or fails. -::: +:::caution A ReadWriteOnce claim in use cannot be read -## Step 3: Ensure the claim is unused +These claims attach to one **node** at a time. If a Pod still mounts the claim, the backup Pod never becomes Ready unless it happens to schedule onto that same node, and the function reports that rather than hanging. +Scale the workload down first. See [Step 3](#step-3-delete). -If the application still exists, scale it to zero (or delete it **without** expecting Epinio to remove data PVCs): +::: -```bash -# Example for a leftover StatefulSet from the stateful chart -kubectl scale statefulset -n --replicas=0 -kubectl wait --for=delete pod -n -l app.kubernetes.io/name= --timeout=120s -``` +:::note Why this uses `kubectl exec` rather than an attached Pod -If only orphan Pods remain: +Streaming the archive out of `kubectl run -i` looks simpler and quietly produces a **truncated tarball**: the container starts writing before `kubectl` finishes attaching, and the output produced in that window is lost. +The symptom is a `couldn't fetch pre-attach logs` warning and an archive that fails `gzip -t`. On a small volume it can appear to work. +`kubectl exec` against an already-running Pod has no such race and is binary-safe, which is the same mechanism `kubectl cp` uses. -```bash -kubectl delete pod -n --grace-period=0 --force -``` +::: -Re-check that no Pod mounts the claim (command in Step 1). +## Step 3: Delete -## Step 4: Delete the PVC +Delete one claim: ```bash -kubectl delete pvc -n +kubectl delete pvc -n ``` -Delete several labeled app-data claims at once: +Delete every dangling claim of both kinds: ```bash -kubectl delete pvc -n -l app.kubernetes.io/name= +epinio-dangling-pvcs --names | while read -r ns name; do + kubectl delete pvc -n "$ns" "$name" +done ``` -Delete a known staging cache claim in the Epinio namespace: +Preview it first. The same loop with `--dry-run=client` changes nothing: ```bash -kubectl delete pvc -n epinio --ignore-not-found -kubectl delete pvc -n epinio --ignore-not-found +epinio-dangling-pvcs --names | while read -r ns name; do + kubectl delete pvc -n "$ns" "$name" --dry-run=client +done ``` -:::caution Deletion is irreversible for the claim - -Whether the underlying PersistentVolume and its data are removed depends on the StorageClass `reclaimPolicy`. -See Step 5 for `Retain`. - -::: - -## Step 5: Clean up `Retain` PersistentVolumes (if needed) +:::caution A claim in use will not go away -If the StorageClass uses `Retain`, deleting the PVC leaves a Released PV and the disk data: +Deleting a claim that a Pod still mounts leaves it in `Terminating` until that Pod is gone. +Scale the workload down first: ```bash -kubectl get pv -kubectl describe pv +kubectl get statefulset -n -l app.kubernetes.io/name= +kubectl scale statefulset -n --replicas=0 +kubectl wait --for=delete pod -n -l app.kubernetes.io/name= --timeout=120s ``` -Only after you are sure the data can go: +Force-deleting a Pod (`--grace-period=0 --force`) removes the Pod object without waiting for the kubelet to unmount, which can leave the volume attached and the claim stuck. Reserve it for a node that is genuinely unreachable. -```bash -kubectl delete pv -``` - -Your storage provider may still need a separate volume delete; follow that provider’s process if the PV does not free capacity on its own. +::: -## Verification +## Check the volumes were actually reclaimed -Confirm the claims are gone: +Deleting a claim does not guarantee the disk came back. That is decided by the StorageClass `reclaimPolicy` on the PersistentVolume behind it. ```bash -kubectl get pvc -n -l app.kubernetes.io/name= -kubectl get pvc -n epinio | grep -E '|cache|sourceblobs' || true +kubectl get pv -o custom-columns=\ +'NAME:.metadata.name,STATUS:.status.phase,POLICY:.spec.persistentVolumeReclaimPolicy,CLAIM:.spec.claimRef.name' \ + | awk 'NR==1 || $2=="Released"' ``` -If you deleted data for an app you still intend to run, push or restage it and confirm a **new** empty claim is provisioned (StatefulSet charts create a fresh PVC on the next deploy). - -## Cleanup +- **`Retain`**: a `Released` volume here is expected. The data is intact and waiting for you. Remove it with `kubectl delete pv `, then reclaim the disk through your storage provider if it does not free on its own. +- **`Delete`**: a `Released` volume here is a **fault**. The provisioner should have removed the volume and its data and did not, so that disk is still allocated with nothing pointing at it. `kubectl delete pv` removes the API object but does not necessarily reclaim the underlying disk; check the provisioner's logs and your storage backend. -Remove the inspect Job after you are done: +## Verification ```bash -kubectl delete job epinio-pvc-inspect -n --ignore-not-found +epinio-dangling-pvcs ``` +Empty output means nothing is dangling. If you deleted data for an application you still intend to run, push it again and confirm a **new** empty claim is provisioned. StatefulSet charts create a fresh one on the next deploy. + ## Related -- [What deletion removes](../../developer/concepts/applications/applications.mdx#what-deletion-removes) — Epinio’s built-in `--delete-pvc` / `--delete-image` behavior -- [Storage lifecycle](../../../reference/concepts/storage.md#storage-lifecycle) — capacity planning for leftover data volumes -- [Migrating from MinIO to SeaweedFS](../networking/migrate_minio_to_seaweedfs.md) — same Job-based operator pattern for storage work +- [What deletion removes](../../developer/concepts/applications/applications.mdx#what-deletion-removes): Epinio's built-in `--delete-pvc` / `--delete-image` behavior +- [Storage lifecycle](../../../reference/concepts/storage.md#storage-lifecycle): capacity planning for leftover data volumes +- [Migrating from MinIO to SeaweedFS](../networking/migrate_minio_to_seaweedfs.md): the Job-based pattern for larger storage work