diff --git a/docs/how-to/developer/concepts/applications/applications.mdx b/docs/how-to/developer/concepts/applications/applications.mdx index 87a769d3fc..09fa73b158 100644 --- a/docs/how-to/developer/concepts/applications/applications.mdx +++ b/docs/how-to/developer/concepts/applications/applications.mdx @@ -258,7 +258,9 @@ Epinio selects an application's data volumes by the `app.kubernetes.io/name` lab the application's namespace. A volume your chart provisions without that label is not deleted, and has to be removed with `kubectl`. See [storage lifecycle](../../../../reference/concepts/storage.md#storage-lifecycle) for how -this affects capacity planning. +this affects capacity planning, and +[Removing leftover PVCs without Epinio](../../../operator/operations/remove_pvcs_without_epinio.md) +for a kubectl Job-based cleanup when the application is already gone or Epinio is unavailable. ## Additional Actions diff --git a/docs/how-to/operator/operations/remove_pvcs_without_epinio.md b/docs/how-to/operator/operations/remove_pvcs_without_epinio.md new file mode 100644 index 0000000000..dd0a8104de --- /dev/null +++ b/docs/how-to/operator/operations/remove_pvcs_without_epinio.md @@ -0,0 +1,262 @@ +--- +sidebar_label: Removing leftover PVCs without Epinio +sidebar_position: 26 +title: Removing leftover PVCs without Epinio +description: Find, back up, and delete Epinio PersistentVolumeClaims that no longer belong to an application, using kubectl alone +keywords: [epinio, kubernetes, pvc, storage, cleanup, statefulset, staging] +doc-type: [how-to] +doc-topic: [epinio, how-to, operations, remove-pvcs] +doc-persona: [epinio-operator] +--- + +Epinio leaves two kinds of PersistentVolumeClaim behind, for two different reasons. +This guide finds both with one command, backs up the ones worth keeping, and deletes them using only `kubectl`. + +:::tip Prefer Epinio when you can + +If the application still exists and you are on Epinio **1.14.2+**, delete it with its data in one step: + +```bash +epinio app delete --delete-pvc +``` + +Use this guide when the application is already gone, when Epinio is unavailable, or when a claim was left behind by a delete that did not finish. + +::: + +## The two kinds of leftover claim + +Almost every question about leftover claims comes from treating these as one thing. They are not. + +| | Application data | Staging cache | +|---|---|---| +| Lives in | the application namespace | the Epinio install namespace (default `epinio`) | +| Named | `stateful--` | `---` | +| Labeled | `app.kubernetes.io/name=` | **nothing**: no labels, no owner references | +| Holds | data your application wrote | buildpack layers from the last build | +| Left behind because | you deleted the app without `--delete-pvc` (**by design**) | a delete did not finish (**a fault**) | +| Worth backing up | yes | no, the next build rebuilds it | +| Created by | the chart's `volumeClaimTemplates` | every staging run | + +The consequences of that table are what make the two cases feel inconsistent: + +- **Application data claims are labeled**, because Kubernetes copies the StatefulSet's selector onto every claim its template creates. That label is how `--delete-pvc` finds them. +- **Staging claims are not labeled at all.** Epinio finds them by recomputing their name from the application, so a staging claim whose application is gone can no longer be found by anything. It is invisible, and it keeps its disk. + +:::caution Platform storage is not in scope + +SeaweedFS, the registry, and `image-export-pvc` belong to the Epinio installation itself. +They are never leftovers. Do not delete them. + +::: + +## Step 1: Identify + +This function lists every claim that no longer has an application behind it, of either kind. +Paste it into your shell once; the later steps reuse it. + +```bash +# Lists Epinio PersistentVolumeClaims that no longer have an application. +# Usage: epinio-dangling-pvcs [--names] +epinio-dangling-pvcs() { + local install_ns="${EPINIO_NAMESPACE:-epinio}" expected apps + + # Every staging claim name the surviving applications can account for. + expected=$(kubectl get apps.application.epinio.io -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace} {.metadata.name}{"\n"}{end}' | + while read -r ns app; do + [ -n "$app" ] || continue + for kind in cache sourceblobs; do + n="$ns-$kind-$app" + printf '%s-%s\n' "${n:0:22}" "$(printf '%s' "$n" | sha1sum | cut -d' ' -f1)" + done + done) + + apps=$(kubectl get apps.application.epinio.io -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}') + + kubectl get pvc -A -o json | jq -r \ + --arg install_ns "$install_ns" --arg expected "$expected" \ + --arg apps "$apps" --arg names "${1:-}" ' + ($expected | split("\n") | map(select(length > 0))) as $ok | + ($apps | split("\n") | map(select(length > 0))) as $live | + [ .items[] + | . as $p + | $p.metadata.namespace as $ns + | $p.metadata.name as $n + | ($p.metadata.labels // {}) as $l + | (if ($ns == $install_ns and ($n | test("-(cache|sourceblobs)-"))) + then (if ($ok | index($n)) then empty else {kind:"staging", app:"-"} end) + elif ($l["app.kubernetes.io/name"] and ($l["app.kubernetes.io/instance"] | not)) + then (if ($live | index($ns + "/" + $l["app.kubernetes.io/name"])) + then empty + else {kind:"app-data", app:$l["app.kubernetes.io/name"]} end) + else empty end) as $hit + | {ns:$ns, name:$n, kind:$hit.kind, app:$hit.app, + size:($p.status.capacity.storage // "-"), phase:$p.status.phase} + ] as $rows | + if $names == "--names" then + $rows[] | "\(.ns) \(.name)" + else + (["KIND","NAMESPACE","CLAIM","SIZE","APP"] | @tsv), + ($rows[] | [.kind, .ns, .name, .size, .app] | @tsv) + end' +} +``` + +Run it: + +```bash +epinio-dangling-pvcs | column -t -s$'\t' +``` + +``` +KIND NAMESPACE CLAIM SIZE APP +staging epinio workspace-cache-exampl-1f9956f9... 1Gi - +app-data workspace stateful-r-09b90aa6...-0 1Gi my-stateful-app +``` + +Anything not listed still has an application that owns it. Claims belonging to Epinio **services** are skipped, because a service chart also sets `app.kubernetes.io/instance` and this function treats that as a sign the claim is owned by something else. + +:::note Custom application charts + +The `app-data` check assumes the claim carries `app.kubernetes.io/name` and no `app.kubernetes.io/instance`, which is what the bundled charts produce. +A custom chart that sets extra labels is skipped rather than listed. That errs toward leaving claims alone, so review a custom chart's claims by hand. + +::: + +## Step 2: Back up + +Only worth doing for `app-data` claims. A staging cache is rebuilt by the next `epinio push`, so there is nothing in it to lose. + +```bash +# Copies the contents of a claim into a local tarball. +# Usage: epinio-pvc-backup +epinio-pvc-backup() { + local ns="$1" claim="$2" out="$3" pod rc elsewhere + + if [ -z "$ns" ] || [ -z "$claim" ] || [ -z "$out" ]; then + echo "usage: epinio-pvc-backup " >&2 + return 2 + fi + + if ! kubectl get pvc -n "$ns" "$claim" >/dev/null 2>&1; then + echo "No claim '$claim' in namespace '$ns'." >&2 + elsewhere=$(kubectl get pvc -A \ + -o jsonpath="{range .items[?(@.metadata.name=='$claim')]}{.metadata.namespace}{'\n'}{end}" 2>/dev/null) + if [ -n "$elsewhere" ]; then + echo "It is in namespace '$elsewhere'. A staging claim is named after the" >&2 + echo "application's namespace but lives in the Epinio install namespace." >&2 + fi + return 1 + fi + + pod="pvc-backup-$RANDOM$RANDOM" + kubectl run "$pod" -n "$ns" --restart=Never --quiet --image=busybox:1.36 \ + --overrides="$(printf '%s' '{"spec":{"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"CLAIM"}}], + "containers":[{"name":"b","image":"busybox:1.36","command":["sleep","3600"], + "volumeMounts":[{"name":"d","mountPath":"/d","readOnly":true}]}]}}' | sed "s/CLAIM/$claim/")" >/dev/null || return 1 + + if kubectl wait --for=condition=Ready "pod/$pod" -n "$ns" --timeout=5m >&2; then + kubectl exec -n "$ns" "$pod" -- tar -czf - -C /d . > "$out"; rc=$? + else + echo "the backup Pod never became Ready. Inspect it with:" >&2 + echo " kubectl describe pod -n $ns $pod" >&2 + rc=1 + fi + + kubectl delete pod -n "$ns" "$pod" --wait=false >/dev/null 2>&1 + [ "$rc" -eq 0 ] && echo "wrote $out ($(wc -c < "$out") bytes)" >&2 + return "$rc" +} +``` + +Pass the namespace from the `NAMESPACE` column of Step 1, not the one embedded in the claim's name: + +```bash +epinio-pvc-backup workspace stateful-r-09b90aa6...-0 my-app-data.tgz +tar -tzf my-app-data.tgz +``` + +The claim is mounted read-only, and the Pod is removed whether the copy succeeds or fails. + +:::caution A ReadWriteOnce claim in use cannot be read + +These claims attach to one **node** at a time. If a Pod still mounts the claim, the backup Pod never becomes Ready unless it happens to schedule onto that same node, and the function reports that rather than hanging. +Scale the workload down first. See [Step 3](#step-3-delete). + +::: + +:::note Why this uses `kubectl exec` rather than an attached Pod + +Streaming the archive out of `kubectl run -i` looks simpler and quietly produces a **truncated tarball**: the container starts writing before `kubectl` finishes attaching, and the output produced in that window is lost. +The symptom is a `couldn't fetch pre-attach logs` warning and an archive that fails `gzip -t`. On a small volume it can appear to work. +`kubectl exec` against an already-running Pod has no such race and is binary-safe, which is the same mechanism `kubectl cp` uses. + +::: + +## Step 3: Delete + +Delete one claim: + +```bash +kubectl delete pvc -n +``` + +Delete every dangling claim of both kinds: + +```bash +epinio-dangling-pvcs --names | while read -r ns name; do + kubectl delete pvc -n "$ns" "$name" +done +``` + +Preview it first. The same loop with `--dry-run=client` changes nothing: + +```bash +epinio-dangling-pvcs --names | while read -r ns name; do + kubectl delete pvc -n "$ns" "$name" --dry-run=client +done +``` + +:::caution A claim in use will not go away + +Deleting a claim that a Pod still mounts leaves it in `Terminating` until that Pod is gone. +Scale the workload down first: + +```bash +kubectl get statefulset -n -l app.kubernetes.io/name= +kubectl scale statefulset -n --replicas=0 +kubectl wait --for=delete pod -n -l app.kubernetes.io/name= --timeout=120s +``` + +Force-deleting a Pod (`--grace-period=0 --force`) removes the Pod object without waiting for the kubelet to unmount, which can leave the volume attached and the claim stuck. Reserve it for a node that is genuinely unreachable. + +::: + +## Check the volumes were actually reclaimed + +Deleting a claim does not guarantee the disk came back. That is decided by the StorageClass `reclaimPolicy` on the PersistentVolume behind it. + +```bash +kubectl get pv -o custom-columns=\ +'NAME:.metadata.name,STATUS:.status.phase,POLICY:.spec.persistentVolumeReclaimPolicy,CLAIM:.spec.claimRef.name' \ + | awk 'NR==1 || $2=="Released"' +``` + +- **`Retain`**: a `Released` volume here is expected. The data is intact and waiting for you. Remove it with `kubectl delete pv `, then reclaim the disk through your storage provider if it does not free on its own. +- **`Delete`**: a `Released` volume here is a **fault**. The provisioner should have removed the volume and its data and did not, so that disk is still allocated with nothing pointing at it. `kubectl delete pv` removes the API object but does not necessarily reclaim the underlying disk; check the provisioner's logs and your storage backend. + +## Verification + +```bash +epinio-dangling-pvcs +``` + +Empty output means nothing is dangling. If you deleted data for an application you still intend to run, push it again and confirm a **new** empty claim is provisioned. StatefulSet charts create a fresh one on the next deploy. + +## Related + +- [What deletion removes](../../developer/concepts/applications/applications.mdx#what-deletion-removes): Epinio's built-in `--delete-pvc` / `--delete-image` behavior +- [Storage lifecycle](../../../reference/concepts/storage.md#storage-lifecycle): capacity planning for leftover data volumes +- [Migrating from MinIO to SeaweedFS](../networking/migrate_minio_to_seaweedfs.md): the Job-based pattern for larger storage work diff --git a/docs/reference/concepts/storage.md b/docs/reference/concepts/storage.md index e2a957eda3..61a0df5ee4 100644 --- a/docs/reference/concepts/storage.md +++ b/docs/reference/concepts/storage.md @@ -141,7 +141,9 @@ Epinio selects an application's data volumes by the `app.kubernetes.io/name` lab the application's namespace. A volume provisioned by a chart without that label is never deleted by Epinio. See [deleting an application](../../how-to/developer/concepts/applications/applications.mdx#what-deletion-removes) -for how to request each cleanup. +for how to request each cleanup, and +[Removing leftover PVCs without Epinio](../../how-to/operator/operations/remove_pvcs_without_epinio.md) +to reclaim claims with `kubectl` when Epinio cannot delete them. ## Storage Calculation Formulas @@ -307,7 +309,7 @@ Total Storage (GB) ≈ N apps × (0.05 + 1 + 1 + 0.5 × 3) = N apps × 3.55 GB 1. **Old Source Blobs**: Implement lifecycle policies to delete old S3 objects 2. **Old Images**: Configure registry garbage collection for unused images -3. **Unused PVCs**: Clean up PVCs left behind by deleted applications, see [Storage lifecycle](#storage-lifecycle) +3. **Unused PVCs**: Clean up PVCs left behind by deleted applications — see [Storage lifecycle](#storage-lifecycle) and [Removing leftover PVCs without Epinio](../../how-to/operator/operations/remove_pvcs_without_epinio.md) 4. **Build Cache**: Periodically clear build caches for applications that haven't been rebuilt recently ## Example Scenarios