From 26b8f53a4f0dc20ed206c24cdaf25a3528d5e91e Mon Sep 17 00:00:00 2001 From: Dmitrii Creed Date: Fri, 21 Aug 2026 22:20:22 +0400 Subject: [PATCH] fix(docker): stop the distro-upgrade layer being cached forever `simplecontainer/github-actions:latest` ships python3 3.14.5-r0 -- 12 HIGH, 32 MEDIUM, 4 LOW across python3/pyc/python3-pyc/python3-pycache-pyc0 -- while Alpine v3.24/main has served 3.14.7-r1 for a while. Our own binary in that image scans 0; every finding is the base OS layer. The cause is the build cache, not a dependency version. Release, staging and preview builds all pass `cache-from: type=gha`, and the final stage's base is digest-pinned with an `apk update && apk upgrade && apk add ...` RUN string that never changes. So that layer's cache key is permanently stable and the upgrade has not actually executed since the first ever build. From the release log for 076224a: --cache-from type=gha,scope=prod-github-actions #13 [stage-1 2/8] RUN apk update && apk upgrade --no-cache && apk add ... #13 CACHED `--no-cache` on the apk line is not related -- it governs apk's own index cache, not Docker layers. Fix: name the final stage `runtime` in all five Dockerfiles and pass `no-cache-filters: runtime` to docker/build-push-action in push.yaml, build-staging.yml and branch-preview.yaml. Scoped to the final stage deliberately. The expensive work -- the SHA-256 verified Pulumi and gcloud downloads, `gcloud components install` and the slimming pass -- lives in `builder` and keeps its cache. Measured on github-actions.Dockerfile with a warm builder: fully cached (today) 7.3 s python3 3.14.5-r0 48 os findings --no-cache-filter runtime 41.1 s python3 3.14.7-r1 0 os findings 34 seconds per image. Verified across all five: caddy, cloud-helpers, kubectl, github-actions and github-actions-staging each build clean with the filter and report 0 OS-package findings. Remaining findings are upstream binaries we do not compile -- gcloud-crc32c and gke-gcloud-auth-plugin (17), kubectl (17), cel-go in the Caddy binary (1) -- all already documented in #387. Why this matters beyond python3: git, openssh-client, curl, libssl3, libcrypto3 and aws-cli sit in the same frozen layer. The next advisory in any of them would have been just as invisible, in the image that runs every consumer's deploy holding their cloud credentials. caddy and cloud-helpers only looked healthy because #387 happened to change instructions upstream of their upgrade layers, which busted the chain by accident. Not switching these to distroless or scratch: `aws` is a `#!/usr/bin/python3` script, `gcloud` is a shell script that needs a Python interpreter, and SC shells out to bash, git, ssh, pulumi and gcloud via Pulumi local.Command. A shell-less base cannot run this image. Scratch is worth considering for cloud-helpers (a single static Go binary, already at 0 findings) and possibly caddy -- tracked separately, not mixed into a cache fix. Signed-off-by: Dmitrii Creed --- .github/workflows/branch-preview.yaml | 12 ++++++++++++ .github/workflows/build-staging.yml | 14 ++++++++++++++ .github/workflows/push.yaml | 12 ++++++++++++ caddy.Dockerfile | 10 +++++++++- cloud-helpers.aws.Dockerfile | 10 +++++++++- github-actions-staging.Dockerfile | 6 +++++- github-actions.Dockerfile | 10 +++++++++- kubectl.Dockerfile | 10 +++++++++- 8 files changed, 79 insertions(+), 5 deletions(-) diff --git a/.github/workflows/branch-preview.yaml b/.github/workflows/branch-preview.yaml index 14fdb9a4..1fd76852 100644 --- a/.github/workflows/branch-preview.yaml +++ b/.github/workflows/branch-preview.yaml @@ -362,6 +362,18 @@ jobs: type=gha,scope=preview-${{ matrix.cache_scope }} type=gha,scope=prod-${{ matrix.cache_scope }} cache-to: type=gha,mode=max,scope=preview-${{ matrix.cache_scope }} + # Never cache-hit the distro-upgrade layer. `runtime` is the final + # stage in every one of these Dockerfiles; its base is digest-pinned + # and its `apk/dnf upgrade` RUN string never changes, so with + # `cache-from` its key is permanently stable and the upgrade silently + # stops executing after the first ever build. That is how + # `simplecontainer/github-actions:latest` came to ship python3 + # 3.14.5-r0 (12 HIGH) while Alpine already served 3.14.7-r1 — and it + # would have frozen git / openssh-client / curl / libssl3 the same way. + # Scoped to the final stage on purpose: the expensive builder stage + # (SHA-verified Pulumi + gcloud download, install and slim) keeps its + # cache. Measured cost of re-running `runtime` alone: ~34 s. + no-cache-filters: runtime provenance: false # Phase 2 attestation (mirrors push.yaml). Preview builds get the SAME # security guarantees as production releases so consumers pin-testing a diff --git a/.github/workflows/build-staging.yml b/.github/workflows/build-staging.yml index c8333610..452e9bc2 100644 --- a/.github/workflows/build-staging.yml +++ b/.github/workflows/build-staging.yml @@ -119,6 +119,18 @@ jobs: type=gha,scope=staging-github-actions type=gha,scope=prod-github-actions cache-to: type=gha,mode=max,scope=staging-github-actions + # Never cache-hit the distro-upgrade layer. `runtime` is the final + # stage in every one of these Dockerfiles; its base is digest-pinned + # and its `apk/dnf upgrade` RUN string never changes, so with + # `cache-from` its key is permanently stable and the upgrade silently + # stops executing after the first ever build. That is how + # `simplecontainer/github-actions:latest` came to ship python3 + # 3.14.5-r0 (12 HIGH) while Alpine already served 3.14.7-r1 — and it + # would have frozen git / openssh-client / curl / libssl3 the same way. + # Scoped to the final stage on purpose: the expensive builder stage + # (SHA-verified Pulumi + gcloud download, install and slim) keeps its + # cache. Measured cost of re-running `runtime` alone: ~34 s. + no-cache-filters: runtime provenance: false - name: Build and push caddy staging image @@ -136,6 +148,8 @@ jobs: type=gha,scope=staging-caddy type=gha,scope=prod-caddy cache-to: type=gha,mode=max,scope=staging-caddy + # Same reason as above — see the first build step in this file. + no-cache-filters: runtime provenance: false # Phase 2: sign + SBOM + SLSA provenance for staging images. Staging has diff --git a/.github/workflows/push.yaml b/.github/workflows/push.yaml index a2889de5..024896b3 100644 --- a/.github/workflows/push.yaml +++ b/.github/workflows/push.yaml @@ -474,6 +474,18 @@ jobs: # Release builds read and write `prod-*` only. cache-from: type=gha,scope=prod-${{ matrix.image }} cache-to: type=gha,mode=max,scope=prod-${{ matrix.image }} + # Never cache-hit the distro-upgrade layer. `runtime` is the final + # stage in every one of these Dockerfiles; its base is digest-pinned + # and its `apk/dnf upgrade` RUN string never changes, so with + # `cache-from` its key is permanently stable and the upgrade silently + # stops executing after the first ever build. That is how + # `simplecontainer/github-actions:latest` came to ship python3 + # 3.14.5-r0 (12 HIGH) while Alpine already served 3.14.7-r1 — and it + # would have frozen git / openssh-client / curl / libssl3 the same way. + # Scoped to the final stage on purpose: the expensive builder stage + # (SHA-verified Pulumi + gcloud download, install and slim) keeps its + # cache. Measured cost of re-running `runtime` alone: ~34 s. + no-cache-filters: runtime provenance: false # Phase 2 attestation: keyless cosign sign + CycloneDX SBOM + SLSA L3 # provenance. The publish step above is the gating job. Sign/attest steps diff --git a/caddy.Dockerfile b/caddy.Dockerfile index 0acb3bf1..b6886bf6 100644 --- a/caddy.Dockerfile +++ b/caddy.Dockerfile @@ -111,7 +111,15 @@ RUN --mount=type=cache,target=/go/pkg/mod,sharing=locked \ # falls back to local-filesystem cert storage, so a multi-replica parent # stack gets per-pod ACME state and risks Let's Encrypt rate-limit lockout. -FROM caddy:2.11.4@sha256:df7f1c2fb114453b951de51a98efc010db1655a92c2e86be6706714e2417a78d +# The final stage is named `runtime` so CI can pass +# `no-cache-filters: runtime` to docker/build-push-action. Without it the +# distro-upgrade layer below is cached FOREVER: the base is digest-pinned and +# the RUN string never changes, so its cache key is permanently stable and +# `apk upgrade` never actually executes again. `simplecontainer/github-actions:latest` +# shipped python3 3.14.5-r0 (12 HIGH) for exactly this reason while Alpine +# already served 3.14.7-r1. Note `--no-cache` on the apk line is unrelated — it +# governs apk's own index cache, not Docker layers. +FROM caddy:2.11.4@sha256:df7f1c2fb114453b951de51a98efc010db1655a92c2e86be6706714e2417a78d AS runtime RUN apk update && apk upgrade --no-cache && rm -rf /var/cache/apk/* diff --git a/cloud-helpers.aws.Dockerfile b/cloud-helpers.aws.Dockerfile index 7b6215b1..4935def3 100644 --- a/cloud-helpers.aws.Dockerfile +++ b/cloud-helpers.aws.Dockerfile @@ -1,5 +1,13 @@ # Refresh: docker buildx imagetools inspect public.ecr.aws/lambda/provided:al2023 -FROM public.ecr.aws/lambda/provided:al2023@sha256:5f3ae3216e07bb3677cc4dfa0c7867973f7e536abb114d6b44a7b8c558824812 +# The final stage is named `runtime` so CI can pass +# `no-cache-filters: runtime` to docker/build-push-action. Without it the +# distro-upgrade layer below is cached FOREVER: the base is digest-pinned and +# the RUN string never changes, so its cache key is permanently stable and +# `dnf upgrade` never actually executes again. `simplecontainer/github-actions:latest` +# shipped python3 3.14.5-r0 (12 HIGH) for exactly this reason while Alpine +# already served 3.14.7-r1. Note `--no-cache` on the apk line is unrelated — it +# governs apk's own index cache, not Docker layers. +FROM public.ecr.aws/lambda/provided:al2023@sha256:5f3ae3216e07bb3677cc4dfa0c7867973f7e536abb114d6b44a7b8c558824812 AS runtime # Pull post-tag distro fixes (e.g. glibc CVE-2026-4046 once published to AL2023 dnf). RUN dnf upgrade -y --setopt=tsflags=nodocs \ diff --git a/github-actions-staging.Dockerfile b/github-actions-staging.Dockerfile index 0cb12c2e..78e5ad08 100644 --- a/github-actions-staging.Dockerfile +++ b/github-actions-staging.Dockerfile @@ -73,7 +73,11 @@ RUN rm -rf \ && rm -rf /tmp/* /var/tmp/* # ── runtime ───────────────────────────────────────────────────────────────── -FROM alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b +# Named `runtime` so CI can pass `no-cache-filters: runtime`. See +# github-actions.Dockerfile for why: without it the apk upgrade layer below sits +# behind a digest-pinned base with an unchanging RUN string, so its cache key +# never moves and the upgrade never actually re-runs. +FROM alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b AS runtime # aws-cli needed by Pulumi local.Command shell-outs (e.g. `aws s3 sync` in the # static-website template at pkg/clouds/pulumi/aws/static_website.go). diff --git a/github-actions.Dockerfile b/github-actions.Dockerfile index b1e328d4..8647c505 100644 --- a/github-actions.Dockerfile +++ b/github-actions.Dockerfile @@ -84,7 +84,15 @@ RUN rm -rf \ && rm -rf /tmp/* /var/tmp/* # ── runtime ───────────────────────────────────────────────────────────────── -FROM alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b +# The final stage is named `runtime` so CI can pass +# `no-cache-filters: runtime` to docker/build-push-action. Without it the +# distro-upgrade layer below is cached FOREVER: the base is digest-pinned and +# the RUN string never changes, so its cache key is permanently stable and +# `apk upgrade` never actually executes again. `simplecontainer/github-actions:latest` +# shipped python3 3.14.5-r0 (12 HIGH) for exactly this reason while Alpine +# already served 3.14.7-r1. Note `--no-cache` on the apk line is unrelated — it +# governs apk's own index cache, not Docker layers. +FROM alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b AS runtime # python3 stays — gcloud invokes it. py3-pip / binutils / upx confined to builder. # aws-cli needed by Pulumi local.Command shell-outs (e.g. `aws s3 sync` in the diff --git a/kubectl.Dockerfile b/kubectl.Dockerfile index 2125d88a..930ebb44 100644 --- a/kubectl.Dockerfile +++ b/kubectl.Dockerfile @@ -1,5 +1,13 @@ # Refresh: docker buildx imagetools inspect alpine/kubectl:latest -FROM alpine/kubectl:latest@sha256:5d380d18d2509483aef3df54d676c767d798d55ec9f3e02dabfa4c88fe6559bd +# The final stage is named `runtime` so CI can pass +# `no-cache-filters: runtime` to docker/build-push-action. Without it the +# distro-upgrade layer below is cached FOREVER: the base is digest-pinned and +# the RUN string never changes, so its cache key is permanently stable and +# `apk upgrade` never actually executes again. `simplecontainer/github-actions:latest` +# shipped python3 3.14.5-r0 (12 HIGH) for exactly this reason while Alpine +# already served 3.14.7-r1. Note `--no-cache` on the apk line is unrelated — it +# governs apk's own index cache, not Docker layers. +FROM alpine/kubectl:latest@sha256:5d380d18d2509483aef3df54d676c767d798d55ec9f3e02dabfa4c88fe6559bd AS runtime # apk upgrade pulls post-tag distro fixes (e.g. nghttp2 CVE-2026-27135 at scan time). RUN apk update \