Skip to content

ci: bound the apt install and prefer the canonical Ubuntu archive - #133

Merged
abrichr merged 1 commit into
mainfrom
claude/bound-apt-install
Aug 19, 2026
Merged

ci: bound the apt install and prefer the canonical Ubuntu archive#133
abrichr merged 1 commit into
mainfrom
claude/bound-apt-install

Conversation

@abrichr

@abrichr abrichr commented Aug 19, 2026

Copy link
Copy Markdown
Member

What

In db-backup.yml, job dump has timeout-minutes: 30, but the step Install age had no bound of its own.

  • Step Install age: no timeout -> 5 minutes, plus the mirror preference and a bounded apt-get update retry.

Runner OS for the changed job: ubuntu-latest. Both jobs in this file (dump, record-alert) are ubuntu-latest; only dump changed.

Why

This is the production database backup. A job-level timeout is not enough: one hung apt call spends the whole 30 minutes and no recovery point is taken that day. The job fails on a timeout that names the wrong cause.

The hosted runner resolves its Ubuntu mirror through /etc/apt/apt-mirrors.txt, which points at azure.archive.ubuntu.com. That mirror fails intermittently. apt does fall back to the canonical archive, but it spends the whole budget getting there. Two incidents on 2026-08-19: in openadapt-flow a TeX install ran 3h11m and hit the run limit, and in openadapt-capture a test job ran over an hour on main.

Why 5 minutes, not 10

Deliberately tighter than the pattern's default, because here the apt step competes with the recovery point itself:

  • age is a single small package. The comparable apt step in openadapt-capture takes 19s, so 5 minutes is roughly a 15x margin.
  • It leaves 25 of the 30 job minutes for the dump, encryption, checksum, and upload -- the actual backup.
  • A tighter bound strictly dominates the old behaviour. If apt hangs, the old code burned 30 minutes and the backup failed anyway; now it fails in 5 and the failure names the real cause.

Blast radius: only the apt step

age is the encryption tool, so its installation is backup-critical and was treated as such:

  • The package name and flags are identical: sudo apt-get install -y -qq age. The -qq on both calls is preserved.
  • archive.ubuntu.com is the canonical source that azure.archive.ubuntu.com mirrors, so the same suite resolves the same age package. It is the same build it has always been.
  • Untouched: the environment validation, the GitHub environment gate, the AWS OIDC role assumption, the Supabase CLI pin, the source/recipient validation, the three supabase db dump calls, the tar, the age -R encryption, the manifest, verify-artifact, prepare-single-put, the S3 put-object with its SHA-256 and expected-bucket-owner, verify-single-put, and the alert job. git diff touches one step.
  • Three failed update attempts fail the step closed (exit 1) rather than continuing toward a backup that cannot be encrypted.

No other step in this file runs apt. I checked; Install age is the only one.

Tests

db-backup.yml triggers only on workflow_dispatch and schedule, so it does not run on this PR, and I did not dispatch it -- doing so would execute a real backup against the production database. It was verified statically and by simulation instead:

  • actionlint .github/workflows/db-backup.yml -- clean (shellcheck present, so the run blocks were shell-linted too).
  • python3 -c "import yaml; yaml.safe_load(open(F))" -- parses.
  • bash -n on all 6 run blocks in the file -- all parse.
  • The exact Install age script was extracted from the YAML and run under bash -e (matching GitHub's default shell) against a stubbed apt-get/sudo/sleep/sed, in four paths:
scenario result
update succeeds first try install -y -qq age runs, exit 0
update fails 2x then succeeds 2 warnings, install runs, exit 0
update fails 3x 3 warnings, 1 error, exit 1, install never runs
update fine, install fails exit 100 -- set -e correctly restored, failure propagates

No existing check was weakened or removed.

🤖 Generated with Claude Code

The hosted runner resolves its Ubuntu mirror through /etc/apt/apt-mirrors.txt,
which points at azure.archive.ubuntu.com. That mirror fails intermittently and
apt spends minutes of retries before it falls back, so an unbounded package
step can consume a whole run: in openadapt-flow a TeX install ran 3h11m and hit
the run limit, and in openadapt-capture a test job ran over an hour on main.

The dump job has a 30-minute budget but the age install had no bound of its
own, so one hung apt call could spend the whole budget and no recovery point
would be taken that day. Bound the step to 5 minutes and prefer the canonical
archive.

Five minutes is deliberately tight. `age` is a single small package, and a
tighter bound on the install leaves 25 of the 30 job minutes for the dump,
encryption, checksum, and upload, which are the actual recovery point. A slow
mirror now fails this step fast instead of starving the backup.

Only the apt step changes. The dump, the age encryption, the manifest, the
checksum, the S3 upload, and every credential path are untouched, and the
package name and its flags are identical, so the encryption tool is the same
build it has always been. Three failed update attempts fail the step closed
rather than continuing toward a backup that cannot be encrypted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@abrichr
abrichr merged commit 36d7f10 into main Aug 19, 2026
5 checks passed
@abrichr
abrichr deleted the claude/bound-apt-install branch August 19, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant