ci: bound the apt install and prefer the canonical Ubuntu archive - #133
Merged
Conversation
The hosted runner resolves its Ubuntu mirror through /etc/apt/apt-mirrors.txt, which points at azure.archive.ubuntu.com. That mirror fails intermittently and apt spends minutes of retries before it falls back, so an unbounded package step can consume a whole run: in openadapt-flow a TeX install ran 3h11m and hit the run limit, and in openadapt-capture a test job ran over an hour on main. The dump job has a 30-minute budget but the age install had no bound of its own, so one hung apt call could spend the whole budget and no recovery point would be taken that day. Bound the step to 5 minutes and prefer the canonical archive. Five minutes is deliberately tight. `age` is a single small package, and a tighter bound on the install leaves 25 of the 30 job minutes for the dump, encryption, checksum, and upload, which are the actual recovery point. A slow mirror now fails this step fast instead of starving the backup. Only the apt step changes. The dump, the age encryption, the manifest, the checksum, the S3 upload, and every credential path are untouched, and the package name and its flags are identical, so the encryption tool is the same build it has always been. Three failed update attempts fail the step closed rather than continuing toward a backup that cannot be encrypted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
In
db-backup.yml, jobdumphastimeout-minutes: 30, but the step Install age had no bound of its own.Install age: no timeout -> 5 minutes, plus the mirror preference and a boundedapt-get updateretry.Runner OS for the changed job:
ubuntu-latest. Both jobs in this file (dump,record-alert) areubuntu-latest; onlydumpchanged.Why
This is the production database backup. A job-level timeout is not enough: one hung apt call spends the whole 30 minutes and no recovery point is taken that day. The job fails on a timeout that names the wrong cause.
The hosted runner resolves its Ubuntu mirror through
/etc/apt/apt-mirrors.txt, which points atazure.archive.ubuntu.com. That mirror fails intermittently. apt does fall back to the canonical archive, but it spends the whole budget getting there. Two incidents on 2026-08-19: inopenadapt-flowa TeX install ran 3h11m and hit the run limit, and inopenadapt-capturea test job ran over an hour onmain.Why 5 minutes, not 10
Deliberately tighter than the pattern's default, because here the apt step competes with the recovery point itself:
ageis a single small package. The comparable apt step in openadapt-capture takes 19s, so 5 minutes is roughly a 15x margin.Blast radius: only the apt step
ageis the encryption tool, so its installation is backup-critical and was treated as such:sudo apt-get install -y -qq age. The-qqon both calls is preserved.archive.ubuntu.comis the canonical source thatazure.archive.ubuntu.commirrors, so the same suite resolves the sameagepackage. It is the same build it has always been.supabase db dumpcalls, the tar, theage -Rencryption, the manifest,verify-artifact,prepare-single-put, the S3put-objectwith its SHA-256 and expected-bucket-owner,verify-single-put, and the alert job.git difftouches one step.No other step in this file runs apt. I checked;
Install ageis the only one.Tests
db-backup.ymltriggers only onworkflow_dispatchandschedule, so it does not run on this PR, and I did not dispatch it -- doing so would execute a real backup against the production database. It was verified statically and by simulation instead:actionlint .github/workflows/db-backup.yml-- clean (shellcheck present, so therunblocks were shell-linted too).python3 -c "import yaml; yaml.safe_load(open(F))"-- parses.bash -non all 6runblocks in the file -- all parse.Install agescript was extracted from the YAML and run underbash -e(matching GitHub's default shell) against a stubbedapt-get/sudo/sleep/sed, in four paths:install -y -qq ageruns, exit 0set -ecorrectly restored, failure propagatesNo existing check was weakened or removed.
🤖 Generated with Claude Code