ci: prune third-party apt sources before installing build deps - #519
Merged
Merged
Conversation
The runner image ships Google Chrome and Microsoft apt sources this project does not consume. `apt-get update` exits 100 when any source fails, so once Chrome's index began serving a Packages.gz whose hash did not match, every job died in its first step: pkg-config, mise and sccache never installed and every later step failed with exit 127. Drop the sources this project does not use, then update and install unmasked, so a real failure of the Ubuntu archive or of the packages themselves still fails the job. Ubuntu's own entries are matched by content rather than by filename: the image reaches the archive through the mirrorlist mirror+file:/etc/apt/apt-mirrors.txt, and which file holds them has moved between sources.list and sources.list.d across releases. When nothing matches, the step fails loudly instead of installing from a stale index. Refs #517
The guard kept a whole source file when any line in it matched, which fails in the direction that matters: a deb822 file holding an Ubuntu stanza and a Google Chrome stanza was kept intact, so the broken source survived and the outage was not fixed. A third-party list whose comment merely named an Ubuntu host was kept for the same reason. Classify a source entry at a time instead -- a deb/deb-src line, or a deb822 stanza keyed on its URIs field -- and rewrite a file that carries both rather than keeping it whole. Comments no longer classify anything. Matching is case-folded, so an uppercase URI is still recognised. Read the files through sudo and keep awk's own failures distinct from "this file has no Ubuntu entry": a grep that cannot read a file exits 2, which is falsy, so the previous shape silently classified an unreadable Ubuntu source as third-party and deleted it. Classification now completes for every file before anything is modified, so a file that cannot be read aborts the step with nothing pruned. Removal still unlinks the path itself, so a symlink out of sources.list.d loses only the link. Refs #517
Refusing to install when classification recognised no Ubuntu source was meant to stop the step from working off a stale index, but it introduced a new way to break everything: a re-hosted archive, a renamed mirrorlist or any other runner-image change the classifier does not know would turn a working build into a total outage across all six jobs -- the same failure this guard exists to prevent. Prune nothing in that case and carry on instead, emitting a warning that classification stood down. A classifier miss then degrades to exactly the behaviour this step had before the guard existed, rather than to an outage, and the property the exit was protecting is kept anyway: an archive that is genuinely unreachable, or a package that cannot be installed, still fails loudly at the unmasked update and install. Refs #517
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every CI job whose first step is Install system build deps has been failing on every branch
since 2026-09-09, in that step, before a line of gamut is compiled:
apt-get updateexits 100 when any configured source fails. The failing source is theGoogle Chrome apt repository preinstalled on the GitHub runner image — nothing this
repository configures. Because the step was
sudo apt-get update && sudo apt-get install -y …,that exit aborted the step,
pkg-config/mise/sccache never installed, and every later stepdied with
sccache: command not found(exit 127).This change makes the step depend only on the sources gamut actually consumes. Each of the six
jobs carrying the step (
ci.yml: lint, coverage;extended.yml: benches, real-dng;mutants.yml: incremental, full) now removes the apt sources this project does not use, thenruns
apt-get updateandapt-get installunmasked — no|| true, no--allow-*, noignored exit code. A genuine failure of the Ubuntu archive, or of the packages themselves,
still fails the job exactly as before.
Ubuntu's own sources are recognised by content, not by filename, because the runner image
reaches the archive through a mirrorlist rather than a literal archive URL. From the failing
run's own log (image
ubuntu24/20260907.300, Ubuntu 24.04 noble):So the Ubuntu entries carry
mirror+file:/etc/apt/apt-mirrors.txt, and a naive "keep anythingmentioning
ubuntu.com" filter would have deleted them. The guard keeps any source filematching the mirrorlist or an
*.ubuntu.comhost, insources.listorsources.list.dalike, and fails loudly if that leaves nothing rather than installing from a stale index.
Each kept/removed file is echoed, so every run records what it pruned.
Urgency and merge order
The outage was transient and has since resolved on its own. Runs from about 18:00 on
2026-09-09 are green repository-wide without this having merged — Google's index evidently
stopped serving the mismatched hash. So this pull request is now protection against
recurrence, not a blocker, and there was room to get it right rather than land it fast.
That reframes, but does not remove, the reason to merge it: the step as written on
masterstill takes every job down whenever any third-party source the runner image ships has a bad
index, which is outside this project's control and has no warning. Should it recur before this
lands, branches recover with no code change of their own — re-run their failed jobs:
or push any commit (including a merge of
master), which starts a fresh run.Review follow-up (commits 2 and 3)
6474ab2was reviewed read-only: nothing Medium or above, and the reviewer confirmed bysimulation across 18 fixture layouts that the six blocks are byte-identical, that no trigger,
matrix, permission, concurrency, cache key, job name or step name changed, that the guard
precedes any
apt-get, and that a symlink out ofsources.list.dunlinks only the link. ThreeLow findings were raised and are fixed here:
ba3c223).grep -q … "$f"kept a wholefile when any line matched, which failed in the direction that matters: a deb822 file
holding an Ubuntu stanza and a Google Chrome stanza was kept intact, so the broken source
survived and the outage was not fixed; a third-party list whose comment named an Ubuntu host
was kept too. Classification now works an entry at a time — a
deb/deb-srcline, or adeb822 stanza keyed on its
URIs:field — and rewrites a mixed file rather than keeping itwhole. Comments no longer classify anything, and matching is case-folded. The safe direction
is preserved: a file containing an Ubuntu entry is never deleted.
ba3c223).grepinside anifisexempt from errexit and its status 2 (I/O error) is falsy, so a source the script could not
read was classified third-party and deleted. Sources are now read through
sudo, awk's ownfailures are kept distinct from the "no Ubuntu entry here" signal (exit 10), and classification
completes for every file before anything is modified — so an unreadable source aborts the step
with nothing pruned.
f60a984). Thekept == 0 → exit 1guard wouldconvert a future runner-image change (a re-hosted archive, a renamed mirrorlist) from a working
build into a total outage across all six jobs — a new way to break everything, introduced by a
fix for something that breaks everything. If pruning would leave no Ubuntu source the step now
prunes nothing, emits
::warning::, and proceeds. A classifier miss degrades to exactly thebehaviour this step had before the guard existed, while an archive that is genuinely
unreachable still fails loudly at the unmasked
update/install.Deferred to #518: three of the six copies of this block —
extended.yml's two jobs andmutants.yml's full-workspace job — cannot be exercised by any pull request, so driftbetween the copies is invisible until a
masteror scheduled run. That is the concrete cost thecomposite action in #518 buys down; a workflow-lint gate would also have caught R1 and R2.
Validation
Local, in the worktree:
python3 -c "yaml.safe_load(...)"over all three workflows — pass: each parses, and eachstill exposes exactly two
apt-getsteps under the same job and step names as before.ci.yml'slintstep,/etc/aptrebound to afixture tree,
sudo/apt-getstubbed), run against both follow-up commits — pass,9/9 each:
sources.list, Chrome + Microsoft insources.list.d) →keeps
sources.listbyte-for-byte, removes both;sources.list.d/ubuntu.sources→ kept; literalarchive.ubuntu.com→ kept;Chrome stanza gone and the Ubuntu stanza intact;
archive.ubuntu.com→ removed anyway;ports.ubuntu.com,security.ubuntu.com,deb-src,[arch=…]option blocks and amulti-stanza deb822 file → all survive;
HTTP://ARCHIVE.UBUNTU.COM→ recognised as Ubuntu's;(verified that mode 000 does block reads in the harness environment, so this exercises the
real path rather than a stub);
::warning::emitted,
apt-get updatestill runs, exit 0.list, and
prune=precedesapt-getin every one.mise run check-commits→no errors in 3 commits— pass.Not run, deliberately:
mise run lint/mise run test/mise run fmt-check. This changetouches no Rust and nothing rustfmt sees.
The acceptance test is this pull request's own CI run, and it passed. The step can only get
past its first minute if
pkg-confignow installs, so a job that goes on to compile theworkspace is direct proof the outage is fixed. Observed on this branch at
6474ab2(CI run
34386394665, Mutants run34386394668):These are the results for
6474ab2. The two review-follow-up commits were re-run and therevised guard passes the same way, at
f60a984(CI run34390387549, Mutants run34390387313):So the per-entry classifier, the read-failure handling and the stand-down all survive a real
runner:
Clippy & Doctestsagain spends eighteen minutes compiling the workspace, which is onlyreachable through a successful
apt-get installbehind the rewritten guard.Six of the seven required checks are terminal passes. Before this change the same jobs died in
under a minute inside Install system build deps;
Clippy & Doctestsalone now spends twentyminutes actually compiling the workspace, which is only reachable through a successful
apt-get install.Coverage (test gate)runs the instrumented workspace build and legitimatelytakes about three hours (it ran 2h59m on #485 before the outage), so it had not reported yet; it
shares the identical step with the jobs that already passed, so it is not a blocker for this
claim.
Risks and rollout
ships). Nothing in these jobs
apt-get installs from them; every other tool arrives via mise,dtolnay/rust-toolchain,mozilla-actions/sccache-action, or a vendored submodule. Therunner VM is ephemeral, so the removal does not outlive the job.
*.ubuntu.comhosts, the guardfails the step with an explicit message. That is the deliberate trade: a loud failure beats a
silent install from a stale index.
sudo apt-get update && sudo apt-get install -y ….pkg-configfrom mise instead of apt) was not taken:mise.tomlstill records thatpkg-config"lacks a clean mise backend", and nothing foundhere overturns that.
anchors and a composite action would live under
.github/actions/, outside this change'smanifest.
Issue
Closes #517.
Decisions taken
Unresolved review notes
None.