Overhaul the runner fleet: arm64, 8 replicas, per-replica caches, and hbf support - #3
Open
persello wants to merge 9 commits into
Open
Overhaul the runner fleet: arm64, 8 replicas, per-replica caches, and hbf support#3persello wants to merge 9 commits into
persello wants to merge 9 commits into
Conversation
Select architecture-specific downloads from BuildKit's TARGETARCH so the image builds natively on arm64 (e.g. Apple Silicon via OrbStack) as well as amd64: - GitHub Actions runner: linux-x64 / linux-arm64 - Pkl: pkl-linux-amd64 / pkl-linux-aarch64 TARGETARCH is declared without a default, since a default shadows the value the builder injects and would silently fetch amd64 binaries into an arm64 image. Steps fall back to `dpkg --print-architecture` when it is unset so non-BuildKit builds still resolve the host architecture. Downloads now use `curl -f` so a 404 fails the build instead of writing an HTML error page in place of the binary. The Rust, espup and uv toolchains already resolve their own host architecture, and entrypoint.sh has no architecture assumptions. Verified by building --platform linux/arm64 and running the image: pkl 0.30.1 (native), runner 2.336.0, rustc/cargo-nextest on aarch64-unknown-linux-gnu, and the esp toolchain with xtensa-esp-elf-gcc 15.2.0. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
uv installed itself and maturin under /root/.local, and the image copied
that tree into /home/runner/.local. The copy brought along maturin's
launcher symlink, which points at an absolute path inside
/root/.local/share/uv/tools. /root is mode 0700, so the unprivileged
runner user that actually executes jobs could not traverse it:
$ gosu runner maturin --version
error: exec: "maturin": executable file not found in $PATH
It worked as root, which is why this went unnoticed. Both architectures
were affected.
Install uv into /usr/local/bin and its tools into /opt/uv (via
UV_INSTALL_DIR / UV_TOOL_BIN_DIR / UV_TOOL_DIR) so they sit on the shared
PATH and are readable by every user. This also removes the need to copy
the tree into the runner's home at all.
Verified in the arm64 image: `gosu runner maturin --version` reports
1.14.1 and `maturin list-python` resolves CPython 3.12.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
A runner executes one job at a time, so parallelism is purely the replica count. Default it to 4 and bound what each replica may consume. Sized for a 16-core / 64 GB host: 4 replicas x 4 CPUs x 10 GB, leaving headroom for the host OS. CARGO_BUILD_JOBS is pinned to RUNNER_CPUS because cargo otherwise sizes its thread pool from the host core count, so each replica would spawn ~16 threads and N replicas would oversubscribe the machine N-fold; the cpus limit alone only throttles the result rather than preventing the thrashing. All three knobs are overridable via RUNNER_COUNT / RUNNER_CPUS / RUNNER_MEMORY. Replicas now share a cargo-registry volume instead of each re-downloading the full dependency set. Only the registry is shared, not the whole CARGO_HOME: cargo locks that directory so concurrent access is safe, whereas a shared target/ dir would race. entrypoint.sh repairs ownership of the registry volume when it comes back root-owned, which happens for a volume not seeded from the image and would otherwise silently break every build. Also drops the runner-data volume, which was declared but never mounted. Verified on the arm64 image: compose applies the limits outside swarm (NanoCpus=4000000000, Memory=10737418240) across 4 replicas; the runner user can write to the registry volume both when seeded from the image and after the root-owned repair path; and a crate fetched in one container is served to a second via `cargo fetch --offline`, confirming real sharing. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
Every job in the firmware repo's firmware_ci.yml targets `runs-on: labels: [fw-builder]`. With no labels set, a runner registers successfully and then sits idle forever, since no job ever matches it. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
Observed CI state shows runs queuing while host CPU sits idle, so job throughput is runner-starved rather than compute-bound. A single run only reaches 5 concurrent jobs, but firmware_ci.yml keys its concurrency group per branch, so several runs execute simultaneously and jobs queue globally — more replicas do get used. Memory is the binding constraint: 8 x 6 GB = 48 GB of the ~58 GB the OrbStack VM exposes, leaving host headroom. CPUs are oversubscribed 1:1 (8 x 2 = 16) since jobs spend much of their wall time on network and link steps rather than pegged compute. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
Containers are not long-lived, so $HOME/.cache/sccache in the writable layer is discarded on every recreate and each new container recompiles from cold. setup-rust-dual in the firmware repo configures a 25 GB sccache there and describes it as living on persistent runner storage, so that cache is worth keeping across recreates. It cannot be one shared volume. sccache maintains its LRU index in memory per server process, so several containers pointed at one cache directory evict against each other and corrupt it. Since every replica of a scaled service shares one set of volumes, `deploy.replicas` cannot express per-replica storage — the replicas are now eight explicit services built from a YAML anchor, each with its own sccache volume. The crate registry stays shared, which is safe because cargo locks it. _work is still not persisted: the stale submodule target/ it would preserve is exactly what was breaking builds. The Dockerfile pre-creates the sccache directory so its volume is seeded with runner ownership, and the entrypoint's ownership repair now covers both volume paths — a volume that is non-empty and root-owned is not re-seeded by Docker and would otherwise be unwritable by the runner user. Verified: compose resolves 8 services each with a distinct sccache volume and a shared cargo-registry; the runner user can write the sccache volume when seeded from the image; and with a deliberately root-owned non-empty volume, an unrepaired write fails with EACCES while the entrypoint loop restores ownership and the write succeeds. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
Sharing one cargo-registry volume across the 8 runners introduced a new
CI failure absent from every run before it:
error: could not compile `crc32fast` (lib)
Caused by: could not execute process `.../bin/rustc --crate-name crc32fast
.../registry/src/index.crates.io-*/crc32fast-1.5.0/src/lib.rs`
Caused by: No such file or directory (os error 2)
`could not execute process` appears 0 times across runs predating the
shared volume and immediately after it, with the vanished path inside the
shared registry. Unpacked sources under registry/src are removed
mid-compile when another container's cargo garbage-collects the global
cache, so the rustc spawn fails on a working directory that no longer
exists. Cargo's package-cache lock does not cover a build for its whole
duration, and it cannot arbitrate between separate containers.
Give each runner its own registry volume, matching sccache. This costs N
copies of the crate downloads and removes the only remaining shared
mutable state between concurrently building runners.
Note this is distinct from the pre-existing `could not parse/generate dep
info` failures, which point at a submodule's target/ rather than the
registry and are addressed separately in the firmware repo.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
All eight replicas had been offline for about ten days, which also meant firmware CI could not run at all. The failure was silent: nothing was left running to report it. The runner self-updates in place, and a post-update runner drops a `.runner_migrated` marker beside its config. `config.sh` treats that marker ALONE as proof the runner is already configured -- confirmed by creating only `.runner_migrated` and passing a deliberately bogus token, which fails with "Cannot configure the runner because it is already configured" without even attempting to authenticate. The cleanup here stopped at `.credentials_rsaparams`, so every replica that had auto-updated exited 1 on its next restart. Deleting the marker is correct rather than expedient: this entrypoint always reconfigures from a freshly minted registration token, so no migrated state is worth preserving across a restart. `restart: on-failure:5` turned that per-restart failure into a permanent one -- five retries were spent in seconds, after which Docker left the containers dead. A runner fleet should heal rather than latch off, so it becomes `unless-stopped`. The entrypoint mints one token per start, so even a genuinely broken image loops visibly in the logs instead of failing silently. Default RUNNER_TOKEN and RUNNER_NAME to empty as well. Both are optional when GITHUB_PAT is set, but leaving them unset made `docker compose` print two warnings per service -- sixteen lines that buried the real error underneath. Regression test: plant `.runner_migrated` in a live container, restart it, and confirm the count of "Listening for Jobs" lines increases. Do not test this by grepping `docker logs | tail -N` for that string without counting: a container that booted fine and then broke still has the line in its history, which reports a crash-looping runner as healthy. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf
hbf CI runs on GitHub-hosted runners today and reinstalls its toolchain on every job. Moving it here needs two things the image lacked. Add bun and the Tauri desktop dependencies. `cargo build -p hbf-gui` links against webkit2gtk-4.1 and fails at pkg-config time without the -dev package; librsvg2 and appindicator3 are Tauri's SVG and tray-icon dependencies. bun builds the SvelteKit bundle that `tauri::generate_context!()` embeds at COMPILE time, which makes it a build dependency rather than a test-only tool, and it is pinned to the version hbf CI's `oven-sh/setup-bun` requests so lockfile resolution matches. BUN_INSTALL puts the binary on the shared PATH instead of under /root, which is mode 0700 and so invisible to the unprivileged runner user -- the same trap the uv block already documents. Both layers go AFTER espup deliberately. Docker invalidates every layer below an edited one, and rebuilding the Xtensa toolchain costs many minutes. Verified the `esp` toolchain survived the rebuild untouched. Image grows 8.86 -> 9.6 GB. Add `hbf-builder` to every replica rather than reserving a subset for it. A runner is offered a job only when its labels are a SUPERSET of the job's `runs-on`, so splitting them (1-6 fw-builder, 7-8 hbf-builder) would leave six containers ineligible for hbf work and idle whenever hbf work is all that is queued. Both labels everywhere means any replica serves either repo, and capacity is added by adding replicas. One trap worth recording: `docker compose build` must be run with NO service argument. Each service declares its own `build: .`, so compose tags a separate image per service, and `docker compose build runner-1` silently leaves the other seven on the old image -- they still start and register, so nothing looks wrong until a job needs a tool only the rebuilt image has. hbf CI cannot move here yet: `.github/actions/setup-canvas` downloads pkl-linux-amd64 while these runners are arm64, and it writes to /usr/local/bin, which the runner user cannot do. pkl is also skewed three ways (0.30.1 here, 0.31.1 in hbf CI, 0.32.1 on the dev machine). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine commits that had accumulated locally and were never pushed — the fleet's real configuration existed only on one machine. The last two were written today in response to the fleet having been completely offline.
The outage (3f0f1e6)
All eight replicas had been dead for about ten days, which also meant firmware CI could not run at all. Nothing was left running to report it.
The runner self-updates in place, and a post-update runner drops a
.runner_migratedmarker beside its config.config.shtreats that marker alone as proof the runner is already configured — confirmed by creating only.runner_migratedand passing a deliberately bogus token, which fails withCannot configure the runner because it is already configuredwithout even attempting to authenticate. The cleanup inentrypoint.shstopped at.credentials_rsaparams, so every replica that had auto-updated exited 1 on its next restart.restart: on-failure:5then turned a per-restart failure into a permanent one: five retries were spent in seconds, after which Docker left the containers dead. Nowunless-stopped, so the fleet heals rather than latching off.Also worth knowing: OrbStack itself was
Stopped, and Docker autostarts it on socket access then can drop out mid-command. Checkorb statusbefore diagnosing anything else.hbf support (7adc233)
Adds
bun1.3.14 and the Tauri deps (libwebkit2gtk-4.1-dev,libayatana-appindicator3-dev,librsvg2-dev), and putshbf-builderalongsidefw-builderon every replica.Both labels everywhere is deliberate. A runner is offered a job only when its labels are a superset of the job's
runs-on, so splitting them (1–6fw-builder, 7–8hbf-builder) would leave six containers ineligible for hbf work and idle whenever hbf work is all that's queued. Capacity gets added by adding replicas, not by reserving them.New layers go after espup so a rebuild never invalidates the multi-GB Xtensa toolchain. Verified the
esptoolchain survived untouched. Image grows 8.86 → 9.6 GB.Earlier commits (75ae855…862ab62)
arm64 builds alongside amd64; a fix for maturin being unusable by the runner user; per-replica resource limits and
CARGO_BUILD_JOBS; 8 replicas at 2 CPU / 6 GB; and per-replica sccache and cargo-registry volumes — the shared registry caused real CI failures where unpacked sources underregistry/srcvanished mid-compile when another container's cargo garbage-collected them.Verification
fw-builder,hbf-builder,RestartCount=0..runner_migratedin a live container, restarted, and confirmed the count ofListening for Jobslines increased and the marker was deleted.Two traps recorded in the commit messages
docker compose buildmust be run with NO service argument. Each service declares its ownbuild: ., so compose tags a separate image per service;docker compose build runner-1silently leaves the other seven on the old image. They still start and register, so nothing looks wrong until a job needs a tool only the rebuilt image has. I hit this myself.docker logs | tail -NforListening for Jobs. A container that booted fine and then broke still has that line in its history, so a restart test done this way reports a crash-looping runner as healthy. Count the occurrences and require an increase.Not yet possible
hbf CI cannot move here until:
setup-canvasdownloadspkl-linux-amd64while these runners are arm64; it writes to/usr/local/bin, which the non-rootrunneruser cannot; and pkl is skewed three ways (0.30.1 here, 0.31.1 in hbf CI, 0.32.1 on the dev machine). Unifying pkl touches firmware CI, so it wants its own verification.Issues
No open issues in this repo, so this closes nothing.
🤖 Generated with Claude Code
https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf