ci(fixture-vm): make the nightly actually run — lock, qcow2 attribute, git-dep hash (#362, #365) - #364
ci(fixture-vm): make the nightly actually run — lock, qcow2 attribute, git-dep hash (#362, #365)#364avrabe wants to merge 18 commits into
Conversation
…out (#362) Trace-Topology Fixture Generation has failed 60 of 60 runs, 2026-06-01 through 2026-07-30, without ever passing. The cause is not a defect — it is `trace-fixtures.yml:58` doing exactly its job: ##[error]tools/fixture-vm/flake.lock is missing. ##[error]The nightly build runs only against a committed, ##[error]reviewed lock. The guard is fail-closed by design: the nightly builds ONLY against a committed lock and never re-resolves inputs itself, so the lock is the determinism pin for the guest kernel and the whole closure. It has simply never been satisfied — `tools/fixture-vm/` has held only `flake.nix` since it landed. Produced by the documented procedure: dispatch `fixture-vm flake.lock` (run 30560238740, `success`), download its artifact, commit it here. The pin resolves `flake.nix:70`'s declared `github:NixOS/nixpkgs/ nixos-24.05` to rev b134951a4c9f3c995fd7be05f3243f8ecd65d798. Verified against upstream independently of the container that produced it — that rev exists in NixOS/nixpkgs and is the current tip of nixos-24.05. So this commit chooses nothing; it records what the flake already declares. That tip is dated 2024-12-30. nixos-24.05 stopped moving 19 months ago and is EOL. Bumping to a supported channel is a real decision and is deliberately NOT taken here: it would change the guest kernel, and the guest kernel is what the generated trace fixtures are a function of. Raised for human judgement rather than folded into a red-to-green fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verification result — I said I'd dispatch and report what actually happensI did, and the prediction in the PR body ("it is entirely possible this reveals a Run 30560720500 on The good news is real progress. The fail-closed guard at
Filed with a full diagnosis and a patch sketch checked against the pinned What I changed here as a result
Why I am still proposing to merge itThree reasons, and I'd rather they be judged explicitly than assumed:
The alternative — hold this until #365 is also fixed — bundles a reviewed Standing caveat, unchangedBeyond flake evaluation lie the guest build, the 🤖 Generated with Claude Code |
#365) The nightly builds `.#nixosConfigurations.fixture-vm.config.system.build.qcow2`, but nothing ever defined that attribute. It existed only in this file's header comment and in the workflow's build command. `system.build` is a freeform `lazyAttrsOf raw`, so a missing member is not a typo the module system can catch — it surfaces as `error: attribute 'qcow2' missing` at evaluation, before a single byte is built. That was invisible until #364 committed the flake.lock. The workflow's "Require committed flake.lock" step is fail-closed, so all 60 prior runs died there and nix was never invoked at all. Clearing the outer blocker exposed this inner one for the first time in the workflow's life. Defining it exposed a third blocker: `etch` (pulseengine/rivet, pinned in root Cargo.toml) is the workspace's one git dependency, and git deps carry no checksum in Cargo.lock, so importCargoLock cannot vendor them without an explicit hash. It is vendored even though gen-fixtures never uses it — importCargoLock takes the whole lock file, not the subtree that `cargoBuildFlags` selects. Verified locally, not merely reasoned about. `nix build` needs x86_64-linux, but all three failures are *evaluation* errors and evaluation is platform-independent, so the whole config can be forced on darwin: cd tools/fixture-vm nix eval --raw --no-update-lock-file \ '.#nixosConfigurations.fixture-vm.config.system.build.qcow2.drvPath' now exits 0 and prints a .drv path, warning-free. The etch hash was cross-checked two-sidedly against the *pinned* 24.05 fetchgit rather than trusted from the unstable prefetcher that produced it: the correct hash yields a store path, and a deliberately wrong one errors with `got: sha256-x37urQw...`, re-deriving the same value independently. This does NOT prove the image builds or that the guest boots. Those are x86_64-linux build-time and run-time concerns that only CI can settle. Also drops a rename warning by moving boot.loader.grub.timeout to the bootloader-agnostic boot.loader.timeout, so the build log reads clean and a real warning would stand out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
avrabe
changed the title
Dispatch result: it failed. Blocker #4, and it invalidates a claim in this PR.Run 30562393849 → The diagnosis, and what it is notThis is not an The real fault is one line at the top of nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";Measured, not assumed —
So the flake has been pinned to a toolchain that cannot compile this workspace Why my local eval did not catch it, which is the pointIt couldn't, and I said so before dispatching:
That boundary held exactly. Evaluation is platform-independent, so blockers 1–3 The useful generalisation: a cheap oracle is worth having only if you state Running count
Four deep, each invisible behind the last. Pushing the pin bump next; the I am not flipping "Addresses" to "Closes" and not merging until a dispatch is |
Rivet verification gate✅ 20/20 passed
Filter: Failed artifacts(none) Updated automatically by |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…on 2024
Dispatch 30562393849 failed at build time, four blockers deep:
building '...etch-0.2.0.drv'...
error: builder failed with exit code 101
> feature `edition2024` is required
> ...not stabilized in this version of Cargo (1.77.1).
This is not an `etch` problem and dropping the dependency would not help.
`etch` is simply the first derivation cargo reached; all 23 spar crates
inherit `edition = "2024"` from [workspace.package], so gen-fixtures itself
fails identically a few derivations later.
The fault is the nixpkgs pin, which was doing two unrelated jobs: it defines
the guest OS *and* supplies rustPlatform. Nothing in the flake signalled the
second role, so "pin a stable channel for a reproducible guest" silently also
meant "pin a compiler from December 2024". Measured rustc.version per channel:
nixos-24.05 -> 1.77.2 cannot compile this workspace
nixos-24.11 -> 1.82.0 cannot compile this workspace
nixos-25.05 -> 1.86.0 ok (edition 2024 stabilised in 1.85)
So this build was never possible, which is consistent with a workflow that has
never once succeeded.
Verified against the instantiated derivation rather than inferred from
pkgs.cargo: the gen-fixtures drv now takes cargo-1.86.0.drv and
rustc-wrapper-1.86.0.drv as direct inputs. Evaluation of the qcow2 drvPath is
clean and warning-free at the new pin.
The etch outputHashes value was re-derived under 25.05 rather than carried
over, because fetchgit's defaults could differ between channels and that
failure appears only at build time. Two-sided: 25.05's fetcher independently
reports got: sha256-x37urQw97R/ARqvlVpXpp3tJqbvztbOiUyAGNZItlA0=, the same
value. Note the positive side alone proves nothing here — a fixed-output
derivation short-circuits when the path is already in the store, so only the
deliberately-wrong-hash run actually exercises the fetcher.
Still unproven locally, and only CI can settle it: that the image builds and
that the guest boots.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e edition floor
Run 30563612749 got further than any previous fixture-vm run and then died
inside the derivation:
error: rustc 1.86.0 is not supported by the following package:
smol_str@0.3.6 requires rustc 1.89
This falsifies a claim I wrote into 1ea73b1. That commit moved the pin
24.05 -> 25.05 and recorded, in the flake itself:
nixos-25.05 -> 1.86.0 [OK] cannot compile -> can compile this workspace
That was wrong, and wrong in an instructive way. Edition 2024 requiring
>= 1.85 is a NECESSARY condition that I mistook for THE condition. The
probe genuinely ran and genuinely returned 1.86.0, so the check felt like
verification while silently scoping the claim to the one variable I had
thought of.
The actual constraint is
max(rust-version) over the entire RESOLVED DEPENDENCY CLOSURE
which is an emergent property of Cargo.lock, not of this repo: the
workspace declares no `rust-version` and carries no rust-toolchain.toml,
so nothing states the floor and nothing pins it. Derived mechanically
rather than guessed a second time -- 152 of 235 packages declare one; the
max is 1.89 (smol_str 0.3.6), then 1.87.0 (wasip2, wit-bindgen):
cargo metadata --format-version 1 --locked \
| jq -r '.packages[] | select(.rust_version) | .rust_version' \
| sort -V | tail -1
nixos-25.11 ships rustc 1.91.1 and clears both floors. Confirmed against
the INSTANTIATED derivation, not against `pkgs.rustc`: the
gen-fixtures-0.10.0.drv reached by the disk image now references
cargo-1.91.1.drv and rustc-wrapper-1.91.1.drv.
The etch outputHash was re-derived under the new channel rather than
assumed to survive it, since a fetchgit-defaults change would evaluate
fine and fail only at build time. Checked from the side that cannot pass
vacuously -- a deliberately wrong hash forces a real fetch and reports
`got: sha256-x37urQw97R/ARqvlVpXpp3tJqbvztbOiUyAGNZItlA0=`, identical to
the pinned value. (The happy path proves nothing here: a fixed-output
derivation short-circuits when the store path already exists.)
The comment now carries the derivation command instead of a number to
trust, and records how the earlier check failed. Note the floor MOVES ON
EVERY `cargo update` -- a dependency bump can raise it with no change to
this repo, and nothing gates that today; it surfaces only as a red
nightly here.
This still does NOT prove the image builds or that the guest boots.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dispatch result: failed. Blocker #5, and it falsifies a claim I put in this branch.Reporting the actual outcome as committed, not merging on a green evaluation. Run 30563612749 This makes a sentence in
|
| channel | rustc | verdict |
|---|---|---|
| nixos-24.05 | 1.77.2 | ✗ edition 2024 not stabilised |
| nixos-24.11 | 1.82.0 | ✗ likewise |
| nixos-25.05 | 1.86.0 | ✗ clears the edition, fails the closure |
| nixos-25.11 | 1.91.1 | ✓ clears both |
Pushed as aef79c2: pin → nixos-25.11.
Verified against the instantiated derivation, not pkgs.rustc
Same discipline as blocker #4 — the question is what the toolchain reaching
gen-fixtures is, and pkgs.rustc is not that. The gen-fixtures-0.10.0.drv
reachable from the disk image now references cargo-1.91.1.drv and
rustc-wrapper-1.91.1.drv.
The etch outputHash was re-derived under the new channel rather than
assumed to survive it — a fetchgit-defaults change between channels evaluates
fine and fails only at build time. Checked from the side that cannot pass
vacuously: a deliberately wrong hash forces a real fetch and reports
got: sha256-x37urQw97R/ARqvlVpXpp3tJqbvztbOiUyAGNZItlA0=
identical to the pinned value. The happy path proves nothing here — a
fixed-output derivation short-circuits when the store path already exists, so
"it built" would have been true whether or not the hash was still correct.
The blocker ladder so far
| # | Blocker | Surfaced |
|---|---|---|
| 1 | flake.lock absent; the guard is fail-closed |
60/60 runs, never got past it |
| 2 | system.build.qcow2 never defined |
run 30560720500, first run to reach nix |
| 3 | cargoLock.outputHashes missing for the etch git dep |
local eval |
| 4 | pinned channel's cargo predates edition 2024 | run 30562393849 |
| 5 | closure MSRV 1.89 > channel's rustc 1.86 | run 30563612749 |
Each was invisible behind the one before it. Five deep is not bad luck — it is
what a fail-closed outer guard does to a pipeline nobody has ever run to
completion. Every one of these has been sitting there since the workflow was
written; none of them is a regression.
Known-unfixed, stated so it is not mistaken for coverage
The closure floor moves on every cargo update. A dependency bump can raise
it with no change to this repo and no change to this flake, and nothing gates
that — it surfaces only as a red nightly here, at build time, which is exactly
how blocker #5 got in. The comment in flake.nix therefore carries the
derivation command rather than a number to trust. A real fix is either a
declared rust-version in [workspace.package] or a CI check comparing the two;
both are out of scope for this PR and I would rather name the gap than imply it
is closed.
Re-dispatched as run 30564693866.
Still Addresses, not Closes, and still not merging until a dispatch is
actually green.
🤖 Generated with Claude Code
Blockers #6 and #7 in the same build script, five levels below anything this image is for: spar-trace-topology -> spar-network -> good_lp -> highs -> highs-sys Run 30564693866 got further than any of the previous 60: the 25.11 pin cleared the closure MSRV, every vendored crate compiled, and the build died inside highs-sys' build script with is `cmake` not installed? highs-sys compiles the HiGHS C++ LP solver from vendored source. gen-fixtures never calls a solver — but `--bin gen-fixtures` selects what gets LINKED, not what gets COMPILED, and cargo builds the dependency graph of the whole `-p spar-trace-topology` package. Same shape as the etch vendoring already documented here: the lock file decides what must build, not the binary. `dontUseCmakeConfigure = true` is load-bearing rather than defensive. Adding cmake to nativeBuildInputs makes nixpkgs' setup-hook install cmakeConfigurePhase as this package's configurePhase — it guards on exactly this variable at setup-hook.sh:145, checked in the pinned tree, not from memory — and it would then try to cmake a Rust workspace root that has no CMakeLists.txt. BLOCKER #7 WAS PREDICTED, NOT DISCOVERED ---------------------------------------- Rather than dispatch and wait for the next red run, I read the graph. highs-sys also build-depends on bindgen 0.71, which reaches libclang via clang-sys -> libloading, i.e. it dlopen()s libclang.so while the build script runs. Nothing in the sandbox provides it, so "Unable to find libclang" was queued up to surface the moment the cmake fix cleared the way. `rustPlatform.bindgenHook` exports LIBCLANG_PATH and BINDGEN_EXTRA_CLANG_ARGS and nothing else here sets them. Verified present at the pinned rev. This is the pattern that made this workflow cost 60 red runs to diagnose: each blocker was invisible behind the one before it, so every dispatch bought exactly one bit of information. clang-sys and highs-sys are the only two -sys crates in the graph, so this should close the native-dependency species. WHAT IS STILL UNVERIFIED ------------------------ Evaluation is clean and platform-independent, and it says nothing about whether the image builds or the guest boots. Both are x86_64-linux concerns only CI can settle. "The graph says the dependency exists" is not "the fix is sufficient" — a clean eval has been wrong about this build three times now. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dispatch result: further than ever, still red — blockers #6 and #7Reporting run 30564693866 The 25.11 pin worked. Cargo accepted the closure (no MSRV complaint), every Blocker #6. Note
Blocker #7 was predicted, not discoveredThe thing that has made this PR expensive is that each blocker was invisible
Running tally
Still unverifiedEvaluation is clean, and evaluation is platform-independent — it says nothing Unchanged commitment: still |
The Rust half now builds (blockers #6/#7 closed — gen-fixtures-0.10.0 compiles). Run 30565973744 got all the way to the disk image and died on: error: a 'x86_64-linux' with features {kvm} is required to build '...-nixos-disk-image.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, nixos-test, uid-range} That set is Nix's default minus `kvm`. Nix adds `kvm` only when access("/dev/kvm", R_OK|W_OK) succeeds, and make-disk-image.nix builds through vmTools.runInLinuxVM, whose derivation requires the feature. `--device /dev/kvm` was already passed, and the runner user is in the `kvm` group on the host — but /dev/kvm is root:kvm 0660 and rootless podman drops supplementary groups by default, so the in-container process is in no such group. The device is present and unopenable. `--group-add keep-groups` carries them through. Also add a precondition step that asks Nix for its computed system-features before anything is built. The existing diagnostic was correct and precise; it was just emitted after ~5 minutes of compiling and buried under ~700 `building '/nix/store/...drv'` lines. Same fact, two seconds in, at the top of the log. The probe asserts on Nix's own feature set rather than a shell `[ -w /dev/kvm ]` test — the feature set is what the build is matched against, so a probe of anything else could green while the build fails. The image digest moves to workflow `env:` so the probe and the build it guards cannot drift onto different images. Refs #362, #365. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dispatch result: the Rust half is done — blocker #8 is the disk imageRun 30565973744 It died one layer further out, in the disk image itself: Blocker #8. That feature set is Nix's default minus The device was already being passed. The diagnostic existed; nobody could reach itNix's error is correct, precise, and names the exact missing feature. It was So this commit also adds a precondition step that asks Nix for its computed Status of the new dispatch Running tally
Eight blockers, each invisible until the one before it cleared — the cost of a Still |
Blocker #9. `Locate qcow2` failed on a tree where the image had just been built successfully, which reads like a naming mismatch and is not one. `nix build --out-link /spar/tools/fixture-vm/result` wrote the symlink into the bind-mounted repo, pointing at `/nix/store/…`. That container is `--rm` with its own store; nothing of it exists on the host after the run. The symlink survived, its target did not — the build product was unreachable from the host by construction, so no amount of globbing for a differently named file would have found it. The out-link now goes to /tmp and the image is copied across the mount while the store it lives in is still alive. `Locate qcow2` also stops trusting `cp`'s exit code. A copy that exits 0 says a file arrived, not that a bootable NixOS image did; a truncated or wrong-path copy would have surfaced three steps later, at `Run fixture-vm`, far from its cause. The step now asserts existence and a >= 64 MiB floor (a full NixOS qcow2 is hundreds of MB) and names what went wrong. Neither `nixos.qcow2` nor `result` was ever gitignored. That stayed harmless only because the build had never once got far enough to produce them. Evidence blocker #8 is closed, from the probe's own output in run 30567928227 rather than from the step's exit code: container process: uid=0(root) gid=0(root) groups=0(root),65534(nobody) crw-rw---- 1 nobody nobody 10, 232 Jul 28 04:16 /dev/kvm nix system-features = benchmark big-parallel kvm nixos-test uid-range ok: nix advertises kvm; nixos-disk-image.drv is buildable here The host `kvm` group arrives in the userns as 65534, matching the device's mapped group — which is exactly what `--group-add keep-groups` carries and what its absence dropped. `building '…-nixos-disk-image.drv'` then ran to completion in 13 minutes. Refs #362 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Blocker #8 is closed — measured, not inferred. Blocker #9 is open.Reporting the outcome as committed. Run I said I would read the probe's log rather than infer "keep-groups worked" from The host Blocker #9 —
|
| # | blocker | closed by |
|---|---|---|
| 1 | nightly refuses to run without a committed flake.lock |
f18be8d |
| 2 | system.build.qcow2 is not a stock NixOS option |
1f5ef9a |
| 3 | etch git dependency hash |
1f5ef9a |
| 4 | nixpkgs 24.05's cargo predates edition 2024 | 1ea73b1 |
| 5 | closure MSRV is 1.89, above 25.05's rustc 1.86 | aef79c2 |
| 6 | highs-sys needs cmake |
a33d0d6 |
| 7 | highs-sys needs libclang |
a33d0d6 |
| 8 | rootless podman drops the kvm group |
52fafa0 ✅ verified green |
| 9 | the qcow2 never leaves the container's store | dccc117 |
Dispatched 30569595760.
Still Addresses, not Closes, and still not merging until a dispatch is
actually green. Four steps have still never executed: boot the guest, tshark
the capture, round-trip all four fixtures, upload. Each is a fresh opportunity
for blocker #10. I will report the outcome either way.
Run 30569595760 reached `Run fixture-vm` — a step that had never executed before — printed SeaBIOS, iPXE and "Welcome to GRUB!", then produced nothing at all for 900 seconds and hit the harness timeout. That silence is not the guest hanging at GRUB. It is the point where we went blind. The host side was never wrong: the harness runs `-nographic -serial mon:stdio` and inherits stdout. What survived it was only *firmware* output. Under `-nographic` SeaBIOS redirects the BIOS text console onto the serial port, and GRUB draws through BIOS calls, so GRUB rode that redirect for free. Linux does not use BIOS calls — at handoff it switches to whatever `console=` names, and the default is `tty0`, the VGA framebuffer `-nographic` has disconnected. The kernel, systemd, and gen-fixtures have all been logging into a void. flake.nix set boot.kernelPackages, boot.kernelModules and boot.loader.grub, but never boot.kernelParams. Adding it: boot.kernelParams = [ "console=tty0" "console=ttyS0,115200" ]; Order is load-bearing. Every `console=` receives kernel messages, but the LAST one becomes /dev/console — which is what init and the already-enabled `serial-getty@ttyS0` attach to. ttyS0 must come last or the getty lands on the framebuffer again. This does not by itself prove the guest completes. It makes the next failure legible instead of a 900-second blank, which is the difference between debugging and guessing — the same shape as every other blocker in this series: an operation that produced "nothing happened" rendered identically to one that worked. Refs #362, #365 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reporting the outcome, as committed: the dispatch was not green.Run 30569595760 What passed — including new ground
The qcow2 byte count I said I'd read back once logs were servable: 1,995,309,056 bytes — 1.9 GiB, about 30× the 64 MiB floor that step Step 11 is a step that had never executed in this workflow's history. So The silence is not the guest hanging at GRUB — it is where we go blindThe obvious read is "the guest wedged right after GRUB." I don't think that's The host side is correct and was never the problem: the harness runs Linux does not use BIOS calls. At handoff it switches to whatever "Welcome to GRUB!" is not where the guest stopped. It is the last line we were The fix (8f581aa)boot.kernelParams = [ "console=tty0" "console=ttyS0,115200" ];Order is load-bearing: every What this does and does not claimIt does not claim the guest now completes. I have no evidence for that and That is the same shape as every other blocker in this series, and worth naming Still not merging until a dispatch is actually green. Re-dispatching on 8f581aa |
|
Dispatch outcome on It failed earlier than the commit was written for. What actually happened
Why the KVM precondition passed anywayThis is the part worth writing down, because the precondition step was added specifically to stop this class of failure, and it reported green two minutes earlier: Both readings are correct. The build still cannot open the device, because the process that answered the probe is not the process that runs the build. Nix computes The step's own comment defends the design:
That is right about scheduling and blind to execution. Same shape as every other blocker in this series: an operation that produced "nothing happened" — here, "no KVM for you" — rendered identically to one that worked. What I am changing
Note that TCG is not a usable fallback here regardless — it aborts. So KVM has to genuinely work in the builder; failing loudly at the top is the only correct behaviour. I will report the next dispatch outcome either way, as before. |
Run 30572572863 passed the KVM precondition and then failed the build it
guards:
probe: nix system-features = benchmark big-parallel kvm nixos-test …
build: qemu-system-x86_64: Could not access KVM kernel module:
Permission denied -> falling back to tcg -> SIGABRT (134)
Both readings were correct. /dev/kvm is mode 0660 owned by the host's
root:kvm, neither of which is mapped into the rootless userns — so both
render as 65534/nobody, and access is granted purely by the supplementary
group `--group-add keep-groups` carries in. The container's uid 0 has it:
container process: uid=0(root) gid=0(root) groups=0(root),65534(nobody)
crw-rw---- 1 nobody nobody 10, 232 /dev/kvm
Nix computes `system-features` in that parent process, so access() there
succeeds and `kvm` is advertised. It then runs the builder as a nixbld
build user, which carries no such group, and QEMU's open(O_RDWR) got
EPERM. Scheduling was never the problem; execution was.
The step's own comment defended reading `system-features` as strictly
better than `[ -w /dev/kvm ]`, "the feature set is what the build is
actually matched against". That is true and it is the wrong question.
`system-features` decides whether the derivation may be PLACED here; the
failure was in whether its builder may OPEN the device. Different
principal, different syscall. `[ -w /dev/kvm ]` would have been wrong the
same way, because it too runs as the parent. Neither probe can speak for
the build user, so the probe now IS a build.
tools/fixture-vm/kvm-probe.nix is a dependency-free derivation shaped like
the thing it stands in for: `requiredSystemFeatures = [ "kvm" ]` so Nix
schedules it through the same path as nixos-disk-image.drv, and a builder
that performs the actual open(3<>/dev/kvm). It prints `id` first — the one
line whose absence made this cost twelve minutes and a 700-line log to
see. Verified by instantiation, not by eye: builder /bin/sh, system
x86_64-linux, requiredSystemFeatures=kvm, script text intact through Nix's
'' quoting.
Two supporting changes:
* NIX_CONF_EXTRA hoists the nix.conf settings into workflow `env:`, shared
verbatim by probe and build. They had drifted — the build appended
experimental-features and sandbox=false, the probe appended nothing. This
workflow already hoisted NIX_IMAGE_DIGEST on exactly this reasoning ("a
probe that greens a different image than the one built proves nothing");
differing Nix settings have the same defect.
* `build-users-group =` makes builds run as the container root that
keep-groups actually granted, rather than dropping to nixbld.
The second is a hypothesis, not a finding: I confirmed root can open the
device and the builder cannot, but not that the image sets
build-users-group=nixbld. The new probe settles it either way in seconds
and prints build-users-group alongside the builder's id, so if the guess is
wrong the next run says so precisely instead of failing blind twelve
minutes in.
TCG is not a usable fallback here — with KVM denied QEMU fell back to
software emulation and aborted on an assertion at -smp 32. There is no
degraded mode to tolerate, which is why this stays a hard gate.
This does not by itself prove the guest completes; it removes the blocker
in front of the boot fix landed in 8f581aa, which this run never reached.
Refs #362, #365
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Follow-up: the nixbld${toString n} = { uid = 30000 + n; gid = 30000; groups = [ "nixbld" ]; };
groups = { root.gid = 0; nixbld.gid = 30000; nobody.gid = 65534; };
defaultNixConf = { sandbox = "false"; build-users-group = "nixbld"; … };Three things fall out. 1. The diagnosis is confirmed. The image ships 2. The daemon caveat is refuted. I flagged that if the image ran a nix daemon, appending to 3. An incidental correction. It also explains a cosmetic detail from the failing run's log that looked like noise:
Run 30575158798 is still queued behind the single podman runner, and I will report it either way. But note what its job now is: it is no longer the experiment that decides the diagnosis — that is decided. It is the confirmation that the fix works end to end, and the first chance the boot fix in Still not merging until a dispatch is actually green. |
The NIX_CONF_EXTRA comment claimed `sandbox = false` was required for the
build to work. It is not: the nixos/nix image already ships it. Verified
against the image's own definition rather than inferred — NixOS/nix
docker.nix at 2.24.9 (the version this digest resolves to, per the note in
the run step) defines:
defaultNixConf = {
sandbox = "false";
build-users-group = "nixbld";
...
};
So of the two settings we append, `sandbox = false` is a no-op today and
`build-users-group =` is the one doing the work — it overrides the image's
`build-users-group = nixbld`, which is what keeps the supplementary group
`--group-add keep-groups` carries in, and therefore what makes /dev/kvm
reachable at BUILD time.
The setting stays. NIX_IMAGE_DIGEST is a pin we will bump, and a bump must
not be able to silently flip a default this build depends on; stating the
assumption explicitly is the point. Only the claim about it changes.
Also de-duplicates the second copy of the explanation in the run step,
which is where the wrong claim would otherwise survive the fix — it now
points at NIX_CONF_EXTRA as the single source instead of restating it.
No behaviour change: comments only. NIX_CONF_EXTRA parses byte-identically
(experimental-features / sandbox / build-users-group, empty value intact).
Refs #362
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reporting run 30575158798, as promised — it failedI said I would report it either way and would not merge until a dispatch is
What the run does establish
Step 4 is the fix in Step 5 then ran for 5m21s and built several hundred derivations — the whole
Why it failed — blocker #12, and it is caused by
|
Run 30575158798 got 5m21s into the NixOS graph and then died with error: home directory '/homeless-shelter' exists; please remove it to assure purity of builds without sandboxing Nix runs builders with HOME=/homeless-shelter and refuses to START any build if that path exists, once the sandbox is off — the check sits in local-derivation-goal.cc behind `!useChroot`, so it is a direct consequence of the `sandbox = false` this container relies on. That error names the derivation that TRIPPED over the directory, never the one that made it. Attribution here is by timing: the error fired 5m21s after gen-fixtures started with no intervening `building '…'` line, and the check runs at build START, so the directory was created inside that window. Which tool inside it wrote there is inferred, not measured (cmake's $HOME/.cmake/packages registry is the usual culprit and cmake is on this build's PATH for highs-sys). The fix does not rest on that inference: redirecting HOME covers whichever tool it was. Two alternatives rejected. `rm -rf /homeless-shelter` in the workflow cannot work — the directory appears mid-graph, so clearing it before `nix build` leaves the failing window untouched. Enabling the sandbox would also fix it and may be the better end state, but it would make kvm-probe.nix's out-of-store `builder = "/bin/sh"` illegal and its interaction with nested userns under rootless podman is untested; not bundling that into a dispatch already carrying two other fixes. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The probe exists because the old KVM precondition measured the parent process instead of the build principal. Its own header claimed "it prints `id` first, so the log names that principal outright" — and in run 30575158798 that line printed: sh: line 2: id: command not found builder principal: Nix clears PATH for builders and this derivation is deliberately dependency-free, so there is no coreutils. `echo` still exited 0, so `set -eu` did not catch it. The assertion itself was fine — the real `exec 3<>/dev/kvm` succeeded from the real build principal, which is what the probe is for — but the line whose entire job was to be EVIDENCE rendered "nothing happened" identically to "it worked". That is the same defect the probe was written to fix, one level inside it. Read /proc/self/status with shell built-ins only, and exit non-zero when the read comes back empty: a probe that cannot name the principal it tested produces an unattributable result, which is worse than no result. A diagnostic line is only evidence if its absence fails the build. Also records what the Groups: line is actually for. `build-users-group =` (empty) makes Nix run builders as the invoking user rather than dropping to a nixbld account, which is precisely why the supplementary group that `--group-add keep-groups` carries in survives into the build — and therefore why /dev/kvm is openable at all. Printing Uid/Gid/Groups makes that mechanism visible instead of inferred. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nix throws the impure-HOME error when a build STARTS, so it names the derivation that tripped over the directory and never the one that created it. Working out the creator for run 30575158798 meant reading timestamps against the build graph. Whatever a builder dropped in there is a fingerprint of the tool that wrote it — a .cmake/packages entry, a .cargo, a .npm — so one listing turns that inference into a reading. It has to happen inside the same `sh -c`: the container is --rm with its own store, so a later step has nothing left to look at. The exit status is captured as `nix build … || rc=$?`, not from inside an `if ! …; then rc=$?` block. In the latter, $? is the status of the negation — which SUCCEEDED — so rc would be 0 and the failure path would exit green. Verified by execution rather than by reading: with a stubbed `nix` exiting 42, the fragment exits 42 with the shelter absent, exits 42 and prints the tree with it present, and still reaches the copy when the stub exits 0. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dispatch re-run: red. Not merging. Here is what moved and what did not.Run 30579194737 on I said on this PR: "Both fixes go on this branch and I re-dispatch. Not merging on What the three fixes actually bought
Each fix is discharged by evidence, not by the run being green:
The failure moved roughly six minutes deeper into the build graph. That is the The new failure is a different one
and then, inevitably: The socket refuses because nothing ever bound it. This is the same species as fixture-vm blockers #8 and #11, one axis over. What happens nextThe fix goes on this branch, with a probe that asserts the capabilities are Nothing is blocked by holding this open: 🤖 Generated with Claude Code |
Run 30579194737 got six minutes further than 30575158798 and then died in
nixos-disk-image.drv. The visible line was
qemu-system-x86_64: -chardev socket,id=store,path=virtio-store.sock:
Failed to connect to 'virtio-store.sock': Connection refused
which reads like a socket-path or ordering bug. It is neither. Three lines
above it, twice:
ERROR virtiofsd] can't apply the child capabilities: failed to sync
capabilities with the kernel
virtiofsd is what shares the Nix store into the transient VM that
nixos-disk-image.drv boots. It raises a fixed capability set for its child
with capng_apply(BOTH); raising one that is not already permitted returns
EPERM, so it exits before binding the socket. The refused connection is the
absence of a listener, not a connection problem.
Two of virtiofsd's required capabilities are absent from podman's rootless
default set: MKNOD and DAC_READ_SEARCH. PODMAN_CAPS grants exactly those,
hoisted into the workflow env for the third time and for the same reason as
NIX_IMAGE_DIGEST and NIX_CONF_EXTRA -- the precondition probe has to run
under the same grant as the build it guards. They are confined to the
rootless user namespace and give the container nothing on the host the
invoking user did not already have.
The assertion goes in the BUILD principal, not the container. virtiofsd is
spawned by a Nix builder, so the builder's capability set is the one that
decides -- and testing the container's instead would repeat the mistake this
probe was written to fix, one axis over: /dev/kvm was about the principal's
groups, this is about the same principal's capabilities.
Only the two capabilities the workflow grants are asserted. Both masks are
printed -- the container's from the step, the builder's from the probe --
so a capability missing for some other reason is readable from the log
instead of guessed at from a list of podman defaults recalled from memory,
and so that a grant lost between podman and the builder is distinguishable
from a grant never made. Asserting a capability nobody has shown to be
required would only be a second way to be confidently wrong.
Verified locally by executing the emitted builder script against constructed
/proc/self/status fixtures, not by reading it:
* podman rootless default mask (00000000800405fb) -> exit 1, naming both
DAC_READ_SEARCH and MKNOD
* the same mask with bits 2 and 27 set -> exit 0
* CapEff absent entirely -> exit 1, rather than expanding
to a bare 0x and dying with a parse error that names neither the file
nor the cause
Also checked: the workflow YAML parses, all nine step scripts pass bash -n,
both sh -c bodies pass sh -n and contain zero apostrophes, and both .nix
files parse.
kvm-probe.nix is renamed to build-principal-probe.nix because it no longer
probes only KVM, and a file whose name misdescribes what it measures is the
exact defect this branch keeps fixing.
Refs #362, #365
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eeds
Run 30580839772 got all the way to `<<< NixOS Stage 1 >>>` and then:
waiting for device /dev/vda1 to appear.......................
Timed out waiting for device /dev/vda1, trying to mount anyway.
mount: /mnt-root: fsconfig() failed: /dev/vda1: Can't lookup blockdev.
`nixpkgs.lib.nixosSystem` imports neither profiles/base.nix nor
profiles/qemu-guest.nix, and the latter is the only place the virtio module
set comes from. Measured on the config as it stood:
$ nix eval …#config.boot.initrd.availableKernelModules --json
["ahci","ata_piix","atkbd",…,"nvme",…,"sd_mod","sr_mod",…,"usbhid",…]
IDE, SATA, NVMe and USB keyboards, and not one virtio_*. The harness attaches
the disk `if=virtio`, so /dev/vda1 could not appear no matter how long stage 1
waited. Same command after the import:
["9p","9pnet_virtio","virtio_blk","virtio_mmio","virtio_net",
"virtio_pci","virtio_scsi", …]
WHY IT GOT THIS FAR. Nothing cross-checks `fileSystems."/"` against the drivers
the initrd can actually load, so the image evaluated and built cleanly. SeaBIOS
and GRUB then reached the disk through BIOS int 13h, which works for any
controller — so firmware, bootloader, kernel and initrd all loaded off a disk
the kernel itself could not address. A disk that boots is not a disk the kernel
can mount; every stage before stage 1 was reading it a different way.
The import also carries 9p and 9pnet_virtio, which the /fixtures share needs.
That mount is `nofail`, so without them gen-fixtures would have written its
output into an ordinary directory inside the guest, the host would have
collected nothing, and the step would have reported success. Two failure modes,
one import — and the second is the more dangerous, because it is silent.
NOT VERIFIED HERE: this is measured at the evaluation layer only. Whether the
guest now boots to multi-user.target and writes four fixtures can only be
answered by a dispatch on x86_64-linux.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run 30583284365 failed twice over, and the serial console added in b57200b is the only reason either half is legible. Both defects are in this file. 1. THE UNIT HAD NO PATH TO THE TOOLS THAT WERE IN THE IMAGE. gen-fixtures died on: capability missing: cannot create network namespace (is this runner12 with the `netns` label?): command `ip` failed: could not spawn: No such file or directory (os error 2) `pkgs.iproute2` was in `environment.systemPackages` the whole time. That option does not do what the name suggests for a service: it populates /run/current-system/sw/bin, which is on the PATH of an interactive login shell. A systemd unit gets NixOS's minimal default (coreutils, findutils, gnugrep, gnused, systemd, util-linux) and nothing more. `ip` was in the closure and unreachable from the one process that calls it. `systemd.services.<name>.path` is the unit-scoped list, so that is where the tools now go. `ip netns exec` passes the caller's PATH down to the child, so this one list covers both the direct spawns and the namespaced ones. The list is DERIVED from the source, not recalled: every literal passed to run_cmd / netns_exec / netns_capture / netns_spawn_bg in crates/spar-trace-topology/src/bin/gen-fixtures.rs and src/fixtures/netns.rs. That mattered — writing it from the comment above systemPackages produced a list missing `arping`, because that comment was itself already incomplete. Copying it would have propagated the gap. `arping` turned out to be missing from `environment.systemPackages` too, so it was not in the guest at all, and its call site discards the error (see the companion commit). It has therefore never run, and nothing said so. It comes from iputils rather than pkgs.arping: the call site passes `-I <dev>`, which is iputils' spelling; Habets' arping (pkgs.arping 2.26) uses `-i` and would fail on the interface. Confirmed against the pinned channel — nixos-25.11's iputils is 20250605 and its own longDescription enumerates arping, with no meson flag disabling it. 2. A FAILING RUN COULD NOT POWER THE GUEST OFF. Power-off was `ExecStartPost`, which systemd runs ONLY when ExecStart succeeded. So the unit went to `failed`, nothing shut the machine down, and it idled at the login prompt until the harness killed it at 900s: fixture-vm: error: QEMU guest did not shut down within 900 seconds A failing run was therefore indistinguishable from a hanging one, and cost fifteen minutes to say so. `ExecStopPost` runs on both paths and is documented for exactly this. `TimeoutStopSec = 60` bounds the stop sequence itself, so a wedged poweroff is SIGKILLed rather than drifting out to the harness watchdog. That watchdog stays as the backstop; it just should not be the first thing that notices anything. Refs #362, #365 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run 30583284365 reported a missing `ip` like this:
capability missing: cannot create network namespace (is this runner12
with the `netns` label?): command `ip` failed: could not spawn:
No such file or directory (os error 2)
The sentence blames the runner label. The actual fault was the guest's
systemd unit PATH — `ip` was in the image, just not reachable from the
service. Two different owners, two different fixes, one message. The
os-error-2 tail was the only thing that distinguished them, and it is at
the far end of a line that opens by naming the wrong suspect.
So the two are now different values, not one:
* ToolNotFound — spawn returned ENOENT. The binary is not on PATH. That
is a provisioning fault in the image we build. Its message points at
the unit `path` and states outright that environment.systemPackages
does not reach a unit, because that is the trap that produced it.
* CapabilityMissing — the binary ran and was refused. A privilege fault
in the environment we run under, and the runner-label question is a
fair one to ask.
`diagnose_create` is where the split is enforced: it wraps a failed
`ip netns add` in the capability framing EXCEPT when the inner error is
ToolNotFound, which passes through unwrapped so the message names the
image rather than the label.
Three tests, and they cover two different halves:
* absent_tool_and_refused_tool_are_distinguishable spawns a name that
cannot exist and compares its rendering against a program that runs and
exits non-zero, asserting the two strings differ. This is the detection
half. Mutation-tested: collapsing the ENOENT arm back into
FixtureError::Command — the pre-fix behaviour — fails this test and
only this test, exit 101.
* the two diagnose_create_* tests construct the error directly, so they
survived that mutation. They cover attribution, not detection. Stated
here because "three tests passed" would otherwise imply three
independent guards, and it is two.
Also: gen-fixtures spawned arping behind `let _ =`. Non-fatal is right —
arping exits non-zero with no reply and the capture is still usable — but
non-fatal and unobserved are different things. That discarded Result was
the one signal that would have said arping was absent from the guest
entirely, which it was (see the companion commit); the call has therefore
never done anything and never said so. It now warns, matching the lldpd,
lldpctl and pmc calls around it.
Refs #362, #365
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Neither job had one, so both inherited GitHub's 6-hour default on a self-hosted pool. A wedged job does not just fail late — it holds a runner slot the whole time, and these run on the `podman` label, which is one machine. generate-fixtures already has an inner watchdog: the harness kills the QEMU guest at 900s. That covers the guest and nothing else. The nix build of the qcow2, a podman that will not start, a runner that loses /dev/kvm — all of it sits outside the watchdog and was unbounded. 180 minutes is set against a measured warm-cache run (30583284365: 21:27:49 -> 21:57:50, ~30 min) with room for a cold rustc + nixpkgs build. resolve-lock is a network operation with no watchdog at all; 30 minutes is generous for resolving flake inputs and still bounds a stalled fetch. These are backstops, not the fix for anything currently broken — the two commits before this one address why 30583284365 actually hung. A backstop is worth having anyway: the next unknown failure should cost minutes, not a working day of runner time. Refs #362, #365 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Addresses #362 and #365. Neither is auto-closed: a green dispatch is the gate
for flipping those to
Closes, and that has not happened yet.What
Adds
tools/fixture-vm/flake.lock. One new file, 27 lines. Nothing else.Why the nightly has never passed
Trace-Topology Fixture Generation has concluded
failureon 60 of 60 runs,2026-06-01→2026-07-30. There is no passing run in the window.It is not a defect — it is
trace-fixtures.yml:58doing precisely its job(run 30516478347):
The guard is fail-closed by design: the nightly builds only against a committed
lock and never re-resolves inputs itself, so that lock is the determinism pin for
the pinned nixpkgs, the guest kernel, and the whole closure. It has simply never
been satisfied —
tools/fixture-vm/has contained onlyflake.nixsince itlanded. The workflow told us the exact remedy in every one of those 60 runs.
This follows that remedy literally: dispatch
fixture-vm flake.lock(concluded
success), download its artifact, commit it in a reviewed PR.What is being pinned, and why this commit chooses nothing
flake.nix:70declaresgithub:NixOS/nixpkgs/nixos-24.05. The lock resolves thatto
b134951a4c9f3c995fd7be05f3243f8ecd65d798.I verified that independently of the container that produced it — verifying a
CI artifact with the CI that made it proves nothing. Against the GitHub API
directly:
NixOS/nixpkgsb134951a…,2024-12-30T13:00:28Znixos-24.05So this commit is the deterministic resolution of what the repo already declares,
not a value I selected.
That branch tip is dated 2024-12-30.
nixos-24.05has not moved in 19 monthsand is EOL.
nix flake updatefaithfully resolved the newest commit on the declared branch;that branch just stopped receiving them. Bumping to a supported channel is a real
decision and is not taken here, for one concrete reason: it would change the
guest kernel, and the generated trace fixtures are a function of the guest
kernel. Folding a fixture-changing input bump into a red-to-green CI repair would
hide it.
Two separable questions, and I'd rather they be reviewed as such:
who can weigh fixture stability against an unpatched-CVE build environment.
If the answer to (2) is "bump it," the right shape is: change
flake.nix's ref,re-dispatch, and land the fixture diff visibly in its own PR.
Verification — and its honest limit
30560238740—success.version: 7, singlenixpkgsinput matchingflake.nix.What I have not verified: that the nightly now goes green. This unblocks the
guard at line 58; whether the subsequent steps (guest build,
/dev/kvmboot,fixture generation) succeed has been untestable for as long as the guard has been
failing — no run has ever reached them. It is entirely possible this reveals a
second failure behind the first. That is progress either way: the next failure
will be a real one rather than a missing file, and it will be visible.
I'll dispatch
trace-fixtureson this branch and report what actually happensbefore merging.
What the dispatch found: two more blockers, both now fixed here
The prediction above held. The lock let nix evaluate for the first time in the
workflow's life, and evaluation immediately died on something new.
flake.lockabsent; the line-58 guard is fail-closedsystem.build.qcow2never defined — the attribute the workflow builds existed only in commentscargoLock.outputHashesmissing for theetchgit dependencyEach was invisible behind the one before it. That is the whole reason this took
60 runs to surface: the outer guard failed closed, so nothing downstream was ever
reached, and "never ran" logged identically to "ran and failed."
Blocker 2 (#365).
system.buildis a freeformlazyAttrsOf raw, so a membernobody defines is not a typo the module system catches — it is
error: attribute 'qcow2' missingat evaluation. Fixed by importing nixpkgs' ownmake-disk-image.nixas a build product, the same waynixos-generatorsdefinesits
qcowformat. Two arguments are load-bearing:partitionTableType = "legacy"— not style.rootPartitionis a switch on it(
legacy → "1",efi → "2",hybrid → "3"), andflake.nixhardcodesfileSystems."/" = "/dev/vda1". Any other value partitions cleanly, buildscleanly, and then cannot find its root — a failure that appears at guest boot,
not at build.
copyChannel = false— the defaulttruecopies the whole nixpkgs tree into animage that boots once, runs one oneshot service, and powers off.
Blocker 3.
etch(pulseengine/rivet, pinned in the rootCargo.toml) is theworkspace's one git dependency. Registry crates carry their checksum in
Cargo.lock; git deps do not, soimportCargoLockcannot vendor them without anexplicit hash. Note it is vendored even though
gen-fixturesnever uses it —importCargoLocktakes the entire lock file, not the subtreecargoBuildFlagsselects.
How this was verified without burning CI runs
nix buildneeds x86_64-linux. But all three blockers are evaluation errors,and evaluation is platform-independent — so the whole guest config can be forced
on a macOS laptop in ~2 minutes, instead of one blocker per ~10-minute self-hosted
dispatch:
That now exits 0 and prints a
.drvpath, warning-free. Blocker 3 was found thisway rather than by a CI round-trip.
The
etchhash was cross-checked two-sidedly against the pinned 24.05fetchgit, not trusted from the unstablenix-prefetch-gitthat produced it —a fetcher-behaviour difference between channels would evaluate fine and then fail
at build time in CI. The correct hash yields a store path; a deliberately wrong
one errors with
got: sha256-x37urQw…, re-deriving the same value independently.A one-sided check that only confirms the happy path proves nothing.
What is still unverified: that the image builds and the guest boots. Those
are x86_64-linux build-time and run-time concerns that only CI can settle, and a
clean local eval says nothing about either. Dispatched as
run 30562393849; I
will report what actually happens rather than merging on a green evaluation.
Also folded in:
boot.loader.grub.timeout→boot.loader.timeout, which wasemitting a rename warning on every evaluation. One line, and it keeps the build
log readable so a real warning stands out.
No rivet artifact, deliberately
Matching #353 and #363: workflow/CI-infrastructure changes in this repo carry no
rivet artifacts, and there are no
REQ-CI-*ids. Flagged so the omission reads asa choice.
Meta
Same shape as #361/#363. Neither nightly is a required context, which is why a
job that has never once passed sat red for two months announcing its own fix in
plain text. A gate nobody reads is not a gate.
🤖 Generated with Claude Code