ci: make the four remaining green-no-matter-what checks able to fail

[zah]

Follow-up to #2, which fixed the reproducibility gate. That review turned up four more checks in this pipeline that reported success no matter what they observed. This PR turns each one into an assertion.

Stacked on #2. This branch is cut from ci/reproducibility-gate-must-fail, so #2's commit appears here too — that was necessary for item 2, where the CI step and the Justfile recipe are made to share one implementation. Merge #2 first and this diff reduces to its own commit.

One serious finding, described in full under item 4: coverage generation has been failing outright for some time, and || true hid it. The coverage number published for this repo is fictional.

---

Item 1 — component smoke test could not detect a binary that never starts

ci.yml, job build-matrix, step "Test <component> basic functionality". Every branch ended in || true:

timeout 10s ./build/attestation-agent/bin/attestation-agent || true
timeout 5s  ./build/rust-echo-service/bin/rust-echo-service || true
./build/rust-client/bin/rust-client --help || true

A binary that segfaults on startup passed exactly like one that works.

Landed. The three long-running services must still be alive when timeout kills them; the one-shot CLIs must exit 0 from --help:

set -euo pipefail
ulimit -c 0
component="<matrix.component>"
bin="./build/$component/bin/$component"
test -x "$bin"
case "$component" in
  "attestation-agent")                     kind=service; limit=10 ;;
  "rust-echo-service"|"cpp-echo-service")  kind=service; limit=5  ;;
  *)                                       kind=cli ;;
esac
if [ "$kind" = "service" ]; then
  echo "=== $component: expecting it to stay alive for ${limit}s ==="
  if timeout "${limit}s" "$bin"; then status=0; else status=$?; fi
  if [ "$status" -eq 124 ]; then
    echo "OK: $component was still running after ${limit}s and was killed by timeout"
  else
    echo "FAIL: $component exited on its own with status $status before the ${limit}s deadline."
    exit 1
  fi
else
  ...same shape, asserting status -eq 0 from --help...
fi

Why 124 is the pass condition. GNU timeout returns 124 only when the command was still running at the deadline and had to be killed — which for a server is exactly the healthy outcome. A process that exits on its own returns its own status (0, 1, 101, 139 …), and for a server that means it crashed or refused to start. Verified against every mode:

exit0.sh        -> timeout exit=0      exit1.sh -> timeout exit=1
segv.sh         -> timeout exit=139    (SIGSEGV = 128+11)
sigterm-trap.sh -> timeout exit=124    (traps SIGTERM, exits 0 — still 124)

That third-to-last case matters: cpp-echo-service really does trap SIGTERM and exit 0 cleanly (Shutdown signal 15 received appears in its CI log). timeout still reports 124, so the assertion is correct for it.

A trap found while verifying, and closed. A crashing binary that is slow to dump core can still be dying when the deadline fires, and timeout then reports 124 — the healthy code — for a segfault. This is not hypothetical; it reproduced here. Running the step without ulimit -c 0, five times, against a fixture that segfaults immediately:

run 1 WITHOUT ulimit -c 0: step exit=0   <- a segfaulting service reported HEALTHY
run 2 WITHOUT ulimit -c 0: step exit=1
run 3 WITHOUT ulimit -c 0: step exit=1
run 4 WITHOUT ulimit -c 0: step exit=1
run 5 WITHOUT ulimit -c 0: step exit=1
run 1 WITH    ulimit -c 0: step exit=1
run 2 WITH    ulimit -c 0: step exit=1
run 3 WITH    ulimit -c 0: step exit=1

Hence the ulimit -c 0 line: with core dumps off, a crash is reported immediately and honestly, and the check stops being flaky in the one direction a check must never be flaky.

Mutation verification — item 1

Step body extracted from the committed YAML with yq, with ${{ matrix.component }} substituted exactly as Actions does, run against fixture binaries:

Fixture	Exit
service stays alive	0
service traps SIGTERM and exits 0 (real cpp-echo-service behaviour)	0
service exits 1 immediately ("Address already in use")	1
service segfaults on startup	1
CLI --help exits 0	0
CLI --help exits 2	1

Control — the old body against the same three failing fixtures: 0, 0, 0.

The assertion also fires on the real binaries

Run against the actual built components rather than fixtures:

derivation-hasher --help              -> exit=0    (pass)
rust-client --help                    -> exit=0    (pass)
timeout 5s  rust-echo-service         -> exit=124  (pass: still alive, killed by timeout)
timeout 10s attestation-agent         -> exit=1    (fail)

That last line is a real "service failed to start", caught by the new assertion and invisible under the old one:

[INFO  attestation_agent] Initializing AttestationAgent
Error: Os { code: 98, kind: AddrInUse, message: "Address already in use" }

It is specific to the machine it ran on — port 3000 was already occupied there — not a defect in this repo; on the CI runner the agent binds 3000 and stays up for the full 10 s, which is why this item is expected to stay green. It is worth stating plainly, though, that this check now reports a genuine failure when a port is unavailable, because in that situation the service genuinely did not start. On ephemeral GitHub-hosted runners each job gets a clean VM, so there is no shared-port hazard; on a developer workstation a red result here is the truth, not noise.

---

Item 2 — just ci-docs-reproducibility ran no reproducibility check

Justfile:212 was ci-docs-reproducibility: check generate-docs with an empty body, so just ci-main could not catch a reproducibility regression locally no matter what.

Landed. The check is now scripts/check-reproducibility.sh (the body from #2, unchanged in behaviour), and both callers run it:

    - name: Test reproducible builds
      # Implementation lives in scripts/check-reproducibility.sh so that this
      # gate and the `just ci-docs-reproducibility` recipe run the same code.
      run: ./scripts/check-reproducibility.sh

ci-docs-reproducibility: check generate-docs
    @echo "=== Running Reproducibility Check ==="
    ./scripts/check-reproducibility.sh

Mutation verification — item 2

scripts/check-reproducibility.sh run with a stub nix that materializes the out-links from fixtures:

Fixture	Exit
rust-echo-service output differs	1
both components identical	0
attestation-agent output differs (content + stray file)	1

Control: the old recipe had no body — no check ran, so the local command was incapable of failing.

---

Item 3 — a missing artifact download passed

chmod +x build/bin/* || true
ls -la build/bin/* || true

An empty or partial download passed here and surfaced later as a confusing integration-test failure.

Landed. All five binaries must be present, and the failure message shows what actually arrived:

set -euo pipefail
expected=(attestation-agent cpp-echo-service derivation-hasher rust-client rust-echo-service)
missing=()
for name in "${expected[@]}"; do
  if [ ! -f "build/bin/$name" ]; then missing+=("build/bin/$name"); fi
done
if [ "${#missing[@]}" -ne 0 ]; then
  echo "FAIL: expected binaries are missing from the downloaded artifacts:"
  printf '  %s\n' "${missing[@]}"
  echo "What was actually downloaded:"
  if [ -d build ]; then find build -maxdepth 3 -mindepth 1 | sort; else echo "  (no 'build' directory at all)"; fi
  exit 1
fi
chmod +x build/bin/*
ls -la build/bin/

The list is not guesswork — the previous run's ls output shows exactly these five arriving.

Mutation verification — item 3

Fixture	Exit
all five binaries present	0
rust-echo-service missing	1 (names the missing path, lists what did arrive)
download produced nothing at all	1 (no 'build' directory at all)

Control — the old body against the same two failing fixtures: 0 and 0 (the second printing chmod: cannot access 'build/bin/*' and passing anyway).

---

Item 4 — coverage has been failing silently; the published number is fictional

This is the finding. The step was:

cargo install cargo-tarpaulin || true
cargo tarpaulin --out Xml || true

Both commands have been failing. From the most recent run's Generate test coverage step:

error: failed to compile `cargo-tarpaulin v0.37.2`
Caused by:
  rustc 1.86.0 is not supported by the following packages:
    cargo-platform@0.3.3 requires rustc 1.91
    ...
error: no such command: `tarpaulin`

cargo install cannot build a current cargo-tarpaulin against the pinned 1.86 toolchain, so cargo tarpaulin is not a command, so no cobertura.xml is written — and fail_ci_if_error: false then lets the Codecov step upload nothing without complaint. Every layer of this was suppressed, so coverage silently stopped being measured while continuing to look measured. Whatever coverage figure is associated with this repo does not come from this pipeline.

Worth noting separately: cargo install reached out to crates.io in the middle of an otherwise Nix-pinned build, which is both a supply-chain surface and a source of exactly this kind of drift.

Landed, because it turned out to be fixable rather than merely reportable. cargo-tarpaulin now comes from the dev shell (flake.nix), and generation must succeed:

set -euo pipefail
cd <matrix.project>
cargo tarpaulin --out Xml
test -f cobertura.xml
echo "OK: coverage report generated for <matrix.project>"

Verified locally before landing — nixpkgs 25.05 ships cargo-tarpaulin 0.32.5, which works against the pinned toolchain:

$ nix shell nixpkgs/nixos-25.05#cargo-tarpaulin ... --command cargo tarpaulin --out Xml
test result: ok. 4 passed; 0 failed
60.96% coverage, 114/187 lines covered
TARPAULIN_EXIT=0
-rw-r--r-- 1 zahary users 5977 cobertura.xml

What I deliberately left advisory, and why. fail_ci_if_error: false on the Codecov upload stays, now with a comment saying so. That step posts to a third-party service; an outage or rate limit at codecov.io is not a defect in this repository and should not block a merge. The assertion that actually matters — that coverage was measured and the report exists — now lives in the step above, where a failure means something about our code. Splitting it this way is what makes the advisory half defensible: nothing is hidden any more, because the thing being tolerated is now only the network hop.

Mutation verification — item 4

Fixture	Exit
tarpaulin works, report written	0
tarpaulin missing (today's real CI behaviour)	101
tarpaulin exits 0 but writes no report	1

Control — the old body against the same two failing fixtures: 0 and 0.

---

CI verdict: all 14 checks pass, and this time they did the work

Run 31977587449 — conclusion=success. No badge flips green → red. Every item was measured against the current pipeline before being changed, which is why. The logs show each assertion actually executing rather than passing vacuously:

Item 1 — every component reached a real verdict:

=== cpp-echo-service: expecting it to stay alive for 5s ===
OK: cpp-echo-service was still running after 5s and was killed by timeout     (22:59:00 -> 22:59:05)
=== rust-echo-service: expecting it to stay alive for 5s ===
OK: rust-echo-service was still running after 5s and was killed by timeout    (23:00:29 -> 23:00:34)
=== attestation-agent: expecting it to stay alive for 10s ===
OK: attestation-agent was still running after 10s and was killed by timeout
=== derivation-hasher: expecting '--help' to exit 0 ===
OK: derivation-hasher --help exited 0
=== rust-client: expecting '--help' to exit 0 ===
OK: rust-client --help exited 0

Item 2 — the shared script ran, and Nix genuinely re-executed both derivations:

##[group]Run ./scripts/check-reproducibility.sh
checking outputs of '/nix/store/...-rust-echo-service-0.1.0.drv'...
checking outputs of '/nix/store/...-attestation-agent-0.1.0.drv'...
OK: rust-echo-service is reproducible
OK: attestation-agent is reproducible

Item 3 — all five binaries present and listed.

Item 4 — the headline. For the first time, this pipeline produced actual coverage numbers:

60.96% coverage, 114/187 lines covered   OK: coverage report generated for derivation-hasher
28.16% coverage,  89/316 lines covered   OK: coverage report generated for attestation-agent
47.37% coverage,  63/133 lines covered   OK: coverage report generated for services/rust-echo
 1.41% coverage,   1/71  lines covered   OK: coverage report generated for clients/rust-client

These are the first real measurements the pipeline has emitted. They are also worth reading on their own terms — clients/rust-client at 1.41% is effectively untested, and that fact was previously invisible behind a coverage step that could not fail. Nothing in this PR changes those numbers; it just stops them from being hidden.

All new step bodies and scripts/check-reproducibility.sh pass shellcheck -s bash. No continue-on-error, no || true, no || echo, no skips were added anywhere; the legitimate pkill … || true cleanup at the end of the integration-tests job is untouched.

🤖 Generated with Claude Code