ci: make the reproducibility gate able to fail

[zah]

What was broken

.github/workflows/ci.yml, job documentation-reproducibility, step "Test reproducible builds". The entire assertion was:

diff -r result-1 result-1-rebuild || echo "Build not reproducible for rust-echo-service"
diff -r result-2 result-2-rebuild || echo "Build not reproducible for attestation-agent"

This gate could not detect an irreproducible build, for two independent reasons:

1. The result was swallowed. || echo turns a real difference into a log line and an exit status of 0. A reproducibility regression printed a message and the job stayed green.
2. Nothing was ever rebuilt. The "rebuild" invocations were plain nix build. Nix resolves them to the store path that already exists instead of re-running the derivation, so result-1 and result-1-rebuild were two symlinks to the same store path. diff -r was comparing a directory with itself. Even without the || echo, the check would have passed unconditionally.

Evidence for (2) from the last green CI/CD Pipeline run of this step (run 27944090156, job Documentation & Reproducibility): the first two nix build calls build 15 derivations, and the two -rebuild calls emit nothing at all except a settings warning before the step ends. No rebuild, no diff output.

What changed

Only the one step. No new tolerances, no continue-on-error, no skips.

    - name: Test reproducible builds
      run: |
        set -euo pipefail

        nix build .#rust-echo-service -o result-1
        nix build .#attestation-agent -o result-2

        # `--rebuild` re-executes the derivation and makes Nix compare the fresh
        # output against the one already in the store. Without it the second
        # `nix build` is a no-op that resolves to the *same* store path, so the
        # comparison below would trivially succeed no matter what.
        nix build .#rust-echo-service --rebuild -o result-1-rebuild
        nix build .#attestation-agent --rebuild -o result-2-rebuild

        # A differing pair prints the full diff (diagnostics) and makes the step
        # exit non-zero. This gate must never report success on a real diff.
        pairs=(
          "rust-echo-service|result-1|result-1-rebuild"
          "attestation-agent|result-2|result-2-rebuild"
        )
        rc=0
        for entry in "${pairs[@]}"; do
          IFS='|' read -r component first second <<< "$entry"
          echo "=== Reproducibility check: $component ($first vs $second) ==="
          if diff -r "$first" "$second"; then
            echo "OK: $component is reproducible"
          else
            echo "FAIL: build not reproducible for $component (diff shown above)"
            rc=1
          fi
        done

        if [ "$rc" -ne 0 ]; then
          echo "Reproducibility gate failed: at least one component produced differing outputs."
        fi
        exit "$rc"

Two layers now do real work: --rebuild makes Nix itself fail the step when a re-execution produces a different output, and the explicit diff loop prints the file-level diff and propagates a non-zero exit. Diagnostics are preserved — the diff is still printed in full — but the step can no longer report success on a real difference.

This check may flip green → red. If it does, the failure predates this PR. Nothing here makes the build less reproducible; the check has simply started telling the truth. Please do not "fix" a red result by weakening this step.

Mutation verification

An argument is not evidence, so the new step body was executed against real, differing directories. The body was extracted verbatim from the committed YAML with yq, and run with a stub nix on PATH that materializes the -o <name> out-links from fixture directories — so the shell that runs is byte-for-byte the shell CI runs.

$ yq -r '.jobs["documentation-reproducibility"].steps[]
         | select(.name=="Test reproducible builds") | .run' \
     .github/workflows/ci.yml > /tmp/rv/step-body.sh

Scenario A — an actual differing pair (result-1/bin/rust-echo-service = BUILD-A, result-1-rebuild/bin/rust-echo-service = BUILD-B-DIFFERENT):

$ cd /tmp/rv/caseA && PATH=/tmp/rv/stub-bin:$PATH NIX_STUB_FIXTURES=/tmp/rv/caseA/fixtures \
    bash -euo pipefail /tmp/rv/step-body.sh
=== Reproducibility check: rust-echo-service (result-1 vs result-1-rebuild) ===
diff -r result-1/bin/rust-echo-service result-1-rebuild/bin/rust-echo-service
1c1
< BUILD-A
---
> BUILD-B-DIFFERENT
FAIL: build not reproducible for rust-echo-service (diff shown above)
=== Reproducibility check: attestation-agent (result-2 vs result-2-rebuild) ===
OK: attestation-agent is reproducible
Reproducibility gate failed: at least one component produced differing outputs.
SCENARIO A EXIT CODE = 1

Scenario B — an identical pair:

$ cd /tmp/rv/caseB && PATH=/tmp/rv/stub-bin:$PATH NIX_STUB_FIXTURES=/tmp/rv/caseB/fixtures \
    bash -euo pipefail /tmp/rv/step-body.sh
=== Reproducibility check: rust-echo-service (result-1 vs result-1-rebuild) ===
OK: rust-echo-service is reproducible
=== Reproducibility check: attestation-agent (result-2 vs result-2-rebuild) ===
OK: attestation-agent is reproducible
SCENARIO B EXIT CODE = 0

Scenario C — only the second component drifts (content change + an extra file), proving the loop does not short-circuit after the first pair:

$ cd /tmp/rv/caseC && PATH=/tmp/rv/stub-bin:$PATH NIX_STUB_FIXTURES=/tmp/rv/caseC/fixtures \
    bash -euo pipefail /tmp/rv/step-body.sh
=== Reproducibility check: rust-echo-service (result-1 vs result-1-rebuild) ===
OK: rust-echo-service is reproducible
=== Reproducibility check: attestation-agent (result-2 vs result-2-rebuild) ===
diff -r result-2/bin/attestation-agent result-2-rebuild/bin/attestation-agent
1a2
> EXTRA-FILE-DRIFT
Only in result-2-rebuild/bin: stray-file
FAIL: build not reproducible for attestation-agent (diff shown above)
Reproducibility gate failed: at least one component produced differing outputs.
SCENARIO C EXIT CODE = 1

Control — the old body against the exact same differing pair from Scenario A:

$ cd /tmp/rv/caseA && PATH=/tmp/rv/stub-bin:$PATH NIX_STUB_FIXTURES=/tmp/rv/caseA/fixtures \
    bash -euo pipefail /tmp/rv/step-body-original.sh
diff -r result-1/bin/rust-echo-service result-1-rebuild/bin/rust-echo-service
1c1
< BUILD-A
---
> BUILD-B-DIFFERENT
Build not reproducible for rust-echo-service
ORIGINAL BODY EXIT CODE ON A REAL DIFF = 0

1 vs 0 on identical inputs is the whole point of this PR.

The --rebuild half, verified independently

--rebuild is the part that actually re-executes the derivation, so it was verified on a derivation that is nondeterministic by construction (date +%s%N > $out/stamp):

$ nix build --impure --file nondet2.nix --no-link            # first build
  nix build (first)               exit=0
$ nix build --impure --file nondet2.nix --no-link            # second build, no --rebuild
  nix build (again, NO --rebuild) exit=0   <- what the old workflow did
$ nix build --impure --file nondet2.nix --rebuild --no-link
  nix build --rebuild             exit=1
error: derivation '/nix/store/v2452mz71ad8wi1gsx1n7axzk9713001-nondet-demo-2.drv' may not be deterministic: output '/nix/store/wh1f6isqb0j2xwc6n2sa3nlfbizy6l67-nondet-demo-2' differs

A demonstrably nondeterministic derivation is re-"built" by the old command with exit code 0. --rebuild catches it. Note --rebuild requires the output to already be valid in the store, which is why the plain builds of result-1/result-2 stay and run first.

The new step body also passes shellcheck -s bash cleanly.

Is the build actually reproducible?

Yes — and this PR is the first time that has ever actually been checked.

Before this change there was no signal at all. Because of defect (2), the job never once re-executed either derivation, so the absence of a Build not reproducible line in past logs was not evidence of reproducibility — the check was comparing each out-link with itself.

The hardened step ran on this PR (CI/CD Pipeline run 31975702359, job Documentation & Reproducibility) and produced real verification:

22:29:40 checking outputs of '/nix/store/yaiw7qr9crqbmb2zmcklnfffpxfanysw-rust-echo-service-0.1.0.drv'...
22:31:26 checking outputs of '/nix/store/725dz7vcpiyvjhj0ahz2nc9dl59h739d-attestation-agent-0.1.0.drv'...
22:34:00 === Reproducibility check: rust-echo-service (result-1 vs result-1-rebuild) ===
22:34:00 OK: rust-echo-service is reproducible
22:34:00 === Reproducibility check: attestation-agent (result-2 vs result-2-rebuild) ===
22:34:00 OK: attestation-agent is reproducible

Those checking outputs of ... lines are Nix genuinely re-running each derivation and byte-comparing the result with the store — they have never appeared in this job's logs before. The job's conclusion is success, and this time that means something.

Independently confirmed on a local machine:

$ nix build .#rust-echo-service --no-link          # populate the store
  exit=0  -> /nix/store/j8lk48kqs3ljkf4nbpblx7985a98wc04-rust-echo-service-0.1.0
$ nix build .#rust-echo-service --rebuild --no-link
checking outputs of '/nix/store/yaiw7qr9crqbmb2zmcklnfffpxfanysw-rust-echo-service-0.1.0.drv'...
  REBUILD_EXIT=0

So the good news is that nothing is being hidden today. The point stands regardless: the gate that was supposed to protect this property could not have reported a regression, and now it can.

Other cannot-fail checks in this repo (reported, not changed here)

Found while auditing ci.yml; none are touched by this PR, and each deserves its own decision:

• ci.yml:121-133 — "Test <component> basic functionality" ends every branch in || true. A binary that segfaults on startup passes this step exactly like one that runs fine. A meaningful version would assert the expected exit status (e.g. 124 from timeout for the long-running services, 0 for --help).
• ci.yml:174-186 — coverage generation is cargo install ... || true / cargo tarpaulin ... || true, and the Codecov upload sets fail_ci_if_error: false. Defensible for coverage reporting, but it means a total coverage-tooling outage is invisible.
• ci.yml:213-216 — chmod +x build/bin/* || true / ls -la build/bin/* || true. If the artifact download produced nothing, this step passes and the failure surfaces later as a confusing integration-test error.
• Justfile:212 — ci-docs-reproducibility: check generate-docs is advertised as the local mirror of this CI job but performs no reproducibility check at all, so just ci-main cannot catch a reproducibility regression either.

ci.yml:221-229 (pkill -f ... || true in the always-run cleanup step) is a legitimate use — pkill exits 1 when nothing matched — and is left alone. There is no continue-on-error: anywhere in the repository.

🤖 Generated with Claude Code