ci: perry compile --no-link pays for a full runtime build it discards — 343s nested cargo inside a 300s budget (intermittent compiler-output-regression timeout)

[proggeramlug]

compiler-output-regression has been red on every main sweep for at least 34 hours — continuously from 60922041cd (2026-09-18T17:56) through 3f5b5b1424 (2026-09-20T03:11), 26 sweeps with no green in between. It is part of main-gate, so main-gate is red on every train.

It is not a compiler hang, and the log reads exactly like one:

subprocess.TimeoutExpired: Command '[.../target/debug/perry', 'compile',
  '.../benchmarks/compiler_output/fixtures/h1_native_rep_equivalence.ts',
  '-o', '.../hir-probe.o', '--print-hir', '--no-link', '--no-cache',
  '--verify-native-regions']' timed out after 300 seconds

All four steps fail (native-region proof, native-ABI proof, positive vectorization, HIR fact rewrite) because they share the fixture list and the same fixture stalls each one.

What is actually happening

perry compile is not compiling for 300 seconds. It spends about one second compiling and then blocks on a nested cargo build, because auto-optimize fires and rebuilds the runtime from source.

Reproduced locally with the v0.5.1611 release binary, the same command line:

run	result
as CI runs it	still going at >300 s
PERRY_NO_AUTO_OPTIMIZE=1	1.2 s, rc=0

The stalled process is at 0.0% CPU with a child — the discriminator that separates "spinning" from "waiting". The child is:

cargo build --release -p perry-runtime-static -p perry-stdlib-static
  --no-default-features --features perry-runtime/full,...
  --out-dir .../target/perry-auto-a51d6650d9b10028/release/...

i.e. a full cold-cache runtime build, inside a 300-second budget, on a CI runner, from a debug compiler.

The one-line cause

scripts/compiler_output_harness/capture.py:137 builds the compiler environment as

env = {**os.environ, "PERRY_LLVM_KEEP_IR": "1", "PERRY_NO_CACHE": "1"}

PERRY_NO_AUTO_OPTIMIZE is not among them, and nothing in .github/workflows/test.yml's job env sets it either. --compile-timeout defaults to 300 (cli.py:47).

Several other harnesses in this repo already set it deliberately — gc_instrument_smoke.sh:40, gc_root_dominance_corpus.sh:320, run_doc_tests.sh:22 — and auto_opt_app_patterns.sh:53 goes the other way and refuses to run if it is set, because testing that path is its whole point. So the convention exists; this harness is simply missing from it.

The fix needs one judgement call, not a blanket export

The capture steps that failed all pass --no-link --print-hir: for those, auto-optimize cannot affect the artifact under test and suppressing it is unambiguously right.

But the harness also has --run-timeout and run_perf_stat paths that execute a produced binary, where the linked runtime is part of what is being measured. Do not set PERRY_NO_AUTO_OPTIMIZE=1 globally in capture.py without establishing which suites run a binary and whether the auto-optimized runtime is load-bearing for them. Getting this wrong swaps a loud red for a quiet measurement change.

Also worth deciding: whether a 300-second compile timeout should be allowed to encompass a nested cargo build at all, or whether perry compile --no-link should decline to auto-optimize on its own — the second would fix the class rather than this instance.

Incidental

Each triggered run leaves a target/perry-auto-<hash>/ directory behind; the one my local reproduction created was 377 MB after a partial build. On CI that is inside the job, but it is the same leak tracked for the gap sweep.

[proggeramlug]

Correction to this issue's framing — the title and the "34 hours" claim were wrong, and I wrote them.

I sampled one failing run (bd24fb32a2), found TimeoutExpired in it, confirmed the job's conclusion was failure across 26 consecutive sweeps, and wrote that up as one cause. That was checking whether the failing set was non-empty instead of whether it was the same set.

Measured across six recent main sweeps, counting occurrences in each compiler-output-regression job log:

SHA	TimeoutExpired	unexpected_missed_reason_kinds	raw-field-get	no_dynamic_property_runtime
dd00a00305	0	7	8	3
3f5b5b1424	0	7	8	3
bd24fb32a2	2	6	4	3
14ca314481	0	7	8	3
14ca314481	0	7	8	3
2f9dc8e692	0	7	8	3

The timeout is intermittent — one run in six, and the run I happened to open. The persistent red is a set of structural verdicts that has nothing to do with auto-optimize, tracked separately as #10784.

This issue stays open on its own merits, rescoped. The mechanism is real and independently confirmed: run_pipeline.rs:6169 resolves optimized_libs unconditionally, while --no-link's early return is at :6730 — so a --no-link compile performs the entire runtime build and then discards it 560 lines later. PERRY_RUNTIME_DIR does not suppress it. Measured on an M-series mini, the nested build alone takes 343 s, against a 300 s --compile-timeout: it cannot fit the budget it is charged to, on hardware faster than a GitHub runner. That is why it fires sometimes rather than always — it is a cache-state race, not a constant.

Two further corrections to what I originally wrote here:

• I said the four failing steps "pass --no-link --print-hir" and that suppressing auto-optimize was therefore unambiguously safe for them. Each of those steps runs both a --no-link probe and a linking compile whose binary is then executed — no step passes --skip-run, and --perf-counters off disables only run_perf_stat, not run_benchmark. The suppression is safe for the probe, not for the step.
• Suppressing it on the linking compile would be actively wrong. runtime_budgets' allocations_traced / gc_collections_traced / write_barriers_traced are derived (analyzers.py:434-481) solely from the executed binary's PERRY_GC_TRACE stderr, and optimized_libs/freshness.rs:288-293 adds perry-runtime/diagnostics to the rebuild because the compiler sees PERRY_GC_TRACE=1 — a derivation that runs only on the auto-optimize branch. A runtime emitting no trace scores 0 on all three and passes every budget vacuously, since they are maxima. That is CLAUDE.md failure mode useEffect + setState panics with RefCell already borrowed on macOS ARM64 #4.

So the fix sets PERRY_NO_AUTO_OPTIMIZE=1 at exactly one call site — capture.py:242, the HIR probe — and deliberately not at capture.py:261. repsel_census.py:681 already sets it and has done for a while, with a docstring making the same argument; this was the one site missing from an existing convention. A test asserts the classification per call site and was sabotage-checked: moving the suppression to the linking compile turns it red.

Known remainder, not closed by the fix: the cold build still happens, just in the linking compile's budget instead of the probe's. Only one perry-auto hash is ever created across all 23 workloads, so CI pays exactly one cold build; the fix changes which step absorbs it, not whether 300 s is enough. The durable fix is a pre-warm step in test.yml with its own explicit budget — asserting target/perry-auto-* exists afterwards, or it is a no-op nobody notices.

Worth its own issue: --no-link could decline to auto-optimize outright — a two-line guard at run_pipeline.rs:6169 — which fixes the class rather than this instance. The risk is the undefined-symbol stub scan at :6179, which consumes optimized_libs.runtime to seed its defined-symbol set; skipping the rebuild falls back to find_runtime_library and could change the synthesized stubs in an emitted object. repsel_determinism and repsel_knob_isolation hash --no-link objects, so their baselines would move once.

One incidental worth recording: when the harness times out, Python kills perry but the nested cargo survives as an orphan and keeps building.

[proggeramlug]

Fixed on main as of v0.5.1616 (8e9f6f09e8), via merge train 237 (#10792).

compiler-output-regression now passes both suites — the first time since 2026-09-02, eighteen days. It is a main-gate input, so every merge in that window was bypassing it.

The refreshed gate was proven able to fail, on the assembled tree rather than only per-branch:

clean      → rc=0, "failed_workloads": []
sabotaged  → rc=1   (one require_records consumer reverted to js_object_get_field_by_name_f64)

That check took four attempts, and the three that failed are worth recording because each one looked like a pass: BSD sed has no 0,/re/ address so the file was never modified; the next edit landed on a comment rather than the consumer = line; and the third used the wrong entry point, which produced zero bytes of output and exited 0. Asserting that an edit landed is not sufficient — it has to land on the load-bearing line, and the harness has to actually be running.

Both fixes are in the train body in full, including the reasoning that PERRY_NO_AUTO_OPTIMIZE=1 is set at the HIR probe and deliberately not at the linking compile, since runtime_budgets are maxima derived from the executed binary's PERRY_GC_TRACE output and would pass vacuously against a runtime that emits no trace.

Two things left open rather than quietly closed:

The cold build still happens, just in the linking compile's 300 s budget instead of the probe's. Only one perry-auto hash is created across all 23 workloads, so CI pays exactly one cold build — the fix changes which step absorbs it, not whether 300 s is enough. The durable answer is a pre-warm step in test.yml with its own explicit budget, asserting target/perry-auto-* exists afterwards or it is a no-op nobody notices.

--no-link could decline to auto-optimize outright — a two-line guard at run_pipeline.rs:6169 — which fixes the class rather than this instance. The risk is the undefined-symbol stub scan at :6179, which consumes optimized_libs.runtime to seed its defined-symbol set; skipping the rebuild falls back to find_runtime_library and could change synthesized stubs in an emitted object. repsel_determinism and repsel_knob_isolation hash --no-link objects, so their baselines would move once. Worth its own issue.

And the structural finding, which this PR deliberately did not attempt: allowed_missed_reason_kinds is a module-wide sweep and main's event loop guarantees a five-remark floor in every program, so for any fixture without un-vectorizable loops of its own the list detects compiler boilerplate rather than the fixture. Excluding main is necessary but not sufficient — the GC loop back-edge poll injects control flow into the fixture's own functions, accounting for 2 of native_owned_typed_views' 3 control_flow remarks. -fsave-optimization-record=yaml gives exact per-function attribution with stable keys and is the right mechanism, ~150–250 lines, but the real cost is re-deriving all 25 workloads' lists. Its own issue, so that diff is reviewable on its own.

[proggeramlug]

Landed via merge train 243 (#10838) as v0.5.1622 — ba303c1f5f.

This should make compiler-output-regression green for the first time in weeks. Its structural verdicts were already fixed as of v0.5.1618 (failed_workloads: []); the only thing left was the 2 × TimeoutExpired residual recorded when #10782 was first closed — the cold build moving from the probe's budget into the linking compile's.

The pre-warm found something my design missed: there are two cold builds, not one. The perry-auto-<hash> directory name excludes the cross-feature set while the build stamp inside it includes it, and loop_bound_semantics is the one gated workload with no *_traced budget — so it wants the other feature set, hits a stamp mismatch, and re-runs cargo. A single-warm step would have left exactly one gate to absorb a cold build, and the red would have looked unchanged.

Both liveness assertions are in and both are load-bearing: the presence check (a perry-auto-*/ with a build stamp and a non-empty archive) and the verification compile that must land under half the 300 s budget. The second is the one that actually discriminates — presence can pass on a stale directory.

A gate caught the new step, and that is the gate working. compiler_output_step_liveness.py pins the job's post-build step inventory and refused the addition. Registered in COMPILER_SUBJECTS rather than exempted, since the pre-warm depends on the build and already carries the matching guard; the gate now passes at 10 subjects and its own self-test still passes at 8 cases, which I checked separately because a registry edit that broke the checker would be worse than the drift it hides.

Worth stating why it has to be listed rather than exempted: the pre-warm carries its own assertions and can fail, so it is a subject. If it were ever silently removed, every gate below would quietly inherit the timeout it exists to prevent — and the failure would look exactly like the red it just fixed.

On provenance: this work came from an agent that died on an authentication error after committing but before pushing. I recovered the commit, rebased it twice as main moved beneath it, and pushed. Since it never reached lint, I verified the workflow change here instead — test.yml parses at 28 jobs, the pre-warm sits at step 10 with all seven gates at 11–17, and ci_plan.py --self-test passes.

#10804 rode along: six areas, 197 fixtures weighted to its Ptr<Shape> surface, zero unexplained regressions.