Description
Problem
e2e-test-hooks is a cargo feature that compiles two scripted-failure seams into rocm:
engines/lemonade/src/lib.rs:435— makes the lemonade backend install fail on demand (with the env-var name declared at:43)apps/rocm/src/main.rs:5005—scripted_backend_failure, which lets the no-GPU pre-flight be bypassed so the run can reach the install phase at all
It exists to serve one scenario: @id:serve-lemonade-preparation-recovery (serve-18), which is tagged @requires-no-gpu and therefore runs on the blocking every-PR lane.
The cost of that one scenario is spread across the repo:
| site | count |
|---|---|
feature declarations (apps/rocm, engines/lemonade Cargo.toml) |
2 |
#[cfg] / cfg! seams |
3 |
--features rocm/e2e-test-hooks in workflows (nightly.yml x5, e2e-selfhosted.yml x5) |
10 |
xtask/src/e2e.rs:96 build flag |
1 |
| contract tests asserting a lane must be hook-ful | 2 |
| contract test asserting a lane must be hook-free | 1 |
defensive assertion in tests/e2e-cucumber/tests/e2e/serving_steps.rs:946-966 |
1 |
Two consequences follow from the feature existing at all.
Binaries under test are not the binaries shipped. Every lane that builds with the feature packages and installs a binary that differs from a release build. #342 fixed this for the Windows lifecycle lane. It is still true on Linux: build-and-test runs cargo build --workspace in the dev profile only, so the release build happens inside cargo xtask e2e — with the feature — and the Linux lifecycle lane installs a hook-carrying binary through the real installer.
The contract tests encode the asymmetry instead of removing it. One test asserts the Windows lane must be hook-free, two assert self-hosted and nightly must be hook-ful, and ci.yml's Linux lane is asserted by neither. That gap is the same one that hid the duplicate-build drift #342 fixed, one lane over. Adding a fourth contract test would make the asymmetry permanent rather than fix it.
Proposal
Delete the feature. The enabling step is to stop requiring serve-18 to run on a GPU-less lane:
- Retag serve-18 from
@requires-no-gputo@requires-gpu. - Drive its failure with a planted fake
lemond/lemonadepair plus a hand-written install manifest, instead of a compiled-in seam. This is viable becauseresolve_runtime()(engines/lemonade/src/lib.rs:1294-1304) checks onlyread_manifest()andmanifest.lemond.is_file()— it does not verify a SHA, version, or signature — andserve_http()reachesensure_best_llamacpp_backendat:626without going throughprepare_embeddable, so the SHA-pinned archive download is never on this path. - Delete both seams, the feature declarations, the 11 build-flag sites, and the three contract tests, which all become statements about a feature that no longer exists.
Approaches already ruled out, so they are not re-proposed:
- Mock the failing request. The install failure boundary is a subprocess (
run_lemonade_backend_install,:2786), not HTTP.MockServermocks a layer that does not exist yet at that point in the run. - Prewarm the cache on hosted lanes.
$RUNNER_WORKSPACEis ephemeral on hosted runners, and on the prewarmed self-hosted lanesxtask e2e-prewarmalready ranrocm engines install, so there is nothing left to fail. - Drop the network (e.g. sandlock). Wrong layer on hosted runners — the archive download fails first, before the backend install — and skipped entirely on prewarmed lanes.
- Mock
verify_sha256. Any such mechanism is itself a seam, and it moves the failure mode from fail-closed to fail-open inside the supply-chain verification path. See AGENTS.md section 9.
Tradeoff
Serve-18 loses every-PR coverage and moves to a gated lane. That is a real cost and the reason the seams were written in the first place; it should be an explicit maintainer decision, not a side effect of a cleanup. The counter-argument is that the coverage it currently provides on the every-PR lane is coverage of a code path that only exists in test builds, and that the scenario is rare enough that gated-lane latency is acceptable.
Out of scope
- Sharing built binaries across the Linux
build-and-testandE2E testsjobs (~1m25 each). Pure perf, independent of this feature. - ci: cache rustc with sccache on compile-heavy jobs #125 (Windows sccache, 1299s -> 418s). Separately approved and mergeable; still the largest available build-time win and needs no new engineering.
Follow-up to #342, which raised this in review.
Activity
rominf commentedon Sep 14, 2026
Heads-up for whoever picks this up: #380 landed a new consumer of
e2e-test-hooks, so the surface this issue proposes to delete has grown since it was filed.It adds a logical-clock seam to the dashboard daemon —
#[cfg(feature = "e2e-test-hooks")]plus aROCM_CLI_DASH_TEST_CLOCK_OFFSET_PATHenv var — to makedash-08/dash-09deterministic (#379). It is cleanly gated and compiled out in production builds, so it is not a correctness concern; it just means removing the feature is no longer only about the Lemonade backend-install seam.Concretely, this issue now has to answer an extra question: how do
dash-08/dash-09stay deterministic without the feature? Deleting it as-is re-exposes them to the wall-clock race that #379 tracked, which reproduces reliably (see the recipe on that issue). Options are roughly to promote the clock seam to a non-test-gated injection point, to widen the window through the ordinarydashboard.daemon.instance_tick_secsconfig, or to move the boundary coverage down to the already-deterministicrocm-dash-coreobservation tests and keep only the wiring at E2E level.Worth folding into the plan before the retag-and-delete sequence, since it changes the order: the dash replacement has to land before the feature comes out, the same way the serve-18 retag does.
test(e2e): drive the Lemonade recovery scenario with a planted runtime
rominf commentedon Sep 22, 2026
Heads-up for anyone picking this up from the issue text: the enumeration above is no longer complete, and the central premise no longer holds.
This issue argues from "It exists to serve one scenario" and lists every site in a table. That was accurate when written. It stopped being accurate on 2026-09-14, when #380 ("test(dash): make throughput expiry deterministic") added a second, unrelated consumer at
apps/rocm/src/dash.rs, gating the daemon's logical observation clock behind the same feature.The consequence is not cosmetic. Deleting the feature per the proposal above silently turns that clock off, which breaks
dash-08(dash-gen-tps-held-after-scrape-failure) anddash-09(dash-gen-tps-expiry-boundary). Both are@requires-os:linux, so they run on the blocking every-PR lane — and they are the same two scenarios #412 had to stabilise after they went red onmain. So the naive reading of step 3 ("delete both seams") trades one gated-lane scenario for two blocking-lane regressions.One detail that makes the fix easier than it first looks:
RunnerOptions::test_clock_offset_pathinrocm-dash-daemonwas never feature-gated. It is a plainpubfield documented as test-only. Only the CLI's environment-variable read was gated, so the feature bought less isolation than the site count suggests — the daemon already ships that affordance in every build.#351 resolves this rather than working around it, by applying the same treatment to both seams under one principle worth stating here since it is the thing this issue is really about:
Under that principle the dash clock becomes a file at a fixed path inside the telemetry state dir, planted by the harness and never created by
rocm— the direct analogue of the planted Lemonade manifest this issue already proposes.One trap worth recording for whoever implements it: the path must be returned only when the file exists.
cycle_timestampselects wall time only onNone, so aSome(path)pointing at an absent file leaves the daemon on its defaultFreeRunning(0)logical clock. Simply ungating the old environment variable would therefore take production offUtc::now().No change requested to the proposal's direction — it is still the right call. This is only to stop the site table and the "one scenario" framing from under-scoping the work for the next reader.
test(e2e): drive the Lemonade recovery scenario with a planted runtime
test(e2e): drive the Lemonade recovery scenario with a planted runtime
test(e2e): drive the Lemonade recovery scenario with a planted runtime
test(e2e): drive the Lemonade recovery scenario with a planted runtime