Skip to content

Remove the e2e-test-hooks feature #349

Open

Description

@rominf

Problem

e2e-test-hooks is a cargo feature that compiles two scripted-failure seams into rocm:

  • engines/lemonade/src/lib.rs:435 — makes the lemonade backend install fail on demand (with the env-var name declared at :43)
  • apps/rocm/src/main.rs:5005 — scripted_backend_failure, which lets the no-GPU pre-flight be bypassed so the run can reach the install phase at all

It exists to serve one scenario: @id:serve-lemonade-preparation-recovery (serve-18), which is tagged @requires-no-gpu and therefore runs on the blocking every-PR lane.

The cost of that one scenario is spread across the repo:

site count
feature declarations (apps/rocm, engines/lemonade Cargo.toml) 2
#[cfg] / cfg! seams 3
--features rocm/e2e-test-hooks in workflows (nightly.yml x5, e2e-selfhosted.yml x5) 10
xtask/src/e2e.rs:96 build flag 1
contract tests asserting a lane must be hook-ful 2
contract test asserting a lane must be hook-free 1
defensive assertion in tests/e2e-cucumber/tests/e2e/serving_steps.rs:946-966 1

Two consequences follow from the feature existing at all.

Binaries under test are not the binaries shipped. Every lane that builds with the feature packages and installs a binary that differs from a release build. #342 fixed this for the Windows lifecycle lane. It is still true on Linux: build-and-test runs cargo build --workspace in the dev profile only, so the release build happens inside cargo xtask e2e — with the feature — and the Linux lifecycle lane installs a hook-carrying binary through the real installer.

The contract tests encode the asymmetry instead of removing it. One test asserts the Windows lane must be hook-free, two assert self-hosted and nightly must be hook-ful, and ci.yml's Linux lane is asserted by neither. That gap is the same one that hid the duplicate-build drift #342 fixed, one lane over. Adding a fourth contract test would make the asymmetry permanent rather than fix it.

Proposal

Delete the feature. The enabling step is to stop requiring serve-18 to run on a GPU-less lane:

  1. Retag serve-18 from @requires-no-gpu to @requires-gpu.
  2. Drive its failure with a planted fake lemond/lemonade pair plus a hand-written install manifest, instead of a compiled-in seam. This is viable because resolve_runtime() (engines/lemonade/src/lib.rs:1294-1304) checks only read_manifest() and manifest.lemond.is_file() — it does not verify a SHA, version, or signature — and serve_http() reaches ensure_best_llamacpp_backend at :626 without going through prepare_embeddable, so the SHA-pinned archive download is never on this path.
  3. Delete both seams, the feature declarations, the 11 build-flag sites, and the three contract tests, which all become statements about a feature that no longer exists.

Approaches already ruled out, so they are not re-proposed:

  • Mock the failing request. The install failure boundary is a subprocess (run_lemonade_backend_install, :2786), not HTTP. MockServer mocks a layer that does not exist yet at that point in the run.
  • Prewarm the cache on hosted lanes. $RUNNER_WORKSPACE is ephemeral on hosted runners, and on the prewarmed self-hosted lanes xtask e2e-prewarm already ran rocm engines install, so there is nothing left to fail.
  • Drop the network (e.g. sandlock). Wrong layer on hosted runners — the archive download fails first, before the backend install — and skipped entirely on prewarmed lanes.
  • Mock verify_sha256. Any such mechanism is itself a seam, and it moves the failure mode from fail-closed to fail-open inside the supply-chain verification path. See AGENTS.md section 9.

Tradeoff

Serve-18 loses every-PR coverage and moves to a gated lane. That is a real cost and the reason the seams were written in the first place; it should be an explicit maintainer decision, not a side effect of a cleanup. The counter-argument is that the coverage it currently provides on the every-PR lane is coverage of a code path that only exists in test builds, and that the scenario is rare enough that gated-lane latency is acceptable.

Out of scope

  • Sharing built binaries across the Linux build-and-test and E2E tests jobs (~1m25 each). Pure perf, independent of this feature.
  • ci: cache rustc with sccache on compile-heavy jobs #125 (Windows sccache, 1299s -> 418s). Separately approved and mergeable; still the largest available build-time win and needs no new engineering.

Follow-up to #342, which raised this in review.

Activity

  1. rominf commented on Sep 14, 2026

    @rominf
    CollaboratorAuthor

    Heads-up for whoever picks this up: #380 landed a new consumer of e2e-test-hooks, so the surface this issue proposes to delete has grown since it was filed.

    It adds a logical-clock seam to the dashboard daemon — #[cfg(feature = "e2e-test-hooks")] plus a ROCM_CLI_DASH_TEST_CLOCK_OFFSET_PATH env var — to make dash-08/dash-09 deterministic (#379). It is cleanly gated and compiled out in production builds, so it is not a correctness concern; it just means removing the feature is no longer only about the Lemonade backend-install seam.

    Concretely, this issue now has to answer an extra question: how do dash-08/dash-09 stay deterministic without the feature? Deleting it as-is re-exposes them to the wall-clock race that #379 tracked, which reproduces reliably (see the recipe on that issue). Options are roughly to promote the clock seam to a non-test-gated injection point, to widen the window through the ordinary dashboard.daemon.instance_tick_secs config, or to move the boundary coverage down to the already-deterministic rocm-dash-core observation tests and keep only the wiring at E2E level.

    Worth folding into the plan before the retag-and-delete sequence, since it changes the order: the dash replacement has to land before the feature comes out, the same way the serve-18 retag does.

  2. added a commit that references this issue on Sep 22, 2026
    1c2600b
  3. rominf commented on Sep 22, 2026

    @rominf
    CollaboratorAuthor

    Heads-up for anyone picking this up from the issue text: the enumeration above is no longer complete, and the central premise no longer holds.

    This issue argues from "It exists to serve one scenario" and lists every site in a table. That was accurate when written. It stopped being accurate on 2026-09-14, when #380 ("test(dash): make throughput expiry deterministic") added a second, unrelated consumer at apps/rocm/src/dash.rs, gating the daemon's logical observation clock behind the same feature.

    The consequence is not cosmetic. Deleting the feature per the proposal above silently turns that clock off, which breaks dash-08 (dash-gen-tps-held-after-scrape-failure) and dash-09 (dash-gen-tps-expiry-boundary). Both are @requires-os:linux, so they run on the blocking every-PR lane — and they are the same two scenarios #412 had to stabilise after they went red on main. So the naive reading of step 3 ("delete both seams") trades one gated-lane scenario for two blocking-lane regressions.

    One detail that makes the fix easier than it first looks: RunnerOptions::test_clock_offset_path in rocm-dash-daemon was never feature-gated. It is a plain pub field documented as test-only. Only the CLI's environment-variable read was gated, so the feature bought less isolation than the site count suggests — the daemon already ships that affordance in every build.

    #351 resolves this rather than working around it, by applying the same treatment to both seams under one principle worth stating here since it is the thing this issue is really about:

    The goal is no compile-time divergence between the binary under test and the binary shipped. It is not "no test affordances at all". An affordance present identically in every build, which activates only on state a user's machine never has, satisfies the goal; a #[cfg(feature)] does not, because it makes the tested binary a different binary.

    Under that principle the dash clock becomes a file at a fixed path inside the telemetry state dir, planted by the harness and never created by rocm — the direct analogue of the planted Lemonade manifest this issue already proposes.

    One trap worth recording for whoever implements it: the path must be returned only when the file exists. cycle_timestamp selects wall time only on None, so a Some(path) pointing at an absent file leaves the daemon on its default FreeRunning(0) logical clock. Simply ungating the old environment variable would therefore take production off Utc::now().

    No change requested to the proposal's direction — it is still the right call. This is only to stop the site table and the "one scenario" framing from under-scoping the work for the next reader.

  4. added a commit that references this issue on Sep 24, 2026
    13dd7b3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions