Skip to content

test(e2e): drive the Lemonade recovery scenario with a planted runtime (ROCMAI-669) - #351

Open
rominf wants to merge 10 commits into
mainfrom
chore/remove-e2e-test-hooks
Open

rominf wants to merge 10 commits into
mainfrom
chore/remove-e2e-test-hooks

Conversation

@rominf

@rominf rominf commented Sep 7, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #349. This converts one consumer of the e2e-test-hooks feature, the Lemonade recovery scenario, and does not remove the feature: it still has other consumers, among them the simulated host root (#510) that docs/release-trust.md relies on being compiled out of release builds, so removing it is the last step of #349.

Problem

@id:serve-lemonade-preparation-recovery (serve-18) needs Lemonade's llama.cpp backend install to fail twice, so it can pin the retry count and the recovery command. It got that failure from two #[cfg(feature = "e2e-test-hooks")] branches in product code:

  • engines/lemonade/src/lib.rs failed the Install RPC when ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE was set;
  • apps/rocm/src/serve_cmd.rs waived rocm serve's no-GPU pre-flight for the same variable.

So the scenario tested a code path that exists only in test builds.

Change

Both branches are deleted. The failure now happens where a broken install fails: at a non-zero child process. The pre-flight's other E2E waivers (the scripted free-space override and the OOM-launch fault injection) are untouched.

  • The runtime. fake-lemonade is a new [[bin]] of the suite, planted as the runtime's lemonade and lemond:

    • lemonade backends reports llamacpp:rocm as installable;
    • lemonade backends install exits non-zero and prints a fixed refusal line;
    • each call is classified by its subcommand, the first word after the global --host/--port options, not by a word anywhere in its arguments (unit-tested in the bin);
    • lemond stays alive until its parent is gone (Unix re-parenting) or a 10-minute cap passes, so a failed teardown leaves nothing running.

    The install manifest is written beside it. serve skips its own install only when Detect reports exactly lemonade-embeddable-<pinned version>, and anything else silently starts the real multi-gigabyte install, so the planted env_id takes the version from the workspace pin file runtime-deps.toml, read as data. The suite stays black-box, with no rocm-deps import, and a version bump needs no copy updated.

  • --wait. serve-18 runs rocm serve --managed --wait. The install fails inside the spawned engine process, and --wait makes a death at any point fail the serve. An engine that dies within the first 200 ms fails it with its log tail; one that dies later fails it with a pointer to rocm logs --service <id>, which the step follows, as a user would, before asserting the retry and recovery lines.

  • Proof the fake was used. serve-18 asserts the fake's refusal line appears exactly twice, through a constant shared by the fixture and the step. If the planted env_id ever stopped matching Detect, the scenario fails fast instead of downloading.

  • The GPU. The pre-flight runs unchanged, against one KFD GPU node planted under the simulated host root (ROCM_CLI_TEST_HOST_ROOT). The root holds nothing else, so the scenario needs no GPU, and a GPU runner's own devices do not take part. XDG_RUNTIME_DIR also points inside the scenario, so the fake server writes nothing outside it.

  • The gate. @requires-no-gpu → @requires-os:linux.

  • The planted env_id. The engine's manifest writer and rocm's managed_engine_runtime_id both build the id through a new rocm_deps::lemonade_env_id, so the two sides of that exact-string comparison cannot drift apart. The suite repeats the shape, since it imports nothing from the workspace; a comment names the helper it mirrors.

  • The backends table is a fixture file, tests/e2e-cucumber/fixtures/lemonade_backends_table.txt. The fake prints it, and the engine's unit test reads the same file with include_str!, so the table the scenario relies on is also checked against the engine's parser on every unit-test lane. The pre-push hook's CROSS_CRATE_READS lists it, so editing it runs rocm-engine-lemonade's tests.

Comments in the workflow, the contract test, serve_cmd.rs and docs/testing.md that cited the deleted seam now cite what remains. docs/architecture.md and the suite README list the two new files.

Why this doesn't weaken verification

resolve_runtime accepts the planted manifest because it only checks that the manifest parses and that lemond is a file. No checksum, version or signature is on this path; the sha-pinned archive download lives in prepare_embeddable, which serve does not reach once Detect matches. The no-GPU pre-flight is no longer waived for this scenario. It passes for the same reason it passes on a real GPU host.

Known gap, not fixed here

Without --wait, rocm serve --managed notices a managed engine that dies only within its 200 ms post-spawn check, on every platform. An engine that dies later is reported as readiness: starting with exit 0 once the 45 s readiness probe gives up, and the failure is visible only in the service log. Reproduction on Linux: plant a Lemonade runtime whose backends install fails after a 1 s delay (serve-18's fixture with a sleep added), then run rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed. It prints readiness: starting and exits 0 after about 45 s (measured: 45.3 s). The same death with --wait fails the serve. serve-18 uses --wait for that reason, and so does not cover the default path. The default-path defect needs its own fix.

Test plan

  • serve-18 passes locally via cargo xtask e2e on Linux on both failure paths: with the real fixture (the engine dies inside the 200 ms window; the serve step takes 0.2 s) and with the fixture's install delayed by 1 s (the engine dies after it; the serve step takes 45 s and the recovery lines come from rocm logs). Before --wait, the second path failed with serve unexpectedly succeeded, and so did the first on this host under load.
  • Mutation checks:
    • fake install succeeds: the scenario fails, with no retry announcement;
    • GPU not planted: the scenario fails with its named pre-flight message;
    • fixture table reports installed: the engine unit test fails;
    • tag changed to @requires-no-gpu or removed: the tag pin fails;
    • the fake's status match moved from the subcommand position to the last argument: its classification test fails.
  • Gate: fmt; both clippy invocations; cargo test --workspace --all-targets; xtask check-architecture-doc, check-crate-edges, check-module-size, manifest --check.
  • Not verified locally: the scenario on a GPU host or under WSL2 (left to the MI300X and WSL2 lanes). cargo test --workspace has three failures on this host in files the change does not touch, all from the sandbox: two build.rs git tests (global commit signing inside temp repos) and one AIM docker stop test (podman installed as docker). cargo xtask smoke passes every check up to its bubblewrap sandbox step, which this host cannot run (no network namespace).

Related

@rominf

rominf commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main (6ad833f1) and repaired. Head is now 1c2600ba, a single commit.

What the rebase dropped

The two ci(windows) commits at the base of this branch, both fully landed on main as da157ca3 (#342). Checked against main's files rather than the commit messages — main's version is a strict superset: same $targetDir block in ci.yml, plus two assertions this branch's copy lacked (it pins both exported binary paths, and pins the Test-Path/throw fail-fast guard). Nothing was unlanded, so nothing was carried forward.

That also retires the "Depends on #342" note at the top of the description — #342 has merged.

What the rebase preserved

Four conflicts, each resolved by keeping both sides rather than taking one:

sccache (#125) is untouched: 87 occurrences in ci.yml on both main and this head.

Three things needed updating because main moved underneath the original patch:

  • The MI350P lanes in e2e-selfhosted.yml and nightly.yml (ci(e2e): add an MI350P self-hosted E2E lane #341) still passed --features rocm/e2e-test-hooks. Left alone they would have failed the repurposed contract test.
  • install_lifecycle.feature's CONSTRAINT comment described the hook-carrying/hook-free lane split this PR abolishes, so it is removed.
  • The dashboard clock — see below.

The dash clock was a second consumer

#380 added a #[cfg(feature = "e2e-test-hooks")] gate in apps/rocm/src/dash.rs after this branch was cut, so deleting the feature broke clippy -D warnings (unexpected cfg condition value) and would have silently disabled the deterministic clock for dash-08/dash-09 on the blocking lane.

Rather than keep the feature alive for it, the same technique this PR already uses for Lemonade is applied to dash. The principle: what must not diverge between the tested and the shipped binary is the compile. That is not the same as "no test affordances" — a path- or state-based affordance is present identically in every build and inert until something plants the state, whereas a #[cfg(feature)] makes the tested binary a different binary.

So rocm dash now reads the directive from <ROCM_CLI_DATA_DIR>/telemetry/test-clock-offset, inside the data root it already resolves, and the harness plants it exactly as it plants the Lemonade manifest. No env var, no feature. The daemon side needed no change — RunnerOptions::test_clock_offset_path already ships unconditionally.

The existence check there is load-bearing rather than an optimisation: that field selects wall time only on None, so handing the daemon a path whose file is absent would leave production on the default FreeRunning(0) logical clock.

Why serve-18 was failing

Not infrastructure — a real defect in this PR. All three red lanes reported the same single unexpected failure, and the logs showed the planted runtime being ignored while a real multi-gigabyte backend install ran and succeeded, so serve returned 0.

Before serve prepares an engine it calls Detect and skips its own install only when the reported env_id equals the lemonade-embeddable-{LEMONADE_VERSION} that managed_engine_runtime_id builds. The planted manifest carried the sentinel "e2e-planted", which does not fail loudly — it just lets serve fall through to the real install. The manifest now derives that id from rocm_deps::LEMONADE_VERSION rather than copying it, so a version bump cannot reintroduce the fallthrough.

With the install skipped, serve_http resolves the planted runtime and reaches ensure_best_llamacpp_backend, whose backends install runs the fixture and fails at a non-zero child process — which is the premise the scenario asserts.

The planting location was already correct: this scenario does not opt into the shared runtime tree, so the CLI resolves no env root and lemonade_root is the isolated engine dir. (Planting into the shared tree would have been wrong anyway — other scenarios serve from it.)

Status of the three previously-red lanes

Lane Before Now
E2E tests (GPU) → now E2E tests (MI300X) 1 unexpected failure green — serve-18 passes, 0 unexpected failures
E2E tests (rad3 R9700) 1 unexpected failure green — serve-18 passes, 0 unexpected failures
E2E tests (Strix Halo, Windows) 1 unexpected failure still queued at time of writing

Also green on this head: E2E tests (mock lane), E2E tests (MI350P) — serve-18 passes there too — and E2E tests (Strix Halo, WSL2). Every required check is green, including clippy, windows-build-and-test, Test (affected crates) and Commit signatures + sign-off. E2E tests (Strix Halo, Ubuntu) is also still queued.

Verified locally

cargo clippy --workspace --all-targets -- -D warnings and cargo clippy -p e2e-cucumber --test e2e -- -D warnings clean; cargo fmt --all --check clean; cargo check -p e2e-cucumber --test e2e explicitly (that target is test = false); feature_naming 4/4, so no stale @id/index rows after the rebase; xtask 138/138.

dash-08 and dash-09 were run on this Linux host and pass, as does the whole dash feature (16 scenarios, 102 steps, 0 failures). Worth noting that a local run cannot reproduce the concurrency of the mock lane, which is the load that made these two flaky before — the lane itself is the authority, and it is green on this head.

cargo test --workspace --all-targets is clean apart from three rocm-dash-tui agent tests failing with cannot create token cache dir: Permission denied. That is a sandbox filesystem restriction in my environment, in a crate this diff does not touch. I did not verify them against an unmodified main checkout, so I am reporting the cause rather than asserting they are pre-existing.

Not verified locally: serve-18 itself — this host has no usable AMD GPU, which is precisely why the scenario is now @requires-gpu. The three GPU lanes above are what exercised it.

@rominf
rominf marked this pull request as ready for review September 22, 2026 13:12
@rominf
rominf requested a review from a team as a code owner September 22, 2026 13:13
@rominf
rominf requested a review from tomastola September 22, 2026 13:13

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran a code review pass on this in addition to my own read of the diff, and it surfaced two things worth looking at before merge.

The main one: dash_test_clock_offset_path in apps/rocm/src/dash.rs now checks for a file on every rocm dash launch, in every build, not just E2E builds. If that file (telemetry/test-clock-offset) ever ends up in a real user's data directory (synced or copied from a dev or CI machine, some future bug that touches that path), the daemon silently drops onto a frozen logical clock instead of wall time. There's no log line when this happens, only a doc comment saying it should never occur outside the E2E harness. A one line warn log when the file is found and used would make this fail loudly instead of silently, without touching the test seam design itself.

The second one is minor, and actually already flagged by the author in a comment: the clock file name and its data/telemetry subpath are duplicated as literals between dash.rs and the E2E's dash_steps.rs, kept in sync only by that comment. Not asking to block on this one, just noting it since the review turned it up.

CI is otherwise clean aside from the Strix Halo Windows and Ubuntu self hosted lanes, which I checked are flaky on main independent of this PR (3 of the last 4 completed main runs failed or were cancelled on those two lanes), so that's not a signal against this change.

Comment thread apps/rocm/src/dash.rs Outdated
/// `TestClockDirective::FreeRunning(0)` logical clock. Returning the path
/// unconditionally would therefore take production off `Utc::now()`.
fn dash_test_clock_offset_path(paths: &AppPaths) -> Option<PathBuf> {
let path = paths.telemetry_state_dir().join(DASH_TEST_CLOCK_FILE);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every build now takes this path, not just E2E builds. If telemetry/test-clock-offset ever exists in a real user's data dir, the dashboard silently freezes its telemetry clock instead of using wall time, with nothing logged. Worth a warn log when this file is found and used, so a stray file fails loudly instead of silently skewing displayed timestamps.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and fixed in 7841012 — this is the inherent cost of the state-based seam and it should not be paid silently.

dash_test_clock_offset_path now logs at WARN when it finds the file, naming the path:

WARN rocm::dash: dashboard telemetry clock is in TEST mode, not wall time: displayed
timestamps come from a logical clock read from this file. Only the E2E harness creates
it; delete it to restore wall-clock timestamps.
  test_clock_file=/…/telemetry/test-clock-offset

Why tracing and not stderr. logging.rs makes the client subscriber file-only on purpose — the TUI owns the raw-mode terminal and a stray stdout/stderr write corrupts the display. The only production caller of this function is maybe_spawn_embedded_daemon on the rocm dash path, so stderr is exactly the wrong channel here; a line printed there would also land on the PTY grid that the dash-* scenarios parse and assert against. The WARN lands in ~/.rocm/logs/rocm-cli.log.<date>, which is where you would look when investigating "my timestamps are wrong" — and it survives the alt-screen switch, which a pre-launch eprintln would not.

Verified live, not just by unit test. Planting telemetry/test-clock-offset under a throwaway ROCM_CLI_DATA_DIR and launching the release rocm dash under a pty produced exactly the record above in the client log; the same launch with no file planted produced an empty log. logging::init runs at the top of run(), before subcommand dispatch, so the subscriber is installed before this callsite is reached, and DEFAULT_FILTER (warn,rocm=info,…) admits the rocm::dash target.

Tests. Two, using a thread-local capturing subscriber (tracing::subscriber::with_default, so a concurrently-running test cannot see or pollute the records): one asserts the WARN fires and names the file when it is present, one asserts nothing at all is logged when it is absent — a warning on every launch would be noise and would drain the signal out of the first.

Both were mutation-probed rather than just observed green: deleting the tracing::warn! turns the first test red (expected a WARN record, got: ), and moving the warn above the existence check so it fires unconditionally turns the second red. So neither passes vacuously.

The existence check itself is unchanged — cycle_timestamp picks wall time only on None, so returning Some for an absent file would put production on FreeRunning(0). Only the logging is new.

/// `<ROCM_CLI_DATA_DIR>/telemetry/test-clock-offset` and falls back to wall time
/// when it is absent, so a rename on either side silently drops the whole
/// mechanism rather than failing loudly.
const DASH_CLOCK_OFFSET_FILE: &str = "test-clock-offset";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This risk is already spelled out in the comment above (rename on either side silently breaks the mechanism). Not blocking, just flagging that the fix (a shared constant, or a debug_assert tying the two together) is still open.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closed this one too, with the debug_assert-style option rather than the shared constant — reasoning, since you offered both:

A shared constant is not reachable here. apps/rocm is a binary crate, so the suite cannot import DASH_TEST_CLOCK_FILE at all; making it importable means hosting it in a library and adding a first-party edge from e2e-cucumber to that library. #422 landed xtask check-crate-edges on main while this branch sat, which deliberately makes every such edge a reviewed decision — spending one on a single string, to couple the black-box suite to a production crate it otherwise doesn't link, seemed like the wrong trade.

So instead dash::tests::e2e_harness_plants_the_file_rocm_dash_reads in apps/rocm/src/dash.rs reads the step file's source with include_str! and pins the literals to what rocm dash actually resolves. That is the same cross-tree reach the suite already makes (installer_fixture.rs does it for install.ps1), it adds no dependency, and it runs in the every-PR unit lane rather than only where the dash scenarios run.

It checks two things:

  • the harness's DASH_CLOCK_OFFSET_FILE declaration parsed out of the source, compared against DASH_TEST_CLOCK_FILE — parsed, not substring-matched, so an unrelated constant containing the same text can't satisfy it;
  • every directory between the data root and the file, derived from AppPaths::telemetry_state_dir() and required to appear in order inside dash_clock_path's own body. Anchoring on the builder rather than the whole file matters: a stray .join("telemetry") elsewhere in those ~1.2k lines would otherwise satisfy it. Taking the whole relative path rather than its last component matters too — telemetry_state_dir() being moved (say to data/dash/telemetry) leaves the leaf name intact, and an earlier version of this test passed under that mutation while the mechanism was broken.

Mutation-probed in all four directions: renaming the constant on the production side, renaming it on the harness side, relocating telemetry_state_dir(), and moving the .join out of the builder while leaving a decoy elsewhere in the file — each fails with a message naming the mismatch.

It does impose two constraints on that file, both written into the comment above the constant: keep the declaration on one line, and keep dash_clock_path building the path from .join(..) links. If either is broken the guard fails loudly with an explanatory message rather than passing silently, which is the failure mode I wanted.

One related thing the rebase surfaced, since it is the same gate: this PR's own e2e-cucumber -> rocm-deps edge (the pinned Lemonade version behind the planted runtime's env_id) was not in the allowlist, so check-crate-edges failed on the rebased branch even though the rebase itself was conflict-free. That entry is added in 7841012.

@rominf
rominf force-pushed the chore/remove-e2e-test-hooks branch from 1c2600b to 7841012 Compare September 24, 2026 08:05
@rominf

rominf commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — both findings were right, and the first one is the real cost of choosing a state-based seam over a compile-time one, so it shouldn't be paid silently. Pushed as 7841012, on top of a rebase onto current main.

The warn log. dash_test_clock_offset_path now logs at WARN when it finds the file, naming the path and saying the dashboard clock is in test mode rather than wall time. It goes through tracing (file-only client log) rather than stderr, because the only caller sits on the rocm dash path where the TUI owns the terminal — details and the live end-to-end evidence are in the thread. The existence check itself is untouched.

The duplicated literals. Took the debug_assert-style option rather than the shared constant: apps/rocm is a binary crate, so a shared constant would mean a new first-party crate edge, which #422's check-crate-edges guard (landed on main while this sat) exists to make a reviewed decision. A unit test reads the step file's source instead and pins both literals — anchored inside the harness's own path builder, and covering the whole directory chain, so a relocation is caught as well as a rename. Reasoning and the mutation probes are in that thread.

Rebase. Onto 8788394c, conflict-free — but not therefore free. #224's removal of the WSL setup script survives (the deliberate "no ROCDXG provisioning here" notes in both WSL lanes are intact, and the only remaining references to the script are the assertions that it must not be named), and git grep e2e-test-hooks finds no #[cfg(feature = ...)] anywhere — only two doc-comment mentions, both added by this PR to explain what replaced the seam. What the clean rebase did hide is that this PR's own e2e-cucumber -> rocm-deps edge was missing from the new allowlist, so cargo xtask check-crate-edges failed on the rebased branch; that entry is added here.

Ran locally on this head: dash-08 and dash-09 green (2 scenarios, 22 steps, 0 unexpected failures) — they are on the blocking every-PR lane, so I didn't want to push a dash change without executing them. Plus both CI clippy invocations, fmt --check, cargo check --workspace --all-targets and -p e2e-cucumber --test e2e separately, xtask manifest --check, xtask check-crate-edges, smoke_local.py, and the workspace test suite.

Also closed out one loose end from the previous description: the three rocm-dash-tui agent-test failures in my sandbox are now established as pre-existing rather than assumed — the same three fail identically on a clean checkout of origin/main.

Leaving both threads open for you to close.

@rominf

rominf commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator Author

CI status correction — two things changed after this review was written

Flagging this because the review reasonably discounted the self-hosted lanes, and both facts behind that have since moved. Neither was wrong at the time.

1. E2E tests (Strix Halo, Windows) became a required check today (2026-09-24). When this review was filed it was advisory, so discounting it was correct then. It is now in branch protection's required-status-check list, so a failure there blocks the merge.

2. That lane has since gone green on main three times running. The review noted 3 of the last 4 completed main runs failed or were cancelled on these lanes — accurate when written. The precise picture for the Windows lane specifically:

main run Strix Halo, Windows
2026-09-23 07:59 → 15:06 (5 runs) never completed (runs cancelled by supersession)
2026-09-23 17:08 success
2026-09-24 05:48 success
2026-09-24 07:10 success

The first completed result landed about three hours after this review. So the flakiness read was sound at the time and simply predates the evidence.

The Ubuntu lane is a different story and the original read still holds there: it fails at GPU preflight under contention, is red on main today, and is not required. Not chasing it.

What that means for this PR

E2E tests (Strix Halo, Windows) is failing here on a real regression in this PR's own scenario, not lane noise:

  • serve-18 / @id:serve-lemonade-preparation-recovery reports serve unexpectedly succeeded — rc 0, empty stderr.
  • The backend install is never attempted, so the failure the scenario exists to assert never happens. The one Installing Lemonade llamacpp:rocm backend... line in the log is the lane's pre-warm step, not the scenario.
  • It is Windows-specific: E2E tests (MI300X) and E2E tests (rad3 R9700) are green on this same head, so it is neither a rebase-wide break nor a side effect of the two review fixes (a warn! in dash.rs, #[cfg(test)] tests, a comment, and one crate-edge allowlist entry — none reachable from rocm serve).

This was invisible until now because the Windows lane was cancelled on the previous head, so this PR has never had Windows data for the planted-runtime rework.

Working hypothesis, not yet confirmed: #404 changed llamacpp:rocm backend alignment to depend on the active ROCm SDK, and this scenario's planted runtime deliberately reports llamacpp rocm as installable to make the engine attempt an install. If that alignment now short-circuits when no SDK is active, the install would be skipped exactly as observed. Investigating that interaction now — I will report back either way rather than leaving it as a guess.

Not proposing to retag or weaken the scenario while the cause is unknown.

@rominf

rominf commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: my #404 hypothesis was wrong, and the real cause is a pre-existing Windows defect

Correcting the guess I posted above before it misleads anyone: #404 is not involved. It touched crates/rocm-core/src/lib.rs, docs/engine-plugins.md, engines/lemonade/src/lib.rs, runtime_setup.feature and runtime_steps.rs — not apps/rocm/src/main.rs, where the divergence actually lives. Apologies for pointing at it; I flagged it as unconfirmed and it did not survive contact with the code.

The real mechanism

spawn_managed_engine_child in apps/rocm/src/main.rs is asymmetric across platforms, and has been since the initial import — months before any recent change:

  • non-Windows: command.spawn() → sleep(200ms) → child.try_wait(); if the child already exited it bail!s with managed_engine_startup_failure_detail(status, &record.log_path), which tails the child's service log into the parent's error.
  • Windows: rocm_core::spawn_detached_no_inherit(...), which returns a bare PID. There is no Child, and therefore no liveness or exit check whatsoever.

After the spawn the parent waits up to 45s for HTTP readiness, records the status, and returns Ok(...) regardless. So on Windows a managed engine that dies during startup yields exit code 0 with readiness: starting — exactly the observed serve unexpectedly succeeded, rc 0, empty stderr, ~48s.

This is a real user-visible defect on main, independent of this PR: on Windows, rocm serve reports success when the managed engine died at startup.

Why this PR surfaced it

The deleted e2e-test-hooks seam injected its failure at the top of install_response — inside the Install RPC, which ensure_self_managed_engine_ready calls in the parent before any spawn. It failed synchronously on every platform, so the Windows gap was never exercised.

The planted runtime deliberately moves the failure to the fake's backends install, which is reached from the engine's Serve path — inside the detached child. serve-18 therefore now depends on managed-spawn startup-failure detection, which Windows does not implement. The scenario is correct; it is asserting behaviour the platform does not provide.

Why there is no test-side fix

Both seam-side options are closed, checked rather than assumed:

  • Run serve in the foreground. run_attached_service calls the same spawn_managed_engine_child. Every rocm serve path on Windows goes through the same detached primitive.
  • Make the failure happen in the parent's Install RPC instead. install_response → prepare_embeddable_with calls ensure_cached_archive(..., expected_sha256, download) before its needs_extraction check, so the Install RPC always demands the sha256-pinned embeddable archive. A planted runtime cannot forge that, and forcing this path would trigger the real multi-gigabyte download this design exists to avoid.

What I am not doing

I am not retagging or weakening serve-18 while the cause is understood but unfixed, and I am not bolting a production Windows fix onto a test-infrastructure PR — that change deserves its own PR, its own tests, and a maintainer with a Windows host.

Flagging it here for a maintainer decision rather than picking one unilaterally. Happy to open a separate issue for the spawn_managed_engine_child gap if that is wanted.

Observed vs inferred, stated plainly: every file/line and git fact above is observed. That the fake's backends install failure occurs inside the detached child and is invisible to the parent is inferred from the two branches plus the rc 0 / empty-parent-stderr evidence — I have no Windows host and did not read the child's service log, which the job log does not include.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 7841012

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Deletes the e2e-test-hooks cargo feature of the rocm crate and both seams it gated, replacing them with state-based affordances: the Lemonade recovery scenario now drives a planted fake runtime whose backends install exits non-zero, and rocm dash reads its logical-clock directive from <data>/telemetry/test-clock-offset (warning at WARN when present) instead of a compiled-in env-var branch. No blocking findings: the code itself is in good shape, and the two red checks at this head are recorded as an observation rather than as a finding of mine, since I could not attribute either to this diff. Verified: in a scratch copy of the tree I ran the dash unit-test module (22/22 pass, including all three new tests) plus two mutation experiments — renaming the production DASH_TEST_CLOCK_FILE constant, and relocating the harness's telemetry path segment — and the new cross-file guard correctly fails on both, so it is not vacuous; I separately confirmed AppPaths exposes exactly the three fields the new test constructs and telemetry_state_dir() is data_dir/telemetry, that managed_engine_runtime_id("lemonade") builds exactly the lemonade-embeddable-{LEMONADE_VERSION} string the planted manifest writes, that LemonadeInstallManifest's eight fields match the planted JSON name-for-name with no serde renames, that the engine's path helpers resolve to where the harness plants, that Duration::from_mins compiles on the pinned 1.96.0 toolchain, that the feature deletion leaves no live references anywhere in the tree, and that the workflow-contract and crate-edges checks pass; leak and prompt-injection scans of the diff were clean. The full suite, the e2e suite and anything needing a GPU were not run here. CI conclusions this review worked from: 26 success, 1 skipped, 2 failure. Blocking: 0 · Non-blocking: 6.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • Two checks are failing at this head and one is pending. I could not determine whether the failures are real regressions from this change or infrastructure noise, and I am deliberately not naming or guessing at jobs — but the change touches several workflow files and the contract tests that police them, so a lane-level failure is exactly the shape a regression here would take, and the two logs are worth reading directly rather than assuming they are unrelated.

  • engines/lemonade/src/lib.rs:~6806 — e2e_fake_runtime_table_drives_a_rocm_backend_install hardcodes a copy of the fixture's backends table rather than reading it, so editing the table in tests/e2e-cucumber/src/bin/fake-lemonade.rs would break the now-GPU-gated scenario while this test stays green; the doc comment's "keep this table byte-identical" is an unenforced instruction, which is precisely the pattern this same PR mechanically eliminates on the dash side — mirror that by include_str!-ing the fixture and extracting its printed rows.

  • apps/rocm/src/dash.rs:105 — the new WARN goes to tracing, which on the rocm dash path lands in a log file, so a user staring at skewed timestamps still sees nothing on screen; the terminal-corruption reasoning is sound, but a line in the TUI's own status area would actually reach the person the warning is for.

  • tests/e2e-cucumber/features/model_serving.feature:187 — nothing asserts this scenario still carries @requires-gpu, so a future tag edit would silently change which lane runs it; the repo already has the precedent for pinning a tag against the real feature file in tests/e2e-cucumber/src/expectation.rs.

  • engines/lemonade/src/lib.rs:~642 — pre-existing, but this PR makes the path routine: the ensure_best_llamacpp_backend failure returns via ? without terminating the spawned lemond, unlike the two sibling error paths just below it that do; in the scenario the harness's own teardown reaps it, so nothing leaks in test, but the asymmetry in production code is worth a follow-up.

  • xtask/src/workflow_contract.rs:~1292 — the ci.yml lifecycle lane's build check remains a substring match while the two other workflows now get the stricter whole-line one; the asymmetry predates this PR, but this is the change that introduces the stronger form and could cheaply extend it.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 7841012

This is a formal review recording our position on the record. It is deliberately non-gating: this automation files no approval, so no approving review will appear here whatever the outcome, and the merge decision stays with a human reviewer.

Blocking: 0 · Non-blocking: 6. This round reviewed removal of a compile-time test seam in favour of state-based affordances, with the new cross-file guard shown to be non-vacuous by two mutation experiments run in a scratch copy.

Check conclusions this review worked from, read at this exact head immediately before publishing: 26 success, 1 skipped, 2 failure.

The full findings, including the non-blocking items, are in the review comment posted alongside this one.

@rominf

rominf commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Splitting this up — it has three blockers welded together

Rebasing this onto current main (b0d598d0) surfaced a second structural problem, and taken with the Windows one it convinced me this PR is the wrong container for the work. Marking it draft rather than leaving it sitting as ready for review.

What the rebase found

The rebase itself was nearly clean — one additive conflict in tests/e2e-cucumber/Cargo.toml, where main added the fake-tailscale bin and this branch adds fake-lemonade. But a clean rebase is not a working one.

#421 landed two new consumers of the feature this PR deletes, in the newest commit on main:

  • apps/rocm/src/comfyui.rs:36,41 — comfyui_source_archive_url(), behind ROCM_CLI_COMFYUI_SOURCE_ARCHIVE_URL_OVERRIDE
  • apps/rocm/src/main.rs:9549,9554 — torch_runtime_dep_checks_disabled(), behind ROCM_CLI_DISABLE_TORCH_RUNTIME_DEP_CHECKS

cargo clippy --workspace --all-targets -- -D warnings fails on the rebased branch with four unexpected cfg condition value errors. And silencing them by simply deleting the attributes would be worse than the compile error: the #[cfg(not(..))] bodies become the only ones, so #421's ComfyUI scenario would fetch the real GitHub archive instead of its loopback fixture, and therock_steps.rs would run real system package installs. Both are on the every-PR mock lane.

That is the same thing that happened with #380's dash clock while this branch sat. Two consumers added in the time this has been open — the removal is chasing a target that keeps moving, which is itself a signal.

The three blockers

Piece Blocker
dash logical clock none — mergeable today
Lemonade planted runtime the Windows managed-spawn gap
e2e-test-hooks removal a moving consumer list; correctly the last step, not the first

Bundled, they all inherit the worst one.

What I have done

  • refactor(dash): read the test clock from state, not a compiled-in seam #466 — the dash clock conversion on its own, rebased onto current main. Both review threads here informed it and it carries both fixes: the WARN, and the cross-file guard pinning the harness literals. dash-08/dash-09 pass locally, both clippy invocations and fmt are clean, and the guard is mutation-probed. It has no GPU or Windows dependency, so it can land independently.
  • serve: on Windows a managed engine that dies at startup is reported as success #467 — the spawn_managed_engine_child gap, filed as its own issue as offered above. On Windows there is no liveness check after the detached spawn, and start_managed_service returns Ok(..) regardless of the readiness outcome, so rocm serve exits 0 when the managed engine dies at startup. That is a real user-visible defect on main independent of this PR, and it is what serve-18 is now unlucky enough to depend on.

What happens to this one

Parked as draft, not closed — the planted-runtime work is sound and I would rather re-land it than rewrite it. It needs #467 fixed first, because the scenario legitimately asserts behaviour Windows does not currently implement, and the Windows lane is now a required check.

The feature removal should come last, as a small mechanical PR converting the remaining consumers in one go, so it lands before main can overtake it again.

I have not pushed the rebase here — it does not compile for the reason above, and a broken head would be worse than a stale one. It is kept locally if it is useful later.

Nothing here needs a decision from a reviewer right now; #466 is the piece that is ready.

@rominf

rominf commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Head is now e97c827c. The PR is re-scoped and rebuilt on main (1c33248c), and the description is rewritten to match. It stays a draft.

What changed in scope.

What carried over. Compared with the old diff over the kept files:

  • Kept, with prose changes only: fake-lemonade, the planted manifest with env_id derived from rocm_deps, the serving_steps.rs changes, removal of the seam and the bypass, and the engine unit test.
  • Different:
    • the table is now a raw const read via include_str!;
    • rocm-deps is a dev-dependency, so the crate-edges allowlist entry is not needed;
    • the serve runs through run_rocm_with_scenario_env, because it now carries the host root;
    • the Lemonade crate keeps its feature, since it now only forwards to rocm-core;
    • comments are ASCII, for the fix(output): replace non-ASCII punctuation with ASCII in CLI output text #683 contract.
  • Dropped: the dash files, the workflow/xtask feature removal, and the install_lifecycle.feature note, which is still true.

One design change from the old version: no GPU needed. Previously, removing the pre-flight bypass forced serve-18 onto @requires-gpu, which lost the every-PR cadence. Instead, the harness now plants one KFD GPU node under the simulated host root, so the unchanged pre-flight passes on any Linux host. The tag becomes @requires-os:linux:

Verified locally on a GPU-less host. Mutations:

  • fake install succeeds: the scenario fails;
  • GPU not planted: the scenario fails with its named pre-flight message.

siloteemu's non-blocking items (automated review on 7841012c):

Item Disposition
two failing checks at that head that was serve-18 on Strix Windows (#467), analysed in the two comments above it. The re-scoped scenario no longer runs on Windows.
engine test copies the fixture table fixed: the_e2e_fake_runtime_table_drives_a_rocm_backend_install reads BACKENDS_TABLE from fake-lemonade.rs via include_str!; editing the fixture's table fails it (mutation-checked)
WARN not visible in the TUI dashboard clock, moved to #466; out of scope here
nothing pins serve-18's gate tag fixed: the_lemonade_recovery_scenario_runs_on_every_linux_lane_and_not_on_windows in expectation.rs reads the real tag line; changing it to @requires-no-gpu or dropping it fails (mutation-checked). The tag pinned is @requires-os:linux, not @requires-gpu, because of the design change above.
lemond left running when ensure_best_llamacpp_backend fails follow-up filed as #729 (also covers wait_for_lemonade_cli_status's early return)
contract asymmetry: substring vs whole-line build check moot here: this PR no longer edits those assertions, so it stays with the feature-removal step

Why still a draft. It no longer has to come last: it removes no feature, and the scenario does not need #437 to pass. I am keeping it a draft as asked until #437 (the fix for #467) lands. Once that happens it can come out of draft as-is, and extending serve-18 to Windows would be a follow-up.

Gate on this head:

  • fmt;
  • both clippy invocations, on Linux and on the Windows GNU target, plus Windows cargo check -D warnings;
  • rocm-engine-lemonade, rocm --bin rocm, e2e-cucumber --lib + feature_naming, check --test e2e;
  • xtask tests, manifest --check, check-crate-edges;
  • serve-18 via cargo xtask e2e.

@rominf
rominf marked this pull request as ready for review October 26, 2026 12:46
@rominf rominf changed the title test(e2e): drive the Lemonade recovery scenario with a planted runtime test(e2e): drive the Lemonade recovery scenario with a planted runtime (ROCMAI-669) Oct 26, 2026

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · e97c827

Review — needs work

Full review of the whole change (12 files, 1c33248c318c72a9e837627ff71476c8583f1930...e97c827c0b69d684f64524794ed3d94116e86ae0), fanned out across three coupled-file-group workers plus centralized cross-cutting checks. Part of #349 (open tracking issue), explicitly re-scoped to convert one consumer of the e2e-test-hooks seam rather than remove the feature — the description says so and the diff matches that scope exactly. PR title cites ROCMAI-669; that identifier is an internal tracker reference not retrieved for this review.

Blocking

  • CI check "E2E tests (MI300X, current)" is red at the reviewed head commit — CI status for e97c827c0b69d684f64524794ed3d94116e86ae0 (not a source location; run: https://github.com/ROCm/rocm-cli/actions/runs/40315449640/job/121546925225).
    gh pr view 351 --json statusCheckRollup,headRefOid confirms headRefOid matches the reviewed commit and this check's conclusion is FAILURE, while every other GPU/OS E2E lane (Strix Halo ×4, W7900, Windows, WSL2) passed on the same run. The job log itself returned 403 in this environment, so the root cause could not be read directly — but the mechanism this PR introduces is a plausible direct cause, not just an unrelated flake: the scenario's gate changed from @requires-no-gpu to @requires-os:linux, which is exactly what newly exposes serve-18 to a real-GPU lane (MI300X) for the first time. Previously this scenario never ran there at all. Worth ruling out before assuming it's infra noise, since (unlike the Strix Halo/Windows self-hosted flakiness a prior reviewer already characterized as pre-existing) this specific lane has no established flake history against this scenario — it's new exposure.
    Confidence 100 · architectural · Fix: pull the MI300X job log and confirm whether the planted fake runtime / simulated host-root GPU node interacts badly with a lane that also has a real AMD GPU and real Lemonade state, before merging.

Non-blocking

  • Stale "scripted failure seams" wording in an assert message, contradicting the doc comment immediately above it — xtask/src/workflow_contract.rs:296 (the assert! message inside assert_prebuilt_e2e_lanes_enable_test_hooks), vs. the doc comment at xtask/src/workflow_contract.rs:273-274.
    This PR correctly rewrote the function's doc comment from "the scripted Lemonade backend-install failure" to "the simulated host root, ROCM_CLI_TEST_HOST_ROOT" (the compiled-in failure branch is exactly what this PR removes). The assert! panic message below it in the same function was not updated and still reads "...without it the suite's scripted failure seams are compiled out:\n{block}", describing the now-deleted mechanism. A future reader hitting this assertion failure gets pointed at machinery that no longer exists.
    Confidence 85 · mechanical · Fix: update the assert message to match the revised doc comment (e.g. "...without it the suite's test seams, such as the simulated host root, are compiled out").

Decisions for the author

  • fake-lemonade.rs's run_lemonade_cli silently succeeds on any subcommand it doesn't recognize — tradeoff. This keeps the fixture small and avoids breaking the scenario if the engine issues some other incidental call, but it also means a future, unexpected invocation shape from the real engine (a new flag, a new subcommand) would pass through unnoticed rather than failing loudly, which could mask a drift between the fixture and the real lemonade CLI contract it stands in for. The alternative (fail loudly on anything unrecognized) was deliberately not chosen here (the code comments explain the three calls that matter and say "any other subcommand: succeed quietly"); confirm that's still the intended tradeoff as the fixture's surface grows.

Positive signals

  • The new backend_alignment.rs unit test pulls fake-lemonade.rs's BACKENDS_TABLE via include_str! rather than a hand-copied literal, so the fixture and the parser it's meant to drive cannot silently drift apart — verified the extraction is byte-exact and the parser assertions are correct against it.
  • The planted runtime's env_id (format!("lemonade-embeddable-{}", rocm_deps::LEMONADE_VERSION) in the test fixture) is verified byte-identical to the format string the real installer uses in engines/lemonade/src/install.rs:201, and the match against Detect was traced through to confirm the scenario actually skips the real multi-gigabyte install rather than silently falling through to it — exactly the property the PR's description claims.
  • Every cross-referenced issue in the description (#467, #437, #729, #510, #466) checks out factually against live GitHub state — titles, open/merged status, and the relationships claimed between them are all accurate.

Deployment notes

Both removed code paths (engines/lemonade/src/lib.rs's scripted install failure, apps/rocm/src/serve_cmd.rs's pre-flight waiver) were compiled only under the e2e-test-hooks feature, so this change has no behavioral effect on release/production builds.

What this covered

Every file in the diff, read completely at the PR head plus surrounding code (the full serve() pre-flight, the Lemonade install/detect flow, the E2eWorld harness, the tag-resolution machinery in expectation.rs), via three independent fanned-out workers grouped by contract coupling (engine/fixture contract; harness-and-steps; feature-tag-and-CI-gating), each of which activated the review-code skill itself and verified its own candidates before reporting. Centralized checks on top: live CI status for the exact head commit, a repo-wide grep confirming no dead references remain to the removed env var/feature seam, independent tracing of the env_id format-string parity between the fixture and the real installer, verification of the AGENTS.md crate-edge and GitHub-Actions-pinning conventions against this diff, and DCO/commit-quality checks (single well-described commit, signed off).

Did not run: a deep git log/blame history pass beyond confirming this is a single commit with no revision churn to review; a prior-PR-review-comments pass on these files (not retrievable at reasonable cost here); local build or test execution (no Rust toolchain available in this environment) — relied on live CI results instead, per the skill's guidance to take CI at face value rather than re-deriving its coverage.

Not reconciled against any PR discussion or review threads — those were deliberately excluded as input to this review's findings.

@tomastola tomastola left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings.

Non-blocking

  • MI300X lane red is unrelated to this PR. E2E tests (MI300X, current) (job 121546925225) fails only on therock-next-09 (a vLLM uv pip install --reinstall step), which this diff does not touch. serve-18 passed on that same lane. The identical single unexpected failure appears on main (runs 40360152181, 40357677968, 40354126258) and on other branches in the same window (e.g. jobs 121795242877, 121882265448, 121704235820). It should not gate this PR.
  • The fake lemond never exits (tests/e2e-cucumber/src/bin/fake-lemonade.rs, run_lemond). The comment says rocm serve terminates it when the serve fails, but the description says #729 tracks a lemond left running when the backend install fails during serve, which is this scenario's path. Until #729 lands, every Linux lane run can leave a sleeping process behind, and the scenario now runs on every Linux lane. Consider bounding the fake's lifetime, and aligning the comment with what serve does today.
  • The planted env_id repeats the format string managed_engine_runtime_id uses (plant_failing_lemonade_runtime in tests/e2e-cucumber/tests/e2e.rs; apps/rocm/src/engines_cmd.rs:323). Deriving the version from rocm_deps covers a version bump but not a change to the lemonade-embeddable- prefix. If Detect stopped matching, serve would start the real multi-gigabyte install instead of failing fast. An assertion that the install attempted was the fake's (it already prints a distinctive stderr line) would make that fail quickly.
  • tests/e2e-cucumber/README.md (around lines 244-251) lists the fixture binaries (fake-uv, fake-python, fake-tailscale) and does not mention fake-lemonade.

Decisions for the author

  • The engine unit test reads the e2e crate's fake-lemonade.rs with include_str! and extracts the raw-string literal textually. The table is then checked on every unit-test lane with no copy to drift. The cost is that a product crate's unit tests depend on a path in tests/ and on the literal's exact spelling. Please confirm that is intended.
  • The e2e crate gains a rocm-deps dev-dependency, while the README's "Black-box only … No imports from the rocm-cli codebase" principle stays as written. rocm-core is already such an edge. Either confirm this extension or adjust the README line.
  • serve-18 now runs on every Linux lane (GPU and GPU-less) and WSL2, where it previously ran only on the no-GPU lane. The new gate is pinned in expectation.rs. This is more cadence, and each run starts the fake server noted above.

Good things

  • The tag gate is pinned against the real feature-file tag line, so an edit fails a test.
  • The no-GPU pre-flight is no longer waived; it runs against a GPU planted under the simulated host root.
  • Both e2e-test-hooks branches are removed cleanly, and no references to ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE remain.

@rominf
rominf force-pushed the chore/remove-e2e-test-hooks branch from e97c827 to 07bc2be Compare November 3, 2026 13:52

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 07bc2be

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

DEGRADED REVIEW (single-context; see What this covered).

Full review of the whole change (14 files, 9875940e...07bc2be8, i.e. everything added since the last nowycondro review at e97c827c, re-verified from scratch). Part of #349 (open tracking issue); converts one consumer (serve-18) of the e2e-test-hooks seam, explicitly re-scoped not to remove the feature itself — description and diff match. ROCMAI-669 (Jira, retrieved) describes exactly this converted scope and is "In Progress", consistent with an in-flight step of #349 (also open). Neither claims this closes either tracker item; no mismatch.

Blocking

  • The PR currently has real, substantive merge conflicts against the current tip of main, in apps/rocm/src/serve_cmd.rs, xtask/src/workflow_contract.rs, and .github/workflows/e2e-selfhosted.yml. Confirmed by actually merging (git merge-tree), not inferred from mergeable_state: dirty alone.
    apps/rocm/src/serve_cmd.rs (no-GPU pre-flight bail): main has since landed #284, which added a sibling test hook e2e_simulate_oom_launch() into the exact same if this PR edits (now !cpu_only && !scripted_backend_failure && !e2e_simulate_oom_launch() && !reuse_existing && host_has_no_usable_gpu on main, vs. this PR's !cpu_only && !reuse_existing && host_has_no_usable_gpu). A careless resolution (taking either side wholesale) either keeps the dead scripted_backend_failure this PR removes, or silently drops e2e_simulate_oom_launch() that #284 added — the correct merge keeps !e2e_simulate_oom_launch() and drops only !scripted_backend_failure.
    xtask/src/workflow_contract.rs and .github/workflows/e2e-selfhosted.yml: main independently reworded the same doc comment / CI comment this PR rewords (to document the new rocm/e2e-oom-fault-injection floor), and main's own wording still says "the suite's scripted failure seams are compiled out" / "the scripted Lemonade backend-install failure" — the exact stale phrasing this PR's second commit fixed in its own branch, now re-introduced by the conflict unless resolved deliberately.
    Confidence 100 · architectural · Fix: rebase onto current main, keep e2e_simulate_oom_launch() intact in the pre-flight condition, and reconcile the doc/assert/CI-comment wording so it documents both the oom-fault-injection floor (from #284) and the simulated-host-root rewording (from this PR) without reintroducing "scripted failure seams" language for the now-deleted Lemonade seam.

Non-blocking

None.

Decisions for the author

  • rocm-deps dev-dependency vs. the e2e README's "black-box only" principle — tradeoff. tests/e2e-cucumber/Cargo.toml adds rocm-deps as a dev-dependency so the planted runtime's env_id tracks rocm_deps::LEMONADE_VERSION instead of a copy. tests/e2e-cucumber/README.md's Design Principles still read "No imports from the rocm-cli codebase", with only rocm-core called out as an existing exception (for a harness unit test reading a planted record back through rocm_core::ManagedServiceRecord). rocm-deps is a second first-party edge the README text doesn't acknowledge. It's a reasonable extension of the same precedent (verified: the derived env_id format is byte-identical to apps/rocm/src/engines_cmd.rs::managed_engine_runtime_id's, and xtask check-crate-edges exempts dev-dependencies by design), but confirm it's intended and consider updating the README line.
  • The fake lemond's lifetime bound relies mostly on a 10-minute backstop in practice — tradeoff, already self-documented. fake-lemonade.rs's run_lemond exits on Unix re-parenting or after LEMOND_MAX_LIFETIME (10 min). Since #729 (open) means the real engine does not stop lemond when backend install fails — this scenario's own path — re-parenting may not occur until the whole test harness tears down, leaving the 10-minute cap as the actual bound on every Linux-lane run of this scenario until #729 lands. The commit message already states this tradeoff plainly; flagging only so it's confirmed as accepted rather than overlooked.
  • fake-lemonade.rs's run_lemonade_cli succeeds quietly on any unrecognized subcommand — tradeoff. Keeps the fixture small, but a future unexpected call shape from the real engine would pass through unnoticed rather than surfacing a fixture/contract drift. The in-file comment already explains this choice; confirm it's still intended as the fixture's surface grows.

Positive signals

  • The planted manifest's JSON shape is a byte-exact match for LemonadeInstallManifest's real fields (engines/lemonade/src/install.rs), and resolve_runtime() genuinely only checks manifest.lemond.is_file() — verified directly in the source, confirming the PR description's "why this doesn't weaken verification" claim rather than taking it on trust.
  • The planted KFD node's properties content matches the real per-line key value parser in crates/rocm-core/src/host_gpu.rs exactly (and the existing unit tests there use the identical format), so the simulated GPU is recognized for the reason the PR says it is.
  • expectation.rs's new test derives the scenario's tags from the real .feature file text (feature_tags, matched by @id:) rather than a hardcoded string, so it is a genuine pin against tag drift, not a vacuous one.
  • The second commit (07bc2be8) cleanly resolved every non-blocking item a prior round of review raised against e97c827c: the stale "scripted failure seams" assert message, the fake lemond never exiting, the env_id-drift blind spot (now caught by asserting the fake's install-refusal string appears exactly twice), and the missing fake-lemonade README entry.
  • The previously-flagged MI300X CI failure at e97c827c is resolved: at this head, E2E tests (MI300X, current) is green. The one remaining failing self-hosted check, E2E tests (Strix Halo, Windows, current), was traced via its job log to runtime-resolve-canonical-release (a TheRock wheel-install scenario this diff does not touch) failing as "a regression" — and the E2E self-hosted workflow has failed on every one of the last 10 completed runs on main itself, so this is pre-existing lane breakage, not something this change introduced.

Deployment notes

No behavioral change to release/production builds: both deleted branches were compiled only under #[cfg(feature = "e2e-test-hooks")]. No new runtime dependency; rocm-deps is test-only (dev-dependency).

What this covered

DEGRADED REVIEW: fan-out to independent sub-workers was attempted twice (3 workers, grouped by coupling: product-code removal; e2e harness/fixture binary; feature-gating/docs/CI/xtask), and both attempts failed outright on a shared, environment-wide concurrent-subagent ceiling unrelated to this task (the tool explicitly said not to retry a third time). This review therefore ran single-context, with the same file groups and checks the workers would have covered done directly: every file in the diff read completely at the PR head (apps/rocm/src/serve_cmd.rs, engines/lemonade/src/{lib.rs,llamacpp_backend.rs}, tests/e2e-cucumber/{Cargo.toml,README.md,src/bin/fake-lemonade.rs,src/expectation.rs,src/fixture_binary.rs,features/model_serving.feature,tests/e2e.rs,tests/e2e/serving_steps.rs}, xtask/src/workflow_contract.rs, .github/workflows/e2e-selfhosted.yml, Cargo.lock), plus surrounding/callee code (engines/lemonade/src/install.rs, engines/lemonade/src/process.rs, apps/rocm/src/engines_cmd.rs, crates/rocm-core/src/host_gpu.rs, crates/rocm-core/src/hardware_root.rs, xtask/src/crate_edges.rs), live CI status plus one job log for the exact head commit, a 10-run history comparison of the self-hosted workflow on main, a repo-wide grep for dangling references to the removed mechanism, and a real three-way merge against origin/main to check mergeability.
Did not run: a deep git log/blame pass beyond spot checks (used to locate the sibling #284 PR causing the merge conflict); a prior-PR-review-comments pass on these files beyond what the merge-conflict investigation surfaced; local build or test execution (no Rust toolchain invoked; relied on live CI, consistent with the skill's guidance). Not reconciled against this PR's own review discussion/threads — deliberately excluded as input to this review's findings, per method.

🤖 Generated with Claude Code

`@id:serve-lemonade-preparation-recovery` (serve-18) needs Lemonade's
llama.cpp backend install to fail twice, so it can pin the retry count and
the recovery command. Two `#[cfg(feature = "e2e-test-hooks")]` branches
provided that: one in the Lemonade engine's Install RPC that failed on
`ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE`, and one in `rocm serve` that
waived its no-GPU pre-flight for the same variable. Both are deleted. The
`e2e-test-hooks` feature stays: it has other consumers (the simulated host
root among them), and removing it is the last step of #349, not this one.

The failure now happens where a broken install fails, at a non-zero child
process:

- `fake-lemonade`, a `[[bin]]` of the suite, is planted as the runtime's
  `lemonade` and `lemond`. `lemonade backends` reports `llamacpp:rocm` as
  installable and `lemonade backends install` exits non-zero; `lemond` stays
  alive until killed.
- The install manifest is written beside it. Its `env_id` is derived from
  `rocm_deps::LEMONADE_VERSION` (a dev-dependency, so no crate-edge entry):
  `serve` skips its own install only when `Detect` reports exactly that id,
  and a placeholder silently runs the real multi-gigabyte install instead.
- The pre-flight runs unchanged against one KFD GPU node planted under the
  simulated host root. The root holds nothing else, so the scenario needs no
  GPU and a GPU runner's devices do not take part. `XDG_RUNTIME_DIR` points
  inside the scenario too, so the fake server writes nothing outside it.

The scenario is now `@requires-os:linux` instead of `@requires-no-gpu`. It
still runs on the GPU-less every-PR lane, and now on the Linux GPU lanes and
WSL2 as well. It stays off Windows: the host root re-roots Linux paths only,
and there `rocm serve` does not notice a managed engine dying at startup
(#467), so the failure would never reach the user. A test in
`expectation.rs` pins that gate against the real tag line.

The engine's unit test now reads the backends table out of the fixture's
source with `include_str!`, instead of a copy kept in step by a comment.
Workflow and contract-test comments that cited the deleted seam as their
example now cite the host root.

Checked by mutation: the scenario fails when the fake install succeeds, and
fails with its named pre-flight message when the GPU is not planted.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…runtime

The fake `lemond` now exits once its parent is gone (Unix re-parenting)
and after a 10-minute backstop, so a run whose best-effort teardown fails
leaves no process behind while #729 is open. serve-18 now asserts that
the planted runtime refused both backend installs, through a constant
shared between the fixture and the step. If the planted env_id ever
stopped matching Detect, the scenario now fails fast instead of starting
the real multi-gigabyte install. Also fixes a stale "scripted failure
seams" assert message and lists fake-lemonade among the README's fixture
binaries.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The e2e suite is black-box and imports nothing from the rocm-cli
codebase, per its README. The planted Lemonade runtime's env_id was
the exception: it took the version from a rocm-deps dev-dependency.

It now reads the version from runtime-deps.toml, the workspace pin
file that rocm-deps' build script generates LEMONADE_VERSION from, so
the planted id still follows a version bump with no copy to keep in
step. A missing or misspelled pin panics with the file's path. The
README's black-box principle names this as the way a scenario gets a
pinned version.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The engine's unit test pulled the fake runtime's `lemonade backends`
table out of fake-lemonade.rs by matching the text of a raw-string
literal, so it depended on how that literal was spelled. The table is
now a data file under the e2e fixtures. The fake binary prints it with
include_str!, and the engine test reads the same file, as rocm-core's
fix tests already read the PowerShell cases there. The pre-push hook
lists the table as read by rocm-engine-lemonade, so editing it runs
that crate's tests.

Also drops three comments that still described serve-18's removed
failure hook as waiving the no-GPU pre-flight. Since the rebase, the
free-space override is the only scripted waiver left.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the chore/remove-e2e-test-hooks branch from 07bc2be to 895c4fb Compare November 16, 2026 11:00
@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the reviews. The branch is rebased onto current main; here is where each point landed.

@nowycondro, review on 07bc2be8 (merge conflicts with #284). Rebased. The no-GPU pre-flight in apps/rocm/src/serve_cmd.rs keeps !e2e_simulate_oom_launch() from #284 and drops only the removed scripted_backend_failure; the free-space override is now the one other test-hook waiver, under its own name, scripted_free_space. The doc comment on the reuse pre-gate names both remaining hooks. The doc comment and assert message in xtask/src/workflow_contract.rs, and the comment in .github/workflows/e2e-selfhosted.yml, keep main's oom-fault-injection floor and cite the simulated host root as the example seam. None of them describes the deleted Lemonade seam any more.

rocm-deps dev-dependency vs. the black-box principle (@tomastola, @nowycondro). Removed. The suite now reads the Lemonade version from runtime-deps.toml, the workspace pin file that rocm-deps' build script generates LEMONADE_VERSION from, through the new tests/e2e-cucumber/src/runtime_pins.rs. So the planted env_id still follows a version bump with no copy to keep in step, and a missing or misspelled pin panics naming the file. The README's black-box principle now names this as the way a scenario gets a pinned version.

The engine unit test reading fake-lemonade.rs with include_str! (@tomastola). Kept include_str!, but it now reads a data file instead of Rust source. The table moved to tests/e2e-cucumber/fixtures/lemonade_backends_table.txt: fake-lemonade prints it, and the test in engines/lemonade/src/llamacpp_backend.rs parses the same file, so the fake and the parser cannot drift and the check runs on every unit-test lane. It no longer depends on how a raw-string literal is spelled, which was the fragile part. Product-crate tests already read fixtures from that directory (rocm-core's PowerShell cases), and the pre-push hook's CROSS_CRATE_READS lists the table as an input of rocm-engine-lemonade, so editing it runs that crate's tests. Mutation-checked: switching the table's installable to installed turns the test red.

Addressed in cd025acc and unchanged by the rebase (@tomastola's other points):

  • The fake lemond exits once it is re-parented, and after a 10-minute cap in any case. Until lemonade: a failed backend install during serve leaves the spawned lemond running #729 lands that cap is the real bound when serve fails, and the comment says so.
  • A drifted lemonade-embeddable- prefix can no longer fall through to the real install unnoticed: serve-18 asserts the fake's install-refusal line appears exactly twice.
  • fake-lemonade is listed with the other fixture binaries in the e2e README, and the stale "scripted failure seams" assert message is fixed.

Also in this push: docs/architecture.md lists the two new suite files, which check-architecture-doc requires now that it covers tests/e2e-cucumber/src. The planted runtime no longer pre-creates state/ and logs/: the engine's launch creates both itself, so only the manifest directory is needed.

Not changed:

  • run_lemonade_cli still succeeds quietly on subcommands it does not recognise. The scenario depends only on the three calls the comment lists; failing on anything incidental would tie the fixture to how the engine happens to call things rather than to the behaviour under test.
  • serve-18 runs on every Linux and WSL2 lane, which is intended: the pre-flight runs against a GPU planted under the simulated host root instead of being waived. It stays off native Windows until fix(serve): watch a Windows engine through its handle and keep a concurrent stop (ROCMAI-675) #437 fixes serve: on Windows a managed engine that dies at startup is reported as success #467.
  • serve-18 now fails on serve's engine start path (ensure_best_llamacpp_backend), not in the Install RPC: because the planted env_id matches, serve skips its own install. The retry and recovery lines still come from the same install_llamacpp_backend_with_retry, and more real code runs on the way (GPU resolution, spawning lemond, relaying the startup failure).
  • The MI300X failure at e97c827c was therock-next-09, which also failed on main; serve-18 passed on that lane.

Verified locally on the new head: serve-18 via cargo xtask e2e (1 scenario, 0 unexpected failures); fmt; both clippy invocations; cargo test --workspace --all-targets; check-architecture-doc, check-crate-edges, check-module-size, manifest --check. Host limits, in code this change does not touch: two build_metadata git_watch_paths tests (global commit signing in temporary repos), one service_control AIM stop test (podman installed as docker), and cargo xtask smoke stopping at its bubblewrap sandbox step (no network namespace on this host); every smoke check before it passed.

@rominf
rominf requested a review from juhovainio November 16, 2026 11:04
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff end to end; nothing here looks wrong in the logic, and replacing the test-only install-failure hook with a planted runtime is a clear improvement. The points below are small, but I'd like them settled before merge.

  • The new count of 2 on the fake's refusal line (inline) depends on the engine retrying exactly once and relaying child stderr once per attempt. Please confirm it against the first green e2e run.
  • The src/runtime_pins.rs row in docs/architecture.md is out of alphabetical order (inline).
  • Please grep docs/testing.md, docs/manual-testing.md and the README for leftover mentions of ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE, and for serve-18 being described as @requires-no-gpu. The gate is now @requires-os:linux, so serve-18 depends on the ROCM_CLI_TEST_HOST_ROOT seam.
  • The fake lemonade matches status and backends anywhere in argv instead of as the subcommand. Harmless today, but easy to tighten.
  • Worth noting: the fake lemond is reaped only on a Unix parent change or after the 10-minute cap.

The e2e lanes were still pending when I looked, so serve-18's result on CI is not confirmed yet.

// Both attempts must have been refused by the planted runtime. Without its
// line, `serve` did not accept the planted runtime and attempted some other
// install -- for a mismatched `env_id`, the real multi-gigabyte one.
assert_eq!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The count of 2 assumes the engine retries exactly once and relays the child's stderr once per attempt. Please confirm against the first green e2e run (the mutation check in the PR text is the only evidence so far).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed against CI, and the count is now exercised on both failure paths.

  • CI, at 895c4fb0: serve-18 passed, with the == 2 assertion, on E2E tests (both Ubuntu LTSes) and on E2E tests (Strix Halo, WSL2). Each log shows Then serving stops after one automatic retry passing.
  • Why it is exactly 2: install_llamacpp_backend_with_retry calls the install closure once, and once more on failure, and never again. Each call runs lemonade backends install with inherited stderr (backend_install_command), and the fake prints its refusal line once per run. The step also pins retrying once to exactly one.

Checking this turned up a real flake, now fixed in e0c4e8d. The install fails inside the spawned engine, and without --wait rocm serve notices only an engine that dies within its 200 ms post-spawn check. On a host at load ~18 the engine missed that window, and serve reported readiness: starting with exit 0 after 50 s. serve-18 now runs serve --managed --wait, and follows the rocm logs --service <id> pointer when the engine died after the window. Both paths pass locally (serve step 0.26 s, and 45.3 s with the fixture's install delayed 1 s), with the same counts. The default (no --wait) behaviour is a real defect, recorded in the PR body as a known gap for its own fix.

Comment thread docs/architecture.md
| [`src/quit_and_wait.rs`](../tests/e2e-cucumber/src/quit_and_wait.rs) | Retry loop for quitting a TUI that may open a quit-confirm prompt first. | flat |
| [`src/reader_failure.rs`](../tests/e2e-cucumber/src/reader_failure.rs) | Failure state shared between a background terminal reader and its waiters. | flat |
| [`src/runtime_root.rs`](../tests/e2e-cucumber/src/runtime_root.rs) | `check_runtime_root_not_nested`: judges the active runtime folder `rocm examine` prints and rejects a doubly nested `runtimes/wheel` path. | flat |
| [`src/runtime_pins.rs`](../tests/e2e-cucumber/src/runtime_pins.rs) | Reads a pinned runtime version out of the workspace-root [`runtime-deps.toml`](../runtime-deps.toml) as plain data, so a planted runtime keyed by that pin (the Lemonade `env_id`) follows a version bump without importing `rocm-deps`. | flat |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this row sorts before src/runtime_root.rs in the surrounding alphabetical table.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6898ef9: the runtime_pins.rs row now sorts before runtime_root.rs.

The fake matched `status` and `backends` anywhere in its arguments, so an
argument that only equalled one of them would have been answered as that
subcommand. It now skips the global --host/--port options the engine
passes and classifies the call by the word that follows, in a pure
function covered by unit tests. Any `backends` call other than `install`
still lists, so a flag the engine adds later gets the table rather than
an empty success.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The e2e harness table is alphabetical.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (20 files, ee605065…895c4fb0), fanned out to three coupled-file-group workers (engine/fixture contract; e2e harness/steps/feature; product-code/CI/xtask/docs — the third could not run as an independent worker, see What this covered). Part of an open tracking issue for retiring a compile-time E2E test-hook feature, explicitly scoped to convert one consumer (the Lemonade preparation-recovery scenario) rather than remove the feature itself — description and diff match exactly.

Blocking

  • A real, current merge conflict against main, confined to a doc comment — apps/rocm/src/serve_cmd.rs:868-877
    Confirmed independently via git merge-tree --write-tree origin/main <head> against today's tip, and cross-checked against the GitHub API, which reports mergeable: CONFLICTING / mergeStateStatus: DIRTY for this exact head. The conflict is confined to the prose paragraph describing the pre-flight bail's skip conditions: main independently reworded the same paragraph when it added OOM-launch fault-injection "either mode" support, and this PR's own rewording (retiring a "backend-failure scenario" mention) overlaps the same lines. The executable code just below (the let scripted_free_space = … / if !cpu_only && … block) auto-merges cleanly — only the comment conflicts. A careless resolution taking main's side wholesale would reintroduce the stale "backend-failure scenario" wording this PR correctly retires.
    Confidence 100 · mechanical · Fix: rebase onto current main and hand-merge this one paragraph so it keeps this PR's retirement of the stale wording while adding back the "(either mode)" detail main introduced for OOM-launch. Nothing else in the tree conflicts — two other files an earlier round of feedback flagged as conflicting now merge cleanly, independently re-verified.

Non-blocking

  • fake-lemonade's CLI dispatch matches tokens anywhere in argv, not the exact subcommand position — tests/e2e-cucumber/src/bin/fake-lemonade.rs:99-110
    Scans the whole argument vector instead of checking a specific position. Works for today's three call shapes but a future one carrying both matched tokens could silently hit the wrong branch.
    Confidence 75 · logic · Fix: parse structurally (skip flag/value pairs, then match the first remaining token exactly).
  • A table row in docs/architecture.md is out of alphabetical order — docs/architecture.md:810
    The new runtime_pins.rs row sits after runtime_root.rs; it sorts before it alphabetically.
    Confidence 90 · mechanical · Fix: move the row above runtime_root.rs.
  • The planted env_id format string is now duplicated a third time, with no compile-time tie to production — tests/e2e-cucumber/tests/e2e.rs:754, vs. engines/lemonade/src/install.rs and apps/rocm/src/engines_cmd.rs's managed_engine_runtime_id
    A drift between the three literals would make serve silently fall through to a real multi-gigabyte install instead of the fixture. Mitigated — a runtime assertion elsewhere in this diff makes such a drift fail loudly — but nothing ties the literals together at build time.
    Confidence 76 · architectural · Fix: a shared helper in the pin crate would remove the duplication by construction; if that's not viable under the harness's black-box rule, cross-reference the sites in comments.
  • A source binary is read twice when planting it under two names — tests/e2e-cucumber/tests/e2e.rs:749-750
    Harmless (test-only, tiny binary) but avoidable.
    Confidence 70 · mechanical · Fix: read once, write both destinations from the same buffer.

Decisions for the author

  • This PR's new test does not itself prove the removed scripted-failure seam is dead — tradeoff. The new scenario setup never touches the retired env var, so reverting only the production removal (while keeping this PR's new test infrastructure) would leave the scenario passing unchanged, since it exercises the new planted-runtime path rather than the old one. Expected for a dead-code removal — its safety rests on the repo-wide absence of remaining references and on CI passing, not on this test — but worth confirming that's the intended coverage story.
  • The loose argv matching above may be an accepted simplification — confirm it's intentional given today's call shapes never collide, rather than something to tighten later.
  • A lane with real network egress could turn an env_id drift into a slow timeout rather than a fast failure — in this sandboxed CI a fallback to the real install has nowhere to download from, so a drift fails fast; a lane with egress would instead hang. Worth confirming a bounded timeout wraps this scenario everywhere it runs.

Positive signals

  • The scenario's lane gate is pinned against the real feature-file text (not a hardcoded copy), so editing the tag or the scenario id changes the test's own input and flips its assertions.
  • The planted GPU is traced end-to-end as a real mechanism, not a no-op: a planted KFD sysfs node reaches the real GPU-detection parser and resolves the pre-flight's "no usable GPU" check correctly on every lane the scenario now runs on, GPU-bearing or not.
  • The planted runtime's verification surface and manifest shape are confirmed field-for-field against the real installer code, matching the PR description's "why this doesn't weaken verification" claim directly rather than on trust.
  • A shared fixture file for the backend-status table is read by both the fake binary and a new engine unit test via include_str!, so the table and the parser it drives cannot silently drift apart.
  • Grepping the whole tree at the head finds no remaining reference anywhere to the removed env var or its old gate — a clean removal with no orphaned comments, docs, or contract tests describing the retired mechanism.
  • Black-box test-harness discipline is maintained and strengthened: a new helper reads a version pin from the workspace pin file as plain text rather than importing a product crate, and the harness docs are updated to describe the pattern; no new dependency was added.
  • CI is fully green at this exact head, including self-hosted GPU lanes an earlier round of review found flaky — resolved now.
  • Every issue the description cites checks out factually against live state, and one of them is a real platform gap this PR's own work surfaced and the author filed candidly rather than working around quietly.

Deployment notes

No behavioral change to release/production builds: the removed branches and the renamed pre-flight gate variable were compiled only under the test-only feature already gating this area. No new runtime dependency; the harness gains only a new test-only binary target.

What this covered

Every file in the diff (20 files) read completely, plus surrounding/callee code: the full pre-flight function and its doc comments, the real installer's runtime-resolution logic, the GPU-detection chain, and the e2e harness's world/setup code. Work was fanned out to three coupled-file-group workers. Two ran independently; the third (serve command pre-flight, workflow/xtask contract checks, CI workflow yaml, and docs) could not be spawned as an independent worker — the spawn and one respawn both hit a concurrency ceiling shared with unrelated sessions in this environment — and was reviewed directly in this context instead, so that slice was not an independent pass.

Centralized checks: live CI status for the exact head commit (fully green, no failures); independent re-verification of mergeability against the current tip of main via a real three-way merge tool, cross-checked against the forge's own mergeability status (both agree, see Blocking); a repo-wide grep for dangling references to the removed mechanism; retrieval of every issue the description cites.

Did not run: the test suite locally (no toolchain available in this environment) — relied on live CI and static tracing instead; a deep history pass beyond what was needed to diagnose the merge conflict.

Not reconciled against this PR's own review discussion or threads — deliberately excluded as input to these findings, per method.


🤖 Automated by an AI agent

Comment thread apps/rocm/src/serve_cmd.rs Outdated
Comment on lines +868 to +877
// (probe returns `None`). The E2E-only OOM-launch fault injection stands in
// for the GPU its lane does not have.
//
// The scripted free-space override opens the same waiver: it exists to refuse a
// The scripted free-space override waives it too: it exists to refuse a
// Lemonade install on a drive that cannot hold it, which `rocm serve` only
// reaches past this bail.
let scripted_backend_failure = cfg!(feature = "e2e-test-hooks")
&& [
"ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE",
"ROCM_E2E_LEMONADE_FREE_SPACE_BYTES",
]
.iter()
.any(|variable| std::env::var_os(variable).is_some());
let scripted_free_space = cfg!(feature = "e2e-test-hooks")
&& std::env::var_os("ROCM_E2E_LEMONADE_FREE_SPACE_BYTES").is_some();
if !cpu_only
&& !scripted_backend_failure
&& !scripted_free_space

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this PR has a real, current merge conflict against main, confined to this paragraph. Confirmed independently via git merge-tree --write-tree origin/main pr/351 against today's tip, and cross-checked against the GitHub API (mergeable: CONFLICTING, mergeStateStatus: DIRTY).

main independently reworded this same paragraph when it added OOM-launch fault-injection "either mode" support, and this PR's own rewording (retiring the "backend-failure scenario" mention) overlaps the same lines. The executable code just below (the let scripted_free_space = … / if !cpu_only && … block) auto-merges cleanly — only this prose conflicts.

A careless resolution that takes main's side wholesale would reintroduce the stale "backend-failure scenario" wording this PR correctly retires. The correct resolution keeps this PR's retirement of that wording while adding back the "(either mode)" detail main introduced for OOM-launch. Confidence 100.

(FYI: the other two files an earlier round of feedback flagged as conflicting, xtask/src/workflow_contract.rs and .github/workflows/e2e-selfhosted.yml, now merge cleanly — main's edits there landed in non-overlapping regions, re-verified independently.)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in the merge 1174198 (a merge commit, not a rebase, since your review is on the current head). The merged comment keeps this PR's removal of the backend-failure seam and takes main's "(either mode)" wording: "The E2E-only launch fault injection (either mode) stands in for the GPU its lane does not have." The executable block merged cleanly and is unchanged: !scripted_free_space && !e2e_simulate_oom_launch() && …. The doc on reuse_pregate_opens, which names the same hook, now says "either mode" too. git grep finds no remaining mention of the removed seam.

const LEMOND_MAX_LIFETIME: Duration = Duration::from_mins(10);

fn run_lemonade_cli(args: &[String]) -> ExitCode {
if let Some(index) = args.iter().position(|arg| arg == "backends") {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this dispatch matches tokens anywhere in argv (args.iter().position(|arg| arg == "backends"), and args.iter().any(|arg| arg == "status") below) rather than checking a fixed subcommand position. It happens to work for the three argv shapes the real engine builds today, but a future call shape carrying both tokens (e.g. a --host/--port value that happens to equal one of the matched words) would silently hit the wrong branch. Confidence 75.

Fix suggestion: parse structurally (skip flag/value pairs, then match the first remaining token exactly) rather than scanning the whole vector.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 90dee28. The fake now skips the global --host/--port options the engine passes and matches the first remaining word exactly: status alone, backends install …, or any other backends call (which still lists, so a flag the engine might add later gets the table rather than an empty success). Anything else succeeds quietly, as before. The classification is a pure function with unit tests in the bin, run by cargo test. I checked that moving the status match off the subcommand position turns them red.

Comment thread docs/architecture.md
| [`src/quit_and_wait.rs`](../tests/e2e-cucumber/src/quit_and_wait.rs) | Retry loop for quitting a TUI that may open a quit-confirm prompt first. | flat |
| [`src/reader_failure.rs`](../tests/e2e-cucumber/src/reader_failure.rs) | Failure state shared between a background terminal reader and its waiters. | flat |
| [`src/runtime_root.rs`](../tests/e2e-cucumber/src/runtime_root.rs) | `check_runtime_root_not_nested`: judges the active runtime folder `rocm examine` prints and rejects a doubly nested `runtimes/wheel` path. | flat |
| [`src/runtime_pins.rs`](../tests/e2e-cucumber/src/runtime_pins.rs) | Reads a pinned runtime version out of the workspace-root [`runtime-deps.toml`](../runtime-deps.toml) as plain data, so a planted runtime keyed by that pin (the Lemonade `env_id`) follows a version bump without importing `rocm-deps`. | flat |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this row is out of alphabetical order — it's placed after runtime_root.rs, but "runtime_pins" sorts before "runtime_root". Confidence 90.

Fix: move this row above the runtime_root.rs row.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6898ef9 (same fix as your earlier comment on this row).


let version = e2e_cucumber::runtime_pins::pinned_version("lemonade");
let manifest = serde_json::json!({
"env_id": format!("lemonade-embeddable-{version}"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this env_id format string (lemonade-embeddable-<version>) is now duplicated a third time, with no compile-time tie to the two production copies (engines/lemonade/src/install.rs's format!("lemonade-embeddable-{version}") and apps/rocm/src/engines_cmd.rs's managed_engine_runtime_id). A drift between any of the three literals would make serve silently fall through to a real multi-gigabyte install instead of using this fixture.

This is mitigated — serving_steps.rs asserts the fake's refusal line appears exactly twice, so a drift fails loudly rather than silently — but nothing ties the three literals together at build time. Confidence 76.

Fix suggestion: a shared rocm_deps::lemonade_env_id(version) helper would remove the duplication by construction; if that's not viable under this harness's no-rocm-deps-import rule, cross-reference the three sites in doc comments so an edit to one is visibly incomplete without the others.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took the helper suggestion for the two production copies, in 37b509d. rocm_deps::lemonade_env_id(version) now builds the id in both places: the engine's manifest writer (install.rs) and rocm's managed_engine_runtime_id. Those are the two sides of the exact-string comparison that decides whether serve skips its install, so they can no longer drift. rocm-deps already holds the other helpers derived from the pin (archive name, download URL), and docs/architecture.md's entry for it now lists the id. The engines_cmd.rs test keeps its literal on purpose, so a change to the helper's shape fails there.

The e2e copy stays: the suite imports nothing from the workspace (README design principles), which is why it reads the pin as data. A comment beside it now names rocm_deps::lemonade_env_id as the helper it mirrors, and serve-18's refusal count catches a drift at run time.

Comment thread tests/e2e-cucumber/tests/e2e.rs Outdated
Comment on lines +749 to +750
install_fixture_binary("fake-lemonade", &lemond);
install_fixture_binary("fake-lemonade", &lemonade);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: these two calls each independently read the same source binary's bytes from disk to plant it under two names. Harmless (test-only, tiny binary) but avoidable. Confidence 70.

Fix suggestion: read the source bytes once and write both destinations from the same buffer.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b59a9ec: the new install_fixture_binary_as reads the fixture once and writes both names from the same buffer. install_fixture_binary keeps its single-destination form for the other callers.

Resolves the one conflict, in the comment above serve's no-GPU pre-flight
bail: main named the launch fault injection's two modes, and this branch
had dropped the removed backend-failure seam from the same sentence. The
merged comment keeps both changes, and the reuse pre-gate's doc now says
"either mode" too.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The engine writes `lemonade-embeddable-<version>` into the install
manifest, and rocm compares that id with what the engine's Detect
reports to decide whether its own install is already done. Each side
spelled the shape out, so the two could drift apart and send serve to
the real multi-gigabyte install. Both now call
rocm_deps::lemonade_env_id. The e2e suite still repeats the shape,
since it imports nothing from the workspace; a comment there names the
helper it mirrors, and serve-18's refusal count catches a drift.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The planted runtime installs the same fixture binary as both `lemond`
and `lemonade`. install_fixture_binary_as reads the fixture once and
writes every destination from that buffer; install_fixture_binary keeps
its single-destination form for the other callers. Also names
rocm_deps::lemonade_env_id as the helper the planted env_id mirrors.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The backend install fails inside the spawned managed engine. Without
--wait, rocm serve notices an engine that dies only within its 200 ms
post-spawn check and reports a later death as `starting` with exit 0
once the 45 s readiness probe gives up. On a loaded host the engine
missed that window and serve-18 failed with "serve unexpectedly
succeeded". With --wait a later death fails the serve with a pointer to
`rocm logs --service <id>`; the step follows that pointer, as a user
would, before asserting the retry and recovery lines. An engine that
dies inside the window still fails the serve with its log tail.

engine_death_report tells the two messages apart and is unit-tested on
the CLI's wording; a failed serve carrying neither fails the step and
names both, instead of surfacing as a missing retry line.

The default (no --wait) behaviour is a product defect and is left for
its own fix.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

@juhovainio thanks for both rounds. The inline points each have a reply in their own thread, naming the fix commit. On this review's "Decisions for the author":

  • The new test does not prove the removed seam is dead. Yes, and that is intended. It is a deletion, so what guards it is that nothing references the seam any more (git grep finds neither ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE nor scripted_backend_failure) and that the build and clippy pass. serve-18 covers the behaviour that replaced it.
  • The loose argv matching. Tightened rather than kept: see the thread on fake-lemonade.rs.
  • A lane with egress turning an env_id drift into a slow failure. It would. serve-18's serve has no per-step bound, so on a lane with egress a drift would start the real install, and only the job's timeout-minutes would stop it. Two changes since your review make a drift less likely: the engine and rocm now share rocm_deps::lemonade_env_id, and the planted copy names that helper. A bounded serve for this one scenario is possible, but it would also cap the 45 s --wait path, so I have not added one here.

Separately, checking your count-of-2 question turned up a flake in serve-18. It also exposed a default-path defect in rocm serve --managed. Details are in that thread and in the PR body's "Known gap" section. serve-18 now runs with --wait, and the defect is recorded for its own fix.

@rominf
rominf requested a review from juhovainio November 17, 2026 13:22

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings, but I'm requesting changes for the items below.

  1. Overstated claim (e2e.rs comment and PR body): a drifted planted env_id does not "fail fast instead of downloading". serve starts the real install first and the refusal-count assertion only fails afterwards. Reword or add an earlier guard (inline).
  2. Red-run evidence: I can't see in the PR text that the new tests were shown to fail with their target broken (e.g. drifted env_id, --wait removed). Please state it.
  3. LogTailQuoted branch: confirm the Then-steps read stderr there (inline).
  4. Lane coverage: serve-18 now also runs on the MI300X and Strix Halo WSL2 lanes. The planted host root should isolate it, but CI hasn't confirmed that yet.
  5. Slow path: under load the --wait path can spend the full 45 s readiness window on the blocking lane. The step comment documents it; just noting the cost.

CI: nothing failing so far; build-and-test, Windows, Strix/Windows E2E and coverage were still running when I looked, and MI300X E2E is queued on the downed runner pool. ROCMAI-669 scope matches.

// Mirrors `rocm_deps::lemonade_env_id`, which the engine and `rocm`
// both use; the suite cannot import it. A drift sends `serve` to
// the real install, which serve-18's refusal count catches.
"env_id": format!("lemonade-embeddable-{version}"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment (and the PR body's "fails fast instead of downloading") overstates this. If the planted env_id drifts, serve first starts the real multi-GB install, and serve-18's refusal count only fails afterwards. Please reword to "fails late, after the real install attempt", or add an earlier guard.

let (logs, logs_stderr, logs_rc) =
crate::run_rocm(world, &["logs", "--service", &service_id]);
assert_eq!(
logs_rc, 0,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the LogTailQuoted branch nothing is appended to stdout, unlike the LogsPointer branch. Please confirm the retry/recovery Then-steps read stderr (where the quoted log tail lives), or append it so both paths feed the assertions identically.

// Both attempts must have been refused by the planted runtime. Without its
// line, `serve` did not accept the planted runtime and attempted some other
// install -- for a mismatched `env_id`, the real multi-gigabyte one.
assert_eq!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This count also proves the planted runtime was used, but it only fails after any real install it would otherwise have run. Consider saying so in the message so a mismatch isn't mistaken for a fast failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants