Repository navigation
Conversation
90bf93d to
1c2600b
Compare
|
Rebased onto current What the rebase droppedThe two That also retires the "Depends on #342" note at the top of the description — #342 has merged. What the rebase preservedFour conflicts, each resolved by keeping both sides rather than taking one:
Three things needed updating because
The dash clock was a second consumer#380 added a Rather than keep the feature alive for it, the same technique this PR already uses for Lemonade is applied to dash. The principle: what must not diverge between the tested and the shipped binary is the compile. That is not the same as "no test affordances" — a path- or state-based affordance is present identically in every build and inert until something plants the state, whereas a So The existence check there is load-bearing rather than an optimisation: that field selects wall time only on Why serve-18 was failingNot infrastructure — a real defect in this PR. All three red lanes reported the same single unexpected failure, and the logs showed the planted runtime being ignored while a real multi-gigabyte backend install ran and succeeded, so Before With the install skipped, The planting location was already correct: this scenario does not opt into the shared runtime tree, so the CLI resolves no env root and Status of the three previously-red lanes
Also green on this head: Verified locally
Not verified locally: serve-18 itself — this host has no usable AMD GPU, which is precisely why the scenario is now |
juhovainio
left a comment
There was a problem hiding this comment.
I ran a code review pass on this in addition to my own read of the diff, and it surfaced two things worth looking at before merge.
The main one: dash_test_clock_offset_path in apps/rocm/src/dash.rs now checks for a file on every rocm dash launch, in every build, not just E2E builds. If that file (telemetry/test-clock-offset) ever ends up in a real user's data directory (synced or copied from a dev or CI machine, some future bug that touches that path), the daemon silently drops onto a frozen logical clock instead of wall time. There's no log line when this happens, only a doc comment saying it should never occur outside the E2E harness. A one line warn log when the file is found and used would make this fail loudly instead of silently, without touching the test seam design itself.
The second one is minor, and actually already flagged by the author in a comment: the clock file name and its data/telemetry subpath are duplicated as literals between dash.rs and the E2E's dash_steps.rs, kept in sync only by that comment. Not asking to block on this one, just noting it since the review turned it up.
CI is otherwise clean aside from the Strix Halo Windows and Ubuntu self hosted lanes, which I checked are flaky on main independent of this PR (3 of the last 4 completed main runs failed or were cancelled on those two lanes), so that's not a signal against this change.
| /// `TestClockDirective::FreeRunning(0)` logical clock. Returning the path | ||
| /// unconditionally would therefore take production off `Utc::now()`. | ||
| fn dash_test_clock_offset_path(paths: &AppPaths) -> Option<PathBuf> { | ||
| let path = paths.telemetry_state_dir().join(DASH_TEST_CLOCK_FILE); |
There was a problem hiding this comment.
Every build now takes this path, not just E2E builds. If telemetry/test-clock-offset ever exists in a real user's data dir, the dashboard silently freezes its telemetry clock instead of using wall time, with nothing logged. Worth a warn log when this file is found and used, so a stray file fails loudly instead of silently skewing displayed timestamps.
There was a problem hiding this comment.
Agreed, and fixed in 7841012 — this is the inherent cost of the state-based seam and it should not be paid silently.
dash_test_clock_offset_path now logs at WARN when it finds the file, naming the path:
WARN rocm::dash: dashboard telemetry clock is in TEST mode, not wall time: displayed
timestamps come from a logical clock read from this file. Only the E2E harness creates
it; delete it to restore wall-clock timestamps.
test_clock_file=/…/telemetry/test-clock-offset
Why tracing and not stderr. logging.rs makes the client subscriber file-only on purpose — the TUI owns the raw-mode terminal and a stray stdout/stderr write corrupts the display. The only production caller of this function is maybe_spawn_embedded_daemon on the rocm dash path, so stderr is exactly the wrong channel here; a line printed there would also land on the PTY grid that the dash-* scenarios parse and assert against. The WARN lands in ~/.rocm/logs/rocm-cli.log.<date>, which is where you would look when investigating "my timestamps are wrong" — and it survives the alt-screen switch, which a pre-launch eprintln would not.
Verified live, not just by unit test. Planting telemetry/test-clock-offset under a throwaway ROCM_CLI_DATA_DIR and launching the release rocm dash under a pty produced exactly the record above in the client log; the same launch with no file planted produced an empty log. logging::init runs at the top of run(), before subcommand dispatch, so the subscriber is installed before this callsite is reached, and DEFAULT_FILTER (warn,rocm=info,…) admits the rocm::dash target.
Tests. Two, using a thread-local capturing subscriber (tracing::subscriber::with_default, so a concurrently-running test cannot see or pollute the records): one asserts the WARN fires and names the file when it is present, one asserts nothing at all is logged when it is absent — a warning on every launch would be noise and would drain the signal out of the first.
Both were mutation-probed rather than just observed green: deleting the tracing::warn! turns the first test red (expected a WARN record, got: ), and moving the warn above the existence check so it fires unconditionally turns the second red. So neither passes vacuously.
The existence check itself is unchanged — cycle_timestamp picks wall time only on None, so returning Some for an absent file would put production on FreeRunning(0). Only the logging is new.
| /// `<ROCM_CLI_DATA_DIR>/telemetry/test-clock-offset` and falls back to wall time | ||
| /// when it is absent, so a rename on either side silently drops the whole | ||
| /// mechanism rather than failing loudly. | ||
| const DASH_CLOCK_OFFSET_FILE: &str = "test-clock-offset"; |
There was a problem hiding this comment.
This risk is already spelled out in the comment above (rename on either side silently breaks the mechanism). Not blocking, just flagging that the fix (a shared constant, or a debug_assert tying the two together) is still open.
There was a problem hiding this comment.
Closed this one too, with the debug_assert-style option rather than the shared constant — reasoning, since you offered both:
A shared constant is not reachable here. apps/rocm is a binary crate, so the suite cannot import DASH_TEST_CLOCK_FILE at all; making it importable means hosting it in a library and adding a first-party edge from e2e-cucumber to that library. #422 landed xtask check-crate-edges on main while this branch sat, which deliberately makes every such edge a reviewed decision — spending one on a single string, to couple the black-box suite to a production crate it otherwise doesn't link, seemed like the wrong trade.
So instead dash::tests::e2e_harness_plants_the_file_rocm_dash_reads in apps/rocm/src/dash.rs reads the step file's source with include_str! and pins the literals to what rocm dash actually resolves. That is the same cross-tree reach the suite already makes (installer_fixture.rs does it for install.ps1), it adds no dependency, and it runs in the every-PR unit lane rather than only where the dash scenarios run.
It checks two things:
- the harness's
DASH_CLOCK_OFFSET_FILEdeclaration parsed out of the source, compared againstDASH_TEST_CLOCK_FILE— parsed, not substring-matched, so an unrelated constant containing the same text can't satisfy it; - every directory between the data root and the file, derived from
AppPaths::telemetry_state_dir()and required to appear in order insidedash_clock_path's own body. Anchoring on the builder rather than the whole file matters: a stray.join("telemetry")elsewhere in those ~1.2k lines would otherwise satisfy it. Taking the whole relative path rather than its last component matters too —telemetry_state_dir()being moved (say todata/dash/telemetry) leaves the leaf name intact, and an earlier version of this test passed under that mutation while the mechanism was broken.
Mutation-probed in all four directions: renaming the constant on the production side, renaming it on the harness side, relocating telemetry_state_dir(), and moving the .join out of the builder while leaving a decoy elsewhere in the file — each fails with a message naming the mismatch.
It does impose two constraints on that file, both written into the comment above the constant: keep the declaration on one line, and keep dash_clock_path building the path from .join(..) links. If either is broken the guard fails loudly with an explanatory message rather than passing silently, which is the failure mode I wanted.
One related thing the rebase surfaced, since it is the same gate: this PR's own e2e-cucumber -> rocm-deps edge (the pinned Lemonade version behind the planted runtime's env_id) was not in the allowlist, so check-crate-edges failed on the rebased branch even though the rebase itself was conflict-free. That entry is added in 7841012.
1c2600b to
7841012
Compare
|
Thanks — both findings were right, and the first one is the real cost of choosing a state-based seam over a compile-time one, so it shouldn't be paid silently. Pushed as 7841012, on top of a rebase onto current The warn log. The duplicated literals. Took the Rebase. Onto Ran locally on this head: Also closed out one loose end from the previous description: the three Leaving both threads open for you to close. |
CI status correction — two things changed after this review was writtenFlagging this because the review reasonably discounted the self-hosted lanes, and both facts behind that have since moved. Neither was wrong at the time. 1. 2. That lane has since gone green on
The first completed result landed about three hours after this review. So the flakiness read was sound at the time and simply predates the evidence. The Ubuntu lane is a different story and the original read still holds there: it fails at GPU preflight under contention, is red on What that means for this PR
This was invisible until now because the Windows lane was cancelled on the previous head, so this PR has never had Windows data for the planted-runtime rework. Working hypothesis, not yet confirmed: #404 changed Not proposing to retag or weaken the scenario while the cause is unknown. |
Follow-up: my #404 hypothesis was wrong, and the real cause is a pre-existing Windows defectCorrecting the guess I posted above before it misleads anyone: #404 is not involved. It touched The real mechanism
After the spawn the parent waits up to 45s for HTTP readiness, records the status, and returns This is a real user-visible defect on Why this PR surfaced itThe deleted The planted runtime deliberately moves the failure to the fake's Why there is no test-side fixBoth seam-side options are closed, checked rather than assumed:
What I am not doingI am not retagging or weakening serve-18 while the cause is understood but unfixed, and I am not bolting a production Windows fix onto a test-infrastructure PR — that change deserves its own PR, its own tests, and a maintainer with a Windows host. Flagging it here for a maintainer decision rather than picking one unilaterally. Happy to open a separate issue for the Observed vs inferred, stated plainly: every file/line and git fact above is observed. That the fake's |
|
🔴 Automated review · pr-review-watcher · 7841012 This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. SummaryDeletes the e2e-test-hooks cargo feature of the rocm crate and both seams it gated, replacing them with state-based affordances: the Lemonade recovery scenario now drives a planted fake runtime whose 🚫 Blocking (must fix before merge)None. Non-blocking
|
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 7841012
This is a formal review recording our position on the record. It is deliberately non-gating: this automation files no approval, so no approving review will appear here whatever the outcome, and the merge decision stays with a human reviewer.
Blocking: 0 · Non-blocking: 6. This round reviewed removal of a compile-time test seam in favour of state-based affordances, with the new cross-file guard shown to be non-vacuous by two mutation experiments run in a scratch copy.
Check conclusions this review worked from, read at this exact head immediately before publishing: 26 success, 1 skipped, 2 failure.
The full findings, including the non-blocking items, are in the review comment posted alongside this one.
Splitting this up — it has three blockers welded togetherRebasing this onto current What the rebase foundThe rebase itself was nearly clean — one additive conflict in #421 landed two new consumers of the feature this PR deletes, in the newest commit on
That is the same thing that happened with #380's dash clock while this branch sat. Two consumers added in the time this has been open — the removal is chasing a target that keeps moving, which is itself a signal. The three blockers
Bundled, they all inherit the worst one. What I have done
What happens to this oneParked as draft, not closed — the planted-runtime work is sound and I would rather re-land it than rewrite it. It needs #467 fixed first, because the scenario legitimately asserts behaviour Windows does not currently implement, and the Windows lane is now a required check. The feature removal should come last, as a small mechanical PR converting the remaining consumers in one go, so it lands before I have not pushed the rebase here — it does not compile for the reason above, and a broken head would be worse than a stale one. It is kept locally if it is useful later. Nothing here needs a decision from a reviewer right now; #466 is the piece that is ready. |
7841012 to
e97c827
Compare
|
Head is now What changed in scope.
What carried over. Compared with the old diff over the kept files:
One design change from the old version: no GPU needed. Previously, removing the pre-flight bypass forced serve-18 onto
Verified locally on a GPU-less host. Mutations:
siloteemu's non-blocking items (automated review on
Why still a draft. It no longer has to come last: it removes no feature, and the scenario does not need #437 to pass. I am keeping it a draft as asked until #437 (the fix for #467) lands. Once that happens it can come out of draft as-is, and extending serve-18 to Windows would be a follow-up. Gate on this head:
|
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · e97c827
Review — needs work
Full review of the whole change (12 files, 1c33248c318c72a9e837627ff71476c8583f1930...e97c827c0b69d684f64524794ed3d94116e86ae0), fanned out across three coupled-file-group workers plus centralized cross-cutting checks. Part of #349 (open tracking issue), explicitly re-scoped to convert one consumer of the e2e-test-hooks seam rather than remove the feature — the description says so and the diff matches that scope exactly. PR title cites ROCMAI-669; that identifier is an internal tracker reference not retrieved for this review.
Blocking
- CI check "E2E tests (MI300X, current)" is red at the reviewed head commit — CI status for
e97c827c0b69d684f64524794ed3d94116e86ae0(not a source location; run: https://github.com/ROCm/rocm-cli/actions/runs/40315449640/job/121546925225).
gh pr view 351 --json statusCheckRollup,headRefOidconfirmsheadRefOidmatches the reviewed commit and this check's conclusion isFAILURE, while every other GPU/OS E2E lane (Strix Halo ×4, W7900, Windows, WSL2) passed on the same run. The job log itself returned 403 in this environment, so the root cause could not be read directly — but the mechanism this PR introduces is a plausible direct cause, not just an unrelated flake: the scenario's gate changed from@requires-no-gputo@requires-os:linux, which is exactly what newly exposesserve-18to a real-GPU lane (MI300X) for the first time. Previously this scenario never ran there at all. Worth ruling out before assuming it's infra noise, since (unlike the Strix Halo/Windows self-hosted flakiness a prior reviewer already characterized as pre-existing) this specific lane has no established flake history against this scenario — it's new exposure.
Confidence 100 · architectural · Fix: pull the MI300X job log and confirm whether the planted fake runtime / simulated host-root GPU node interacts badly with a lane that also has a real AMD GPU and real Lemonade state, before merging.
Non-blocking
- Stale "scripted failure seams" wording in an assert message, contradicting the doc comment immediately above it —
xtask/src/workflow_contract.rs:296(theassert!message insideassert_prebuilt_e2e_lanes_enable_test_hooks), vs. the doc comment atxtask/src/workflow_contract.rs:273-274.
This PR correctly rewrote the function's doc comment from "the scripted Lemonade backend-install failure" to "the simulated host root,ROCM_CLI_TEST_HOST_ROOT" (the compiled-in failure branch is exactly what this PR removes). Theassert!panic message below it in the same function was not updated and still reads "...without it the suite's scripted failure seams are compiled out:\n{block}", describing the now-deleted mechanism. A future reader hitting this assertion failure gets pointed at machinery that no longer exists.
Confidence 85 · mechanical · Fix: update the assert message to match the revised doc comment (e.g. "...without it the suite's test seams, such as the simulated host root, are compiled out").
Decisions for the author
fake-lemonade.rs'srun_lemonade_clisilently succeeds on any subcommand it doesn't recognize — tradeoff. This keeps the fixture small and avoids breaking the scenario if the engine issues some other incidental call, but it also means a future, unexpected invocation shape from the real engine (a new flag, a new subcommand) would pass through unnoticed rather than failing loudly, which could mask a drift between the fixture and the reallemonadeCLI contract it stands in for. The alternative (fail loudly on anything unrecognized) was deliberately not chosen here (the code comments explain the three calls that matter and say "any other subcommand: succeed quietly"); confirm that's still the intended tradeoff as the fixture's surface grows.
Positive signals
- The new
backend_alignment.rsunit test pullsfake-lemonade.rs'sBACKENDS_TABLEviainclude_str!rather than a hand-copied literal, so the fixture and the parser it's meant to drive cannot silently drift apart — verified the extraction is byte-exact and the parser assertions are correct against it. - The planted runtime's
env_id(format!("lemonade-embeddable-{}", rocm_deps::LEMONADE_VERSION)in the test fixture) is verified byte-identical to the format string the real installer uses inengines/lemonade/src/install.rs:201, and the match againstDetectwas traced through to confirm the scenario actually skips the real multi-gigabyte install rather than silently falling through to it — exactly the property the PR's description claims. - Every cross-referenced issue in the description (#467, #437, #729, #510, #466) checks out factually against live GitHub state — titles, open/merged status, and the relationships claimed between them are all accurate.
Deployment notes
Both removed code paths (engines/lemonade/src/lib.rs's scripted install failure, apps/rocm/src/serve_cmd.rs's pre-flight waiver) were compiled only under the e2e-test-hooks feature, so this change has no behavioral effect on release/production builds.
What this covered
Every file in the diff, read completely at the PR head plus surrounding code (the full serve() pre-flight, the Lemonade install/detect flow, the E2eWorld harness, the tag-resolution machinery in expectation.rs), via three independent fanned-out workers grouped by contract coupling (engine/fixture contract; harness-and-steps; feature-tag-and-CI-gating), each of which activated the review-code skill itself and verified its own candidates before reporting. Centralized checks on top: live CI status for the exact head commit, a repo-wide grep confirming no dead references remain to the removed env var/feature seam, independent tracing of the env_id format-string parity between the fixture and the real installer, verification of the AGENTS.md crate-edge and GitHub-Actions-pinning conventions against this diff, and DCO/commit-quality checks (single well-described commit, signed off).
Did not run: a deep git log/blame history pass beyond confirming this is a single commit with no revision churn to review; a prior-PR-review-comments pass on these files (not retrievable at reasonable cost here); local build or test execution (no Rust toolchain available in this environment) — relied on live CI results instead, per the skill's guidance to take CI at face value rather than re-deriving its coverage.
Not reconciled against any PR discussion or review threads — those were deliberately excluded as input to this review's findings.
There was a problem hiding this comment.
No blocking findings.
Non-blocking
- MI300X lane red is unrelated to this PR.
E2E tests (MI300X, current)(job 121546925225) fails only ontherock-next-09(a vLLMuv pip install --reinstallstep), which this diff does not touch. serve-18 passed on that same lane. The identical single unexpected failure appears onmain(runs 40360152181, 40357677968, 40354126258) and on other branches in the same window (e.g. jobs 121795242877, 121882265448, 121704235820). It should not gate this PR. - The fake
lemondnever exits (tests/e2e-cucumber/src/bin/fake-lemonade.rs,run_lemond). The comment saysrocm serveterminates it when the serve fails, but the description says #729 tracks alemondleft running when the backend install fails during serve, which is this scenario's path. Until #729 lands, every Linux lane run can leave a sleeping process behind, and the scenario now runs on every Linux lane. Consider bounding the fake's lifetime, and aligning the comment with what serve does today. - The planted
env_idrepeats the format stringmanaged_engine_runtime_iduses (plant_failing_lemonade_runtimeintests/e2e-cucumber/tests/e2e.rs;apps/rocm/src/engines_cmd.rs:323). Deriving the version fromrocm_depscovers a version bump but not a change to thelemonade-embeddable-prefix. If Detect stopped matching, serve would start the real multi-gigabyte install instead of failing fast. An assertion that the install attempted was the fake's (it already prints a distinctive stderr line) would make that fail quickly. tests/e2e-cucumber/README.md(around lines 244-251) lists the fixture binaries (fake-uv,fake-python,fake-tailscale) and does not mentionfake-lemonade.
Decisions for the author
- The engine unit test reads the e2e crate's
fake-lemonade.rswithinclude_str!and extracts the raw-string literal textually. The table is then checked on every unit-test lane with no copy to drift. The cost is that a product crate's unit tests depend on a path intests/and on the literal's exact spelling. Please confirm that is intended. - The e2e crate gains a
rocm-depsdev-dependency, while the README's "Black-box only … No imports from the rocm-cli codebase" principle stays as written.rocm-coreis already such an edge. Either confirm this extension or adjust the README line. - serve-18 now runs on every Linux lane (GPU and GPU-less) and WSL2, where it previously ran only on the no-GPU lane. The new gate is pinned in
expectation.rs. This is more cadence, and each run starts the fake server noted above.
Good things
- The tag gate is pinned against the real feature-file tag line, so an edit fails a test.
- The no-GPU pre-flight is no longer waived; it runs against a GPU planted under the simulated host root.
- Both
e2e-test-hooksbranches are removed cleanly, and no references toROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILUREremain.
e97c827 to
07bc2be
Compare
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 07bc2be
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
DEGRADED REVIEW (single-context; see What this covered).
Full review of the whole change (14 files, 9875940e...07bc2be8, i.e. everything added since the last nowycondro review at e97c827c, re-verified from scratch). Part of #349 (open tracking issue); converts one consumer (serve-18) of the e2e-test-hooks seam, explicitly re-scoped not to remove the feature itself — description and diff match. ROCMAI-669 (Jira, retrieved) describes exactly this converted scope and is "In Progress", consistent with an in-flight step of #349 (also open). Neither claims this closes either tracker item; no mismatch.
Blocking
- The PR currently has real, substantive merge conflicts against the current tip of
main, inapps/rocm/src/serve_cmd.rs,xtask/src/workflow_contract.rs, and.github/workflows/e2e-selfhosted.yml. Confirmed by actually merging (git merge-tree), not inferred frommergeable_state: dirtyalone.
apps/rocm/src/serve_cmd.rs(no-GPU pre-flight bail):mainhas since landed #284, which added a sibling test hooke2e_simulate_oom_launch()into the exact sameifthis PR edits (now!cpu_only && !scripted_backend_failure && !e2e_simulate_oom_launch() && !reuse_existing && host_has_no_usable_gpuonmain, vs. this PR's!cpu_only && !reuse_existing && host_has_no_usable_gpu). A careless resolution (taking either side wholesale) either keeps the deadscripted_backend_failurethis PR removes, or silently dropse2e_simulate_oom_launch()that #284 added — the correct merge keeps!e2e_simulate_oom_launch()and drops only!scripted_backend_failure.
xtask/src/workflow_contract.rsand.github/workflows/e2e-selfhosted.yml:mainindependently reworded the same doc comment / CI comment this PR rewords (to document the newrocm/e2e-oom-fault-injectionfloor), andmain's own wording still says "the suite's scripted failure seams are compiled out" / "the scripted Lemonade backend-install failure" — the exact stale phrasing this PR's second commit fixed in its own branch, now re-introduced by the conflict unless resolved deliberately.
Confidence 100 · architectural · Fix: rebase onto currentmain, keepe2e_simulate_oom_launch()intact in the pre-flight condition, and reconcile the doc/assert/CI-comment wording so it documents both the oom-fault-injection floor (from #284) and the simulated-host-root rewording (from this PR) without reintroducing "scripted failure seams" language for the now-deleted Lemonade seam.
Non-blocking
None.
Decisions for the author
rocm-depsdev-dependency vs. the e2e README's "black-box only" principle — tradeoff.tests/e2e-cucumber/Cargo.tomladdsrocm-depsas a dev-dependency so the planted runtime'senv_idtracksrocm_deps::LEMONADE_VERSIONinstead of a copy.tests/e2e-cucumber/README.md's Design Principles still read "No imports from the rocm-cli codebase", with onlyrocm-corecalled out as an existing exception (for a harness unit test reading a planted record back throughrocm_core::ManagedServiceRecord).rocm-depsis a second first-party edge the README text doesn't acknowledge. It's a reasonable extension of the same precedent (verified: the derivedenv_idformat is byte-identical toapps/rocm/src/engines_cmd.rs::managed_engine_runtime_id's, andxtask check-crate-edgesexempts dev-dependencies by design), but confirm it's intended and consider updating the README line.- The fake
lemond's lifetime bound relies mostly on a 10-minute backstop in practice — tradeoff, already self-documented.fake-lemonade.rs'srun_lemondexits on Unix re-parenting or afterLEMOND_MAX_LIFETIME(10 min). Since #729 (open) means the real engine does not stoplemondwhen backend install fails — this scenario's own path — re-parenting may not occur until the whole test harness tears down, leaving the 10-minute cap as the actual bound on every Linux-lane run of this scenario until #729 lands. The commit message already states this tradeoff plainly; flagging only so it's confirmed as accepted rather than overlooked. fake-lemonade.rs'srun_lemonade_clisucceeds quietly on any unrecognized subcommand — tradeoff. Keeps the fixture small, but a future unexpected call shape from the real engine would pass through unnoticed rather than surfacing a fixture/contract drift. The in-file comment already explains this choice; confirm it's still intended as the fixture's surface grows.
Positive signals
- The planted manifest's JSON shape is a byte-exact match for
LemonadeInstallManifest's real fields (engines/lemonade/src/install.rs), andresolve_runtime()genuinely only checksmanifest.lemond.is_file()— verified directly in the source, confirming the PR description's "why this doesn't weaken verification" claim rather than taking it on trust. - The planted KFD node's
propertiescontent matches the real per-linekey valueparser incrates/rocm-core/src/host_gpu.rsexactly (and the existing unit tests there use the identical format), so the simulated GPU is recognized for the reason the PR says it is. expectation.rs's new test derives the scenario's tags from the real.featurefile text (feature_tags, matched by@id:) rather than a hardcoded string, so it is a genuine pin against tag drift, not a vacuous one.- The second commit (
07bc2be8) cleanly resolved every non-blocking item a prior round of review raised againste97c827c: the stale "scripted failure seams" assert message, the fakelemondnever exiting, theenv_id-drift blind spot (now caught by asserting the fake's install-refusal string appears exactly twice), and the missingfake-lemonadeREADME entry. - The previously-flagged MI300X CI failure at
e97c827cis resolved: at this head,E2E tests (MI300X, current)is green. The one remaining failing self-hosted check,E2E tests (Strix Halo, Windows, current), was traced via its job log toruntime-resolve-canonical-release(a TheRock wheel-install scenario this diff does not touch) failing as "a regression" — and theE2E self-hostedworkflow has failed on every one of the last 10 completed runs onmainitself, so this is pre-existing lane breakage, not something this change introduced.
Deployment notes
No behavioral change to release/production builds: both deleted branches were compiled only under #[cfg(feature = "e2e-test-hooks")]. No new runtime dependency; rocm-deps is test-only (dev-dependency).
What this covered
DEGRADED REVIEW: fan-out to independent sub-workers was attempted twice (3 workers, grouped by coupling: product-code removal; e2e harness/fixture binary; feature-gating/docs/CI/xtask), and both attempts failed outright on a shared, environment-wide concurrent-subagent ceiling unrelated to this task (the tool explicitly said not to retry a third time). This review therefore ran single-context, with the same file groups and checks the workers would have covered done directly: every file in the diff read completely at the PR head (apps/rocm/src/serve_cmd.rs, engines/lemonade/src/{lib.rs,llamacpp_backend.rs}, tests/e2e-cucumber/{Cargo.toml,README.md,src/bin/fake-lemonade.rs,src/expectation.rs,src/fixture_binary.rs,features/model_serving.feature,tests/e2e.rs,tests/e2e/serving_steps.rs}, xtask/src/workflow_contract.rs, .github/workflows/e2e-selfhosted.yml, Cargo.lock), plus surrounding/callee code (engines/lemonade/src/install.rs, engines/lemonade/src/process.rs, apps/rocm/src/engines_cmd.rs, crates/rocm-core/src/host_gpu.rs, crates/rocm-core/src/hardware_root.rs, xtask/src/crate_edges.rs), live CI status plus one job log for the exact head commit, a 10-run history comparison of the self-hosted workflow on main, a repo-wide grep for dangling references to the removed mechanism, and a real three-way merge against origin/main to check mergeability.
Did not run: a deep git log/blame pass beyond spot checks (used to locate the sibling #284 PR causing the merge conflict); a prior-PR-review-comments pass on these files beyond what the merge-conflict investigation surfaced; local build or test execution (no Rust toolchain invoked; relied on live CI, consistent with the skill's guidance). Not reconciled against this PR's own review discussion/threads — deliberately excluded as input to this review's findings, per method.
🤖 Generated with Claude Code
`@id:serve-lemonade-preparation-recovery` (serve-18) needs Lemonade's llama.cpp backend install to fail twice, so it can pin the retry count and the recovery command. Two `#[cfg(feature = "e2e-test-hooks")]` branches provided that: one in the Lemonade engine's Install RPC that failed on `ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE`, and one in `rocm serve` that waived its no-GPU pre-flight for the same variable. Both are deleted. The `e2e-test-hooks` feature stays: it has other consumers (the simulated host root among them), and removing it is the last step of #349, not this one. The failure now happens where a broken install fails, at a non-zero child process: - `fake-lemonade`, a `[[bin]]` of the suite, is planted as the runtime's `lemonade` and `lemond`. `lemonade backends` reports `llamacpp:rocm` as installable and `lemonade backends install` exits non-zero; `lemond` stays alive until killed. - The install manifest is written beside it. Its `env_id` is derived from `rocm_deps::LEMONADE_VERSION` (a dev-dependency, so no crate-edge entry): `serve` skips its own install only when `Detect` reports exactly that id, and a placeholder silently runs the real multi-gigabyte install instead. - The pre-flight runs unchanged against one KFD GPU node planted under the simulated host root. The root holds nothing else, so the scenario needs no GPU and a GPU runner's devices do not take part. `XDG_RUNTIME_DIR` points inside the scenario too, so the fake server writes nothing outside it. The scenario is now `@requires-os:linux` instead of `@requires-no-gpu`. It still runs on the GPU-less every-PR lane, and now on the Linux GPU lanes and WSL2 as well. It stays off Windows: the host root re-roots Linux paths only, and there `rocm serve` does not notice a managed engine dying at startup (#467), so the failure would never reach the user. A test in `expectation.rs` pins that gate against the real tag line. The engine's unit test now reads the backends table out of the fixture's source with `include_str!`, instead of a copy kept in step by a comment. Workflow and contract-test comments that cited the deleted seam as their example now cite the host root. Checked by mutation: the scenario fails when the fake install succeeds, and fails with its named pre-flight message when the GPU is not planted. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…runtime The fake `lemond` now exits once its parent is gone (Unix re-parenting) and after a 10-minute backstop, so a run whose best-effort teardown fails leaves no process behind while #729 is open. serve-18 now asserts that the planted runtime refused both backend installs, through a constant shared between the fixture and the step. If the planted env_id ever stopped matching Detect, the scenario now fails fast instead of starting the real multi-gigabyte install. Also fixes a stale "scripted failure seams" assert message and lists fake-lemonade among the README's fixture binaries. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The e2e suite is black-box and imports nothing from the rocm-cli codebase, per its README. The planted Lemonade runtime's env_id was the exception: it took the version from a rocm-deps dev-dependency. It now reads the version from runtime-deps.toml, the workspace pin file that rocm-deps' build script generates LEMONADE_VERSION from, so the planted id still follows a version bump with no copy to keep in step. A missing or misspelled pin panics with the file's path. The README's black-box principle names this as the way a scenario gets a pinned version. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The engine's unit test pulled the fake runtime's `lemonade backends` table out of fake-lemonade.rs by matching the text of a raw-string literal, so it depended on how that literal was spelled. The table is now a data file under the e2e fixtures. The fake binary prints it with include_str!, and the engine test reads the same file, as rocm-core's fix tests already read the PowerShell cases there. The pre-push hook lists the table as read by rocm-engine-lemonade, so editing it runs that crate's tests. Also drops three comments that still described serve-18's removed failure hook as waiving the no-GPU pre-flight. Since the rebase, the free-space override is the only scripted waiver left. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
07bc2be to
895c4fb
Compare
|
Thanks for the reviews. The branch is rebased onto current @nowycondro, review on
The engine unit test reading Addressed in
Also in this push: Not changed:
Verified locally on the new head: serve-18 via |
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
juhovainio
left a comment
There was a problem hiding this comment.
Read the diff end to end; nothing here looks wrong in the logic, and replacing the test-only install-failure hook with a planted runtime is a clear improvement. The points below are small, but I'd like them settled before merge.
- The new count of 2 on the fake's refusal line (inline) depends on the engine retrying exactly once and relaying child stderr once per attempt. Please confirm it against the first green e2e run.
- The
src/runtime_pins.rsrow indocs/architecture.mdis out of alphabetical order (inline). - Please grep
docs/testing.md,docs/manual-testing.mdand the README for leftover mentions ofROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE, and for serve-18 being described as@requires-no-gpu. The gate is now@requires-os:linux, so serve-18 depends on theROCM_CLI_TEST_HOST_ROOTseam. - The fake
lemonadematchesstatusandbackendsanywhere in argv instead of as the subcommand. Harmless today, but easy to tighten. - Worth noting: the fake
lemondis reaped only on a Unix parent change or after the 10-minute cap.
The e2e lanes were still pending when I looked, so serve-18's result on CI is not confirmed yet.
| // Both attempts must have been refused by the planted runtime. Without its | ||
| // line, `serve` did not accept the planted runtime and attempted some other | ||
| // install -- for a mismatched `env_id`, the real multi-gigabyte one. | ||
| assert_eq!( |
There was a problem hiding this comment.
The count of 2 assumes the engine retries exactly once and relays the child's stderr once per attempt. Please confirm against the first green e2e run (the mutation check in the PR text is the only evidence so far).
There was a problem hiding this comment.
Confirmed against CI, and the count is now exercised on both failure paths.
- CI, at
895c4fb0: serve-18 passed, with the== 2assertion, onE2E tests(both Ubuntu LTSes) and onE2E tests (Strix Halo, WSL2). Each log showsThen serving stops after one automatic retrypassing. - Why it is exactly 2:
install_llamacpp_backend_with_retrycalls the install closure once, and once more on failure, and never again. Each call runslemonade backends installwith inherited stderr (backend_install_command), and the fake prints its refusal line once per run. The step also pinsretrying onceto exactly one.
Checking this turned up a real flake, now fixed in e0c4e8d. The install fails inside the spawned engine, and without --wait rocm serve notices only an engine that dies within its 200 ms post-spawn check. On a host at load ~18 the engine missed that window, and serve reported readiness: starting with exit 0 after 50 s. serve-18 now runs serve --managed --wait, and follows the rocm logs --service <id> pointer when the engine died after the window. Both paths pass locally (serve step 0.26 s, and 45.3 s with the fixture's install delayed 1 s), with the same counts. The default (no --wait) behaviour is a real defect, recorded in the PR body as a known gap for its own fix.
| | [`src/quit_and_wait.rs`](../tests/e2e-cucumber/src/quit_and_wait.rs) | Retry loop for quitting a TUI that may open a quit-confirm prompt first. | flat | | ||
| | [`src/reader_failure.rs`](../tests/e2e-cucumber/src/reader_failure.rs) | Failure state shared between a background terminal reader and its waiters. | flat | | ||
| | [`src/runtime_root.rs`](../tests/e2e-cucumber/src/runtime_root.rs) | `check_runtime_root_not_nested`: judges the active runtime folder `rocm examine` prints and rejects a doubly nested `runtimes/wheel` path. | flat | | ||
| | [`src/runtime_pins.rs`](../tests/e2e-cucumber/src/runtime_pins.rs) | Reads a pinned runtime version out of the workspace-root [`runtime-deps.toml`](../runtime-deps.toml) as plain data, so a planted runtime keyed by that pin (the Lemonade `env_id`) follows a version bump without importing `rocm-deps`. | flat | |
There was a problem hiding this comment.
Nit: this row sorts before src/runtime_root.rs in the surrounding alphabetical table.
There was a problem hiding this comment.
Fixed in 6898ef9: the runtime_pins.rs row now sorts before runtime_root.rs.
The fake matched `status` and `backends` anywhere in its arguments, so an argument that only equalled one of them would have been answered as that subcommand. It now skips the global --host/--port options the engine passes and classifies the call by the word that follows, in a pure function covered by unit tests. Any `backends` call other than `install` still lists, so a flag the engine adds later gets the table rather than an empty success. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The e2e harness table is alphabetical. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (20 files, ee605065…895c4fb0), fanned out to three coupled-file-group workers (engine/fixture contract; e2e harness/steps/feature; product-code/CI/xtask/docs — the third could not run as an independent worker, see What this covered). Part of an open tracking issue for retiring a compile-time E2E test-hook feature, explicitly scoped to convert one consumer (the Lemonade preparation-recovery scenario) rather than remove the feature itself — description and diff match exactly.
Blocking
- A real, current merge conflict against
main, confined to a doc comment —apps/rocm/src/serve_cmd.rs:868-877
Confirmed independently viagit merge-tree --write-tree origin/main <head>against today's tip, and cross-checked against the GitHub API, which reportsmergeable: CONFLICTING/mergeStateStatus: DIRTYfor this exact head. The conflict is confined to the prose paragraph describing the pre-flight bail's skip conditions:mainindependently reworded the same paragraph when it added OOM-launch fault-injection "either mode" support, and this PR's own rewording (retiring a "backend-failure scenario" mention) overlaps the same lines. The executable code just below (thelet scripted_free_space = …/if !cpu_only && …block) auto-merges cleanly — only the comment conflicts. A careless resolution takingmain's side wholesale would reintroduce the stale "backend-failure scenario" wording this PR correctly retires.
Confidence 100 · mechanical · Fix: rebase onto currentmainand hand-merge this one paragraph so it keeps this PR's retirement of the stale wording while adding back the "(either mode)" detailmainintroduced for OOM-launch. Nothing else in the tree conflicts — two other files an earlier round of feedback flagged as conflicting now merge cleanly, independently re-verified.
Non-blocking
fake-lemonade's CLI dispatch matches tokens anywhere inargv, not the exact subcommand position —tests/e2e-cucumber/src/bin/fake-lemonade.rs:99-110
Scans the whole argument vector instead of checking a specific position. Works for today's three call shapes but a future one carrying both matched tokens could silently hit the wrong branch.
Confidence 75 · logic · Fix: parse structurally (skip flag/value pairs, then match the first remaining token exactly).- A table row in
docs/architecture.mdis out of alphabetical order —docs/architecture.md:810
The newruntime_pins.rsrow sits afterruntime_root.rs; it sorts before it alphabetically.
Confidence 90 · mechanical · Fix: move the row aboveruntime_root.rs. - The planted
env_idformat string is now duplicated a third time, with no compile-time tie to production —tests/e2e-cucumber/tests/e2e.rs:754, vs.engines/lemonade/src/install.rsandapps/rocm/src/engines_cmd.rs'smanaged_engine_runtime_id
A drift between the three literals would makeservesilently fall through to a real multi-gigabyte install instead of the fixture. Mitigated — a runtime assertion elsewhere in this diff makes such a drift fail loudly — but nothing ties the literals together at build time.
Confidence 76 · architectural · Fix: a shared helper in the pin crate would remove the duplication by construction; if that's not viable under the harness's black-box rule, cross-reference the sites in comments. - A source binary is read twice when planting it under two names —
tests/e2e-cucumber/tests/e2e.rs:749-750
Harmless (test-only, tiny binary) but avoidable.
Confidence 70 · mechanical · Fix: read once, write both destinations from the same buffer.
Decisions for the author
- This PR's new test does not itself prove the removed scripted-failure seam is dead — tradeoff. The new scenario setup never touches the retired env var, so reverting only the production removal (while keeping this PR's new test infrastructure) would leave the scenario passing unchanged, since it exercises the new planted-runtime path rather than the old one. Expected for a dead-code removal — its safety rests on the repo-wide absence of remaining references and on CI passing, not on this test — but worth confirming that's the intended coverage story.
- The loose
argvmatching above may be an accepted simplification — confirm it's intentional given today's call shapes never collide, rather than something to tighten later. - A lane with real network egress could turn an
env_iddrift into a slow timeout rather than a fast failure — in this sandboxed CI a fallback to the real install has nowhere to download from, so a drift fails fast; a lane with egress would instead hang. Worth confirming a bounded timeout wraps this scenario everywhere it runs.
Positive signals
- The scenario's lane gate is pinned against the real feature-file text (not a hardcoded copy), so editing the tag or the scenario id changes the test's own input and flips its assertions.
- The planted GPU is traced end-to-end as a real mechanism, not a no-op: a planted KFD sysfs node reaches the real GPU-detection parser and resolves the pre-flight's "no usable GPU" check correctly on every lane the scenario now runs on, GPU-bearing or not.
- The planted runtime's verification surface and manifest shape are confirmed field-for-field against the real installer code, matching the PR description's "why this doesn't weaken verification" claim directly rather than on trust.
- A shared fixture file for the backend-status table is read by both the fake binary and a new engine unit test via
include_str!, so the table and the parser it drives cannot silently drift apart. - Grepping the whole tree at the head finds no remaining reference anywhere to the removed env var or its old gate — a clean removal with no orphaned comments, docs, or contract tests describing the retired mechanism.
- Black-box test-harness discipline is maintained and strengthened: a new helper reads a version pin from the workspace pin file as plain text rather than importing a product crate, and the harness docs are updated to describe the pattern; no new dependency was added.
- CI is fully green at this exact head, including self-hosted GPU lanes an earlier round of review found flaky — resolved now.
- Every issue the description cites checks out factually against live state, and one of them is a real platform gap this PR's own work surfaced and the author filed candidly rather than working around quietly.
Deployment notes
No behavioral change to release/production builds: the removed branches and the renamed pre-flight gate variable were compiled only under the test-only feature already gating this area. No new runtime dependency; the harness gains only a new test-only binary target.
What this covered
Every file in the diff (20 files) read completely, plus surrounding/callee code: the full pre-flight function and its doc comments, the real installer's runtime-resolution logic, the GPU-detection chain, and the e2e harness's world/setup code. Work was fanned out to three coupled-file-group workers. Two ran independently; the third (serve command pre-flight, workflow/xtask contract checks, CI workflow yaml, and docs) could not be spawned as an independent worker — the spawn and one respawn both hit a concurrency ceiling shared with unrelated sessions in this environment — and was reviewed directly in this context instead, so that slice was not an independent pass.
Centralized checks: live CI status for the exact head commit (fully green, no failures); independent re-verification of mergeability against the current tip of main via a real three-way merge tool, cross-checked against the forge's own mergeability status (both agree, see Blocking); a repo-wide grep for dangling references to the removed mechanism; retrieval of every issue the description cites.
Did not run: the test suite locally (no toolchain available in this environment) — relied on live CI and static tracing instead; a deep history pass beyond what was needed to diagnose the merge conflict.
Not reconciled against this PR's own review discussion or threads — deliberately excluded as input to these findings, per method.
🤖 Automated by an AI agent
| // (probe returns `None`). The E2E-only OOM-launch fault injection stands in | ||
| // for the GPU its lane does not have. | ||
| // | ||
| // The scripted free-space override opens the same waiver: it exists to refuse a | ||
| // The scripted free-space override waives it too: it exists to refuse a | ||
| // Lemonade install on a drive that cannot hold it, which `rocm serve` only | ||
| // reaches past this bail. | ||
| let scripted_backend_failure = cfg!(feature = "e2e-test-hooks") | ||
| && [ | ||
| "ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE", | ||
| "ROCM_E2E_LEMONADE_FREE_SPACE_BYTES", | ||
| ] | ||
| .iter() | ||
| .any(|variable| std::env::var_os(variable).is_some()); | ||
| let scripted_free_space = cfg!(feature = "e2e-test-hooks") | ||
| && std::env::var_os("ROCM_E2E_LEMONADE_FREE_SPACE_BYTES").is_some(); | ||
| if !cpu_only | ||
| && !scripted_backend_failure | ||
| && !scripted_free_space |
There was a problem hiding this comment.
Blocking: this PR has a real, current merge conflict against main, confined to this paragraph. Confirmed independently via git merge-tree --write-tree origin/main pr/351 against today's tip, and cross-checked against the GitHub API (mergeable: CONFLICTING, mergeStateStatus: DIRTY).
main independently reworded this same paragraph when it added OOM-launch fault-injection "either mode" support, and this PR's own rewording (retiring the "backend-failure scenario" mention) overlaps the same lines. The executable code just below (the let scripted_free_space = … / if !cpu_only && … block) auto-merges cleanly — only this prose conflicts.
A careless resolution that takes main's side wholesale would reintroduce the stale "backend-failure scenario" wording this PR correctly retires. The correct resolution keeps this PR's retirement of that wording while adding back the "(either mode)" detail main introduced for OOM-launch. Confidence 100.
(FYI: the other two files an earlier round of feedback flagged as conflicting, xtask/src/workflow_contract.rs and .github/workflows/e2e-selfhosted.yml, now merge cleanly — main's edits there landed in non-overlapping regions, re-verified independently.)
There was a problem hiding this comment.
Resolved in the merge 1174198 (a merge commit, not a rebase, since your review is on the current head). The merged comment keeps this PR's removal of the backend-failure seam and takes main's "(either mode)" wording: "The E2E-only launch fault injection (either mode) stands in for the GPU its lane does not have." The executable block merged cleanly and is unchanged: !scripted_free_space && !e2e_simulate_oom_launch() && …. The doc on reuse_pregate_opens, which names the same hook, now says "either mode" too. git grep finds no remaining mention of the removed seam.
| const LEMOND_MAX_LIFETIME: Duration = Duration::from_mins(10); | ||
|
|
||
| fn run_lemonade_cli(args: &[String]) -> ExitCode { | ||
| if let Some(index) = args.iter().position(|arg| arg == "backends") { |
There was a problem hiding this comment.
Non-blocking: this dispatch matches tokens anywhere in argv (args.iter().position(|arg| arg == "backends"), and args.iter().any(|arg| arg == "status") below) rather than checking a fixed subcommand position. It happens to work for the three argv shapes the real engine builds today, but a future call shape carrying both tokens (e.g. a --host/--port value that happens to equal one of the matched words) would silently hit the wrong branch. Confidence 75.
Fix suggestion: parse structurally (skip flag/value pairs, then match the first remaining token exactly) rather than scanning the whole vector.
There was a problem hiding this comment.
Fixed in 90dee28. The fake now skips the global --host/--port options the engine passes and matches the first remaining word exactly: status alone, backends install …, or any other backends call (which still lists, so a flag the engine might add later gets the table rather than an empty success). Anything else succeeds quietly, as before. The classification is a pure function with unit tests in the bin, run by cargo test. I checked that moving the status match off the subcommand position turns them red.
| | [`src/quit_and_wait.rs`](../tests/e2e-cucumber/src/quit_and_wait.rs) | Retry loop for quitting a TUI that may open a quit-confirm prompt first. | flat | | ||
| | [`src/reader_failure.rs`](../tests/e2e-cucumber/src/reader_failure.rs) | Failure state shared between a background terminal reader and its waiters. | flat | | ||
| | [`src/runtime_root.rs`](../tests/e2e-cucumber/src/runtime_root.rs) | `check_runtime_root_not_nested`: judges the active runtime folder `rocm examine` prints and rejects a doubly nested `runtimes/wheel` path. | flat | | ||
| | [`src/runtime_pins.rs`](../tests/e2e-cucumber/src/runtime_pins.rs) | Reads a pinned runtime version out of the workspace-root [`runtime-deps.toml`](../runtime-deps.toml) as plain data, so a planted runtime keyed by that pin (the Lemonade `env_id`) follows a version bump without importing `rocm-deps`. | flat | |
There was a problem hiding this comment.
Non-blocking: this row is out of alphabetical order — it's placed after runtime_root.rs, but "runtime_pins" sorts before "runtime_root". Confidence 90.
Fix: move this row above the runtime_root.rs row.
There was a problem hiding this comment.
Fixed in 6898ef9 (same fix as your earlier comment on this row).
|
|
||
| let version = e2e_cucumber::runtime_pins::pinned_version("lemonade"); | ||
| let manifest = serde_json::json!({ | ||
| "env_id": format!("lemonade-embeddable-{version}"), |
There was a problem hiding this comment.
Non-blocking: this env_id format string (lemonade-embeddable-<version>) is now duplicated a third time, with no compile-time tie to the two production copies (engines/lemonade/src/install.rs's format!("lemonade-embeddable-{version}") and apps/rocm/src/engines_cmd.rs's managed_engine_runtime_id). A drift between any of the three literals would make serve silently fall through to a real multi-gigabyte install instead of using this fixture.
This is mitigated — serving_steps.rs asserts the fake's refusal line appears exactly twice, so a drift fails loudly rather than silently — but nothing ties the three literals together at build time. Confidence 76.
Fix suggestion: a shared rocm_deps::lemonade_env_id(version) helper would remove the duplication by construction; if that's not viable under this harness's no-rocm-deps-import rule, cross-reference the three sites in doc comments so an edit to one is visibly incomplete without the others.
There was a problem hiding this comment.
Took the helper suggestion for the two production copies, in 37b509d. rocm_deps::lemonade_env_id(version) now builds the id in both places: the engine's manifest writer (install.rs) and rocm's managed_engine_runtime_id. Those are the two sides of the exact-string comparison that decides whether serve skips its install, so they can no longer drift. rocm-deps already holds the other helpers derived from the pin (archive name, download URL), and docs/architecture.md's entry for it now lists the id. The engines_cmd.rs test keeps its literal on purpose, so a change to the helper's shape fails there.
The e2e copy stays: the suite imports nothing from the workspace (README design principles), which is why it reads the pin as data. A comment beside it now names rocm_deps::lemonade_env_id as the helper it mirrors, and serve-18's refusal count catches a drift at run time.
| install_fixture_binary("fake-lemonade", &lemond); | ||
| install_fixture_binary("fake-lemonade", &lemonade); |
There was a problem hiding this comment.
Non-blocking: these two calls each independently read the same source binary's bytes from disk to plant it under two names. Harmless (test-only, tiny binary) but avoidable. Confidence 70.
Fix suggestion: read the source bytes once and write both destinations from the same buffer.
There was a problem hiding this comment.
Fixed in b59a9ec: the new install_fixture_binary_as reads the fixture once and writes both names from the same buffer. install_fixture_binary keeps its single-destination form for the other callers.
Resolves the one conflict, in the comment above serve's no-GPU pre-flight bail: main named the launch fault injection's two modes, and this branch had dropped the removed backend-failure seam from the same sentence. The merged comment keeps both changes, and the reuse pre-gate's doc now says "either mode" too. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The engine writes `lemonade-embeddable-<version>` into the install manifest, and rocm compares that id with what the engine's Detect reports to decide whether its own install is already done. Each side spelled the shape out, so the two could drift apart and send serve to the real multi-gigabyte install. Both now call rocm_deps::lemonade_env_id. The e2e suite still repeats the shape, since it imports nothing from the workspace; a comment there names the helper it mirrors, and serve-18's refusal count catches a drift. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The planted runtime installs the same fixture binary as both `lemond` and `lemonade`. install_fixture_binary_as reads the fixture once and writes every destination from that buffer; install_fixture_binary keeps its single-destination form for the other callers. Also names rocm_deps::lemonade_env_id as the helper the planted env_id mirrors. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The backend install fails inside the spawned managed engine. Without --wait, rocm serve notices an engine that dies only within its 200 ms post-spawn check and reports a later death as `starting` with exit 0 once the 45 s readiness probe gives up. On a loaded host the engine missed that window and serve-18 failed with "serve unexpectedly succeeded". With --wait a later death fails the serve with a pointer to `rocm logs --service <id>`; the step follows that pointer, as a user would, before asserting the retry and recovery lines. An engine that dies inside the window still fails the serve with its log tail. engine_death_report tells the two messages apart and is unit-tested on the CLI's wording; a failed serve carrying neither fails the step and names both, instead of surfacing as a missing retry line. The default (no --wait) behaviour is a product defect and is left for its own fix. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
|
@juhovainio thanks for both rounds. The inline points each have a reply in their own thread, naming the fix commit. On this review's "Decisions for the author":
Separately, checking your count-of-2 question turned up a flake in serve-18. It also exposed a default-path defect in |
juhovainio
left a comment
There was a problem hiding this comment.
No blocking findings, but I'm requesting changes for the items below.
- Overstated claim (
e2e.rscomment and PR body): a drifted plantedenv_iddoes not "fail fast instead of downloading".servestarts the real install first and the refusal-count assertion only fails afterwards. Reword or add an earlier guard (inline). - Red-run evidence: I can't see in the PR text that the new tests were shown to fail with their target broken (e.g. drifted
env_id,--waitremoved). Please state it. LogTailQuotedbranch: confirm the Then-steps read stderr there (inline).- Lane coverage: serve-18 now also runs on the MI300X and Strix Halo WSL2 lanes. The planted host root should isolate it, but CI hasn't confirmed that yet.
- Slow path: under load the
--waitpath can spend the full 45 s readiness window on the blocking lane. The step comment documents it; just noting the cost.
CI: nothing failing so far; build-and-test, Windows, Strix/Windows E2E and coverage were still running when I looked, and MI300X E2E is queued on the downed runner pool. ROCMAI-669 scope matches.
| // Mirrors `rocm_deps::lemonade_env_id`, which the engine and `rocm` | ||
| // both use; the suite cannot import it. A drift sends `serve` to | ||
| // the real install, which serve-18's refusal count catches. | ||
| "env_id": format!("lemonade-embeddable-{version}"), |
There was a problem hiding this comment.
The comment (and the PR body's "fails fast instead of downloading") overstates this. If the planted env_id drifts, serve first starts the real multi-GB install, and serve-18's refusal count only fails afterwards. Please reword to "fails late, after the real install attempt", or add an earlier guard.
| let (logs, logs_stderr, logs_rc) = | ||
| crate::run_rocm(world, &["logs", "--service", &service_id]); | ||
| assert_eq!( | ||
| logs_rc, 0, |
There was a problem hiding this comment.
In the LogTailQuoted branch nothing is appended to stdout, unlike the LogsPointer branch. Please confirm the retry/recovery Then-steps read stderr (where the quoted log tail lives), or append it so both paths feed the assertions identically.
| // Both attempts must have been refused by the planted runtime. Without its | ||
| // line, `serve` did not accept the planted runtime and attempted some other | ||
| // install -- for a mismatched `env_id`, the real multi-gigabyte one. | ||
| assert_eq!( |
There was a problem hiding this comment.
This count also proves the planted runtime was used, but it only fails after any real install it would otherwise have run. Consider saying so in the message so a mismatch isn't mistaken for a fast failure.
Part of #349. This converts one consumer of the
e2e-test-hooksfeature, the Lemonade recovery scenario, and does not remove the feature: it still has other consumers, among them the simulated host root (#510) thatdocs/release-trust.mdrelies on being compiled out of release builds, so removing it is the last step of #349.Problem
@id:serve-lemonade-preparation-recovery(serve-18) needs Lemonade's llama.cpp backend install to fail twice, so it can pin the retry count and the recovery command. It got that failure from two#[cfg(feature = "e2e-test-hooks")]branches in product code:engines/lemonade/src/lib.rsfailed the Install RPC whenROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILUREwas set;apps/rocm/src/serve_cmd.rswaivedrocm serve's no-GPU pre-flight for the same variable.So the scenario tested a code path that exists only in test builds.
Change
Both branches are deleted. The failure now happens where a broken install fails: at a non-zero child process. The pre-flight's other E2E waivers (the scripted free-space override and the OOM-launch fault injection) are untouched.
The runtime.
fake-lemonadeis a new[[bin]]of the suite, planted as the runtime'slemonadeandlemond:lemonade backendsreportsllamacpp:rocmas installable;lemonade backends installexits non-zero and prints a fixed refusal line;--host/--portoptions, not by a word anywhere in its arguments (unit-tested in the bin);lemondstays alive until its parent is gone (Unix re-parenting) or a 10-minute cap passes, so a failed teardown leaves nothing running.The install manifest is written beside it.
serveskips its own install only whenDetectreports exactlylemonade-embeddable-<pinned version>, and anything else silently starts the real multi-gigabyte install, so the plantedenv_idtakes the version from the workspace pin fileruntime-deps.toml, read as data. The suite stays black-box, with norocm-depsimport, and a version bump needs no copy updated.--wait. serve-18 runsrocm serve --managed --wait. The install fails inside the spawned engine process, and--waitmakes a death at any point fail the serve. An engine that dies within the first 200 ms fails it with its log tail; one that dies later fails it with a pointer torocm logs --service <id>, which the step follows, as a user would, before asserting the retry and recovery lines.Proof the fake was used. serve-18 asserts the fake's refusal line appears exactly twice, through a constant shared by the fixture and the step. If the planted
env_idever stopped matchingDetect, the scenario fails fast instead of downloading.The GPU. The pre-flight runs unchanged, against one KFD GPU node planted under the simulated host root (
ROCM_CLI_TEST_HOST_ROOT). The root holds nothing else, so the scenario needs no GPU, and a GPU runner's own devices do not take part.XDG_RUNTIME_DIRalso points inside the scenario, so the fake server writes nothing outside it.The gate.
@requires-no-gpu→@requires-os:linux.E2E testslane, onE2E tests (MI300X)and onE2E tests (Strix Halo, WSL2).rocm servedoes not notice a managed engine dying at startup (serve: on Windows a managed engine that dies at startup is reported as success #467, fix in fix(serve): watch a Windows engine through its handle and keep a concurrent stop (ROCMAI-675) #437), so the failure would never reach the user.expectation.rspins this gate against the real tag line.The planted
env_id. The engine's manifest writer androcm'smanaged_engine_runtime_idboth build the id through a newrocm_deps::lemonade_env_id, so the two sides of that exact-string comparison cannot drift apart. The suite repeats the shape, since it imports nothing from the workspace; a comment names the helper it mirrors.The backends table is a fixture file,
tests/e2e-cucumber/fixtures/lemonade_backends_table.txt. The fake prints it, and the engine's unit test reads the same file withinclude_str!, so the table the scenario relies on is also checked against the engine's parser on every unit-test lane. The pre-push hook'sCROSS_CRATE_READSlists it, so editing it runsrocm-engine-lemonade's tests.Comments in the workflow, the contract test,
serve_cmd.rsanddocs/testing.mdthat cited the deleted seam now cite what remains.docs/architecture.mdand the suite README list the two new files.Why this doesn't weaken verification
resolve_runtimeaccepts the planted manifest because it only checks that the manifest parses and thatlemondis a file. No checksum, version or signature is on this path; the sha-pinned archive download lives inprepare_embeddable, whichservedoes not reach onceDetectmatches. The no-GPU pre-flight is no longer waived for this scenario. It passes for the same reason it passes on a real GPU host.Known gap, not fixed here
Without
--wait,rocm serve --managednotices a managed engine that dies only within its 200 ms post-spawn check, on every platform. An engine that dies later is reported asreadiness: startingwith exit 0 once the 45 s readiness probe gives up, and the failure is visible only in the service log. Reproduction on Linux: plant a Lemonade runtime whosebackends installfails after a 1 s delay (serve-18's fixture with a sleep added), then runrocm serve Qwen3-0.6B-GGUF --engine lemonade --managed. It printsreadiness: startingand exits 0 after about 45 s (measured: 45.3 s). The same death with--waitfails the serve. serve-18 uses--waitfor that reason, and so does not cover the default path. The default-path defect needs its own fix.Test plan
serve-18passes locally viacargo xtask e2eon Linux on both failure paths: with the real fixture (the engine dies inside the 200 ms window; the serve step takes 0.2 s) and with the fixture's install delayed by 1 s (the engine dies after it; the serve step takes 45 s and the recovery lines come fromrocm logs). Before--wait, the second path failed withserve unexpectedly succeeded, and so did the first on this host under load.installed: the engine unit test fails;@requires-no-gpuor removed: the tag pin fails;statusmatch moved from the subcommand position to the last argument: its classification test fails.fmt; both clippy invocations;cargo test --workspace --all-targets;xtask check-architecture-doc,check-crate-edges,check-module-size,manifest --check.cargo test --workspacehas three failures on this host in files the change does not touch, all from the sandbox: twobuild.rsgit tests (global commit signing inside temp repos) and one AIMdockerstop test (podman installed asdocker).cargo xtask smokepasses every check up to its bubblewrap sandbox step, which this host cannot run (no network namespace).Related
None), so no planted GPU would be needed.lemondleft running when the backend install fails during serve. The fake's lifetime cap covers the scenario until then.