Repository navigation
Conversation
3388ba7 to
2df8565
Compare
3c6e9a1 to
124b181
Compare
|
Rebased onto
Local results are in the updated description. The full mock suite ran 237 scenarios. The 3 unexpected failures were all local: a port already held on the shared test machine, and two load flakes that pass on their own. The PR stays in draft until required CI is green. |
tomastola
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (base 1c33248c, head 124b1815).
Blocking
- examine-17's Then step can pass without asserting anything —
tests/e2e-cucumber/tests/e2e/examine_steps.rs:1014-1022(assert_gpus_match_kfd_nodes)
The step returns silently whenkfd_gpu_node_versions(...)isNoneor empty, and again whenhost_has_lspciis false. The scenario now plants eight simulated GPUs, so those early returns are exactly what hides a planted tree the CLI no longer reads. If the planting,world.host_pathor the CLI's probe root regress, the scenario stays green. The siblingassert_gpu_target_matches_kfdgot a "nothing may quietly decline" guard in this PR; this step did not.
Confidence 70 · logic · Fix: whenworld.simulated_hostisSome, assert the node count equalshost.gpus.len()and fail instead of returning. Skip thelspcibail-out for simulated hosts.
Non-blocking
fake-host-toolanswers plausibly for argv nobody planted —tests/e2e-cucumber/src/bin/fake-host-tool.rs:43-49
Lookup falls back from the full command line to the bare tool name, souname -norlspciwith different flags return the canned-a/-nn -Dtext with exit 0. If the CLI changes itslspciflags, the stand-in will not disagree, which weakens the drift protection this PR is built around.
Confidence 70 · logic · Fix: fall back to the bare name only for argument-less tools (mpirun,lsmod); otherwise exit 127 with a loud stderr message.- Stale tag name in comments —
.github/workflows/ci.yml:1527says "@requires-gpuscenarios resolve to skip here", andfeatures/dash.feature:329says "@requires-wslrather than@requires-gpu"; the skipping tag is now@requires-real-gpu.
Confidence 85 · mechanical · Fix: rename in both. plant_simulated_hostpanics on a second call into the same root —simulated_host.rs:437(link(), symlink already exists). Loud, but undocumented; two Given steps in one scenario would trigger it.
Confidence 70 · mechanical · Fix: document it, or fail with a message naming the double plant.- Unused
Platform::Wsl2 { rocdxg_ready: false }variant —simulated_host.rs:137, 247-250, 317hasldconfig/rocdxgbranches nothing constructs, so they are untested dead code.
Confidence 70 · mechanical · Fix: drop the variant and branches, or add a scenario that uses it. - README does not tell GPU-box users about
E2E_HARDWARE— the tag table mentions it, but the environment-variable table and run examples do not. A plaincargo xtask e2eon a GPU machine now skips every@requires-real-gpuscenario without saying why where people look.
Confidence 70 · documentation · Fix: add anE2E_HARDWARErow (defaultsimulated) and anE2E_HARDWARE=real cargo xtask e2eexample.
Decisions for the author
- Coverage moves off real hardware, and Windows/macOS lose scenarios — tradeoff
- serve-13 (
serve-no-gpu-fails-fast) was@requires-no-gpu, so it ran on any no-GPU lane including Windows and macOS; it is now@requires-os:linuxon a simulated host. examine-10 went from@requires-bare-metal(skips only on WSL) to@requires-os:linux, so--no-frameworksis no longer checked on Windows or macOS. - serve-13/15/16/19/20/21, examine-10/11/17, diagnose-08/15/16/17/18 and driver-install-01..03 no longer run against a real GPU or WSL host. Only examine-04 has a real twin (examine-29); examine-30/31 check layout assumptions only. Real masked-device
serverefusals and real-WSLdiagnose/driver-plan paths run in no lane. - The simulated run is hermetic and runs on every PR, which is a real gain. Cheap mitigations: a
@requires-real-gpucopy of serve-15 and a@requires-wslcopy of diagnose-17, as examine-29 does for examine-04; keep a non-simulated no-GPU variant for Windows/macOS. Note the description's "each moved scenario still asserts the same behaviour" and "gain coverage" do not mention the Windows/macOS loss.
- serve-13 (
- Fake vLLM ignores everything except
--hostand--port— tradeoff (src/bin/fake-vllm.rs:30-36)
serve-11 proves selection, runtime lookup, launch and readiness, but not that the CLI passes sane vLLM flags;--port=Nwould be ignored and fall back to 8000. Recording argv under the root and asserting the expected flags would close this; acceptable if the flags are covered elsewhere. - serve-11 failed in the test plan on a port conflict (127.0.0.1:11435) — non-blocking-improvement
A fixed port in a hermetic harness invites this on shared runners; an ephemeral port would remove the flake.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 124b181
I read the full diff for this PR (simulated-host infrastructure, expectation/tag plumbing, CI workflow changes, and the migrated e2e feature/step files), cloned the repo at the PR head commit, and cross-checked the two concerns from @tomastola's existing CHANGES_REQUESTED review against the current code. No Rust toolchain is available in this environment, so nothing was compiled or executed — everything below is from static reading of the diff and surrounding code.
Confirmed still open (both from the existing review)
-
examine-17's assertion can still pass without checking anything —
tests/e2e-cucumber/tests/e2e/examine_steps.rs:1012-1016(assert_gpus_match_kfd_nodes). The function returns silently ifkfd_gpu_node_versions(...)isNone/empty, before ever comparing against the planted host. Compare this to its siblingassert_gpu_target_matches_kfd(same file, lines 945-956), which was given an explicitif let Some(expected) = world.simulated_host....gfx_target() { assert_eq!(...) }guard so "nothing below may quietly decline to check it."assert_gpus_match_kfd_nodeshas no equivalent direct comparison againstworld.simulated_host.as_ref().map(|s| s.host.gpus.len()). The feature comment atfeatures/examine.feature:226-227("the step's premises always hold") documents the intent but the code doesn't enforce it — a regression in planting,world.host_path, or the CLI's probe root would make this scenario pass vacuously instead of failing. Confidence: high (70-80%), this is a real gap, not a false positive. -
Coverage-loss tradeoff is real and still present — verified via
git grepat head:features/model_serving.feature:135(serve-13) now carries@requires-os:linuxonly, down from@requires-no-gpu(which also ran on Windows/macOS).features/examine.feature:111(examine-10) is similarly now@requires-os:linux, down from@requires-bare-metal(which only skipped WSL). Both comments in the feature files candidly explain the simulated premise, but the net effect for those two scenarios is that Windows/macOS lanes no longer exercise GPU-required-refusal and--no-frameworksbehavior. This is a tradeoff for the author/reviewers to accept or mitigate (e.g. a non-simulated Windows/macOS variant), not something I'd block on myself, but it is accurately described in the prior review and still applies at this head commit.
Additional observation (non-blocking, low confidence)
tests/e2e-cucumber/src/simulated_host.rs(filter_real_path/link_stand_ins, around line 402) rebuilds the filtered-PATH fixture from scratch on every scenario invocation — a freshread_dir+stat+symlink pass over every PATH directory per scenario. With 19+ scenarios now going through this path per run, this adds I/O overhead to the per-PR suite. This is a performance nit, not a correctness issue, and I did not measure actual wall-clock impact.
Not independently re-verified
I did not re-derive every item in the prior review's non-blocking list (fake-host-tool argv fallback, stale @requires-gpu tag mentions in comments, plant_simulated_host panic-on-double-call, unused Wsl2 { rocdxg_ready: false } variant, README gap, fake-vllm flag handling, serve-11 port conflict) — a spot read of the code is consistent with those being accurate, but I did not independently trace each one line-by-line given the medium-effort budget for this pass. I found no additional correctness bugs beyond what's above and what's already flagged.
🤖 Generated with Claude Code
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 124b181
Review — needs work
Full review of the whole change, run as five independent file-grouped passes plus consolidation.
Implements ROCMAI-677 ("E2E: describe GPU hosts with simulated machines so GPU-visibility scenarios run on every PR"); the PR description's claims (HardwareMode gating, simulated_host.rs layout, fake-vllm/fake-host-tool stand-ins, nineteen scenarios moved to simulated machines, examine-29/30/31 real-hardware/drift checks) match what the diff contains, verified across passes with no mismatch found.
Blocking
- CI check "E2E tests (MI300X, current)" is failing on the reviewed commit —
.github/workflows/e2e-selfhosted.yml(self-hosted MI300X lane); run: https://github.com/ROCm/rocm-cli/actions/runs/40314879688/job/121545019785
Confirmed viagh pr view 517 --json statusCheckRollup(conclusion: FAILURE) and the rendered Actions page, which shows the "Run E2E tests on GPU hardware" step exiting with code 1; every other check on the PR is green. Full job logs were not retrievable from this environment (the GitHub Actions REST log/run API endpoints returned 403/Access Denied through an egress proxy), so the specific failing scenario or assertion could not be identified. This PR's owne2e-selfhosted.ymlchanges are what first setE2E_HARDWARE: realon this exact lane, so the failure may be the first real exercise of the new@requires-real-gpugating on that hardware rather than an unrelated flake — that is plausible, not established.
Confidence 100 · architectural · Fix: obtain the job's actual log/annotation (needs sign-in or maintainer access), determine whether it's a genuine regression in the new real-GPU gating/hardware-state assumptions or a known self-hosted-runner flake, and resolve or re-run before merge.
Non-blocking
ldconfigcache header reports a fixed count of 5 libraries that doesn't match the number actually written —tests/e2e-cucumber/src/simulated_host.rs:308-320. The cannedldconfig -poutput always prints "5 libs found in cache" but only 4 unconditional entries are written (a 5th,librocdxg.so, is added only forPlatform::Wsl2 { rocdxg_ready: true }), so on bare-metal and WSL2-not-ready machines the printed header disagrees with the printed list below it. No consumer currently parses the header count (rocm-core's checks match by soname substring), so nothing currently breaks, but it's a fidelity gap in a fixture whose whole point is to match realldconfigoutput.
Confidence 80 · mechanical · Fix: compute the header count from the number of lines actually emitted rather than hardcoding it.bare_metal_layout_drift's GPU-topology checks early-return when the simulated side reports zero GPUs —tests/e2e-cucumber/src/simulated_host.rs:520-557(let Some((_, simulated_props)) = simulated_gpus.first() else { return drift; }). Two drift checks (non-zerogfx_target_versionon the CPU KFD node; DRM card vendor/device id shape) never execute when comparing against a no-GPU simulated host. Every current call site only drives this withinstinct_mi300x(1), so it's unreachable today, but it's a latent gap if this drift check is ever reused for a no-GPU scenario.
Confidence 70 · logic · Fix: run those two checks unconditionally, independent of whether the simulated side has a GPU.fake-host-tool.rsredeclaresTOOL_OUTPUT_DIRandTEST_HOST_ROOT_ENVinstead of importing them fromsimulated_host.rs—tests/e2e-cucumber/src/bin/fake-host-tool.rs:22,24vstests/e2e-cucumber/src/simulated_host.rs:38,41. The two copies are held in sync only by a "must match" doc comment with no compiler enforcement; the sibling binaryfake-vllm.rs:17proves a[[bin]]target in this crate canuse e2e_cucumber::...to reach the crate's own lib, so this duplication isn't architecturally forced (unlike the separate, legitimately-forced duplication againstrocm_core::hardware_root::TEST_HOST_ROOT_ENV, whichfake-host-tool.rscannot reach).
Confidence 75 · mechanical · Fix: import the two constants frome2e_cucumber::simulated_hostinstead of redefining them.
Decisions for the author
@id:examine-detects-gpu-and-driver(examine-04) keeps its scenario id while changing what it verifies — tradeoff. Before this PR the id denoted "verified against the real installed GPU" (rare,@requires-gpu-gated, self-hosted-only runs); after, the same id denotes "verified against a simulated Instinct GPU" (frequent,@requires-os:linux, every Linux PR lane), with a new, narrower id (examine-detects-this-machines-gpu-and-driver, examine-29) now covering what the old id used to mean, and a nightly drift check (examine-30/31) mitigating simulated/real divergence. This is well-reasoned and documented in the feature-file comments, but if any external dashboard or flaky-test tracker keys history by scenario id, examine-04's historical baseline will now be read against a very different population of runs. Worth confirming no such tracker exists, or that the discontinuity is acceptable.
Positive signals
bare_metal_layout_drift,wsl_layout_drift, andlspci_drift(simulated_host.rs) are explicit, tested mechanisms for catching simulation/reality divergence — e.g. a regression test pins the exactgfx_target_version-in-propertiesvs standalone-file bug that motivated simulating hardware in the first place — directly addressing the core risk that a simulated-host suite could drift from what real hardware actually does.HardwareMode::parse(expectation.rs) fails closed: an unrecognizedE2E_HARDWAREvalue panics the run rather than silently defaulting tosimulated, so a typo can't quietly skip every real-GPU scenario while CI stays green — and the one call site (tests/e2e.rs) is wired to actually propagate that panic rather than swallow it.assert_vllm_default's new simulated-only readiness assertion (serving_steps.rs) closes a real gap:rocm serveexits 0 regardless of readiness, so without this check a simulated vLLM that never came up would have passed silently.feature_body(expectation.rs) reads the actual.featurefile text to assert a scenario's premise line is really present, so deleting the underlying Gherkin step would fail this test rather than silently certifying stale coverage.
Deployment notes
None — no product code changes; this PR only changes which CI lanes exercise which existing e2e scenarios and adds new test infrastructure.
What this covered
Every file in the diff, read completely at PR head 124b1815e0ef5359513dece166627e3f81f9e04b against base d59f596f6e08297c207f008021eecb37defc3558 (32 files, +1991/-261), split across five independent file-grouped passes: (1) tests/e2e-cucumber/src/simulated_host.rs; (2) tests/e2e-cucumber/src/expectation.rs + mock_server.rs + lib.rs; (3) tests/e2e-cucumber/src/bin/fake-host-tool.rs + fake-vllm.rs + tests/e2e/simulated_host_steps.rs + tests/e2e.rs; (4) the remaining tests/e2e/*_steps.rs files plus all ten changed .feature files; (5) docs/testing.md, docs/ci-hardware-testing.md, the three .github/workflows/*.yml files, tests/e2e-cucumber/Cargo.toml, tests/e2e-cucumber/README.md. Each pass also read relevant surrounding/consumer code outside its primary scope (e.g. crates/rocm-core/src/examine.rs, openmpi.rs, apps/rocm/src/service_launch.rs, engines/vllm/src/runtime.rs) to verify specific cross-file claims. Step 7 (design review) questions were answered from material the five passes already surfaced (scenario-id continuity, simulation fidelity, real-vs-simulated gating design) rather than a separate fan-out round.
CI: gh pr view 517 --json statusCheckRollup read in full; one check (E2E tests, MI300X, current) reports FAILURE, all others SUCCESS/SKIPPED — reflected above as a blocking finding. Job logs were not retrievable (GitHub Actions REST log/run endpoints returned 403 from this environment's egress proxy); only the rendered Actions page's step name and exit code were available.
Did not run: the history pass (git log/git blame on modified regions) and prior-changes pass (review comments on earlier changes to these files) were not run this round — simulated_host.rs, the fake binaries, and the simulated-host step definitions are new files with no history, and the remaining changed files' history was not separately mined given the depth already reached by the code and impact-analysis passes; re-run these specifically if deeper confidence is wanted. No cargo build/cargo test/clippy was executed in this environment; all findings were settled by reading source against source, not execution — the PR's own description reports cargo test, clippy, and a 237-scenario local run as green except three explained, non-blocking flakes (serve-11, comfyui-04, automations-04), consistent with what the diff shows but not independently re-run here.
Ticket ROCMAI-677 retrieved by key (title, status, issue type only — no comments): "E2E: describe GPU hosts with simulated machines so GPU-visibility scenarios run on every PR," status In Progress. Matches the change's own description of itself.
Not reconciled against any PR discussion, review comments, or earlier review rounds — none were read, per method.
124b181 to
6a09714
Compare
|
@tomastola thanks. Rebased onto main and addressed in 3cd2e06, with the scenario numbering and tags in 6a09714:
From the later automated review: the ldconfig header count is computed from its entries, the drift check's CPU-node/DRM checks run without a simulated GPU, and fake-host-tool imports its constants. After the rebase the PR's scenarios are examine-31/32/33 (main now has examine-29/30). The MI300X failure was therock-next-09's vllm/amd-aiter pin mismatch, fixed on main by #755 and picked up by the rebase. The coverage trade-offs you raised (twins for serve-13/serve-15/diagnose-17, fake-vllm flags, serve-11's fixed port) I'll answer separately. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 6a09714
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (base 9875940e, head 6a09714). Implements ROCMAI-677 ("E2E: describe GPU hosts with simulated machines so GPU-visibility scenarios run on every PR"); the ticket title/status match the PR's own description, and the description's claims (HardwareMode gating, simulated_host.rs layout, fake-vllm/fake-host-tool stand-ins, nineteen scenarios moved to simulated machines, examine-29/32/33 real-hardware/drift checks) match what the diff contains.
Earlier rounds on this PR (at older commits) flagged a vacuous-assertion gap in assert_gpus_match_kfd_nodes and an MI300X CI failure. Both are resolved at this head: assert_gpus_match_kfd_nodes now carries the same "never decline to check on a simulated host" guard its sibling assert_gpu_target_matches_kfd already had, independently re-verified by two blind passes below. CI has moved on to a different, newly-identified failure (below).
Blocking
-
A
#[given(...)]step-text rename broke 3 Windows-only scenarios, and this is the confirmed root cause of the live CI failure on the Windows self-hosted lane —tests/e2e-cucumber/tests/e2e/examine_steps.rs:67renamed the step from"a machine with an AMD GPU"to"this machine has an AMD GPU"(confirmed against the base commit, which still has the old text). Two of the Gherkin call sites were updated to match (examine.feature:486,runtime_setup.feature:186), but three were not:diagnose.feature:647(@id:diagnose-windows-display-driver-advice-follows-the-inspection, scenario diagnose-41),diagnose.feature:755(@id:diagnose-windows-hip-sdk-advice-follows-the-inspection, scenario diagnose-48), andexamine.feature:453(@id:examine-windows-driver-version-date-and-age, scenario examine-29) still sayGiven a machine with an AMD GPU, which no longer binds to any step function. All three are tagged@requires-os:windows @requires-real-gpu, which is exactly why this was invisible in the author's local WSL2 test run (those scenarios skip there) and only surfaces on the Windows GPU lane.
Verified by direct evidence, not just reading: the "E2E tests (Strix Halo, Windows, current)" job on this exact commit (run 41652611022, job 125974649969) isFAILURE, and its log's own reconciliation output names precisely these three scenarios —'diagnose-windows-display-driver-advice-follows-the-inspection' was expected to pass on this host but FAILED — a regression.,'diagnose-windows-hip-sdk-advice-follows-the-inspection' was ... FAILED,'examine-windows-driver-version-date-and-age' was ... FAILED— a 1:1 match with the three stale call sites.
Confidence 100 · mechanical · Fix: update the threeGivenlines toGiven this machine has an AMD GPU. -
CI is not green on this head beyond the above, and the gaps need triage, not just the rename fix — the same Windows job also reports a 4th failure,
'runtime-resolve-canonical-release'(runtime_setup.feature:186, tag@requires-gfx-target), which already uses the correct renamed step text, so it is not explained by the finding above; its cause is still open. Separately, two other self-hosted jobs — "E2E tests (Strix Halo, Ubuntu, release, current)" and "E2E tests (Strix Halo, Ubuntu, nightly, current)" — are alsoFAILURE, but their logs show##[error]No space left on deviceduring action checkout, i.e. runner disk exhaustion unrelated to this PR's code. Two more self-hosted jobs ("E2E tests (MI300X, current)", "E2E tests (W7900, current)") are stillQUEUEDand have not reported at all as of this review. So: one confirmed code regression (above), one unexplained scenario failure, two infra flakes needing a runner-side fix or re-run, and two jobs pending — the PR cannot be called green yet on any of these grounds independently.
Confidence 100 (all read directly from the live check rollup and job logs) · architectural · Fix: fix the rename (above); investigateruntime-resolve-canonical-releaseseparately; re-run or escalate the two disk-space failures; wait for MI300X/W7900 to complete before merge.
Non-blocking
SimulatedGpu::gfx_target()formats the packed version's "minor" digit in decimal, not hex, diverging from the production formula it exists to mirror —tests/e2e-cucumber/src/simulated_host.rs:138-142(format!("gfx{}{}{:x}", v / 10_000, (v / 100) % 100, v % 100)). The production decoder it stands in for,gfx_target_from_gc_version(crates/rocm-core/src/host_gpu.rs:1568-1573), hex-formats both the minor and revision digits (format!("gfx{major}{minor:x}{revision:x}")) — its own test (gc_version_encodes_components_above_nine_as_hex_digits) exists specifically because decimal and hex only agree below 10. Verified directly by comparing both formulas side by side. Currently dormant: the module's only two GPU constants (mi300x: minor=4,strix_halo: minor=5) have single-digit minors, so decimal and hex coincide and no test catches it — but this is test-oracle logic consumed byexamine_steps.rsandserving_steps.rsto compute the expected value the CLI's real output is checked against, so a future GPU constant with minor ≥ 10 would silently assert the wrong expectation.
Confidence 85 · logic · Fix: change toformat!("gfx{}{:x}{:x}", v / 10_000, (v / 100) % 100, v % 100), matchinggfx_target_from_gc_version.TEST_HOST_ROOT_ENVis duplicated betweensimulated_host.rsandrocm_core::hardware_rootwith only a "must match" doc comment, not a compiler/test-enforced invariant —tests/e2e-cucumber/src/simulated_host.rs:40-42. A working precedent for closing exactly this gap already exists in the same crate:tests/e2e-cucumber/src/mock_server.rs's own#[cfg(test)]block already reaches therocm-coredev-dependency (rocm_core::ManagedServiceRecord), so a#[test]assertingTEST_HOST_ROOT_ENV == rocm_core::hardware_root::TEST_HOST_ROOT_ENVis reachable the same way. If the two literals are ever edited independently, the CLI binary would silently stop reading the simulated-host directory this whole PR plants, with nothing to notice.
Confidence 75 · architectural · Fix: add that one-line equality test tosimulated_host.rs's existingmod tests.bare_metal_layout_drift's reference-node selection relies on non-deterministicread_dirorder —tests/e2e-cucumber/src/simulated_host.rs:530-552(simulated_gpus.first(), no sort). The sibling production function this mirrors (detect_kfd_gfx_target_inincrates/rocm-core/src/host_gpu.rs) explicitly sorts nodes before picking "first," with a comment notingread_dirorder is filesystem-defined. This drift-check helper has no equivalent sort, so its choice of "the" simulated GPU node to compare every real node's properties against could vary run to run once a simulated host has ≥2 heterogeneous GPU nodes. Dormant today: every current caller uses either a single-GPU host or homogeneous multi-GPU hosts.
Confidence 70 · mechanical · Fix: sortgpu_nodes's entries (e.g. by node directory name) before taking.first().SimulatedHost's publicplatform/gpusfields let a caller construct a self-inconsistent machine —tests/e2e-cucumber/src/simulated_host.rs:166-171. Nothing stopsSimulatedHost { platform: Platform::Wsl2, gpus: vec![SimulatedGpu::mi300x(..)] }outside the three documented constructors;plant_wsl2(same file, 379-393) ignoresself.gpusentirely and unconditionally writes the Strix-Halo-specific Windows-inventory line, so such a host's reported GPU and its simulated Windows inventory answer would contradict each other. Not currently reachable — every call site goes throughinstinct_mi300x/bare_metal_without_gpu/wsl2_strix_halo.
Confidence 70 · architectural · Fix: make the fields private behind the existing constructors, or haveplant_wsl2derive its answer fromself.gpus.ScenarioDecl.requires_real_gpu ⇒ requires_gpuis enforced only insidefrom_tags, not by the type —tests/e2e-cucumber/src/expectation.rs:111-219(public independentboolfields), set together only inside the tag-parsing loop (:259-263);resolve()'s two gates key offrequires_gpualone and never re-derive it fromrequires_real_gpu. The file already has precedent for partial struct-literal construction of aScenarioDeclelsewhere in its own tests, so a future hand-built literal could silently desync the two flags, which would make a@requires-real-gpuscenario resolve toExpectPasson a GPU-less real-hardware host instead of skipping. Not currently reachable — the only production constructor isfrom_tags.
Confidence 70 · architectural · Fix: computerequires_gpufromrequires_real_gpu(private field + accessor), ordebug_assert!the implication at the top ofresolve().
Decisions for the author
- Coverage moves off real/WSL hardware for most of the 19 migrated scenarios, narrowing two of them from a broader platform tag to Linux-only — tradeoff. Verified directly against the diff:
serve-13(serve-no-gpu-fails-fast) moved from@requires-no-gputo@requires-os:linux;examine-10(examine-can-skip-framework-probing) moved from@requires-bare-metal(which only skipped WSL) to@requires-os:linux. Of the nineteen moved scenarios, onlyexamine-04keeps a real-hardware twin (examine-29); the nightly checks (examine-32/examine-33) verify fixture-layout fidelity, not scenario behavior, on real hardware. This is a reasonable tradeoff — hermetic, always-on coverage gained vs. real/WSL exercising lost for most moved scenarios — and the PR description's framing ("the moved scenarios gain coverage") doesn't mention the Linux-only narrowing for these two. Worth the author confirming this is accepted as-is rather than mitigated with a@requires-real-gpu/non-simulated twin for either. - The simulated vLLM stand-in ignores every flag except
--host/--port— tradeoff (tests/e2e-cucumber/src/bin/fake-vllm.rs).serve-11genuinely proves engine selection, runtime lookup, managed launch, and readiness-wait (the fake binds the CLI-passed host/port rather than a hardcoded default, confirmed againstengines/vllm/src/process.rs'svllm_serve_args), but does not prove the CLI passes correct additional vLLM flags — a wrong--max-model-lenor similar would go unnoticed here. Acceptable if flag correctness is covered elsewhere; worth confirming that scope boundary is intentional.
Positive signals
assert_gpu_target_matches_kfdandassert_gpus_match_kfd_nodes(examine_steps.rs) both now correctly guard every early return against vacuously passing on a simulated host — independently re-traced by two blind passes in this round, each confirming no early-return path discharges examine-04/examine-17 without having actually checked the planted topology.fake-host-tool's dispatch answers only the exact command line it was told to expect, with one narrow, explicitly-justified exception (powershell.exe, which must accept a whole inventory script) — directly defeating the "stand-in silently papers over a changed CLI invocation" failure mode, backed by a dedicated unit test (a_stand_in_answers_only_the_command_line_it_was_planted_for).bare_metal_layout_drift/wsl_layout_drift/lspci_drift, exercised nightly against real hardware (examine-32/33), is a well-targeted regression guard: its own test suite pins the exact historical bug (gfx_target_versionread from a standalone file no kernel has) that motivated building this simulated-host infrastructure in the first place.HardwareMode::parsefails closed on an unrecognizedE2E_HARDWAREvalue (panics rather than defaulting tosimulated), consistent with this project's no-silent-GPU-fallback convention, and the one production call site (tests/e2e.rs) propagates that panic rather than swallowing it.tui_driver.rs'sisolate_env_removals()addition closes a real, pre-existing isolation gap (a PTY-driven serve on a simulated host could previously inherit the runner's ownHIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES), bringing the PTY path in line with the piped path's existingisolate_cmdguard.- Docs/CI consistency is well maintained:
docs/testing.md,docs/ci-hardware-testing.md, andtests/e2e-cucumber/README.mdall describe the current@requires-real-gpu/E2E_HARDWAREsemantics without drifting from each other or from the actual tag names in the source, and every self-hosted GPU job in both workflow files correctly sets/forwardsE2E_HARDWARE: real.
Deployment notes
No product code changes — this PR only changes test infrastructure and which CI lanes exercise which existing e2e scenarios. Do not merge until the Blocking items are resolved: the confirmed step-text regression, the unexplained runtime-resolve-canonical-release failure, the two disk-space infra failures, and the two still-pending self-hosted jobs (MI300X, W7900).
What this covered
Every file in the diff (33 files, +2150/-282) read completely at PR head 6a0971404e3ddd4738074745f26c841d3f1613ed against merge-base 9875940ebc7d1dbf9db0706e01d29842e8ec1d27, by six independent blind fan-out workers grouped by coupling: (1) simulated_host.rs; (2) expectation.rs + mock_server.rs + lib.rs; (3) the fake binaries + simulated_host_steps.rs + tests/e2e.rs; (4) diagnose/driver/examine step files + their feature files; (5) serve/runtime/comfyui/tui step files + their feature files; (6) docs + CI workflow files + Cargo.toml + README. Each worker also read relevant surrounding/consumer production code (crates/rocm-core/src/{host_gpu,examine,diagnose,fix}.rs, apps/rocm/src/service_launch.rs, engines/vllm/src/{runtime,process}.rs) to verify cross-file claims rather than trusting the test code's own framing. I then independently re-verified the highest-value candidates myself (the step-text rename, the gfx_target hex/decimal divergence, the serve-13/examine-10 tag narrowing) directly against the source and the live CI logs before scoring them.
CI was read live via gh pr view 517 --json statusCheckRollup and GitHub Actions job logs, twice (once mid-review, once immediately before publishing) — the head commit had not moved between the two checks.
Caveat on blindness: before delegating the file-grouped passes above, this orchestrating context necessarily read the full bodies of three prior reviews on this PR (pinned to an older commit, 124b181) in order to perform the mandated check for whether this PR had already been reviewed at its current head — it had not, since the head has since advanced to 6a09714. That prior-review content was used only to decide whether to proceed, and afterward to sanity-check which previously-raised issues are now fixed (confirmed: the assert_gpus_match_kfd_nodes vacuous-pass gap and the ldconfig header-count mismatch are both resolved at this head); it was not given to any of the six fan-out workers, who read the code fresh with no knowledge of prior rounds, and none of this review's findings reproduce a prior round's finding. Did not run: the history/prior-changes passes were not run as a separate dedicated pass beyond git log spot-checks already folded into the workers' own investigation, since most touched files are new. No cargo build/cargo test/clippy was executed in this environment (no Rust toolchain available) — all non-CI findings were settled by reading source against source, not execution. Not reconciled against any PR discussion or review comments — none were read as part of producing these findings, per method.
🤖 Generated with Claude Code
6a09714 to
5836191
Compare
|
@tomastola the trade-offs from your review, answered. The blocking items were answered in my earlier comment. Rebased onto main at 3e00084. main has since added scenarios, so this PR's scenarios are renumbered; the numbers below are current. Edited after posting: an earlier version of this comment said the no-GPU refusal is held on native Windows, which is wrong; see the first point.
The description now says plainly what moved off real hardware and what has no twin. |
|
@nowycondro answers to the automated reviews of 124b181 and 6a09714. Rebased onto main at 3e00084. This PR's scenarios are renumbered after main's newer ones; the numbers below are current. Blocking items from the 6a09714 review:
Non-blocking items, all in adbb7d2:
Decisions left to the author:
Also changed since 6a09714, from a full review of my own:
|
GPU scenarios are about to stop depending on the runner's hardware: they will plant a simulated host instead, so they run on GitHub-hosted runners on every PR. A few premises cannot be simulated, such as a model actually generating tokens on the device, and those need a way to say so. @requires-real-gpu implies @requires-gpu and also skips unless the run sets E2E_HARDWARE=real. The default is `simulated`. Any other value aborts the run, because a misspelt `real` would otherwise skip every real-GPU scenario and still report green. Every self-hosted GPU lane, per-PR and nightly, sets E2E_HARDWARE=real now, and the WSL lanes forward it through WSLENV. Nothing carries the tag yet, so no scenario changes lanes in this commit. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
GPU scenarios could only run where the runner had the right hardware. The CLI reads the real machine, so detection and refusal paths that never touch the device were confined to the GPU lanes, multi-GPU premises to the hosts with several cards, and WSL premises to the one WSL lane. Scenarios now name the machine they need in a Given step, for example "a machine with eight AMD Instinct GPUs", "a bare-metal Linux machine with an AMD GPU" or "a WSL machine with an AMD GPU passed through". The step plants a directory laid out the way the kernel lays out /dev, /sys and /proc, points ROCM_CLI_TEST_HOST_ROOT at it, and isolates the commands from the runner: - Commands get a PATH with the hardware tools removed (uname, lspci, ldconfig, rocminfo, amd-smi, the kernel log, the WSL interop shells and others). The tools the simulated machine has come back as a fake-host-tool stand-in that answers from canned output. - The runner's GPU visibility masks and ROCm variables are unset. - HOME is moved into the scenario. "A managed runtime is active" plants a runtime record whose vllm is a fake-vllm stand-in. A serve on a simulated machine therefore runs the real engine selection, runtime lookup, launch and readiness wait, and only the process at the end is simulated. Eighteen scenarios move onto simulated machines and run on every Linux lane, including the GitHub-hosted one: examine-04/10/11/17, diagnose-08/15/16/17/18, driver-install-01..03 and serve-11/13/15/19/20/21. examine-49 repeats examine-04 against the real machine, so a real GPU lane still covers detection there, including the Windows lane. serve-16 stays on real hardware: it also runs on native Windows, which a simulated machine cannot describe. Every scenario left on @requires-gpu needs a physical device or a runtime really installed for one, so those become @requires-real-gpu. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Scenarios on a simulated machine fail only when the CLI misreads the fixture. If a kernel or driver update moves something the fixture assumes, those scenarios keep passing while the CLI fails on hardware. That is how the KFD target came to be read from a file no kernel has. Two @nightly @requires-real-gpu scenarios now hold the fixture against a real machine. On a bare-metal GPU host they check: - each KFD GPU node states gfx_target_version inside its properties file - the property keys the fixture writes are present - gpu_id files exist - render nodes match drm_render_minor - CPU nodes report a target of 0 - the AMD DRM cards have vendor and device files - `lspci -nn -D` lines have the expected shape On a WSL2 distribution they check /dev/dxg, libdxcore.so and the microsoft-standard-WSL2 kernel string. Each difference is reported along with what to update. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
examine-17's Then step returned silently when the KFD topology read came back empty, so a planting or probe-root regression left it green. On a simulated machine it now requires as many GPU nodes as were planted. fake-host-tool answers only the exact command line planted (powershell.exe excepted) and exits 127 naming anything else, so a changed CLI invocation can't get stale canned text. The ldconfig -p header counts its entries, the layout drift check runs its CPU-node and DRM checks without a simulated GPU, the unused WSL "rocdxg not ready" variant is gone, a second simulated-host plant fails fast, and the README documents E2E_HARDWARE. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Seven scenarios main added since this branch was cut still carried @requires-gpu although each needs a physical device: the Windows driver and HIP SDK checks, the AIM serving pair, the live-reuse --gpu validation, and the ComfyUI runtime-preservation install. They now carry @requires-real-gpu like every other such scenario, so a simulated run on a GPU box skips them. docs/testing.md names the new tag and the E2E_HARDWARE=real opt-in for the ComfyUI run. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Fifteen scenarios main added since the last rebase still carried @requires-gpu although each serves a model on the device through a really installed runtime: the no-background-helper check, the explicit port refusals for a managed service and an untracked process, the automatic port fallback, the explicit port that differs from a live equivalent, the five Anthropic-endpoint scenarios, the small assistant served by its short name, and the refusal of a repo with no quantization variant (whose GPU only gets it past the earlier gate, on the native Windows lane it is meant for), the two registered-GGUF serves and the dash Lemonade journey. A default (simulated) run on a GPU box would still have served on the physical device with them; they now carry @requires-real-gpu like every other such scenario. Every self-hosted lane, the merge-queue ones included, sets E2E_HARDWARE=real, so they keep running where they ran before. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
From review of the simulated-machine harness: - A simulated GPU's expected gfx target rendered the minor digit in decimal while the product renders both minor and stepping as hex, so a future GPU with a component above nine would have been checked against the wrong target. It now matches the product (90010 is gfx90a). - The root variable the harness sets is now tied to the one the CLI reads by a test, instead of by a comment. - The nightly drift check sorts KFD nodes before choosing the simulated node every real one is compared with, as the product does, instead of taking whichever the filesystem lists first. - A simulated machine is built only through its constructors, so its platform and GPUs cannot disagree. - The GPU gate treats @requires-real-gpu as needing a GPU on its own, rather than relying on the tag parser to have set both flags. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A simulated run was described as checking the same thing on a GPU box as on a GitHub-hosted runner, but two setup steps still decided by whether the runner had a GPU: "a model is served in the background" served a real model and "a managed runtime is active" installed a real SDK whenever one was present, so chat-06, chat-07 and serve-49 did real GPU work on a GPU box in the default mode. Both now take their GPU-less path unless the run opted into real hardware and the host has a GPU. Their xfail rows follow the same mode: a matrix condition can now name `hardware = "real"`, and the chat-06/07 rows do. Otherwise chat-06, passing against the mock server in a simulated run on a vLLM box, would read as a stale XPASS of its vLLM row and fail the run. E2E_HARDWARE is read in one place, which also refuses a value that is not valid Unicode instead of treating it as unset. The docs now name the probes a simulated machine does not replace, including the DRM VRAM fallback serve uses when amd-smi is missing. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A self-hosted lane that dropped E2E_HARDWARE, or a WSL lane that stopped forwarding it through WSLENV, would skip every @requires-real-gpu scenario and still go green: the hardware-mode parser rejects a misspelling but cannot see an absence. The workflow contract now derives the self-hosted lanes from both GPU workflows and requires each to set it in its job env, each WSL lane to forward it into the guest, and no GitHub-hosted job to set it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The drift checks run only nightly on real hardware, where a branch that stopped firing reads as "no drift". wsl_layout_drift had no test at all, and the bare-metal checks for /dev/kfd and the amdgpu module, a missing gpu_id, a render minor with no render node, a DRM device id in another form and an lspci listing with no AMD GPU were untested. Each now has a test that fails when its branch is disabled. The reference node is chosen in numeric order, as the product orders KFD nodes, rather than by path, which put node 10 before node 2. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
"The inspection reports that the driver is available" passed whenever the output contained "driver_status", which rocm examine prints whatever it found, not_detected included, so examine-04 and its real-hardware twin could not fail on the driver half of their name. The step now reads the status: on a simulated machine it must be the one the planted platform implies (amdgpu_available on bare metal, wsl_rocdxg_ready on WSL2), and on the real machine one of the statuses that mean a usable driver on Linux, WSL2 or Windows. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
serve-19 moved onto a simulated two-GPU machine, which left its single-GPU case, a mask naming device 0 refusing ordinal 1 on a host with one device, running nowhere while its comment still claimed it. serve-59 is that case on a simulated one-GPU machine, and a unit test holds it to one GPU. The shared Then step no longer accepts "out of range": on one GPU ordinal 1 is out of range with or without the mask, so accepting it let a broken mask check pass there. With serve-19 and serve-20 planting their own GPUs, no scenario carries @requires-multi-gpu any more. The tag, its gate, the capability probe's GPU count that served only it, and their tests are removed rather than kept for a scenario nobody has named. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Moving scenarios onto simulated machines left some premises checked only against a simulation: - serve-61 repeats serve-15's masked-device refusal on a real Linux GPU host, and diagnose-65 repeats diagnose-17's WSL platform guard on a real WSL2 distribution. - serve-60 is serve-13's GPU-required refusal on the runner's own GPU-less Linux host. It is Linux only: off Linux there is no usable-GPU probe, so a GPU-less Windows host is not refused today. serve-11 proved engine selection, launch and readiness, but the vLLM stand-in bound whatever it was given, so wrong launch flags went unnoticed. fake-vllm now records its arguments beside itself, and the scenario checks the model, host, port and --enforce-eager. It serves on a free port instead of the shared default, and no longer runs the GPU lanes' port cleanup, which SIGKILLs any listener on the serve and assistant ports and would kill a developer's own server. The stand-in is installed with the shared fixture installer rather than a symlink. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Comments and docs still described the suite before simulated machines: comments naming plain @requires-gpu where a scenario is now @requires-real-gpu (ComfyUI's runtime install, the low-VRAM vLLM scenario, the Lemonade backend-alignment notes), unit tests citing tag sets their scenarios no longer carry, a fake-host-tool header listing tools it is not linked as, a CI comment crediting simulated machines with covering the modprobe directories that none plants, and a runtime-lifecycle note saying no mock engine could answer readiness. The drift checks were said to hold "every fixture assumption"; the docs now say what they compare and what they do not. The step list in the testing doc names the two-GPU machine, and expectations.toml names the real-hardware gate. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The shared-GPU-queue lanes run scripts/run-e2e-lane.sh on real GPUs outside any workflow, and it never set E2E_HARDWARE, so every @requires-real-gpu scenario would have skipped there and the lane stayed green. The script now exports E2E_HARDWARE=real unless the caller set it, and the contract test fails if that export goes. The same test only looked for E2E_HARDWARE in the env of jobs that are not self-hosted, so a workflow-level env: or a run: script setting it passed. It now checks everything in a workflow outside the self-hosted GPU jobs, with a unit test for each of the three places. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
serve-11 always plants a simulated Instinct machine, so the steps only it uses no longer need a branch for the runner's own GPU, and its @requires-engine:vllm tag gated nothing: the engine gate applies only to GPU scenarios, and the engine now follows the simulated GPU. The When step now refuses to run without a simulated machine. Three doc comments had ended up on the wrong item (real_gpu_host, SIMULATED_RUNTIME_KEY, the single-GPU serve-53 test) and are moved back to the items they describe. Comments and docs still described serve-50/51/52 as @requires-gpu, chat-06/07 as serving for real on any GPU host, diagnose-28 as carrying the Linux and bare-metal pair, and a shorter list of tools taken off PATH than HOST_TOOLS holds. The probe comments now say that the e2e suite answers ldconfig with a stand-in on PATH, and that the absolute fallbacks still reach the real linker. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The WSL ROCDXG check asks `ldconfig -p` and, when PATH has none, falls back to /sbin/ldconfig and /usr/sbin/ldconfig. On a simulated host the e2e suite answers the bare `ldconfig` with a stand-in on PATH, but a simulated machine without one would fall through to the runner's own linker and read its cache. With a test host root set, only the `ldconfig` on PATH is asked now, so its absence reads as "could not ask". A release build has no test host root, so its behaviour is unchanged. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
examine-10 planted a simulated bare-metal machine because the framework probe used to stop on WSL2 before it reached the framework step. It no longer does: bare-metal Linux, WSL2 and native Windows all reach it, so the --no-frameworks choice is acted on everywhere. Planting a Linux machine only narrowed the scenario to Linux lanes and dropped the Windows lanes it used to run on. It now plants nothing and carries no platform tag. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
5836191 to
04b3d4e
Compare
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
Why
Many e2e scenarios only need the CLI to see a particular GPU machine: eight Instinct GPUs, a visibility mask over two devices, a WSL2 distribution with GPU passthrough, a bare-metal host with the in-tree driver. Because the CLI reads the runner's real hardware, those scenarios could only run on a self-hosted lane that happened to have that hardware, and some (multi-GPU, WSL) ran on one lane or none. Detection and refusal paths that never touch the device were therefore checked rarely, and never on the GitHub-hosted
E2E testslane that runs on every PR.What changes
Simulated machines. A Given step now names the machine a scenario needs ("a machine with eight AMD Instinct GPUs", "a WSL machine with an AMD GPU passed through", "a Linux machine with no AMD GPU", ...). It plants a directory laid out the way the kernel lays out
/dev,/sysand/proc(KFD topology withgfx_target_versioninside each node'sproperties, DRM cards,/dev/kfdor/dev/dxg,/proc/version,/etc/os-release) and points thee2e-test-hooksbuild'sROCM_CLI_TEST_HOST_ROOTat it. Commands then get aPATHwith the machine-describing tools removed (uname,lspci,ldconfig,rocminfo,amd-smi,mpirun, the kernel log, the WSL interop shells, and every/mntentry); the tools the simulated machine has come back asfake-host-tool, which answers only the exact command line planted for it and exits 127 naming anything else, so a changed CLI invocation cannot get stale canned text. The runner's GPU visibility and ROCm variables are unset andHOMEmoves into the scenario.Given a managed runtime is activeon a simulated machine plants a runtime whosevllmisfake-vllm: a serve runs the CLI's real engine selection, runtime lookup, managed launch and readiness wait, andfake-vllmrecords its argv so serve-11 checks the model, host, port and--enforce-eagerit was launched with.Moved onto simulated machines, and so onto every Linux lane including the GitHub-hosted one: examine-04/11/17, diagnose-08/15/16/17/18, driver-install-01..03, serve-11/13/15/19/20/21, plus serve-59 (serve-19's single-GPU case). examine-10 (
--no-frameworks) plants nothing and drops its platform tag, since every platform reaches the framework step; it now runs on every lane, Windows included.E2E_HARDWARE=simulated|realand@requires-real-gpu. A scenario whose premise cannot be simulated (a model generating tokens on a device, a runtime really installed for it) is tagged@requires-real-gpu: it needs a GPU and also skips unless the run setsE2E_HARDWARE=real. The default issimulated; any other value, including one that is not valid Unicode, aborts the run, so a misspeltrealcannot skip every real-GPU scenario and still go green. In a simulated run, the two setup steps that used to reach the runner's GPU (a real background serve, a real SDK install) take their GPU-less path, so a GPU box runs what a GitHub-hosted runner runs; xfail rows for scenarios that serve for real only in arealrun sayhardware = "real"in their condition. No feature file carries plain@requires-gpuany more, and@requires-multi-gpuwith its capability-probe GPU count is removed: the multi-GPU scenarios plant their own GPUs.Every self-hosted GPU lane in
e2e-selfhosted.ymlandnightly.ymlsetsE2E_HARDWARE: real(the WSL lanes forward it throughWSLENV), andscripts/run-e2e-lane.sh, which the shared GPU queue runs outside any workflow, exports it unless the caller chose a mode. A workflow-contract test fails when a self-hosted lane drops it, a WSL lane stops forwarding it, the lane script stops exporting it or sets it again afterwards, or anything outside a self-hosted GPU job sets it.Real-hardware twins and a nightly drift check. Simulated scenarios fail only when the CLI misreads the fixture, so the fixture is held against real machines:
@nightly)gfx_target_versioninsideproperties, the property keys the fixture writes,gpu_id, render minors, CPU nodes, DRM vendor and device files,lspci -nn -Dshape@nightly)/dev/dxg,libdxcore.so, themicrosoft-standard-WSL2kernel stringThe other moved scenarios have no twin: multi-GPU mask semantics (serve-19/20/21), the default-engine vLLM launch on Instinct (serve-11), the WSL driver-install plan and remedy (driver-install-01..03, diagnose-18), WSL detection on a real guest (examine-11) and multi-GPU listing (examine-17) are checked only against simulated machines, and the drift between those machines and real ones only nightly.
One product change. With a simulated root set in a hook build, the WSL ROCDXG check asks only the
ldconfigonPATHand skips its/sbinand/usr/sbinfallbacks, so a simulated machine without one reads as "could not ask" instead of answering from the runner's linker cache. A build without the hook has no root and is unchanged. The rest of the product diff is comments naming the new tag.Not simulated: package-manager state, discovered ROCm installs, the identity of the user running the suite (
$USER,id -Gn,staton device nodes), andlsmod(so a simulated machine reports module state from/proc/modulesand sysfs only). Container detection reads under the host root, so a simulated machine is never in a container.docs/testing.mdlists the probes a simulated machine does not replace.Running locally: a plain
cargo xtask e2eon a GPU box runs insimulatedmode and skips every@requires-real-gpuscenario;E2E_HARDWARE=real cargo xtask e2eruns them. Simulated scenarios need arocmbuilt withrocm/e2e-test-hooks, whichcargo xtask e2ebuilds.Scenario coverage (AGENTS.md §3)
No
rocmcommand output changes. This PR changes which machines existing scenarios run against and adds examine-49/50/51, serve-59/60/61 and diagnose-65. The moved scenarios run on every Linux lane; the real-hardware twins run on the self-hosted GPU lanes (@requires-real-gpu), serve-60 on GPU-less Linux lanes such as the GitHub-hosted one, and the nightly drift checks on the nightly GPU and WSL lanes.Not covered: the GPU-required refusal on a GPU-less native Windows host. Off Linux the CLI has no usable-GPU probe and reads "cannot tell" as "not known to be absent", so
rocm servelaunches there instead of refusing. serve-60 is therefore@requires-os:linux, and that Windows behaviour is a product gap this PR does not change.Verification
cargo xtask e2e(simulated mode) on a Linux host with a GPU: 404 scenarios, all passing except serve-42 (@id:serve-wsl-refusal-names-the-wsl-driver-status), which fails the same way on unmodifiedmainon that host: its hand-planted WSL machine leaves the runner's GPU tools onPATH, so serve finds the real GPU. It passes on GPU-less lanes.cargo test --workspace --all-targets,cargo test -p rocm-core --lib --features e2e-test-hooks,cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --check,cargo xtask check-module-size,check-architecture-docandcheck-crate-edges. On the test host, threeapps/rocmtests this PR does not touch fail for host reasons: twobuild.rstests whose temporary repository cannot commit under the host's signing config, and one that needsdockerto honourDOCKER_HOST(it is podman there).New tests were each broken against their target and seen to fail: the
hardwarexfail key, thePATHfilter (/mnt, non-executables),fake-host-tool's refusal of an unplanted command line, the WSL Windows-inventory answer, the linker-cache probe's simulated-root switch, the lane-script override check, and serve-59 with the CLI's visible-set check disabled.Not verified locally: examine-49/50/51, diagnose-65 and serve-61 need hardware this host does not have; the self-hosted lanes run them.
If this PR fixes a bug, searched
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (n/a: no bug fix; the chat-06/07 rows are narrowed tohardware = "real".)If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per
docs/architecture.md. (n/a)If this PR adds, renames or removes a production source file (or a
.rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry, and the prose of any entry whose file changed role still describes it. (simulated_host.rs,simulated_host_steps.rs,fake-host-tool.rs,fake-vllm.rsadded; thecapability.rsentries no longer mention a GPU count.)Every new or changed user-facing message was read against the code path that runs after it. (n/a: no
rocmmessage changes.)