Skip to content

test(e2e): describe GPU hosts with simulated machines (ROCMAI-677) - #517

Draft
rominf wants to merge 18 commits into
mainfrom
test/e2e-simulated-machines
Draft

rominf wants to merge 18 commits into
mainfrom
test/e2e-simulated-machines

Conversation

@rominf

@rominf rominf commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Many e2e scenarios only need the CLI to see a particular GPU machine: eight Instinct GPUs, a visibility mask over two devices, a WSL2 distribution with GPU passthrough, a bare-metal host with the in-tree driver. Because the CLI reads the runner's real hardware, those scenarios could only run on a self-hosted lane that happened to have that hardware, and some (multi-GPU, WSL) ran on one lane or none. Detection and refusal paths that never touch the device were therefore checked rarely, and never on the GitHub-hosted E2E tests lane that runs on every PR.

What changes

Simulated machines. A Given step now names the machine a scenario needs ("a machine with eight AMD Instinct GPUs", "a WSL machine with an AMD GPU passed through", "a Linux machine with no AMD GPU", ...). It plants a directory laid out the way the kernel lays out /dev, /sys and /proc (KFD topology with gfx_target_version inside each node's properties, DRM cards, /dev/kfd or /dev/dxg, /proc/version, /etc/os-release) and points the e2e-test-hooks build's ROCM_CLI_TEST_HOST_ROOT at it. Commands then get a PATH with the machine-describing tools removed (uname, lspci, ldconfig, rocminfo, amd-smi, mpirun, the kernel log, the WSL interop shells, and every /mnt entry); the tools the simulated machine has come back as fake-host-tool, which answers only the exact command line planted for it and exits 127 naming anything else, so a changed CLI invocation cannot get stale canned text. The runner's GPU visibility and ROCm variables are unset and HOME moves into the scenario. Given a managed runtime is active on a simulated machine plants a runtime whose vllm is fake-vllm: a serve runs the CLI's real engine selection, runtime lookup, managed launch and readiness wait, and fake-vllm records its argv so serve-11 checks the model, host, port and --enforce-eager it was launched with.

Moved onto simulated machines, and so onto every Linux lane including the GitHub-hosted one: examine-04/11/17, diagnose-08/15/16/17/18, driver-install-01..03, serve-11/13/15/19/20/21, plus serve-59 (serve-19's single-GPU case). examine-10 (--no-frameworks) plants nothing and drops its platform tag, since every platform reaches the framework step; it now runs on every lane, Windows included.

E2E_HARDWARE=simulated|real and @requires-real-gpu. A scenario whose premise cannot be simulated (a model generating tokens on a device, a runtime really installed for it) is tagged @requires-real-gpu: it needs a GPU and also skips unless the run sets E2E_HARDWARE=real. The default is simulated; any other value, including one that is not valid Unicode, aborts the run, so a misspelt real cannot skip every real-GPU scenario and still go green. In a simulated run, the two setup steps that used to reach the runner's GPU (a real background serve, a real SDK install) take their GPU-less path, so a GPU box runs what a GitHub-hosted runner runs; xfail rows for scenarios that serve for real only in a real run say hardware = "real" in their condition. No feature file carries plain @requires-gpu any more, and @requires-multi-gpu with its capability-probe GPU count is removed: the multi-GPU scenarios plant their own GPUs.

Every self-hosted GPU lane in e2e-selfhosted.yml and nightly.yml sets E2E_HARDWARE: real (the WSL lanes forward it through WSLENV), and scripts/run-e2e-lane.sh, which the shared GPU queue runs outside any workflow, exports it unless the caller chose a mode. A workflow-contract test fails when a self-hosted lane drops it, a WSL lane stops forwarding it, the lane script stops exporting it or sets it again afterwards, or anything outside a self-hosted GPU job sets it.

Real-hardware twins and a nightly drift check. Simulated scenarios fail only when the CLI misreads the fixture, so the fixture is held against real machines:

Scenario Checks on real hardware
examine-49 examine-04's detection against this machine's own GPU and driver, on every GPU lane including Windows
serve-60 serve-13's GPU-required refusal on the runner's own GPU-less Linux host (Linux only: see Not covered below)
serve-61 serve-15's every-GPU-masked refusal on a real Linux GPU host
diagnose-65 diagnose-17's WSL platform guard on a real WSL2 distribution
examine-50 (@nightly) the bare-metal layout: gfx_target_version inside properties, the property keys the fixture writes, gpu_id, render minors, CPU nodes, DRM vendor and device files, lspci -nn -D shape
examine-51 (@nightly) the WSL2 layout: /dev/dxg, libdxcore.so, the microsoft-standard-WSL2 kernel string

The other moved scenarios have no twin: multi-GPU mask semantics (serve-19/20/21), the default-engine vLLM launch on Instinct (serve-11), the WSL driver-install plan and remedy (driver-install-01..03, diagnose-18), WSL detection on a real guest (examine-11) and multi-GPU listing (examine-17) are checked only against simulated machines, and the drift between those machines and real ones only nightly.

One product change. With a simulated root set in a hook build, the WSL ROCDXG check asks only the ldconfig on PATH and skips its /sbin and /usr/sbin fallbacks, so a simulated machine without one reads as "could not ask" instead of answering from the runner's linker cache. A build without the hook has no root and is unchanged. The rest of the product diff is comments naming the new tag.

Not simulated: package-manager state, discovered ROCm installs, the identity of the user running the suite ($USER, id -Gn, stat on device nodes), and lsmod (so a simulated machine reports module state from /proc/modules and sysfs only). Container detection reads under the host root, so a simulated machine is never in a container. docs/testing.md lists the probes a simulated machine does not replace.

Running locally: a plain cargo xtask e2e on a GPU box runs in simulated mode and skips every @requires-real-gpu scenario; E2E_HARDWARE=real cargo xtask e2e runs them. Simulated scenarios need a rocm built with rocm/e2e-test-hooks, which cargo xtask e2e builds.

Scenario coverage (AGENTS.md §3)

No rocm command output changes. This PR changes which machines existing scenarios run against and adds examine-49/50/51, serve-59/60/61 and diagnose-65. The moved scenarios run on every Linux lane; the real-hardware twins run on the self-hosted GPU lanes (@requires-real-gpu), serve-60 on GPU-less Linux lanes such as the GitHub-hosted one, and the nightly drift checks on the nightly GPU and WSL lanes.

Not covered: the GPU-required refusal on a GPU-less native Windows host. Off Linux the CLI has no usable-GPU probe and reads "cannot tell" as "not known to be absent", so rocm serve launches there instead of refusing. serve-60 is therefore @requires-os:linux, and that Windows behaviour is a product gap this PR does not change.

Verification

  • cargo xtask e2e (simulated mode) on a Linux host with a GPU: 404 scenarios, all passing except serve-42 (@id:serve-wsl-refusal-names-the-wsl-driver-status), which fails the same way on unmodified main on that host: its hand-planted WSL machine leaves the runner's GPU tools on PATH, so serve finds the real GPU. It passes on GPU-less lanes.

  • cargo test --workspace --all-targets, cargo test -p rocm-core --lib --features e2e-test-hooks, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --check, cargo xtask check-module-size, check-architecture-doc and check-crate-edges. On the test host, three apps/rocm tests this PR does not touch fail for host reasons: two build.rs tests whose temporary repository cannot commit under the host's signing config, and one that needs docker to honour DOCKER_HOST (it is podman there).

  • New tests were each broken against their target and seen to fail: the hardware xfail key, the PATH filter (/mnt, non-executables), fake-host-tool's refusal of an unplanted command line, the WSL Windows-inventory answer, the linker-cache probe's simulated-root switch, the lane-script override check, and serve-59 with the CLI's visible-set check disabled.

  • Not verified locally: examine-49/50/51, diagnose-65 and serve-61 need hardware this host does not have; the self-hosted lanes run them.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (n/a: no bug fix; the chat-06/07 rows are narrowed to hardware = "real".)

  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (n/a)

  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry, and the prose of any entry whose file changed role still describes it. (simulated_host.rs, simulated_host_steps.rs, fake-host-tool.rs, fake-vllm.rs added; the capability.rs entries no longer mention a GPU count.)

  • Every new or changed user-facing message was read against the code path that runs after it. (n/a: no rocm message changes.)

@rominf
rominf force-pushed the test/e2e-host-root branch 5 times, most recently from 3388ba7 to 2df8565 Compare October 9, 2026 07:57
Base automatically changed from test/e2e-host-root to main October 12, 2026 09:55
@rominf
rominf force-pushed the test/e2e-simulated-machines branch from 3c6e9a1 to 124b181 Compare October 26, 2026 11:24
@rominf

rominf commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main at 1c33248. Head is now 124b181. What changed:

  • Dropped the first commit ("let e2e builds re-root the hardware probes"). test(core): let e2e builds re-root the hardware probes #510 landed it on main in a later form.
  • E2E_HARDWARE: real is set on main's current lane set: per-PR MI300X, Strix Ubuntu, Strix Windows, WSL and W7900, plus nightly W7900. The per-PR R9700 and MI350P lanes no longer exist.
  • Scenarios main added with @requires-gpu since this PR was written are now @requires-real-gpu: examine-19, serve-23, serve-24 and runtime-lifecycle-12. runtime-16 keeps main's @requires-os:windows, and therock-next-09 keeps main's removal of @nightly.
  • diagnose-13 is no longer moved. vllm: point import failures at rocm diagnose #646 made its checker apply on WSL too, so it already runs on every Linux lane without a simulated machine. That makes 19 moved scenarios, not 20.
  • This PR's examine scenarios are renumbered examine-29/30/31, because main took 18-28. Cross-references now use @id: instead of numbers.
  • Removed this PR's own copy of the fixture-binary lookup in favour of main's fixture_binary::fixture_binary.
  • Corrected docs that main's changes had made wrong:
    • Container detection now reads under the host root, so a simulated machine is never in a container.
    • The user's identity ($USER, id -Gn, stat on device nodes) is not simulated.
    • docs/ci-hardware-testing.md and the WSL lane comments no longer say the WSL diagnose and driver scenarios run only on the WSL lane.
  • When the simulated vLLM never becomes ready, the serve-11 failure message now includes the service log.

Local results are in the updated description. The full mock suite ran 237 scenarios. The 3 unexpected failures were all local: a port already held on the shared test machine, and two load flakes that pass on their own. The PR stays in draft until required CI is green.

@rominf
rominf marked this pull request as ready for review October 26, 2026 11:50
@rominf
rominf requested a review from a team as a code owner October 26, 2026 11:50
@rominf
rominf requested a review from tomastola October 26, 2026 11:50
@rominf rominf changed the title test(e2e): describe GPU hosts with simulated machines test(e2e): describe GPU hosts with simulated machines (ROCMAI-677) Oct 26, 2026

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (base 1c33248c, head 124b1815).

Blocking

  • examine-17's Then step can pass without asserting anything — tests/e2e-cucumber/tests/e2e/examine_steps.rs:1014-1022 (assert_gpus_match_kfd_nodes)
    The step returns silently when kfd_gpu_node_versions(...) is None or empty, and again when host_has_lspci is false. The scenario now plants eight simulated GPUs, so those early returns are exactly what hides a planted tree the CLI no longer reads. If the planting, world.host_path or the CLI's probe root regress, the scenario stays green. The sibling assert_gpu_target_matches_kfd got a "nothing may quietly decline" guard in this PR; this step did not.
    Confidence 70 · logic · Fix: when world.simulated_host is Some, assert the node count equals host.gpus.len() and fail instead of returning. Skip the lspci bail-out for simulated hosts.

Non-blocking

  • fake-host-tool answers plausibly for argv nobody planted — tests/e2e-cucumber/src/bin/fake-host-tool.rs:43-49
    Lookup falls back from the full command line to the bare tool name, so uname -n or lspci with different flags return the canned -a / -nn -D text with exit 0. If the CLI changes its lspci flags, the stand-in will not disagree, which weakens the drift protection this PR is built around.
    Confidence 70 · logic · Fix: fall back to the bare name only for argument-less tools (mpirun, lsmod); otherwise exit 127 with a loud stderr message.
  • Stale tag name in comments — .github/workflows/ci.yml:1527 says "@requires-gpu scenarios resolve to skip here", and features/dash.feature:329 says "@requires-wsl rather than @requires-gpu"; the skipping tag is now @requires-real-gpu.
    Confidence 85 · mechanical · Fix: rename in both.
  • plant_simulated_host panics on a second call into the same root — simulated_host.rs:437 (link(), symlink already exists). Loud, but undocumented; two Given steps in one scenario would trigger it.
    Confidence 70 · mechanical · Fix: document it, or fail with a message naming the double plant.
  • Unused Platform::Wsl2 { rocdxg_ready: false } variant — simulated_host.rs:137, 247-250, 317 has ldconfig/rocdxg branches nothing constructs, so they are untested dead code.
    Confidence 70 · mechanical · Fix: drop the variant and branches, or add a scenario that uses it.
  • README does not tell GPU-box users about E2E_HARDWARE — the tag table mentions it, but the environment-variable table and run examples do not. A plain cargo xtask e2e on a GPU machine now skips every @requires-real-gpu scenario without saying why where people look.
    Confidence 70 · documentation · Fix: add an E2E_HARDWARE row (default simulated) and an E2E_HARDWARE=real cargo xtask e2e example.

Decisions for the author

  • Coverage moves off real hardware, and Windows/macOS lose scenarios — tradeoff
    • serve-13 (serve-no-gpu-fails-fast) was @requires-no-gpu, so it ran on any no-GPU lane including Windows and macOS; it is now @requires-os:linux on a simulated host. examine-10 went from @requires-bare-metal (skips only on WSL) to @requires-os:linux, so --no-frameworks is no longer checked on Windows or macOS.
    • serve-13/15/16/19/20/21, examine-10/11/17, diagnose-08/15/16/17/18 and driver-install-01..03 no longer run against a real GPU or WSL host. Only examine-04 has a real twin (examine-29); examine-30/31 check layout assumptions only. Real masked-device serve refusals and real-WSL diagnose/driver-plan paths run in no lane.
    • The simulated run is hermetic and runs on every PR, which is a real gain. Cheap mitigations: a @requires-real-gpu copy of serve-15 and a @requires-wsl copy of diagnose-17, as examine-29 does for examine-04; keep a non-simulated no-GPU variant for Windows/macOS. Note the description's "each moved scenario still asserts the same behaviour" and "gain coverage" do not mention the Windows/macOS loss.
  • Fake vLLM ignores everything except --host and --port — tradeoff (src/bin/fake-vllm.rs:30-36)
    serve-11 proves selection, runtime lookup, launch and readiness, but not that the CLI passes sane vLLM flags; --port=N would be ignored and fall back to 8000. Recording argv under the root and asserting the expected flags would close this; acceptable if the flags are covered elsewhere.
  • serve-11 failed in the test plan on a port conflict (127.0.0.1:11435) — non-blocking-improvement
    A fixed port in a hermetic harness invites this on shared runners; an ephemeral port would remove the flake.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 124b181

I read the full diff for this PR (simulated-host infrastructure, expectation/tag plumbing, CI workflow changes, and the migrated e2e feature/step files), cloned the repo at the PR head commit, and cross-checked the two concerns from @tomastola's existing CHANGES_REQUESTED review against the current code. No Rust toolchain is available in this environment, so nothing was compiled or executed — everything below is from static reading of the diff and surrounding code.

Confirmed still open (both from the existing review)

  • examine-17's assertion can still pass without checking anything — tests/e2e-cucumber/tests/e2e/examine_steps.rs:1012-1016 (assert_gpus_match_kfd_nodes). The function returns silently if kfd_gpu_node_versions(...) is None/empty, before ever comparing against the planted host. Compare this to its sibling assert_gpu_target_matches_kfd (same file, lines 945-956), which was given an explicit if let Some(expected) = world.simulated_host....gfx_target() { assert_eq!(...) } guard so "nothing below may quietly decline to check it." assert_gpus_match_kfd_nodes has no equivalent direct comparison against world.simulated_host.as_ref().map(|s| s.host.gpus.len()). The feature comment at features/examine.feature:226-227 ("the step's premises always hold") documents the intent but the code doesn't enforce it — a regression in planting, world.host_path, or the CLI's probe root would make this scenario pass vacuously instead of failing. Confidence: high (70-80%), this is a real gap, not a false positive.

  • Coverage-loss tradeoff is real and still present — verified via git grep at head: features/model_serving.feature:135 (serve-13) now carries @requires-os:linux only, down from @requires-no-gpu (which also ran on Windows/macOS). features/examine.feature:111 (examine-10) is similarly now @requires-os:linux, down from @requires-bare-metal (which only skipped WSL). Both comments in the feature files candidly explain the simulated premise, but the net effect for those two scenarios is that Windows/macOS lanes no longer exercise GPU-required-refusal and --no-frameworks behavior. This is a tradeoff for the author/reviewers to accept or mitigate (e.g. a non-simulated Windows/macOS variant), not something I'd block on myself, but it is accurately described in the prior review and still applies at this head commit.

Additional observation (non-blocking, low confidence)

  • tests/e2e-cucumber/src/simulated_host.rs (filter_real_path/link_stand_ins, around line 402) rebuilds the filtered-PATH fixture from scratch on every scenario invocation — a fresh read_dir+stat+symlink pass over every PATH directory per scenario. With 19+ scenarios now going through this path per run, this adds I/O overhead to the per-PR suite. This is a performance nit, not a correctness issue, and I did not measure actual wall-clock impact.

Not independently re-verified

I did not re-derive every item in the prior review's non-blocking list (fake-host-tool argv fallback, stale @requires-gpu tag mentions in comments, plant_simulated_host panic-on-double-call, unused Wsl2 { rocdxg_ready: false } variant, README gap, fake-vllm flag handling, serve-11 port conflict) — a spot read of the code is consistent with those being accurate, but I did not independently trace each one line-by-line given the medium-effort budget for this pass. I found no additional correctness bugs beyond what's above and what's already flagged.

🤖 Generated with Claude Code

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 124b181

Review — needs work

Full review of the whole change, run as five independent file-grouped passes plus consolidation.
Implements ROCMAI-677 ("E2E: describe GPU hosts with simulated machines so GPU-visibility scenarios run on every PR"); the PR description's claims (HardwareMode gating, simulated_host.rs layout, fake-vllm/fake-host-tool stand-ins, nineteen scenarios moved to simulated machines, examine-29/30/31 real-hardware/drift checks) match what the diff contains, verified across passes with no mismatch found.

Blocking

  • CI check "E2E tests (MI300X, current)" is failing on the reviewed commit — .github/workflows/e2e-selfhosted.yml (self-hosted MI300X lane); run: https://github.com/ROCm/rocm-cli/actions/runs/40314879688/job/121545019785
    Confirmed via gh pr view 517 --json statusCheckRollup (conclusion: FAILURE) and the rendered Actions page, which shows the "Run E2E tests on GPU hardware" step exiting with code 1; every other check on the PR is green. Full job logs were not retrievable from this environment (the GitHub Actions REST log/run API endpoints returned 403/Access Denied through an egress proxy), so the specific failing scenario or assertion could not be identified. This PR's own e2e-selfhosted.yml changes are what first set E2E_HARDWARE: real on this exact lane, so the failure may be the first real exercise of the new @requires-real-gpu gating on that hardware rather than an unrelated flake — that is plausible, not established.
    Confidence 100 · architectural · Fix: obtain the job's actual log/annotation (needs sign-in or maintainer access), determine whether it's a genuine regression in the new real-GPU gating/hardware-state assumptions or a known self-hosted-runner flake, and resolve or re-run before merge.

Non-blocking

  • ldconfig cache header reports a fixed count of 5 libraries that doesn't match the number actually written — tests/e2e-cucumber/src/simulated_host.rs:308-320. The canned ldconfig -p output always prints "5 libs found in cache" but only 4 unconditional entries are written (a 5th, librocdxg.so, is added only for Platform::Wsl2 { rocdxg_ready: true }), so on bare-metal and WSL2-not-ready machines the printed header disagrees with the printed list below it. No consumer currently parses the header count (rocm-core's checks match by soname substring), so nothing currently breaks, but it's a fidelity gap in a fixture whose whole point is to match real ldconfig output.
    Confidence 80 · mechanical · Fix: compute the header count from the number of lines actually emitted rather than hardcoding it.
  • bare_metal_layout_drift's GPU-topology checks early-return when the simulated side reports zero GPUs — tests/e2e-cucumber/src/simulated_host.rs:520-557 (let Some((_, simulated_props)) = simulated_gpus.first() else { return drift; }). Two drift checks (non-zero gfx_target_version on the CPU KFD node; DRM card vendor/device id shape) never execute when comparing against a no-GPU simulated host. Every current call site only drives this with instinct_mi300x(1), so it's unreachable today, but it's a latent gap if this drift check is ever reused for a no-GPU scenario.
    Confidence 70 · logic · Fix: run those two checks unconditionally, independent of whether the simulated side has a GPU.
  • fake-host-tool.rs redeclares TOOL_OUTPUT_DIR and TEST_HOST_ROOT_ENV instead of importing them from simulated_host.rs — tests/e2e-cucumber/src/bin/fake-host-tool.rs:22,24 vs tests/e2e-cucumber/src/simulated_host.rs:38,41. The two copies are held in sync only by a "must match" doc comment with no compiler enforcement; the sibling binary fake-vllm.rs:17 proves a [[bin]] target in this crate can use e2e_cucumber::... to reach the crate's own lib, so this duplication isn't architecturally forced (unlike the separate, legitimately-forced duplication against rocm_core::hardware_root::TEST_HOST_ROOT_ENV, which fake-host-tool.rs cannot reach).
    Confidence 75 · mechanical · Fix: import the two constants from e2e_cucumber::simulated_host instead of redefining them.

Decisions for the author

  • @id:examine-detects-gpu-and-driver (examine-04) keeps its scenario id while changing what it verifies — tradeoff. Before this PR the id denoted "verified against the real installed GPU" (rare, @requires-gpu-gated, self-hosted-only runs); after, the same id denotes "verified against a simulated Instinct GPU" (frequent, @requires-os:linux, every Linux PR lane), with a new, narrower id (examine-detects-this-machines-gpu-and-driver, examine-29) now covering what the old id used to mean, and a nightly drift check (examine-30/31) mitigating simulated/real divergence. This is well-reasoned and documented in the feature-file comments, but if any external dashboard or flaky-test tracker keys history by scenario id, examine-04's historical baseline will now be read against a very different population of runs. Worth confirming no such tracker exists, or that the discontinuity is acceptable.

Positive signals

  • bare_metal_layout_drift, wsl_layout_drift, and lspci_drift (simulated_host.rs) are explicit, tested mechanisms for catching simulation/reality divergence — e.g. a regression test pins the exact gfx_target_version-in-properties vs standalone-file bug that motivated simulating hardware in the first place — directly addressing the core risk that a simulated-host suite could drift from what real hardware actually does.
  • HardwareMode::parse (expectation.rs) fails closed: an unrecognized E2E_HARDWARE value panics the run rather than silently defaulting to simulated, so a typo can't quietly skip every real-GPU scenario while CI stays green — and the one call site (tests/e2e.rs) is wired to actually propagate that panic rather than swallow it.
  • assert_vllm_default's new simulated-only readiness assertion (serving_steps.rs) closes a real gap: rocm serve exits 0 regardless of readiness, so without this check a simulated vLLM that never came up would have passed silently.
  • feature_body (expectation.rs) reads the actual .feature file text to assert a scenario's premise line is really present, so deleting the underlying Gherkin step would fail this test rather than silently certifying stale coverage.

Deployment notes

None — no product code changes; this PR only changes which CI lanes exercise which existing e2e scenarios and adds new test infrastructure.

What this covered

Every file in the diff, read completely at PR head 124b1815e0ef5359513dece166627e3f81f9e04b against base d59f596f6e08297c207f008021eecb37defc3558 (32 files, +1991/-261), split across five independent file-grouped passes: (1) tests/e2e-cucumber/src/simulated_host.rs; (2) tests/e2e-cucumber/src/expectation.rs + mock_server.rs + lib.rs; (3) tests/e2e-cucumber/src/bin/fake-host-tool.rs + fake-vllm.rs + tests/e2e/simulated_host_steps.rs + tests/e2e.rs; (4) the remaining tests/e2e/*_steps.rs files plus all ten changed .feature files; (5) docs/testing.md, docs/ci-hardware-testing.md, the three .github/workflows/*.yml files, tests/e2e-cucumber/Cargo.toml, tests/e2e-cucumber/README.md. Each pass also read relevant surrounding/consumer code outside its primary scope (e.g. crates/rocm-core/src/examine.rs, openmpi.rs, apps/rocm/src/service_launch.rs, engines/vllm/src/runtime.rs) to verify specific cross-file claims. Step 7 (design review) questions were answered from material the five passes already surfaced (scenario-id continuity, simulation fidelity, real-vs-simulated gating design) rather than a separate fan-out round.

CI: gh pr view 517 --json statusCheckRollup read in full; one check (E2E tests, MI300X, current) reports FAILURE, all others SUCCESS/SKIPPED — reflected above as a blocking finding. Job logs were not retrievable (GitHub Actions REST log/run endpoints returned 403 from this environment's egress proxy); only the rendered Actions page's step name and exit code were available.

Did not run: the history pass (git log/git blame on modified regions) and prior-changes pass (review comments on earlier changes to these files) were not run this round — simulated_host.rs, the fake binaries, and the simulated-host step definitions are new files with no history, and the remaining changed files' history was not separately mined given the depth already reached by the code and impact-analysis passes; re-run these specifically if deeper confidence is wanted. No cargo build/cargo test/clippy was executed in this environment; all findings were settled by reading source against source, not execution — the PR's own description reports cargo test, clippy, and a 237-scenario local run as green except three explained, non-blocking flakes (serve-11, comfyui-04, automations-04), consistent with what the diff shows but not independently re-run here.

Ticket ROCMAI-677 retrieved by key (title, status, issue type only — no comments): "E2E: describe GPU hosts with simulated machines so GPU-visibility scenarios run on every PR," status In Progress. Matches the change's own description of itself.

Not reconciled against any PR discussion, review comments, or earlier review rounds — none were read, per method.

@rominf
rominf force-pushed the test/e2e-simulated-machines branch from 124b181 to 6a09714 Compare November 3, 2026 14:19
@rominf

rominf commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

@tomastola thanks. Rebased onto main and addressed in 3cd2e06, with the scenario numbering and tags in 6a09714:

  • examine-17 could pass vacuously: its step now fails on a simulated machine unless it reads back as many KFD GPU nodes as were planted (the lspci bail-out was already skipped there). Pointing the probe at a missing directory turns it red; the old code passed silently.
  • fake-host-tool's bare-name fallback: it answers only the exact command line it was planted for, with powershell.exe the one argv-independent exception, and exits 127 naming anything else. a_stand_in_answers_only_the_command_line_it_was_planted_for goes red if the fallback is widened again.
  • Stale @requires-gpu comments in ci.yml and dash.feature now say @requires-real-gpu, and main's seven newer scenarios that need a physical device are tagged @requires-real-gpu too, so no feature file carries plain @requires-gpu.
  • A second simulated-host plant fails fast, naming both machines.
  • The unused Wsl2 { rocdxg_ready: false } variant is gone.
  • README: an E2E_HARDWARE row and an E2E_HARDWARE=real cargo xtask e2e example.

From the later automated review: the ldconfig header count is computed from its entries, the drift check's CPU-node/DRM checks run without a simulated GPU, and fake-host-tool imports its constants. After the rebase the PR's scenarios are examine-31/32/33 (main now has examine-29/30). The MI300X failure was therock-next-09's vllm/amd-aiter pin mismatch, fixed on main by #755 and picked up by the rebase.

The coverage trade-offs you raised (twins for serve-13/serve-15/diagnose-17, fake-vllm flags, serve-11's fixed port) I'll answer separately.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 6a09714

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (base 9875940e, head 6a09714). Implements ROCMAI-677 ("E2E: describe GPU hosts with simulated machines so GPU-visibility scenarios run on every PR"); the ticket title/status match the PR's own description, and the description's claims (HardwareMode gating, simulated_host.rs layout, fake-vllm/fake-host-tool stand-ins, nineteen scenarios moved to simulated machines, examine-29/32/33 real-hardware/drift checks) match what the diff contains.

Earlier rounds on this PR (at older commits) flagged a vacuous-assertion gap in assert_gpus_match_kfd_nodes and an MI300X CI failure. Both are resolved at this head: assert_gpus_match_kfd_nodes now carries the same "never decline to check on a simulated host" guard its sibling assert_gpu_target_matches_kfd already had, independently re-verified by two blind passes below. CI has moved on to a different, newly-identified failure (below).

Blocking

  • A #[given(...)] step-text rename broke 3 Windows-only scenarios, and this is the confirmed root cause of the live CI failure on the Windows self-hosted lane — tests/e2e-cucumber/tests/e2e/examine_steps.rs:67 renamed the step from "a machine with an AMD GPU" to "this machine has an AMD GPU" (confirmed against the base commit, which still has the old text). Two of the Gherkin call sites were updated to match (examine.feature:486, runtime_setup.feature:186), but three were not: diagnose.feature:647 (@id:diagnose-windows-display-driver-advice-follows-the-inspection, scenario diagnose-41), diagnose.feature:755 (@id:diagnose-windows-hip-sdk-advice-follows-the-inspection, scenario diagnose-48), and examine.feature:453 (@id:examine-windows-driver-version-date-and-age, scenario examine-29) still say Given a machine with an AMD GPU, which no longer binds to any step function. All three are tagged @requires-os:windows @requires-real-gpu, which is exactly why this was invisible in the author's local WSL2 test run (those scenarios skip there) and only surfaces on the Windows GPU lane.
    Verified by direct evidence, not just reading: the "E2E tests (Strix Halo, Windows, current)" job on this exact commit (run 41652611022, job 125974649969) is FAILURE, and its log's own reconciliation output names precisely these three scenarios — 'diagnose-windows-display-driver-advice-follows-the-inspection' was expected to pass on this host but FAILED — a regression., 'diagnose-windows-hip-sdk-advice-follows-the-inspection' was ... FAILED, 'examine-windows-driver-version-date-and-age' was ... FAILED — a 1:1 match with the three stale call sites.
    Confidence 100 · mechanical · Fix: update the three Given lines to Given this machine has an AMD GPU.

  • CI is not green on this head beyond the above, and the gaps need triage, not just the rename fix — the same Windows job also reports a 4th failure, 'runtime-resolve-canonical-release' (runtime_setup.feature:186, tag @requires-gfx-target), which already uses the correct renamed step text, so it is not explained by the finding above; its cause is still open. Separately, two other self-hosted jobs — "E2E tests (Strix Halo, Ubuntu, release, current)" and "E2E tests (Strix Halo, Ubuntu, nightly, current)" — are also FAILURE, but their logs show ##[error]No space left on device during action checkout, i.e. runner disk exhaustion unrelated to this PR's code. Two more self-hosted jobs ("E2E tests (MI300X, current)", "E2E tests (W7900, current)") are still QUEUED and have not reported at all as of this review. So: one confirmed code regression (above), one unexplained scenario failure, two infra flakes needing a runner-side fix or re-run, and two jobs pending — the PR cannot be called green yet on any of these grounds independently.
    Confidence 100 (all read directly from the live check rollup and job logs) · architectural · Fix: fix the rename (above); investigate runtime-resolve-canonical-release separately; re-run or escalate the two disk-space failures; wait for MI300X/W7900 to complete before merge.

Non-blocking

  • SimulatedGpu::gfx_target() formats the packed version's "minor" digit in decimal, not hex, diverging from the production formula it exists to mirror — tests/e2e-cucumber/src/simulated_host.rs:138-142 (format!("gfx{}{}{:x}", v / 10_000, (v / 100) % 100, v % 100)). The production decoder it stands in for, gfx_target_from_gc_version (crates/rocm-core/src/host_gpu.rs:1568-1573), hex-formats both the minor and revision digits (format!("gfx{major}{minor:x}{revision:x}")) — its own test (gc_version_encodes_components_above_nine_as_hex_digits) exists specifically because decimal and hex only agree below 10. Verified directly by comparing both formulas side by side. Currently dormant: the module's only two GPU constants (mi300x: minor=4, strix_halo: minor=5) have single-digit minors, so decimal and hex coincide and no test catches it — but this is test-oracle logic consumed by examine_steps.rs and serving_steps.rs to compute the expected value the CLI's real output is checked against, so a future GPU constant with minor ≥ 10 would silently assert the wrong expectation.
    Confidence 85 · logic · Fix: change to format!("gfx{}{:x}{:x}", v / 10_000, (v / 100) % 100, v % 100), matching gfx_target_from_gc_version.
  • TEST_HOST_ROOT_ENV is duplicated between simulated_host.rs and rocm_core::hardware_root with only a "must match" doc comment, not a compiler/test-enforced invariant — tests/e2e-cucumber/src/simulated_host.rs:40-42. A working precedent for closing exactly this gap already exists in the same crate: tests/e2e-cucumber/src/mock_server.rs's own #[cfg(test)] block already reaches the rocm-core dev-dependency (rocm_core::ManagedServiceRecord), so a #[test] asserting TEST_HOST_ROOT_ENV == rocm_core::hardware_root::TEST_HOST_ROOT_ENV is reachable the same way. If the two literals are ever edited independently, the CLI binary would silently stop reading the simulated-host directory this whole PR plants, with nothing to notice.
    Confidence 75 · architectural · Fix: add that one-line equality test to simulated_host.rs's existing mod tests.
  • bare_metal_layout_drift's reference-node selection relies on non-deterministic read_dir order — tests/e2e-cucumber/src/simulated_host.rs:530-552 (simulated_gpus.first(), no sort). The sibling production function this mirrors (detect_kfd_gfx_target_in in crates/rocm-core/src/host_gpu.rs) explicitly sorts nodes before picking "first," with a comment noting read_dir order is filesystem-defined. This drift-check helper has no equivalent sort, so its choice of "the" simulated GPU node to compare every real node's properties against could vary run to run once a simulated host has ≥2 heterogeneous GPU nodes. Dormant today: every current caller uses either a single-GPU host or homogeneous multi-GPU hosts.
    Confidence 70 · mechanical · Fix: sort gpu_nodes's entries (e.g. by node directory name) before taking .first().
  • SimulatedHost's public platform/gpus fields let a caller construct a self-inconsistent machine — tests/e2e-cucumber/src/simulated_host.rs:166-171. Nothing stops SimulatedHost { platform: Platform::Wsl2, gpus: vec![SimulatedGpu::mi300x(..)] } outside the three documented constructors; plant_wsl2 (same file, 379-393) ignores self.gpus entirely and unconditionally writes the Strix-Halo-specific Windows-inventory line, so such a host's reported GPU and its simulated Windows inventory answer would contradict each other. Not currently reachable — every call site goes through instinct_mi300x/bare_metal_without_gpu/wsl2_strix_halo.
    Confidence 70 · architectural · Fix: make the fields private behind the existing constructors, or have plant_wsl2 derive its answer from self.gpus.
  • ScenarioDecl.requires_real_gpu ⇒ requires_gpu is enforced only inside from_tags, not by the type — tests/e2e-cucumber/src/expectation.rs:111-219 (public independent bool fields), set together only inside the tag-parsing loop (:259-263); resolve()'s two gates key off requires_gpu alone and never re-derive it from requires_real_gpu. The file already has precedent for partial struct-literal construction of a ScenarioDecl elsewhere in its own tests, so a future hand-built literal could silently desync the two flags, which would make a @requires-real-gpu scenario resolve to ExpectPass on a GPU-less real-hardware host instead of skipping. Not currently reachable — the only production constructor is from_tags.
    Confidence 70 · architectural · Fix: compute requires_gpu from requires_real_gpu (private field + accessor), or debug_assert! the implication at the top of resolve().

Decisions for the author

  • Coverage moves off real/WSL hardware for most of the 19 migrated scenarios, narrowing two of them from a broader platform tag to Linux-only — tradeoff. Verified directly against the diff: serve-13 (serve-no-gpu-fails-fast) moved from @requires-no-gpu to @requires-os:linux; examine-10 (examine-can-skip-framework-probing) moved from @requires-bare-metal (which only skipped WSL) to @requires-os:linux. Of the nineteen moved scenarios, only examine-04 keeps a real-hardware twin (examine-29); the nightly checks (examine-32/examine-33) verify fixture-layout fidelity, not scenario behavior, on real hardware. This is a reasonable tradeoff — hermetic, always-on coverage gained vs. real/WSL exercising lost for most moved scenarios — and the PR description's framing ("the moved scenarios gain coverage") doesn't mention the Linux-only narrowing for these two. Worth the author confirming this is accepted as-is rather than mitigated with a @requires-real-gpu/non-simulated twin for either.
  • The simulated vLLM stand-in ignores every flag except --host/--port — tradeoff (tests/e2e-cucumber/src/bin/fake-vllm.rs). serve-11 genuinely proves engine selection, runtime lookup, managed launch, and readiness-wait (the fake binds the CLI-passed host/port rather than a hardcoded default, confirmed against engines/vllm/src/process.rs's vllm_serve_args), but does not prove the CLI passes correct additional vLLM flags — a wrong --max-model-len or similar would go unnoticed here. Acceptable if flag correctness is covered elsewhere; worth confirming that scope boundary is intentional.

Positive signals

  • assert_gpu_target_matches_kfd and assert_gpus_match_kfd_nodes (examine_steps.rs) both now correctly guard every early return against vacuously passing on a simulated host — independently re-traced by two blind passes in this round, each confirming no early-return path discharges examine-04/examine-17 without having actually checked the planted topology.
  • fake-host-tool's dispatch answers only the exact command line it was told to expect, with one narrow, explicitly-justified exception (powershell.exe, which must accept a whole inventory script) — directly defeating the "stand-in silently papers over a changed CLI invocation" failure mode, backed by a dedicated unit test (a_stand_in_answers_only_the_command_line_it_was_planted_for).
  • bare_metal_layout_drift/wsl_layout_drift/lspci_drift, exercised nightly against real hardware (examine-32/33), is a well-targeted regression guard: its own test suite pins the exact historical bug (gfx_target_version read from a standalone file no kernel has) that motivated building this simulated-host infrastructure in the first place.
  • HardwareMode::parse fails closed on an unrecognized E2E_HARDWARE value (panics rather than defaulting to simulated), consistent with this project's no-silent-GPU-fallback convention, and the one production call site (tests/e2e.rs) propagates that panic rather than swallowing it.
  • tui_driver.rs's isolate_env_removals() addition closes a real, pre-existing isolation gap (a PTY-driven serve on a simulated host could previously inherit the runner's own HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES), bringing the PTY path in line with the piped path's existing isolate_cmd guard.
  • Docs/CI consistency is well maintained: docs/testing.md, docs/ci-hardware-testing.md, and tests/e2e-cucumber/README.md all describe the current @requires-real-gpu/E2E_HARDWARE semantics without drifting from each other or from the actual tag names in the source, and every self-hosted GPU job in both workflow files correctly sets/forwards E2E_HARDWARE: real.

Deployment notes

No product code changes — this PR only changes test infrastructure and which CI lanes exercise which existing e2e scenarios. Do not merge until the Blocking items are resolved: the confirmed step-text regression, the unexplained runtime-resolve-canonical-release failure, the two disk-space infra failures, and the two still-pending self-hosted jobs (MI300X, W7900).

What this covered

Every file in the diff (33 files, +2150/-282) read completely at PR head 6a0971404e3ddd4738074745f26c841d3f1613ed against merge-base 9875940ebc7d1dbf9db0706e01d29842e8ec1d27, by six independent blind fan-out workers grouped by coupling: (1) simulated_host.rs; (2) expectation.rs + mock_server.rs + lib.rs; (3) the fake binaries + simulated_host_steps.rs + tests/e2e.rs; (4) diagnose/driver/examine step files + their feature files; (5) serve/runtime/comfyui/tui step files + their feature files; (6) docs + CI workflow files + Cargo.toml + README. Each worker also read relevant surrounding/consumer production code (crates/rocm-core/src/{host_gpu,examine,diagnose,fix}.rs, apps/rocm/src/service_launch.rs, engines/vllm/src/{runtime,process}.rs) to verify cross-file claims rather than trusting the test code's own framing. I then independently re-verified the highest-value candidates myself (the step-text rename, the gfx_target hex/decimal divergence, the serve-13/examine-10 tag narrowing) directly against the source and the live CI logs before scoring them.

CI was read live via gh pr view 517 --json statusCheckRollup and GitHub Actions job logs, twice (once mid-review, once immediately before publishing) — the head commit had not moved between the two checks.

Caveat on blindness: before delegating the file-grouped passes above, this orchestrating context necessarily read the full bodies of three prior reviews on this PR (pinned to an older commit, 124b181) in order to perform the mandated check for whether this PR had already been reviewed at its current head — it had not, since the head has since advanced to 6a09714. That prior-review content was used only to decide whether to proceed, and afterward to sanity-check which previously-raised issues are now fixed (confirmed: the assert_gpus_match_kfd_nodes vacuous-pass gap and the ldconfig header-count mismatch are both resolved at this head); it was not given to any of the six fan-out workers, who read the code fresh with no knowledge of prior rounds, and none of this review's findings reproduce a prior round's finding. Did not run: the history/prior-changes passes were not run as a separate dedicated pass beyond git log spot-checks already folded into the workers' own investigation, since most touched files are new. No cargo build/cargo test/clippy was executed in this environment (no Rust toolchain available) — all non-CI findings were settled by reading source against source, not execution. Not reconciled against any PR discussion or review comments — none were read as part of producing these findings, per method.


🤖 Generated with Claude Code

@rominf

rominf commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

@tomastola the trade-offs from your review, answered. The blocking items were answered in my earlier comment. Rebased onto main at 3e00084. main has since added scenarios, so this PR's scenarios are renumbered; the numbers below are current. Edited after posting: an earlier version of this comment said the no-GPU refusal is held on native Windows, which is wrong; see the first point.

  • Windows lost the no-GPU refusal and --no-frameworks:
    • serve-60 (@id:serve-no-gpu-fails-fast-on-this-host, @requires-no-gpu @requires-os:linux) is serve-13 without a simulated machine: the refusal on the runner's own GPU-less Linux host, such as the GitHub-hosted one. (61689f8)
    • It cannot cover native Windows. Off Linux the CLI has no usable-GPU probe, and rocm serve reads "cannot tell" as "not known to be absent", so a GPU-less Windows host launches instead of refusing. That is a product gap, not something this PR changes, so the GPU-required refusal still has no Windows scenario; the description says so.
    • examine-10 no longer plants a machine or carries a platform tag. The framework probe now runs on bare-metal Linux, WSL2 and native Windows alike, so it runs on every lane, Windows included. (04b3d4e)
    • macOS is out of scope here: AGENTS.md lists Windows and Linux as the supported hosts.
  • Real-hardware twins:
    • serve-61 is serve-15's every-GPU-masked refusal on a real Linux GPU host.
    • diagnose-65 is diagnose-17's platform guard on a real WSL2 distribution.
    • examine-49 is examine-04 against the runner's own GPU and driver.
    • The scenarios still without a twin are listed in the description: the multi-GPU mask cases (serve-19/20/21), serve-11, driver-install-01..03, diagnose-18, examine-11 and examine-17. They are checked only against simulated machines, with the nightly drift checks examine-50/51 as the backstop.
  • fake-vllm ignored its flags: it now records the arguments it was launched with. serve-11 asserts the model, host, port and --enforce-eager, and dropping --enforce-eager from the launch turns it red. (61689f8)
  • serve-11's fixed port: it serves on a free port. It also no longer runs the GPU lanes' port cleanup, which kills any listener on the serve ports. (61689f8)

The description now says plainly what moved off real hardware and what has no twin.

@rominf

rominf commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

@nowycondro answers to the automated reviews of 124b181 and 6a09714. Rebased onto main at 3e00084. This PR's scenarios are renumbered after main's newer ones; the numbers below are current.

Blocking items from the 6a09714 review:

  • Step-text rename broke three Windows scenarios: fixed. No feature file uses Given a machine with an AMD GPU any more, including main's newer runtime-32.
  • runtime-09 failing on the Windows lane: not this PR. The preview resolved TheRock packages through a package mirror configured on that runner, so canonical_source named the mirror rather than the canonical index the step expects.
  • The Strix Halo Ubuntu lanes: they failed with "No space left on device" during checkout, which is runner infrastructure.
  • CI on the new head: pending; I'll follow the checks to completion.

Non-blocking items, all in adbb7d2:

  • gfx_target() renders minor and stepping as hex, as the product does (90010 is gfx90a).
  • TEST_HOST_ROOT_ENV is tied to rocm_core's constant by a test.
  • The drift check orders KFD nodes numerically before choosing the reference node (77d0619).
  • SimulatedHost is built only through its constructors.
  • The GPU gate treats @requires-real-gpu as needing a GPU on its own.

Decisions left to the author:

  • Coverage trade-off and the fake-vllm flags: answered in my reply to tomastola above. serve-60 holds the no-GPU refusal on GPU-less Linux hosts (not on native Windows, where the CLI has no usable-GPU probe; see that reply), examine-10 runs everywhere, and serve-11 asserts the launch flags.
  • examine-04 keeps its id: the id names the behaviour, detecting the GPU and driver, which examine-04 still checks. The real-machine check has its own id, examine-detects-this-machines-gpu-and-driver (examine-49). The reports in this repo key on ids within a run. I don't know of anything that keeps history by id; if there is something, examine-04's history would start mixing simulated runs at this merge.

Also changed since 6a09714, from a full review of my own:

  • A simulated run no longer serves or installs on the runner's GPU. A matrix row can now say hardware = "real", and the chat-06/07 rows do, so a simulated run on a vLLM box does not read chat-06's pass as a stale XPASS.
  • fake-host-tool answers only the exact planted command line and exits 127 otherwise. Its refusal, the PATH filter and the WSL Windows-inventory answer each have a test that fails when its branch is broken.
  • serve-59 is serve-19's case on a one-GPU machine. The shared Then no longer accepts "out of range", which would have let a broken mask check pass there.
  • @requires-multi-gpu and its capability-probe GPU count are removed: the multi-GPU scenarios plant their own GPUs.
  • scripts/run-e2e-lane.sh exports E2E_HARDWARE=real for the shared-queue lanes. The workflow contract fails when it stops, when it runs the suite first, or when any other line names the variable.
  • In a hook build with a simulated root, the WSL ROCDXG check asks only the ldconfig on PATH, not the real host's /sbin copies (dbbb576). A build without the hook is unchanged.

GPU scenarios are about to stop depending on the runner's hardware: they
will plant a simulated host instead, so they run on GitHub-hosted runners
on every PR. A few premises cannot be simulated, such as a model actually
generating tokens on the device, and those need a way to say so.

@requires-real-gpu implies @requires-gpu and also skips unless the run
sets E2E_HARDWARE=real. The default is `simulated`. Any other value aborts
the run, because a misspelt `real` would otherwise skip every real-GPU
scenario and still report green.

Every self-hosted GPU lane, per-PR and nightly, sets E2E_HARDWARE=real
now, and the WSL lanes forward it through WSLENV. Nothing carries the tag
yet, so no scenario changes lanes in this commit.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
GPU scenarios could only run where the runner had the right hardware.
The CLI reads the real machine, so detection and refusal paths that never
touch the device were confined to the GPU lanes, multi-GPU premises to
the hosts with several cards, and WSL premises to the one WSL lane.

Scenarios now name the machine they need in a Given step, for example "a
machine with eight AMD Instinct GPUs", "a bare-metal Linux machine with
an AMD GPU" or "a WSL machine with an AMD GPU passed through". The step
plants a directory laid out the way the kernel lays out /dev, /sys and
/proc, points ROCM_CLI_TEST_HOST_ROOT at it, and isolates the commands
from the runner:
- Commands get a PATH with the hardware tools removed (uname, lspci,
  ldconfig, rocminfo, amd-smi, the kernel log, the WSL interop shells
  and others). The tools the simulated machine has come back as a
  fake-host-tool stand-in that answers from canned output.
- The runner's GPU visibility masks and ROCm variables are unset.
- HOME is moved into the scenario.

"A managed runtime is active" plants a runtime record whose vllm is a
fake-vllm stand-in. A serve on a simulated machine therefore runs the
real engine selection, runtime lookup, launch and readiness wait, and
only the process at the end is simulated.

Eighteen scenarios move onto simulated machines and run on every Linux
lane, including the GitHub-hosted one: examine-04/10/11/17,
diagnose-08/15/16/17/18, driver-install-01..03 and
serve-11/13/15/19/20/21. examine-49 repeats examine-04 against the
real machine, so a real GPU lane still covers detection there,
including the Windows lane. serve-16 stays on real hardware: it also
runs on native Windows, which a simulated machine cannot describe.
Every scenario left on @requires-gpu needs a physical device or a
runtime really installed for one, so those become @requires-real-gpu.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Scenarios on a simulated machine fail only when the CLI misreads the
fixture. If a kernel or driver update moves something the fixture
assumes, those scenarios keep passing while the CLI fails on hardware.
That is how the KFD target came to be read from a file no kernel has.

Two @nightly @requires-real-gpu scenarios now hold the fixture against
a real machine.

On a bare-metal GPU host they check:
- each KFD GPU node states gfx_target_version inside its properties file
- the property keys the fixture writes are present
- gpu_id files exist
- render nodes match drm_render_minor
- CPU nodes report a target of 0
- the AMD DRM cards have vendor and device files
- `lspci -nn -D` lines have the expected shape

On a WSL2 distribution they check /dev/dxg, libdxcore.so and the
microsoft-standard-WSL2 kernel string.

Each difference is reported along with what to update.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
examine-17's Then step returned silently when the KFD topology read
came back empty, so a planting or probe-root regression left it green.
On a simulated machine it now requires as many GPU nodes as were
planted. fake-host-tool answers only the exact command line planted
(powershell.exe excepted) and exits 127 naming anything else, so a
changed CLI invocation can't get stale canned text. The ldconfig -p
header counts its entries, the layout drift check runs its CPU-node and
DRM checks without a simulated GPU, the unused WSL "rocdxg not ready"
variant is gone, a second simulated-host plant fails fast, and the
README documents E2E_HARDWARE.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Seven scenarios main added since this branch was cut still carried
@requires-gpu although each needs a physical device: the Windows driver
and HIP SDK checks, the AIM serving pair, the live-reuse --gpu
validation, and the ComfyUI runtime-preservation install. They now carry
@requires-real-gpu like every other such scenario, so a simulated run on
a GPU box skips them. docs/testing.md names the new tag and the
E2E_HARDWARE=real opt-in for the ComfyUI run.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Fifteen scenarios main added since the last rebase still carried
@requires-gpu although each serves a model on the device through a
really installed runtime: the no-background-helper check, the explicit
port refusals for a managed service and an untracked process, the
automatic port fallback, the explicit port that differs from a live
equivalent, the five Anthropic-endpoint scenarios, the small assistant
served by its short name, and the refusal of a repo with no
quantization variant (whose GPU only gets it past the earlier gate,
on the native Windows lane it is meant for), the two registered-GGUF
serves and the dash Lemonade journey. A default
(simulated) run on a GPU box would still have served on the physical
device with them; they now carry @requires-real-gpu like every other
such scenario. Every self-hosted lane, the merge-queue ones included,
sets E2E_HARDWARE=real, so they keep running where they ran before.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
From review of the simulated-machine harness:

- A simulated GPU's expected gfx target rendered the minor digit in
  decimal while the product renders both minor and stepping as hex, so
  a future GPU with a component above nine would have been checked
  against the wrong target. It now matches the product (90010 is
  gfx90a).
- The root variable the harness sets is now tied to the one the CLI
  reads by a test, instead of by a comment.
- The nightly drift check sorts KFD nodes before choosing the simulated
  node every real one is compared with, as the product does, instead of
  taking whichever the filesystem lists first.
- A simulated machine is built only through its constructors, so its
  platform and GPUs cannot disagree.
- The GPU gate treats @requires-real-gpu as needing a GPU on its own,
  rather than relying on the tag parser to have set both flags.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A simulated run was described as checking the same thing on a GPU box as
on a GitHub-hosted runner, but two setup steps still decided by whether
the runner had a GPU: "a model is served in the background" served a
real model and "a managed runtime is active" installed a real SDK
whenever one was present, so chat-06, chat-07 and serve-49 did real GPU
work on a GPU box in the default mode. Both now take their GPU-less path
unless the run opted into real hardware and the host has a GPU.

Their xfail rows follow the same mode: a matrix condition can now name
`hardware = "real"`, and the chat-06/07 rows do. Otherwise chat-06,
passing against the mock server in a simulated run on a vLLM box,
would read as a stale XPASS of its vLLM row and fail the run.

E2E_HARDWARE is read in one place, which also refuses a value that is
not valid Unicode instead of treating it as unset. The docs now name the
probes a simulated machine does not replace, including the DRM VRAM
fallback serve uses when amd-smi is missing.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A self-hosted lane that dropped E2E_HARDWARE, or a WSL lane that stopped
forwarding it through WSLENV, would skip every @requires-real-gpu
scenario and still go green: the hardware-mode parser rejects a
misspelling but cannot see an absence. The workflow contract now derives
the self-hosted lanes from both GPU workflows and requires each to set
it in its job env, each WSL lane to forward it into the guest, and no
GitHub-hosted job to set it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The drift checks run only nightly on real hardware, where a branch that
stopped firing reads as "no drift". wsl_layout_drift had no test at
all, and the bare-metal checks for /dev/kfd and the amdgpu module, a
missing gpu_id, a render minor with no render node, a DRM device id in
another form and an lspci listing with no AMD GPU were untested. Each
now has a test that fails when its branch is disabled.

The reference node is chosen in numeric order, as the product orders KFD
nodes, rather than by path, which put node 10 before node 2.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
"The inspection reports that the driver is available" passed whenever
the output contained "driver_status", which rocm examine prints whatever
it found, not_detected included, so examine-04 and its real-hardware
twin could not fail on the driver half of their name. The step now
reads the status: on a simulated machine it must be the one the planted
platform implies (amdgpu_available on bare metal, wsl_rocdxg_ready on
WSL2), and on the real machine one of the statuses that mean a usable
driver on Linux, WSL2 or Windows.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
serve-19 moved onto a simulated two-GPU machine, which left its
single-GPU case, a mask naming device 0 refusing ordinal 1 on a host
with one device, running nowhere while its comment still claimed it.
serve-59 is that case on a simulated one-GPU machine, and a unit test
holds it to one GPU. The shared Then step no longer accepts "out of
range": on one GPU ordinal 1 is out of range with or without the mask,
so accepting it let a broken mask check pass there.

With serve-19 and serve-20 planting their own GPUs, no scenario carries
@requires-multi-gpu any more. The tag, its gate, the capability probe's
GPU count that served only it, and their tests are removed rather than
kept for a scenario nobody has named.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Moving scenarios onto simulated machines left some premises checked
only against a simulation:

- serve-61 repeats serve-15's masked-device refusal on a real Linux GPU
  host, and diagnose-65 repeats diagnose-17's WSL platform guard on a
  real WSL2 distribution.
- serve-60 is serve-13's GPU-required refusal on the runner's own
  GPU-less Linux host. It is Linux only: off Linux there is no
  usable-GPU probe, so a GPU-less Windows host is not refused today.

serve-11 proved engine selection, launch and readiness, but the vLLM
stand-in bound whatever it was given, so wrong launch flags went
unnoticed. fake-vllm now records its arguments beside itself, and the
scenario checks the model, host, port and --enforce-eager. It serves on
a free port instead of the shared default, and no longer runs the GPU
lanes' port cleanup, which SIGKILLs any listener on the serve and
assistant ports and would kill a developer's own server. The stand-in
is installed with the shared fixture installer rather than a symlink.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Comments and docs still described the suite before simulated machines:
comments naming plain @requires-gpu where a scenario is now
@requires-real-gpu (ComfyUI's runtime install, the low-VRAM vLLM
scenario, the Lemonade backend-alignment notes), unit tests citing tag
sets their scenarios no longer carry, a fake-host-tool header listing
tools it is not linked as, a CI comment crediting simulated machines
with covering the modprobe directories that none plants, and a
runtime-lifecycle note saying no mock engine could answer readiness.
The drift checks were said to hold "every fixture assumption"; the docs
now say what they compare and what they do not. The step list in the
testing doc names the two-GPU machine, and expectations.toml names the
real-hardware gate.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The shared-GPU-queue lanes run scripts/run-e2e-lane.sh on real GPUs
outside any workflow, and it never set E2E_HARDWARE, so every
@requires-real-gpu scenario would have skipped there and the lane
stayed green. The script now exports E2E_HARDWARE=real unless the
caller set it, and the contract test fails if that export goes.

The same test only looked for E2E_HARDWARE in the env of jobs that are
not self-hosted, so a workflow-level env: or a run: script setting it
passed. It now checks everything in a workflow outside the self-hosted
GPU jobs, with a unit test for each of the three places.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
serve-11 always plants a simulated Instinct machine, so the steps only
it uses no longer need a branch for the runner's own GPU, and its
@requires-engine:vllm tag gated nothing: the engine gate applies only
to GPU scenarios, and the engine now follows the simulated GPU. The
When step now refuses to run without a simulated machine.

Three doc comments had ended up on the wrong item (real_gpu_host,
SIMULATED_RUNTIME_KEY, the single-GPU serve-53 test) and are moved back
to the items they describe. Comments and docs still described
serve-50/51/52 as @requires-gpu, chat-06/07 as serving for real on any
GPU host, diagnose-28 as carrying the Linux and bare-metal pair, and a
shorter list of tools taken off PATH than HOST_TOOLS holds. The probe
comments now say that the e2e suite answers ldconfig with a stand-in on
PATH, and that the absolute fallbacks still reach the real linker.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The WSL ROCDXG check asks `ldconfig -p` and, when PATH has none, falls
back to /sbin/ldconfig and /usr/sbin/ldconfig. On a simulated host the
e2e suite answers the bare `ldconfig` with a stand-in on PATH, but a
simulated machine without one would fall through to the runner's own
linker and read its cache. With a test host root set, only the
`ldconfig` on PATH is asked now, so its absence reads as "could not
ask". A release build has no test host root, so its behaviour is
unchanged.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
examine-10 planted a simulated bare-metal machine because the framework
probe used to stop on WSL2 before it reached the framework step. It no
longer does: bare-metal Linux, WSL2 and native Windows all reach it, so
the --no-frameworks choice is acted on everywhere. Planting a Linux
machine only narrowed the scenario to Linux lanes and dropped the
Windows lanes it used to run on. It now plants nothing and carries no
platform tag.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the test/e2e-simulated-machines branch from 5836191 to 04b3d4e Compare November 16, 2026 12:19
@github-actions

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants