Skip to content

test(e2e): require a platform decision on every capability-tagged scenario (ROCMAI-438) - #1074

Draft
rominf wants to merge 6 commits into
mainfrom
ROCMAI-438-test-platform-gated-scenarios
Draft

rominf wants to merge 6 commits into
mainfrom
ROCMAI-438-test-platform-gated-scenarios

Conversation

@rominf

@rominf rominf commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Why

A capability tag such as @requires-gpu asks what hardware the host has, not which OS it runs, so the native Windows GPU lane satisfies it. examine-lists-every-gpu-with-its-address-and-target (examine-17) carried only @requires-gpu, so it ran on the Windows GPU lane and passed with every step green while checking nothing: its step reads the KFD topology and returns early when /sys/class/kfd is absent, which it always is on native Windows (and on WSL2, which exposes no KFD nodes).

Platform gating lives in three independently maintained places: cfg in source, scenario tags, and lane configuration. Nothing tied the scenario side to the platform, so a scenario whose premise exists on one OS only could quietly run, and pass, everywhere a capability tag lets it. Applying the rule across the suite also showed that the AIM and vLLM scenarios, which the CLI refuses on native Windows, recorded that premise nowhere in their tags.

What changes

  • A guard requires a platform decision on every capability-tagged scenario. tests/e2e-cucumber/tests/platform_tags.rs reads the .feature files the harness reads by default (recursively, any extension case), expands Scenario Outlines the way cucumber does, and parses each scenario's tags with the harness's own ScenarioDecl::from_tags. Classification destructures ScenarioDecl without .., so a new tag field fails to compile until it is classified as a capability, an OS decision, or neither. The guard fails on a capability-tagged scenario with neither @requires-os:<os> nor @requires-wsl that is not in ALLOWED_ANY_OS (including one with no @id), on two scenarios sharing an @id, on a tag that ScenarioDecl reads placed on a Feature: or Rule: line (the harness builds each ScenarioDecl from the scenario's own tags only, so it never sees those; cucumber's own @serial, @retry and --tags tags there are legitimate and not reported), on finding no .feature file or no scenario, and on an allowlist entry that is duplicated, has an empty reason, names no scenario, names one with no capability tag, or names one whose tags already decide its OS. Each check is a pure helper with its own test against fixtures. ALLOWED_ANY_OS holds 39 entries, each an (id, reason) pair whose reason says what the scenario's steps assert and why that holds on every OS a lane runs it on.
  • examine-17 is pinned to bare-metal Linux, and its step refuses to pass anywhere else. It gains @requires-os:linux @requires-bare-metal. The KFD read now takes a BareMetalLinux token that only the host check this_host_is_bare_metal_linux constructs, so reaching the step on any other host (an explicit filter that bypasses the tags, or tags dropped later) panics instead of returning early with nothing asserted. The check's accept/refuse logic is unit-tested on every host through driver_host_guard.rs.
  • AIM and vLLM scenarios carry @requires-os:linux. That covers serve-26 and serve-27 (AIM), and the seven @requires-engine:vllm scenarios that lacked it: serve-05, serve-11, runtime-03, runtime-04, runtime-18, runtime-lifecycle-12 and therock-next-09. The eighth, serve-23 (serve-vllm-low-vram-oom-guidance), already carried it. WSL2 is not excluded, since the CLI there is a Linux build.
  • The no-GPU refusal scenarios serve-13 and serve-29 carry @requires-os:linux. serve-no-gpu-fails-fast and serve-unrelated-live-service-still-fails-fast keyed their refusal on the usable-GPU probe, which exists only on Linux (WSL2 included); on native Windows it is a stub answering "unknown", which does not block a launch, so the refusal never fires there. They join serve-28, serve-30 and serve-32, which already carried the tag, and docs/testing.md now says all four no-GPU refusal scenarios do.
  • examine-10 drops @requires-bare-metal. The framework probe honours --framework skip on every platform, WSL2 included, so it now carries no capability tag and needs no allowlist entry.
  • The rule is written down once. docs/testing.md gains "Capability-tagged scenarios name their OS": the rule, everything the guard fails on, what it leaves to review (whether a Linux scenario also needs @requires-bare-metal), the run command and where it runs in CI. AGENTS.md §3's "Platform-gated code gets a test that runs on its gate" becomes "Platform-conditional behavior is verified on the platform it targets": it keeps the previous content and adds that a platform-specific change names its target platform and the check that runs there and would fail without it, and that a test returning early on the target is not that check. A short "A capability tag is not a platform tag" paragraph points at the testing doc, and §13 item 14 follows. The PR template gains one checklist item, CONTRIBUTING's "What reviewers look for" one bullet, and the @requires-os row of the e2e suite README's tags table a pointer to the testing doc. docs/architecture.md has an entry for the new file and updated entries for the host-guard files. Two doc comments in src/expectation.rs are corrected: the parser yields tags without the leading @, and the comment now lists what examine skips on WSL2. diagnose-28's comment now names the scenario that covers its WSL2 contract.
  • Docs scope the no-GPU refusal and the AIM nightly lanes to Linux. README.md's "Reuse is exempt from the no-GPU refusal" and docs/manual-testing.md's recipe for it now say the refusal applies on Linux, WSL2 included: on native Windows probe_usable_amd_gpu_indices answers unknown rather than "no GPU", so serve does not refuse, and the manual recipe's native Windows variant, which could never trigger the refusal, is removed. docs/testing.md's AIM paragraph now names serve-26/27's @requires-os:linux tag, the nightly lanes that run them, that the Windows nightly lane is excluded by the tag, and that on the WSL2 nightly lane they run only if docker info succeeds in its distro, which that workflow does not set up.

An allowlist was chosen over a no-op tag such as @any-os: each decision carries a written reason a reviewer can check, and it avoids churning dozens of tag lines in .feature files that many open PRs also touch.

The Windows lint gate for cfg(windows) code (warnings-as-errors) is unchanged: the windows-debug job already runs cargo clippy --workspace --all-targets -- -D warnings on windows-latest.

Platform

Lane effects of the tag changes:

Scenario Before After
examine-17 ran on every GPU lane, vacuously off bare-metal Linux bare-metal Linux GPU lanes only
examine-10 skipped on WSL2 also runs on the WSL2 lane (it already ran on native Windows)
serve-26, serve-27 (AIM, @nightly) eligible on the nightly E2E tests (Strix Halo, Windows, incl. nightly-only) lane, whose host reports AIM available, where the CLI refuses AIM skip on native Windows
seven vLLM scenarios skipped on native Windows by the harness's engine check still skipped on native Windows, now with a reason naming the OS; the tag records the premise
serve-13, serve-29 (no-GPU refusal, @requires-no-gpu) allowlisted as any-OS, but no native Windows lane ran them: the Windows GPU lanes have a GPU and skip @requires-no-gpu @requires-os:linux; still run on the Linux no-GPU mock e2e job, so no lane loses coverage

No scenario moves off the blocking mock e2e job: every scenario that gains a tag here either already carries @requires-gpu, which resolves to skip on that job's GPU-less host, or gains only @requires-os:linux, which that Ubuntu-hosted job satisfies (serve-13 and serve-29); examine-10 only loses one.

  • Target platform: native Windows, where examine-17 and the AIM scenarios stop running. The check that runs there and would fail without the change is the E2E tests (Strix Halo, Windows, current) lane. In run 43451380081, which tested dee0e6f (the only later tag change pins serve-13 and serve-29 to Linux, and that GPU host skips both as @requires-no-gpu either way), that job's e2e-gpu-strix-windows-report artifact has platform.json resolve examine-lists-every-gpu-with-its-address-and-target to skip with reason "requires os 'linux'; this host is 'windows'", and the scenario is absent from report.json. The same platform.json shows the host reporting AIM as available and serve-26/27 resolving to aim, skipped only as nightly-only, which is the nightly lane's case in the table. serve-missing-gguf-variant-refused (serve-56) and examine-can-skip-framework-probing (examine-10) pass there.
  • Target platform: WSL2, for examine-10. In the same run's E2E tests (Strix Halo, WSL2, current) job, examine-10 resolves to pass and passes.
  • Target platform: bare-metal Linux, for the KFD step's host check. It runs on the bare-metal Linux GPU lane, E2E tests (MI300X, current). Not verified on hardware: the E2E tests (MI300X, current) run on the current head is pending, so no result is claimed for it here, and the host check has not yet been seen to accept a bare-metal Linux host in a real run. Off bare-metal Linux its refusal is exercised by the unit test of the check, not by an e2e run.
  • The guard is OS-neutral. It runs in the windows-debug job's cargo test --workspace --all-targets on every pull request that changes Rust or a .feature file, and on the Linux Test (affected crates) job when the change selects e2e-cucumber.

Verification

  • cargo test -p e2e-cucumber --test platform_tags --test feature_naming --test driver_host_guard: 13, 4 and 3 tests pass. The Feature:/Rule: check is tested in both directions: every tag ScenarioDecl reads is reported there, and @serial, @retry, @retry(2) and a plain filter tag are not.
  • Mutation testing: a single-branch mutation of each guard check turns a test red. Examples are dropping requires_wsl from the OS decision, counting an @id as duplicated only past two uses, and letting the host check accept WSL2. The one surviving mutation is passing constant host facts inside this_host_is_bare_metal_linux instead of the probed ones, which only a run of the real binary on a non-bare-metal-Linux host would catch.
  • Three capability-tagged scenarios landed on main after this branch was cut, and the guard flagged each when the branch was rebased. Each was checked to run with real assertions on the native Windows GPU lane before being allowlisted: all its steps passed there, and none of them has an early return. serve-small-assistant-short-name and runtime-resolve-earlier-release were checked in merge-queue run 43277547876's e2e-gpu-strix-windows-report. serve-missing-gguf-variant-refused (serve-56) was checked in merge-queue run 43411859952's e2e-gpu-strix-windows-report, where all five of its steps passed. Its feature-file comment says it carries no OS tag on purpose, and its variant check is OS-neutral code that refuses before any server starts.
  • Run locally and passing: cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings (the harness target --all-targets skips), cargo fmt --all --check, cargo xtask check-architecture-doc, cargo xtask check-module-size, cargo test -p xtask hardware_testing_docs_cover_all_self_hosted_platforms, and the pre-commit hooks including lychee.
  • Not verified locally: cargo test --workspace --all-targets and python scripts/smoke_local.py. They fail on the local host only in code this PR does not touch, because of host setup: a noexec /tmp, a global commit-signing config picked up by a git-fixture test, podman's docker emulation, and bubblewrap network-namespace limits. CI is authoritative for those. The licence-header hook was not run locally because its tool is not installed; CI runs it.
  • Not tested: that the KFD step actually calls the host check. The BareMetalLinux parameter is the enforcement, since the KFD read cannot be called without the token, and removing the parameter is a signature change visible in review.

No Gherkin scenario is added because the CLI's behaviour is unchanged. This is test plumbing, scenario tags and one step's precondition, plus documentation.

Not fixed here

Instances of the same class, left as follow-ups:

  • examine-detects-gpu-and-driver (examine-04) stays allowlisted for its gfx-target assertion, but its KFD target cross-check returns early wherever it cannot read a usable KFD target, which includes native Windows and WSL2; the doc comment on assert_gpu_target_matches_kfd lists the exact cases. Its driver step also passes whenever examine prints a driver_status: line, which it does on every host, so it asserts no driver state on any OS.
  • Unix-only tests of behaviour that also matters on Windows: restoring_a_marker_replaces_the_file_rather_than_rewriting_it_in_place (apps/rocm/src/runtimes_cmd.rs), replacing_a_record_swaps_the_file_rather_than_rewriting_it_in_place (apps/rocm/src/remote/session.rs), and the runtime_library_path_env PATH merge in crates/rocm-core/src/examine/framework.rs, whose tests are all #[cfg(unix)].
  • The Windows Ctrl-C / Ctrl-Break exit-code mapping in crates/rocm-dash-tui/src/app/terminal.rs has no test on the Windows job.
  • Some shared serve steps that serve-56 uses assert less than their names say. assert_serve_refused ("serving is refused before any engine starts") checks only for a non-zero exit. assert_no_quantization_suggested checks only that Q4_K_M and Q4_0 are absent. assert_no_service_recorded checks that the services/ directory is empty, but its services list --all check looks only for vllm-. serve-56's own variant-advice step is what actually pins its refusal.
  • Doc drift outside this change's claims: tests/e2e-cucumber/README.md still says the catalog rules WSL2 out of scope, and its tags-table rows for @requires-bare-metal and @requires-engine do not say that the tag leaves the OS undecided, nor list aim among the engines. The "windows-debug runs on every change" wording in xtask/src/prepush.rs and a .github/workflows/ci.yml comment is looser than the job's path filter.
  • Small review nits: the vLLM comment in model_serving.feature says the skip on native Windows now names the OS, but on a Windows host with no GPU the GPU skip reason fires first; one docs/testing.md paragraph line is not reflowed; and AGENTS.md §3's "A capability tag is not a platform tag" sentence can be read as pairing @requires-bare-metal with @requires-wsl.
  • The ALLOWED_ANY_OS reason for runtime-lemonade-backend-alignment-opt-out does not cover WSL2. Backend alignment runs only for the rocm llama.cpp backend (should_attempt_llamacpp_backend_alignment in engines/lemonade/src/backend_alignment.rs), and WSL2 falls back to Vulkan, so there the "packaged pin survives" step cannot fail and only the stderr announcement is checked. The scenario may need @requires-bare-metal.
  • The ALLOWED_ANY_OS reason for runtime-install-sdk-overwrite-requires-yes says the consent refusal runs "before any download", but on a cold host the launcher step can still fetch first (the scenario's own comment in runtime_setup.feature says so). The refusal runs before the SDK download and any change on disk.
  • The second commit's message says the guard fails on any tag on a Feature: or Rule: line. A later commit narrowed that to the tags ScenarioDecl reads, which is what the branch does, so a squash message should use this description's wording.
  • The WSL2 GPU lane ran without a detected GPU in the run for an earlier head of this branch (6133a60), whose tag changes are identical to the current head's: its GPU scenarios skipped as "requires an AMD GPU; none detected". That lane's green result therefore says nothing about GPU scenarios on WSL2 for this change; its no-GPU scenarios did run and pass there.
  • The allowlist reason for serve-absent-gpu-index-rejected overstates the check on native Windows when the adapter listing does not answer.
  • The expectation.rs doc comment says unit fixtures write tags with the leading @, while most fixtures write both shapes.
  • In docs/testing.md, the sentence about examine-17's KFD check sits just before "Do not run these scenarios on a non-Windows host" and reads as part of the Windows set.
  • The EVERY_DECL_TAG comment in platform_tags.rs says it lists "every tag from_tags reads", but the list is hand-written, so nothing ties it to the parser.
  • A comment above serve-28 in model_serving.feature misnumbers serve-29 and serve-31.
  • The // No-GPU refusal header in ALLOWED_ANY_OS now heads only serve-lemonade-preparation-recovery.
  • tests/e2e-cucumber/tests/feature_naming.rs scans .feature files line by line, not through the gherkin parser and the harness's file discovery, so its view of the suite can differ from what the harness runs. platform_tags.rs now matches the harness.
  • Tagging reasons and comments do not yet name the code symbol that makes them true, which AGENTS.md §3 asks of a comment that justifies a decision: the feature-file comments, most ALLOWED_ANY_OS reasons, and the requires_bare_metal doc.
  • docs/testing.md's AIM paragraph paraphrases the family rule in engine_available("aim") as "a Radeon or Strix Halo GPU", and says aim_serve_scenarios_stay_off_the_per_pr_lanes pins the tagging, although that test does not assert the OS tag.
  • The first sentence of docs/testing.md's "The Serve Reuse Exemption From The No-GPU Refusal" section, above the serve-28..32 list, states the refusal with no platform scope; only the paragraph after the list says it is Linux-only.
  • @requires-os: accepts any value, so a misspelt OS still counts as a platform decision for the guard.

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation).
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen.
  • Comments and doc lines this PR adds or changes that state how the code behaves were checked against that code, and each that is the reason for a decision names the file and the function, constant or test that makes it true (AGENTS.md §3).
  • If this PR changes behavior on only one platform, the description names that platform and the check that runs there and would fail without the change (AGENTS.md §3).
  • Any existing scenario this PR moves off the blocking mock e2e job (for example by a tag) is justified as a deletion (AGENTS.md §8).
  • Commits answering review feedback were diffed whole against the reviewed head and widen no documented user-facing option's meaning or authority (AGENTS.md §11).
  • Every commit message on the branch describes what its own commit does in words still true at the branch head (AGENTS.md §5).

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@rominf
rominf force-pushed the ROCMAI-438-test-platform-gated-scenarios branch 5 times, most recently from d5bb407 to f3230c5 Compare November 17, 2026 12:04
…nario (ROCMAI-438)

A capability tag such as @requires-gpu asks what the host has, not
which OS it runs, so the Windows GPU lane satisfies it. That is how
examine-lists-every-gpu-with-its-address-and-target ran on Windows and
passed without checking anything: its KFD step returns early when
/sys/class/kfd is absent. The scenario now carries @requires-os:linux
and @requires-bare-metal.

A new platform_tags.rs guard fails on any capability-tagged scenario
that has neither an OS-deciding tag nor a reviewed ALLOWED_ANY_OS
entry, and on allowlist entries that are stale or contradict the
scenario's tags. AGENTS.md section 3, the PR template and CONTRIBUTING
now state the rule: a platform-specific change names its target
platform and the check that runs there and would fail without it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…cenarios (ROCMAI-438)

The AIM and vLLM scenarios now carry @requires-os:linux, because the
CLI refuses both engines on native Windows. For vLLM this records the
premise without moving a lane, since the harness already counts vLLM
unavailable there. For AIM it does move one: the harness counted AIM
available wherever Docker runs, so a native Windows host with Docker
would have run serve-26/27 into that refusal. examine-10 drops
@requires-bare-metal, because the framework probe honours the choice
on WSL2 too.

The guard is rewritten as pure helpers with a test for each branch:
an exhaustive ScenarioDecl destructure, so a new tag must be
classified; discovery and Scenario Outline expansion done the way the
harness does them; and failures for a duplicate @id, any tag on a
Feature or Rule line, and an empty suite.

The KFD topology step now needs a BareMetalLinux token that only the
host check mints. Off bare-metal Linux it panics instead of returning
early and passing with nothing asserted.

Allowlist reasons now say what each scenario's steps actually assert,
and the docs state the rule once, in docs/testing.md.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ROCMAI-438)

main added serve-56 with a capability tag and, deliberately per its
comment, no OS tag, so the guard flagged it on rebase. Its variant
check is OS-neutral code that refuses before any server starts, and
the native Windows GPU lane ran all five of its steps with real
assertions in merge-queue run 43411859952.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…OCMAI-438)

The Feature/Rule-line check reported every tag found there, but
cucumber's own runner legitimately reads tags at that level: @serial,
@Retry and whatever a --tags filter matches. It now reports only tags
that ScenarioDecl::from_tags reads, which are the ones the harness never
sees off a scenario line. The predicate destructures ScenarioDecl
without `..`, so a new field fails to compile until it is counted. New
tests pin both directions: every tag in the shared ScenarioDecl tag
table is reported, and runner tags on Feature and Rule lines are not.

The bare-metal host check carries a targeted
cfg_attr(test, allow(dead_code)) instead of a keep-alive line in its
unit test, since only the driver_host_guard target includes the file
without calling a step.

assert_gpu_target_matches_kfd's doc comment is now the single source
for when the KFD cross-check returns early (no KFD node with a nonzero
gfx_target_version, as on native Windows and WSL2, or a revision of 10
or more). The examine-04 feature comment and its ALLOWED_ANY_OS reason
point to it instead of saying "off Linux", which WSL2 contradicts.

AGENTS.md, the e2e README and docs/testing.md name @requires-wsl as the
OS-deciding alternative to @requires-os, and docs/testing.md states the
windows-debug claim accurately: it runs
cargo test --workspace --all-targets when a change touches its heavy
paths, which every Rust and .feature change does. The model_serving
comment says the vLLM tag changes no outcome, only the skip reason.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
serve-13 and serve-29 were allowlisted as OS-neutral, but the no-GPU
refusal keys on the usable-GPU probe, which exists only on Linux (WSL2
included). On native Windows that probe is a stub answering unknown, so
the refusal never fires there and the allowlist reason was false.

Tag both @requires-os:linux and drop their allowlist entries; they keep
running on the Linux no-GPU mock lane, and the docs now name all four
Linux-tagged no-GPU scenarios.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…CMAI-438)

The no-GPU serve refusal relies on a usable-GPU probe that only Linux
(WSL2 included) implements, so README.md and the manual-testing recipe no
longer imply it fires on native Windows. docs/testing.md now names
serve-26/27's @requires-os:linux tag and states which nightly lanes run
them, which exclude them, and the Docker condition on the WSL2 lane.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-438-test-platform-gated-scenarios branch from 04bee67 to 6e13a7e Compare November 17, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant