Repository navigation
Conversation
Contributor
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
rominf
force-pushed
the
ROCMAI-438-test-platform-gated-scenarios
branch
5 times, most recently
from
November 17, 2026 12:04
d5bb407 to
f3230c5
Compare
…nario (ROCMAI-438) A capability tag such as @requires-gpu asks what the host has, not which OS it runs, so the Windows GPU lane satisfies it. That is how examine-lists-every-gpu-with-its-address-and-target ran on Windows and passed without checking anything: its KFD step returns early when /sys/class/kfd is absent. The scenario now carries @requires-os:linux and @requires-bare-metal. A new platform_tags.rs guard fails on any capability-tagged scenario that has neither an OS-deciding tag nor a reviewed ALLOWED_ANY_OS entry, and on allowlist entries that are stale or contradict the scenario's tags. AGENTS.md section 3, the PR template and CONTRIBUTING now state the rule: a platform-specific change names its target platform and the check that runs there and would fail without it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…cenarios (ROCMAI-438) The AIM and vLLM scenarios now carry @requires-os:linux, because the CLI refuses both engines on native Windows. For vLLM this records the premise without moving a lane, since the harness already counts vLLM unavailable there. For AIM it does move one: the harness counted AIM available wherever Docker runs, so a native Windows host with Docker would have run serve-26/27 into that refusal. examine-10 drops @requires-bare-metal, because the framework probe honours the choice on WSL2 too. The guard is rewritten as pure helpers with a test for each branch: an exhaustive ScenarioDecl destructure, so a new tag must be classified; discovery and Scenario Outline expansion done the way the harness does them; and failures for a duplicate @id, any tag on a Feature or Rule line, and an empty suite. The KFD topology step now needs a BareMetalLinux token that only the host check mints. Off bare-metal Linux it panics instead of returning early and passing with nothing asserted. Allowlist reasons now say what each scenario's steps actually assert, and the docs state the rule once, in docs/testing.md. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ROCMAI-438) main added serve-56 with a capability tag and, deliberately per its comment, no OS tag, so the guard flagged it on rebase. Its variant check is OS-neutral code that refuses before any server starts, and the native Windows GPU lane ran all five of its steps with real assertions in merge-queue run 43411859952. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…OCMAI-438) The Feature/Rule-line check reported every tag found there, but cucumber's own runner legitimately reads tags at that level: @serial, @Retry and whatever a --tags filter matches. It now reports only tags that ScenarioDecl::from_tags reads, which are the ones the harness never sees off a scenario line. The predicate destructures ScenarioDecl without `..`, so a new field fails to compile until it is counted. New tests pin both directions: every tag in the shared ScenarioDecl tag table is reported, and runner tags on Feature and Rule lines are not. The bare-metal host check carries a targeted cfg_attr(test, allow(dead_code)) instead of a keep-alive line in its unit test, since only the driver_host_guard target includes the file without calling a step. assert_gpu_target_matches_kfd's doc comment is now the single source for when the KFD cross-check returns early (no KFD node with a nonzero gfx_target_version, as on native Windows and WSL2, or a revision of 10 or more). The examine-04 feature comment and its ALLOWED_ANY_OS reason point to it instead of saying "off Linux", which WSL2 contradicts. AGENTS.md, the e2e README and docs/testing.md name @requires-wsl as the OS-deciding alternative to @requires-os, and docs/testing.md states the windows-debug claim accurately: it runs cargo test --workspace --all-targets when a change touches its heavy paths, which every Rust and .feature change does. The model_serving comment says the vLLM tag changes no outcome, only the skip reason. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
serve-13 and serve-29 were allowlisted as OS-neutral, but the no-GPU refusal keys on the usable-GPU probe, which exists only on Linux (WSL2 included). On native Windows that probe is a stub answering unknown, so the refusal never fires there and the allowlist reason was false. Tag both @requires-os:linux and drop their allowlist entries; they keep running on the Linux no-GPU mock lane, and the docs now name all four Linux-tagged no-GPU scenarios. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…CMAI-438) The no-GPU serve refusal relies on a usable-GPU probe that only Linux (WSL2 included) implements, so README.md and the manual-testing recipe no longer imply it fires on native Windows. docs/testing.md now names serve-26/27's @requires-os:linux tag and states which nightly lanes run them, which exclude them, and the Docker condition on the WSL2 lane. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf
force-pushed
the
ROCMAI-438-test-platform-gated-scenarios
branch
from
November 17, 2026 14:18
04bee67 to
6e13a7e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A capability tag such as
@requires-gpuasks what hardware the host has, not which OS it runs, so the native Windows GPU lane satisfies it.examine-lists-every-gpu-with-its-address-and-target(examine-17) carried only@requires-gpu, so it ran on the Windows GPU lane and passed with every step green while checking nothing: its step reads the KFD topology and returns early when/sys/class/kfdis absent, which it always is on native Windows (and on WSL2, which exposes no KFD nodes).Platform gating lives in three independently maintained places:
cfgin source, scenario tags, and lane configuration. Nothing tied the scenario side to the platform, so a scenario whose premise exists on one OS only could quietly run, and pass, everywhere a capability tag lets it. Applying the rule across the suite also showed that the AIM and vLLM scenarios, which the CLI refuses on native Windows, recorded that premise nowhere in their tags.What changes
tests/e2e-cucumber/tests/platform_tags.rsreads the.featurefiles the harness reads by default (recursively, any extension case), expands Scenario Outlines the way cucumber does, and parses each scenario's tags with the harness's ownScenarioDecl::from_tags. Classification destructuresScenarioDeclwithout.., so a new tag field fails to compile until it is classified as a capability, an OS decision, or neither. The guard fails on a capability-tagged scenario with neither@requires-os:<os>nor@requires-wslthat is not inALLOWED_ANY_OS(including one with no@id), on two scenarios sharing an@id, on a tag thatScenarioDeclreads placed on aFeature:orRule:line (the harness builds eachScenarioDeclfrom the scenario's own tags only, so it never sees those; cucumber's own@serial,@retryand--tagstags there are legitimate and not reported), on finding no.featurefile or no scenario, and on an allowlist entry that is duplicated, has an empty reason, names no scenario, names one with no capability tag, or names one whose tags already decide its OS. Each check is a pure helper with its own test against fixtures.ALLOWED_ANY_OSholds 39 entries, each an(id, reason)pair whose reason says what the scenario's steps assert and why that holds on every OS a lane runs it on.@requires-os:linux @requires-bare-metal. The KFD read now takes aBareMetalLinuxtoken that only the host checkthis_host_is_bare_metal_linuxconstructs, so reaching the step on any other host (an explicit filter that bypasses the tags, or tags dropped later) panics instead of returning early with nothing asserted. The check's accept/refuse logic is unit-tested on every host throughdriver_host_guard.rs.@requires-os:linux. That covers serve-26 and serve-27 (AIM), and the seven@requires-engine:vllmscenarios that lacked it: serve-05, serve-11, runtime-03, runtime-04, runtime-18, runtime-lifecycle-12 and therock-next-09. The eighth, serve-23 (serve-vllm-low-vram-oom-guidance), already carried it. WSL2 is not excluded, since the CLI there is a Linux build.@requires-os:linux.serve-no-gpu-fails-fastandserve-unrelated-live-service-still-fails-fastkeyed their refusal on the usable-GPU probe, which exists only on Linux (WSL2 included); on native Windows it is a stub answering "unknown", which does not block a launch, so the refusal never fires there. They join serve-28, serve-30 and serve-32, which already carried the tag, anddocs/testing.mdnow says all four no-GPU refusal scenarios do.@requires-bare-metal. The framework probe honours--framework skipon every platform, WSL2 included, so it now carries no capability tag and needs no allowlist entry.docs/testing.mdgains "Capability-tagged scenarios name their OS": the rule, everything the guard fails on, what it leaves to review (whether a Linux scenario also needs@requires-bare-metal), the run command and where it runs in CI. AGENTS.md §3's "Platform-gated code gets a test that runs on its gate" becomes "Platform-conditional behavior is verified on the platform it targets": it keeps the previous content and adds that a platform-specific change names its target platform and the check that runs there and would fail without it, and that a test returning early on the target is not that check. A short "A capability tag is not a platform tag" paragraph points at the testing doc, and §13 item 14 follows. The PR template gains one checklist item, CONTRIBUTING's "What reviewers look for" one bullet, and the@requires-osrow of the e2e suite README's tags table a pointer to the testing doc.docs/architecture.mdhas an entry for the new file and updated entries for the host-guard files. Two doc comments insrc/expectation.rsare corrected: the parser yields tags without the leading@, and the comment now lists whatexamineskips on WSL2. diagnose-28's comment now names the scenario that covers its WSL2 contract.docs/manual-testing.md's recipe for it now say the refusal applies on Linux, WSL2 included: on native Windowsprobe_usable_amd_gpu_indicesanswers unknown rather than "no GPU", soservedoes not refuse, and the manual recipe's native Windows variant, which could never trigger the refusal, is removed.docs/testing.md's AIM paragraph now names serve-26/27's@requires-os:linuxtag, the nightly lanes that run them, that the Windows nightly lane is excluded by the tag, and that on the WSL2 nightly lane they run only ifdocker infosucceeds in its distro, which that workflow does not set up.An allowlist was chosen over a no-op tag such as
@any-os: each decision carries a written reason a reviewer can check, and it avoids churning dozens of tag lines in.featurefiles that many open PRs also touch.The Windows lint gate for
cfg(windows)code (warnings-as-errors) is unchanged: thewindows-debugjob already runscargo clippy --workspace --all-targets -- -D warningsonwindows-latest.Platform
Lane effects of the tag changes:
@nightly)E2E tests (Strix Halo, Windows, incl. nightly-only)lane, whose host reports AIM available, where the CLI refuses AIM@requires-no-gpu)@requires-no-gpu@requires-os:linux; still run on the Linux no-GPU mocke2ejob, so no lane loses coverageNo scenario moves off the blocking mock
e2ejob: every scenario that gains a tag here either already carries@requires-gpu, which resolves to skip on that job's GPU-less host, or gains only@requires-os:linux, which that Ubuntu-hosted job satisfies (serve-13 and serve-29); examine-10 only loses one.E2E tests (Strix Halo, Windows, current)lane. In run 43451380081, which tested dee0e6f (the only later tag change pins serve-13 and serve-29 to Linux, and that GPU host skips both as@requires-no-gpueither way), that job'se2e-gpu-strix-windows-reportartifact hasplatform.jsonresolveexamine-lists-every-gpu-with-its-address-and-targettoskipwith reason "requires os 'linux'; this host is 'windows'", and the scenario is absent fromreport.json. The sameplatform.jsonshows the host reporting AIM as available and serve-26/27 resolving toaim, skipped only as nightly-only, which is the nightly lane's case in the table.serve-missing-gguf-variant-refused(serve-56) andexamine-can-skip-framework-probing(examine-10) pass there.E2E tests (Strix Halo, WSL2, current)job, examine-10 resolves topassand passes.E2E tests (MI300X, current). Not verified on hardware: theE2E tests (MI300X, current)run on the current head is pending, so no result is claimed for it here, and the host check has not yet been seen to accept a bare-metal Linux host in a real run. Off bare-metal Linux its refusal is exercised by the unit test of the check, not by an e2e run.windows-debugjob'scargo test --workspace --all-targetson every pull request that changes Rust or a.featurefile, and on the LinuxTest (affected crates)job when the change selectse2e-cucumber.Verification
cargo test -p e2e-cucumber --test platform_tags --test feature_naming --test driver_host_guard: 13, 4 and 3 tests pass. TheFeature:/Rule:check is tested in both directions: every tagScenarioDeclreads is reported there, and@serial,@retry,@retry(2)and a plain filter tag are not.requires_wslfrom the OS decision, counting an@idas duplicated only past two uses, and letting the host check accept WSL2. The one surviving mutation is passing constant host facts insidethis_host_is_bare_metal_linuxinstead of the probed ones, which only a run of the real binary on a non-bare-metal-Linux host would catch.mainafter this branch was cut, and the guard flagged each when the branch was rebased. Each was checked to run with real assertions on the native Windows GPU lane before being allowlisted: all its steps passed there, and none of them has an early return.serve-small-assistant-short-nameandruntime-resolve-earlier-releasewere checked in merge-queue run 43277547876'se2e-gpu-strix-windows-report.serve-missing-gguf-variant-refused(serve-56) was checked in merge-queue run 43411859952'se2e-gpu-strix-windows-report, where all five of its steps passed. Its feature-file comment says it carries no OS tag on purpose, and its variant check is OS-neutral code that refuses before any server starts.cargo clippy --workspace --all-targets -- -D warnings,cargo clippy -p e2e-cucumber --test e2e -- -D warnings(the harness target--all-targetsskips),cargo fmt --all --check,cargo xtask check-architecture-doc,cargo xtask check-module-size,cargo test -p xtask hardware_testing_docs_cover_all_self_hosted_platforms, and the pre-commit hooks includinglychee.cargo test --workspace --all-targetsandpython scripts/smoke_local.py. They fail on the local host only in code this PR does not touch, because of host setup: anoexec/tmp, a global commit-signing config picked up by a git-fixture test, podman's docker emulation, and bubblewrap network-namespace limits. CI is authoritative for those. The licence-header hook was not run locally because its tool is not installed; CI runs it.BareMetalLinuxparameter is the enforcement, since the KFD read cannot be called without the token, and removing the parameter is a signature change visible in review.No Gherkin scenario is added because the CLI's behaviour is unchanged. This is test plumbing, scenario tags and one step's precondition, plus documentation.
Not fixed here
Instances of the same class, left as follow-ups:
examine-detects-gpu-and-driver(examine-04) stays allowlisted for its gfx-target assertion, but its KFD target cross-check returns early wherever it cannot read a usable KFD target, which includes native Windows and WSL2; the doc comment onassert_gpu_target_matches_kfdlists the exact cases. Its driver step also passes wheneverexamineprints adriver_status:line, which it does on every host, so it asserts no driver state on any OS.restoring_a_marker_replaces_the_file_rather_than_rewriting_it_in_place(apps/rocm/src/runtimes_cmd.rs),replacing_a_record_swaps_the_file_rather_than_rewriting_it_in_place(apps/rocm/src/remote/session.rs), and theruntime_library_path_envPATH merge incrates/rocm-core/src/examine/framework.rs, whose tests are all#[cfg(unix)].crates/rocm-dash-tui/src/app/terminal.rshas no test on the Windows job.assert_serve_refused("serving is refused before any engine starts") checks only for a non-zero exit.assert_no_quantization_suggestedchecks only thatQ4_K_MandQ4_0are absent.assert_no_service_recordedchecks that theservices/directory is empty, but itsservices list --allcheck looks only forvllm-. serve-56's own variant-advice step is what actually pins its refusal.tests/e2e-cucumber/README.mdstill says the catalog rules WSL2 out of scope, and its tags-table rows for@requires-bare-metaland@requires-enginedo not say that the tag leaves the OS undecided, nor listaimamong the engines. The "windows-debugruns on every change" wording inxtask/src/prepush.rsand a.github/workflows/ci.ymlcomment is looser than the job's path filter.model_serving.featuresays the skip on native Windows now names the OS, but on a Windows host with no GPU the GPU skip reason fires first; onedocs/testing.mdparagraph line is not reflowed; and AGENTS.md §3's "A capability tag is not a platform tag" sentence can be read as pairing@requires-bare-metalwith@requires-wsl.ALLOWED_ANY_OSreason forruntime-lemonade-backend-alignment-opt-outdoes not cover WSL2. Backend alignment runs only for therocmllama.cpp backend (should_attempt_llamacpp_backend_alignmentinengines/lemonade/src/backend_alignment.rs), and WSL2 falls back to Vulkan, so there the "packaged pin survives" step cannot fail and only the stderr announcement is checked. The scenario may need@requires-bare-metal.ALLOWED_ANY_OSreason forruntime-install-sdk-overwrite-requires-yessays the consent refusal runs "before any download", but on a cold host the launcher step can still fetch first (the scenario's own comment inruntime_setup.featuresays so). The refusal runs before the SDK download and any change on disk.Feature:orRule:line. A later commit narrowed that to the tagsScenarioDeclreads, which is what the branch does, so a squash message should use this description's wording.serve-absent-gpu-index-rejectedoverstates the check on native Windows when the adapter listing does not answer.expectation.rsdoc comment says unit fixtures write tags with the leading@, while most fixtures write both shapes.docs/testing.md, the sentence about examine-17's KFD check sits just before "Do not run these scenarios on a non-Windows host" and reads as part of the Windows set.EVERY_DECL_TAGcomment inplatform_tags.rssays it lists "every tagfrom_tagsreads", but the list is hand-written, so nothing ties it to the parser.model_serving.featuremisnumbers serve-29 and serve-31.// No-GPU refusalheader inALLOWED_ANY_OSnow heads onlyserve-lemonade-preparation-recovery.tests/e2e-cucumber/tests/feature_naming.rsscans.featurefiles line by line, not through the gherkin parser and the harness's file discovery, so its view of the suite can differ from what the harness runs.platform_tags.rsnow matches the harness.ALLOWED_ANY_OSreasons, and therequires_bare_metaldoc.docs/testing.md's AIM paragraph paraphrases the family rule inengine_available("aim")as "a Radeon or Strix Halo GPU", and saysaim_serve_scenarios_stay_off_the_per_pr_lanespins the tagging, although that test does not assert the OS tag.docs/testing.md's "The Serve Reuse Exemption From The No-GPU Refusal" section, above the serve-28..32 list, states the refusal with no platform scope; only the paragraph after the list says it is Linux-only.@requires-os:accepts any value, so a misspelt OS still counts as a platform decision for the guard.Checklist
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows.docs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation)..rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry (cargo xtask check-architecture-docfails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.e2ejob (for example by a tag) is justified as a deletion (AGENTS.md §8).