diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 0e065d02e..ad70adaca 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -9,6 +9,7 @@ SPDX-License-Identifier: MIT - [ ] If this PR adds, renames or removes a production source file (or a `.rs` file under `xtask/src` or `tests/e2e-cucumber`), `docs/architecture.md` has the matching entry (`cargo xtask check-architecture-doc` fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. - [ ] Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will *not* happen. - [ ] Comments and doc lines this PR adds or changes that state how the code behaves were checked against that code, and each that is the reason for a decision names the file and the function, constant or test that makes it true (AGENTS.md §3). +- [ ] If this PR changes behavior on only one platform, the description names that platform and the check that runs there and would fail without the change (AGENTS.md §3). - [ ] Any existing scenario this PR moves off the blocking mock `e2e` job (for example by a tag) is justified as a deletion (AGENTS.md §8). - [ ] Commits answering review feedback were diffed whole against the reviewed head and widen no documented user-facing option's meaning or authority (AGENTS.md §11). - [ ] Every commit message on the branch describes what its own commit does in words still true at the branch head (AGENTS.md §5). diff --git a/AGENTS.md b/AGENTS.md index e340022c7..468a2534a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -166,12 +166,23 @@ data. Fixing only the state a reviewer named invites the next round to name its when a finding is one instance of a class, fix the class and list the sibling cases in the reply. -**Platform-gated code gets a test that runs on its gate.** When a `cfg(windows)` path -cannot be driven by e2e, put its logic behind a pure helper or an injectable function -(a script runner, a clock) so a unit test runs on the Windows CI job, including the error -and timeout paths. A value matched across files (a version tag, a sentinel string) gets -one shared constant and a test tying the producer to the consumer. If a path stays -untested, say which and why in the PR text. +**Platform-conditional behavior is verified on the platform it targets.** A change that +takes effect on one platform only — a `cfg(windows)` or `cfg(unix)` branch, a read of an +interface one OS alone has (sysfs, a Win32 call), a scenario whose premise is +OS-specific — names in the PR text that platform and the check that runs there and would +fail without the change. A test that runs only elsewhere is not that check, and neither +is one that runs on the target but returns early when its premise is missing. For a +`cfg(windows)` path e2e cannot drive, put its logic behind a pure helper or an injectable +function (a script runner, a clock) so a unit test runs on the Windows CI job, including +the error and timeout paths. A value matched across files (a version tag, a sentinel +string) gets one shared constant and a test tying the producer to the consumer. If a path +stays untested, say which and why in the PR text. + +**A capability tag is not a platform tag.** A scenario whose premise exists on one OS only +carries `@requires-os:` (or `@requires-wsl` where the premise is WSL2 itself), plus +`@requires-bare-metal` where WSL2 lacks the interface, even when `@requires-gpu` or another +capability tag already gates it; "Capability-tagged scenarios name their OS" in +[docs/testing.md](docs/testing.md) says why and what checks it. ## 4) Live State Verification Before Any External Claim @@ -562,7 +573,7 @@ Run this checklist before any upstream push, PR update, issue update, or public 11. User approval obtained for any high-blast-radius external action — *§1* 12. Force-push's live review/approval state checked before proposing or executing one — *§1* 13. Review comments checked for unreplied threads before resuming review-feedback work — *§11* -14. New tests shown to fail when their target is broken; state matrix written for new state or placeholders; platform-gated logic covered by a test that runs on its gate — *§3* +14. New tests shown to fail when their target is broken; state matrix written for new state or placeholders; platform-conditional behavior verified by a check that runs on its target platform — *§3* 15. Old claims grepped across docs and PR body; PR body matches the final diff — *§5* 16. Subprocess/FFI checklist applied to any new process or driver call — *§6* 17. Comments and doc lines this change adds or edits that state how the code behaves checked against that code; each that is the reason for a decision names where the code makes it true — *§3* diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f0e141c65..4f85089cd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -138,6 +138,7 @@ Pull requests are normally squash-merged, and by default the squash commit's mes ### What reviewers look for - Tests cover the new behavior +- Platform-specific changes name the platform they target and a check that runs there and would fail without them - No secrets, credentials, or internal hostnames in committed files - Third-party dependencies recorded in `Cargo.lock` - License headers on new source files (see `licenserc.toml`) diff --git a/README.md b/README.md index 80116ff60..73736c834 100644 --- a/README.md +++ b/README.md @@ -1518,7 +1518,8 @@ when `ROCR_VISIBLE_DEVICES` is set, because the two orderings can diverge. #### Reuse is exempt from the no-GPU refusal -Under the default `gpu_required` policy, `rocm serve` on a host with no usable +Under the default `gpu_required` policy, on Linux (including WSL2), where the +CLI can probe which GPUs are usable, `rocm serve` on a host with no usable AMD GPU fails fast with `no usable AMD GPU detected`, before any engine is prepared or downloaded. A `rocm serve` that matches an **already-running managed service for the same engine and model** now skips that refusal and diff --git a/docs/architecture.md b/docs/architecture.md index 41e9758cd..17ec6974c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -844,9 +844,10 @@ Gherkin `.feature` files under `features/` backed by Rust step functions that dr | File | Responsibility | Class | | --- | --- | --- | -| [`tests/driver_host_guard.rs`](../tests/e2e-cucumber/tests/driver_host_guard.rs) | Ordinary test that the Windows driver-install steps run the host guard before calling the CLI, checked against the step source. | flat | +| [`tests/driver_host_guard.rs`](../tests/e2e-cucumber/tests/driver_host_guard.rs) | Ordinary tests of the host checks in `tests/e2e/host_guard.rs`: that the Windows driver-install steps run theirs before calling the CLI (checked against the step source), that it refuses Linux and WSL, and, through the test inside `host_guard.rs`, that the KFD topology step's check refuses any host but bare-metal Linux. | flat | | [`tests/e2e.rs`](../tests/e2e-cucumber/tests/e2e.rs) | The cucumber test binary: the `E2eWorld` state and its cleanup on drop, the `run_rocm` helpers (including `run_rocm_with_scenario_env_off_executor`, which waits for a long CLI run on a blocking thread so it does not stall concurrent scenarios), the runner `main`, and reconciliation of results with expectations. | flat | | [`tests/feature_naming.rs`](../tests/e2e-cucumber/tests/feature_naming.rs) | Drift guard, in the ordinary `cargo test` set, for the `-` scenario names and ids in the `.feature` files. | flat | +| [`tests/platform_tags.rs`](../tests/e2e-cucumber/tests/platform_tags.rs) | Drift guard, in the ordinary `cargo test` set, that every capability-tagged scenario also carries an OS-deciding tag or a reviewed `ALLOWED_ANY_OS` entry; [testing.md](testing.md) lists everything it fails on. | flat | | [`tests/skill_reference.rs`](../tests/e2e-cucumber/tests/skill_reference.rs) | Drift guard, in the ordinary `cargo test` set, that the catalog table in [`reference.md`](../skills/rocm-doctor/reference.md) uses the vocabulary `rocm fix` prints. | flat | #### `tests/e2e/` @@ -866,7 +867,7 @@ Gherkin `.feature` files under `features/` backed by Rust step functions that dr | [`tests/e2e/driver_steps.rs`](../tests/e2e-cucumber/tests/e2e/driver_steps.rs) | Steps for [`driver_install.feature`](../tests/e2e-cucumber/features/driver_install.feature): previewing and approving GPU driver installation plans on WSL, Linux and Windows hosts. | flat | | [`tests/e2e/engines_steps.rs`](../tests/e2e-cucumber/tests/e2e/engines_steps.rs) | Steps for [`engine_shell.feature`](../tests/e2e-cucumber/features/engine_shell.feature): `rocm engines shell` under a pseudo-terminal, since the defect shows only in what the terminal renders; and the vLLM steps of `serve-62` in [`model_serving.feature`](../tests/e2e-cucumber/features/model_serving.feature), which run `rocm engines list` against a stand-in interpreter. | flat | | [`tests/e2e/examine_steps.rs`](../tests/e2e-cucumber/tests/e2e/examine_steps.rs) | Steps for [`examine.feature`](../tests/e2e-cucumber/features/examine.feature): `rocm examine`, the active runtime's provenance, version and help output, engine listing and the driver plan preview. | flat | -| [`tests/e2e/host_guard.rs`](../tests/e2e-cucumber/tests/e2e/host_guard.rs) | The host check the Windows driver-install steps run before any CLI call, kept free of the cucumber world so a separate test can include it. | flat | +| [`tests/e2e/host_guard.rs`](../tests/e2e-cucumber/tests/e2e/host_guard.rs) | The host checks steps run before relying on an OS-specific premise (the Windows driver-install steps, the KFD topology step, which cannot read sysfs without the token its check returns), kept free of the cucumber world so a separate test can include it. | flat | | [`tests/e2e/lemonade_install_steps.rs`](../tests/e2e-cucumber/tests/e2e/lemonade_install_steps.rs) | Steps for [`lemonade_installs.feature`](../tests/e2e-cucumber/features/lemonade_installs.feature): what the user is told about the Lemonade copies on disk, from planted trees and install records. | flat | | [`tests/e2e/lifecycle_steps.rs`](../tests/e2e-cucumber/tests/e2e/lifecycle_steps.rs) | Steps for [`install_lifecycle.feature`](../tests/e2e-cucumber/features/install_lifecycle.feature): `xtask package`, the root installer and the installed binaries, from install through uninstall. | flat | | [`tests/e2e/logs_steps.rs`](../tests/e2e-cucumber/tests/e2e/logs_steps.rs) | Steps for [`logs.feature`](../tests/e2e-cucumber/features/logs.feature): `rocm logs` searches over command-log files planted inside the tail window it reads. | flat | diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 6559430f4..c1473ef1f 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -1400,10 +1400,9 @@ Expected result: **Linux or WSL only.** Unlike the rest of this file, the commands below are bash, not PowerShell, and they pin `--engine vllm`, which is unsupported on -native Windows. The behaviour itself is not platform-specific; only this recipe -for reproducing it is. On native Windows, substitute `--engine lemonade` -throughout and set the mask with `$env:HIP_VISIBLE_DEVICES = ""` (and clear it -with `Remove-Item Env:HIP_VISIBLE_DEVICES` afterwards). +native Windows. The refusal itself is Linux-only too: on native Windows +`probe_usable_amd_gpu_indices` answers unknown rather than "no GPU", so `serve` +does not refuse there and this section has nothing to check. This is a change to a refusal you may be relying on, so check both halves. Mask every device to stand in for a host with no usable AMD GPU — the same condition @@ -1959,7 +1958,7 @@ Expected result: The automated scenario is the `@nightly` `serve-aim-key-guards-every-route` in [model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature), -which runs only on a nightly host with Docker and a Radeon or Strix Halo GPU. +which runs only on a Linux nightly host with Docker and a Radeon or Strix Halo GPU. ## 12. Background Automation Helper diff --git a/docs/testing.md b/docs/testing.md index 98c683862..957f341c9 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -248,14 +248,67 @@ stops being needed, or whose file moves. A new field reuses the helper; a new exception is a reviewed entry in that list. Where it runs: the pre-push hook (`cargo xtask pre-push-check`) selects `xtask` -for any Rust change, the `windows-debug` job runs `cargo test --workspace` on -every pull request that changes Rust, and the merge queue and post-merge runs -test the whole workspace. The Linux pull-request lane (`cargo xtask affected`) -selects `xtask` only for a change in `xtask` or in a crate it depends on, or for -a file it reads, so a pull request that touches nothing but +for any Rust change, the `windows-debug` job runs +`cargo test --workspace --all-targets` on every pull request that changes Rust, +and the merge queue and post-merge runs test the whole workspace. The Linux +pull-request lane (`cargo xtask affected`) selects `xtask` only for a change in +`xtask` or in a crate it depends on, or for a file it reads, so a pull request +that touches nothing but `crates/rocm-dash-tui` does not run this test on Linux. The module-import contract above has the same property. +### Capability-tagged scenarios name their OS + +A capability tag (`@requires-gpu`, `@requires-engine:`, `@requires-docker` +and the rest) asks what the host has, not which OS it runs, so the Windows GPU +lane runs every `@requires-gpu` scenario. A scenario whose premise exists on +one OS only also carries `@requires-os:`, or `@requires-wsl` where the +premise is WSL2 itself. `@requires-os:linux` still admits WSL2, which reports +itself as Linux, so add `@requires-bare-metal` too when WSL2 lacks the +interface. examine-17 is the case that prompted this: its step reads the KFD +topology, found none on the Windows lane, and passed without asserting +anything. + +[platform_tags.rs](../tests/e2e-cucumber/tests/platform_tags.rs) reads the +`.feature` files the e2e harness reads by default (every `*.feature` under +`tests/e2e-cucumber/features/`, recursively, extension in any case), expands +Scenario Outlines the way cucumber does, and parses each scenario's tags with +the harness's own `ScenarioDecl::from_tags`. A capability tag there is any +`@requires-*` tag `ScenarioDecl` reads except `@requires-os:` and +`@requires-wsl`, the two that decide the OS. It fails on: + +- a capability-tagged scenario with neither OS-deciding tag whose `@id` is not + in its `ALLOWED_ANY_OS` list, including one with no `@id` at all +- two scenarios with the same `@id` +- a tag `ScenarioDecl` reads on a `Feature:` or `Rule:` line, which it never + sees there: the harness builds each scenario's `ScenarioDecl` from that + scenario's own tags. Other tags there are not reported, since cucumber's own + runner reads Feature and Rule tags for `@serial`, `@retry` and `--tags` +- finding no `.feature` file, or no scenario, which would leave every other + check passing over nothing +- an `ALLOWED_ANY_OS` entry that is listed twice, has an empty reason, names no + scenario, names one with no capability tag, or names one whose tags already + decide its OS + +Each entry's reason says why the scenario's steps hold on every OS a lane runs +it on. The guard checks only that the OS is decided: whether a Linux scenario +also needs `@requires-bare-metal` is left to review, and it pins that tag only +on examine-17. Run it on its own with: + +```bash +cargo test -p e2e-cucumber --test platform_tags +``` + +It reads tags, not steps, so whether an entry's reason is still true is a +review question: a step that starts reading something only one OS has turns +its entry wrong without failing anything here. Where it runs: the +`windows-debug` job runs `cargo test --workspace --all-targets` on every pull +request that changes Rust or a `.feature` file, both of which its `heavy` path +filter matches, so a tag-only edit is checked there. The Linux pull-request +lane runs it only when the change selects `e2e-cucumber` and passes that lane's +Rust path filter, which a `.feature` file alone does not, and the pre-push hook +does not run `e2e-cucumber`'s tests. + ### Tests that run a fake executable Plant a stand-in `python`, `uv`, `amd-smi`, ... with @@ -1182,8 +1235,8 @@ filter, as before, including a lockfile-only push. The hook is an early warning, and CI is not narrowed everywhere. On pull requests the `clippy` job lints the whole workspace and the `test` job (`Test (affected crates)`) runs only the crates -`cargo xtask affected` selects. The whole workspace's tests run in the `windows-debug` job on -every change, and in the merge queue and on `main`, where the Linux `test` job is forced to +`cargo xtask affected` selects. The whole workspace's tests run in the `windows-debug` job when +a change touches its `heavy` paths, as every Rust change does, and in the merge queue and on `main`, where the Linux `test` job is forced to `--workspace`. Known limits, unchanged by the narrowing: the steps check the working tree rather than the pushed @@ -1832,12 +1885,17 @@ AIM serve and restart canaries and the AIM endpoint-key check) are `@nightly` for cost: each pulls the AIM image and a 9B checkpoint on a cold runner and may wait up to 1200s for readiness, which on the Strix Ubuntu job, back when it ran on pull requests (about 25 minutes without -them, 45 minute cap) used up the whole budget. They still need `@requires-engine:aim`, -i.e. Docker reachable and a Radeon or Strix Halo GPU, so they run only where -`E2E_INCLUDE_NIGHTLY=1` is set on such a host: the nightly Strix Halo lane, or an -`include_nightly` dispatch of `e2e-selfhosted.yml` (whose Strix Ubuntu job then -gets a 125 minute cap). `aim_serve_scenarios_stay_off_the_per_pr_lanes` pins -the tagging and that they stay reachable when the opt-in is set. +them, 45 minute cap) used up the whole budget. They still need +`@requires-engine:aim`, i.e. Docker reachable and a Radeon or Strix Halo GPU, +and `@requires-os:linux` (the CLI refuses AIM on native Windows), so they run +only where `E2E_INCLUDE_NIGHTLY=1` is set on such a host, which includes the +nightly Strix Halo Ubuntu lane (`e2e-gpu-nightly-strix`) and the Strix Ubuntu +job of an `include_nightly` dispatch of `e2e-selfhosted.yml` (which then gets a +125 minute cap). The Windows nightly lane (`e2e-gpu-nightly-strix-windows`) is +excluded by the OS tag; on the WSL2 nightly lane (`e2e-wsl-nightly`) they run +only if `docker info` succeeds in its distro, which the workflow does not set +up. `aim_serve_scenarios_stay_off_the_per_pr_lanes` pins the tagging and that +they stay reachable when the opt-in is set. All three scenarios skip on a host without Docker; the `e2e-gpu-nightly-strix` job logs whether Docker is reachable ("Report whether Docker is reachable"). @@ -3668,8 +3726,10 @@ this harness's platform filter, so selecting a Windows scenario by name or tag on Linux runs it against Linux behavior instead of skipping it. The Windows install steps also require a native Windows build before any CLI invocation. [driver_host_guard.rs](../tests/e2e-cucumber/tests/driver_host_guard.rs) checks -that ordering without running an installer. Do not run these scenarios on a -non-Windows host. +that ordering without running an installer. The same test target also tests the +native-Windows check itself, and the check that the KFD topology step +(examine-17) runs to refuse any host but bare-metal Linux. +Do not run these scenarios on a non-Windows host. `serve-16` (`@id:serve-absent-gpu-index-rejected`, [model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature)) runs @@ -4796,8 +4856,9 @@ selects nothing and the run fails with `0 scenarios executed` (see replaces it. Those four carry `@requires-no-gpu` (their premise is a host with no usable AMD -GPU), so they get their live coverage on the GitHub-hosted mock lane; serve-28, -serve-30 and serve-32 also carry `@requires-os:linux`. +GPU), so they get their live coverage on the GitHub-hosted mock lane. All four +also carry `@requires-os:linux`: the usable-GPU probe the no-GPU refusal keys on +exists only on Linux, so on native Windows that refusal never fires. The fifth is the pre-gate's other half, and it is the one that needs real hardware: diff --git a/tests/e2e-cucumber/README.md b/tests/e2e-cucumber/README.md index c68381db7..0d8cc6a25 100644 --- a/tests/e2e-cucumber/README.md +++ b/tests/e2e-cucumber/README.md @@ -195,7 +195,7 @@ Scenarios carry stable-id and capability tags: | `@requires-dir-permissions` | Premise is a directory that refuses new files once its write bits are cleared. Resolves to **skip** for a user those bits do not bind: root, which is what the self-hosted WSL2 guest runs the suite as. The capability probe's `dir_permissions_bind` clears the write bits on a directory under the temp root and tries to create a file in it. `@requires-os:linux` cannot express this: root on Linux is still `linux`, and the scenario would pass without testing anything. Always **skip** on Windows, which has no such bits. | | `@requires-case-sensitive-fs` | Premise is two files whose names differ only in letter case. Resolves to **skip** where the scenarios' temp root folds case. The capability probe's `case_sensitive_fs` writes two such names in its own temp dir, which follows `$TMPDIR` like every scenario's isolated root, and checks they stayed two files. `@requires-os:linux` cannot express this: `$TMPDIR` on Linux, WSL2 included, can sit on a casefolded directory or a mounted Windows drive. | | `@requires-engine:` | Pins the serve engine. Resolves to skip where that engine can't start (e.g. vLLM on a lemonade-only Strix host). | -| `@requires-os:` | Premise is OS-specific; skip on other OSes. | +| `@requires-os:` | Premise is OS-specific; skip on other OSes. A capability tag such as `@requires-gpu` does not decide the OS, so an OS-specific scenario carries this too (or `@requires-wsl` where the premise is WSL2 itself): see "Capability-tagged scenarios name their OS" in [docs/testing.md](../../docs/testing.md). | | `@requires-oom-fault-injection` | Needs the `rocm` binary under test to carry the test-only `e2e-oom-fault-injection` hook, which fabricates a managed launch so `rocm serve`'s summary can be verified without a GPU: one that runs out of GPU memory (the OOM guidance), or, in its `ready` mode, one that becomes ready and then fails its smoke test. Unlike every other capability this one **cannot be probed** — no product command exposes it — so `xtask e2e` reports it via `ROCM_E2E_OOM_FAULT_INJECTION=1`, set only when xtask compiled the binary itself with the feature. A prebuilt `ROCM_CLI_BINARY` (the self-hosted lanes' shipping release build) has the hook compiled out, so these scenarios resolve to **skip** there and run on the mock lane. | | `@serve-timeout:` | Lengthen the serve-readiness wait for a genuinely slow serve (e.g. a large model). | | `@nightly` | Expensive scenario skipped by default; included when `E2E_INCLUDE_NIGHTLY=1`. | diff --git a/tests/e2e-cucumber/features/diagnose.feature b/tests/e2e-cucumber/features/diagnose.feature index b2f76c363..cdc1ab599 100644 --- a/tests/e2e-cucumber/features/diagnose.feature +++ b/tests/e2e-cucumber/features/diagnose.feature @@ -420,9 +420,9 @@ Feature: Diagnosing failures and listing fixes # a fix the CLI applies, so the report of a change that never happened looked # like success. # @requires-bare-metal because the entry under test is scoped to bare-metal - # Linux and Windows. On WSL it is refused at the platform gate instead, which - # is a different contract with its own scenario — and the right one, since the - # catalog does not claim this remedy applies there. + # Linux and Windows. On WSL it is declined as not applicable instead, which is + # the contract diagnose-11 covers (with another entry) — and the right one, + # since the catalog does not claim this remedy applies there. @id:diagnose-fix-needing-an-argument-says-so @requires-bare-metal Scenario: diagnose-28 - A fix that needs more information says what it needs and changes nothing Given a user who has chosen a fix that cannot run until it is told what to act on diff --git a/tests/e2e-cucumber/features/examine.feature b/tests/e2e-cucumber/features/examine.feature index 3426caa4f..497ea3a0b 100644 --- a/tests/e2e-cucumber/features/examine.feature +++ b/tests/e2e-cucumber/features/examine.feature @@ -24,8 +24,12 @@ Feature: GPU detection and system inspection # from the KFD topology in sysfs, while `examine` reaches its answer through the # CLI's own probe. Detection used to look for `gfx_target_version` as a standalone # file, which no kernel exposes, and silently fell back to decoding a GC IP - # version -- naming an MI300X `gfx943` instead of `gfx942`. Only the GPU lane can - # exercise this; there is no KFD topology to read on the mock lane. + # version -- naming an MI300X `gfx943` instead of `gfx942`. The cross-check + # returns early wherever it cannot read a usable KFD target, so on native + # Windows and WSL2 it checks nothing; `assert_gpu_target_matches_kfd` in + # examine_steps.rs lists the exact cases. The driver step passes on the + # `driver_status` line `examine` always prints, so where the cross-check + # returns early the scenario asserts only that `examine` names a gfx target. @id:examine-detects-gpu-and-driver @requires-gpu Scenario: examine-04 - System inspection detects the GPU and driver Given a machine with an AMD GPU @@ -101,15 +105,9 @@ Feature: GPU detection and system inspection # Leaving the frameworks out is the variant pinned here: the outcome is # identical on every host, whereas asserting that a *named* framework was - # probed would depend on what happens to be installed. - # - # @requires-bare-metal because the probe never reaches its framework step on - # WSL2 — it returns as soon as it recognises the platform — so `framework` - # stays "unknown" there whatever the flag says. That the probe gives up that - # early is its own defect, tracked separately; this scenario is about whether - # the choice is reachable, and it cannot answer that where no choice is acted - # on at all. - @id:examine-can-skip-framework-probing @requires-bare-metal + # probed would depend on what happens to be installed. The probe honours the + # choice on every platform, WSL2 included, so the scenario carries no tag. + @id:examine-can-skip-framework-probing Scenario: examine-10 - The user can leave the frameworks out of the inspection When the user inspects the system without probing frameworks Then the inspection reports that it skipped the frameworks @@ -220,11 +218,17 @@ Feature: GPU detection and system inspection # fill the field. Where the name does resolve a target, the kernel's target # replaces it (see examine-57). # - # The step no-ops where a premise does not hold -- no readable KFD topology, - # no `lspci` to supply PCI addresses, or a topology whose nodes disagree on a - # target -- because on such a host the answer it would otherwise flag is the - # correct one. - @id:examine-lists-every-gpu-with-its-address-and-target @requires-gpu + # The KFD topology exists only on bare-metal Linux -- native Windows has no + # sysfs and WSL2 exposes no KFD nodes -- so the scenario is tagged for that + # host. `@requires-gpu` alone let the Windows GPU lane run it, where the step + # found no topology and passed without asserting anything. + # + # The step still no-ops where a premise does not hold on Linux -- no readable + # KFD topology, no `lspci` to supply PCI addresses, or a topology whose nodes + # disagree on a target -- because on such a host the answer it would + # otherwise flag is the correct one. That early return covers a Linux host + # missing a premise; it is not a substitute for the OS tags. + @id:examine-lists-every-gpu-with-its-address-and-target @requires-gpu @requires-os:linux @requires-bare-metal Scenario: examine-17 - The machine-readable report lists every GPU the kernel sees Given a machine with an AMD GPU When the user inspects the system both for reading and for scripting diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 2370b2079..050f2eec5 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -45,7 +45,13 @@ Feature: Model serving # caught before merge, while the heavier `@merge-queue` serves # (`serve-default-engine-working-endpoint`, `serve-default-engine-inference`, # and `serve-readiness-contract`) run only in the merge queue (not on MI300X). - @id:serve-vllm-inference @requires-gpu @requires-engine:vllm + # Every `@requires-engine:vllm` scenario also carries `@requires-os:linux`, + # which records that its premise is Linux-only: the CLI refuses vLLM on native + # Windows (serve-33). The tag changes no outcome today, since the harness + # already counts vLLM unavailable on native Windows and skips these there; the + # skip there now names the OS rather than the engine. WSL2 is not excluded: + # the CLI there is a Linux build and serves vLLM. + @id:serve-vllm-inference @requires-gpu @requires-engine:vllm @requires-os:linux Scenario: serve-05 - A served model responds to inference requests on vLLM Given a managed runtime is active And a model is being served on GPU @@ -115,7 +121,7 @@ Feature: Model serving # only applies where vLLM is the effective engine — `@requires-engine:vllm` # skips it on lemonade-default hosts (Strix Halo), where asserting a vLLM # default would be a guaranteed false failure. - @id:serve-vllm-default-on-instinct @requires-gpu @requires-engine:vllm + @id:serve-vllm-default-on-instinct @requires-gpu @requires-engine:vllm @requires-os:linux Scenario: serve-11 - vLLM is the default serving engine on Instinct Given a managed runtime is active When the user serves a vLLM-capable model without specifying an engine @@ -136,8 +142,12 @@ Feature: Model serving # GPU-required enforcement (EAI-7400). Under the GPU-required default, a host # with no usable AMD GPU must refuse to serve — before any engine is prepared or # launched — with an actionable message, never a CPU or device-0 fallback. Runs - # on the no-GPU mock host, so it gates every PR (@requires-no-gpu, no GPU needed). - @id:serve-no-gpu-fails-fast @requires-no-gpu + # on the Linux no-GPU mock host, so it gates every PR there (@requires-no-gpu, no + # GPU needed). + # @requires-os:linux because the usable-GPU probe the refusal keys on exists only + # on Linux (WSL2 included); on native Windows it answers "unknown", so this + # refusal never fires there. + @id:serve-no-gpu-fails-fast @requires-no-gpu @requires-os:linux Scenario: serve-13 - Serving is refused on a host with no AMD GPU When the user serves a model under the GPU-required default Then serving is refused before any engine starts @@ -381,7 +391,13 @@ Feature: Model serving # ROCM_CLI_AIM_READY_TIMEOUT_SECS (see isolate_cmd) — a cold-start HF weight # download for this 9B model legitimately exceeds the engine's default # 5-minute readiness cap. - @id:serve-aim-inference @requires-gpu @requires-engine:aim @serve-timeout:1200 @nightly + # `@requires-os:linux` on both, and on serve-63: the CLI refuses AIM on native + # Windows, while the harness counts AIM as available wherever Docker runs and + # the GPU family fits, whatever the OS. Without the tag a native Windows host + # with Docker and such a GPU would run them and meet that refusal; with it + # they skip there. + # WSL2 is not excluded, since the CLI there is a Linux build. + @id:serve-aim-inference @requires-gpu @requires-engine:aim @requires-os:linux @serve-timeout:1200 @nightly Scenario: serve-26 - A model served on AIM responds to inference requests Given a managed runtime is active And a model is being served on aim @@ -396,7 +412,7 @@ Feature: Model serving # on (`stop_managed_services`, via the `stop_aim_service` step), and the # same serve+inference fixture `serve-26` uses, rather than adding a new # stop/restart mechanism just for AIM. - @id:serve-aim-lifecycle-parity @requires-gpu @requires-engine:aim @serve-timeout:1200 @nightly + @id:serve-aim-lifecycle-parity @requires-gpu @requires-engine:aim @requires-os:linux @serve-timeout:1200 @nightly Scenario: serve-27 - A stopped AIM service can be restarted and serves inference again Given a managed runtime is active And a model is being served on aim @@ -476,7 +492,10 @@ Feature: Model serving # the bail left all three refusal scenarios green. Planting a configured setup # venv that the registry has not yet recorded is what makes the write reachable, # and the last Then is what refuses it. - @id:serve-unrelated-live-service-still-fails-fast @requires-no-gpu + # + # @requires-os:linux for the same reason as serve-13: the usable-GPU probe the + # refusal keys on exists only on Linux, so on native Windows it never fires. + @id:serve-unrelated-live-service-still-fails-fast @requires-no-gpu @requires-os:linux Scenario: serve-29 - A live service for another model does not soften the no-GPU refusal Given a live managed Lemonade serve for an unrelated model And setup names an unregistered runtime folder @@ -997,7 +1016,7 @@ Feature: Model serving # Not `@merge-queue`: the merge queue runs no lane where vLLM is the engine # (the MI300X lane does not run on `merge_group`), so under that tag it # skipped everywhere. - @id:serve-vllm-key-guards-every-route @requires-gpu @requires-engine:vllm @nightly + @id:serve-vllm-key-guards-every-route @requires-gpu @requires-engine:vllm @requires-os:linux @nightly Scenario: serve-61 - A key-protected vLLM endpoint refuses keyless requests outside the OpenAI API Given a managed runtime is active And a model is being served on GPU behind an API key @@ -1031,7 +1050,7 @@ Feature: Model serving # AIM engine crate); this proves a real AIM image actually loads the # middleware. `@nightly` for the same reason as serve-26: it pulls the image # and a 9B checkpoint on a cold runner. - @id:serve-aim-key-guards-every-route @requires-gpu @requires-engine:aim @serve-timeout:1200 @nightly + @id:serve-aim-key-guards-every-route @requires-gpu @requires-engine:aim @requires-os:linux @serve-timeout:1200 @nightly Scenario: serve-63 - A key-protected AIM endpoint refuses keyless requests outside the OpenAI API Given a managed runtime is active And a model is being served on aim behind an API key diff --git a/tests/e2e-cucumber/features/runtime_lifecycle.feature b/tests/e2e-cucumber/features/runtime_lifecycle.feature index 19bfe62a5..471d4b735 100644 --- a/tests/e2e-cucumber/features/runtime_lifecycle.feature +++ b/tests/e2e-cucumber/features/runtime_lifecycle.feature @@ -161,7 +161,7 @@ Feature: Runtime lifecycle state machine # design and it is never stale — on a lemonade-default host the report would # correctly say `services_on_previous_runtime: 0` and asserting a restart there # would be a guaranteed false failure. Same reasoning as `serve-11`. - @id:runtime-lifecycle-activate-restart-services-succeeds @requires-gpu @requires-engine:vllm + @id:runtime-lifecycle-activate-restart-services-succeeds @requires-gpu @requires-engine:vllm @requires-os:linux Scenario: runtime-lifecycle-12 - Activating with --restart-services --yes stops and respawns the running server Given a managed runtime is active And a model is being served on GPU diff --git a/tests/e2e-cucumber/features/runtime_setup.feature b/tests/e2e-cucumber/features/runtime_setup.feature index 2114a7b35..226d71a1c 100644 --- a/tests/e2e-cucumber/features/runtime_setup.feature +++ b/tests/e2e-cucumber/features/runtime_setup.feature @@ -59,7 +59,7 @@ Feature: Runtime configuration # extensions in the runtime: a reinstall that leaves a torch and torchvision pair # unable to register their operators, or a torch the vLLM build does not match, # fails here instead of passing and breaking every serve. - @id:runtime-sdk-reinstall-keeps-engine-consistent @requires-gpu @requires-engine:vllm @nightly + @id:runtime-sdk-reinstall-keeps-engine-consistent @requires-gpu @requires-engine:vllm @requires-os:linux @nightly Scenario: runtime-03 - Reinstalling the SDK leaves the installed engine able to use the GPU Given a managed runtime with an inference engine already installed When the user installs the SDK again @@ -93,7 +93,7 @@ Feature: Runtime configuration # Same lane as scenario 4 and for the same reasons: a real SDK install and a # real engine, on the serialized nightly GPU runners. `@requires-engine:vllm` # because only vLLM shares the runtime environment the alignment writes into. - @id:runtime-torch-alignment-opt-out @requires-gpu @requires-engine:vllm @nightly + @id:runtime-torch-alignment-opt-out @requires-gpu @requires-engine:vllm @requires-os:linux @nightly Scenario: runtime-04 - Opting out of the torch alignment keeps the torch the user installed Given a managed runtime with an inference engine already installed And the user has opted out of realigning torch @@ -392,7 +392,7 @@ Feature: Runtime configuration # The other half: that a runtime installed without the toolchain can actually # serve. vLLM compiles Triton kernels at runtime, which is the case most # likely to need `devel`, so it is the one worth proving end to end. - @id:runtime-install-sdk-serves-without-toolchain @requires-gpu @requires-engine:vllm @nightly + @id:runtime-install-sdk-serves-without-toolchain @requires-gpu @requires-engine:vllm @requires-os:linux @nightly Scenario: runtime-18 - A runtime-only SDK install serves vLLM inference Given a machine with no CLI-managed runtimes When the user installs the SDK diff --git a/tests/e2e-cucumber/features/therock_next_generation.feature b/tests/e2e-cucumber/features/therock_next_generation.feature index 6e38e6c1f..b1c19bf71 100644 --- a/tests/e2e-cucumber/features/therock_next_generation.feature +++ b/tests/e2e-cucumber/features/therock_next_generation.feature @@ -171,7 +171,7 @@ Feature: TheRock "next" ROCm 10 install layout # this line and still be unservable, so this scenario reuses the same # serve/chat steps every other engine-canary scenario does rather than # stopping at "install reported the right pins". - @id:therock-next-09-live-install-reports-vllm-rocm10x-discovery-pins @requires-gpu @requires-engine:vllm + @id:therock-next-09-live-install-reports-vllm-rocm10x-discovery-pins @requires-gpu @requires-engine:vllm @requires-os:linux Scenario: therock-next-09 - Installing vLLM against a live ROCm 10 preview runtime reports the discovery pins Given a machine with no CLI-managed runtimes When the user installs the SDK from the ROCm 10 preview source with no family override diff --git a/tests/e2e-cucumber/src/expectation.rs b/tests/e2e-cucumber/src/expectation.rs index 0351bbe30..acdb51fd2 100644 --- a/tests/e2e-cucumber/src/expectation.rs +++ b/tests/e2e-cucumber/src/expectation.rs @@ -18,9 +18,9 @@ use serde::Deserialize; use crate::capability::{HostCapability, docker_available}; -/// Tag prefixes in their normalized bare form. cucumber-rs currently supplies -/// scenario tags with a leading `@`; [`ScenarioDecl::from_tags`] accepts either -/// shape so direct unit fixtures and runtime parser output cannot drift. +/// Tag prefixes in their normalized bare form. The gherkin parser yields +/// scenario tags without the leading `@` and unit fixtures write it; +/// [`ScenarioDecl::from_tags`] accepts either shape so the two cannot drift. const ID_PREFIX: &str = "id:"; const REQUIRES_ENGINE_PREFIX: &str = "requires-engine:"; const REQUIRES_OS_PREFIX: &str = "requires-os:"; @@ -98,9 +98,13 @@ pub struct ScenarioDecl { /// apply, so a scenario needing one of those bare-metal matches has no /// premise. (Keyword-only checks explicitly registered for WSL — e.g. the /// vLLM startup-OOM entry — do run there; those belong under - /// `@requires-os:linux`, not this tag.) And `examine`'s probe returns as - /// soon as it recognises WSL2, before most of its steps run, so a scenario - /// asserting anything those steps populate has none either. + /// `@requires-os:linux`, not this tag.) And `examine` skips its bare-metal + /// probes on WSL2 — the CPU, the PCI and KFD GPU enumeration, kernel + /// modules, the user's render and video groups, device nodes, Secure Boot + /// and the amdgpu kernel log — so the CPU details, the `gpus` list and + /// what those other probes report are absent there, and a scenario + /// asserting one of them has no premise either. Whether the host has an + /// AMD GPU is still reported, from the Windows side's detection. /// /// This is a SKIP and not an xfail row: the diagnose half is deliberate and /// separately unit-tested, and recording it as a known bug would tell every @@ -182,8 +186,9 @@ pub struct ScenarioDecl { } impl ScenarioDecl { - /// Parse a scenario's declaration from its gherkin tags, accepting the - /// leading `@` shape cucumber-rs supplies at runtime and bare unit fixtures. + /// Parse a scenario's declaration from its gherkin tags, accepting them as + /// the gherkin parser yields them at runtime (no leading `@`) and with the + /// leading `@` unit fixtures write. pub fn from_tags>(tags: &[S]) -> Self { let mut id = None; let mut requires_docker = false; diff --git a/tests/e2e-cucumber/tests/driver_host_guard.rs b/tests/e2e-cucumber/tests/driver_host_guard.rs index d88445713..f96a95f2f 100644 --- a/tests/e2e-cucumber/tests/driver_host_guard.rs +++ b/tests/e2e-cucumber/tests/driver_host_guard.rs @@ -2,9 +2,14 @@ // // SPDX-License-Identifier: MIT +//! Tests of the host checks in `e2e/host_guard.rs`, which this file includes. +//! //! The Windows install scenarios must refuse a Linux/WSL host before any CLI -//! call, even when an explicit cucumber filter bypasses the platform tags. -//! Check the actual step wiring without ever invoking a driver installer. +//! call, even when an explicit cucumber filter bypasses the platform tags: this +//! file checks the actual step wiring without ever invoking a driver installer, +//! and that the check refuses Linux and WSL. Including `host_guard.rs` also +//! runs its own test of the KFD topology step's check, which refuses any host +//! but bare-metal Linux. #[path = "e2e/host_guard.rs"] mod host_guard; diff --git a/tests/e2e-cucumber/tests/e2e/examine_steps.rs b/tests/e2e-cucumber/tests/e2e/examine_steps.rs index a2288e098..fbdb083e8 100644 --- a/tests/e2e-cucumber/tests/e2e/examine_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/examine_steps.rs @@ -1078,6 +1078,11 @@ fn lowest_kfd_gpu_node_gfx_target_version() -> Option { /// (9.4.2 -> gfx942). Revisions from 10 up use a lettered form (9.0.10 -> /// gfx90a) whose mapping belongs to the CLI, and copying it here would just /// restate the code under test, so those hosts keep the looser assertion above. +/// +/// So it returns early, asserting only that a target was reported, in two +/// cases: no KFD node reports a nonzero `gfx_target_version` (the topology is +/// unreadable, as on native Windows and WSL2, or names no GPU node), or the +/// lowest such node's revision is 10 or higher. #[then("the inspection names the GPU target that the kernel reports")] async fn assert_gpu_target_matches_kfd(world: &mut E2eWorld) { let output = world.cli_output.as_ref().expect("no command was run"); @@ -1103,11 +1108,13 @@ async fn assert_gpu_target_matches_kfd(world: &mut E2eWorld) { /// The `gfx_target_version` of every GPU the KFD topology describes, read /// straight from sysfs. /// -/// `None` when the topology is unreadable, which is the normal case off Linux. -/// CPU nodes report a `gfx_target_version` of `0` and are skipped, so the length -/// is the kernel's own GPU count and the values say whether those GPUs are all -/// the same part. -fn kfd_gpu_node_versions() -> Option> { +/// `None` when the topology is unreadable. It takes a +/// `host_guard::BareMetalLinux`, which only the host check makes, so it cannot +/// be called off bare-metal Linux, where it would always be `None`. CPU nodes +/// report a `gfx_target_version` of `0` and are skipped, so the length is the +/// kernel's own GPU count and the values say whether those GPUs are all the +/// same part. +fn kfd_gpu_node_versions(_host: &super::host_guard::BareMetalLinux) -> Option> { let mut versions = Vec::new(); for entry in std::fs::read_dir("/sys/class/kfd/kfd/topology/nodes") .ok()? @@ -1140,7 +1147,12 @@ fn host_has_lspci() -> bool { #[then("it lists one AMD GPU per kernel GPU node, each with its PCI address and gfx target")] async fn assert_gpus_match_kfd_nodes(world: &mut E2eWorld) { - let Some(versions) = kfd_gpu_node_versions().filter(|versions| !versions.is_empty()) else { + // Off bare-metal Linux there is no KFD topology, so the early returns below + // would pass this step without asserting anything. They are for a Linux + // host missing a premise; any other host is a tagging mistake. + let host = super::host_guard::this_host_is_bare_metal_linux(); + let Some(versions) = kfd_gpu_node_versions(&host).filter(|versions| !versions.is_empty()) + else { return; }; if !host_has_lspci() { diff --git a/tests/e2e-cucumber/tests/e2e/host_guard.rs b/tests/e2e-cucumber/tests/e2e/host_guard.rs index da02480ef..d76ca49ef 100644 --- a/tests/e2e-cucumber/tests/e2e/host_guard.rs +++ b/tests/e2e-cucumber/tests/e2e/host_guard.rs @@ -2,11 +2,13 @@ // // SPDX-License-Identifier: MIT -//! The host check the Windows driver-install steps run before any CLI call. +//! Host checks steps run before relying on an OS-specific premise: the +//! Windows driver-install steps before any CLI call, and the KFD topology step +//! before reading sysfs. //! //! Kept free of the cucumber world so `driver_host_guard.rs` can include it and -//! call it with an OS name. The `e2e` target has `harness = false`, so a test -//! inside it would never run. +//! test each check with the host facts as arguments. The `e2e` target has +//! `harness = false`, so a test inside it would never run. /// Panic unless `os` names native Windows. WSL reports `linux`. pub fn require_os_is_windows(os: &str) { @@ -15,3 +17,52 @@ pub fn require_os_is_windows(os: &str) { "this driver-install step requires native Windows" ); } + +/// Proof that this host was checked and is bare-metal Linux. Only +/// [`this_host_is_bare_metal_linux`] makes one, so a function that takes one +/// (`kfd_gpu_node_versions` in `examine_steps.rs`) cannot be called without +/// the check having passed. That parameter is the whole enforcement: no test +/// checks that a step calls the check, and dropping the parameter is a +/// signature change a reviewer sees. +pub struct BareMetalLinux(()); + +/// The check behind [`this_host_is_bare_metal_linux`], taking the host facts +/// as arguments so it can be tested on any host. Private, so no step can hand +/// it facts of its own. +fn bare_metal_linux(os: &str, is_wsl: bool) -> BareMetalLinux { + assert!( + os == "linux" && !is_wsl, + "this step is tagged @requires-os:linux @requires-bare-metal, but runs on \ + os={os:?} wsl={is_wsl}: its premise exists only on bare-metal Linux" + ); + BareMetalLinux(()) +} + +/// Panic unless this host is Linux and not WSL2, as the harness's capability +/// probe reports it. For a step whose scenario is tagged +/// `@requires-os:linux @requires-bare-metal`: reaching it anywhere else means +/// the tags were bypassed (an explicit filter) or dropped, and the step would +/// otherwise find no premise and pass without asserting. +// Only `driver_host_guard.rs`, which includes this file under the test harness +// and calls no step, leaves it unused. +#[cfg_attr(test, allow(dead_code))] +pub fn this_host_is_bare_metal_linux() -> BareMetalLinux { + let cap = e2e_cucumber::capability::host_capability(); + bare_metal_linux(&cap.os_family, cap.is_wsl) +} + +// Run by `driver_host_guard.rs`, which includes this file and has the test +// harness; the `e2e` target has none, so they never run there. +#[cfg(test)] +mod tests { + #[test] + fn the_bare_metal_linux_check_accepts_only_bare_metal_linux() { + super::bare_metal_linux("linux", false); + for (os, is_wsl) in [("linux", true), ("windows", false), ("macos", false)] { + assert!( + std::panic::catch_unwind(|| super::bare_metal_linux(os, is_wsl)).is_err(), + "{os:?} wsl={is_wsl} must be refused" + ); + } + } +} diff --git a/tests/e2e-cucumber/tests/platform_tags.rs b/tests/e2e-cucumber/tests/platform_tags.rs new file mode 100644 index 000000000..94a51175c --- /dev/null +++ b/tests/e2e-cucumber/tests/platform_tags.rs @@ -0,0 +1,802 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Drift guard: a scenario gated on a host capability also says which OS it +//! is for, or is reviewed as OS-neutral. +//! +//! A capability tag (`@requires-gpu`, `@requires-engine:`, …) asks what the +//! host *has*, never what it *is*: the Windows GPU lane has a GPU, so it runs +//! every `@requires-gpu` scenario. A scenario whose premise exists on one OS +//! only — `/sys/class/kfd`, `lspci`, a Win32 call — then reaches a host that +//! cannot hold it, and a step that returns early when its premise is missing +//! passes there without asserting anything. `examine-17` did exactly that on +//! the Windows lane, reported green. +//! +//! So every capability-tagged scenario must either carry a tag that decides +//! its OS (`@requires-os:` or `@requires-wsl`) or be listed in +//! [`ALLOWED_ANY_OS`] with the reason its assertions hold on every OS a lane +//! runs it on. docs/testing.md lists everything this fails on. +//! +//! What it reads is what the harness's `filter_run` reads under its default +//! discovery: every `*.feature` file (extension in any case) under `features/`, +//! recursively, with Scenario Outlines expanded by cucumber's own +//! `expand_examples` so an `Examples:` block's tags join its scenarios' tags, +//! and each scenario's tags parsed by the same [`ScenarioDecl::from_tags`]. A +//! feature file run through cucumber's `--input` instead is not read here. +//! +//! This runs in the ordinary `cargo test` set (unlike the `e2e` target, which +//! needs a real `rocm` binary), so a missing tag is caught without a suite run. + +use std::collections::{BTreeMap, BTreeSet}; +use std::path::{Path, PathBuf}; + +use cucumber::feature::Ext as _; +use cucumber::gherkin::{self, GherkinEnv}; +use e2e_cucumber::expectation::ScenarioDecl; + +/// For the remote scenarios, which share one reason. +const REMOTE_OVER_SSH: &str = "drives the CLI's SSH client against a Linux container and asserts \ + the client's output, its session list and the container's own state. The client's \ + Unix-only parts (SSH connection multiplexing, the credential file's 0600 mode) are \ + asserted by no step. Only lanes that set E2E_INCLUDE_DOCKER run it, which today are the \ + GitHub-hosted Ubuntu lanes, so a native Windows client is never exercised here"; + +/// Capability-tagged scenarios reviewed as meaningful on every OS a lane runs +/// them on, by `@id`, with the reason. An entry is a claim about the steps: if +/// one starts asserting something only one OS has, the entry is wrong and the +/// scenario needs `@requires-os:` instead. +const ALLOWED_ANY_OS: &[(&str, &str)] = &[ + // bench.feature + ( + "bench-load-real-serve", + "benchmarks whichever engine the host serves with, through the endpoint the CLI prints", + ), + // diagnose.feature + ( + "diagnose-fix-needing-an-argument-says-so", + "`fix-9-igpu-dgpu` needs `--device-index` on both `linux` and `windows`, and the step \ + asserts exit code 0, the `Flags:` line and no claimed change, the same on both. \ + @requires-bare-metal keeps it off WSL2, which the CLI reports as `wsl`, a platform \ + the entry has no row for: there it is declined as not applicable, the contract \ + diagnose-fix-inapplicable-here-is-declined-not-attempted covers with another entry", + ), + // examine.feature + ( + "examine-detects-gpu-and-driver", + "asserts that `examine` names a gfx target, which it prints on every OS. The driver \ + step passes when the output contains `amdgpu` or `driver_status` anywhere, and \ + `examine` prints a `driver_status:` line on every host whatever its value, so it \ + asserts no driver state on any OS; the KFD target cross-check returns early \ + wherever it cannot read a usable KFD target, so on native Windows and WSL2 it \ + checks nothing (`assert_gpu_target_matches_kfd` in examine_steps.rs lists the \ + exact cases)", + ), + ( + "examine-framework-names-the-interpreter-that-answered", + "the managed runtime's interpreter is reported on every OS a runtime installs on", + ), + // model_serving.feature — the default engine, whichever the host has + ( + "serve-large-model-inference", + "serves through the host's default engine; inference is checked over HTTP", + ), + ( + "serve-default-engine-working-endpoint", + "serves through the host's default engine; the endpoint is checked over HTTP", + ), + ( + "serve-default-engine-inference", + "serves through the host's default engine; inference is checked over HTTP", + ), + ( + "serve-small-assistant-short-name", + "serves through the host's default engine; the step asserts the exact model ref the \ + CLI resolved for that engine, and inference is checked over HTTP", + ), + ( + "serve-readiness-contract", + "the readiness contract is the CLI's own, the same on every OS and engine", + ), + ( + "serve-explicit-port-reserved", + "the managed-port reservation is checked from service records, the same on every OS", + ), + ( + "serve-explicit-port-in-use-by-untracked-process", + "the port is held by a loopback listener the step opens, which works on every OS", + ), + ( + "serve-auto-port-skips-occupied", + "the port is held by a loopback listener the step opens, which works on every OS", + ), + ( + "serve-absent-gpu-index-rejected", + "the --gpu index check runs before any engine starts, on every OS; the step accepts \ + each lane's refusal wording", + ), + ( + "serve-anthropic-endpoint-advertised", + "the Anthropic-compatible proxy is the CLI's own, in front of any engine on any OS", + ), + ( + "serve-anthropic-messages-inference", + "the Anthropic-compatible proxy is the CLI's own, in front of any engine on any OS", + ), + ( + "serve-anthropic-endpoint-readiness", + "the Anthropic-compatible proxy is the CLI's own, in front of any engine on any OS", + ), + ( + "serve-anthropic-messages-tool-use", + "the Anthropic-compatible proxy is the CLI's own, in front of any engine on any OS", + ), + ( + "serve-anthropic-messages-streaming", + "the Anthropic-compatible proxy is the CLI's own, in front of any engine on any OS", + ), + // model_serving.feature — Lemonade, which serves on Linux and Windows + ( + "serve-lemonade-inference", + "Lemonade serves on Linux and Windows; inference is checked over HTTP", + ), + ( + "serve-hf-checkpoint-inference", + "Lemonade serves on Linux and Windows; inference is checked over HTTP", + ), + ( + "serve-missing-gguf-variant-refused", + "serve may install Lemonade first, but the variant check is OS-neutral code (no cfg \ + branch) that refuses before any server starts; the steps assert the same on every \ + OS, none conditionally: a non-zero exit with the `owner/repo:` advice that \ + pins the refusal, neither `Q4_K_M` nor `Q4_0` named, and no record under the \ + isolated data dir's `services/`", + ), + ( + "serve-stop-leaves-no-background-helper", + "Lemonade serves on Linux and Windows; the step reads serve's output, the helper's \ + autostart claim and runtime state under the scenario's own data directory, and \ + `rocm daemon --status`, which the CLI keeps the same way on both", + ), + // No-GPU refusal + ( + "serve-lemonade-preparation-recovery", + "the retry and recovery message are OS-neutral; the failure is injected before any \ + download", + ), + // remote.feature + ("remote-doctor-refuses-to-install", REMOTE_OVER_SSH), + ("remote-serve-publishes-endpoint", REMOTE_OVER_SSH), + ("remote-status-reports-both-halves", REMOTE_OVER_SSH), + ("remote-attach-restores-endpoint", REMOTE_OVER_SSH), + ("remote-stop-clears-both", REMOTE_OVER_SSH), + ("remote-doctor-reports-remote-health", REMOTE_OVER_SSH), + ( + "remote-serve-refuses-reusing-already-serving-session", + REMOTE_OVER_SSH, + ), + // runtime_setup.feature + ( + "runtime-install-sdk-active", + "the default SDK install and its registration run on Linux and Windows alike", + ), + ( + "runtime-path-not-nested", + "reads `active_runtime_root` from `examine`, which every OS prints", + ), + ( + "runtime-update-reports-freshness", + "the update check compares the active runtime with the channel index on every OS", + ), + ( + "runtime-resolve-canonical-release", + "a dry-run that resolves the release index for the detected target, on every OS", + ), + ( + "runtime-resolve-earlier-release", + "a dry-run that resolves the pinned earlier release from the earlier stream's \ + aggregate index, which publishes for Linux and Windows alike", + ), + ( + "runtime-install-sdk-overwrite-requires-yes", + "the consent refusal is OS-neutral code that runs before any download", + ), + ( + "runtime-install-sdk-overwrite-with-yes", + "the default SDK reinstall runs on Linux and Windows alike", + ), + ( + "runtime-lemonade-backend-alignment-opt-out", + "Lemonade and the default SDK both install on Linux and Windows", + ), + // therock_next_generation.feature + ( + "therock-next-07-live-install-auto-detects-arch", + "installs with --format wheel, which Windows supports (only the tarball format is \ + Linux-only); the GPU probe runs through the runtime's torch on any OS", + ), +]; + +/// One parsed scenario: the feature file it lives in, its name, and its tags as +/// the harness reads them. +struct Parsed { + file: String, + name: String, + decl: ScenarioDecl, +} + +fn features_dir() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("features") +} + +/// Every `*.feature` file under `dir`, at any depth, matching the extension in +/// any case — the files cucumber's default discovery (`*.feature`, +/// case-insensitive) hands the harness. +fn feature_files(dir: &Path, out: &mut Vec) { + for entry in std::fs::read_dir(dir).unwrap_or_else(|e| panic!("{}: {e}", dir.display())) { + let entry = entry.expect("dir entry"); + let path = entry.path(); + let file_type = entry.file_type().expect("file type"); + if file_type.is_dir() { + feature_files(&path, out); + } else if file_type.is_file() + && path + .extension() + .is_some_and(|ext| ext.eq_ignore_ascii_case("feature")) + { + out.push(path); + } + } +} + +/// What [`parse_dir`] read: every scenario the harness would filter, and +/// every tag `ScenarioDecl` reads found where the harness never hands it one. +struct Suite { + scenarios: Vec, + /// Tags `ScenarioDecl::from_tags` reads, found on a `Feature:` or `Rule:` + /// line. The harness passes it only `scenario.tags`, so it never sees + /// these. Other tags there are not collected: cucumber's own runner reads + /// Feature and Rule tags too, for `@serial`, `@retry` and `--tags`. + misplaced_tags: Vec, +} + +/// The real suite. Panics where [`parse_dir`] fails, since every check here +/// would otherwise pass over nothing. +fn parsed_suite() -> Suite { + parse_dir(&features_dir()).unwrap_or_else(|e| panic!("{e}")) +} + +fn parsed_scenarios() -> Vec { + parsed_suite().scenarios +} + +/// Every scenario under `dir` the harness would filter, including those under +/// a `Rule:` and each expanded row of a Scenario Outline, with each scenario's +/// own tags plus its `Examples:` tags once expanded. Fails where there is no +/// feature file or no scenario, either of which would leave every check here +/// passing over nothing. +fn parse_dir(dir: &Path) -> Result { + let mut paths = Vec::new(); + feature_files(dir, &mut paths); + paths.sort(); + if paths.is_empty() { + return Err(format!( + "no .feature files found under {} — every check here would pass vacuously", + dir.display(), + )); + } + let mut suite = Suite { + scenarios: Vec::new(), + misplaced_tags: Vec::new(), + }; + for path in paths { + let feature = gherkin::Feature::parse_path(&path, GherkinEnv::default()) + .map_err(|e| format!("{}: {e}", path.display()))? + .expand_examples() + .map_err(|e| format!("{}: {e}", path.display()))?; + let file = path + .strip_prefix(dir) + .unwrap_or(&path) + .to_string_lossy() + .into_owned(); + let headers = std::iter::once((&feature.keyword, &feature.name, &feature.tags)) + .chain(feature.rules.iter().map(|r| (&r.keyword, &r.name, &r.tags))); + for (keyword, name, tags) in headers { + for tag in tags.iter().filter(|tag| scenario_decl_reads(tag)) { + suite + .misplaced_tags + .push(format!("{file}: {keyword} {name:?} carries @{tag}")); + } + } + let scenarios = feature + .scenarios + .iter() + .chain(feature.rules.iter().flat_map(|r| r.scenarios.iter())); + for scenario in scenarios { + suite.scenarios.push(Parsed { + file: file.clone(), + name: scenario.name.clone(), + decl: ScenarioDecl::from_tags(&scenario.tags), + }); + } + } + if suite.scenarios.is_empty() { + return Err(format!( + "no scenarios parsed from {} — every check here would pass vacuously", + dir.display(), + )); + } + Ok(suite) +} + +/// What a scenario's tags say about where it runs. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct Placement { + /// Gated on something the host has: every `@requires-*` tag except the + /// two that decide the OS. + capability_gated: bool, + /// The tags pin the OS: `@requires-os:` or `@requires-wsl`. + /// `@requires-bare-metal` does not: it skips WSL2 only, so native Windows + /// still runs it — and `@requires-os:linux` still admits WSL2, which + /// reports itself as Linux. + platform_decided: bool, +} + +/// Classify a scenario's declaration. Destructured without `..`, so a field +/// added to `ScenarioDecl` fails to compile here until it is put on one side or +/// the other. `@nightly`, `@lifecycle`, `@merge-queue` and `@serve-timeout` +/// choose when a scenario runs, not where, so they are neither. +const fn placement(d: &ScenarioDecl) -> Placement { + let ScenarioDecl { + id: _, + requires_gpu, + requires_multi_gpu, + requires_gfx_target, + requires_no_gpu, + requires_bare_metal, + requires_wsl, + requires_oom_fault_injection, + requires_case_sensitive_fs, + requires_dir_permissions, + requires_engine, + requires_os, + serve_timeout_secs: _, + requires_docker, + nightly: _, + lifecycle: _, + merge_queue: _, + } = d; + Placement { + capability_gated: *requires_gpu + || *requires_multi_gpu + || *requires_gfx_target + || *requires_no_gpu + || *requires_bare_metal + || requires_engine.is_some() + || *requires_docker + || *requires_oom_fault_injection + || *requires_case_sensitive_fs + || *requires_dir_permissions, + platform_decided: requires_os.is_some() || *requires_wsl, + } +} + +/// Whether `ScenarioDecl::from_tags` reads anything from `tag`. Destructured +/// without `..`, like [`placement`], so a field added to `ScenarioDecl` fails +/// to compile here until it is counted. +fn scenario_decl_reads(tag: &str) -> bool { + let ScenarioDecl { + id, + requires_gpu, + requires_multi_gpu, + requires_gfx_target, + requires_no_gpu, + requires_bare_metal, + requires_wsl, + requires_oom_fault_injection, + requires_case_sensitive_fs, + requires_dir_permissions, + requires_engine, + requires_os, + serve_timeout_secs, + requires_docker, + nightly, + lifecycle, + merge_queue, + } = ScenarioDecl::from_tags(&[tag]); + id.is_some() + || requires_gpu + || requires_multi_gpu + || requires_gfx_target + || requires_no_gpu + || requires_bare_metal + || requires_wsl + || requires_oom_fault_injection + || requires_case_sensitive_fs + || requires_dir_permissions + || requires_engine.is_some() + || requires_os.is_some() + || serve_timeout_secs.is_some() + || requires_docker + || nightly + || lifecycle + || merge_queue +} + +fn label(p: &Parsed) -> String { + let id = p.decl.id.as_deref().unwrap_or(""); + format!("@id:{id} ({}: {:?})", p.file, p.name) +} + +/// Capability-tagged scenarios that neither decide their OS nor are listed in +/// `allowed`, each with what to do about it. +fn undecided_scenarios(scenarios: &[Parsed], allowed: &[(&str, &str)]) -> Vec { + let allowed: BTreeSet<&str> = allowed.iter().map(|(id, _)| *id).collect(); + scenarios + .iter() + .filter(|p| { + let placement = placement(&p.decl); + placement.capability_gated && !placement.platform_decided + }) + .filter_map(|p| match p.decl.id.as_deref() { + None => Some(format!( + "{}: add an @id:- tag and then an OS tag, or an \ + ALLOWED_ANY_OS entry under that id", + label(p) + )), + Some(id) if !allowed.contains(id) => Some(format!( + "{}: add an OS tag, or add the id to ALLOWED_ANY_OS with a reason", + label(p) + )), + Some(_) => None, + }) + .collect() +} + +/// `@id`s that more than one scenario carries. The harness fails a run on the +/// first one; here it would also let one scenario's allowlist entry vouch for +/// another. +fn duplicate_ids(scenarios: &[Parsed]) -> Vec { + let mut by_id: BTreeMap<&str, Vec<&Parsed>> = BTreeMap::new(); + for p in scenarios { + if let Some(id) = p.decl.id.as_deref() { + by_id.entry(id).or_default().push(p); + } + } + by_id + .into_values() + .filter(|ps| ps.len() > 1) + .map(|ps| ps.into_iter().map(label).collect::>().join(" and ")) + .collect() +} + +/// Entries in `allowed` that are duplicated, unexplained, name no scenario, or +/// name one that needs no entry. +fn allowlist_problems(scenarios: &[Parsed], allowed: &[(&str, &str)]) -> Vec { + let by_id: BTreeMap<&str, &Parsed> = scenarios + .iter() + .filter_map(|p| p.decl.id.as_deref().map(|id| (id, p))) + .collect(); + let mut seen = BTreeSet::new(); + let mut problems = Vec::new(); + for (id, reason) in allowed { + if !seen.insert(*id) { + problems.push(format!("{id}: listed more than once")); + } + if reason.trim().is_empty() { + problems.push(format!("{id}: has no reason")); + } + match by_id.get(id) { + None => problems.push(format!("{id}: no scenario has this @id — drop the entry")), + Some(p) if !placement(&p.decl).capability_gated => problems.push(format!( + "{}: carries no capability tag, so it needs no entry — drop it", + label(p) + )), + Some(p) if placement(&p.decl).platform_decided => problems.push(format!( + "{}: its tags already decide its OS, which contradicts an \"any OS\" entry — \ + drop the entry or the OS tag", + label(p) + )), + Some(_) => {} + } + } + problems +} + +#[test] +fn capability_tagged_scenarios_decide_their_os_or_are_reviewed() { + let violations = undecided_scenarios(&parsed_scenarios(), ALLOWED_ANY_OS); + assert!( + violations.is_empty(), + "{} capability-tagged scenario(s) neither decide their OS nor are reviewed as \ + OS-neutral. A capability tag is not a platform tag: the Windows GPU lane satisfies \ + `@requires-gpu`, so a step whose premise is OS-specific runs there and can pass \ + without asserting anything (AGENTS.md §3). The OS tags are @requires-os: and \ + @requires-wsl; @requires-os:linux still admits WSL2, so add @requires-bare-metal \ + too when WSL2 lacks the interface. An ALLOWED_ANY_OS entry, in \ + tests/e2e-cucumber/tests/platform_tags.rs, states why the steps hold on every OS:\n {}", + violations.len(), + violations.join("\n "), + ); +} + +#[test] +fn scenario_ids_are_unique() { + let duplicates = duplicate_ids(&parsed_scenarios()); + assert!( + duplicates.is_empty(), + "scenario @ids must be unique — the harness fails the run on a repeat:\n {}", + duplicates.join("\n "), + ); +} + +#[test] +fn feature_and_rule_lines_carry_no_tags() { + let misplaced = parsed_suite().misplaced_tags; + assert!( + misplaced.is_empty(), + "the harness builds each scenario's ScenarioDecl from that scenario's own tags, \ + so a tag ScenarioDecl reads never reaches it from a Feature: or Rule: line — put \ + it on each scenario instead (cucumber's own tags, such as @serial, are not \ + reported):\n {}", + misplaced.join("\n "), + ); +} + +#[test] +fn allowlist_entries_are_live_and_needed() { + let problems = allowlist_problems(&parsed_scenarios(), ALLOWED_ANY_OS); + assert!( + problems.is_empty(), + "ALLOWED_ANY_OS in tests/e2e-cucumber/tests/platform_tags.rs has stale or \ + contradictory entries (AGENTS.md §3):\n {}", + problems.join("\n "), + ); +} + +/// The guard is only as good as the parse under it: this pins the scenario +/// that motivated it, so a parse that silently drops tags cannot leave every +/// other check passing over nothing. +#[test] +fn the_kfd_topology_scenario_is_pinned_to_bare_metal_linux() { + const ID: &str = "examine-lists-every-gpu-with-its-address-and-target"; + let scenarios = parsed_scenarios(); + let examine_17 = scenarios + .iter() + .find(|p| p.decl.id.as_deref() == Some(ID)) + .unwrap_or_else(|| panic!("examine.feature no longer has @id:{ID}")); + assert_eq!( + examine_17.decl.requires_os.as_deref(), + Some("linux"), + "{}: reads /sys/class/kfd, which only Linux has", + label(examine_17), + ); + assert!( + examine_17.decl.requires_bare_metal, + "{}: WSL2 has no KFD topology either, so it needs @requires-bare-metal", + label(examine_17), + ); +} + +// The checks above run over the real suite, which holds no violation, so they +// cannot show that a broken check would notice one. These feed each check a +// fixture holding exactly one. + +fn fixture(tags: &[&str]) -> Parsed { + Parsed { + file: "fixture.feature".to_owned(), + name: tags.join(" "), + decl: ScenarioDecl::from_tags(tags), + } +} + +/// (tag, capability_gated, platform_decided). Every tag `from_tags` reads. +const EVERY_DECL_TAG: &[(&str, bool, bool)] = &[ + ("@requires-gpu", true, false), + ("@requires-multi-gpu", true, false), + ("@requires-gfx-target", true, false), + ("@requires-no-gpu", true, false), + ("@requires-bare-metal", true, false), + ("@requires-engine:aim", true, false), + ("@requires-docker", true, false), + ("@requires-oom-fault-injection", true, false), + ("@requires-case-sensitive-fs", true, false), + ("@requires-dir-permissions", true, false), + ("@requires-os:linux", false, true), + ("@requires-wsl", false, true), + ("@id:fixture-01", false, false), + ("@serve-timeout:60", false, false), + ("@nightly", false, false), + ("@lifecycle", false, false), + ("@merge-queue", false, false), +]; + +#[test] +fn every_tag_is_classified_as_a_capability_an_os_decision_or_neither() { + for &(tag, capability_gated, platform_decided) in EVERY_DECL_TAG { + assert_eq!( + placement(&ScenarioDecl::from_tags(&[tag])), + Placement { + capability_gated, + platform_decided + }, + "{tag}", + ); + } +} + +#[test] +fn every_tag_scenario_decl_reads_is_counted_and_runner_tags_are_not() { + for &(tag, _, _) in EVERY_DECL_TAG { + assert!(scenario_decl_reads(tag), "{tag} is read by ScenarioDecl"); + } + for tag in ["@serial", "@retry", "@retry(2)", "@smoke"] { + assert!( + !scenario_decl_reads(tag), + "{tag} is not read by ScenarioDecl" + ); + } +} + +#[test] +fn an_undecided_scenario_is_reported_and_an_id_less_one_is_told_to_add_an_id() { + let scenarios = [ + fixture(&["@id:listed", "@requires-gpu"]), + fixture(&["@id:decided", "@requires-gpu", "@requires-os:linux"]), + fixture(&["@id:ungated"]), + fixture(&["@id:unlisted", "@requires-gpu"]), + fixture(&["@requires-gpu"]), + ]; + let violations = undecided_scenarios(&scenarios, &[("listed", "a reason")]); + assert_eq!(violations.len(), 2, "{violations:#?}"); + assert!( + violations[0].starts_with("@id:unlisted ") && violations[0].contains("ALLOWED_ANY_OS"), + "{violations:#?}" + ); + assert!( + violations[1].starts_with("@id: ") && violations[1].contains("add an @id"), + "{violations:#?}" + ); +} + +#[test] +fn a_repeated_id_is_reported() { + let scenarios = [ + fixture(&["@id:twice", "@requires-gpu"]), + fixture(&["@id:once"]), + fixture(&["@id:twice"]), + ]; + let duplicates = duplicate_ids(&scenarios); + assert_eq!(duplicates.len(), 1, "{duplicates:#?}"); + assert!(duplicates[0].starts_with("@id:twice "), "{duplicates:#?}"); +} + +#[test] +fn each_kind_of_bad_allowlist_entry_is_reported() { + let scenarios = [ + fixture(&["@id:needed", "@requires-gpu"]), + fixture(&["@id:ungated"]), + fixture(&["@id:decided", "@requires-gpu", "@requires-wsl"]), + ]; + assert_eq!( + allowlist_problems(&scenarios, &[("needed", "a reason")]), + Vec::::new(), + "a needed entry with a reason is fine", + ); + let cases: &[(&[(&str, &str)], &str)] = &[ + ( + &[("needed", "a reason"), ("needed", "a reason")], + "needed: listed more than once", + ), + (&[("needed", " ")], "needed: has no reason"), + (&[("gone", "a reason")], "gone: no scenario has this @id"), + (&[("ungated", "a reason")], "carries no capability tag"), + (&[("decided", "a reason")], "already decide its OS"), + ]; + for &(allowed, expected) in cases { + let problems = allowlist_problems(&scenarios, allowed); + assert_eq!(problems.len(), 1, "{allowed:?}: {problems:#?}"); + assert!( + problems[0].contains(expected), + "{allowed:?}: expected {expected:?}, got {problems:#?}" + ); + } +} + +/// A fresh, empty directory under the test target's own scratch space. +fn scratch_dir(name: &str) -> PathBuf { + let dir = Path::new(env!("CARGO_TARGET_TMPDIR")).join(name); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("fixture dir"); + dir +} + +#[test] +fn discovery_reads_what_the_harness_reads() { + // A nested file with an upper-case extension, an outline whose Examples + // block carries the OS tag, and a scenario under a Rule: each is something + // the harness's default discovery and outline expansion pick up. + let dir = scratch_dir("platform_tags_discovery"); + std::fs::create_dir_all(dir.join("nested")).expect("fixture dir"); + std::fs::write( + dir.join("nested").join("Outline.FEATURE"), + "Feature: fixture\n\ + \n\ + \x20 @id:fixture-outline @requires-gpu\n\ + \x20 Scenario Outline: fixture-01 - row \n\ + \x20 Given a step\n\ + \n\ + \x20 @requires-os:linux\n\ + \x20 Examples:\n\ + \x20 | n |\n\ + \x20 | 1 |\n\ + \n\ + \x20 Rule: a rule\n\ + \n\ + \x20 @id:fixture-under-rule @requires-gpu\n\ + \x20 Scenario: fixture-02 - under a rule\n\ + \x20 Given a step\n", + ) + .expect("fixture file"); + std::fs::write(dir.join("ignored.txt"), "not a feature").expect("fixture file"); + let scenarios = parse_dir(&dir).expect("fixture parses").scenarios; + assert_eq!(scenarios.len(), 2, "one expanded row and one rule scenario"); + let row = &scenarios[0]; + assert_eq!(row.decl.id.as_deref(), Some("fixture-outline")); + assert_eq!( + row.decl.requires_os.as_deref(), + Some("linux"), + "the Examples tag must reach the row" + ); + assert!(Path::new(&row.file).ends_with("nested/Outline.FEATURE")); + let under_rule = &scenarios[1]; + assert_eq!(under_rule.decl.id.as_deref(), Some("fixture-under-rule")); + assert!( + placement(&under_rule.decl).capability_gated, + "a scenario under a Rule is checked like any other" + ); +} + +#[test] +fn a_directory_with_no_feature_file_or_no_scenario_is_refused() { + let empty = scratch_dir("platform_tags_empty"); + std::fs::write(empty.join("notes.txt"), "not a feature").expect("fixture file"); + let err = parse_dir(&empty) + .err() + .expect("no feature file must be refused"); + assert!(err.contains("no .feature files"), "{err}"); + + let bare = scratch_dir("platform_tags_no_scenarios"); + std::fs::write(bare.join("bare.feature"), "Feature: nothing in it\n").expect("fixture file"); + let err = parse_dir(&bare).err().expect("no scenario must be refused"); + assert!(err.contains("no scenarios parsed"), "{err}"); +} + +#[test] +fn a_scenario_decl_tag_on_a_feature_or_rule_line_is_reported_and_a_runner_tag_is_not() { + // `@serial`, `@retry` and a tag only a `--tags` filter would match are read + // by cucumber's runner at Feature and Rule level, so they are not reported. + let dir = scratch_dir("platform_tags_misplaced"); + std::fs::write( + dir.join("misplaced.feature"), + "@serial @requires-gpu @retry(2) @nightly @smoke\n\ + Feature: tagged at the top\n\ + \n\ + \x20 @retry @id:fixture-rule @serial\n\ + \x20 Rule: tagged rule\n\ + \n\ + \x20 @id:fixture-01\n\ + \x20 Scenario: fixture-01 - a scenario\n\ + \x20 Given a step\n", + ) + .expect("fixture file"); + let misplaced = parse_dir(&dir).expect("fixture parses").misplaced_tags; + assert_eq!( + misplaced, + [ + "misplaced.feature: Feature \"tagged at the top\" carries @requires-gpu", + "misplaced.feature: Feature \"tagged at the top\" carries @nightly", + "misplaced.feature: Rule \"tagged rule\" carries @id:fixture-rule", + ], + ); +}