Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ SPDX-License-Identifier: MIT
- [ ] If this PR adds, renames or removes a production source file (or a `.rs` file under `xtask/src` or `tests/e2e-cucumber`), `docs/architecture.md` has the matching entry (`cargo xtask check-architecture-doc` fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.
- [ ] Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will *not* happen.
- [ ] Comments and doc lines this PR adds or changes that state how the code behaves were checked against that code, and each that is the reason for a decision names the file and the function, constant or test that makes it true (AGENTS.md §3).
- [ ] If this PR changes behavior on only one platform, the description names that platform and the check that runs there and would fail without the change (AGENTS.md §3).
- [ ] Any existing scenario this PR moves off the blocking mock `e2e` job (for example by a tag) is justified as a deletion (AGENTS.md §8).
- [ ] Commits answering review feedback were diffed whole against the reviewed head and widen no documented user-facing option's meaning or authority (AGENTS.md §11).
- [ ] Every commit message on the branch describes what its own commit does in words still true at the branch head (AGENTS.md §5).
25 changes: 18 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,23 @@ data. Fixing only the state a reviewer named invites the next round to name its
when a finding is one instance of a class, fix the class and list the sibling cases in
the reply.

**Platform-gated code gets a test that runs on its gate.** When a `cfg(windows)` path
cannot be driven by e2e, put its logic behind a pure helper or an injectable function
(a script runner, a clock) so a unit test runs on the Windows CI job, including the error
and timeout paths. A value matched across files (a version tag, a sentinel string) gets
one shared constant and a test tying the producer to the consumer. If a path stays
untested, say which and why in the PR text.
**Platform-conditional behavior is verified on the platform it targets.** A change that
takes effect on one platform only — a `cfg(windows)` or `cfg(unix)` branch, a read of an
interface one OS alone has (sysfs, a Win32 call), a scenario whose premise is
OS-specific — names in the PR text that platform and the check that runs there and would
fail without the change. A test that runs only elsewhere is not that check, and neither
is one that runs on the target but returns early when its premise is missing. For a
`cfg(windows)` path e2e cannot drive, put its logic behind a pure helper or an injectable
function (a script runner, a clock) so a unit test runs on the Windows CI job, including
the error and timeout paths. A value matched across files (a version tag, a sentinel
string) gets one shared constant and a test tying the producer to the consumer. If a path
stays untested, say which and why in the PR text.

**A capability tag is not a platform tag.** A scenario whose premise exists on one OS only
carries `@requires-os:<os>` (or `@requires-wsl` where the premise is WSL2 itself), plus
`@requires-bare-metal` where WSL2 lacks the interface, even when `@requires-gpu` or another
capability tag already gates it; "Capability-tagged scenarios name their OS" in
[docs/testing.md](docs/testing.md) says why and what checks it.

## 4) Live State Verification Before Any External Claim

Expand Down Expand Up @@ -562,7 +573,7 @@ Run this checklist before any upstream push, PR update, issue update, or public
11. User approval obtained for any high-blast-radius external action — *§1*
12. Force-push's live review/approval state checked before proposing or executing one — *§1*
13. Review comments checked for unreplied threads before resuming review-feedback work — *§11*
14. New tests shown to fail when their target is broken; state matrix written for new state or placeholders; platform-gated logic covered by a test that runs on its gate — *§3*
14. New tests shown to fail when their target is broken; state matrix written for new state or placeholders; platform-conditional behavior verified by a check that runs on its target platform — *§3*
15. Old claims grepped across docs and PR body; PR body matches the final diff — *§5*
16. Subprocess/FFI checklist applied to any new process or driver call — *§6*
17. Comments and doc lines this change adds or edits that state how the code behaves checked against that code; each that is the reason for a decision names where the code makes it true — *§3*
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ Pull requests are normally squash-merged, and by default the squash commit's mes
### What reviewers look for

- Tests cover the new behavior
- Platform-specific changes name the platform they target and a check that runs there and would fail without them
- No secrets, credentials, or internal hostnames in committed files
- Third-party dependencies recorded in `Cargo.lock`
- License headers on new source files (see `licenserc.toml`)
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1518,7 +1518,8 @@ when `ROCR_VISIBLE_DEVICES` is set, because the two orderings can diverge.

#### Reuse is exempt from the no-GPU refusal

Under the default `gpu_required` policy, `rocm serve` on a host with no usable
Under the default `gpu_required` policy, on Linux (including WSL2), where the
CLI can probe which GPUs are usable, `rocm serve` on a host with no usable
AMD GPU fails fast with `no usable AMD GPU detected`, before any engine is
prepared or downloaded. A `rocm serve` that matches an **already-running
managed service for the same engine and model** now skips that refusal and
Expand Down
5 changes: 3 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -844,9 +844,10 @@ Gherkin `.feature` files under `features/` backed by Rust step functions that dr

| File | Responsibility | Class |
| --- | --- | --- |
| [`tests/driver_host_guard.rs`](../tests/e2e-cucumber/tests/driver_host_guard.rs) | Ordinary test that the Windows driver-install steps run the host guard before calling the CLI, checked against the step source. | flat |
| [`tests/driver_host_guard.rs`](../tests/e2e-cucumber/tests/driver_host_guard.rs) | Ordinary tests of the host checks in `tests/e2e/host_guard.rs`: that the Windows driver-install steps run theirs before calling the CLI (checked against the step source), that it refuses Linux and WSL, and, through the test inside `host_guard.rs`, that the KFD topology step's check refuses any host but bare-metal Linux. | flat |
| [`tests/e2e.rs`](../tests/e2e-cucumber/tests/e2e.rs) | The cucumber test binary: the `E2eWorld` state and its cleanup on drop, the `run_rocm` helpers (including `run_rocm_with_scenario_env_off_executor`, which waits for a long CLI run on a blocking thread so it does not stall concurrent scenarios), the runner `main`, and reconciliation of results with expectations. | flat |
| [`tests/feature_naming.rs`](../tests/e2e-cucumber/tests/feature_naming.rs) | Drift guard, in the ordinary `cargo test` set, for the `<feature-key>-<NN>` scenario names and ids in the `.feature` files. | flat |
| [`tests/platform_tags.rs`](../tests/e2e-cucumber/tests/platform_tags.rs) | Drift guard, in the ordinary `cargo test` set, that every capability-tagged scenario also carries an OS-deciding tag or a reviewed `ALLOWED_ANY_OS` entry; [testing.md](testing.md) lists everything it fails on. | flat |
| [`tests/skill_reference.rs`](../tests/e2e-cucumber/tests/skill_reference.rs) | Drift guard, in the ordinary `cargo test` set, that the catalog table in [`reference.md`](../skills/rocm-doctor/reference.md) uses the vocabulary `rocm fix` prints. | flat |

#### `tests/e2e/`
Expand All @@ -866,7 +867,7 @@ Gherkin `.feature` files under `features/` backed by Rust step functions that dr
| [`tests/e2e/driver_steps.rs`](../tests/e2e-cucumber/tests/e2e/driver_steps.rs) | Steps for [`driver_install.feature`](../tests/e2e-cucumber/features/driver_install.feature): previewing and approving GPU driver installation plans on WSL, Linux and Windows hosts. | flat |
| [`tests/e2e/engines_steps.rs`](../tests/e2e-cucumber/tests/e2e/engines_steps.rs) | Steps for [`engine_shell.feature`](../tests/e2e-cucumber/features/engine_shell.feature): `rocm engines shell` under a pseudo-terminal, since the defect shows only in what the terminal renders; and the vLLM steps of `serve-62` in [`model_serving.feature`](../tests/e2e-cucumber/features/model_serving.feature), which run `rocm engines list` against a stand-in interpreter. | flat |
| [`tests/e2e/examine_steps.rs`](../tests/e2e-cucumber/tests/e2e/examine_steps.rs) | Steps for [`examine.feature`](../tests/e2e-cucumber/features/examine.feature): `rocm examine`, the active runtime's provenance, version and help output, engine listing and the driver plan preview. | flat |
| [`tests/e2e/host_guard.rs`](../tests/e2e-cucumber/tests/e2e/host_guard.rs) | The host check the Windows driver-install steps run before any CLI call, kept free of the cucumber world so a separate test can include it. | flat |
| [`tests/e2e/host_guard.rs`](../tests/e2e-cucumber/tests/e2e/host_guard.rs) | The host checks steps run before relying on an OS-specific premise (the Windows driver-install steps, the KFD topology step, which cannot read sysfs without the token its check returns), kept free of the cucumber world so a separate test can include it. | flat |
| [`tests/e2e/lemonade_install_steps.rs`](../tests/e2e-cucumber/tests/e2e/lemonade_install_steps.rs) | Steps for [`lemonade_installs.feature`](../tests/e2e-cucumber/features/lemonade_installs.feature): what the user is told about the Lemonade copies on disk, from planted trees and install records. | flat |
| [`tests/e2e/lifecycle_steps.rs`](../tests/e2e-cucumber/tests/e2e/lifecycle_steps.rs) | Steps for [`install_lifecycle.feature`](../tests/e2e-cucumber/features/install_lifecycle.feature): `xtask package`, the root installer and the installed binaries, from install through uninstall. | flat |
| [`tests/e2e/logs_steps.rs`](../tests/e2e-cucumber/tests/e2e/logs_steps.rs) | Steps for [`logs.feature`](../tests/e2e-cucumber/features/logs.feature): `rocm logs` searches over command-log files planted inside the tail window it reads. | flat |
Expand Down
9 changes: 4 additions & 5 deletions docs/manual-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -1400,10 +1400,9 @@ Expected result:

**Linux or WSL only.** Unlike the rest of this file, the commands below are
bash, not PowerShell, and they pin `--engine vllm`, which is unsupported on
native Windows. The behaviour itself is not platform-specific; only this recipe
for reproducing it is. On native Windows, substitute `--engine lemonade`
throughout and set the mask with `$env:HIP_VISIBLE_DEVICES = ""` (and clear it
with `Remove-Item Env:HIP_VISIBLE_DEVICES` afterwards).
native Windows. The refusal itself is Linux-only too: on native Windows
`probe_usable_amd_gpu_indices` answers unknown rather than "no GPU", so `serve`
does not refuse there and this section has nothing to check.

This is a change to a refusal you may be relying on, so check both halves. Mask
every device to stand in for a host with no usable AMD GPU — the same condition
Expand Down Expand Up @@ -1959,7 +1958,7 @@ Expected result:

The automated scenario is the `@nightly` `serve-aim-key-guards-every-route` in
[model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature),
which runs only on a nightly host with Docker and a Radeon or Strix Halo GPU.
which runs only on a Linux nightly host with Docker and a Radeon or Strix Halo GPU.

## 12. Background Automation Helper

Expand Down
95 changes: 78 additions & 17 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,14 +248,67 @@ stops being needed, or whose file moves. A new field reuses the helper; a new
exception is a reviewed entry in that list.

Where it runs: the pre-push hook (`cargo xtask pre-push-check`) selects `xtask`
for any Rust change, the `windows-debug` job runs `cargo test --workspace` on
every pull request that changes Rust, and the merge queue and post-merge runs
test the whole workspace. The Linux pull-request lane (`cargo xtask affected`)
selects `xtask` only for a change in `xtask` or in a crate it depends on, or for
a file it reads, so a pull request that touches nothing but
for any Rust change, the `windows-debug` job runs
`cargo test --workspace --all-targets` on every pull request that changes Rust,
and the merge queue and post-merge runs test the whole workspace. The Linux
pull-request lane (`cargo xtask affected`) selects `xtask` only for a change in
`xtask` or in a crate it depends on, or for a file it reads, so a pull request
that touches nothing but
`crates/rocm-dash-tui` does not run this test on Linux. The module-import
contract above has the same property.

### Capability-tagged scenarios name their OS

A capability tag (`@requires-gpu`, `@requires-engine:<e>`, `@requires-docker`
and the rest) asks what the host has, not which OS it runs, so the Windows GPU
lane runs every `@requires-gpu` scenario. A scenario whose premise exists on
one OS only also carries `@requires-os:<os>`, or `@requires-wsl` where the
premise is WSL2 itself. `@requires-os:linux` still admits WSL2, which reports
itself as Linux, so add `@requires-bare-metal` too when WSL2 lacks the
interface. examine-17 is the case that prompted this: its step reads the KFD
topology, found none on the Windows lane, and passed without asserting
anything.

[platform_tags.rs](../tests/e2e-cucumber/tests/platform_tags.rs) reads the
`.feature` files the e2e harness reads by default (every `*.feature` under
`tests/e2e-cucumber/features/`, recursively, extension in any case), expands
Scenario Outlines the way cucumber does, and parses each scenario's tags with
the harness's own `ScenarioDecl::from_tags`. A capability tag there is any
`@requires-*` tag `ScenarioDecl` reads except `@requires-os:<os>` and
`@requires-wsl`, the two that decide the OS. It fails on:

- a capability-tagged scenario with neither OS-deciding tag whose `@id` is not
in its `ALLOWED_ANY_OS` list, including one with no `@id` at all
- two scenarios with the same `@id`
- a tag `ScenarioDecl` reads on a `Feature:` or `Rule:` line, which it never
sees there: the harness builds each scenario's `ScenarioDecl` from that
scenario's own tags. Other tags there are not reported, since cucumber's own
runner reads Feature and Rule tags for `@serial`, `@retry` and `--tags`
- finding no `.feature` file, or no scenario, which would leave every other
check passing over nothing
- an `ALLOWED_ANY_OS` entry that is listed twice, has an empty reason, names no
scenario, names one with no capability tag, or names one whose tags already
decide its OS

Each entry's reason says why the scenario's steps hold on every OS a lane runs
it on. The guard checks only that the OS is decided: whether a Linux scenario
also needs `@requires-bare-metal` is left to review, and it pins that tag only
on examine-17. Run it on its own with:

```bash
cargo test -p e2e-cucumber --test platform_tags
```

It reads tags, not steps, so whether an entry's reason is still true is a
review question: a step that starts reading something only one OS has turns
its entry wrong without failing anything here. Where it runs: the
`windows-debug` job runs `cargo test --workspace --all-targets` on every pull
request that changes Rust or a `.feature` file, both of which its `heavy` path
filter matches, so a tag-only edit is checked there. The Linux pull-request
lane runs it only when the change selects `e2e-cucumber` and passes that lane's
Rust path filter, which a `.feature` file alone does not, and the pre-push hook
does not run `e2e-cucumber`'s tests.

### Tests that run a fake executable

Plant a stand-in `python`, `uv`, `amd-smi`, ... with
Expand Down Expand Up @@ -1182,8 +1235,8 @@ filter, as before, including a lockfile-only push.

The hook is an early warning, and CI is not narrowed everywhere. On pull requests the `clippy`
job lints the whole workspace and the `test` job (`Test (affected crates)`) runs only the crates
`cargo xtask affected` selects. The whole workspace's tests run in the `windows-debug` job on
every change, and in the merge queue and on `main`, where the Linux `test` job is forced to
`cargo xtask affected` selects. The whole workspace's tests run in the `windows-debug` job when
a change touches its `heavy` paths, as every Rust change does, and in the merge queue and on `main`, where the Linux `test` job is forced to
`--workspace`.

Known limits, unchanged by the narrowing: the steps check the working tree rather than the pushed
Expand Down Expand Up @@ -1832,12 +1885,17 @@ AIM serve and restart canaries and the AIM endpoint-key check) are `@nightly`
for cost: each pulls the AIM image and a 9B
checkpoint on a cold runner and may wait up to 1200s for readiness, which on the
Strix Ubuntu job, back when it ran on pull requests (about 25 minutes without
them, 45 minute cap) used up the whole budget. They still need `@requires-engine:aim`,
i.e. Docker reachable and a Radeon or Strix Halo GPU, so they run only where
`E2E_INCLUDE_NIGHTLY=1` is set on such a host: the nightly Strix Halo lane, or an
`include_nightly` dispatch of `e2e-selfhosted.yml` (whose Strix Ubuntu job then
gets a 125 minute cap). `aim_serve_scenarios_stay_off_the_per_pr_lanes` pins
the tagging and that they stay reachable when the opt-in is set.
them, 45 minute cap) used up the whole budget. They still need
`@requires-engine:aim`, i.e. Docker reachable and a Radeon or Strix Halo GPU,
and `@requires-os:linux` (the CLI refuses AIM on native Windows), so they run
only where `E2E_INCLUDE_NIGHTLY=1` is set on such a host, which includes the
nightly Strix Halo Ubuntu lane (`e2e-gpu-nightly-strix`) and the Strix Ubuntu
job of an `include_nightly` dispatch of `e2e-selfhosted.yml` (which then gets a
125 minute cap). The Windows nightly lane (`e2e-gpu-nightly-strix-windows`) is
excluded by the OS tag; on the WSL2 nightly lane (`e2e-wsl-nightly`) they run
only if `docker info` succeeds in its distro, which the workflow does not set
up. `aim_serve_scenarios_stay_off_the_per_pr_lanes` pins the tagging and that
they stay reachable when the opt-in is set.

All three scenarios skip on a host without Docker; the `e2e-gpu-nightly-strix` job
logs whether Docker is reachable ("Report whether Docker is reachable").
Expand Down Expand Up @@ -3668,8 +3726,10 @@ this harness's platform filter, so selecting a Windows scenario by name or tag
on Linux runs it against Linux behavior instead of skipping it. The Windows
install steps also require a native Windows build before any CLI invocation.
[driver_host_guard.rs](../tests/e2e-cucumber/tests/driver_host_guard.rs) checks
that ordering without running an installer. Do not run these scenarios on a
non-Windows host.
that ordering without running an installer. The same test target also tests the
native-Windows check itself, and the check that the KFD topology step
(examine-17) runs to refuse any host but bare-metal Linux.
Do not run these scenarios on a non-Windows host.

`serve-16` (`@id:serve-absent-gpu-index-rejected`,
[model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature)) runs
Expand Down Expand Up @@ -4796,8 +4856,9 @@ selects nothing and the run fails with `0 scenarios executed` (see
replaces it.

Those four carry `@requires-no-gpu` (their premise is a host with no usable AMD
GPU), so they get their live coverage on the GitHub-hosted mock lane; serve-28,
serve-30 and serve-32 also carry `@requires-os:linux`.
GPU), so they get their live coverage on the GitHub-hosted mock lane. All four
also carry `@requires-os:linux`: the usable-GPU probe the no-GPU refusal keys on
exists only on Linux, so on native Windows that refusal never fires.

The fifth is the pre-gate's other half, and it is the one that needs real
hardware:
Expand Down
Loading
Loading