diff --git a/tests/e2e-cucumber/README.md b/tests/e2e-cucumber/README.md index eb88d0857e..23e19878c8 100644 --- a/tests/e2e-cucumber/README.md +++ b/tests/e2e-cucumber/README.md @@ -104,10 +104,18 @@ E2E_INCLUDE_LIFECYCLE=1 E2E_ONLY_LIFECYCLE=1 cargo xtask e2e ## Tags and per-scenario expectations -There is no tag-filter tiering. Each CI job runs the **whole** suite -(`cargo xtask e2e`, no `-t` filter); the harness resolves every scenario to -**pass / xfail / skip** at runtime from its capability tags plus the known-bug -matrix, then reconciles the actual result against that expectation. +Each CI job runs the suite as a whole (`cargo xtask e2e`, no `-t` filter); the +harness resolves every scenario to **pass / xfail / skip** at runtime from its +capability tags plus the known-bug matrix, then reconciles the actual result +against that expectation. + +Four tags tier the suite at runtime rather than by a `-t` filter, so a scenario +carrying one runs only where its env var is set: `@nightly` +(`E2E_INCLUDE_NIGHTLY`), `@lifecycle` (`E2E_INCLUDE_LIFECYCLE`), +`@requires-docker` (`E2E_INCLUDE_DOCKER`) and `@merge-queue` (`E2E_MERGE_QUEUE`, +set on the self-hosted merge-queue lanes). A `@merge-queue` scenario does not run +on ordinary per-PR CI, only on the self-hosted lanes in +[`.github/workflows/e2e-selfhosted.yml`](../../.github/workflows/e2e-selfhosted.yml). ### Naming @@ -147,6 +155,8 @@ Scenarios carry stable-id and capability tags: | `@requires-os:` | Premise is OS-specific; skip on other OSes. | | `@requires-oom-fault-injection` | Needs the `rocm` binary under test to carry the test-only `e2e-oom-fault-injection` hook, which fabricates a managed launch that runs out of GPU memory so the OOM guidance can be verified without a GPU. Unlike every other capability this one **cannot be probed** — no product command exposes it — so `xtask e2e` reports it via `ROCM_E2E_OOM_FAULT_INJECTION=1`, set only when xtask compiled the binary itself with the feature. A prebuilt `ROCM_CLI_BINARY` (the self-hosted lanes' shipping release build) has the hook compiled out, so these scenarios resolve to **skip** there and run on the mock lane. | | `@serve-timeout:` | Lengthen the serve-readiness wait for a genuinely slow serve (e.g. a large model). | +| `@requires-no-gpu` | Premise is a host with no usable AMD GPU (e.g. a refusal that only happens without one). The inverse of `@requires-gpu`; resolves to **skip** on a GPU host. | +| `@merge-queue` | Too expensive for per-PR CI (a real serve). Skipped unless `E2E_MERGE_QUEUE` is set, which every self-hosted lane in [`e2e-selfhosted.yml`](../../.github/workflows/e2e-selfhosted.yml) except the MI300X one does on a `merge_group` event. | | `@nightly` | Expensive scenario skipped by default; included when `E2E_INCLUDE_NIGHTLY=1`. | | `@lifecycle` | Expensive, OS-mutating release-lifecycle scenario (packaging + real installer + install/uninstall). Skipped by default; included when `E2E_INCLUDE_LIFECYCLE=1`. `E2E_ONLY_LIFECYCLE=1` selects only this set without bypassing expectation resolution. A caller can additionally fold specific non-`@lifecycle` scenarios into that restricted set via `E2E_ALSO_RUN_IDS` (comma-separated `@id`s), without tagging them `@lifecycle` — which would also pull them out of the full-suite lanes that don't set `E2E_INCLUDE_LIFECYCLE`. `ci.yml`'s Windows lane uses this to fold in the hermetic comfyui-01/02/03 scenarios and the native-Windows vLLM refusal scenarios serve-33/34/35. | @@ -159,6 +169,20 @@ matches a condition is expected to fail (xfail); if it then passes, that is an bug. See `src/expectation.rs` for the resolver and `expectations.toml`'s header for the condition grammar. +When a comment here points at product code, **cite a symbol, never a line +number** — `select_runtime_update_source` in [`apps/rocm/src/main.rs`](../../apps/rocm/src/main.rs), not +`main.rs:16427`. `main.rs` alone is many thousands of lines and +moves every week; nothing checks these numbers, so a stale one sends the reader +somewhere confidently wrong, which is worse than no pointer at all. + +When a scenario's comment block says **"Expected to FAIL"**, write it directly +above that scenario's `@id:` tag line with **no blank line between them**. A test +binds each such claim to the tags immediately beneath it and fails if the row has +since been deleted — so the prose cannot quietly come to say the opposite of what +the suite enforces. A blank line breaks that binding, and the same test then +fails naming the comment's own file and line rather than letting the scenario +drop out of the check unnoticed. + On a GPU host, a `serve` precondition that never publishes its model is relaunched once — but only for a scenario expected to pass; a known bug keeps its shortened `serve_timeout_secs` and fails on the first attempt. The stalled diff --git a/tests/e2e-cucumber/expectations.toml b/tests/e2e-cucumber/expectations.toml index fec5e0154f..47d3445ca3 100644 --- a/tests/e2e-cucumber/expectations.toml +++ b/tests/e2e-cucumber/expectations.toml @@ -164,3 +164,150 @@ bug = "EAI-7423" reason = "Lemonade managed serve reaches ready then shuts down immediately, so the benchmark never reaches a live model." serve_timeout_secs = 90 flaky = true + +# ── Contracts found by walking the public README end to end ───────────────── +# +# The rows below came out of one walkthrough of the README against a nightly build +# on an MI300X pod (EAI-8024 tracks the set). Most scenarios assert behaviour the +# walkthrough leads a reader to expect; a few are contracts the walkthrough implies +# rather than states (diagnose and fix agreeing, a usable device not being +# diagnosed as a permission failure, a second server avoiding a taken address), and +# one is a proposal the README does not make (uninstall stopping what it manages, +# below). Each row says the product does not do that yet — so the day one is fixed +# its row goes stale and deletes itself. + +# --- EAI-8020: `diagnose` and `fix` each carry their own copy of the +# render-group remedy, and the verification step has drifted between them +# (`diagnose` adds `ls -l /dev/kfd`). Both strings are constants, so this holds +# wherever the catalog runs on bare metal; `@requires-os:linux` and +# `@requires-bare-metal` keep it off Windows and WSL2, where the recipe does not apply. Remove when the two are one. --- +[["diagnose-and-fix-agree-on-how-to-verify"]] +when = {} +bug = "EAI-8020" +reason = "The diagnosis and the fix preview print different verification steps for the same remedy." + +# --- EAI-8009: with no `--model`, the local chat provider only looks for the +# built-in Lemonade assistant model, so a ready managed service running anything +# else is ignored and the user is told nothing is running. The scenario plants +# its own ready service, so this needs no hardware and holds on every lane. +# Remove when chatting locally uses whichever local server is ready. --- +[["chat-local-uses-the-running-server"]] +when = {} +bug = "EAI-8009" +reason = "The local chat provider only accepts the built-in assistant model, so a ready local server running any other model is not used." + +# --- EAI-7423, not EAI-8007. This scenario's premise is a REAL serve on whatever +# engine the host defaults to, so on a lemonade LINUX host it inherits the same +# managed-serve bug as its siblings above: the serve reaches ready and is shut +# down ~0.08s later, so the endpoint never becomes reachable and the precondition +# — not the contract under test — is what fails. Measured on Strix Halo Ubuntu: +# passed in run 196, failed in run 210 with "endpoint did not serve model +# Qwen3-0.6B after 2 attempt(s) of 300s", which is that signature exactly. +# +# Scoped and flagged identically to the other EAI-7423 rows: os=linux because +# native Windows lemonade serve works (it passes on Strix Windows), gfx* because +# a real GPU host is needed at all, and `flaky = true` because which of these +# pass varies run to run — a deterministic-XPASS row would fail that lane on +# unrelated PRs. Delete this row with the other EAI-7423 ones, not with EAI-8007. +[["serve-services-stop-reports-what-it-stopped"]] +when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*" } +bug = "EAI-7423" +reason = "Lemonade managed serve reaches ready then shuts down immediately, so the service this scenario needs to stop never comes up." +serve_timeout_secs = 90 +flaky = true + +# For the contract this scenario actually exists to hold: +# EAI-8007 has NO row on purpose. `services stop` reported stopping 0 processes +# for a service it did in fact stop on the pod, but neither fixture reproduces it +# here: a plain child process registered as a managed service was counted +# correctly in a one-off no-GPU check during development, and so was a real vLLM +# serve (MI300X, run 188). The shipped scenario is the real-serve one, which is +# `@requires-gpu @merge-queue`, so no blocking lane holds this contract. It ships +# as a guard rather than an expected failure, because a row would be a stale +# XPASS and would fail the lane. The ticket keeps the +# pod observation; what distinguishes that host is still unknown. + +# --- EAI-8014: a PROPOSED contract, not a README one. The README documents that +# `uninstall` warns it does not stop managed background processes; this row pins +# the stronger behaviour the ticket asks for — `uninstall` removes the managed +# files and reports completion without stopping the servers it manages, and takes +# the service records with it, so the supported way to stop them goes too. The +# warning is documented behaviour, so this stays a row only until the product and +# README change together. No hardware needed. Remove when uninstall stops what it +# manages first, or when the contract is dropped. --- +[["serve-uninstall-stops-what-it-manages"]] +when = {} +bug = "EAI-8014" +reason = "Uninstall reports completion while leaving the servers it manages running, and removes the records needed to stop them." + +# --- EAI-8020. The other half of the same divergence: `diagnose` names +# the device's actual owning group while the `fix` recipe is hard-coded to +# `render`, so the two propose different changes wherever the device's group is +# not `render`. Needs a real device node, hence the GPU lane. Measured on the +# MI300X lane (E2E self-hosted ROCm/rocm-cli run 192) and the Strix Halo Ubuntu lane +# (run 196), failing for this reason on both, with 0 XPASS. The group is substituted +# by a `stat` shim, so the outcome does not depend on the host. --- +[["diagnose-and-fix-agree-on-the-remedy-command"]] +when = {} +bug = "EAI-8020" +reason = "The diagnosis proposes adding the user to the device's own group while the fix is hard-coded to `render`." + +# --- EAI-8021. The render-group check only ever ADDS score for missing +# group membership and never credits access that demonstrably works, so a device +# the user can already read and write is still diagnosed as a permission problem. +# The scenario substitutes a world-writable device answer and asserts the +# inspection agrees before relying on it. It shims BOTH `stat` (device +# `crw-rw-rw- root root`) and `id` (groups `users` — none of the conventional +# ones), so the runner's own group membership cannot reach the result: the +# outcome is the same on every bare-metal Linux GPU host. If this XPASSes, the +# product changed; it is not a host difference, and NOT a reason to add +# `flaky = true`. Measured on MI300X (run 192) and Strix Halo Ubuntu (run 196), +# failing for this reason on both, with 0 XPASS. --- +[["diagnose-credits-a-usable-device"]] +when = {} +bug = "EAI-8021" +reason = "A device the user can already read and write is still offered as a group-permission cause." + +# --- EAI-8006: `--gpu auto` reports `selected none` on a machine that has a GPU, +# because the CLI cannot get an answer out of `amd-smi` — on the pod the bundled +# copy could not execute at all (its `#!/usr/bin/env python3` shebang found no +# interpreter), even though the CLI has a working interpreter recorded in its own +# config. +# +# os = "windows" is MEASURED, not guessed, and the three lanes separate the two +# candidate explanations cleanly: +# MI300X gfx943 linux PASSES (runs 190/191/192) +# Strix Halo Ubuntu gfx1151 linux PASSES (run 196) +# Strix Halo Windows gfx1151 windows FAILS (run 196) — `gpu: auto (selected none)` +# Same GPU family on both Strix lanes, so this is the OS, not the family — which +# is why the row is not scoped with therock_family the way +# `examine-both-forms-agree-on-gpu` is. Remove it when a Windows host names the +# device it picked. --- +[["serve-auto-gpu-selection-names-a-device"]] +when = { os = "windows" } +bug = "EAI-8006" +reason = "Letting the CLI choose a GPU reports that it selected none on a Windows machine that has one." +serve_timeout_secs = 180 + +# --- EAI-8016. The serve path has no address-availability check: the +# default port is used whether or not something already holds it, so a second +# server is handed the address the first is on. The scenario holds the address +# itself rather than starting a second real server. Measured on all three GPU +# lanes before the scenario became `@merge-queue`: MI300X (run 192), Strix Halo +# Ubuntu (run 196) and Strix Halo Windows (runs 196 and 210). It failed for this +# reason on each, Windows included, with 0 XPASS. --- +[["serve-second-server-gets-a-free-port"]] +when = {} +bug = "EAI-8016" +reason = "A server started without an address is given the default one even when it is already in use." +serve_timeout_secs = 120 + +# EAI-8017 and EAI-8018 have NO rows and NO scenarios, because neither can be +# exercised on any lane available here. Both need a working Lemonade install, +# and Lemonade's own launcher cannot start on either lane: on the MI300X runner +# `lemond` wants GLIBC_2.38 / GLIBCXX_3.4.32, which that OS does not have +# (measured, runs 190/191), and in a plain container it wants +# libdrm_amdgpu.so.1. Both die before reaching the behaviour under test, so a +# scenario would pass vacuously — worse than no scenario, because it would read +# as coverage. They were reproduced by hand on the pod (ubuntu 24.04) and stay +# on their tickets until a lane can run Lemonade. diff --git a/tests/e2e-cucumber/features/chat.feature b/tests/e2e-cucumber/features/chat.feature index 3f83c17592..c377bc2bf0 100644 --- a/tests/e2e-cucumber/features/chat.feature +++ b/tests/e2e-cucumber/features/chat.feature @@ -150,3 +150,14 @@ Feature: Chat and endpoint detection Then the planning assistant was asked exactly once And the plan text returned is the one the planning assistant gave And the action returned is the one the planning assistant gave + + # Expected to FAIL. Scenario 8 names the model and works; this one leaves the + # choice to the CLI and is told nothing is running — on the very machine whose + # services list shows a ready local server. Asking to chat locally should use + # the local server that is there, whichever model the user chose to serve. + @id:chat-local-uses-the-running-server + Scenario: chat-13 - Chatting locally uses the local server that is already running + Given a model is being served + And the model is registered with the CLI + When the user sends a one-shot chat prompt without naming a model + Then the CLI prints the assistant's reply diff --git a/tests/e2e-cucumber/features/diagnose.feature b/tests/e2e-cucumber/features/diagnose.feature index 01bfafb32a..7ebf7d4c83 100644 --- a/tests/e2e-cucumber/features/diagnose.feature +++ b/tests/e2e-cucumber/features/diagnose.feature @@ -748,3 +748,51 @@ Feature: Diagnosing failures and listing fixes Given a machine with an AMD GPU When the user diagnoses this machine in machine-readable form, with no symptom Then the diagnosis lists the HIP SDK cause only when the inspection finds something that needs it + + # Expected to FAIL. `diagnose` and `fix` are two views of the same remedy: the + # first tells the user what will put the machine right, while the second is the + # command they are told to run. They must not leave the user with two different + # definitions of what proves the remedy worked. + @id:diagnose-and-fix-agree-on-how-to-verify @requires-os:linux @requires-bare-metal + Scenario: diagnose-49 - Diagnosing a problem and previewing its fix agree on how to verify it + Given a user who hit a device-permission failure + When the user compares the diagnosis with the matching fix preview + Then both give the same way to verify that the fix worked + + # A proposed remedy has to name a group that actually exists on the machine. + # When the device belongs to a group the local group database does not name, + # `stat` answers the literal `UNKNOWN`, and the diagnosis used to print that + # lookup failure as if it were a group the user could join. Since fixed on main + # (the lookup failure now falls back to the conventional group), so this ships + # as a guard rather than an expected failure. + @id:diagnose-commands-name-a-real-group @requires-gpu @requires-bare-metal @requires-os:linux + Scenario: diagnose-50 - Every group named in a diagnosis is one the machine recognises + Given the GPU device belongs to a group the machine cannot name + When the user asks the CLI to diagnose a device-permission failure + Then every group named in the remedy exists on the machine + + # Expected to FAIL on every bare-metal Linux GPU host. The device's group is + # substituted (a `stat` shim names a real group other than the hard-coded + # default), so the premise does not depend on the host, and the CLI's own + # inspection is checked to see it first. The diagnosis promises one command + # and its matching fix previews another. This is distinct from scenario + # diagnose-49: even after the verify text agrees, the actual change must agree + # too. + @id:diagnose-and-fix-agree-on-the-remedy-command @requires-gpu @requires-bare-metal @requires-os:linux + Scenario: diagnose-51 - Diagnosing a problem and previewing its fix agree on the remedy command + Given the GPU device belongs to a recognised non-default group + When the user compares the diagnosis with the matching fix preview + Then both give the same command for applying the remedy + + # Expected to FAIL on every bare-metal Linux GPU host. The usable device is + # substituted, not a property of the host: `stat` and `id` shims make the + # device world-readable and writable and put the user in none of the + # conventional groups, and the CLI's own inspection is checked to agree first. + # Membership in one conventional group is only a means to access the device, + # not the outcome. A diagnosis should credit the observable access the user + # has instead of recommending a permission repair for a usable device. + @id:diagnose-credits-a-usable-device @requires-gpu @requires-bare-metal @requires-os:linux + Scenario: diagnose-52 - A usable GPU device is not diagnosed as a permission failure + Given the user can already read and write the GPU device + When the user asks the CLI to diagnose the machine + Then adding the user to a device group is not offered as a cause diff --git a/tests/e2e-cucumber/features/examine.feature b/tests/e2e-cucumber/features/examine.feature index 83d89d2306..bee280e152 100644 --- a/tests/e2e-cucumber/features/examine.feature +++ b/tests/e2e-cucumber/features/examine.feature @@ -469,3 +469,29 @@ Feature: GPU detection and system inspection Given a machine with a ROCm install that was not set up by the CLI When the user inspects the system in machine-readable form Then the inspection reports a current install-method name + + # The help's own worked examples are the first thing a new user copies, so a + # model named there has to be one this CLI can actually serve. The check + # accepts either form the README documents — a name the model listing knows, + # or an explicit `owner/repo` reference — and so does not prescribe which + # model the examples should use. + # + # Since fixed on main (EAI-8011 swapped the unresolvable example name for a + # real catalog alias), so this ships as a guard rather than an expected + # failure. + @id:examine-help-serve-example-names-a-resolvable-model + Scenario: examine-31 - Every model the help offers as an example is one the CLI can resolve + When the user reads the serve examples the help offers + Then every model named there is one the CLI can resolve + + # Running the CLI with no subcommand and running its dashboard command are two + # different things. When the help describes them as the same one, a reader has + # no way to learn what the plain command does or that there is anything else + # to reach. + # + # Since fixed on main (the same EAI-8011 change now describes bare `rocm` as + # the launcher), so this ships as a guard rather than an expected failure. + @id:examine-help-describes-the-default-command + Scenario: examine-32 - The help tells the two ways of opening a screen apart + When the user asks for help + Then running the CLI with no subcommand is not described as the dashboard command diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 539644f521..0271e61779 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -670,3 +670,78 @@ Feature: Model serving # suppression was untested: deleting the guard left the scenario green and a # `rocm daemon` outliving the checkout. And no background automation daemon was left behind + + # `serve` advertises a set of device policies, and the list the user is offered + # has to be the list the command accepts. Runs on the no-GPU lane, where a + # refusal that names the policy itself is cleanly distinguishable from the + # ordinary "this machine has no GPU" refusal every policy gets there. + # + # Since fixed on main (EAI-8019 made `--device` a clap `ValueEnum`, so the + # advertised list and the accepted list are one list), so this ships as a + # guard rather than an expected failure. + @id:serve-rejects-no-advertised-device-policy @requires-no-gpu + Scenario: serve-42 - Every device policy the serve command offers is one it accepts + When the user serves a model under each device policy the command offers + Then no policy is refused for being that policy + + # A guard, not a finding. Stopping a running server was reported as having + # stopped nothing on the pod this set came from, but neither fixture tried here + # reproduces that. Why it ships as a guard rather than an expected failure, and + # what was measured on which lane, is recorded once at the EAI-8007 note in + # `expectations.toml` — not restated here, so the two cannot drift. + # + # It holds that contract unconditionally EXCEPT on the lemonade-default Linux + # GPU lane, where a separate serve bug (EAI-7423) stops the service coming up + # at all and a `flaky` row covers the precondition failing; see that row. + # + # @merge-queue because it serves a real model and the Strix Halo lanes have + # little headroom left on main alone, so three more real serves follow + # serve-06/07/08 onto the merge-queue path where the budget for them lives. + @id:serve-services-stop-reports-what-it-stopped @requires-gpu @merge-queue + Scenario: serve-43 - Stopping a running server reports that it stopped it + Given a managed runtime is active + And a model is being served on GPU + When the user stops the server that is running + Then the CLI reports that it stopped a process + And the stopped server no longer answers + + # Expected to FAIL. A PROPOSED contract, not one the README promises: the + # README says `uninstall` warns that it does not stop managed background + # processes. This scenario asserts the stronger behaviour — that removing the + # CLI's managed files reports completion while the server it was managing is + # left running, and with the records gone, the supported way to stop it has + # been removed along with them. The removal keeps the binaries and the cache, + # so it reaches only this scenario's own config and data directories. It is + # still a real `rocm uninstall`, so it runs with the other `@lifecycle` + # scenarios rather than on every Linux lane. + @id:serve-uninstall-stops-what-it-manages @requires-os:linux @lifecycle + Scenario: serve-44 - Removing the CLI's managed files stops the servers it manages + Given a local server this machine manages is running + When the user removes the CLI's managed files + Then the server is no longer running + And the removal is reported as complete + + # Expected to FAIL on Windows. Asking the CLI to choose a GPU on a machine that + # has one must end with a device chosen: reporting that it selected none and + # carrying on leaves the user unable to tell which GPU their model will run on, + # or whether it will run on one at all. Both Linux lanes name the device, so + # this scenario also guards them. + # @merge-queue for the serve-cost reason on `@id:serve-services-stop-reports-what-it-stopped`. + @id:serve-auto-gpu-selection-names-a-device @requires-gpu @merge-queue + Scenario: serve-45 - Letting the CLI choose the GPU names the device it chose + Given a managed runtime is active + And a machine with an AMD GPU + When the user serves a model letting the CLI choose the GPU + Then the plan names the device it chose + + # Expected to FAIL on a GPU host. A second server started while the usual + # address is already taken is handed that same address anyway, so it collides + # with the server already there. Either outcome is fine — pick a free address, + # or say the usual one is busy — but silently reusing it is not. + # @merge-queue for the serve-cost reason on `@id:serve-services-stop-reports-what-it-stopped`. + @id:serve-second-server-gets-a-free-port @requires-gpu @merge-queue + Scenario: serve-46 - A second server does not take an address already in use + Given a managed runtime is active + And the address a new server would use is already taken + When the user serves a model without choosing an address + Then the new server does not try to use the taken address diff --git a/tests/e2e-cucumber/src/expectation.rs b/tests/e2e-cucumber/src/expectation.rs index df78c81db4..6f09e99c31 100644 --- a/tests/e2e-cucumber/src/expectation.rs +++ b/tests/e2e-cucumber/src/expectation.rs @@ -358,6 +358,11 @@ impl Expectations { self.by_id.get(id).map_or(&[], Vec::as_slice) } + /// Every scenario id this file declares a condition for. + pub fn declared_ids(&self) -> impl Iterator { + self.by_id.keys().map(String::as_str) + } + /// The shortest `serve_timeout_secs` among the conditions matching this host /// for a scenario, if any. A known bug that manifests as a serve which never /// becomes ready should fail fast rather than burn the full cold-start window @@ -2245,6 +2250,368 @@ flaky = true )); } + /// The `@id:` tags declared on tag lines. Tag lines only, the way + /// `scenarios_of` in tests/feature_naming.rs reads them: this set decides + /// whether a row is an orphan, so an `@id:` written inside a `#` comment + /// must not vouch for a row that measures nothing. + fn tag_ids(text: &str) -> std::collections::BTreeSet { + let mut ids = std::collections::BTreeSet::new(); + for line in text.lines() { + let line = line.trim(); + if !line.starts_with('@') { + continue; + } + for tag in line.split_whitespace() { + if let Some(id) = tag.strip_prefix("@id:") { + ids.insert(id.to_owned()); + } + } + } + ids + } + + #[test] + fn a_commented_out_id_does_not_declare_a_scenario() { + let text = " # @id:ghost\n @requires-os:linux @id:real\n Scenario: x\n"; + let ids = tag_ids(text); + assert!(ids.contains("real"), "{ids:?}"); + assert!(!ids.contains("ghost"), "{ids:?}"); + } + + /// A row whose scenario has been renamed or deleted never matches anything, + /// so it can neither xfail nor go stale: it just sits there stating a bug + /// that nothing measures. Nothing at runtime notices — resolution is keyed + /// FROM the scenario TO this file, never the other way — so the orphan is + /// caught here instead. + #[test] + fn every_expectation_row_names_a_scenario_that_exists() { + let mut scenario_ids = std::collections::BTreeSet::new(); + for path in feature_files() { + let text = std::fs::read_to_string(&path) + .unwrap_or_else(|e| panic!("could not read {}: {e}", path.display())); + scenario_ids.extend(tag_ids(&text)); + } + assert!( + !scenario_ids.is_empty(), + "read no `@id:` tags at all, so this check would pass vacuously" + ); + + let m = Expectations::parse(include_str!("../expectations.toml")).unwrap(); + // Both sides, not just the scenarios. An `expectations.toml` that + // declared nothing would make `orphans` empty and this check green — + // it would be reporting "no stale rows" about a file it never read. + // A sibling test would catch the empty parse, but this one should not + // depend on that to mean what it says. + let declared: Vec<&str> = m.declared_ids().collect(); + assert!( + !declared.is_empty(), + "expectations.toml declared no rows at all, so this check would pass vacuously" + ); + let orphans: Vec<&str> = declared + .iter() + .copied() + .filter(|id| !scenario_ids.contains(*id)) + .collect(); + assert!( + orphans.is_empty(), + "expectations.toml declares rows for scenarios that no longer exist: {orphans:?}\n\ + Delete the row, or fix the id if the scenario was renamed." + ); + } + + /// Every `.feature` file the suite runs. + /// + /// `features/` is flat, as `scenarios_of` in tests/feature_naming.rs also + /// assumes, so this does not recurse. It refuses to run rather than quietly + /// covering less if that ever stops being true: a subdirectory would simply + /// vanish from the checks below, and a check that silently stops looking at + /// something is worse than no check at all. + fn feature_files() -> Vec { + feature_files_in(&std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("features")) + } + + /// The directory scan itself, split out so the subdirectory refusal can be + /// exercised against a temporary tree — against the real `features/` it + /// could only ever fire by someone breaking the repository. + fn feature_files_in(dir: &std::path::Path) -> Vec { + let mut paths = Vec::new(); + for entry in std::fs::read_dir(dir).expect("no features directory") { + let path = entry.expect("unreadable features directory entry").path(); + assert!( + !path.is_dir(), + "features/ has grown a subdirectory ({}), which this scan does not descend \ + into — make it recursive, here and in tests/feature_naming.rs, before moving \ + any .feature file into one", + path.display() + ); + if path.extension().is_some_and(|ext| ext == "feature") { + paths.push(path); + } + } + assert!( + !paths.is_empty(), + "found no .feature files in {}", + dir.display() + ); + paths + } + + #[test] + fn the_feature_scan_takes_the_flat_files_it_finds() { + let dir = tempfile::tempdir().expect("no temp dir"); + std::fs::write(dir.path().join("a.feature"), "Feature: a\n").unwrap(); + std::fs::write(dir.path().join("notes.md"), "ignored\n").unwrap(); + // A file named exactly `.feature` has no extension, so this scan skips + // it. Pinned on both sides — `tests/feature_naming.rs` once matched on a + // `.feature` string suffix instead, which would have taken this one and + // left the two scans covering different sets. + std::fs::write(dir.path().join(".feature"), "not a feature file\n").unwrap(); + let found = feature_files_in(dir.path()); + assert_eq!(found.len(), 1, "{found:?}"); + assert!(found[0].ends_with("a.feature"), "{found:?}"); + } + + #[test] + #[should_panic(expected = "found no .feature files")] + fn the_feature_scan_refuses_a_directory_with_no_feature_files() { + // The assertion this pins reports an emptied `features/` once, at the + // scan, naming the directory. It is not what stops the two callers + // agreeing with everything — both already guard themselves ("read no + // `@id:` tags at all, so this check would pass vacuously" and its + // counterpart over claimed ids), so they fail either way; this just + // fails first and says what actually went wrong. + // + // Unreachable against the real `features/`, so pinned here — and + // pinned the same way in `tests/feature_naming.rs`, whose scan carries + // the same assertion. + // + // The fixture writes a non-`.feature` file on purpose: what the scan + // refuses is an empty RESULT, not an empty directory. + let dir = tempfile::tempdir().expect("no temp dir"); + std::fs::write(dir.path().join("notes.md"), "ignored\n").unwrap(); + let _ = feature_files_in(dir.path()); + } + + #[test] + #[should_panic(expected = "has grown a subdirectory")] + fn the_feature_scan_refuses_a_subdirectory_rather_than_skipping_it() { + // The branch that stops this scan quietly covering less than it claims. + // Unreachable against the real `features/`, so it is pinned here. + let dir = tempfile::tempdir().expect("no temp dir"); + std::fs::write(dir.path().join("a.feature"), "Feature: a\n").unwrap(); + std::fs::create_dir(dir.path().join("nested")).unwrap(); + let _ = feature_files_in(dir.path()); + } + + /// What one feature file claims about expected failures. + struct FailureClaims { + /// Scenario ids introduced by a comment block claiming "Expected to + /// FAIL". + ids: Vec, + /// 1-indexed lines whose claim reached no scenario id. + unbound: Vec, + } + + /// Scan one feature file for "Expected to FAIL" claims. + /// + /// A scenario is written comment block → tag line(s) → `Scenario:`, and the + /// block is contiguous with the tags it introduces, so a blank line ends a + /// claim's reach: a file header or a note left in a previous scenario's + /// body describes something other than whatever comes next. + /// + /// A claim that reaches nothing is REPORTED, not dropped. Dropping it is + /// how a scenario leaves this check's coverage without anyone noticing — + /// one blank line between a comment and its tag line would be enough, and + /// a blank line is the same character the file already uses to separate + /// scenarios. Reporting it also keeps the failure honest about where the + /// problem is: the comment, not the scenario that happens to follow it. + /// + /// This is line-shaped, not a Gherkin parse. It assumes the phrase appears + /// only in real `#` comments (not inside a docstring or table cell) and + /// that `Scenario:`/`Scenario Outline:` are the only block terminators — + /// `Background:` is not treated as one. Neither construct exists anywhere in + /// `features/` today, and both would fail loudly (an unbound claim) rather + /// than silently if one appeared, but a reader adding either should revisit + /// this scan first. + fn claimed_failure_ids(text: &str) -> FailureClaims { + let (mut ids, mut unbound) = (Vec::new(), Vec::new()); + let (mut claim_lines, mut pending_ids): (Vec, Vec) = + (Vec::new(), Vec::new()); + for (index, line) in text.lines().enumerate() { + let line = line.trim(); + if line.is_empty() { + // Clearing `pending_ids` here means a blank line between a tag + // line and its `Scenario:` would drop the ids too. Deliberate, + // and no longer silent: the claim above them then reaches + // nothing and is reported as unbound. + unbound.append(&mut claim_lines); + pending_ids.clear(); + } else if line.starts_with('#') { + if line.contains("Expected to FAIL") { + claim_lines.push(index + 1); + } + } else if line.starts_with('@') { + for tag in line.split_whitespace() { + if let Some(id) = tag.strip_prefix("@id:") { + pending_ids.push(id.to_owned()); + } + } + } else if line.starts_with("Scenario:") || line.starts_with("Scenario Outline:") { + if !claim_lines.is_empty() { + if pending_ids.is_empty() { + unbound.append(&mut claim_lines); + } else { + ids.extend(pending_ids.iter().cloned()); + } + } + claim_lines.clear(); + pending_ids.clear(); + } + } + unbound.append(&mut claim_lines); + FailureClaims { ids, unbound } + } + + #[test] + fn a_contiguous_comment_block_binds_to_its_scenario() { + let claims = claimed_failure_ids( + "Feature: f\n\ + \n\ + \x20 # Expected to FAIL. The thing is broken.\n\ + \x20 @id:thing-is-broken\n\ + \x20 Scenario: f-01 - The thing works\n\ + \x20 Given a thing\n", + ); + assert_eq!(claims.ids, ["thing-is-broken"]); + assert!(claims.unbound.is_empty(), "{:?}", claims.unbound); + } + + #[test] + fn a_file_header_mentioning_the_phrase_condemns_no_scenario() { + let claims = claimed_failure_ids( + "# Historical note: f-01 was once Expected to FAIL.\n\ + \n\ + Feature: f\n\ + \n\ + \x20 @id:thing-works\n\ + \x20 Scenario: f-01 - The thing works\n\ + \x20 Given a thing\n", + ); + // Not attributed to `thing-works`, the scenario declared below it. + assert!(claims.ids.is_empty(), "{:?}", claims.ids); + // But not swallowed either: it is reported against its own line. + assert_eq!(claims.unbound, [1]); + } + + #[test] + fn a_comment_a_blank_line_from_its_tag_is_reported_not_dropped() { + let claims = claimed_failure_ids( + "Feature: f\n\ + \n\ + \x20 # Expected to FAIL. The thing is broken.\n\ + \n\ + \x20 @id:thing-is-broken\n\ + \x20 Scenario: f-01 - The thing works\n\ + \x20 Given a thing\n", + ); + // The scenario has silently left the guard's coverage in every earlier + // version of this scan. It must not do so silently. + assert!(claims.ids.is_empty(), "{:?}", claims.ids); + assert_eq!(claims.unbound, [3]); + } + + #[test] + fn a_claim_does_not_inherit_a_stale_id_from_across_a_blank_line() { + // Pins the `pending_ids.clear()` half of the blank-line arm, which the + // other scan tests leave dead: without it `stale-id` would still be + // pending when the claim below is bound, and the claim would be + // attributed to a scenario it says nothing about — silently, and with + // the row check then vouching for the wrong id. + let claims = claimed_failure_ids( + "Feature: f\n\ + \n\ + \x20 @id:stale-id\n\ + \n\ + \x20 # Expected to FAIL. The thing is broken.\n\ + \x20 Scenario: f-01 - The thing works\n", + ); + assert!(claims.ids.is_empty(), "{:?}", claims.ids); + assert_eq!(claims.unbound, [5]); + } + + #[test] + fn a_claim_dangling_at_end_of_file_is_reported_not_dropped() { + // The `Scenario:` that would have bound this claim was deleted, leaving + // the comment as the last thing in the file. Covers the flush after the + // loop, which the blank-line and `Scenario:` cases never reach. + let claims = claimed_failure_ids( + "Feature: f\n\ + \n\ + \x20 # Expected to FAIL. The thing is broken.\n\ + \x20 @id:thing-is-broken\n", + ); + assert!(claims.ids.is_empty(), "{:?}", claims.ids); + assert_eq!(claims.unbound, [3]); + } + + /// The inverse orphan: a scenario whose comment still says "Expected to + /// FAIL" after its row has been deleted. + /// + /// `expectations.toml` owns that state, but the feature files restate it in + /// prose for the reader, and prose does not move when a row does. When a bug + /// is fixed the harness flips correctly and silently — leaving a comment that + /// tells the next reader the opposite of what the suite now enforces. Four + /// rows were deleted while this file was being written, so the drift is not + /// hypothetical. + #[test] + fn every_expected_to_fail_comment_still_has_a_row() { + let mut claimed = Vec::new(); + let mut unbound = Vec::new(); + for path in feature_files() { + let text = std::fs::read_to_string(&path) + .unwrap_or_else(|e| panic!("could not read {}: {e}", path.display())); + let scan = claimed_failure_ids(&text); + claimed.extend(scan.ids); + unbound.extend( + scan.unbound + .into_iter() + .map(|number| format!("{}:{number}", path.display())), + ); + } + // Before asking whether the claims still have rows, insist that every + // claim reached a scenario at all. A claim that reaches none is not + // harmless: it means that scenario is no longer covered by this check, + // and the whole point of the check is that nothing drops out of it + // quietly. Naming the file and line sends the reader to the comment + // itself rather than to whichever scenario happened to follow it. + assert!( + unbound.is_empty(), + "these lines say 'Expected to FAIL' but bind to no scenario id, so the scenario they \ + describe is not covered by this check: {unbound:?}\n\ + A claim reaches only the tag line directly beneath it — no blank line in between. \ + Move the comment against its tag line, or reword it if it is not a claim about a \ + specific scenario." + ); + assert!( + !claimed.is_empty(), + "found no scenario claiming 'Expected to FAIL', so this check would pass vacuously" + ); + + let m = Expectations::parse(include_str!("../expectations.toml")).unwrap(); + let declared: std::collections::BTreeSet<&str> = m.declared_ids().collect(); + let lying: Vec<&String> = claimed + .iter() + .filter(|id| !declared.contains(id.as_str())) + .collect(); + assert!( + lying.is_empty(), + "these scenarios still say 'Expected to FAIL' but have no expectations row, so the \ + comment asserts the opposite of what the suite enforces: {lying:?}\n\ + Delete the comment along with the row — the bug it described is fixed, and the \ + scenario now guards the fix." + ); + } + #[test] fn glob_matches_family() { assert!(glob_match("gfx94*", "gfx942")); @@ -2252,4 +2619,132 @@ flaky = true assert!(!glob_match("gfx94*", "gfx1151")); assert!(glob_match("gfx1151", "gfx1151")); } + + /// `is_wsl` is what tells a WSL2 host from bare metal — `os_family` is + /// `linux` on both. A row conditioned on it must hold on one and not the + /// other, in both polarities, or the key is decorative. + #[test] + fn a_row_conditioned_on_wsl_holds_only_there() { + let d = decl(&["id:x"]); + for (value, xfails_on) in [(true, "wsl2"), (false, "strix-ubuntu")] { + let m = Expectations::parse(&format!( + "[[\"x\"]]\nwhen = {{ is_wsl = {value} }}\nbug = \"EAI-1\"\n\ + reason = \"r\"\n" + )) + .unwrap(); + let other = if value { "strix-ubuntu" } else { "wsl2" }; + assert!( + matches!( + resolve(&d, &cap(xfails_on), &m, Included::default()), + Expectation::ExpectXfail { .. } + ), + "is_wsl = {value} should hold on {xfails_on}" + ); + assert_eq!( + resolve(&d, &cap(other), &m, Included::default()), + Expectation::ExpectPass, + "is_wsl = {value} should not hold on {other}" + ); + } + } + + /// The typo this guards against is the one the `Condition` doc describes: a + /// misspelled key parses to an all-`None` condition, which matches every + /// host, turning one row into a permanent suite-wide xfail. Asserted on both + /// structs, since `XfailEntry` carries the row's own keys. + #[test] + fn a_misspelled_key_is_rejected_rather_than_matching_everything() { + let typo_in_condition = Expectations::parse( + "[[\"x\"]]\nwhen = { engine = \"vllm\" }\nbug = \"EAI-1\"\nreason = \"r\"\n", + ); + assert!( + typo_in_condition.is_err(), + "`engine` is not `effective_engine`; accepting it would xfail every platform" + ); + let typo_in_entry = Expectations::parse( + "[[\"x\"]]\nwhen = { os = \"linux\" }\nbug = \"EAI-1\"\nreason = \"r\"\n\ + flakey = true\n", + ); + assert!( + typo_in_entry.is_err(), + "`flakey` is not `flaky`; accepting it would silently drop the tolerance it asks for" + ); + } + + /// The engine gate is deliberately conjoined with `requires_gpu`: a scenario + /// that never serves must not be skipped because some engine cannot start. + /// Both sides are asserted on the same host and the same unstartable engine + /// — vLLM on Windows, which is the real case — so dropping the conjunct + /// changes exactly one of them. + #[test] + fn only_gpu_scenarios_are_gated_on_the_engine_starting() { + let m = Expectations::parse("").unwrap(); + let host = cap("strix-windows"); + assert!( + !host.engine_available("vllm"), + "this test needs an engine that cannot start here" + ); + let gpu = decl(&["id:x", "requires-gpu", "requires-engine:vllm"]); + assert!( + matches!( + resolve(&gpu, &host, &m, Included::default()), + Expectation::Skip { .. } + ), + "a GPU scenario cannot run where its engine will not start" + ); + let no_gpu = decl(&["id:x", "requires-engine:vllm"]); + assert_eq!( + resolve(&no_gpu, &host, &m, Included::default()), + Expectation::ExpectPass, + "a scenario that never serves has no stake in whether the engine starts" + ); + } + + /// `ResolvedScenario` is what the platform report is built from, so a row + /// that loses its bug id or its flaky flag here loses it in the artifact a + /// reader uses to tell an expected failure from a real one. + #[test] + fn the_report_row_carries_each_expectations_own_metadata() { + let xfail = ResolvedScenario::new( + "x", + "f", + "s", + "vllm", + &Expectation::ExpectXfail { + bug: "EAI-1".into(), + reason: "r".into(), + flaky: true, + }, + ); + assert_eq!(xfail.bug.as_deref(), Some("EAI-1")); + assert_eq!(xfail.reason.as_deref(), Some("r")); + assert!(xfail.flaky); + assert_eq!(xfail.expected, "xfail"); + assert_eq!(xfail.id, "x"); + assert_eq!(xfail.feature, "f"); + assert_eq!(xfail.scenario, "s"); + assert_eq!(xfail.effective_engine, "vllm"); + + // A skip keeps its reason but has no bug, and a pass has neither — the + // two `None`s a reader distinguishes an expected failure by. + let skipped = ResolvedScenario::new( + "x", + "f", + "s", + "vllm", + &Expectation::Skip { + reason: "no gpu".into(), + }, + ); + assert_eq!(skipped.expected, "skip"); + assert_eq!(skipped.bug, None); + assert_eq!(skipped.reason.as_deref(), Some("no gpu")); + assert!(!skipped.flaky); + + let passing = ResolvedScenario::new("x", "f", "s", "vllm", &Expectation::ExpectPass); + assert_eq!(passing.expected, "pass"); + assert_eq!(passing.bug, None); + assert_eq!(passing.reason, None); + assert!(!passing.flaky); + } } diff --git a/tests/e2e-cucumber/src/mock_server.rs b/tests/e2e-cucumber/src/mock_server.rs index 3a2844d87f..7363fe6709 100644 --- a/tests/e2e-cucumber/src/mock_server.rs +++ b/tests/e2e-cucumber/src/mock_server.rs @@ -754,6 +754,12 @@ impl MockServer { /// record shape. #[derive(Debug, Clone, Copy)] pub struct ServiceRecordOptions { + /// Id of the recorded service, used both as the record's own `service_id` + /// and as its file name. Defaults to `e2e-mock`, which the cucumber + /// teardown recognises as the planted record with no process behind it and + /// leaves alone; a scenario that registers a REAL process it owns passes its + /// own id so that teardown also tries to stop it. + pub service_id: &'static str, /// Engine name recorded for the service. Defaults to `"vllm"`, the engine /// the mock server stands in for. A scenario that needs the CLI to match /// this record against a *different* engine — e.g. the self-managing @@ -801,6 +807,7 @@ pub struct ServiceRecordOptions { impl Default for ServiceRecordOptions { fn default() -> Self { Self { + service_id: "e2e-mock", engine: "vllm", status: "ready", startup_phase: None, @@ -840,11 +847,12 @@ pub fn write_service_record_with( options: ServiceRecordOptions, ) { std::fs::create_dir_all(services_dir).expect("failed to create services dir"); + let manifest = services_dir.join(format!("{}.json", options.service_id)); // The CLI only extracts host:port from `endpoint_url` and appends // `/v1/models` for its readiness probe, which the mock serves. let record = json!({ - "service_id": "e2e-mock", + "service_id": options.service_id, "engine": options.engine, "model_ref": model, "canonical_model_id": model, @@ -868,13 +876,13 @@ pub fn write_service_record_with( "engine_recipe_json": null, "restart_count": 0, "last_restart_unix_ms": null, - "manifest_path": services_dir.join("e2e-mock.json"), - "log_path": services_dir.join("e2e-mock.log"), - "engine_state_path": services_dir.join("e2e-mock.state.json"), + "manifest_path": manifest, + "log_path": services_dir.join(format!("{}.log", options.service_id)), + "engine_state_path": services_dir.join(format!("{}.state.json", options.service_id)), "created_at_unix_ms": 1_700_000_000_000_u64, }); std::fs::write( - services_dir.join("e2e-mock.json"), + &manifest, serde_json::to_vec_pretty(&record).expect("failed to serialize record"), ) .expect("failed to write service record"); diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 2cb763cf3b..d78db7d2e5 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -75,6 +75,26 @@ pub struct E2eWorld { pub cli_output: Option, pub cli_outputs: Option>, pub cli_stderr: Option, + /// Output of a SECOND command, for a scenario whose whole point is that two + /// commands agree — a diagnosis against a fix preview, help examples against + /// the model listing, the human `examine` report against its `--json` form. + /// Usually that command's stdout; where a scenario does not care which + /// stream carried a line, it may hold both concatenated (the fix preview + /// does). + /// + /// Put it HERE rather than in `cli_stderr`. The rule that slot keeps is + /// "this is what the command wrote to stderr", and a reader settles it with + /// `grep -rnE '^\s*world\.cli_stderr = ' tests/e2e-cucumber/` — anchored to + /// the assignment so this comment is not itself a hit. Every result should + /// be storing a stderr capture. Borrowing that slot for a second command's + /// output breaks the rule at a distance: the step reading it back looks like + /// it is making a claim about stderr. This field exists so nothing has to. + /// + /// Readability only — NOT a bug fix. Every assertion that moved here would + /// pass unchanged if the same strings went back into `cli_stderr`; what + /// changes is whether a reader of those steps is misled about which stream + /// they are asserting on. + pub cli_other_output: Option, pub cli_rc: Option, /// Wall-clock time the last measured `rocm` invocation took, for scenarios /// where the *duration* bounds the behaviour under test — @@ -189,12 +209,88 @@ pub struct E2eWorld { /// refusal envelope (it carries a `refused` key). Recorded when the preview /// is read, so a later step can check the card against what the CLI said. pub dash_report_refused: Option, + /// An address a scenario is deliberately holding, so a serve started without + /// one meets an address that is already in use. Released with the World. + pub occupied_address: Option, + /// A `PATH` a scenario prepared for the `rocm` invocations that follow (see + /// [`stat_shim_path`]). Carried on the World because the situation it sets up + /// belongs to the Given, while the invocations that must see it are in later + /// steps. + pub path_override: Option, + /// Plain child processes a scenario spawned itself and registered with the + /// CLI as managed services, so a step can assert the PRODUCT stopped them. + /// + /// Killed in `Drop` as a panic-path backstop only — never as the thing under + /// test. The scenarios that use these assert the child is gone *before* + /// teardown runs, because a `Drop` that cleans up after the product failed + /// would turn the very defect being pinned into a pass. + pub owned_processes: Vec, /// The `rocmd run` a daemon-driven automations scenario started, with the /// local webhook it listens on. Held here so `Drop` can kill and reap it /// whatever the scenario's outcome. pub automation_daemon: Option, } +/// A child process this scenario owns, kept alive until the product is asked to +/// stop it (or teardown reaps it). +#[derive(Debug)] +pub struct OwnedProcess { + pub child: std::process::Child, +} + +impl OwnedProcess { + /// The child's PID. + pub fn pid(&self) -> u32 { + self.child.id() + } + + /// Whether the process is still running. + /// + /// This handle is the child's parent, so reaping it is what turns "signalled + /// and gone" into an observable exit — a terminated child stays a zombie + /// until then, and asking the OS whether the PID exists would still say yes. + /// Polls briefly: the product signals the process and this only has to + /// outlast the moment between the signal landing and the kernel finishing + /// with it, not any grace period the product itself waits out. + pub async fn is_running(&mut self) -> bool { + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + loop { + match self.child.try_wait() { + Ok(Some(_)) => return false, + Ok(None) if std::time::Instant::now() < deadline => { + tokio::time::sleep(std::time::Duration::from_millis(100)).await; + } + Ok(None) => return true, + // Not treated as "stopped". The only caller asserts that the + // product stopped this process, so answering `false` here is + // answering "the product did its job" — which would turn a wait + // this harness could not perform into a pass for the very defect + // the scenario exists to catch. `Child` caches the status once + // it has reaped, so a repeated call returns `Ok(Some(_))` rather + // than an error and there is no benign case left to absorb. + // + // Panicking is safe here specifically: nothing calls this from + // `Drop` (the only caller is the step behind + // `@id:serve-uninstall-stops-what-it-manages`), so there is no + // unwind-during-drop abort to worry about, and cucumber catches + // a step panic per scenario — the failure is reported and this + // handle's `Drop` still runs its kill/wait cleanup. + Err(error) => panic!( + "could not determine whether pid {} is still running: {error}", + self.pid() + ), + } + } + } +} + +impl Drop for OwnedProcess { + fn drop(&mut self) { + let _ = self.child.kill(); + let _ = self.child.wait(); + } +} + /// One scenario's resolved expectation plus the identity needed to report it. /// /// Recorded by `filter_run`, which sees every scenario — including the ones it @@ -327,6 +423,7 @@ impl Default for E2eWorld { cli_output: None, cli_outputs: None, cli_stderr: None, + cli_other_output: None, cli_rc: None, cli_elapsed: None, remote_env: Vec::new(), @@ -349,6 +446,9 @@ impl Default for E2eWorld { helper_identity: None, recorded_process: None, dash_report_refused: None, + occupied_address: None, + path_override: None, + owned_processes: Vec::new(), automation_daemon: None, } } @@ -640,6 +740,53 @@ impl E2eWorld { self.register_mock_service_with(ServiceRecordOptions::default()); } + /// Spawn a plain, long-lived child process and register it with the CLI as a + /// ready managed service, exactly as a real `rocm serve --managed` would + /// record its own processes. Returns the child's PID. + /// + /// The child is a sleeper rather than a server: every scenario that uses + /// this asks whether the CLI *stopped the process it recorded*, and a + /// listening socket would add a second thing to get wrong without making + /// that question any easier to answer. Both PID fields point at the child so + /// the record looks like one the CLI itself wrote. + /// + /// Unix only — Windows has no equivalent one-liner sleeper on PATH, and the + /// scenarios that call this are `@requires-os:linux`. + pub fn start_managed_service_process(&mut self, service_id: &'static str, model: &str) -> u32 { + use std::process::Stdio; + + let root = self.isolated_root.as_ref().expect("no isolated root"); + let services = root.path().join("data").join("services"); + // Long enough to outlive any scenario, short enough that a leaked child + // on a persistent runner reaps itself rather than lingering forever. + let child = std::process::Command::new("sleep") + .arg("600") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .expect("failed to spawn the scenario's managed-service process"); + let pid = child.id(); + self.owned_processes.push(OwnedProcess { child }); + + // The port is never connected to; it only has to be a plausible value in + // a well-formed record. + write_service_record_with( + &services, + model, + 18_765, + ServiceRecordOptions { + service_id, + status: "ready", + startup_phase: None, + supervisor_pid: pid, + engine_pid: Some(pid), + ..ServiceRecordOptions::default() + }, + ); + pid + } + pub fn register_mock_service_with(&self, options: ServiceRecordOptions) { let root = self.isolated_root.as_ref().expect("no isolated root"); let mock = self.mock.as_ref().expect("no mock server running"); @@ -721,7 +868,8 @@ impl Drop for E2eWorld { // hence the returned status is discarded here. The serve retry, where a // failed stop changes the next attempt's meaning, does read it. if let Some(root) = &self.isolated_root { - stop_managed_services(root.path()); + let env = self.isolate_env(); + stop_managed_services(root.path(), &env); } // A scenario that made the audit directory read-only must give it back // before the TempDir below tries to remove it, even if a step panicked. @@ -744,7 +892,10 @@ impl Drop for E2eWorld { /// record existed yet, or the stop failed, the next attempt runs against a device /// the previous one still owns, and the run must say so rather than let it look /// like a genuinely broken serve. -fn stop_managed_services(root: &std::path::Path) -> String { +fn stop_managed_services( + root: &std::path::Path, + env: &[(&'static str, std::ffi::OsString)], +) -> String { let services_dir = root.join("data").join("services"); let entries = match std::fs::read_dir(&services_dir) { Ok(entries) => entries, @@ -776,9 +927,14 @@ fn stop_managed_services(root: &std::path::Path) -> String { } let mut cmd = std::process::Command::new(rocm_binary()); cmd.args(["services", "stop", service_id, "--yes"]); - cmd.env("ROCM_CLI_CONFIG_DIR", root.join("config")); - cmd.env("ROCM_CLI_DATA_DIR", root.join("data")); - cmd.env("ROCM_CLI_CACHE_DIR", root.join("cache")); + // The scenario's own environment, not a hand-rolled subset of it. The + // three directory variables were previously rebuilt here, which quietly + // dropped everything else `isolate_env` sets — including the shared + // cache redirection a CI lane exports — so this teardown was talking to + // a slightly different machine than the scenario it is cleaning up after. + for (key, value) in env { + cmd.env(key, value); + } outcomes.push(match cmd.output() { Ok(out) if out.status.success() => format!("{service_id}: stopped"), Ok(out) => format!( @@ -999,6 +1155,134 @@ pub fn run_rocm_with_env_and_unset( ) } +/// Write `script` as the executable `name` in `bin`, and return the `PATH` that +/// puts `bin` first. +#[cfg(unix)] +fn install_shim(bin: &std::path::Path, name: &str, script: &str) -> String { + use std::os::unix::fs::PermissionsExt as _; + + let shim = bin.join(name); + std::fs::write(&shim, script) + .unwrap_or_else(|err| panic!("failed to write the {name} shim: {err}")); + std::fs::set_permissions(&shim, std::fs::Permissions::from_mode(0o755)) + .unwrap_or_else(|err| panic!("failed to mark the {name} shim executable: {err}")); + + let existing = std::env::var("PATH").unwrap_or_default(); + format!("{}:{existing}", bin.display()) +} + +/// Install a `stat` shim on the front of `PATH` that answers for ONE device path +/// and delegates every other invocation to the real binary, and return the `PATH` +/// value to hand the child. +/// +/// The CLI learns a device's mode and owning group by shelling out to `stat`, and +/// several contracts are about what it then does with unusual answers — an owning +/// group the machine cannot name, or one that is not the conventional default. +/// Those states are properties of a machine's device nodes, not something a test +/// can arrange on a shared runner, so the answer is substituted instead. +/// +/// What this proves and what it does not: it exercises how the CLI HANDLES such +/// an answer, not that any particular host produces one. `stat` really does print +/// `UNKNOWN` for an id with no entry in the group database, which is where the +/// substituted answer comes from. +/// +/// Scoped as tightly as possible: only the named path is answered for, so every +/// other `stat` the CLI makes still reads the real filesystem. +/// +/// Unix only — panics elsewhere. Every scenario that reaches it is +/// `@requires-os:linux`, so the panic is unreachable rather than a limitation. +#[cfg(unix)] +pub fn stat_shim_path( + world: &E2eWorld, + device: &str, + mode: &str, + owner: &str, + group: &str, +) -> String { + let root = world.isolated_root.as_ref().expect("no isolated root"); + let bin = root.path().join("stat-shim"); + std::fs::create_dir_all(&bin).expect("failed to create the stat shim directory"); + + let real = ["/usr/bin/stat", "/bin/stat", "/usr/local/bin/stat"] + .into_iter() + .find(|candidate| std::path::Path::new(candidate).is_file()) + .expect("no real `stat` binary to delegate to"); + // Answers only the one call this scenario is substituting: the CLI's + // `stat -c %A|%U|%G ` in `rocm_core`'s `stat_device`. + // Both the format AND the path must match — keying on the path alone would + // hand this `mode|owner|group` line to any other `stat` of the same device + // added later, in whatever format it asked for, and the caller would parse + // the wrong shape without anything saying so. Anything else is the real + // binary's business. + let script = format!( + "#!/bin/sh\nfmt=\nwant=\nfor arg in \"$@\"; do\n \ + if [ \"$arg\" = '%A|%U|%G' ]; then fmt=1; fi\n \ + if [ \"$arg\" = \"{device}\" ]; then want=1; fi\ndone\n\ + if [ -n \"$fmt\" ] && [ -n \"$want\" ]; then\n \ + printf '%s|%s|%s\\n' '{mode}' '{owner}' '{group}'\n exit 0\nfi\nexec {real} \"$@\"\n" + ); + install_shim(&bin, "stat", &script) +} + +#[cfg(not(unix))] +pub fn stat_shim_path( + _world: &E2eWorld, + _device: &str, + _mode: &str, + _owner: &str, + _group: &str, +) -> String { + unreachable!("the stat shim is only reached by @requires-os:linux scenarios") +} + +/// Add an `id` shim to the same directory [`stat_shim_path`] uses, reporting the +/// current user as belonging to `groups` instead of their real ones, and return +/// the `PATH` carrying both shims. +/// +/// The CLI reads group membership by shelling out to `id -Gn`. Whether the user +/// running the suite belongs to a device group is a property of the runner, and +/// on the GPU runner they already do — so a contract about what the CLI concludes +/// for a user who does NOT can only be measured by substituting the answer. Call +/// after [`stat_shim_path`], whose directory this reuses. +#[cfg(unix)] +pub fn id_shim_path(world: &E2eWorld, groups: &[&str]) -> String { + let root = world.isolated_root.as_ref().expect("no isolated root"); + let bin = root.path().join("stat-shim"); + std::fs::create_dir_all(&bin).expect("failed to create the shim directory"); + + let real = ["/usr/bin/id", "/bin/id"] + .into_iter() + .find(|candidate| std::path::Path::new(candidate).is_file()) + .expect("no real `id` binary to delegate to"); + // `-Gn` is the only form the CLI asks for; every other invocation is the real + // binary's business. + let script = format!( + "#!/bin/sh\nif [ \"$1\" = \"-Gn\" ]; then\n echo '{}'\n exit 0\nfi\nexec {real} \"$@\"\n", + groups.join(" ") + ); + install_shim(&bin, "id", &script) +} + +#[cfg(not(unix))] +pub fn id_shim_path(_world: &E2eWorld, _groups: &[&str]) -> String { + unreachable!("the id shim is only reached by @requires-os:linux scenarios") +} + +/// Every group name the machine's own group database knows. +/// +/// Read from `/etc/group` rather than asked of a tool, so the check does not +/// depend on `getent` being installed. Unix only, for the same reason as +/// [`stat_shim_path`]. +pub fn machine_group_names() -> Vec { + std::fs::read_to_string("/etc/group") + .unwrap_or_default() + .lines() + .filter_map(|line| line.split(':').next()) + .filter(|name| !name.trim().is_empty()) + .map(str::to_owned) + .collect() +} + /// Run `rocm` with the behavioral fixture established by a Given step, then /// consume it so it cannot leak into a later action in the same scenario. pub fn run_rocm_with_scenario_env(world: &mut E2eWorld, args: &[&str]) -> (String, String, i32) { diff --git a/tests/e2e-cucumber/tests/e2e/chat_steps.rs b/tests/e2e-cucumber/tests/e2e/chat_steps.rs index ed401a4306..5fe4b43836 100644 --- a/tests/e2e-cucumber/tests/e2e/chat_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/chat_steps.rs @@ -91,6 +91,17 @@ async fn user_sends_oneshot_chat(world: &mut E2eWorld) { world.cli_output = Some(stdout); } +#[when("the user sends a one-shot chat prompt without naming a model")] +async fn user_sends_oneshot_chat_without_model(world: &mut E2eWorld) { + // The same command as the scenario above, minus `--model`. Leaving the model + // out is the ordinary way to say "use whatever is running here", and the + // services list on this machine shows exactly one ready local server. + let (stdout, stderr, rc) = + crate::run_rocm(world, &["chat", "--provider", "local", "--prompt", "Hello"]); + world.cli_output = Some(format!("{stdout}{stderr}")); + world.cli_rc = Some(rc); +} + #[when("the user pipes a one-shot chat prompt through the CLI")] async fn user_pipes_oneshot_chat(world: &mut E2eWorld) { // Drive `rocm chat` with the prompt on stdin and no `--prompt`, matching the diff --git a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs index a12eae28a4..0c31d95f79 100644 --- a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs @@ -8,14 +8,15 @@ use crate::E2eWorld; use crate::e2e::lifecycle_steps; use crate::e2e::tui_driver::{TuiSession, default_timeout}; -/// A symptom string that scores a catalog match on both Linux and Windows. It -/// keys off `check_1_arch_not_in_wheel` (a `LINUX_AND_WINDOWS` checker), which -/// scores 50 on the `HSA_STATUS_ERROR_INVALID_ISA` keyword regardless of host -/// state — the covered-arch penalty only applies when a framework arch list is -/// present, so with none installed the match always renders. (The earlier -/// "/dev/kfd" symptom keyed only off the Linux-only render-group checker and so -/// produced no match on Windows.) The specific fix-id is environment-dependent, -/// so scenarios assert the shape of a match, not the id. +/// A symptom string that scores a catalog match on every platform these +/// scenarios run on. It keys off `check_1_arch_not_in_wheel`, registered for +/// `["linux", "windows", "wsl"]` in `CHECKERS` (`crates/rocm-core/src/diagnose.rs`), +/// which scores 50 on the `HSA_STATUS_ERROR_INVALID_ISA` keyword regardless of +/// host state — the covered-arch penalty only applies when a framework arch +/// list is present, so with none installed the match always renders. (The +/// earlier "/dev/kfd" symptom keyed only off the Linux-only render-group +/// checker and so produced no match on Windows.) The specific fix-id is +/// environment-dependent, so scenarios assert the shape of a match, not the id. const KNOWN_SYMPTOM: &str = "HSA_STATUS_ERROR_INVALID_ISA"; /// The error text a vLLM engine-startup import failure leaves behind, as a user @@ -35,12 +36,16 @@ const ENGINE_IMPORT_FIX_ID: &str = "fix-17-torch-dlpack"; const PREVIEW_FIX_ID: &str = "fix-1-arch"; /// A recipe that would really change the machine, used to prove the CLI asks -/// first. Of the entries the CLI carries out, this is the only one that -/// reaches the confirmation gate on a host with nothing installed: -/// `fix-2-unset-override` -/// never calls it on Linux, `fix-4-render-group` exits early once the user is -/// already in the groups, and `fix-6-path` exits early with "no ROCm install -/// found". This one needs only `--device-index`, which the scenario supplies. +/// first. Of the four AUTO recipes this is the only one that reaches the +/// confirmation gate on a host with nothing installed: `fix-2-unset-override` +/// never calls it on Linux (`run_unset_override_linux` only prints), `fix-6-path` +/// exits 3 with "No ROCm install found; nothing to add to PATH", and +/// `fix-4-render-group` exits 3 unless `usermod` — and `sudo`, when not root — +/// are both on `$PATH`, which is why the command-failure scenario below has to +/// shim them in. (`run_render_group` has no group-membership check at all; an +/// earlier version of this comment said it exited early "once the user is +/// already in the groups", which was never true.) This one needs only +/// `--device-index`, which the scenario supplies. const MUTATING_FIX_ID: &str = "fix-9-igpu-dgpu"; /// The recipe used to prove a failed helper command is explained on stderr with @@ -1642,6 +1647,458 @@ async fn assert_interactive_decline_reported(world: &mut E2eWorld) { ); } +// ── Diagnosis and fix as two views of one remedy ─────────────────── + +/// The remedy both commands describe for a device-permission problem: the CLI +/// can apply it, and both `diagnose` and `fix` have their own copy of what it +/// does — which is exactly why the two can drift apart. +const DEVICE_PERMISSION_FIX_ID: &str = "fix-4-render-group"; + +/// A symptom that makes the device-permission cause score on any bare-metal +/// Linux host (the check is not registered on WSL or Windows), so the +/// comparison never depends on what happens to be wrong with the runner. +const DEVICE_PERMISSION_SYMPTOM: &str = "unable to open /dev/kfd"; + +/// The device whose owning group the remedy is about. +const GPU_DEVICE: &str = "/dev/kfd"; + +/// The field the machine-readable inspection describes [`GPU_DEVICE`] under. +const GPU_DEVICE_FIELD: &str = "kfd"; + +/// The group the fix recipe is hard-coded to add the user to. +const DEFAULT_DEVICE_GROUP: &str = "render"; + +/// The line a diagnosis prints for how to check the remedy worked. +const DIAGNOSE_VERIFY_PREFIX: &str = "verify after fix:"; + +/// The line a fix preview prints for the same thing. +const FIX_VERIFY_PREFIX: &str = "Verify:"; + +/// The value after `label` on the first line that carries it. +/// +/// A bare label with nothing after it is `None`, not `Some("")`. The callers +/// compare two of these for equality, and two empty strings compare equal — so +/// returning `Some("")` would let a diagnosis and a fix preview that BOTH said +/// nothing pass as agreeing. `None` sends the caller to its own "gave no way to +/// verify this" panic, which is what an empty label actually means. +fn labelled(text: &str, label: &str) -> Option { + text.lines() + .filter_map(|line| line.trim().strip_prefix(label)) + .map(|value| value.trim().to_owned()) + .find(|value| !value.is_empty()) +} + +/// The part of a diagnosis that belongs to one cause. +/// +/// A diagnosis reports several causes, each with its own plan and its own way to +/// verify it, and which one comes first depends on the machine. Narrowing to the +/// named cause first is what keeps the comparison about the remedy under test +/// rather than about whichever cause happened to rank highest here. +fn section_for(report: &str, fix_id: &str) -> Option { + let start = report.lines().position(|line| line.contains(fix_id))?; + // Causes are separated by a blank line, so the next one ends this section. + let body: Vec<&str> = report + .lines() + .skip(start) + .take_while(|line| !line.trim().is_empty()) + .collect(); + Some(body.join("\n")) +} + +/// The `usermod` invocation a report proposes, wherever it prints it. Both +/// commands render it differently — one as a shell line under the plan, the +/// other as the command it is about to run — so match on the invocation itself. +fn proposed_group_command(text: &str) -> Option { + text.lines() + .map(str::trim) + .find(|line| line.contains("usermod") && line.contains("-G")) + .map(|line| { + // Strip whichever prefix the renderer used ("$ ", "Will run: ") so the + // two are compared as the same kind of thing. + line.trim_start_matches("$ ") + .trim_start_matches("Will run: ") + .to_owned() + }) +} + +/// The group names a proposed command would add the user to. +/// +/// Reads both spellings of the flag, and both of its forms: `-G a,b` / +/// `--groups a,b` as separate tokens, and `-aG a,b` where the flag is bundled +/// with another. [`proposed_group_command`] matches any line containing `-G`, +/// which includes all of them — so recognising only the separate `-G` token +/// would return NOTHING for a reworded command, and an empty list compares +/// equal to another empty list. Both callers then pass while measuring nothing. +/// The callers additionally reject an empty result, so a form not handled here +/// fails loudly instead. +fn groups_in_command(command: &str) -> Vec { + let mut tokens = command.split_whitespace(); + let names = loop { + let Some(token) = tokens.next() else { + return Vec::new(); + }; + // `-G`/`--groups` take the list as the NEXT token; a bundled `-aG` may + // carry it directly (`-aGrender,video`) or in the next token. + if token == "-G" || token == "--groups" { + break tokens.next().unwrap_or_default(); + } + if let Some(rest) = token.strip_prefix("--groups=") { + break rest; + } + if token.starts_with('-') && !token.starts_with("--") && token.contains('G') { + let (_, rest) = token.split_at(token.find('G').expect("checked above") + 1); + break if rest.is_empty() { + tokens.next().unwrap_or_default() + } else { + rest + }; + } + }; + names + .trim_matches('"') + .split(',') + .filter(|name| !name.trim().is_empty()) + .map(|name| name.trim().to_owned()) + .collect() +} + +/// Run the diagnosis and the matching fix preview, and stash both. +fn compare_diagnosis_with_fix(world: &mut E2eWorld, path_override: Option<&str>) { + let args = ["diagnose", "--symptom", DEVICE_PERMISSION_SYMPTOM]; + let (diagnosis, _, _) = match path_override { + Some(path) => crate::run_rocm_with_env(world, &args, &[("PATH", path)]), + None => crate::run_rocm(world, &args), + }; + // No `--yes`, so nothing is applied; the preview is the user-facing account + // of what the fix would do. + // + // The shim goes to BOTH sides. Today the fix recipe never consults the + // device, so passing it only to `diagnose` would happen to work — but the + // whole point of this pair is that the two disagree, and the natural fix is + // to make `fix` read what `diagnose` read. On the day that lands, a shim on + // one side only would silently turn this into a shimmed-against-real + // comparison and the scenario would report a divergence that is the + // fixture's, not the product's. + let fix_args = ["fix", DEVICE_PERMISSION_FIX_ID, "--dry-run"]; + let (preview, preview_err, _) = match path_override { + Some(path) => crate::run_rocm_with_env(world, &fix_args, &[("PATH", path)]), + None => crate::run_rocm(world, &fix_args), + }; + world.cli_output = Some(diagnosis); + world.cli_other_output = Some(format!("{preview}{preview_err}")); +} + +#[given("a user who hit a device-permission failure")] +async fn user_hit_device_permission_failure(_world: &mut E2eWorld) { + // States the premise; carries no fixture. The symptom is + // `DEVICE_PERMISSION_SYMPTOM`, which the `When` this precedes passes + // directly. That constant is also used by a second `When` further down, + // which runs its own `diagnose` invocation under a shimmed PATH — so it + // belongs to the pair of them, not to this step. + // + // This used to write `world.model_name`, which nothing read: the step LOOKED + // like it parameterised the comparison, so changing the constant here would + // have had no effect and said nothing about it. Left empty rather than + // wired up, because a Given that fed only one of the two would make the + // constant look local to this scenario when it is not. +} + +#[when("the user compares the diagnosis with the matching fix preview")] +async fn user_compares_diagnosis_with_fix(world: &mut E2eWorld) { + let path_override = world.path_override.clone(); + compare_diagnosis_with_fix(world, path_override.as_deref()); +} + +#[then("both give the same way to verify that the fix worked")] +async fn assert_verification_agrees(world: &mut E2eWorld) { + let diagnosis = world.cli_output.as_ref().expect("no diagnosis output"); + let preview = world + .cli_other_output + .as_ref() + .expect("no fix preview output"); + let cause = section_for(diagnosis, DEVICE_PERMISSION_FIX_ID).unwrap_or_else(|| { + panic!( + "the symptom did not produce the device-permission cause, so there is nothing to \ + compare:\n{diagnosis}" + ) + }); + let from_diagnosis = labelled(&cause, DIAGNOSE_VERIFY_PREFIX) + .unwrap_or_else(|| panic!("the diagnosis gave no way to verify the fix:\n{cause}")); + let from_preview = labelled(preview, FIX_VERIFY_PREFIX) + .unwrap_or_else(|| panic!("the fix preview gave no way to verify itself:\n{preview}")); + // A user told two different things to run cannot know which one settles it. + assert_eq!( + from_diagnosis, from_preview, + "the diagnosis and the fix preview give different ways to verify the same remedy" + ); +} + +#[then("both give the same command for applying the remedy")] +async fn assert_remedy_command_agrees(world: &mut E2eWorld) { + let diagnosis = world.cli_output.as_ref().expect("no diagnosis output"); + let preview = world + .cli_other_output + .as_ref() + .expect("no fix preview output"); + let cause = section_for(diagnosis, DEVICE_PERMISSION_FIX_ID).unwrap_or_else(|| { + panic!( + "the symptom did not produce the device-permission cause, so there is nothing to \ + compare:\n{diagnosis}" + ) + }); + let from_diagnosis = proposed_group_command(&cause) + .unwrap_or_else(|| panic!("the diagnosis proposed no group command:\n{cause}")); + let from_preview = proposed_group_command(preview) + .unwrap_or_else(|| panic!("the fix preview proposed no group command:\n{preview}")); + // Compare the groups rather than the whole line: the two renderings name the + // user differently (one leaves `$USER` for the shell, the other resolves it), + // and that difference is presentation, not disagreement about the remedy. + let diagnosis_groups = groups_in_command(&from_diagnosis); + let preview_groups = groups_in_command(&from_preview); + // Two empty lists compare equal. Without this the scenario would pass — and + // its expectation row would go stale — the moment a reworded command stopped + // being parsed, while the divergence it exists to measure went unread. + for (side, groups, line) in [ + ("diagnosis", &diagnosis_groups, &from_diagnosis), + ("fix preview", &preview_groups, &from_preview), + ] { + assert!( + !groups.is_empty(), + "no group names could be read out of the {side}'s command, so the two sides cannot \ + be compared. This is a harness read failure, not a verdict:\n {line}" + ); + } + assert_eq!( + diagnosis_groups, preview_groups, + "the diagnosis and the fix preview would add the user to different groups:\n \ + diagnosis: {from_diagnosis}\n preview: {from_preview}" + ); +} + +// ── Device group states induced for the machine under test ───────── + +#[given("the GPU device belongs to a group the machine cannot name")] +async fn given_unnameable_device_group(world: &mut E2eWorld) { + // What `stat` itself prints for an owning id with no entry in the group + // database — the state a container inherits when the device's group exists + // on the host but not inside. + let path = crate::stat_shim_path(world, GPU_DEVICE, "crw-rw----", "root", "UNKNOWN"); + assert_cli_sees_device_group(world, &path, "UNKNOWN"); + world.path_override = Some(path); +} + +/// Prove the CLI's own inspection reads `group` as the owner of the GPU device +/// through `path`, so a shim that stops answering fails the premise instead of +/// letting the scenario pass on the host's real device state. +fn assert_cli_sees_device_group(world: &E2eWorld, path: &str, group: &str) { + let (report, _, _) = crate::run_rocm_with_env(world, &["examine", "--json"], &[("PATH", path)]); + let json: serde_json::Value = serde_json::from_str(&report) + .unwrap_or_else(|e| panic!("`examine --json` did not emit valid JSON ({e}):\n{report}")); + let stated = json + .get(GPU_DEVICE_FIELD) + .and_then(|device| device.get("owner_group")) + .and_then(serde_json::Value::as_str); + assert_eq!( + stated, + Some(group), + "the inspection does not report {GPU_DEVICE} as owned by group {group:?}, so the \ + substituted premise did not take:\n{report}" + ); +} + +#[given("the GPU device belongs to a recognised non-default group")] +async fn given_non_default_device_group(world: &mut E2eWorld) { + // A real group on this machine other than the one the fix recipe assumes. + // `video` is excluded too: the remedy adds it alongside, so using it would + // make the two sides agree by coincidence. + // + // Prefer a group a device node plausibly belongs to, so the situation reads + // as one a machine really presents rather than as an arbitrary substitution; + // `/etc/group`'s own order would otherwise hand this `root` or `daemon`. + // Any other group still satisfies the premise — what the scenario needs is a + // group the machine names and the recipe does not assume — so an unusual + // host falls back to the first such entry rather than failing. + let names = crate::machine_group_names(); + let usable = |name: &String| name != DEFAULT_DEVICE_GROUP && name != "video"; + let group = ["kvm", "input", "disk", "dialout", "plugdev", "users"] + .into_iter() + .find_map(|preferred| { + names + .iter() + .find(|name| name.as_str() == preferred) + .cloned() + }) + .or_else(|| names.iter().find(|name| usable(name)).cloned()) + .expect("this machine names no group other than the default"); + assert!( + usable(&group), + "picked {group:?} as the non-default device group, which the remedy adds anyway" + ); + let path = crate::stat_shim_path(world, GPU_DEVICE, "crw-rw----", "root", &group); + assert_cli_sees_device_group(world, &path, &group); + world.path_override = Some(path); +} + +#[given("the user can already read and write the GPU device")] +async fn given_usable_device(world: &mut E2eWorld) { + // Two halves of one situation: the device is open to everyone, AND the user + // is in none of the groups conventionally used to reach it. Both are + // substituted rather than hoped for — the GPU runner's user IS in those + // groups, which makes the finding score zero there for a reason that has + // nothing to do with the access the scenario is about (measured: it passed + // on MI300X until the group answer was substituted too). + crate::stat_shim_path(world, GPU_DEVICE, "crw-rw-rw-", "root", "root"); + let path = crate::id_shim_path(world, &["users"]); + // Prove the premise through the CLI's own eyes before relying on it: if the + // inspection does not agree the device is usable, the scenario would be + // measuring something else. + let (report, _, _) = + crate::run_rocm_with_env(world, &["examine", "--json"], &[("PATH", path.as_str())]); + let json: serde_json::Value = serde_json::from_str(&report) + .unwrap_or_else(|e| panic!("`examine --json` did not emit valid JSON ({e}):\n{report}")); + // Read the GPU device the scenario shimmed, by name. The report describes + // several devices — the single `kfd` and each entry of `render_devices` — + // and they all carry these same field names, so a search for the first one + // anywhere in the document would be answered by whichever device happened to + // come first rather than by the one under test. + let device = json.get(GPU_DEVICE_FIELD).unwrap_or_else(|| { + panic!("`examine --json` describes no `{GPU_DEVICE_FIELD}` device:\n{report}") + }); + for field in ["user_can_read", "user_can_write"] { + let stated = device.get(field).and_then(serde_json::Value::as_bool); + assert_eq!( + stated, + Some(true), + "the inspection does not report {GPU_DEVICE} as {field}, so the premise this scenario \ + needs is absent:\n{report}" + ); + } + world.path_override = Some(path); +} + +#[when("the user asks the CLI to diagnose a device-permission failure")] +async fn user_diagnoses_device_permission(world: &mut E2eWorld) { + let path = world + .path_override + .clone() + .expect("no shimmed PATH prepared"); + let (stdout, _, rc) = crate::run_rocm_with_env( + world, + &["diagnose", "--symptom", DEVICE_PERMISSION_SYMPTOM], + &[("PATH", path.as_str())], + ); + world.cli_output = Some(stdout); + world.cli_rc = Some(rc); +} + +#[when("the user asks the CLI to diagnose the machine")] +async fn user_diagnoses_machine(world: &mut E2eWorld) { + let path = world + .path_override + .clone() + .expect("no shimmed PATH prepared"); + // No symptom: the question is what this machine looks like, not what a + // particular error message suggests. + let (stdout, _, rc) = + crate::run_rocm_with_env(world, &["diagnose", "--json"], &[("PATH", path.as_str())]); + world.cli_output = Some(stdout); + world.cli_rc = Some(rc); +} + +#[then("every group named in the remedy exists on the machine")] +async fn assert_named_groups_exist(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no diagnose output"); + let cause = section_for(output, DEVICE_PERMISSION_FIX_ID).unwrap_or_else(|| { + panic!("the symptom did not produce the device-permission cause:\n{output}") + }); + let command = proposed_group_command(&cause) + .unwrap_or_else(|| panic!("the diagnosis proposed no group command:\n{cause}")); + let proposed = groups_in_command(&command); + // Both sides are guarded because an empty either side makes the check below + // vacuous: no proposed groups means nothing is tested, and no known groups + // means everything is reported unknown. `machine_group_names` swallows a + // read error into an empty list, so the second is a real possibility on a + // host where `/etc/group` cannot be read — a harness failure that must not + // be reported as the product naming groups that do not exist. + assert!( + !proposed.is_empty(), + "no group names could be read out of the proposed command, so nothing was checked. \ + This is a harness read failure, not a verdict:\n {command}" + ); + let known = crate::machine_group_names(); + assert!( + !known.is_empty(), + "this machine's group database could not be read, so no proposed group can be judged \ + against it. This is a harness read failure, not a verdict:\n {command}" + ); + let unknown: Vec = proposed + .into_iter() + .filter(|group| !known.iter().any(|name| name == group)) + .collect(); + // Handing the user a command naming a group that does not exist gives them + // something that cannot work and no way to see why. + assert!( + unknown.is_empty(), + "the remedy tells the user to join {unknown:?}, which this machine does not \ + recognise:\n {command}" + ); +} + +#[then("adding the user to a device group is not offered as a cause")] +async fn assert_no_device_group_remedy(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no diagnose output"); + let report: serde_json::Value = + serde_json::from_str(output).expect("diagnose --json did not emit valid JSON"); + // An absent entry is the outcome asked for, so it must be absent because the + // checks ran and cleared it — not because the run failed or routed out + // before evaluating anything. Not `!matched.is_empty()`: with no symptom + // given, a healthy machine correctly matches nothing at all. + assert_eq!( + world.cli_rc, + Some(0), + "diagnose should exit 0 (it is a query):\n{output}" + ); + assert!( + report + .get("out_of_scope") + .is_none_or(serde_json::Value::is_null), + "this bare-metal Linux host was routed out of scope, so the device checks never \ + ran and an absent cause proves nothing:\n{output}" + ); + let matched = report + .get("matched") + .and_then(|m| m.as_array()) + .expect("diagnose JSON has no 'matched' array"); + let offered = matched.iter().find(|entry| { + entry.get("id").and_then(serde_json::Value::as_str) == Some(DEVICE_PERMISSION_FIX_ID) + }); + // Absent means not offered; present without a readable score is a report + // this step cannot judge, and is not allowed to read as a zero. + let score = offered.map_or(0, |entry| { + entry + .get("score") + .and_then(serde_json::Value::as_i64) + .unwrap_or_else(|| { + panic!("the '{DEVICE_PERMISSION_FIX_ID}' entry has no score:\n{output}") + }) + }); + // Belonging to a conventional group is one way to reach the device, not the + // point of doing so. With access already demonstrated, repairing permissions + // is not a cause of anything, and offering it sends the user to change their + // group membership for no reason. + // + // Deliberately "not offered" rather than "not ranked first": the check adds + // score for missing membership and never subtracts for demonstrated access, + // so a fix that only demoted this cause below the others would leave the + // user still told to change their groups. Scoring it zero is the outcome + // this asks for, and the step says so in those words. + assert_eq!( + score, 0, + "the device is readable and writable, yet the diagnosis still offers \ + '{DEVICE_PERMISSION_FIX_ID}' as a cause (score {score})" + ); +} + #[then("the file the fix would have changed is untouched")] async fn assert_rc_file_untouched(world: &mut E2eWorld) { let rc_file = fix_rc_file(world); diff --git a/tests/e2e-cucumber/tests/e2e/examine_steps.rs b/tests/e2e-cucumber/tests/e2e/examine_steps.rs index f3a948d5e3..7be0d79750 100644 --- a/tests/e2e-cucumber/tests/e2e/examine_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/examine_steps.rs @@ -122,6 +122,128 @@ async fn user_asks_help(world: &mut E2eWorld) { world.cli_output = Some(stdout); } +// ── What the help itself promises ────────────────────────────────── + +/// Model references named after `rocm serve ` in an examples block, from both +/// the top-level help and `serve`'s own. A worked example is copied verbatim, so +/// the name in it is the one that has to work. +fn serve_example_models(help: &str) -> Vec { + help.lines() + .filter_map(|line| line.trim().strip_prefix("rocm serve ")) + .filter_map(|rest| rest.split_whitespace().next()) + // The generic placeholders the help uses to describe the FORM of a model + // reference are not names anyone is expected to run. + .filter(|model| !model.starts_with('<')) + .map(str::to_owned) + .collect() +} + +/// Every name the CLI's own model listing answers to: each recipe's canonical id +/// plus its aliases, as `rocm model --verbose` prints them. +fn known_model_names(listing: &str) -> Vec { + let mut names = Vec::new(); + for line in listing.lines() { + let Some((head, rest)) = line.trim().split_once(" aliases=[") else { + continue; + }; + names.push(head.trim().to_owned()); + if let Some((aliases, _)) = rest.split_once(']') { + names.extend(aliases.split(',').map(|alias| alias.trim().to_owned())); + } + } + names +} + +#[when("the user reads the serve examples the help offers")] +async fn user_reads_serve_examples(world: &mut E2eWorld) { + // Both example blocks that name a model: the top-level one a new user meets + // first, and `serve`'s own. + let (general, _, _) = crate::run_rocm(world, &["help"]); + let (serve, _, _) = crate::run_rocm(world, &["serve", "--help"]); + let (listing, _, _) = crate::run_rocm(world, &["model", "--verbose"]); + world.cli_output = Some(format!("{general}\n{serve}")); + world.cli_other_output = Some(listing); +} + +#[then("every model named there is one the CLI can resolve")] +async fn assert_example_models_resolve(world: &mut E2eWorld) { + let help = world.cli_output.as_ref().expect("no help output"); + let listing = world.cli_other_output.as_ref().expect("no model listing"); + let examples = serve_example_models(help); + assert!( + !examples.is_empty(), + "no `rocm serve ` example found in the help:\n{help}" + ); + let known = known_model_names(listing); + assert!( + !known.is_empty(), + "could not read any model name out of the listing:\n{listing}" + ); + // Two forms are legitimate, and the README documents both: a name this CLI's + // own catalog answers to, or an explicit `owner/repo` reference to fetch. The + // check accepts either, so it says the example must WORK without dictating + // which model it should be. + let unresolvable: Vec<&String> = examples + .iter() + .filter(|model| { + !model.contains('/') && !known.iter().any(|name| name.eq_ignore_ascii_case(model)) + }) + .collect(); + assert!( + unresolvable.is_empty(), + "the help offers {unresolvable:?} as models to serve, but they are neither a name the \ + model listing knows nor an `owner/repo` reference — copying the example verbatim \ + cannot work.\nmodel listing knows: {known:?}" + ); +} + +#[then("running the CLI with no subcommand is not described as the dashboard command")] +async fn assert_default_command_described_distinctly(world: &mut E2eWorld) { + let help = world.cli_output.as_ref().expect("no help output"); + // What the help says the dedicated dashboard command does. Taken from the + // help itself rather than hardcoded, so this stays true if the wording + // changes. + let dash_row = help + .lines() + .map(str::trim) + .find(|line| line.starts_with("dash ")) + .unwrap_or_default() + .to_ascii_lowercase(); + assert!( + dash_row.contains("dashboard"), + "expected the help to list a `dash` command that opens the dashboard:\n{help}" + ); + // The sentence describing what running the CLI with no subcommand does. + // + // Read from the whole PARAGRAPH the phrase sits in, not from its one line: + // clap wraps `long_about` at the terminal width, and this sentence sits near + // a wrap boundary. Matching a single line would let a future wording that + // does call the plain command the dashboard slip through whenever the wrap + // happened to fall between "subcommand" and "dashboard" — passing while the + // defect this exists to catch is present. + let paragraph: Vec<&str> = help + .split("\n\n") + .find(|block| block.contains("with no subcommand")) + .unwrap_or_default() + .lines() + .map(str::trim) + .collect(); + let default_sentence = paragraph.join(" ").to_ascii_lowercase(); + assert!( + !default_sentence.is_empty(), + "the help does not say what running the CLI with no subcommand does:\n{help}" + ); + // Two commands, two different screens. Describing the plain command as the + // dashboard leaves the reader unable to tell them apart, and unaware that + // the plain command opens something else entirely. + assert!( + !default_sentence.contains("dashboard"), + "the help describes running the CLI with no subcommand as opening the dashboard, which \ + is what it also says `dash` does — the two are different screens:\n \ + default: {default_sentence}\n dash: {dash_row}" + ); +} + #[when("the user previews the driver install plan")] async fn user_previews_driver_install_plan(world: &mut E2eWorld) { // `--dry-run` renders the plan and returns before touching the system, so @@ -469,9 +591,11 @@ async fn assert_install_method_is_current(world: &mut E2eWorld) { async fn user_inspects_both_ways(world: &mut E2eWorld) { let (human, _, _) = crate::run_rocm(world, &["examine"]); let (json, _, rc) = crate::run_rocm(world, &["examine", "--json"]); - // Both are needed by the comparison step; the human form goes in the stderr - // slot rather than adding a World field for one scenario. - world.cli_stderr = Some(human); + // Both are needed by the comparison step. The human form is a second + // command's STDOUT, so it goes in `cli_other_output` — the field that did + // not exist when this step was written, which is why it used to borrow the + // stderr slot. + world.cli_other_output = Some(human); world.cli_output = Some(json); world.cli_rc = Some(rc); } @@ -548,7 +672,7 @@ async fn user_inspects_for_scripting_first(world: &mut E2eWorld) { // second, so the comparison step can hold the two to each other. let (json, _, rc) = crate::run_rocm(world, &["examine", "--json"]); let (human, _, _) = crate::run_rocm(world, &["examine"]); - world.cli_stderr = Some(human); + world.cli_other_output = Some(human); world.cli_output = Some(json); world.cli_rc = Some(rc); } @@ -641,8 +765,11 @@ fn human_states(human: &str, label: &str) -> Option { #[then("the framework report names the runtime's interpreter")] async fn assert_framework_names_the_runtimes_interpreter(world: &mut E2eWorld) { + // `cli_other_output`, not `cli_stderr`: the human form is a second command's + // STDOUT, and the `When` above moved it out of the stderr slot it used to + // borrow. Reading the old slot found `None` on every GPU lane. let human = world - .cli_stderr + .cli_other_output .as_ref() .expect("the human report was not captured"); let value = parsed_json(world); @@ -721,7 +848,7 @@ async fn assert_framework_names_the_runtimes_interpreter(world: &mut E2eWorld) { #[then("the machine-readable form states everything the readable one does")] async fn assert_json_states_what_human_does(world: &mut E2eWorld) { let human = world - .cli_stderr + .cli_other_output .as_ref() .expect("the human report was not captured"); let value = parsed_json(world); @@ -748,7 +875,7 @@ async fn assert_json_states_what_human_does(world: &mut E2eWorld) { async fn assert_json_names_setup_runtime_folder(world: &mut E2eWorld) { let planted = planted_setup_runtime_root(world).display().to_string(); let human = world - .cli_stderr + .cli_other_output .as_ref() .expect("the human report was not captured"); let value = parsed_json(world); @@ -806,7 +933,7 @@ async fn assert_json_keeps_setup_and_active_apart(world: &mut E2eWorld) { #[then("both reports agree on whether this machine has an AMD GPU")] async fn assert_forms_agree_on_gpu(world: &mut E2eWorld) { let human = world - .cli_stderr + .cli_other_output .as_ref() .expect("the human report was not captured"); let json = parsed_json(world); @@ -829,7 +956,7 @@ async fn assert_forms_agree_on_gpu(world: &mut E2eWorld) { #[then("both reports agree on whether this platform is in scope")] async fn assert_forms_agree_on_platform(world: &mut E2eWorld) { let human = world - .cli_stderr + .cli_other_output .as_ref() .expect("the human report was not captured"); let json = parsed_json(world); @@ -2008,7 +2135,10 @@ fn age_expected_from_date(record: &serde_json::Value) -> &'static str { )] async fn assert_examine_reports_driver_version_and_date(world: &mut E2eWorld) { let json = parsed_json(world); - let text = world.cli_stderr.as_ref().expect("no readable inspection"); + let text = world + .cli_other_output + .as_ref() + .expect("no readable inspection"); let record = windows_driver_record(&json).unwrap_or_else(|| { panic!("the machine-readable form carries no Windows driver record:\n{json:#}") }); @@ -2036,7 +2166,10 @@ async fn assert_examine_reports_driver_version_and_date(world: &mut E2eWorld) { #[then("the inspection calls the driver old only when its date is before 27 July 2023")] async fn assert_examine_driver_age_follows_the_date(world: &mut E2eWorld) { let json = parsed_json(world); - let text = world.cli_stderr.as_ref().expect("no readable inspection"); + let text = world + .cli_other_output + .as_ref() + .expect("no readable inspection"); let record = windows_driver_record(&json) .unwrap_or_else(|| panic!("no Windows driver record in the machine-readable form")); let expected = age_expected_from_date(record); diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index 521629c8ed..adff9847ef 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -495,7 +495,7 @@ fn default_engine_serve_target() -> &'static str { fn stop_scenario_services(world: &E2eWorld) -> String { world.isolated_root.as_ref().map_or_else( || "not attempted: scenario has no isolated root".to_owned(), - |root| crate::stop_managed_services(root.path()), + |root| crate::stop_managed_services(root.path(), &world.isolate_env()), ) } @@ -1305,6 +1305,494 @@ async fn user_serves_absent_gpu_index(world: &mut E2eWorld) { world.cli_rc = Some(rc); } +// ── Device policies the command offers ───────────────────────────── + +/// A help line that starts a new option: `--long` or `-x, --long`. +fn is_option_line(line: &str) -> bool { + let line = line.trim_start(); + line.starts_with("--") + || (line.len() > 1 && line.starts_with('-') && line.as_bytes()[1].is_ascii_alphanumeric()) +} + +/// The device policies `serve --help` advertises, read out of the help rather +/// than listed here so the check follows whatever the command currently offers. +/// +/// Scoped to `--device`'s own block. `serve --help` advertises possible values +/// for more than one option — `--engine` renders its list first, from clap's +/// `value_parser` — so taking the first possible-values list in the help returns +/// the engines, and this scenario would then serve `--device lemonade`, meeting +/// an unrelated rejection that happens to read like the one under test. +/// +/// Both of clap's renderings are read, because the choice between them is a +/// property of how the option happens to be declared rather than a contract: +/// +/// * inline — `--device Device policy [possible values: a, b]` +/// * block — a `Possible values:` line followed by one `- name: description` +/// entry per value, which is what clap emits for a documented `ValueEnum` +/// +/// Reading only one of them makes this return nothing the moment the other is +/// used, and returning nothing here does not read as "the command advertises +/// nothing" — it silently removes the very list the scenario exists to check. +/// The caller's guard turns that into a loud failure, but recognising both +/// shapes is what keeps the scenario measuring the contract at all. +fn advertised_device_policies(help: &str) -> Vec { + let lines: Vec<&str> = help.lines().collect(); + let device = lines.iter().position(|line| { + line.trim_start() + .strip_prefix("--device") + .is_some_and(|rest| rest.starts_with([' ', '=', '<']) || rest.is_empty()) + }); + let Some(device) = device else { + return Vec::new(); + }; + // `--device`'s own block: everything up to the next option. A `--device` + // that advertises nothing must not borrow the following option's list. + let block: Vec<&str> = lines + .iter() + .skip(device) + .enumerate() + .take_while(|(offset, line)| *offset == 0 || !is_option_line(line)) + .map(|(_, line)| *line) + .collect(); + + if let Some(values) = block.iter().find_map(|line| { + let (_, rest) = line.split_once("[possible values:")?; + let (values, _) = rest.split_once(']')?; + Some( + values + .split(',') + .map(|value| value.trim().to_owned()) + .filter(|value| !value.is_empty()) + .collect::>(), + ) + }) && !values.is_empty() + { + return values; + } + + let Some(heading) = block + .iter() + .position(|line| line.trim().eq_ignore_ascii_case("possible values:")) + else { + return Vec::new(); + }; + block + .iter() + .skip(heading + 1) + // `filter_map` rather than `map_while`: a long description wraps onto a + // continuation line, and stopping at the first line that is not an entry + // would drop every policy after it. The block is already bounded to + // `--device`, so there is nothing after it to run into. + .filter_map(|line| { + let entry = line.trim_start().strip_prefix("- ")?; + // `- : `; the description may itself contain a + // colon, so only the first one separates the two. + let name = entry.split_once(':').map_or(entry, |(name, _)| name); + Some(name.trim().to_owned()) + }) + .filter(|value| !value.is_empty()) + .collect() +} + +#[when("the user serves a model under each device policy the command offers")] +async fn user_serves_under_each_device_policy(world: &mut E2eWorld) { + let (help, _, _) = crate::run_rocm(world, &["serve", "--help"]); + let policies = advertised_device_policies(&help); + // A harness guard, not the contract: `serve --help` has always advertised + // some device policy, so an empty list means this step could not read the + // one it renders today, not that the command stopped offering any. Reported + // as its own failure so a help-format change cannot be mistaken for the + // command refusing a policy it advertises. + assert!( + !policies.is_empty(), + "could not read any device policy out of `--device`'s block in `serve --help` — neither \ + an inline `[possible values: ...]` nor a `Possible values:` list of `- name:` entries. \ + This is a harness read failure, not a verdict on the command:\n{help}" + ); + let (model, engine, _) = host_serve_target(); + // One line per policy, so a failure names which one was refused. This runs + // only where there is no GPU, so no serve can actually start and each of + // these returns promptly. + let mut transcript = Vec::new(); + for policy in &policies { + let (stdout, stderr, rc) = crate::run_rocm( + world, + &["serve", model, "--engine", engine, "--device", policy], + ); + transcript.push(format!("{policy}\t{rc}\t{stdout}{stderr}")); + } + world.cli_outputs = Some(transcript); +} + +#[then("no policy is refused for being that policy")] +async fn assert_no_policy_refused(world: &mut E2eWorld) { + let transcript = world + .cli_outputs + .as_ref() + .expect("no serve attempts recorded"); + // On this lane every policy is refused for the same host-level reason — the + // machine has no GPU — and that refusal is not what this scenario is about. + // What it must not find is a refusal aimed at the POLICY: a command that + // offers a choice and then rejects the user for making it. + // + // Matched on the refusal wording rather than on "named the policy and + // failed", because the ordinary host-level refusal names the policy too + // (`device_policy: gpu_required; no active ROCm runtime is configured`), so + // a name-plus-nonzero-rc rule would call every run on this lane a rejection. + // Three wordings are recognised: the two the product uses to turn a policy + // down for being that policy, and clap's own, which is what a value that is + // advertised but not accepted as a value looks like. + // + // The two product wordings are cited so a reword is a grep away rather than + // a silent narrowing. If either moves, this guard quietly falls back to the + // clap branch alone — still a real check, but no longer the one described + // above — so treat a miss here as needing the strings re-sourced, not as + // evidence the command stopped refusing. Both are emitted by + // `parse_device_policy` in `apps/rocm/src/main.rs`: + // "CPU mode is not a fallback path" + // "unsupported device policy" + // Cited by symbol rather than line: an earlier draft gave line numbers, and + // they had drifted by ~170 lines before this PR was through review. + // Matched lowercased, which is why the literals below differ in case from + // the source. + let rejected: Vec<&String> = transcript + .iter() + .filter(|line| { + let lower = line.to_lowercase(); + lower.contains("cpu mode is not a fallback path") + || lower.contains("unsupported device policy") + || (lower.contains("invalid value") && lower.contains("--device")) + }) + .collect(); + assert!( + rejected.is_empty(), + "`serve --help` offers these device policies, but the command refuses one for being \ + what it is:\n{}", + rejected + .iter() + .map(|line| format!(" {line}")) + .collect::>() + .join("\n") + ); +} + +// ── A local server this scenario owns ────────────────────────────── + +/// The recorded id of the server a scenario starts for itself. Distinct from the +/// planted mock's id so the suite's teardown treats it as a real service and +/// tries to stop it too, as a backstop behind the scenario's own assertion. +const OWNED_SERVICE_ID: &str = "e2e-owned-server"; + +#[given("a local server this machine manages is running")] +async fn given_owned_managed_server(world: &mut E2eWorld) { + let pid = world.start_managed_service_process(OWNED_SERVICE_ID, "TestModel/E2E-1B"); + // The CLI must agree the server is there before anything is asked of it, + // otherwise a later "it is gone" would prove nothing. + let (listing, _, _) = crate::run_rocm(world, &["services", "list"]); + assert!( + listing.contains(OWNED_SERVICE_ID), + "the CLI does not list the running server (pid {pid}), so there is nothing to \ + stop:\n{listing}" + ); +} + +#[when("the user stops the server that is running")] +async fn user_stops_running_server(world: &mut E2eWorld) { + // Whichever service this scenario's serve registered. Read from the record + // the CLI wrote in its own (isolated) services directory — the same on-disk + // schema the suite's teardown reads — rather than assuming an id. + let root = world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .join("data") + .join("services"); + let recorded: Vec = std::fs::read_dir(&root) + .into_iter() + .flatten() + .flatten() + .filter_map(|entry| { + // The manifest is `.json`; `.state.json` and other sidecars + // share the extension, so match the manifest name itself. + let name = entry.file_name().into_string().ok()?; + let id = name.strip_suffix(".json")?; + if id.contains('.') { + return None; + } + let bytes = std::fs::read(entry.path()).ok()?; + let record: serde_json::Value = serde_json::from_slice(&bytes).ok()?; + let recorded = record.get("service_id")?.as_str()?; + (recorded == id).then(|| id.to_owned()) + }) + .collect(); + // The server must answer before the stop, or the "no longer answers" check + // that follows could pass on a wrong URL. + let endpoint = world.endpoint.as_ref().expect("no endpoint configured"); + assert!( + model_is_ready(&format!("{endpoint}/models"), None, 30).await, + "{endpoint} does not answer before the stop, so a later silence would prove nothing" + ); + // Exactly one, not the first `read_dir` happens to yield. Directory order is + // unspecified, so with a second record present — a retried serve's leftover, + // or a planted mock alongside a real one — taking the first would stop an + // arbitrary service and the assertion that follows would judge that + // unrelated stop. + assert_eq!( + recorded.len(), + 1, + "expected exactly one managed service under {} so there is no doubt which one is being \ + stopped, found {recorded:?}", + root.display() + ); + let service_id = recorded.into_iter().next().expect("length checked above"); + let (stdout, stderr, rc) = crate::run_rocm(world, &["services", "stop", &service_id, "--yes"]); + world.cli_output = Some(format!("{stdout}{stderr}")); + world.cli_rc = Some(rc); +} + +#[when("the user removes the CLI's managed files")] +async fn user_removes_managed_files(world: &mut E2eWorld) { + // `--keep-binaries` and `--keep-cache` are what keep this safe to run on a + // shared runner. Config and data ARE this scenario's own temporary ones, so + // removing them is real and reaches nothing outside the scenario. The cache + // is NOT: `isolate_env` points `ROCM_CLI_CACHE_DIR` at the persistent + // `E2E_SHARED_CACHE_DIR` whenever CI provides one, which every GPU lane + // does — and this scenario is `@requires-os:linux`, so it runs there. + // Without `--keep-cache` the uninstall would delete the shared download + // cache out from under every later scenario AND every later run. + // + // Neither flag weakens the scenario: what it asserts is whether the CLI + // stops the servers it manages, which has nothing to do with whether it + // also deletes the program or the download cache. The records it must not + // orphan live under the data dir, which is still removed. + let (stdout, stderr, rc) = crate::run_rocm( + world, + &["uninstall", "--yes", "--keep-binaries", "--keep-cache"], + ); + world.cli_output = Some(format!("{stdout}{stderr}")); + world.cli_rc = Some(rc); +} + +#[then("the removal is reported as complete")] +async fn assert_removal_complete(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no uninstall output"); + assert_eq!(world.cli_rc, Some(0), "removal did not succeed:\n{output}"); + assert!( + output.contains("uninstall complete"), + "removal did not report completion:\n{output}" + ); +} + +#[then("the server is no longer running")] +async fn assert_owned_server_stopped(world: &mut E2eWorld) { + let output = world.cli_output.clone().unwrap_or_default(); + // Guards the assertion below against passing vacuously: with no process + // registered there is nothing that could still be running, and the scenario + // would go green having proved nothing. + assert!( + !world.owned_processes.is_empty(), + "this scenario registered no server, so there is nothing to have stopped" + ); + // Asked of the OS, and asked HERE — before this scenario's teardown, which + // also kills what it owns. Deferring the question until after teardown would + // report the harness's own cleanup as the product's work, turning the very + // defect under test into a pass. + let mut still_running = Vec::new(); + for process in &mut world.owned_processes { + if process.is_running().await { + still_running.push(process.pid()); + } + } + assert!( + still_running.is_empty(), + "the CLI reported success but the server it manages is still running (pid \ + {still_running:?}):\n{output}" + ); +} + +#[then("the stopped server no longer answers")] +async fn assert_stopped_server_stops_answering(world: &mut E2eWorld) { + let endpoint = world.endpoint.as_ref().expect("no endpoint configured"); + let url = format!("{endpoint}/models"); + // A signalled server can take a moment to release its port, so the answer is + // given a bounded window to stop. A server that is still listening at the end + // of it was not stopped, whatever the CLI reported. + let deadline = Instant::now() + Duration::from_mins(1); + loop { + if reqwest::get(&url).await.is_err() { + return; + } + assert!( + Instant::now() < deadline, + "the CLI reported the stop, but {url} still answers a minute later" + ); + tokio::time::sleep(Duration::from_secs(2)).await; + } +} + +#[then("the CLI reports that it stopped a process")] +async fn assert_stop_count_reported(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no stop output"); + assert_eq!(world.cli_rc, Some(0), "stopping did not succeed:\n{output}"); + let reported = output + .lines() + .find_map(|line| line.trim().strip_prefix("stopped processes:")) + .map_or_else( + || panic!("the stop gave no account of what it stopped:\n{output}"), + str::trim, + ); + let count: u32 = reported + .parse() + .unwrap_or_else(|_| panic!("could not read the reported count {reported:?}:\n{output}")); + // An operator reading "0" has been told the stop found nothing to do, which + // is the opposite of what happened. + assert!( + count >= 1, + "the server was running and is now stopped, but the CLI reports it stopped \ + {count}:\n{output}" + ); +} + +// ── Choosing a device and an address ─────────────────────────────── + +#[when("the user serves a model letting the CLI choose the GPU")] +async fn user_serves_with_auto_gpu(world: &mut E2eWorld) { + let (model, engine, _) = host_serve_target(); + // `--gpu auto` is the default, named here so the scenario is about the + // choice rather than about what happens to be configured. The plan the CLI + // prints on its way to launching is where it says which device it picked, + // and that line is all this scenario reads — whether the serve then goes on + // to succeed is a different scenario's business. + ensure_serve_port_free().await; + let (stdout, stderr, rc) = crate::run_rocm( + world, + &[ + "serve", + model, + "--engine", + engine, + "--gpu", + "auto", + "--managed", + ], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the plan names the device it chose")] +async fn assert_auto_gpu_named(world: &mut E2eWorld) { + let output = serve_output(world); + let chosen = output + .lines() + .find_map(|line| line.trim().strip_prefix("gpu: auto (selected ")) + .and_then(|rest| rest.split(')').next()) + .map_or_else( + || panic!("the plan does not say which GPU was chosen:\n{output}"), + str::trim, + ); + // "none" on a machine that has a GPU leaves the user unable to tell whether + // their model is about to run on the device at all. + assert_ne!( + chosen, "none", + "this machine has an AMD GPU, but letting the CLI choose one selected \ + nothing:\n{output}" + ); +} + +#[given("the address a new server would use is already taken")] +async fn given_default_serve_address_taken(world: &mut E2eWorld) { + // Hold the address a server started without `--port` would take, so the next + // serve meets exactly the situation a user's second server meets. + ensure_serve_port_free().await; + let listener = std::net::TcpListener::bind(("127.0.0.1", SERVE_PORT)) + .unwrap_or_else(|e| panic!("could not take the default serve address: {e}")); + // Held on the World so it stays taken for the whole scenario and is released + // with it. + // + // Safe only because this scenario is `@requires-gpu`, and every GPU lane is + // serialized to one scenario at a time. `ensure_serve_port_free` clears the + // port with `fuser -k`/`kill -9`, and from the moment this listener exists + // the process holding that port is the cucumber binary itself — so a + // concurrent scenario calling it would kill the whole suite mid-run. The + // serialization is enforced in `e2e.rs`, which pins a GPU lane to one + // scenario regardless of the mock lane's ceiling; if that ever changes, this + // needs an ephemeral port rather than the shared default. + world.occupied_address = Some(listener); +} + +#[when("the user serves a model without choosing an address")] +async fn user_serves_without_choosing_address(world: &mut E2eWorld) { + let (model, engine, _) = host_serve_target(); + // No `--port`, so the CLI picks the address itself — which is the whole + // question. The serve is expected not to come up (something else holds the + // address), so nothing here waits for readiness; the planned address is + // printed before the engine is launched. + let (stdout, stderr, rc) = + crate::run_rocm(world, &["serve", model, "--engine", engine, "--managed"]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the new server does not try to use the taken address")] +async fn assert_new_server_avoids_taken_address(world: &mut E2eWorld) { + let output = serve_output(world); + // The `port:` line read here is the one `serve` prints in its "serve plan" + // block — the `println!` under `serve plan` in `serve` itself + // (`apps/rocm/src/serve_cmd.rs`) — which is what this scenario ran. + // + // Named because grepping the product for `port: {port}` finds a decoy first: + // `run_chat_port_status_tool` writes an identical line for a different + // command entirely. Two reviewers have now landed on that one while checking + // this step, and a third landed on neither because an earlier draft of this + // comment gave line numbers that had already drifted — hence symbols only. + // `serve`'s is indented and the chat tool's is not, which is why + // the `trim()` above is load-bearing rather than defensive — and also why + // matching on indentation instead would be the wrong way to tell them apart, + // since only one of the two can appear in this scenario's output at all. + let planned = output + .lines() + .find_map(|line| line.trim().strip_prefix("port:")) + .map(str::trim); + // Two honest outcomes: pick somewhere else, or say the usual address is + // busy. Planning to use it anyway is the one that collides with the server + // already there. + match planned { + None => { + assert!( + world.cli_rc != Some(0), + "the CLI neither planned an address nor refused:\n{output}" + ); + // A nonzero exit alone is not the second outcome. Any unrelated + // failure — a missing model, a broken runtime — would also plan no + // address and exit nonzero, and would be accepted here as though + // the CLI had honestly reported the conflict. Require it to say so. + // + // Conflict wording, not just the number: an unrelated failure can + // print an endpoint URL that contains the port. A bare "already" + // would admit the serve path's own unrelated uses of the word + // ("already running", "already being served"). + let said = output.to_lowercase(); + assert!( + ["in use", "address already", "port already", "already bound"] + .iter() + .any(|phrase| said.contains(phrase)), + "the CLI refused, but without naming the address that is taken — so this \ + says nothing about whether it noticed the conflict:\n{output}" + ); + } + Some(port) => assert_ne!( + port, + SERVE_PORT.to_string(), + "the address is already taken, but the new server plans to use it \ + anyway:\n{output}" + ), + } +} + /// Serve bound to a public (non-loopback) interface without the public-bind /// opt-in. The bind-host validation is the first thing `serve` does — before any /// engine, model, or GPU work — so this refusal is observable with no GPU (mock diff --git a/tests/e2e-cucumber/tests/feature_naming.rs b/tests/e2e-cucumber/tests/feature_naming.rs index 230f791c8b..422f56a4a1 100644 --- a/tests/e2e-cucumber/tests/feature_naming.rs +++ b/tests/e2e-cucumber/tests/feature_naming.rs @@ -63,16 +63,104 @@ fn features_dir() -> PathBuf { } /// Every `.feature` file actually present, by file name. +/// +/// Flat, not recursive — and it refuses to run rather than quietly covering +/// less if that stops being true. `feature_files` in `src/expectation.rs` makes +/// the same assumption and its refusal message names THIS scan as the other +/// place to fix, so the two guards belong together: a subdirectory that made +/// one of them fail loudly while the other silently skipped it would be the +/// worst of both. fn feature_files() -> Vec { - let mut names: Vec = std::fs::read_dir(features_dir()) - .expect("features dir") - .map(|e| e.expect("dir entry").file_name().to_string_lossy().into()) - .filter(|n: &String| n.ends_with(".feature")) - .collect(); + feature_files_in(&features_dir()) +} + +/// The scan itself, split out so the subdirectory refusal can be exercised +/// against a temporary tree — against the real `features/` it could only fire +/// by someone breaking the repository. +/// +/// Deliberately the same shape as `feature_files_in` in `src/expectation.rs`: +/// same refusal, the same `Path::extension()` test rather than a `.feature` +/// string suffix, and the same non-empty assertion. The extension test is not +/// interchangeable with the suffix one — a file named exactly `.feature` has no +/// extension and would have been taken by one scan and skipped by the other, +/// which is the divergence this pair exists to prevent. +fn feature_files_in(dir: &Path) -> Vec { + let mut names: Vec = Vec::new(); + for entry in std::fs::read_dir(dir).expect("features dir") { + let path = entry.expect("dir entry").path(); + assert!( + !path.is_dir(), + "features/ has grown a subdirectory ({}), which this scan does not descend \ + into — make it recursive, here and in src/expectation.rs, before moving any \ + .feature file into one", + path.display() + ); + if path.extension().is_some_and(|ext| ext == "feature") { + let name = path.file_name().expect("dir entry name").to_string_lossy(); + names.push(name.into_owned()); + } + } + assert!( + !names.is_empty(), + "found no .feature files in {}", + dir.display() + ); names.sort(); names } +#[test] +fn the_feature_scan_takes_the_flat_files_it_finds() { + // Pins FILTERING and MEMBERSHIP, not sort order. `read_dir` order is + // unspecified and happens to come back alphabetical here, so removing + // `names.sort()` leaves this green — verified. Nothing can force a real + // directory to yield entries out of order, so the ordering is asserted for + // a stable comparison rather than because this test proves it. + let dir = tempfile::tempdir().expect("no temp dir"); + std::fs::write(dir.path().join("b.feature"), "Feature: b\n").unwrap(); + std::fs::write(dir.path().join("a.feature"), "Feature: a\n").unwrap(); + std::fs::write(dir.path().join("notes.md"), "ignored\n").unwrap(); + // A file named exactly `.feature` has no extension, so neither this scan nor + // its sibling takes it. Present here so the two stay agreed on that. + std::fs::write(dir.path().join(".feature"), "not a feature file\n").unwrap(); + assert_eq!(feature_files_in(dir.path()), ["a.feature", "b.feature"]); +} + +#[test] +#[should_panic(expected = "found no .feature files")] +fn the_feature_scan_refuses_a_directory_with_no_feature_files() { + // The assertion this pins turns a `features/` that stopped yielding files + // into ONE clear failure naming the directory. It is not what stops the + // other checks passing vacuously — they already fail on their own, just + // confusingly: `feature_files_and_declared_keys_agree` reports the first + // FEATURE_KEYS entry as one that "does not exist", and the other three + // panic inside `scenarios_of` with a read error. Verified by pointing + // `features_dir()` at an empty directory with the assertion deleted: four + // of the five checks fail, none of them saying the directory is empty. + // + // Unreachable against the real directory, so pinned here — and pinned the + // same way in `src/expectation.rs`, whose scan carries the same assertion. + // + // The fixture writes a non-`.feature` file on purpose: what the scan + // refuses is an empty RESULT, not an empty directory, and this covers the + // stronger case. + let dir = tempfile::tempdir().expect("no temp dir"); + std::fs::write(dir.path().join("notes.md"), "ignored\n").unwrap(); + let _ = feature_files_in(dir.path()); +} + +#[test] +#[should_panic(expected = "has grown a subdirectory")] +fn the_feature_scan_refuses_a_subdirectory_rather_than_skipping_it() { + // The branch that stops this scan quietly covering less than it claims. + // Unreachable against the real `features/`, so it is pinned here — the same + // way `src/expectation.rs` pins its twin. + let dir = tempfile::tempdir().expect("no temp dir"); + std::fs::write(dir.path().join("a.feature"), "Feature: a\n").unwrap(); + std::fs::create_dir(dir.path().join("nested")).unwrap(); + let _ = feature_files_in(dir.path()); +} + /// The `@id:` tags and scenario names in one feature file, paired in declaration /// order. Tags precede their scenario, so the most recent id seen belongs to the /// next scenario line.