Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
35c0942
test(e2e): pin the contracts a README walkthrough expects (EAI-8024)
fredespi Aug 12, 2026
f06f417
test(e2e): correct the GPU rows against what run 188 measured
fredespi Aug 12, 2026
e9b1547
test(e2e): drop what the runners cannot exercise, keep what they can
fredespi Aug 12, 2026
de39b90
test(e2e): renumber the serving scenarios around the one main added
fredespi Aug 12, 2026
dd8ef73
test(e2e): scope the auto-GPU-selection contract to Windows (EAI-8006)
fredespi Aug 12, 2026
a77ab32
test(e2e): let the stop-reporting scenario inherit the lemonade serve…
fredespi Aug 13, 2026
e6d2708
test(e2e): move the new real-serve scenarios to the merge-queue path
fredespi Aug 13, 2026
e345143
test(e2e): read the device policies from the right option's help
fredespi Aug 13, 2026
6feed98
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Aug 18, 2026
c60cac7
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Sep 9, 2026
5f27fd3
style(e2e): rustfmt the device-group selection
fredespi Sep 9, 2026
7e25936
test(e2e): prefix the new scenario ids with their feature key
fredespi Sep 9, 2026
0647ff2
test(e2e): bound mock-lane concurrency to the machine
fredespi Sep 9, 2026
f59ce69
test(e2e): observe the gen-tps validity window by sampling, not one look
fredespi Sep 9, 2026
056af16
Revert the gen-tps sampling change, keep the concurrency bound
fredespi Sep 9, 2026
1403b28
test(e2e): declare EAI-7960 and bound mock-lane concurrency
fredespi Sep 9, 2026
4d3669d
test(e2e): fix findings from a review of this round's own changes
fredespi Sep 9, 2026
2e8772b
test(e2e): wait for the engine-shell prompt line, not the banner
fredespi Sep 9, 2026
03f144d
test(e2e): make two guards discriminating rather than silently satisf…
fredespi Sep 10, 2026
19cc220
test(e2e): scope update-02's readback to what its fixture can prove
fredespi Sep 10, 2026
1723f93
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Sep 14, 2026
d6825bb
test(e2e): delete the EAI-7960 rows, which main has made stale
fredespi Sep 14, 2026
542ebe0
test(e2e): drop the trailing blank line left by deleting the last rows
fredespi Sep 14, 2026
ea4fd91
test(e2e): stop bounding mock-lane concurrency, which broke two of ma…
fredespi Sep 14, 2026
a80429f
test(e2e): close three gaps where a guard read more loosely than it c…
fredespi Sep 14, 2026
201f790
test(e2e): cite the two refusal strings, correct the checker's platfo…
fredespi Sep 14, 2026
478f4ea
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 14, 2026
91747cb
test(e2e): wait through the driver, and name what the docs and a grep…
fredespi Sep 14, 2026
93d8dce
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 14, 2026
2b36fec
test(e2e): point the merge-queue docs at the right file, and guard th…
fredespi Sep 15, 2026
22ee61d
test(e2e): stop the new prose guard blaming an innocent scenario
fredespi Sep 15, 2026
d8674b0
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 15, 2026
b6d1fda
test(e2e): make the prose guard report what it cannot bind, and re-so…
fredespi Sep 15, 2026
82506e8
test(e2e): stop claiming uniqueness for a fact recorded in two more p…
fredespi Sep 15, 2026
f713560
test(e2e): retire the cli_stderr exception instead of documenting aro…
fredespi Sep 15, 2026
275f4c0
test(e2e): cite product code by symbol, and stop writing counts that rot
fredespi Sep 15, 2026
5c5ae67
test(e2e): correct the non-blocking reason in the report, and pin the…
fredespi Sep 15, 2026
bc87b89
test(e2e): fix the proven-out reason where it is most visible, and gu…
fredespi Sep 15, 2026
f911952
test(e2e): pin the sibling refusal, and widen the update row to its r…
fredespi Sep 15, 2026
462d9de
test(e2e): make the two feature scans actually agree, and fix the hea…
fredespi Sep 15, 2026
e8482bf
test(e2e): pin the non-empty assertion the last commit added but did …
fredespi Sep 15, 2026
f5629a9
test(e2e): say what the non-empty assertion actually buys, not what i…
fredespi Sep 15, 2026
4e2a22b
docs(ci): re-date the required-check observation, and say how it was …
fredespi Sep 15, 2026
14e0c55
test(xtask): make the two copies of the required-check date fail when…
fredespi Sep 15, 2026
2b84f9c
test(xtask): correct the reason given for flattening the date extractor
fredespi Sep 15, 2026
6cfb628
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 16, 2026
8b71317
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 16, 2026
e9c3356
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 16, 2026
ad44496
style(e2e): apply rustfmt to the merged drain helper
fredespi Sep 16, 2026
13d7a18
test(e2e): retire the dead drain arm the merge left, and tighten four…
fredespi Sep 16, 2026
8ad6f76
test(e2e): narrow the busy-address match, and keep the drain sentinel…
fredespi Sep 16, 2026
c2e929e
test(e2e): say which guards guard a fix, and make the absent label un…
fredespi Sep 17, 2026
0f5394b
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 17, 2026
05717c1
test(e2e): drop an invented bound, and fix two wait diagnostics that …
fredespi Sep 17, 2026
4067420
test(e2e): make the fourth wait diagnostic fit the clause, and state …
fredespi Sep 17, 2026
3d09ade
docs(e2e): mark the neighbouring wait convention deliberate, and cite…
fredespi Sep 17, 2026
3cf2d93
test(e2e): pin the wait diagnostics their doc comments describe
fredespi Sep 17, 2026
807e35b
fix(e2e): build the pinned templates instead of writing them as literals
fredespi Sep 17, 2026
b44bf26
test(e2e): check the doc side too, which the guard's own name claimed…
fredespi Sep 17, 2026
a5cb1fc
test(e2e): pin every fragment, and read the count the doc actually st…
fredespi Sep 17, 2026
97615d6
fix(e2e): read the human report from the slot its own When writes
fredespi Sep 19, 2026
6b3256d
test(e2e): read the code side from code, not from the doc vouching fo…
fredespi Sep 19, 2026
e8626d3
test(e2e): cover the four expectation branches nothing was asserting
fredespi Sep 19, 2026
f87ab1d
test(e2e): close the block-comment hole, and make an absent cause mea…
fredespi Sep 25, 2026
22c7744
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Sep 25, 2026
bd8a44c
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Oct 1, 2026
0fd0ffd
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Oct 27, 2026
dd5fce0
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Oct 27, 2026
94f1c00
test(e2e): address review round on the README-contract pins
fredespi Oct 27, 2026
7ce66d5
style(xtask): rustfmt the ascii-contract directory skip
fredespi Oct 27, 2026
40d58dd
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Oct 28, 2026
1017b64
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Oct 28, 2026
1cb79b8
Merge origin/main into e2e-pin-the-contracts-for-the-13-readme
fredespi Oct 28, 2026
a9114f7
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Oct 30, 2026
ceb3fb1
test(e2e): tighten wording guard and label test, fix doc drift from r…
fredespi Oct 30, 2026
090539f
test(e2e): check the stop really stopped, frame uninstall as a propos…
fredespi Oct 30, 2026
4002c3e
test(e2e): use from_mins for the stop-check deadline (clippy)
fredespi Oct 30, 2026
7768c4d
Merge origin/main; drop work unrelated to the README contracts
fredespi Nov 2, 2026
3012275
e2e: tighten the README-contract steps per review
fredespi Nov 2, 2026
26d16dc
Merge origin/main; renumber this PR's serve scenarios to 33..37
fredespi Nov 3, 2026
de40a17
Merge origin/main; renumber this PR's serve scenarios to 36..40
fredespi Nov 3, 2026
dc5638b
Merge origin/main; gate the uninstall scenario behind @lifecycle
fredespi Nov 3, 2026
1dce98d
Merge origin/main; check substituted device premises, tighten port wo…
fredespi Nov 3, 2026
d180cba
e2e: cite the runs behind the measured rows, and cite stat_device by …
fredespi Nov 3, 2026
97f1082
Merge origin/main; renumber serve scenarios to 42..46, drop the stale…
fredespi Nov 5, 2026
0703a1a
style: end expectations.toml with a single newline
fredespi Nov 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 28 additions & 4 deletions tests/e2e-cucumber/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,18 @@ E2E_INCLUDE_LIFECYCLE=1 E2E_ONLY_LIFECYCLE=1 cargo xtask e2e

## Tags and per-scenario expectations

There is no tag-filter tiering. Each CI job runs the **whole** suite
(`cargo xtask e2e`, no `-t` filter); the harness resolves every scenario to
**pass / xfail / skip** at runtime from its capability tags plus the known-bug
matrix, then reconciles the actual result against that expectation.
Each CI job runs the suite as a whole (`cargo xtask e2e`, no `-t` filter); the
harness resolves every scenario to **pass / xfail / skip** at runtime from its
capability tags plus the known-bug matrix, then reconciles the actual result
against that expectation.

Four tags tier the suite at runtime rather than by a `-t` filter, so a scenario
carrying one runs only where its env var is set: `@nightly`
(`E2E_INCLUDE_NIGHTLY`), `@lifecycle` (`E2E_INCLUDE_LIFECYCLE`),
`@requires-docker` (`E2E_INCLUDE_DOCKER`) and `@merge-queue` (`E2E_MERGE_QUEUE`,
set on the self-hosted merge-queue lanes). A `@merge-queue` scenario does not run
on ordinary per-PR CI, only on the self-hosted lanes in
[`.github/workflows/e2e-selfhosted.yml`](../../.github/workflows/e2e-selfhosted.yml).

### Naming

Expand Down Expand Up @@ -147,6 +155,8 @@ Scenarios carry stable-id and capability tags:
| `@requires-os:<linux\|windows>` | Premise is OS-specific; skip on other OSes. |
| `@requires-oom-fault-injection` | Needs the `rocm` binary under test to carry the test-only `e2e-oom-fault-injection` hook, which fabricates a managed launch that runs out of GPU memory so the OOM guidance can be verified without a GPU. Unlike every other capability this one **cannot be probed** — no product command exposes it — so `xtask e2e` reports it via `ROCM_E2E_OOM_FAULT_INJECTION=1`, set only when xtask compiled the binary itself with the feature. A prebuilt `ROCM_CLI_BINARY` (the self-hosted lanes' shipping release build) has the hook compiled out, so these scenarios resolve to **skip** there and run on the mock lane. |
| `@serve-timeout:<secs>` | Lengthen the serve-readiness wait for a genuinely slow serve (e.g. a large model). |
| `@requires-no-gpu` | Premise is a host with no usable AMD GPU (e.g. a refusal that only happens without one). The inverse of `@requires-gpu`; resolves to **skip** on a GPU host. |
| `@merge-queue` | Too expensive for per-PR CI (a real serve). Skipped unless `E2E_MERGE_QUEUE` is set, which every self-hosted lane in [`e2e-selfhosted.yml`](../../.github/workflows/e2e-selfhosted.yml) except the MI300X one does on a `merge_group` event. |
| `@nightly` | Expensive scenario skipped by default; included when `E2E_INCLUDE_NIGHTLY=1`. |
| `@lifecycle` | Expensive, OS-mutating release-lifecycle scenario (packaging + real installer + install/uninstall). Skipped by default; included when `E2E_INCLUDE_LIFECYCLE=1`. `E2E_ONLY_LIFECYCLE=1` selects only this set without bypassing expectation resolution. A caller can additionally fold specific non-`@lifecycle` scenarios into that restricted set via `E2E_ALSO_RUN_IDS` (comma-separated `@id`s), without tagging them `@lifecycle` — which would also pull them out of the full-suite lanes that don't set `E2E_INCLUDE_LIFECYCLE`. `ci.yml`'s Windows lane uses this to fold in the hermetic comfyui-01/02/03 scenarios and the native-Windows vLLM refusal scenarios serve-33/34/35. |

Expand All @@ -159,6 +169,20 @@ matches a condition is expected to fail (xfail); if it then passes, that is an
bug. See `src/expectation.rs` for the resolver and `expectations.toml`'s header
for the condition grammar.

When a comment here points at product code, **cite a symbol, never a line
number** — `select_runtime_update_source` in [`apps/rocm/src/main.rs`](../../apps/rocm/src/main.rs), not

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new guidance's own example cites the wrong file: select_runtime_update_source is defined in apps/rocm/src/update_cmd.rs, not apps/rocm/src/main.rs (confirmed by grep — the symbol does not appear anywhere in main.rs — and by docs/architecture.md's own note on its Phase 6f relocation out of main.rs). Since this paragraph is specifically teaching "cite a symbol, never a line number" so citations don't go stale, an inaccurate example undercuts the point. Suggest repointing the link/citation at update_cmd.rs.

Confidence 85 · mechanical

`main.rs:16427`. `main.rs` alone is many thousands of lines and
moves every week; nothing checks these numbers, so a stale one sends the reader
somewhere confidently wrong, which is worse than no pointer at all.

When a scenario's comment block says **"Expected to FAIL"**, write it directly
above that scenario's `@id:` tag line with **no blank line between them**. A test
binds each such claim to the tags immediately beneath it and fails if the row has
since been deleted — so the prose cannot quietly come to say the opposite of what
the suite enforces. A blank line breaks that binding, and the same test then
fails naming the comment's own file and line rather than letting the scenario
drop out of the check unnoticed.

On a GPU host, a `serve` precondition that never publishes its model is
relaunched once — but only for a scenario expected to pass; a known bug keeps its
shortened `serve_timeout_secs` and fails on the first attempt. The stalled
Expand Down
147 changes: 147 additions & 0 deletions tests/e2e-cucumber/expectations.toml
Original file line number Diff line number Diff line change
Expand Up @@ -164,3 +164,150 @@ bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the benchmark never reaches a live model."
serve_timeout_secs = 90
flaky = true

# ── Contracts found by walking the public README end to end ─────────────────
#
# The rows below came out of one walkthrough of the README against a nightly build
# on an MI300X pod (EAI-8024 tracks the set). Most scenarios assert behaviour the
# walkthrough leads a reader to expect; a few are contracts the walkthrough implies
# rather than states (diagnose and fix agreeing, a usable device not being
# diagnosed as a permission failure, a second server avoiding a taken address), and
# one is a proposal the README does not make (uninstall stopping what it manages,
# below). Each row says the product does not do that yet — so the day one is fixed
# its row goes stale and deletes itself.

# --- EAI-8020: `diagnose` and `fix` each carry their own copy of the
# render-group remedy, and the verification step has drifted between them
# (`diagnose` adds `ls -l /dev/kfd`). Both strings are constants, so this holds
# wherever the catalog runs on bare metal; `@requires-os:linux` and
# `@requires-bare-metal` keep it off Windows and WSL2, where the recipe does not apply. Remove when the two are one. ---
[["diagnose-and-fix-agree-on-how-to-verify"]]
when = {}
bug = "EAI-8020"
reason = "The diagnosis and the fix preview print different verification steps for the same remedy."

# --- EAI-8009: with no `--model`, the local chat provider only looks for the
# built-in Lemonade assistant model, so a ready managed service running anything
# else is ignored and the user is told nothing is running. The scenario plants
# its own ready service, so this needs no hardware and holds on every lane.
# Remove when chatting locally uses whichever local server is ready. ---
[["chat-local-uses-the-running-server"]]
when = {}
bug = "EAI-8009"
reason = "The local chat provider only accepts the built-in assistant model, so a ready local server running any other model is not used."

# --- EAI-7423, not EAI-8007. This scenario's premise is a REAL serve on whatever
# engine the host defaults to, so on a lemonade LINUX host it inherits the same
# managed-serve bug as its siblings above: the serve reaches ready and is shut
# down ~0.08s later, so the endpoint never becomes reachable and the precondition
# — not the contract under test — is what fails. Measured on Strix Halo Ubuntu:
# passed in run 196, failed in run 210 with "endpoint did not serve model
# Qwen3-0.6B after 2 attempt(s) of 300s", which is that signature exactly.
#
# Scoped and flagged identically to the other EAI-7423 rows: os=linux because
# native Windows lemonade serve works (it passes on Strix Windows), gfx* because
# a real GPU host is needed at all, and `flaky = true` because which of these
# pass varies run to run — a deterministic-XPASS row would fail that lane on
# unrelated PRs. Delete this row with the other EAI-7423 ones, not with EAI-8007.
[["serve-services-stop-reports-what-it-stopped"]]
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*" }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the service this scenario needs to stop never comes up."
serve_timeout_secs = 90
flaky = true

# For the contract this scenario actually exists to hold:
# EAI-8007 has NO row on purpose. `services stop` reported stopping 0 processes
# for a service it did in fact stop on the pod, but neither fixture reproduces it
# here: a plain child process registered as a managed service was counted
# correctly in a one-off no-GPU check during development, and so was a real vLLM
# serve (MI300X, run 188). The shipped scenario is the real-serve one, which is
# `@requires-gpu @merge-queue`, so no blocking lane holds this contract. It ships
# as a guard rather than an expected failure, because a row would be a stale
# XPASS and would fail the lane. The ticket keeps the
# pod observation; what distinguishes that host is still unknown.

# --- EAI-8014: a PROPOSED contract, not a README one. The README documents that
# `uninstall` warns it does not stop managed background processes; this row pins
# the stronger behaviour the ticket asks for — `uninstall` removes the managed
# files and reports completion without stopping the servers it manages, and takes
# the service records with it, so the supported way to stop them goes too. The
# warning is documented behaviour, so this stays a row only until the product and
# README change together. No hardware needed. Remove when uninstall stops what it
# manages first, or when the contract is dropped. ---
[["serve-uninstall-stops-what-it-manages"]]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unconditional rows (when = {}) turn any failure of this scenario, including a failing Given (the services list precondition), into the expected failure. Please show in the PR that the failure is at the final Then and that the premise steps pass.

@fredespi fredespi Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the mock lane at 3012275 (cargo xtask e2e: 5 xfail, 0 XPASS, 0 unexpected), each unconditional row's scenario fails at its final step and every premise step passes:

  • chat-13: Given a model is being served ✔, And the model is registered ✔, When one-shot chat without a model ✔, Then the CLI prints the assistant's reply ✘
  • diagnose-48: Given a device-permission failure ✔, When compare diagnosis with fix preview ✔, Then both give the same way to verify ✘
  • serve-30: Given a managed server is running ✔, When removing managed files ✔, Then removal reported complete ✔, And the server is no longer running ✘

diagnose-50/51 need a real device, and serve-32 needs a merge-queue run, so the mock lane cannot show them. I will add the step-level evidence for them from a GPU-lane run.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This row pins a contract the README contradicts (README says uninstall warns it does not stop managed processes). Confirm a permanent xfail is intended until product and README change together.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's deliberate: the row stays an expected failure until the product and the README change together. It is reported as a proposed contract (EAI-8014), not a README promise, in both the scenario comment and the PR description. Whether to adopt that contract is for the maintainers to decide on EAI-8014. If you'd rather this PR not carry a proposal, I can drop serve-38.

when = {}
bug = "EAI-8014"
reason = "Uninstall reports completion while leaving the servers it manages running, and removes the records needed to stop them."

# --- EAI-8020. The other half of the same divergence: `diagnose` names
# the device's actual owning group while the `fix` recipe is hard-coded to
# `render`, so the two propose different changes wherever the device's group is
# not `render`. Needs a real device node, hence the GPU lane. Measured on the
# MI300X lane (E2E self-hosted ROCm/rocm-cli run 192) and the Strix Halo Ubuntu lane
# (run 196), failing for this reason on both, with 0 XPASS. The group is substituted
# by a `stat` shim, so the outcome does not depend on the host. ---
[["diagnose-and-fix-agree-on-the-remedy-command"]]
when = {}
bug = "EAI-8020"
reason = "The diagnosis proposes adding the user to the device's own group while the fix is hard-coded to `render`."

# --- EAI-8021. The render-group check only ever ADDS score for missing
# group membership and never credits access that demonstrably works, so a device
# the user can already read and write is still diagnosed as a permission problem.
# The scenario substitutes a world-writable device answer and asserts the
Comment thread
fredespi marked this conversation as resolved.
Resolved
# inspection agrees before relying on it. It shims BOTH `stat` (device
# `crw-rw-rw- root root`) and `id` (groups `users` — none of the conventional
# ones), so the runner's own group membership cannot reach the result: the
# outcome is the same on every bare-metal Linux GPU host. If this XPASSes, the
# product changed; it is not a host difference, and NOT a reason to add
# `flaky = true`. Measured on MI300X (run 192) and Strix Halo Ubuntu (run 196),
# failing for this reason on both, with 0 XPASS. ---
[["diagnose-credits-a-usable-device"]]
when = {}
bug = "EAI-8021"
reason = "A device the user can already read and write is still offered as a group-permission cause."

# --- EAI-8006: `--gpu auto` reports `selected none` on a machine that has a GPU,
# because the CLI cannot get an answer out of `amd-smi` — on the pod the bundled
# copy could not execute at all (its `#!/usr/bin/env python3` shebang found no
# interpreter), even though the CLI has a working interpreter recorded in its own
# config.
#
# os = "windows" is MEASURED, not guessed, and the three lanes separate the two
# candidate explanations cleanly:
# MI300X gfx943 linux PASSES (runs 190/191/192)
# Strix Halo Ubuntu gfx1151 linux PASSES (run 196)
# Strix Halo Windows gfx1151 windows FAILS (run 196) — `gpu: auto (selected none)`
# Same GPU family on both Strix lanes, so this is the OS, not the family — which
# is why the row is not scoped with therock_family the way
# `examine-both-forms-agree-on-gpu` is. Remove it when a Windows host names the
# device it picked. ---
[["serve-auto-gpu-selection-names-a-device"]]
Comment thread
fredespi marked this conversation as resolved.
Resolved
when = { os = "windows" }
bug = "EAI-8006"
reason = "Letting the CLI choose a GPU reports that it selected none on a Windows machine that has one."
serve_timeout_secs = 180

# --- EAI-8016. The serve path has no address-availability check: the
# default port is used whether or not something already holds it, so a second
# server is handed the address the first is on. The scenario holds the address
# itself rather than starting a second real server. Measured on all three GPU
# lanes before the scenario became `@merge-queue`: MI300X (run 192), Strix Halo
# Ubuntu (run 196) and Strix Halo Windows (runs 196 and 210). It failed for this
# reason on each, Windows included, with 0 XPASS. ---
[["serve-second-server-gets-a-free-port"]]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This row is when={} but its scenario (serve-32) is @merge-queue, so it is skipped on the MI300X lane the body credits it to, and only non-blocking Strix merge-queue lanes run it. Was it measured anywhere? An unmeasured unscoped row risks an XPASS on lanes where the bug does not reproduce.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair: it is unmeasured in its current form. It was measured before it became @merge-queue: E2E self-hosted runs 192 (MI300X) and 196/210 (Strix Ubuntu and Windows) all failed it for the stated reason (the taken default address still assigned), with 0 XPASS. It went @merge-queue later to keep the Strix lane under its time cap, and since then only a merge_group run sets E2E_MERGE_QUEUE, so neither PR runs nor a workflow_dispatch can run it. The scenario holds the address itself rather than relying on a second real server, so it does not depend on which engine or GPU the lane has, and serve_cmd.rs on main has no port-availability change since. If it does go stale, the XPASS shows up on the non-blocking merge-queue lanes. The description (7768c4d) no longer credits it to the per-PR MI300X lane.

when = {}
bug = "EAI-8016"
reason = "A server started without an address is given the default one even when it is already in use."
serve_timeout_secs = 120

# EAI-8017 and EAI-8018 have NO rows and NO scenarios, because neither can be
# exercised on any lane available here. Both need a working Lemonade install,
# and Lemonade's own launcher cannot start on either lane: on the MI300X runner
# `lemond` wants GLIBC_2.38 / GLIBCXX_3.4.32, which that OS does not have
# (measured, runs 190/191), and in a plain container it wants
# libdrm_amdgpu.so.1. Both die before reaching the behaviour under test, so a
# scenario would pass vacuously — worse than no scenario, because it would read
# as coverage. They were reproduced by hand on the pod (ubuntu 24.04) and stay
# on their tickets until a lane can run Lemonade.
11 changes: 11 additions & 0 deletions tests/e2e-cucumber/features/chat.feature
Original file line number Diff line number Diff line change
Expand Up @@ -150,3 +150,14 @@ Feature: Chat and endpoint detection
Then the planning assistant was asked exactly once
And the plan text returned is the one the planning assistant gave
And the action returned is the one the planning assistant gave

# Expected to FAIL. Scenario 8 names the model and works; this one leaves the
# choice to the CLI and is told nothing is running — on the very machine whose
# services list shows a ready local server. Asking to chat locally should use
# the local server that is there, whichever model the user chose to serve.
@id:chat-local-uses-the-running-server
Scenario: chat-13 - Chatting locally uses the local server that is already running
Given a model is being served
And the model is registered with the CLI
When the user sends a one-shot chat prompt without naming a model
Then the CLI prints the assistant's reply
48 changes: 48 additions & 0 deletions tests/e2e-cucumber/features/diagnose.feature
Original file line number Diff line number Diff line change
Expand Up @@ -748,3 +748,51 @@ Feature: Diagnosing failures and listing fixes
Given a machine with an AMD GPU
When the user diagnoses this machine in machine-readable form, with no symptom
Then the diagnosis lists the HIP SDK cause only when the inspection finds something that needs it

# Expected to FAIL. `diagnose` and `fix` are two views of the same remedy: the
# first tells the user what will put the machine right, while the second is the
# command they are told to run. They must not leave the user with two different
# definitions of what proves the remedy worked.
@id:diagnose-and-fix-agree-on-how-to-verify @requires-os:linux @requires-bare-metal
Scenario: diagnose-49 - Diagnosing a problem and previewing its fix agree on how to verify it
Given a user who hit a device-permission failure
When the user compares the diagnosis with the matching fix preview
Then both give the same way to verify that the fix worked

# A proposed remedy has to name a group that actually exists on the machine.
# When the device belongs to a group the local group database does not name,
# `stat` answers the literal `UNKNOWN`, and the diagnosis used to print that
# lookup failure as if it were a group the user could join. Since fixed on main
# (the lookup failure now falls back to the conventional group), so this ships
# as a guard rather than an expected failure.
@id:diagnose-commands-name-a-real-group @requires-gpu @requires-bare-metal @requires-os:linux
Scenario: diagnose-50 - Every group named in a diagnosis is one the machine recognises
Given the GPU device belongs to a group the machine cannot name
When the user asks the CLI to diagnose a device-permission failure
Then every group named in the remedy exists on the machine

# Expected to FAIL on every bare-metal Linux GPU host. The device's group is
# substituted (a `stat` shim names a real group other than the hard-coded
# default), so the premise does not depend on the host, and the CLI's own
# inspection is checked to see it first. The diagnosis promises one command
# and its matching fix previews another. This is distinct from scenario
# diagnose-49: even after the verify text agrees, the actual change must agree
# too.
@id:diagnose-and-fix-agree-on-the-remedy-command @requires-gpu @requires-bare-metal @requires-os:linux
Scenario: diagnose-51 - Diagnosing a problem and previewing its fix agree on the remedy command
Given the GPU device belongs to a recognised non-default group
When the user compares the diagnosis with the matching fix preview
Then both give the same command for applying the remedy

# Expected to FAIL on every bare-metal Linux GPU host. The usable device is
# substituted, not a property of the host: `stat` and `id` shims make the
# device world-readable and writable and put the user in none of the
# conventional groups, and the CLI's own inspection is checked to agree first.
# Membership in one conventional group is only a means to access the device,
# not the outcome. A diagnosis should credit the observable access the user
# has instead of recommending a permission repair for a usable device.
@id:diagnose-credits-a-usable-device @requires-gpu @requires-bare-metal @requires-os:linux
Scenario: diagnose-52 - A usable GPU device is not diagnosed as a permission failure
Given the user can already read and write the GPU device
When the user asks the CLI to diagnose the machine
Then adding the user to a device group is not offered as a cause
26 changes: 26 additions & 0 deletions tests/e2e-cucumber/features/examine.feature
Original file line number Diff line number Diff line change
Expand Up @@ -469,3 +469,29 @@ Feature: GPU detection and system inspection
Given a machine with a ROCm install that was not set up by the CLI
When the user inspects the system in machine-readable form
Then the inspection reports a current install-method name

# The help's own worked examples are the first thing a new user copies, so a
# model named there has to be one this CLI can actually serve. The check
# accepts either form the README documents — a name the model listing knows,
# or an explicit `owner/repo` reference — and so does not prescribe which
# model the examples should use.
#
# Since fixed on main (EAI-8011 swapped the unresolvable example name for a
# real catalog alias), so this ships as a guard rather than an expected
# failure.
@id:examine-help-serve-example-names-a-resolvable-model
Scenario: examine-31 - Every model the help offers as an example is one the CLI can resolve
When the user reads the serve examples the help offers
Then every model named there is one the CLI can resolve

# Running the CLI with no subcommand and running its dashboard command are two
# different things. When the help describes them as the same one, a reader has
# no way to learn what the plain command does or that there is anything else
# to reach.
#
# Since fixed on main (the same EAI-8011 change now describes bare `rocm` as
# the launcher), so this ships as a guard rather than an expected failure.
@id:examine-help-describes-the-default-command
Scenario: examine-32 - The help tells the two ways of opening a screen apart
When the user asks for help
Then running the CLI with no subcommand is not described as the dashboard command
Loading
Loading