Repository navigation
test(e2e): pin the contracts a README walkthrough expects (EAI-8024) #241
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
35c0942
f06f417
e9b1547
de39b90
dd8ef73
a77ab32
e6d2708
e345143
6feed98
c60cac7
5f27fd3
7e25936
0647ff2
f59ce69
056af16
1403b28
4d3669d
2e8772b
03f144d
19cc220
1723f93
d6825bb
542ebe0
ea4fd91
a80429f
201f790
478f4ea
91747cb
93d8dce
2b36fec
22ee61d
d8674b0
b6d1fda
82506e8
f713560
275f4c0
5c5ae67
bc87b89
f911952
462d9de
e8482bf
f5629a9
4e2a22b
14e0c55
2b84f9c
6cfb628
8b71317
e9c3356
ad44496
13d7a18
8ad6f76
c2e929e
0f5394b
05717c1
4067420
3d09ade
3cf2d93
807e35b
b44bf26
a5cb1fc
97615d6
6b3256d
e8626d3
f87ab1d
22c7744
bd8a44c
0fd0ffd
dd5fce0
94f1c00
7ce66d5
40d58dd
1017b64
1cb79b8
a9114f7
ceb3fb1
090539f
4002c3e
7768c4d
3012275
26d16dc
de40a17
dc5638b
1dce98d
d180cba
97f1082
0703a1a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -164,3 +164,150 @@ bug = "EAI-7423" | |
| reason = "Lemonade managed serve reaches ready then shuts down immediately, so the benchmark never reaches a live model." | ||
| serve_timeout_secs = 90 | ||
| flaky = true | ||
|
|
||
| # ── Contracts found by walking the public README end to end ───────────────── | ||
| # | ||
| # The rows below came out of one walkthrough of the README against a nightly build | ||
| # on an MI300X pod (EAI-8024 tracks the set). Most scenarios assert behaviour the | ||
| # walkthrough leads a reader to expect; a few are contracts the walkthrough implies | ||
| # rather than states (diagnose and fix agreeing, a usable device not being | ||
| # diagnosed as a permission failure, a second server avoiding a taken address), and | ||
| # one is a proposal the README does not make (uninstall stopping what it manages, | ||
| # below). Each row says the product does not do that yet — so the day one is fixed | ||
| # its row goes stale and deletes itself. | ||
|
|
||
| # --- EAI-8020: `diagnose` and `fix` each carry their own copy of the | ||
| # render-group remedy, and the verification step has drifted between them | ||
| # (`diagnose` adds `ls -l /dev/kfd`). Both strings are constants, so this holds | ||
| # wherever the catalog runs on bare metal; `@requires-os:linux` and | ||
| # `@requires-bare-metal` keep it off Windows and WSL2, where the recipe does not apply. Remove when the two are one. --- | ||
| [["diagnose-and-fix-agree-on-how-to-verify"]] | ||
| when = {} | ||
| bug = "EAI-8020" | ||
| reason = "The diagnosis and the fix preview print different verification steps for the same remedy." | ||
|
|
||
| # --- EAI-8009: with no `--model`, the local chat provider only looks for the | ||
| # built-in Lemonade assistant model, so a ready managed service running anything | ||
| # else is ignored and the user is told nothing is running. The scenario plants | ||
| # its own ready service, so this needs no hardware and holds on every lane. | ||
| # Remove when chatting locally uses whichever local server is ready. --- | ||
| [["chat-local-uses-the-running-server"]] | ||
| when = {} | ||
| bug = "EAI-8009" | ||
| reason = "The local chat provider only accepts the built-in assistant model, so a ready local server running any other model is not used." | ||
|
|
||
| # --- EAI-7423, not EAI-8007. This scenario's premise is a REAL serve on whatever | ||
| # engine the host defaults to, so on a lemonade LINUX host it inherits the same | ||
| # managed-serve bug as its siblings above: the serve reaches ready and is shut | ||
| # down ~0.08s later, so the endpoint never becomes reachable and the precondition | ||
| # — not the contract under test — is what fails. Measured on Strix Halo Ubuntu: | ||
| # passed in run 196, failed in run 210 with "endpoint did not serve model | ||
| # Qwen3-0.6B after 2 attempt(s) of 300s", which is that signature exactly. | ||
| # | ||
| # Scoped and flagged identically to the other EAI-7423 rows: os=linux because | ||
| # native Windows lemonade serve works (it passes on Strix Windows), gfx* because | ||
| # a real GPU host is needed at all, and `flaky = true` because which of these | ||
| # pass varies run to run — a deterministic-XPASS row would fail that lane on | ||
| # unrelated PRs. Delete this row with the other EAI-7423 ones, not with EAI-8007. | ||
| [["serve-services-stop-reports-what-it-stopped"]] | ||
| when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*" } | ||
| bug = "EAI-7423" | ||
| reason = "Lemonade managed serve reaches ready then shuts down immediately, so the service this scenario needs to stop never comes up." | ||
| serve_timeout_secs = 90 | ||
| flaky = true | ||
|
|
||
| # For the contract this scenario actually exists to hold: | ||
| # EAI-8007 has NO row on purpose. `services stop` reported stopping 0 processes | ||
| # for a service it did in fact stop on the pod, but neither fixture reproduces it | ||
| # here: a plain child process registered as a managed service was counted | ||
| # correctly in a one-off no-GPU check during development, and so was a real vLLM | ||
| # serve (MI300X, run 188). The shipped scenario is the real-serve one, which is | ||
| # `@requires-gpu @merge-queue`, so no blocking lane holds this contract. It ships | ||
| # as a guard rather than an expected failure, because a row would be a stale | ||
| # XPASS and would fail the lane. The ticket keeps the | ||
| # pod observation; what distinguishes that host is still unknown. | ||
|
|
||
| # --- EAI-8014: a PROPOSED contract, not a README one. The README documents that | ||
| # `uninstall` warns it does not stop managed background processes; this row pins | ||
| # the stronger behaviour the ticket asks for — `uninstall` removes the managed | ||
| # files and reports completion without stopping the servers it manages, and takes | ||
| # the service records with it, so the supported way to stop them goes too. The | ||
| # warning is documented behaviour, so this stays a row only until the product and | ||
| # README change together. No hardware needed. Remove when uninstall stops what it | ||
| # manages first, or when the contract is dropped. --- | ||
| [["serve-uninstall-stops-what-it-manages"]] | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Unconditional rows (
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. From the mock lane at 3012275 (
diagnose-50/51 need a real device, and serve-32 needs a merge-queue run, so the mock lane cannot show them. I will add the step-level evidence for them from a GPU-lane run.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This row pins a contract the README contradicts (README says uninstall warns it does not stop managed processes). Confirm a permanent xfail is intended until product and README change together.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, it's deliberate: the row stays an expected failure until the product and the README change together. It is reported as a proposed contract (EAI-8014), not a README promise, in both the scenario comment and the PR description. Whether to adopt that contract is for the maintainers to decide on EAI-8014. If you'd rather this PR not carry a proposal, I can drop serve-38. |
||
| when = {} | ||
| bug = "EAI-8014" | ||
| reason = "Uninstall reports completion while leaving the servers it manages running, and removes the records needed to stop them." | ||
|
|
||
| # --- EAI-8020. The other half of the same divergence: `diagnose` names | ||
| # the device's actual owning group while the `fix` recipe is hard-coded to | ||
| # `render`, so the two propose different changes wherever the device's group is | ||
| # not `render`. Needs a real device node, hence the GPU lane. Measured on the | ||
| # MI300X lane (E2E self-hosted ROCm/rocm-cli run 192) and the Strix Halo Ubuntu lane | ||
| # (run 196), failing for this reason on both, with 0 XPASS. The group is substituted | ||
| # by a `stat` shim, so the outcome does not depend on the host. --- | ||
| [["diagnose-and-fix-agree-on-the-remedy-command"]] | ||
| when = {} | ||
| bug = "EAI-8020" | ||
| reason = "The diagnosis proposes adding the user to the device's own group while the fix is hard-coded to `render`." | ||
|
|
||
| # --- EAI-8021. The render-group check only ever ADDS score for missing | ||
| # group membership and never credits access that demonstrably works, so a device | ||
| # the user can already read and write is still diagnosed as a permission problem. | ||
| # The scenario substitutes a world-writable device answer and asserts the | ||
|
fredespi marked this conversation as resolved.
Resolved
|
||
| # inspection agrees before relying on it. It shims BOTH `stat` (device | ||
| # `crw-rw-rw- root root`) and `id` (groups `users` — none of the conventional | ||
| # ones), so the runner's own group membership cannot reach the result: the | ||
| # outcome is the same on every bare-metal Linux GPU host. If this XPASSes, the | ||
| # product changed; it is not a host difference, and NOT a reason to add | ||
| # `flaky = true`. Measured on MI300X (run 192) and Strix Halo Ubuntu (run 196), | ||
| # failing for this reason on both, with 0 XPASS. --- | ||
| [["diagnose-credits-a-usable-device"]] | ||
| when = {} | ||
| bug = "EAI-8021" | ||
| reason = "A device the user can already read and write is still offered as a group-permission cause." | ||
|
|
||
| # --- EAI-8006: `--gpu auto` reports `selected none` on a machine that has a GPU, | ||
| # because the CLI cannot get an answer out of `amd-smi` — on the pod the bundled | ||
| # copy could not execute at all (its `#!/usr/bin/env python3` shebang found no | ||
| # interpreter), even though the CLI has a working interpreter recorded in its own | ||
| # config. | ||
| # | ||
| # os = "windows" is MEASURED, not guessed, and the three lanes separate the two | ||
| # candidate explanations cleanly: | ||
| # MI300X gfx943 linux PASSES (runs 190/191/192) | ||
| # Strix Halo Ubuntu gfx1151 linux PASSES (run 196) | ||
| # Strix Halo Windows gfx1151 windows FAILS (run 196) — `gpu: auto (selected none)` | ||
| # Same GPU family on both Strix lanes, so this is the OS, not the family — which | ||
| # is why the row is not scoped with therock_family the way | ||
| # `examine-both-forms-agree-on-gpu` is. Remove it when a Windows host names the | ||
| # device it picked. --- | ||
| [["serve-auto-gpu-selection-names-a-device"]] | ||
|
fredespi marked this conversation as resolved.
Resolved
|
||
| when = { os = "windows" } | ||
| bug = "EAI-8006" | ||
| reason = "Letting the CLI choose a GPU reports that it selected none on a Windows machine that has one." | ||
| serve_timeout_secs = 180 | ||
|
|
||
| # --- EAI-8016. The serve path has no address-availability check: the | ||
| # default port is used whether or not something already holds it, so a second | ||
| # server is handed the address the first is on. The scenario holds the address | ||
| # itself rather than starting a second real server. Measured on all three GPU | ||
| # lanes before the scenario became `@merge-queue`: MI300X (run 192), Strix Halo | ||
| # Ubuntu (run 196) and Strix Halo Windows (runs 196 and 210). It failed for this | ||
| # reason on each, Windows included, with 0 XPASS. --- | ||
| [["serve-second-server-gets-a-free-port"]] | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This row is when={} but its scenario (serve-32) is @merge-queue, so it is skipped on the MI300X lane the body credits it to, and only non-blocking Strix merge-queue lanes run it. Was it measured anywhere? An unmeasured unscoped row risks an XPASS on lanes where the bug does not reproduce.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fair: it is unmeasured in its current form. It was measured before it became @merge-queue: E2E self-hosted runs 192 (MI300X) and 196/210 (Strix Ubuntu and Windows) all failed it for the stated reason (the taken default address still assigned), with 0 XPASS. It went @merge-queue later to keep the Strix lane under its time cap, and since then only a merge_group run sets E2E_MERGE_QUEUE, so neither PR runs nor a workflow_dispatch can run it. The scenario holds the address itself rather than relying on a second real server, so it does not depend on which engine or GPU the lane has, and serve_cmd.rs on main has no port-availability change since. If it does go stale, the XPASS shows up on the non-blocking merge-queue lanes. The description (7768c4d) no longer credits it to the per-PR MI300X lane. |
||
| when = {} | ||
| bug = "EAI-8016" | ||
| reason = "A server started without an address is given the default one even when it is already in use." | ||
| serve_timeout_secs = 120 | ||
|
|
||
| # EAI-8017 and EAI-8018 have NO rows and NO scenarios, because neither can be | ||
| # exercised on any lane available here. Both need a working Lemonade install, | ||
| # and Lemonade's own launcher cannot start on either lane: on the MI300X runner | ||
| # `lemond` wants GLIBC_2.38 / GLIBCXX_3.4.32, which that OS does not have | ||
| # (measured, runs 190/191), and in a plain container it wants | ||
| # libdrm_amdgpu.so.1. Both die before reaching the behaviour under test, so a | ||
| # scenario would pass vacuously — worse than no scenario, because it would read | ||
| # as coverage. They were reproduced by hand on the pod (ubuntu 24.04) and stay | ||
| # on their tickets until a lane can run Lemonade. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This new guidance's own example cites the wrong file:
select_runtime_update_sourceis defined inapps/rocm/src/update_cmd.rs, notapps/rocm/src/main.rs(confirmed by grep — the symbol does not appear anywhere inmain.rs— and bydocs/architecture.md's own note on its Phase 6f relocation out ofmain.rs). Since this paragraph is specifically teaching "cite a symbol, never a line number" so citations don't go stale, an inaccurate example undercuts the point. Suggest repointing the link/citation atupdate_cmd.rs.Confidence 85 · mechanical