Skip to content

test(e2e): drop the ROCMAI-898 xfail rows now that Lemonade serves on Linux (ROCMAI-888) - #1079

Draft
rominf wants to merge 1 commit into
mainfrom
ROCMAI-888-test-drop-rocmai-898-xfail-rows
Draft

rominf wants to merge 1 commit into
mainfrom
ROCMAI-888-test-drop-rocmai-898-xfail-rows

Conversation

@rominf

@rominf rominf commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Why

tests/e2e-cucumber/expectations.toml carried ROCMAI-898 expected-failure rows for every GPU scenario that does a real Lemonade managed serve on a Linux host, because that serve often never exposed the model on the Strix Halo Ubuntu lanes. #963 fixed Lemonade serving on Linux by resolving registered model names, but the rows stayed. All of them were flaky = true, so a pass was tolerated as XPASS and never reported as a stale row. While they stay, a real regression in any of these scenarios on a Linux Lemonade host is reported as an expected failure.

The lanes have passed these scenarios since the fix. In the main push and merge-queue E2E runs from 2026-11-09 to 2026-11-13, the Strix Halo Ubuntu jobs passed every removed scenario that ran, and MI300X passed serve-07 and serve-40, as it also did on this PR's MI300X job (131893614229). The dash-lemonade-managed-inference-journey row, added by #631, passed in all 7 Strix Halo Ubuntu jobs since #631 merged.

What changes

Twelve ROCMAI-898 rows are removed, with their 90s serve waits. These scenarios are now expected to pass on Linux Lemonade hosts:

Scenario Linux Lemonade lanes that run it
serve-07 @id:serve-lemonade-inference Strix Halo Ubuntu; MI300X (pinned to Lemonade)
serve-40 @id:serve-stop-leaves-no-background-helper Strix Halo Ubuntu; MI300X (pinned to Lemonade)
dash-52 @id:dash-lemonade-managed-inference-journey Strix Halo Ubuntu; resolves to expect-pass on MI300X (pinned to Lemonade)
serve-51 @id:serve-anthropic-messages-inference Strix Halo Ubuntu
serve-53 @id:serve-anthropic-messages-tool-use Strix Halo Ubuntu (the vLLM EAI-7223 row stays)
serve-54 @id:serve-anthropic-messages-streaming Strix Halo Ubuntu
chat-06 @id:chat-tool-definitions-accepted Strix Halo Ubuntu (the vLLM EAI-7223 row stays)
chat-07 @id:chat-end-to-end-local-model Strix Halo Ubuntu
bench-06 @id:bench-load-real-serve Strix Halo Ubuntu
serve-09 @id:serve-default-engine-working-endpoint none (@merge-queue)
serve-10 @id:serve-default-engine-inference none (@merge-queue)
serve-12 @id:serve-readiness-contract none (@merge-queue)

The Strix Halo Ubuntu lane runs on main and release/** pushes, nightly and manual dispatch, not on pull requests or in the merge queue, so the effect of this change there shows only after merge or through a dispatch. MI300X runs on pull requests, with vLLM as the default engine, so of these scenarios it serves with Lemonade only the three pinned ones. serve-09, serve-10 and serve-12 are @merge-queue: the only lanes that run merge-queue scenarios are Windows and WSL2, and the WSL2 lane reports no usable GPU, so no Linux Lemonade lane runs those three. Their rows are removed because they described the same fixed serve, not because a lane confirmed them.

Two rows stay, re-attributed to ROCMAI-1023: serve-50 (@id:serve-anthropic-endpoint-advertised) and serve-52 (@id:serve-anthropic-endpoint-readiness). They fail intermittently on Strix Halo Ubuntu with a different symptom: the service is ready, but rocm services list shows no anthropic endpoint: row within the step's 30s poll (serve-50 in jobs 129254733795 and 129309536682; serve-52 in jobs 129309527901, 129318300169, 129700937355 and 130496332636). They stay flaky = true and no longer set a serve-timeout override, since the serve itself succeeds. scripts/xfail_expectations_hint.py therefore still reports live rows for ROCMAI-1023; that is intended.

lemonade_linux_serve_scenarios_have_no_xfail_after_rocmai_898 resolves each of the twelve scenarios from its real tag line on the modelled Linux Lemonade hosts (Strix Halo Ubuntu gfx1151, a WSL2 guest with a usable GPU, gfx1100 bare metal) and, for the Lemonade-pinned serve-07, serve-40 and dash-52, on MI300X, and fails if any row matches. It holds the Lemonade pin per id, so a scenario that loses its @requires-engine:lemonade tag fails it too, and its failure message says how to proceed. anthropic_endpoint_listing_rows_track_rocmai_1023 pins the two kept rows. Tests that used the deleted rows as fixtures now parse inline matrices. docs/testing.md and a model_serving.feature comment that described the old rows are updated.

Verification

  • cargo test -p e2e-cucumber --lib: 366 passed.
  • The guard goes red when a removed row is re-added and when a scenario's Lemonade pin is dropped.
  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo xtask check-architecture-doc, cargo xtask check-module-size and python3 scripts/xfail_expectations_hint.py --self-test pass. The prek hooks pass on the changed files, except the license-header hook (hawkeye), which was not installed locally.
  • Not verified locally: cargo xtask smoke stopped at rocmd sandbox examine snapshot with bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted, a sandbox restriction of the host; this change does not touch that path. cargo test --workspace --all-targets was not run locally; CI covers it.
  • Not verified locally: the scenarios themselves, which need GPU hardware. The evidence that they pass is the lane history in Why.

Scope

No CLI behaviour changes, so no new scenario: only the expectation matrix, its unit tests and the docs change. The affected scenarios and the lanes that run them are in the table above. The nightly workflow's other Linux GPU lanes also stop treating these scenarios as expected failures wherever they serve with Lemonade; the evidence above does not cover those lanes.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (This PR is that cleanup for ROCMAI-898; no ROCMAI-898 row is left.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation). (N/A)
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (N/A: no file added, renamed or removed)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen. (N/A: no user-facing message changed)

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

#963 fixed Lemonade serving on Linux by resolving registered model
names, but the ROCMAI-898 expected-failure rows stayed in
expectations.toml. Every one of them is flaky, so their passes were
tolerated as XPASS and never surfaced as stale rows.

The lanes have been passing these scenarios since. In the main push and
merge-queue E2E runs from #963's merge (2026-11-09) to 2026-11-13, the
Strix Halo Ubuntu jobs passed every removed scenario that ran, MI300X
passed serve-07 and serve-40 (as it also did on this PR's own MI300X
job), and the W7900 jobs in that window passed them too. The
dash-lemonade-managed-inference-journey row, added by #631, has passed
in all seven Strix Halo Ubuntu jobs since that change merged.

Remove twelve rows: the default-engine, readiness-contract, Anthropic
inference/tool-use/streaming, explicit Lemonade inference (serve-07) and
background-helper (serve-40) serving scenarios, the two chat scenarios,
the bench real-serve scenario and the dash managed-inference journey.

Keep serve-50 and serve-52 as flaky rows, re-attributed to ROCMAI-1023:
their serve becomes ready, but `rocm services list` intermittently shows
no `anthropic endpoint:` row within the 30s poll. They set no
serve-timeout override, since the serve itself succeeds.

Add a guard test that resolves each of the twelve scenarios from its real
tag line on the modelled Linux Lemonade hosts (Strix Halo Ubuntu, a WSL2
guest with a usable GPU, gfx1100 bare metal) and, for the ids pinned to
Lemonade, on MI300X, failing if any row matches; the pin is held per id
and the failure message says how to proceed. Tests that used the deleted
rows as fixtures now parse inline matrices. Update docs/testing.md and
a model_serving.feature comment that still described the old rows.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-888-test-drop-rocmai-898-xfail-rows branch from 2bb0673 to 9ba169b Compare November 16, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant