Repository navigation
Conversation
Contributor
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
#963 fixed Lemonade serving on Linux by resolving registered model names, but the ROCMAI-898 expected-failure rows stayed in expectations.toml. Every one of them is flaky, so their passes were tolerated as XPASS and never surfaced as stale rows. The lanes have been passing these scenarios since. In the main push and merge-queue E2E runs from #963's merge (2026-11-09) to 2026-11-13, the Strix Halo Ubuntu jobs passed every removed scenario that ran, MI300X passed serve-07 and serve-40 (as it also did on this PR's own MI300X job), and the W7900 jobs in that window passed them too. The dash-lemonade-managed-inference-journey row, added by #631, has passed in all seven Strix Halo Ubuntu jobs since that change merged. Remove twelve rows: the default-engine, readiness-contract, Anthropic inference/tool-use/streaming, explicit Lemonade inference (serve-07) and background-helper (serve-40) serving scenarios, the two chat scenarios, the bench real-serve scenario and the dash managed-inference journey. Keep serve-50 and serve-52 as flaky rows, re-attributed to ROCMAI-1023: their serve becomes ready, but `rocm services list` intermittently shows no `anthropic endpoint:` row within the 30s poll. They set no serve-timeout override, since the serve itself succeeds. Add a guard test that resolves each of the twelve scenarios from its real tag line on the modelled Linux Lemonade hosts (Strix Halo Ubuntu, a WSL2 guest with a usable GPU, gfx1100 bare metal) and, for the ids pinned to Lemonade, on MI300X, failing if any row matches; the pin is held per id and the failure message says how to proceed. Tests that used the deleted rows as fixtures now parse inline matrices. Update docs/testing.md and a model_serving.feature comment that still described the old rows. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf
force-pushed
the
ROCMAI-888-test-drop-rocmai-898-xfail-rows
branch
from
November 16, 2026 15:34
2bb0673 to
9ba169b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
tests/e2e-cucumber/expectations.tomlcarried ROCMAI-898 expected-failure rows for every GPU scenario that does a real Lemonade managed serve on a Linux host, because that serve often never exposed the model on the Strix Halo Ubuntu lanes. #963 fixed Lemonade serving on Linux by resolving registered model names, but the rows stayed. All of them wereflaky = true, so a pass was tolerated as XPASS and never reported as a stale row. While they stay, a real regression in any of these scenarios on a Linux Lemonade host is reported as an expected failure.The lanes have passed these scenarios since the fix. In the
mainpush and merge-queue E2E runs from 2026-11-09 to 2026-11-13, the Strix Halo Ubuntu jobs passed every removed scenario that ran, and MI300X passed serve-07 and serve-40, as it also did on this PR's MI300X job (131893614229). Thedash-lemonade-managed-inference-journeyrow, added by #631, passed in all 7 Strix Halo Ubuntu jobs since #631 merged.What changes
Twelve ROCMAI-898 rows are removed, with their 90s serve waits. These scenarios are now expected to pass on Linux Lemonade hosts:
@id:serve-lemonade-inference@id:serve-stop-leaves-no-background-helper@id:dash-lemonade-managed-inference-journey@id:serve-anthropic-messages-inference@id:serve-anthropic-messages-tool-use@id:serve-anthropic-messages-streaming@id:chat-tool-definitions-accepted@id:chat-end-to-end-local-model@id:bench-load-real-serve@id:serve-default-engine-working-endpoint@merge-queue)@id:serve-default-engine-inference@merge-queue)@id:serve-readiness-contract@merge-queue)The Strix Halo Ubuntu lane runs on
mainandrelease/**pushes, nightly and manual dispatch, not on pull requests or in the merge queue, so the effect of this change there shows only after merge or through a dispatch. MI300X runs on pull requests, with vLLM as the default engine, so of these scenarios it serves with Lemonade only the three pinned ones. serve-09, serve-10 and serve-12 are@merge-queue: the only lanes that run merge-queue scenarios are Windows and WSL2, and the WSL2 lane reports no usable GPU, so no Linux Lemonade lane runs those three. Their rows are removed because they described the same fixed serve, not because a lane confirmed them.Two rows stay, re-attributed to ROCMAI-1023: serve-50 (
@id:serve-anthropic-endpoint-advertised) and serve-52 (@id:serve-anthropic-endpoint-readiness). They fail intermittently on Strix Halo Ubuntu with a different symptom: the service is ready, butrocm services listshows noanthropic endpoint:row within the step's 30s poll (serve-50 in jobs 129254733795 and 129309536682; serve-52 in jobs 129309527901, 129318300169, 129700937355 and 130496332636). They stayflaky = trueand no longer set a serve-timeout override, since the serve itself succeeds.scripts/xfail_expectations_hint.pytherefore still reports live rows for ROCMAI-1023; that is intended.lemonade_linux_serve_scenarios_have_no_xfail_after_rocmai_898resolves each of the twelve scenarios from its real tag line on the modelled Linux Lemonade hosts (Strix Halo Ubuntu gfx1151, a WSL2 guest with a usable GPU, gfx1100 bare metal) and, for the Lemonade-pinned serve-07, serve-40 and dash-52, on MI300X, and fails if any row matches. It holds the Lemonade pin per id, so a scenario that loses its@requires-engine:lemonadetag fails it too, and its failure message says how to proceed.anthropic_endpoint_listing_rows_track_rocmai_1023pins the two kept rows. Tests that used the deleted rows as fixtures now parse inline matrices.docs/testing.mdand amodel_serving.featurecomment that described the old rows are updated.Verification
cargo test -p e2e-cucumber --lib: 366 passed.cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo xtask check-architecture-doc,cargo xtask check-module-sizeandpython3 scripts/xfail_expectations_hint.py --self-testpass. The prek hooks pass on the changed files, except the license-header hook (hawkeye), which was not installed locally.cargo xtask smokestopped atrocmd sandbox examine snapshotwithbwrap: loopback: Failed RTM_NEWADDR: Operation not permitted, a sandbox restriction of the host; this change does not touch that path.cargo test --workspace --all-targetswas not run locally; CI covers it.Scope
No CLI behaviour changes, so no new scenario: only the expectation matrix, its unit tests and the docs change. The affected scenarios and the lanes that run them are in the table above. The nightly workflow's other Linux GPU lanes also stop treating these scenarios as expected failures wherever they serve with Lemonade; the evidence above does not cover those lanes.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (This PR is that cleanup for ROCMAI-898; no ROCMAI-898 row is left.)docs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation). (N/A).rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry (cargo xtask check-architecture-docfails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (N/A: no file added, renamed or removed)