Skip to content

fix(e2e): get the queue nightly lanes into the reports (serve-57, wiki, pre-warm, fetch) - #1120

Draft
tomastola wants to merge 6 commits into
mainfrom
fix/queue-nightly-reporting
Draft

tomastola wants to merge 6 commits into
mainfrom
fix/queue-nightly-reporting

Conversation

@tomastola

@tomastola tomastola commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Why

The queue nightly (queue run 44002836099) and the Nightly of the same day (44029433287) left no queue lane on either wiki page. Five separate causes, one fix each.

What each part fixes

1. serve-57 chats with the wrong model (tests/e2e-cucumber/src/chat_request.rs, tests/e2e-cucumber/tests/e2e.rs).
request_chat_completion sent the chat to the first model /v1/models lists. The Lemonade router in serve-lemonade-complex-registered-inference (serve-57, nightly) lists more than the scenario's model, so the reply came from Qwen3-0.6B-GGUF and the step failed with "does not identify Qwen3.5-0.8B-GGUF" on all seven queue lanes and four Nightly lanes. The request now takes the scenario's model (world.model_name) and picks the /v1/models entry that is it: an equal id first, else the first model_ids_match accepts. A server that does not list it fails with every model it did list. A scenario that names no model keeps the first-entry behaviour. No expectations.toml row is added: the feature file says serve-57 deliberately has none. Unit tests cover the pick; serve-57 itself only runs on the nightly GPU lanes (@requires-gpu @nightly), so that is where it is exercised.

2. The wiki matrix was skipped after a failed report (.github/workflows/nightly.yml).
publish-e2e-wiki required the report job to have succeeded. The report job fails when a lane ran and uploaded no report (here the Strix WSL2 nightly, part 3), which is the nightly where the grid is most worth reading. It now also runs on failure, still not on cancelled, and keeps the repository, ref and platform-table guards. If the report job fails before writing consolidated/, the job fails at the download instead of publishing. workflow_contract and the wiki docs are updated.

3. A 503 from the nightly index killed a lane (xtask/src/e2e_prewarm.rs).
The Strix WSL2 nightly died on HTTP 503 while fetching https://nightly.repo.amd.com/rocm/whl-next/rocm/ in its pre-warm and uploaded no report. rocm install sdk and rocm update --apply are now retried up to four times, a minute apart, when the CLI reports HTTP 5xx while fetching, matching the retry 4 … shape scripts/run-e2e-lane.sh uses for its fetches. A 404 and every other failure still fail on the first attempt. A test ties the matched text to the source that prints it.

4. A rejected queue run said nothing, and its log could never be read (.github/workflows/nightly.yml, "Fetch the latest queue results").
Run 44002836099 was completed, on main, inside the 24-hour window, with an unexpired artifact and a job log carrying rocm-cli main at <sha>, yet the Nightly reported "none of the newest queue runs … tested main with a readable log and results". Every continue was silent and the job-log fetch discarded its error. Every rejection now prints a ::notice:: naming the run and the reason, the log fetch keeps its error and is retried three times, and the final warning points at those notices.
Cause, found with a temporary probe of the fixed step on this PR (run 44120676454, since removed): with the workflow token the log read failed on all three attempts with the response contains terminal escape sequences; pass --allow-escape-sequences to output it anyway. The runner's gh api refuses a response holding escape sequences, and every job log has them, so the run was always "unreadable". The log is now read with curl -fsSL (follows the redirect to blob storage, sends the token only to api.github.com), which does not depend on the gh version. The contract test pins the curl read. A Nightly was not dispatched from this branch because publish-nightly is not ref-guarded and would publish a release from it; the probe ran the same step instead.

5. vLLM's no-build-pin refusal failed every MI lane's nightly channel (xtask/src/e2e_prewarm.rs, tests/e2e-cucumber/src/{capability,expectation}.rs, crates/e2e-report).
The unpinned nightly SDK is now 10.3.0a…, vLLM has no 10.3 wheels, and rocm engines install vllm refuses with "has no static build pin". Pre-warm treated that as fatal, so mi300x, mi325x, mi350p, mi350x and mi355x never ran their nightly suite. Now, for vLLM on an unpinned nightly SDK only, the pre-warm records the reason in a marker file in the shared runtimes directory, and the suite reports the scenarios that start a vLLM server (id serve-, bench- or chat-, or a vLLM pin) as not applicable with that reason. A pinned SDK, the release channel, any other engine and any other failure stay fatal; GPU scenarios that start no server still run. The marker is cleared at the start of each pre-warm. This is temporary: the comments and docs/ci-hardware-testing.md name the condition for removing it (vLLM wheels for the 10.3 SDK published and a 10.3 row in engines/vllm/src/discover_builds.rs).

Not in this change

  • The GPU memory held on the mi355x node, and whether an absent queue run should read GO-with-warning or NO-GO. Both stay with their owners.
  • A 10.3 row for vLLM's discover table.

Verification

  • cargo test -p e2e-cucumber --lib, cargo test -p xtask (workflow_contract, e2e_prewarm), cargo clippy on xtask, e2e-cucumber (incl. --test e2e), e2e-report, check-module-size and check-architecture-doc pass locally.
  • Breaking pick_model to take the first entry turns its three tests red.
  • The fetch step's script, with gh stubbed, prints a reason for a run still going, an unreadable log and a log naming another branch, and exits 0.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@tomastola
tomastola force-pushed the fix/queue-nightly-reporting branch from decd187 to 62b9d56 Compare November 17, 2026 18:46
request_chat_completion sent the chat to the first model /v1/models lists.
Lemonade's router on its own port lists more than the scenario's model, so
serve-lemonade-complex-registered-inference (serve-57) got a reply from
Qwen3-0.6B-GGUF and failed "does not identify Qwen3.5-0.8B-GGUF" on every
queue lane (queue run 44002836099) and on four nightly lanes (run
44029433287).

The request now takes the scenario's model (world.model_name) and picks the
/v1/models entry that is it: an equal id first, else the first one
model_ids_match accepts. A server that does not list it fails the request
with every model it did list. A scenario that names no model keeps the old
first-entry behaviour.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
… queue run is skipped

Publish E2E support matrix to wiki: also run when the consolidated report job
failed (not cancelled). It fails when a lane ran and uploaded no report, which
is a broken nightly and when the grid is most worth reading; run 44029433287
skipped the page for exactly that. The repository and ref guards and the
no-grid guard in the publish step stay.

Fetch the latest queue results: the job-log read no longer discards its error,
the log read is retried, and every rejection of a candidate run prints a
::notice:: with the reason. Run 44002836099 met every filter the step could be
checked against and was still not imported, and the log could not say which
branch rejected it.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…uild

Retry rocm install sdk and rocm update --apply, up to four attempts a minute
apart, when the CLI reports "HTTP 5xx while fetching": the Strix WSL2 nightly
of run 44029433287 died on a 503 from the nightly index and uploaded no report.
Anything else, a 404 included, still fails on the first attempt.

Treat vLLM's "no static build pin" refusal of an unpinned nightly SDK as not
applicable instead of failing the whole nightly channel, as it has on every MI
lane since the unpinned nightly moved to 10.3 (queue run 44002836099). The
pre-warm records the reason in a marker file in the shared runtimes directory,
and the suite skips the scenarios that serve with vLLM with that reason. A
pinned SDK, the release channel and any other failure stay fatal. Temporary:
remove it once vLLM wheels for the 10.3 SDK exist and the discover table has a
10.3 row.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Review fixes on this branch:

- The wiki docs and a workflow comment still said the matrix publish is
  skipped when the report job fails; they now say what the job does.
- The skip for an engine the pre-warm could not install is tested for the
  pinned-engine, bench and chat cases it covers, not only serve.
- A test ties the two messages the pre-warm matches (vLLM's no-build-pin
  refusal and the CLI's "HTTP <status> while fetching") to the sources that
  print them, so rewording either fails a test instead of silently ending the
  retry or the skip.
- Drop a contract-test clause that matched neither the old nor the new script.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The probe of the fixed fetch step showed why run 44002836099 was rejected
though it met every filter: gh api refuses a response that contains
terminal escape sequences ("pass --allow-escape-sequences"), which every
job log does, so all three log reads failed and the run was skipped as
unreadable. The old `2>/dev/null` hid the message.

curl follows the redirect to blob storage and sends the token only to
api.github.com. The contract test now pins the curl read.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
clippy::too_long_first_doc_paragraph failed the clippy and Windows jobs on
the e2e test target.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola force-pushed the fix/queue-nightly-reporting branch from ac542c5 to d3c1659 Compare November 17, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant