Repository navigation
Conversation
Contributor
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
tomastola
force-pushed
the
fix/queue-nightly-reporting
branch
from
November 17, 2026 18:46
decd187 to
62b9d56
Compare
request_chat_completion sent the chat to the first model /v1/models lists. Lemonade's router on its own port lists more than the scenario's model, so serve-lemonade-complex-registered-inference (serve-57) got a reply from Qwen3-0.6B-GGUF and failed "does not identify Qwen3.5-0.8B-GGUF" on every queue lane (queue run 44002836099) and on four nightly lanes (run 44029433287). The request now takes the scenario's model (world.model_name) and picks the /v1/models entry that is it: an equal id first, else the first one model_ids_match accepts. A server that does not list it fails the request with every model it did list. A scenario that names no model keeps the old first-entry behaviour. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
… queue run is skipped Publish E2E support matrix to wiki: also run when the consolidated report job failed (not cancelled). It fails when a lane ran and uploaded no report, which is a broken nightly and when the grid is most worth reading; run 44029433287 skipped the page for exactly that. The repository and ref guards and the no-grid guard in the publish step stay. Fetch the latest queue results: the job-log read no longer discards its error, the log read is retried, and every rejection of a candidate run prints a ::notice:: with the reason. Run 44002836099 met every filter the step could be checked against and was still not imported, and the log could not say which branch rejected it. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…uild Retry rocm install sdk and rocm update --apply, up to four attempts a minute apart, when the CLI reports "HTTP 5xx while fetching": the Strix WSL2 nightly of run 44029433287 died on a 503 from the nightly index and uploaded no report. Anything else, a 404 included, still fails on the first attempt. Treat vLLM's "no static build pin" refusal of an unpinned nightly SDK as not applicable instead of failing the whole nightly channel, as it has on every MI lane since the unpinned nightly moved to 10.3 (queue run 44002836099). The pre-warm records the reason in a marker file in the shared runtimes directory, and the suite skips the scenarios that serve with vLLM with that reason. A pinned SDK, the release channel and any other failure stay fatal. Temporary: remove it once vLLM wheels for the 10.3 SDK exist and the discover table has a 10.3 row. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Review fixes on this branch: - The wiki docs and a workflow comment still said the matrix publish is skipped when the report job fails; they now say what the job does. - The skip for an engine the pre-warm could not install is tested for the pinned-engine, bench and chat cases it covers, not only serve. - A test ties the two messages the pre-warm matches (vLLM's no-build-pin refusal and the CLI's "HTTP <status> while fetching") to the sources that print them, so rewording either fails a test instead of silently ending the retry or the skip. - Drop a contract-test clause that matched neither the old nor the new script. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The probe of the fixed fetch step showed why run 44002836099 was rejected
though it met every filter: gh api refuses a response that contains
terminal escape sequences ("pass --allow-escape-sequences"), which every
job log does, so all three log reads failed and the run was skipped as
unreadable. The old `2>/dev/null` hid the message.
curl follows the redirect to blob storage and sends the token only to
api.github.com. The contract test now pins the curl read.
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
clippy::too_long_first_doc_paragraph failed the clippy and Windows jobs on the e2e test target. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
tomastola
force-pushed
the
fix/queue-nightly-reporting
branch
from
November 17, 2026 19:14
ac542c5 to
d3c1659
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The queue nightly (queue run 44002836099) and the Nightly of the same day (44029433287) left no queue lane on either wiki page. Five separate causes, one fix each.
What each part fixes
1. serve-57 chats with the wrong model (
tests/e2e-cucumber/src/chat_request.rs,tests/e2e-cucumber/tests/e2e.rs).request_chat_completionsent the chat to the first model/v1/modelslists. The Lemonade router inserve-lemonade-complex-registered-inference(serve-57, nightly) lists more than the scenario's model, so the reply came fromQwen3-0.6B-GGUFand the step failed with "does not identifyQwen3.5-0.8B-GGUF" on all seven queue lanes and four Nightly lanes. The request now takes the scenario's model (world.model_name) and picks the/v1/modelsentry that is it: an equal id first, else the firstmodel_ids_matchaccepts. A server that does not list it fails with every model it did list. A scenario that names no model keeps the first-entry behaviour. Noexpectations.tomlrow is added: the feature file says serve-57 deliberately has none. Unit tests cover the pick; serve-57 itself only runs on the nightly GPU lanes (@requires-gpu @nightly), so that is where it is exercised.2. The wiki matrix was skipped after a failed report (
.github/workflows/nightly.yml).publish-e2e-wikirequired the report job to have succeeded. The report job fails when a lane ran and uploaded no report (here the Strix WSL2 nightly, part 3), which is the nightly where the grid is most worth reading. It now also runs onfailure, still not oncancelled, and keeps the repository, ref and platform-table guards. If the report job fails before writingconsolidated/, the job fails at the download instead of publishing.workflow_contractand the wiki docs are updated.3. A 503 from the nightly index killed a lane (
xtask/src/e2e_prewarm.rs).The Strix WSL2 nightly died on
HTTP 503 while fetching https://nightly.repo.amd.com/rocm/whl-next/rocm/in its pre-warm and uploaded no report.rocm install sdkandrocm update --applyare now retried up to four times, a minute apart, when the CLI reportsHTTP 5xx while fetching, matching theretry 4 …shapescripts/run-e2e-lane.shuses for its fetches. A 404 and every other failure still fail on the first attempt. A test ties the matched text to the source that prints it.4. A rejected queue run said nothing, and its log could never be read (
.github/workflows/nightly.yml, "Fetch the latest queue results").Run 44002836099 was completed, on
main, inside the 24-hour window, with an unexpired artifact and a job log carryingrocm-cli main at <sha>, yet the Nightly reported "none of the newest queue runs … tested main with a readable log and results". Everycontinuewas silent and the job-log fetch discarded its error. Every rejection now prints a::notice::naming the run and the reason, the log fetch keeps its error and is retried three times, and the final warning points at those notices.Cause, found with a temporary probe of the fixed step on this PR (run 44120676454, since removed): with the workflow token the log read failed on all three attempts with
the response contains terminal escape sequences; pass --allow-escape-sequences to output it anyway. The runner'sgh apirefuses a response holding escape sequences, and every job log has them, so the run was always "unreadable". The log is now read withcurl -fsSL(follows the redirect to blob storage, sends the token only to api.github.com), which does not depend on the gh version. The contract test pins the curl read. A Nightly was not dispatched from this branch becausepublish-nightlyis not ref-guarded and would publish a release from it; the probe ran the same step instead.5. vLLM's no-build-pin refusal failed every MI lane's nightly channel (
xtask/src/e2e_prewarm.rs,tests/e2e-cucumber/src/{capability,expectation}.rs,crates/e2e-report).The unpinned nightly SDK is now
10.3.0a…, vLLM has no 10.3 wheels, androcm engines install vllmrefuses with "has no static build pin". Pre-warm treated that as fatal, so mi300x, mi325x, mi350p, mi350x and mi355x never ran their nightly suite. Now, for vLLM on an unpinned nightly SDK only, the pre-warm records the reason in a marker file in the shared runtimes directory, and the suite reports the scenarios that start a vLLM server (idserve-,bench-orchat-, or a vLLM pin) as not applicable with that reason. A pinned SDK, the release channel, any other engine and any other failure stay fatal; GPU scenarios that start no server still run. The marker is cleared at the start of each pre-warm. This is temporary: the comments anddocs/ci-hardware-testing.mdname the condition for removing it (vLLM wheels for the 10.3 SDK published and a 10.3 row inengines/vllm/src/discover_builds.rs).Not in this change
Verification
cargo test -p e2e-cucumber --lib,cargo test -p xtask(workflow_contract,e2e_prewarm),cargo clippyonxtask,e2e-cucumber(incl.--test e2e),e2e-report,check-module-sizeandcheck-architecture-docpass locally.pick_modelto take the first entry turns its three tests red.ghstubbed, prints a reason for a run still going, an unreadable log and a log naming another branch, and exits 0.