Skip to content

ci: a missing queue run makes the release-readiness verdict NO-GO - #1107

Merged
tomastola merged 11 commits into
mainfrom
ci/queue-absent-is-no-go
Nov 17, 2026
Merged

tomastola merged 11 commits into
mainfrom
ci/queue-absent-is-no-go

Conversation

@tomastola

Copy link
Copy Markdown
Collaborator

What

A missing queue run makes the nightly release-readiness verdict NO-GO, with the failing gate queue lanes absent: <reason>. Until now it read GO plus a warning line.

Follow-up to #1063, from review: "absence should be its own gate". It is a separate PR so #1063 does not need another round. Based on ci/retire-selfhosted-nightly-lanes, so the diff here is only this change; after #1063 merges, retarget it to main.

Changes

  • xtask/src/release_readiness.rs: Facts::note_queue_absent adds the gate and sets the verdict; the warning under the table now says the verdict is NO-GO.
  • Test: absence turns a GO into NO-GO with that gate and shows it in the table; no absence changes nothing.
  • docs/nightly-on-the-queue.md: the "nothing to import" paragraph says NO-GO.

Things to know

  • Every night the queue run is late, missing, or from another branch now reads NO-GO. That is the intent: a queue that never ran is not a pass.
  • The nightly report itself still never fails on absence; only the verdict changes.

Verification

cargo test -p xtask (874 pass), cargo clippy -p xtask --all-targets -- -D warnings, cargo fmt --check. CI plumbing only, so no Gherkin scenario.

The R9700, MI350P and W7900 nightly jobs have sat queued for hours and
been cancelled since at least 11-09. Remove them from nightly.yml and
from the consolidated report that waited on them. These lanes now run
on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu,
Windows, WSL2, all on the Strix pool since #1018) are unchanged.

Update the workflow contract tests and the docs to match.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…same lines

Preparation for merging main into this branch without conflicts; the next
commits restore this change's own wording on top of the merge.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…port inputs

Reads a finished run of the shared GPU queue (its per-lane results artifact
and the run's job log) and writes each lane's report files where
`cargo xtask e2e-report` reads them, named for the lane and the queue's gate,
plus a status section for the top of the support matrix. Wired in by the
following commits.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…ry ones out of the verdict

The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X
lane is not the self-hosted one) and marks one the queue does not block on as
advisory. release-readiness leaves advisory rows out of its failure count and
still counts a blocking lane that came back with no result. xtask gains the
`e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly
platform check goes, since #1066 removed the per-PR W7900 lane it exempted.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them
from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X,
MI350X and MI355X, which only the queue runs, would never appear.
e2e-report-nightly now fetches the newest queue-run-results artifact of a main
run that finished in the last 24 hours (and its job log, for the tested branch
and each lane's gate), imports it, and opens the matrix with a Queue lanes
table. A missing, late or malformed queue run never fails the report: the page
says so instead.

The workflow contract pins the new steps and the native Linux preflight lane
counts for this change plus #1066 (four). The docs describe the queue import,
link the queue page instead of "(below; advisory)", and fix the "Of those"
wording in testing.md.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
… skip other-branch runs

Review of the queue import found two gaps against "no GPU row goes missing":

- When no queue run was imported, the Release-Readiness table still read
  "0 failure(s) across all platforms" and GO, and the absence was noted only
  on the E2E-Support-Matrix page. release-readiness now reads the matrix's
  absent note and marks the E2E row and adds a warning under the table. The
  verdict itself is unchanged: absence is still not a failure.
- The fetch step took the single newest queue-run-results artifact and then
  dropped the whole import when its log said another branch, so a later
  dispatch of another branch hid a valid run of main. It now walks the five
  newest runs of the last 24 hours and takes the first that tested main.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
- release_readiness: move queue_absent_reason above the total_e2e_failures
  doc block so each function keeps its own rustdoc.
- nightly: import only a completed queue run (any conclusion), and scan every
  job of the run for the lane matrix instead of assuming the first.
- docs: neutral pool wording, drop the leftover blank line.
- workflow_contract: pin the new fetch-step guards.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
- An unnamed matrix row is now unnamed-row-N and an unnamed artifact
  directory unnamed-dir-N, so plan() can no longer merge two unrelated
  lanes into one.
- Compile the three static regexes once (LazyLock), stop shadowing plan(),
  and share one make_dir helper in write_reports.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
A queue that never ran is not a pass. When the nightly report carries the
'no queue lanes' note, the verdict gets the failing gate
'queue lanes absent: <reason>' and reads NO-GO, instead of GO plus a warning.
Tests and the docs line follow.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from a team as a code owner November 16, 2026 18:08

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The change does what the PR describes: a missing queue run becomes its own failing gate (queue lanes absent: <reason>), the verdict goes to NO-GO, and the report itself still isn't failed. The new test asserts verdict, failing gates and table together, and it goes red if the verdict flip or the gate push is dropped. The narrative job strips .verdict, so it is unaffected. The docs paragraph matches the code.

Non-blocking, none of which need to hold this up:

  1. The doc comment on Facts::queue_absent in xtask/src/release_readiness.rs (around L75-78) still says the verdict does not count that absence as a failure. This PR reverses that, so it is worth rewording. It sits outside the diff hunks, so I can't anchor it inline.
  2. The E2E row in the table stays green while the verdict is NO-GO; the reason shows only in the evidence text, the warning and the Failing gates list. Fine, just a small readability point.
  3. The test resets Facts by hand (verdict = Go; failing_gates.clear()) for the None case; building a fresh Facts would be a cleaner proof.
  4. docs/ci-hardware-testing.md L279 ("never fails the report") is still true, but could add that the verdict is NO-GO to stay in sync.

Note that with the nightly queue absent, every night will now read NO-GO. That is the intended consequence. This is a read of the diff only; CI was still mostly pending when I looked, and the PR is stacked on #1063, so it needs retargeting after that merges.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

Base automatically changed from ci/retire-selfhosted-nightly-lanes to main November 17, 2026 00:24
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

# Conflicts:
#	docs/nightly-on-the-queue.md
#	xtask/src/release_readiness.rs
@tomastola
tomastola force-pushed the ci/queue-absent-is-no-go branch from 48a0d40 to 7cb88d7 Compare November 17, 2026 00:41
@tomastola
tomastola added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 9101286 Nov 17, 2026
40 checks passed
@tomastola
tomastola deleted the ci/queue-absent-is-no-go branch November 17, 2026 01:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants