Repository navigation
ci: a missing queue run makes the release-readiness verdict NO-GO - #1107
Conversation
The R9700, MI350P and W7900 nightly jobs have sat queued for hours and been cancelled since at least 11-09. Remove them from nightly.yml and from the consolidated report that waited on them. These lanes now run on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu, Windows, WSL2, all on the Strix pool since #1018) are unchanged. Update the workflow contract tests and the docs to match. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…same lines Preparation for merging main into this branch without conflicts; the next commits restore this change's own wording on top of the merge. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…port inputs Reads a finished run of the shared GPU queue (its per-lane results artifact and the run's job log) and writes each lane's report files where `cargo xtask e2e-report` reads them, named for the lane and the queue's gate, plus a status section for the top of the support matrix. Wired in by the following commits. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…ry ones out of the verdict The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X lane is not the self-hosted one) and marks one the queue does not block on as advisory. release-readiness leaves advisory rows out of its failure count and still counts a blocking lane that came back with no result. xtask gains the `e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly platform check goes, since #1066 removed the per-PR W7900 lane it exempted. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X, MI350X and MI355X, which only the queue runs, would never appear. e2e-report-nightly now fetches the newest queue-run-results artifact of a main run that finished in the last 24 hours (and its job log, for the tested branch and each lane's gate), imports it, and opens the matrix with a Queue lanes table. A missing, late or malformed queue run never fails the report: the page says so instead. The workflow contract pins the new steps and the native Linux preflight lane counts for this change plus #1066 (four). The docs describe the queue import, link the queue page instead of "(below; advisory)", and fix the "Of those" wording in testing.md. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
… skip other-branch runs Review of the queue import found two gaps against "no GPU row goes missing": - When no queue run was imported, the Release-Readiness table still read "0 failure(s) across all platforms" and GO, and the absence was noted only on the E2E-Support-Matrix page. release-readiness now reads the matrix's absent note and marks the E2E row and adds a warning under the table. The verdict itself is unchanged: absence is still not a failure. - The fetch step took the single newest queue-run-results artifact and then dropped the whole import when its log said another branch, so a later dispatch of another branch hid a valid run of main. It now walks the five newest runs of the last 24 hours and takes the first that tested main. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
- release_readiness: move queue_absent_reason above the total_e2e_failures doc block so each function keeps its own rustdoc. - nightly: import only a completed queue run (any conclusion), and scan every job of the run for the lane matrix instead of assuming the first. - docs: neutral pool wording, drop the leftover blank line. - workflow_contract: pin the new fetch-step guards. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
- An unnamed matrix row is now unnamed-row-N and an unnamed artifact directory unnamed-dir-N, so plan() can no longer merge two unrelated lanes into one. - Compile the three static regexes once (LazyLock), stop shadowing plan(), and share one make_dir helper in write_reports. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
A queue that never ran is not a pass. When the nightly report carries the 'no queue lanes' note, the verdict gets the failing gate 'queue lanes absent: <reason>' and reads NO-GO, instead of GO plus a warning. Tests and the docs line follow. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. The change does what the PR describes: a missing queue run becomes its own failing gate (queue lanes absent: <reason>), the verdict goes to NO-GO, and the report itself still isn't failed. The new test asserts verdict, failing gates and table together, and it goes red if the verdict flip or the gate push is dropped. The narrative job strips .verdict, so it is unaffected. The docs paragraph matches the code.
Non-blocking, none of which need to hold this up:
- The doc comment on
Facts::queue_absentinxtask/src/release_readiness.rs(around L75-78) still says the verdict does not count that absence as a failure. This PR reverses that, so it is worth rewording. It sits outside the diff hunks, so I can't anchor it inline. - The E2E row in the table stays green while the verdict is NO-GO; the reason shows only in the evidence text, the warning and the Failing gates list. Fine, just a small readability point.
- The test resets
Factsby hand (verdict = Go; failing_gates.clear()) for theNonecase; building a freshFactswould be a cleaner proof. docs/ci-hardware-testing.mdL279 ("never fails the report") is still true, but could add that the verdict is NO-GO to stay in sync.
Note that with the nightly queue absent, every night will now read NO-GO. That is the intended consequence. This is a read of the diff only; CI was still mostly pending when I looked, and the PR is stacked on #1063, so it needs retargeting after that merges.
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
Signed-off-by: Tomas Saaristola <tsaarist@amd.com> # Conflicts: # docs/nightly-on-the-queue.md # xtask/src/release_readiness.rs
48a0d40 to
7cb88d7
Compare
What
A missing queue run makes the nightly release-readiness verdict NO-GO, with the failing gate
queue lanes absent: <reason>. Until now it read GO plus a warning line.Follow-up to #1063, from review: "absence should be its own gate". It is a separate PR so #1063 does not need another round. Based on
ci/retire-selfhosted-nightly-lanes, so the diff here is only this change; after #1063 merges, retarget it tomain.Changes
xtask/src/release_readiness.rs:Facts::note_queue_absentadds the gate and sets the verdict; the warning under the table now says the verdict is NO-GO.docs/nightly-on-the-queue.md: the "nothing to import" paragraph says NO-GO.Things to know
Verification
cargo test -p xtask(874 pass),cargo clippy -p xtask --all-targets -- -D warnings,cargo fmt --check. CI plumbing only, so no Gherkin scenario.