Skip to content

ci: feed the queue nightly's lanes into the support matrix and wiki - #1085

Closed
tomastola wants to merge 4 commits into
mainfrom
ci/queue-nightly-in-support-matrix
Closed

tomastola wants to merge 4 commits into
mainfrom
ci/queue-nightly-in-support-matrix

Conversation

@tomastola

Copy link
Copy Markdown
Collaborator

Why

nightly.yml builds the consolidated support matrix from the e2e-gpu-*-report artifacts of its own lanes. publish-e2e-wiki pushes it to the E2E-Support-Matrix wiki page, and release-readiness-nightly / publish-release-readiness-wiki derive the Release-Readiness page and its GO/NO-GO from it.

The shared GPU queue nightly (queue-run.yml, 02:17 UTC) publishes its results as one queue-run-results artifact, which none of that reads. #1063 retires the self-hosted R9700, MI350P and W7900 nightly lanes in favour of the queue, so on its own it would drop those three GPUs from both wiki pages, and MI325X, MI350X and MI355X, which only the queue runs, would never appear.

#1063 should merge only after this PR.

What

e2e-report-nightly now also reads the queue's results:

  • Fetch (Fetch the latest queue results): the newest queue-run-results artifact of a run on this repository's main that finished, and started, in the last 24 hours, whatever the run's conclusion was (a red lane fails the run but its artifact is still uploaded). It is found by the artifact's name, because the workflow contract forbids any other workflow naming the queue workflows or runner. The same run's job log is fetched too: it is the only place the queue prints which branch it tested and which lanes it blocks on.
  • Import (cargo xtask e2e-queue-import, new): copies each lane's report.json, platform.json and commands.jsonl per channel to e2e-queue-<lane>[-advisory]-<channel>-report/, so e2e-report renders them with the other lanes. Nothing else of the artifact is copied (no lane.log, JUnit, HTML or harness diagnostics). The platform_slug is replaced by queue-<lane>-<channel>, because queue lanes of different models report the same slug (the MI325X lane probes as mi300x, the MI350X and MI355X lanes as mi350p), which would fold them, and the self-hosted MI300X lane, into one grid column.
  • Labels: a queue row reads MI325X (queue) Linux (release); a lane the queue marks advisory reads MI325X (queue, advisory) Linux (release).
  • Verdict: release-readiness leaves advisory queue rows out of its failure count. A blocking lane that came back with no result (or lacks a channel after not finishing cleanly) gets an empty report, shown as EMPTY, which already counts as a failure there.
  • Page: the matrix opens with a Queue lanes table (lane, GPU, gate, phase, exit status, channels reported).
  • Missing or late queue run: no run in the window, a run that tested another branch (the branch input of queue-run.yml can), a download or log that fails, or an import that fails never fails the report. The page opens with a note saying that no queue lanes are in it tonight and why. Both new steps are continue-on-error.

Public safety: the artifact is public and already scanned by the queue; only the three report files are used, and the only text taken from the job log (a public log) is the tested branch and commit and each lane's name, phase, exit status and gate, each checked against a strict pattern. A lane name that is not a plain lowercase word is shown as unnamed-N, never echoed.

Also: descriptor.rs in e2e-report names queue lanes and their gate; the docs (nightly-on-the-queue.md, a new section, and ci-hardware-testing.md) and architecture.md are updated; workflow_contract pins the new steps (continue-on-error, fetch filters, no credential beyond the job's read-only token, order, no dependency on the queue's workflows).

On the real 2026-11-13 run (7 lanes) the import produces 13 report directories and a page with the seven lanes; the blocking mi300x (failed suite, nightly channel left no report) and r9700s (no files came back) count towards NO-GO, the advisory lanes do not.

Things to know

  • Policy change: lanes the queue marks advisory (today everything except mi300x and r9700s, so including MI350P and W7900, which counted as failures when they ran self-hosted) no longer count towards the release-readiness verdict. That follows the queue's own gate, as asked; say if you would rather count them.
  • A queue run that is absent from the report does not make the verdict NO-GO; the Queue lanes note is the only sign, so read it before reading a GO.
  • The queue starts at 02:17 UTC and the nightly at 06:00 UTC (its report job runs after its GPU lanes), so a run that takes more than about three and a half hours is not in that night's report.
  • A run re-published by nightly-queue.yml is not imported.
  • The nightly channel of the lanes in the 11-13 run left no report on most lanes, and the W7900 nightly directory carries a manifest that says release, so it shows two (release) columns. That is what the lanes wrote, not something this changes.
  • The page grows: the sample matrix for the 13 queue reports is about 400 KB. Together with the lanes that remain it should stay under the 1 MiB step-summary limit, which would drop only the summary if exceeded.
  • This is CI plumbing with no change to the CLI's observable behaviour, so there is no Gherkin scenario.

Not verified

  • A queue-fed wiki publish can only run on main (publish-e2e-wiki and publish-release-readiness-wiki are gated on it), so the end-to-end path, including the real gh run download and job-log fetch from a hosted runner, has not run. What was exercised: the importer and e2e-report on the real artifact of the 11-13 run and its job log; the two shell steps run locally against a stand-in gh (newest-of-several selection, wrong branch / foreign repository / stale artifact skipped, stale start time, absent reasons, import failure fallback); the contract and unit tests. The first scheduled nightly after merge is the real test; the job's actions: read permission is already granted.
  • The job-log lines that e2e-queue-import reads (tested branch and commit, the lane matrix) are taken from one run's log; a change to the publisher's output format shows up as the "did not test main, or its job log could not be read" note, not as a wrong verdict.
  • cargo test -p xtask: 872 pass, 1 fails here, smoke::tests::run_command_leaves_no_child_running_after_its_deadline, because the sandbox has no kill binary; it is unrelated to this change. cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check are clean. cargo test --workspace --all-targets and cargo xtask smoke were not run.
  • The commits were created through the API and are not signed, so the commit-signatures check cannot pass until they are re-signed.

🤖 by agenthub on AMD AgentHub

…rt inputs

Reads a finished run of the shared GPU queue (its per-lane results artifact
and the run's job log) and writes each lane's report files where
`cargo xtask e2e-report` reads them, named for the lane and the queue's gate,
plus a status section for the top of the support matrix. Not wired in yet.

Signed-off-by: Tomas Saaristola <Tomas.Saaristola@amd.com>
…ry ones out of the verdict

The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X
lane is not the self-hosted one) and marks one the queue does not block on as
advisory. release-readiness leaves advisory rows out of its failure count and
still counts a blocking lane that came back with no result.

Signed-off-by: Tomas Saaristola <Tomas.Saaristola@amd.com>
…se-readiness verdict

e2e-report-nightly fetches the newest queue-run-results artifact of a main run
that finished in the last 24 hours (and that run's job log, for the tested
branch and each lane's gate), imports it with `cargo xtask e2e-queue-import`
and opens the support matrix with a Queue lanes table. A missing, late or
malformed queue run never fails the report: the page says so instead.

Retiring the self-hosted R9700, MI350P and W7900 nightly lanes no longer drops
them from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X,
MI350X and MI355X, which only the queue runs, are on them.

Signed-off-by: Tomas Saaristola <Tomas.Saaristola@amd.com>
…'s lanes

The import steps cannot fail the job, fetch by artifact name from a main run of
this repository inside the 24 hour window, hold no credential beyond the job's
read-only token, and run before the report is built.

Signed-off-by: Tomas Saaristola <Tomas.Saaristola@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Closing: this logic now lives in #1063, the PR that retires the self-hosted lanes, so no PR leaves the wiki missing rows.


🤖 by agenthub on AMD AgentHub

@tomastola tomastola closed this Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant