Skip to content

ci: retire the self-hosted nightly lanes that never get a runner; keep the queue's GPUs in the wiki - #1063

Merged
tomastola merged 9 commits into
mainfrom
ci/retire-selfhosted-nightly-lanes
Nov 17, 2026
Merged

tomastola merged 9 commits into
mainfrom
ci/retire-selfhosted-nightly-lanes

Conversation

@tomastola

@tomastola tomastola commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

What

Removes the self-hosted GPU nightly jobs that never get a runner from nightly.yml: R9700, MI350P and W7900. Since at least 11-09 every scheduled Nightly has left these queued for hours until cancelled. In the same PR, the nightly report now reads the queue's results, so the E2E-Support-Matrix and Release-Readiness wiki pages keep every GPU the queue tests: R9700, MI350P and W7900 stay on them, and MI325X, MI350X and MI355X, which only the queue runs, are added.

Kept as is: the MI300X nightly (and its per-PR lane), the Strix Halo Ubuntu, Windows and WSL2 nightlies (all on the Strix pool since #1018), and the hosted build and publish jobs.

Changes

Retiring the lanes

  • nightly.yml: delete e2e-gpu-nightly-rad3, -mi350p, -w7900; drop them from e2e-report-nightly (needs and LANE_REPORTS).
  • xtask workflow_contract: lane lists and the native Linux preflight lane counts (now four, with ci: remove the W7900 lane from e2e-selfhosted #1066's removal of the per-PR W7900 lane); the queue-doc sentences are pinned to the new wording.
  • Docs (ci-hardware-testing.md, nightly-on-the-queue.md, testing.md, tests/e2e-cucumber/README.md): say these lanes run on the queue, link nightly-on-the-queue.md instead of "(below; advisory)", and fix the "Of those" wording in testing.md.

Feeding the wiki from the queue

  • e2e-report-nightly fetches the newest queue-run-results artifact of a run on main that finished (status == completed), and started, in the last 24 hours (whatever its conclusion: a red lane fails the run, but its artifact is still uploaded), and that run's job log (every job of the run is scanned for the lane matrix, not only the first). It finds the artifact by name, because the workflow contract forbids any other workflow naming the queue workflows or runner. The job log is the only place the queue prints which branch it tested and which lanes it blocks on.
  • New cargo xtask e2e-queue-import copies each lane's report.json, platform.json and commands.jsonl per channel to e2e-queue-<lane>[-advisory]-<channel>-report/, so e2e-report renders them with the other lanes; nothing else of the artifact is copied. It replaces platform_slug with queue-<lane>-<channel>, because queue lanes of different models report the same slug (the MI325X lane probes as mi300x, the MI350X and MI355X lanes as mi350p), which would fold them, and the self-hosted MI300X lane, into one grid column.
  • Rows read MI325X (queue) Linux (release); a lane the queue marks advisory reads (queue, advisory). release-readiness leaves advisory queue rows out of its failure count; a blocking lane that came back with no result (or lacks a channel after not finishing cleanly) gets an empty report, shown as EMPTY, which already counts as a failure there.
  • The matrix opens with a Queue lanes table (lane, GPU, gate, phase, exit status, channels reported).
  • A missing or late queue run, a run that tested another branch (queue-run.yml has a branch input), a download or log that fails, or an import that fails never fails the nightly report: the page opens with a note saying that no queue lanes are in it tonight and why. Both new steps are continue-on-error.
  • Public safety: the artifact is public and already scanned by the queue; only the three report files are used, and the only text taken from the public job log (tested branch and commit, each lane's name, phase, exit status and gate) is checked against a strict pattern first. A lane name that is not a plain lowercase word is shown as unnamed-N, never echoed.
  • workflow_contract pins the new steps (continue-on-error, fetch filters, no credential beyond the job's read-only token, order, no dependency on the queue's workflows); the new xtask code has unit tests on the real 11-13 job-log format.

Lane coverage

Queue run of 11-13 (43351651832), 7 lanes. Only mi300x and r9700s are blocking; the rest are advisory.

Removed lane Covered by Gate 11-13 result
R9700 nightly queue r9700s blocking failed, no files came back
MI350P nightly queue mi350p advisory failed (exit 1), release channel reported
W7900 nightly queue w7900-dual advisory passed (Done, exit 0), both channels reported

Also on the queue and now in the matrix: mi300x (blocking, failed: release suite reported, nightly channel left no report), mi325x, mi350x, mi355x (advisory, failed, release reported). The 11-13 matrix built from this run has 13 queue report directories and a Queue lanes table with all seven lanes; the blocking mi300x and r9700s count towards NO-GO, the advisory lanes do not.

Things to know

  • Policy change: advisory lanes (today everything except mi300x and r9700s, so including MI350P and W7900, which counted when they ran self-hosted) no longer count towards the release-readiness verdict. That follows the queue's own gate.
  • A queue run that is absent from the report does not make the verdict NO-GO; the Queue lanes note is the only sign, so read it before reading a GO.
  • The queue starts at 02:17 UTC and the nightly at 06:00 UTC (its report job runs after its GPU lanes), so a run that takes more than about three and a half hours is not in that night's report. A run re-published by nightly-queue.yml is not imported.
  • The W7900 nightly directory of the 11-13 run carries a manifest that says release, so it shows two (release) columns; that is what the lane wrote.
  • The page grows: the sample matrix for the 13 queue reports is about 400 KB. It should stay under the 1 MiB step-summary limit, which would drop only the summary if exceeded.
  • CI plumbing only, no change to the CLI's observable behaviour, so there is no Gherkin scenario.

Relationship to #1018 and #1066

Rebased onto main after #1018 merged. #1066 (W7900 lane in e2e-selfhosted.yml) has merged; this branch has main merged in (not rebased), because the sandbox could not force-push. The preflight lane counts in workflow_contract.rs and the docs lines both PRs touched are resolved against #1066's wording. An earlier separate PR for the queue logic, #1085, is closed.

Not verified

  • A queue-fed wiki publish can only run on main (publish-e2e-wiki and publish-release-readiness-wiki are gated on it), so the end-to-end path, including the real gh run download and job-log fetch from a hosted runner, has not run. What was exercised: the importer and e2e-report on the real artifact and job log of the 11-13 run; the two new shell steps run locally against a stand-in gh (newest-of-several selection, wrong branch / foreign repository / stale artifact skipped, stale start time, absent reasons, import failure fallback); the unit and contract tests. The first scheduled nightly after merge is the real test; e2e-report-nightly already has actions: read.
  • The job-log lines e2e-queue-import reads (tested branch and commit, the lane matrix) come from one run's log; a change to the publisher's output format shows up as the "did not test main, or its job log could not be read" note, not as a wrong verdict.
  • cargo test -p xtask: 872 pass, 1 fails here, smoke::tests::run_command_leaves_no_child_running_after_its_deadline, because the sandbox has no kill binary; it is unrelated to this change. cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check are clean. cargo test --workspace --all-targets and cargo xtask smoke were not run.
  • commit-signatures is green on the head: the branch has been re-signed.

🤖 by agenthub on AMD AgentHub

CI note

The self-hosted E2E lanes (MI300X, Strix Halo) depend on runner capacity; a lane that waits for a runner is not caused by this change. Every hosted check on the head is green.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

The R9700, MI350P and W7900 nightly jobs have sat queued for hours and
been cancelled since at least 11-09. Remove them from nightly.yml and
from the consolidated report that waited on them. These lanes now run
on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu,
Windows, WSL2, all on the Strix pool since #1018) are unchanged.

Update the workflow contract tests and the docs to match.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…same lines

Preparation for merging main into this branch without conflicts; the next
commits restore this change's own wording on top of the merge.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…port inputs

Reads a finished run of the shared GPU queue (its per-lane results artifact
and the run's job log) and writes each lane's report files where
`cargo xtask e2e-report` reads them, named for the lane and the queue's gate,
plus a status section for the top of the support matrix. Wired in by the
following commits.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…ry ones out of the verdict

The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X
lane is not the self-hosted one) and marks one the queue does not block on as
advisory. release-readiness leaves advisory rows out of its failure count and
still counts a blocking lane that came back with no result. xtask gains the
`e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly
platform check goes, since #1066 removed the per-PR W7900 lane it exempted.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them
from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X,
MI350X and MI355X, which only the queue runs, would never appear.
e2e-report-nightly now fetches the newest queue-run-results artifact of a main
run that finished in the last 24 hours (and its job log, for the tested branch
and each lane's gate), imports it, and opens the matrix with a Queue lanes
table. A missing, late or malformed queue run never fails the report: the page
says so instead.

The workflow contract pins the new steps and the native Linux preflight lane
counts for this change plus #1066 (four). The docs describe the queue import,
link the queue page instead of "(below; advisory)", and fix the "Of those"
wording in testing.md.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola tomastola changed the title ci: retire the self-hosted nightly lanes that never get a runner ci: retire the self-hosted nightly lanes that never get a runner; keep the queue's GPUs in the wiki Nov 13, 2026
@tomastola
tomastola force-pushed the ci/retire-selfhosted-nightly-lanes branch from 68d2482 to 2e20fba Compare November 13, 2026 15:51
@tomastola
tomastola marked this pull request as ready for review November 16, 2026 16:33
@tomastola
tomastola requested a review from a team as a code owner November 16, 2026 16:33
@tomastola
tomastola requested a review from rominf November 16, 2026 16:33
… skip other-branch runs

Review of the queue import found two gaps against "no GPU row goes missing":

- When no queue run was imported, the Release-Readiness table still read
  "0 failure(s) across all platforms" and GO, and the absence was noted only
  on the E2E-Support-Matrix page. release-readiness now reads the matrix's
  absent note and marks the E2E row and adds a warning under the table. The
  verdict itself is unchanged: absence is still not a failure.
- The fetch step took the single newest queue-run-results artifact and then
  dropped the whole import when its log said another branch, so a later
  dispatch of another branch hid a valid run of main. It now walks the five
  newest runs of the last 24 hours and takes the first that tested main.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 16, 2026 16:41

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes for one small item; the rest are non-blocking.

Blocking

  • xtask/src/release_readiness.rs: queue_absent_reason was inserted into the doc block of total_e2e_failures with no blank line, so rustdoc merges the paragraphs onto the new fn and total_e2e_failures loses its doc. Move the new fn, with its own doc, above that block.

Non-blocking

  • The "finished run only" claim is not enforced: the fetch step never checks the run's status, so an in-progress run with an uploaded artifact and a partial job log can be imported (see inline).
  • .jobs[0].id assumes the first job of the queue run is the one that prints the lane matrix. If the run has several jobs, the matrix is missed and every lane reads as blocking, which is a false NO-GO risk. Selecting the job by name, or scanning all jobs, is safer.
  • The PR body is stale: it says commit-signatures cannot pass, but that check is green on this head, and the MI300X CI note is out of date.
  • Wording of the pool name in docs/ci-hardware-testing.md (see inline).
  • docs/ci-hardware-testing.md has a leftover double blank line where the removed paragraph was.

For the author to confirm (policy, already disclosed in the PR body)

  • Advisory lanes no longer count toward the readiness verdict, and an absent queue run gives GO plus a warning rather than NO-GO.
  • A blocking queue lane with no files now makes the verdict NO-GO.

Not verified

  • The real gh run download and job-log fetch on a hosted runner, which can only be proven after merge.
  • Whether other docs or comments still name the removed rad3/mi350p/w7900 nightly lanes.

Comment thread xtask/src/release_readiness.rs Outdated
/// Parsing the already-rendered rows, rather than re-deriving per-platform
/// pass/fail here, keeps this in lockstep with that crate's known-bugs/xfail
/// handling instead of duplicating it.
/// The reason the matrix gives for having no queue lanes, if it says so

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new queue_absent_reason was inserted into the doc block of total_e2e_failures (no blank line), so rustdoc merges all three paragraphs onto queue_absent_reason and total_e2e_failures ends up with no doc. Move this fn (with its own doc) above that block.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d6d85e6: queue_absent_reason now sits above the total_e2e_failures doc block with its own doc, so each function keeps its own rustdoc.

Comment thread .github/workflows/nightly.yml Outdated
for candidate in $candidates; do
rm -rf queue-results/results queue-results/job.log
# A run that started a day ago is not tonight's.
started=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${candidate}" --jq .run_started_at) || continue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs and PR text say only a finished run is imported, but nothing here checks the run's status; the artifact can exist while the run is still going (a partial job log then reads as unknown/blocking lanes). You already fetch the run object: also require .status == "completed" (not .conclusion, since a red run should still be imported).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d6d85e6: the fetch now requires status == completed (not the conclusion, so a red run is still imported). It also scans every job of the run for the lane matrix instead of taking .jobs[0]. Both are pinned in workflow_contract.

Comment thread docs/ci-hardware-testing.md Outdated
dispatch. Its first run, the 11-13 queue nightly
([run 43351651832](https://github.com/ROCm/rocm-cli/actions/runs/43351651832)), got capacity at once and finished `Done`, exit 0.
the merge queue; it keeps its `push`/dispatch triggers and also runs nightly
(`e2e-gpu-nightly-strix` on the same DevLab Dispatch pool labels). There is no

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"DevLab Dispatch" reads like an internal pool name for an upstream doc; if it is not already public wording in this repo, use neutral phrasing ("the shared Strix pool", as nightly-on-the-queue.md does).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"DevLab Dispatch" is already used as the pool name elsewhere in this doc, but the line this PR touched now says "the shared Strix pool" as you suggested.

- release_readiness: move queue_absent_reason above the total_e2e_failures
  doc block so each function keeps its own rustdoc.
- nightly: import only a completed queue run (any conclusion), and scan every
  job of the run for the lane matrix instead of assuming the first.
- docs: neutral pool wording, drop the leftover blank line.
- workflow_contract: pin the new fetch-step guards.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 16, 2026 17:07
@tomastola

Copy link
Copy Markdown
Collaborator Author

Re-review please, @juhovainio: the blocking item and the non-blocking ones are fixed in d6d85e6 (replies on each thread). The two policy points you asked me to confirm stand as written in the PR body: advisory lanes do not count towards the verdict, and an absent queue run gives GO plus a warning. The last item, other docs naming the removed lanes: I did not find any outside this PR's edits when grepping before the push.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-reviewed this at the current head. I found nothing that blocks the merge, but I'm requesting changes because two points need an author decision and a few things are still unproven.

Needs a decision

  • MI350P and W7900 counted toward the readiness verdict while they ran self-hosted. As advisory queue rows they no longer do. The PR body says so, but please confirm that's intended (inline comment).
  • If the queue run never happens, the nightly verdict can still read GO, and only a marker and a warning line say the queue lanes are missing. With the runner outages we've seen, I'd like absence to be its own gate or visibly change the verdict (inline comment).

Worth knowing / smaller items

  • The first scheduled nightly will probably be NO-GO. The 11-13 data in the PR has mi300x and r9700s failing as blocking lanes. That's not a defect, just a heads-up.
  • unnamed-N is numbered separately in parse_log and in the artifact scan, so an unnamed matrix row and an unnamed directory merge into one lane in the Queue lanes table (inline comment).
  • The 13-report matrix is about 400 KB against the 1 MiB step-summary limit, with no guard if more queue lanes are added.
  • The real gh run download, job-log fetch and wiki publish path was only exercised against a stand-in gh, and the PR text says cargo test --workspace and cargo xtask smoke were not run. I'd want those and CI green before merge.
  • Minor: let plan = plan(...) shadows the function, a Regex is rebuilt on each call in render_section and is_run_url, and write_reports repeats the same create_dir_all block.
  • The PR is large (importer, readiness policy change, docs). It's cohesive, so I wouldn't split it now.

I reviewed from the diff only, so I did not check for stale references to the removed lanes elsewhere in the repo, the queue-run.yml schedule claim, or the nightly-queue.yml republish exclusion. Please rely on CI for those.

Good things: the importer validates every log-derived field against strict patterns and copies only three files, the workflow contract test pins step order and the read-only token, and the docs are kept in sync.

.get(5)
.and_then(|c| c.trim().parse::<u64>().ok())
.unwrap_or(0);
if cells

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory queue rows are now excluded from the verdict. This also changes MI350P and W7900, which counted while they ran self-hosted. Disclosed in the PR body; please confirm that is intended.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, intended: the readiness verdict follows the queue's own gate, so a lane the queue lets fail (advisory, today everything except mi300x and r9700s) no longer fails the release verdict, including MI350P and W7900 which counted while they ran self-hosted. It is stated under "Things to know" in the PR body.

);
out.push('\n');

if let Some(reason) = &facts.queue_absent {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the queue lanes are missing the verdict can still be GO and only this warning says so. Given the known runner outages, consider whether absence should be its own gate rather than a note.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: a queue run that never happened is not a pass, so a missing queue run should read NO-GO with "queue lanes absent: ". To keep this PR as it is and mergeable, that follows in a separate small PR based on this branch (it changes only the verdict and its test and docs line).

Comment thread xtask/src/e2e_queue_import.rs Outdated
if !is_lane_name(lane) {
unnamed += 1;
out.lanes.push(MatrixRow {
lane: format!("unnamed-{unnamed}"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unnamed-N is numbered separately in parse_log and in scan(), and plan() merges by name, so an unnamed matrix row and an unnamed directory collapse into one lane in the Queue lanes table.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and fixed in 4684933: an unnamed matrix row is now unnamed-row-N and an unnamed artifact directory unnamed-dir-N, so plan() keeps them as separate lanes (the log row and the directory cannot be matched to each other anyway). The existing test now asserts both names.

- An unnamed matrix row is now unnamed-row-N and an unnamed artifact
  directory unnamed-dir-N, so plan() can no longer merge two unrelated
  lanes into one.
- Compile the three static regexes once (LazyLock), stop shadowing plan(),
  and share one make_dir helper in write_reports.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 16, 2026 17:34
@tomastola

Copy link
Copy Markdown
Collaborator Author

@juhovainio re-review please: 4684933 fixes the unnamed-N collision and the three minor items (let plan shadowing, regexes compiled once via LazyLock, one make_dir helper). cargo test -p xtask and cargo clippy --workspace --all-targets -- -D warnings pass. cargo test --workspace here has 3 failures in the rocm bin, none in code this PR touches (concurrent_active_runtime_marker_writes..., follow_service_logs_keeps_the_hint..., prune_removes_the_install_it_named_and_not_a_case_twin_of_it); I did not run them on main to confirm, so CI is the check. cargo xtask smoke was not run. The 1 MiB step-summary guard is not added: exceeding it drops only the summary, and the lane count is bounded by the queue's lanes. The absence-as-gate point is still open with the author.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found. Four non-blocking points, inline below.

Notes that did not fit an inline comment:

  • The real gh run download, the job-log fetch from a hosted runner and the wiki publish only run on main, so the first scheduled nightly after merge is the real test (the PR body says the same).
  • The step summary is estimated at ~400 KB against the 1 MiB limit; fine today, but it grows with lanes and channels.
  • The branch carries a merge commit and the PR body ends with an attribution footer; maintainers' call whether that matches the repo's conventions.
  • CI was still pending on several checks at review time (commit-signatures, lychee, prek, hawkeye); worth confirming those and the workflow_contract tests go green.

and each lane's name, phase, exit status and gate are taken from it, each
checked against a strict pattern first. A lane whose name is not a plain
lowercase word (letters, digits and dashes, and not ending in `-advisory`,
`-release` or `-nightly`) is shown as `unnamed-1` and so on, as a blocking

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code labels unusable lane names unnamed-row-N (matrix rows) and unnamed-dir-N (artifact directories), not unnamed-1. Worth updating this doc (and the PR body) to match.

let shaped = tokens.len() == 7;
let phase = shaped
.then(|| tokens[2])
.filter(|p| p.bytes().all(|b| b.is_ascii_alphabetic()))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

phase is limited to ASCII letters but not in length, and it ends up in a markdown table cell on a public wiki. Consider capping it (exit is already capped at 4 chars) so the strict-pattern claim holds.

if reported.is_empty() {
// Nothing came back from this lane, whatever it was told to run.
placeholders.push(None);
} else if have_matrix && !clean {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have_matrix && !clean treats an unparsed row (phase None, e.g. the publisher adds a column) the same as a failed lane. A lane that finished Done/0 and ran one channel then gets an EMPTY placeholder for the other, which release-readiness counts as a failure for a blocking lane. Consider adding placeholders only when the phase was parsed and is not Done/0.

}

#[test]
fn total_e2e_failures_leaves_out_the_queue_lanes_the_queue_lets_fail() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test feeds total_e2e_failures the matrix only. The nightly now prepends the ## Queue lanes section (with its own | Lane | GPU | ... | table) before the matrix; a case over that concatenation would pin that the scan can't latch onto the wrong table.

@tomastola
tomastola added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit eb90e41 Nov 17, 2026
41 checks passed
@tomastola
tomastola deleted the ci/retire-selfhosted-nightly-lanes branch November 17, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants