Skip to content

ci: run pull requests on one Linux GPU node and one Windows GPU pool; merge queue drops its native Linux GPU lanes - #1018

Merged
tomastola merged 16 commits into
mainfrom
ci/pr-lanes-two-nodes
Nov 12, 2026
Merged

tomastola merged 16 commits into
mainfrom
ci/pr-lanes-two-nodes

Conversation

@tomastola

@tomastola tomastola commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What

A pull request now runs self-hosted GPU E2E on exactly one Linux node and one Windows node:

PR lane Node Why
E2E tests (MI300X, …) (e2e-gpu) Linux: MI300X (2 runners, mi300x-0/1) The only Linux node where vLLM is the effective engine; Lemonade/llama.cpp run there too.
E2E tests (Strix Halo, Windows, …) + E2E tests (Strix Halo, WSL2, …) Windows: Strix Halo DevLab Dispatch pool Fast (queue ≈1 min), 83% / 95% pass, 5 hung jobs in 2 weeks. WSL2 is a guest on the same Windows host class, so it stays.

Off the PR path and out of the merge queue (they still run on push to main and release/**, on dispatch, and nightly):

  • E2E tests (Strix Halo, Ubuntu, release|nightly, …) (e2e-gpu-strix-ubuntu)
  • E2E tests (W7900, …) (e2e-gpu-w7900)

Mechanically: those two jobs get github.event_name != 'pull_request' in their if:. e2e-report already tolerates skipped lanes (its missing-artifact check skips skipped needs).

Nightly gap found and fixed. e2e-gpu-nightly-strix targeted [self-hosted, linux, strix-halo, native], a static host that is offline. Every scheduled nightly (11-03 … 11-10, all cancelled) had that job queued until the 12 h cap, so Strix Linux had no scheduled coverage at all. It now uses the same devlab-dispatch pool labels as the per-PR lane and drops the static-host HOME/CARGO_HOME/TMPDIR overrides; workflow_contract.rs pins the labels to the per-PR lane like the Windows pair. Not exercised end to end: the nightly job only runs on schedule/dispatch; its body mirrors the per-PR job that runs on that pool today.

Measurements

e2e-selfhosted.yml, pull_request runs, 2026-10-27 → 2026-11-10, 998 runs (every job's final attempt). "Infra" = failed in Set up job / Reclaim GPU or no steps (runner lost/startup). "Test" = failed in the E2E step. "Timeout" = cancelled after running ≥40 min. "Never started" = cancelled still queued (superseded by a newer push). Strix Ubuntu counts two jobs per run (release + nightly legs).

Lane Jobs Pass Test fail Infra fail Timeout Never started Queue wait med / p90 (min) Run med (min)
Strix Halo Ubuntu 1144 438 (38%) 291 19 394 0 16 / 84 30
Strix Halo Windows 572 479 (83%) 86 2 5 0 1 / 13 17
Strix Halo WSL2 572 548 (95%) 24 0 0 0 1 / 13 9
MI300X 548 364 (66%) 106 2 0 76 590 / 1344 26
W7900 507 244 (48%) 61 0 0 202 665 / 1353 27

Last lane to finish (runs with ≥1 finished lane, n=668): W7900 557, MI300X 78, Strix Ubuntu 22, Strix Windows 11, WSL2 0. W7900 is the tail of the PR run most of the time, which is the "blocks PR development" symptom.

Runner capacity (repo runners list, 2026-11-10): Strix pool 6 Linux + 5 Windows ephemeral; mi300x 2; w7900 1; mi350p, r9700, static strix-halo-ubuntu-2 offline. Per-runner MI300X: mi300x-0 27% fail, mi300x-1 10% fail.

Why these nodes

  • Windows: the only Windows GPU pool; fastest queue and cleanest pass rate of any lane.
  • Linux: the three candidates are all poor, and coverage decides it. vLLM only runs on vLLM-eligible families (*-dcgpu, gfx906/908/90a), which among PR lanes is MI300X alone (docs/ci-hardware-testing.md; Strix gfx1151 is Lemonade-only). W7900 has one runner and 70% of its jobs never started. Strix Ubuntu has a pool, but 38% pass: 19% of its jobs hang to the 45 min cap (success p90 is 35 min, so it is a hang, not slowness) and it skips every vLLM scenario. Keeping MI300X keeps vLLM on PRs. Its weakness is queue wait (median ~10 h, p90 ~22 h on 2 runners), which this PR does not fix.

Coverage: suites leaving PR gating

All still run on MI300X (Linux) or Strix Halo Windows/WSL2 on PRs, so vLLM, Lemonade and llama.cpp (via Lemonade) keep a PR lane. What leaves PR gating:

  • Strix Halo (gfx1151) native Linux full suite, release + nightly channel legs: Lemonade/llama.cpp on Linux-on-Strix. Still PR-covered through MI300X (Lemonade runs anywhere) and WSL2 (a Linux guest on Strix). Nightly: e2e-gpu-nightly-strix (see the fix above).
  • W7900 (gfx1100) full suite: Lemonade on RDNA3. Nightly: e2e-gpu-nightly-w7900 (the 11-09 nightly ran it green).

Nothing else changes. Not touched: R9700 and MI350P (already nightly-only, ROCMAI-125; both runner sets are currently offline, so those nightly lanes also sit queued, outside this PR).

Required checks

No change needed: none of the self-hosted lane names is in the main required-checks list (E2E tests, E2E consolidated report are the GitHub-hosted mock lane / report in ci.yml). Verified against the branch-protection API today.

Overlap with open drafts

#999, #1009 and #1011 do not touch the same workflows: they edit scripts/run-e2e-lane.sh, queue-run.yml and docs/nightly-on-the-queue.md. All three, like this PR, edit xtask/src/workflow_contract.rs, in different tests, so expect only trivial textual conflicts there.

Verification

cargo test -p xtask workflow_contract passes (146 tests). Updated mi300x_lane_is_skipped_in_the_merge_queue (both lanes now skip pull_request and merge_group, and the whole if: of each is pinned; see rounds 3 and 4) and extended the pool-label pin. Data caveat: the Actions runs API stops at 1000 results, so the window is the latest 998 runs (14 days), not 21.

Review round 1 (juhovainio)

  • Gate assertion tightened (mi300x_lane_is_skipped_in_the_merge_queue): it now compares the entire if: of e2e-gpu-strix-ubuntu and e2e-gpu-w7900 to the expected text, so the pull_request exclusion must sit in the non-dispatch branch and nothing else may change. I broke the gate and saw it go red: (A) added && github.event_name != 'push' to the W7900 gate, which the old substring assertion would have passed; (B) moved the pull_request exclusion into the dispatch branch of the Strix Ubuntu gate. Both failed the test; restored, 146 pass.
  • Nightly Strix job: new test nightly_strix_ubuntu_job_holds_no_static_host_paths: no actions-runner path and no read of CARGO_HOME/RUSTUP_HOME/TMPDIR/PIP_CACHE_DIR the job does not set. Broke it by reading $TMPDIR in the run step: red. The only HOME reads left are $HOME/.cargo/bin (Rust bootstrap) and HF_HOME, both valid on a pool host.
  • Stale coverage claims: tests/e2e-cucumber/README.md (note under the job table), docs/testing.md (the AIM-cost paragraph), the channel-matrix paragraph and the Triggers paragraph in docs/ci-hardware-testing.md (reflowed), and a TMPDIR comment in scripts/reclaim-gpu.sh. The Platforms and README tables are pinned by a contract test, so I added prose under them rather than a column.

Review round 2 (juhovainio, 87b214e → 2f9c9c6)

  • e2e-report skips lanes whose needs result is skipped (checked on the self-hosted workflow; run 42932854129 succeeded with both lanes skipped).
  • Pool Docker check: a dispatch on the pool (run 43025895270) ran serve-26/27; Docker is reachable, serve-26 passed on both channel legs. serve-27 failed at "Local server stop not confirmed" on both legs (no expectation row; unrelated to the lane change, but the nightly lane will now show it).
  • Nightly path guard now also rejects any $HOME read other than $HOME/.cargo/bin; remaining stale per-PR wording fixed; two prose lines reflowed.

Review round 3 (029c48c)

  • W7900 dropped from merge_group (engineer's decision). It still runs on push to main and release/**, dispatch and nightly. The contract test now pins the whole gate of each lane (Strix Ubuntu: pull_request excluded; W7900: pull_request and merge_group excluded) and that the W7900 job has no E2E_MERGE_QUEUE. Broke the gate (removed the merge_group exclusion): red; restored.
  • Reworded the nightly.yml comment that called e2e-gpu-strix-ubuntu a per-PR lane, plus two comment-only leftovers (xtask/src/e2e_report.rs, tests/e2e-cucumber/src/expectation.rs).
  • The report crate assertion (the_nightly_lanes_publish_the_same_platforms_as_the_per_pr_lanes) is keyed on the artifact names found in each workflow's text, not on if: or events, so skipping lanes on pull_request does not affect it.
  • Nightly path guard matches whole words (HF_TMPDIR is not TMPDIR).

Strix Ubuntu vs MI300X, last 7 days (2026-11-04 14:22Z → 11-11 14:22Z, 944 runs of e2e-selfhosted.yml)

Queue wait = created → started, over jobs that got a runner. "No runner" = cancelled or expired without ever starting. Run time is over passed or failed jobs. Failure rate = failed / (passed + failed). Strix Ubuntu counts two jobs (release + nightly leg) per run.

Lane Event Jobs Queue wait med / p90 (min) Run time med / p90 (min) Pass Failed (test / infra) Failure rate Cancelled at the cap after running No runner
MI300X pull_request 317 720 / 1383 31 / 34 215 46 / 2 18% 0 54
MI300X push 123 705 / 1424 31 / 34 76 13 / 0 15% 0 34
MI300X merge_group 0 (lane skips it)
MI300X all 440 716 / 1389 31 / 34 291 59 / 2 17% 0 88 (20%)
Strix Ubuntu pull_request 682 8 / 69 30 / 34 493 178 / 0 27% 9 0
Strix Ubuntu merge_group 281 34 / 80 13 / 44 181 2 / 0 1% 98 (35%) 0
Strix Ubuntu push 266 40 / 97 30 / 34 203 61 / 0 23% 2 0
Strix Ubuntu all 1229 17 / 75 30 / 35 877 241 / 0 22% 109 (9%) 0
W7900 (for reference) all 519 1287 / 1422 34 / 48 159 43 / 0 21% 0 317 (61%)

Capacity (runner list, 2026-11-11): MI300X 2 runners (both busy); Strix Linux pool 5 registered, 4 online (3 busy), 1 offline (a fresh ephemeral runner per job: 1212 distinct runner names served the 1229 jobs this week); W7900 1 runner.

Could Strix Ubuntu replace MI300X as the per-PR Linux lane?

  • Faster: yes. A PR job starts in ~8 min (p90 ~69) against ~12 h (p90 ~23 h), with the same ~30 min run time, and none are lost without a runner (MI300X: 20%).
  • More reliable: not clearly. On jobs that ran, MI300X fails 17% against 22% on Strix Ubuntu (27% on PRs); the Strix failures are mostly red on main the same day (see Follow-up 1). The lane also hits its 45 min cap: 35% of merge-queue jobs were cancelled at the cap, 9% overall, which would turn a PR verdict into a timeout.
  • Coverage lost (gfx942 → gfx1151): the five vLLM scenarios (serve-vllm-inference, serve-vllm-default-on-instinct, serve-vllm-low-vram-oom-guidance, runtime-lifecycle-activate-restart-services-succeeds, therock-next-09-…vllm-rocm10x-discovery-pins) skip on Strix; serve-rocr-reindexed-gpu-index-rejected (needs more than one GPU) skips on Strix; the whole vLLM runtime path and Instinct (*-dcgpu, gfx942, 8-GPU) hardware paths. Gained: gfx1151 native-Linux APU paths. Lemonade/llama.cpp scenarios and the diagnose/driver-install/examine bare-metal ones run on both.

Review round 4 (5abecbc)

  • e2e-gpu-strix-ubuntu dropped from merge_group (engineer's decision; 35% of its merge-queue jobs, 98 of 281 over 7 days, hit the 45 min cap; non-required, the queue does not wait for it). It still runs on push (main, release/**), dispatch and nightly. The merge queue now runs only the Windows Strix Halo lanes (MI300X and W7900 were already out).
  • Contract test pins both lanes' whole if: gate and that neither job has E2E_MERGE_QUEUE; mutation-checked three ways (exclusion removed, E2E_MERGE_QUEUE restored, exclusion moved to the dispatch branch), all red, restored.
  • The merge queue now has no native Linux GPU lane (MI300X already skipped merge_group): it runs only the Windows Strix and WSL2 lanes. A release/** push still runs all five lanes.

Review round 5

  • Every surface that said Strix Ubuntu runs on merge_group now matches the workflow: this description (the "Off the PR path" line and the Verification section), docs/ci-hardware-testing.md (rows at L39 and L41, the Triggers paragraph), the nightly.yml comment on e2e-gpu-nightly-strix, and the cucumber README. Grepped for merge_group / merge queue across the workflows, docs, README and xtask comments.
  • The docs and this description now state it explicitly: the merge queue has no native Linux GPU lane, only Windows Strix and WSL2.
  • Pool Docker question: serve-26/27 were already exercised on the pool by run 43025895270 (see round 2); nothing new was dispatched.

Review round 6

  • Stale claims fixed: the merge_queue field doc in tests/e2e-cucumber/src/expectation.rs (only Windows Strix and WSL2 run @merge-queue serves), the model_serving.feature AIM comment (Strix Ubuntu no longer a per-PR job), docs/testing.md and the docs/ci-hardware-testing.md merge_group line (WSL2 named). Two badly wrapped lines re-wrapped.
  • Title now says the merge queue also loses its native Linux GPU lanes.
  • Mutation check of nightly_strix_ubuntu_job_holds_no_static_host_paths: (A) changed the job's reclaim step to bash /home/ubuntu/actions-runner/_work/scripts/reclaim-gpu.sh -> red ("the pool host has no /home/ubuntu/actions-runner tree"); (B) changed it to bash $HOME/work/scripts/reclaim-gpu.sh -> red ("reads $HOME outside the Rust bootstrap"). Both restored; cargo test -p xtask workflow_contract 146 pass.
  • Nightly dispatch: not dispatching nightly.yml from this branch, since it builds and publishes nightly release assets. Dispatch run 43025895270 of e2e-selfhosted.yml already ran the same lane steps on the pool: serve-26 passed on both channels (Docker reachable); serve-27 fails at stopping the AIM service (STOP FAILED rc=1), now an xfail row, rocm-cli#1058. The first scheduled nightly after merge is the end-to-end check; I can workflow_dispatch nightly on main after merge if wanted.
  • ci: cancel a PR's queued self-hosted E2E runs when it is merged or closed #1032 (cancel on close) edits the same workflow and is still open; I will rebase onto main once it merges.

Review round 7 (head bfde201, rebased onto main with #1032)

  • No Gherkin scenario: this is a CI-only change (workflow triggers, gates, docs and the workflow contract tests); no product behaviour changes, so there is no user-visible behaviour to specify. The behaviour that matters is pinned by workflow_contract tests (146 -> 153 with ci: cancel a PR's queued self-hosted E2E runs when it is merged or closed #1032's), which I mutation-checked.
  • Merge-queue coverage is intended (engineer's decision): the queue has no native Linux GPU lane, so heavy @merge-queue serves run only on Strix Windows and WSL2. MI300X stays the per-PR Linux lane and is not added to the queue (queue wait about 12 h against the queue's timeout). The only vLLM serve before a change lands is the per-PR canary (serve-05).
  • Expected state of the re-pooled nightly Strix lane: not proven end to end by a nightly.yml run (it publishes release assets, so it is not dispatched from a PR branch). The same lane steps ran on the pool in e2e-selfhosted.yml run 43025895270: serve-26 passed on both legs, serve-27 fails at stopping the AIM service. serve-27 is declared an expected failure in expectations.toml (Linux gfx1151, rocm-cli#1058), so the nightly reports xfail; the job is continue-on-error and cannot block the nightly.
  • Nightly path guard: textual by design; rejecting ${HOME forms is intentional (a pool host provisions no $HOME beyond the Rust bootstrap).
  • e2e-report tolerates skipped lanes: its "every lane that ran uploaded its report" check skips any needs result of skipped, and E2E self-hosted run 43049162834 (a pull_request run with Strix Ubuntu and W7900 skipped) finished with the report job green.
  • Rebase conflicts with ci: cancel a PR's queued self-hosted E2E runs when it is merged or closed #1032 were only the W7900/Strix matrix lines; ci: cancel a PR's queued self-hosted E2E runs when it is merged or closed #1032's closed-PR if table now pins both PR-skipping lanes with their exclusions.

Review round 8 (head b2f23df)

  • Evidence for the serve-27 failure is kept here and in rocm-cli#1058, not in durable comments: run 43025895270 (e2e-selfhosted.yml, platform=strix-ubuntu, include_nightly=true) failed serve-aim-lifecycle-parity on both channel legs at the stop step (STOP FAILED (rc=1) ... Local server stop not confirmed), while serve-aim-inference passed on both. nightly.yml, docs/testing.md, model_serving.feature and expectations.toml now only reference E2E: AIM service stop is not confirmed on Strix Halo Ubuntu (serve-aim-lifecycle-parity, STOP FAILED rc=1) #1058.
  • Wording: a PR runs two Windows-pool jobs (e2e-gpu-strix-windows and e2e-wsl) on separate runners, so the title, workflow header, cucumber README and docs say "one Windows GPU pool" (and "one Linux GPU node" for MI300X).

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@tomastola
tomastola marked this pull request as ready for review November 11, 2026 11:46
@tomastola
tomastola requested a review from a team as a code owner November 11, 2026 11:46
@tomastola
tomastola requested review from juhovainio and rominf and removed request for rominf November 11, 2026 11:46

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head e717523e

This automation never files an approving review — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (single-context pass; justification below).
No ticket named — the description cites internal tracking IDs (ROCMAI-125, EAI-8761, etc.) inline but does not claim to close any GitHub issue.

Blocking

  • docs/ci-hardware-testing.md's "Release-candidate regression gate" row is now stale — docs/ci-hardware-testing.md:41
    This PR redefines "per-PR lanes" (row above, line 39) to mean the narrowed 3-lane set (MI300X + Strix Windows + WSL), explicitly excluding e2e-gpu-strix-ubuntu and e2e-gpu-w7900. The untouched "Release-candidate regression gate" row still reads "The same per-PR lanes," which under the new definition implies the release-candidate gate also narrowed to 3 lanes. It did not: the if: conditions for both e2e-gpu-strix-ubuntu and e2e-gpu-w7900 only exclude github.event_name == 'pull_request', not push, so a push to a release/** branch still runs all 5 self-hosted lanes. The same "Triggers" paragraph this PR edited a few sentences later even says so explicitly: "pushing to a release branch runs the full self-hosted matrix." So the doc now contradicts itself within the same section, and a release manager skimming the table would wrongly conclude Strix Ubuntu/W7900 are no longer exercised before cutting a release tag.
    Confidence 90 · mechanical · Fix: reword the row (e.g. "The full self-hosted matrix (all platforms below, including the two narrowed out of the per-PR gate)") so it doesn't inherit the new, narrower meaning of "per-PR lanes."

Non-blocking

None

Decisions for the author

None

Positive signals

  • The lane-selection choice is backed by a precise, dated 998-run measurement table (pass/fail/timeout/queue-wait per lane) rather than a hunch — unusually rigorous grounding for a CI-scoping change.
  • xtask/src/workflow_contract.rs's updated mi300x_lane_is_skipped_in_the_merge_queue test pins the new per-PR/merge-queue contract as literal substring assertions on each job's if: gate, so a future accidental revert of the pull_request exclusion is caught by CI instead of discovered in production.
  • The offline native-host nightly-Strix bug (every scheduled nightly queued to the 12h cap since 11-03) is called out with its own evidence and fixed as a clearly-labeled drive-by, including an honest "not exercised end to end" caveat for its first scheduled run — not silently bundled in.

Deployment notes

  • After merge, PRs will no longer exercise e2e-gpu-strix-ubuntu or e2e-gpu-w7900; regressions specific to those two lanes surface only on push/merge_group or the nightly run (e2e-gpu-nightly-strix, e2e-gpu-nightly-w7900) — consistent with the author's own "Nightly watch" follow-up item.
  • The repointed e2e-gpu-nightly-strix job (now on the DevLab Dispatch pool) had not executed at review time; its first scheduled run is the real confirmation that the fix works, as the description itself notes.

What this covered

Read all four changed files in full: .github/workflows/e2e-selfhosted.yml (1839 lines), .github/workflows/nightly.yml (~2046 lines — the touched e2e-gpu-nightly-strix job and its per-PR/nightly siblings read closely; the unrelated release/publish jobs skimmed for cross-references), docs/ci-hardware-testing.md (full file), and xtask/src/workflow_contract.rs (the touched tests read in full; the rest of the ~9750-line file searched exhaustively for every reference to the changed job names/gate text rather than read line-by-line, since it is overwhelmingly unrelated pre-existing tests). Reviewed e717523e02a154e8807c0f2c3909367a55e83b07.

Single-context pass, not fanned out: the change is small (4 files, one tightly-coupled workflow pair plus the contract test and doc that describe them) and fits well within what one reviewer can hold completely.

Did not run: cargo test -p xtask workflow_contract was not executed directly (no local checkout in this environment); the new/changed assertions were instead traced by hand against the committed YAML, and the PR's own CI (get_check_runs/get_status, all green, including "E2E consolidated report (self-hosted)") was taken as corroboration that the Rust suite passed. No separate git log/git blame pass was run; the inline comments in both workflow files were read as part of the main pass, which is where the one blocking finding came from.

Deviation to flag: the PR description and title were fetched alongside the diff in the same initial call, so this review was not conducted fully blind to the stated intent as the method prescribes. Findings were nonetheless reached by independently tracing the committed YAML/Rust rather than taken on faith from the description, and the description was deliberately re-checked at the end against the settled findings — it accurately reflects the diff, no scope mismatch found. Not reconciled against any discussion on this PR (one existing comment was left unread, per the method's rule not to read discussion).

Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Fixed in be3088b.

  • Blocking (docs:41): the release-candidate row now reads "The full self-hosted matrix: every platform in the table below, including e2e-gpu-strix-ubuntu and e2e-gpu-w7900, which only skip pull_request". Confirmed against the if: gates: both skip only pull_request, so a release/** push still runs all five lanes.
  • Same class, found while fixing it: the paragraph further down (old lines 108-123) still said the Ubuntu nightly lane pins the native host and that moving it to the pool was "a separate, not-yet-made decision". This PR made that move, so I rewrote it to say all three Strix nightly lanes share the pool labels, pinned by nightly_devlab_dispatch_lanes_share_the_per_pr_pool_labels.
  • Non-blocking: none listed. Docs-only change, no workflow or test edits.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the diff and found nothing that breaks the intent (one Linux and one Windows GPU node per PR), but there are a few things I'd like addressed before merge. Two are inline below.

  • The repointed e2e-gpu-nightly-strix job loses its CARGO_HOME/RUSTUP_HOME/TMPDIR/PIP_CACHE_DIR overrides. Please confirm no later step in that job still reads them, since the job hasn't run since the change.
  • The new contract assertion only pins the string github.event_name != 'pull_request' and the absence of merge_group. A gate that drops push/merge_group some other way would still pass. Please tighten it and say in the PR text that you broke the gate and saw the test go red.
  • Please grep README, docs/testing.md and the Platforms table for per-PR coverage claims about native-Linux Strix and W7900, which this change makes stale. The "Triggers" paragraph in docs/ci-hardware-testing.md is also now one very long line; worth reflowing.

Note the tradeoff: PRs no longer get native-Linux Strix (gfx1151) or W7900 coverage, only push/merge_group/dispatch and nightly do. The PR body discloses that, so I'm treating it as an accepted decision. CI shows no failures so far; the build/test and E2E lanes were still pending when I looked.

Comment thread .github/workflows/nightly.yml Resolved
Comment thread xtask/src/workflow_contract.rs Resolved

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. I traced the changed if: gates (the added github.event_name != 'pull_request' sits in the non-dispatch arm, so push, merge_group, dispatch and release/** still run both lanes), the e2e-report handling of skipped needs, the repointed nightly Strix labels against the per-PR lane, the removed env overrides, and the contract-test substring assertions against the YAML; they agree. The earlier stale release-candidate row is fixed on this head. This was a static review: nothing was run (no Rust toolchain here), and the self-hosted lanes were still in progress when I read CI; the GitHub-hosted checks are green.

Non-blocking, all stale wording that this change leaves behind in docs/ci-hardware-testing.md and nearby:

  • Lines ~88-93: "so the per-PR smoke gate proves the nightly channel boots on at least one platform" is no longer true, since the only lane with the channel matrix no longer runs on pull_request; after this change no PR run exercises the nightly channel. Worth saying so explicitly.
  • Line ~301 still says "all three per-PR Strix lanes (Ubuntu, Windows, WSL2)", and line ~203 says e2e-report covers "the per-PR GPU platforms"; also ~401 ("long-lived native-labeled hosts") and .github/workflows/e2e-selfhosted.yml ~485 ("the only per-PR job with two independent matrix axes").
  • .github/workflows/nightly.yml ~765-770: the comment "no nightly lane runs AIM serve and a pool-based one is needed" is stale now that this job is the pool-based one; the open question is Docker on the pool hosts, which the first scheduled run will answer.
  • Some numbers in the description don't match its own table: W7900 "70% of its jobs never started" is 202 of 507 (about 40%), Strix Ubuntu "19% hang to the cap" is 394 of 1144 (about 34%), and the mi300x-0/mi300x-1 failure counts (129 of 432, 8 of 88) don't add up to the table's 106 test failures plus 2 infra failures. The YAML comment "58-70%" has the same issue. Reconciling or stating the denominators would keep the rationale checkable.
  • Untouched, but now wrong: scripts/reclaim-gpu.sh (~49, ~1394) says the nightly Strix lane redirects TMPDIR to $HOME/actions-runner/tmp, and docs/testing.md (~1245) still calls Ubuntu a per-PR job. The new two-lanes-skip-pull_request assertion lives inside mi300x_lane_is_skipped_in_the_merge_queue, so a failure there will point at the wrong rule; a dedicated test that the three kept lanes still run on pull_request would pin the other half.

🤖 by agenthub on AMD AgentHub

@tomastola

Copy link
Copy Markdown
Collaborator Author

Addressed in 87b214e (signed, with sign-off). All three review points hold and are fixed; both inline threads are replied to and resolved:

  • Nightly Strix job: no remaining reference to the dropped overrides, now pinned by a new test.
  • Gate assertion: the whole if: of both lanes is compared; break-the-gate results are in the PR description.
  • Stale claims: cucumber README, docs/testing.md, the channel-matrix and Triggers paragraphs (reflowed) and a reclaim-gpu.sh comment are updated.

Not rebased onto #1032; whichever merges second needs it.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 12:40

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-reviewed the current head (87b214e). Nothing here is a hard blocker for the change itself, but a few things need an answer or a fix before this goes in, so I'm requesting changes.

To resolve before merge

  • The nightly Strix lane now runs on the pool hosts, and its body hasn't been exercised end to end. Pool hosts are cold; if Docker isn't reachable there, serve-26/27 (AIM, nightly-only) skip silently and the job stays green, so the only place those canaries run stops covering them. A workflow_dispatch run on the pool, or a note in the PR on the Docker question, would settle it (inline comment on the runs-on line).
  • The PR body says e2e-report already tolerates skipped lanes, but that file isn't in the diff. The passing "E2E consolidated report" check is the ci.yml mock-lane report, not the self-hosted one. Please confirm from the self-hosted report path.
  • Please grep the repo for other repeats of the "per-PR Strix Ubuntu / W7900" claim (README.md, docs/nightly-on-the-queue.md, workflow comments) and update any that are now stale. I only saw the diff.
  • Two prose lines were left overlong after the reflow: docs/ci-hardware-testing.md line 320 (inline) and docs/testing.md around line 1245.

Worth confirming (tradeoffs, not blocking)

  • Strix Ubuntu and W7900 still run on push and merge_group. W7900 has a single runner and a very long queue, so the merge queue can still sit on the capacity problem this PR removes from the PR path. They are non-required, so I assume that's intended.
  • No PR lane now exercises the nightly package channel, and with the nightly pools down, effective coverage of it is zero for now. The docs say so; just flagging it.
  • nightly_strix_ubuntu_job_holds_no_static_host_paths only checks the string actions-runner and four env names, so the "no static host paths" wording overstates it. A $HOME/... read other than .cargo/bin would slip through.

What looked good: the gate test compares the whole if: rather than a substring, the nightly lane is pinned to the pool labels by the contract test, and the static-host env is gone. I did not run the contract tests locally; this is a diff read plus CI status.

Comment thread docs/ci-hardware-testing.md Resolved
Comment thread .github/workflows/nightly.yml Resolved

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 87b214e88c3f

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (7 files: 2 workflow YAML, 1 Rust contract-test file, 1 shell script, 3 docs files).
No single ticket named as closed by this PR; the description cites several historical ticket IDs (ROCMAI-125/429/430, EAI-8761/7548/8057) as context for pre-existing design choices this PR builds on, not as something this PR itself closes.

Blocking

None

Non-blocking

  • Self-contradictory "per-PR Strix lanes" claims left in docs/ci-hardware-testing.md — docs/ci-hardware-testing.md:207, :210, :305-306
    Line 305-306 reads "Strix Halo is gfx1151 on every OS, so all three per-PR Strix lanes (Ubuntu, Windows, WSL2) ... are lemonade-only" — directly contradicting this same PR's own edits a few dozen lines earlier (lines 47-53, 88-90) stating e2e-gpu-strix-ubuntu no longer runs on pull_request. Line 207/210 similarly still calls the e2e-report job's 5-lane set "the per-PR GPU platforms," though two of those five (e2e-gpu-strix-ubuntu, e2e-gpu-w7900) are no longer per-PR per this PR's own change. Same root cause (an incomplete "per-PR" sweep in this file), two sites.
    Confidence 85 · logic · Fix: say "two" and drop Ubuntu at 305-306; relabel the 207/210 set as "the self-hosted GPU platforms."
  • docs/testing.md's AIM-serve budget rationale anchors on a job that no longer runs per PR — docs/testing.md:1242-1251
    The paragraph justifies why serve-26/serve-27 stay @nightly by citing e2e-gpu-strix-ubuntu's wall-clock budget "when it still ran per PR" — but that job is now off the PR path, so the paragraph's only concrete evidence no longer describes a currently-at-risk lane. e2e-gpu-strix-windows carries the identical 45-minute cap and still runs per PR today, but goes unmentioned.
    Confidence 70 · logic · Fix: anchor the rationale on e2e-gpu-strix-windows instead (or in addition).
  • Stale TMPDIR-redirect comment in scripts/reclaim-gpu.sh's main() — scripts/reclaim-gpu.sh:1394-1399
    This comment still states in the present tense that "the nightly Strix lane redirects TMPDIR to the runner's home" and "the guard was dead on the one lane that redirects." This PR removes the only TMPDIR override in either workflow file (the nightly Strix lane's env block), so no lane redirects TMPDIR anymore. A near-identical paragraph earlier in the same file (lines 46-54) was correctly updated to past tense; this second occurrence was missed. The script logic itself (scenario_tmp_root()) is unaffected and remains correct.
    Confidence 85 · mechanical · Fix: update to past tense, matching the lines 46-54 edit.
  • Stale "per-PR" comment in xtask/src/e2e_report.rs (file not touched by this PR) — xtask/src/e2e_report.rs, doc-comment above without_channel_matrix_segment
    "...nightly's six GPU lanes and, per ROCMAI-125, e2e-gpu-strix-ubuntu's per-PR [release, nightly] leg..." — this lane's channel matrix is unchanged, but it is no longer a per-PR leg after this PR's gate change. Outside the diff, so missed by the in-PR doc sweep.
    Confidence 85 · mechanical · Fix: drop "per-PR" or note the lane now runs on push/merge_group/dispatch/nightly.

Decisions for the author

  • Keeping MI300X as the sole per-PR Linux GPU lane — tradeoff
    It is the only per-PR lane with vLLM coverage, but the job's own comment already documents its capacity constraint ("limited MI300X runner capacity, the queue entry's job can stay queued long after its PR merged"). Dropping Strix-Ubuntu/W7900 removes their queue problems from the PR path but leaves MI300X's as the one that remains. Worth confirming this is accepted rather than merely the least-bad option by elimination.
  • No per-PR lane now exercises the nightly package channel — tradeoff
    The PR's own docs/ci-hardware-testing.md edit states this plainly: e2e-gpu-strix-ubuntu was the only lane with a [release, nightly] channel matrix, and it's now off the PR path entirely. This is disclosed, not hidden, but worth the author confirming it's an acceptable gap until the next nightly run.

Positive signals

  • The new contract test pins the entire if: gate string for e2e-gpu-strix-ubuntu/e2e-gpu-w7900 (not a substring), which the PR's own commit history shows was deliberately hardened after a manual break-test caught a substring check passing on a wrong gate.
  • nightly_strix_ubuntu_job_holds_no_static_host_paths is a well-targeted regression guard for exactly the failure mode that silently broke the nightly Strix lane before this PR (an unset variable defaulting to empty rather than failing loudly).

Deployment notes

Two previously-per-PR self-hosted lanes (e2e-gpu-strix-ubuntu, e2e-gpu-w7900) stop running on pull_request but keep running on push/merge_group/workflow_dispatch/nightly. Neither lane's check name is in the repository's required-status-check list (per the workflow's own header comment), so this does not change what blocks a merge — but anyone relying on their per-PR signal (e.g. watching their job status on a PR) needs to know to look at push/nightly runs instead.

What this covered

Read completely: .github/workflows/e2e-selfhosted.yml, .github/workflows/nightly.yml, xtask/src/workflow_contract.rs, scripts/reclaim-gpu.sh, docs/ci-hardware-testing.md, docs/testing.md, tests/e2e-cucumber/README.md — via two independent fan-out workers (grouped by coupling: workflow+contract-test, and docs), plus my own direct reads of the key job blocks (e2e-gpu, e2e-gpu-strix-ubuntu, e2e-gpu-w7900, e2e-gpu-nightly-strix, e2e-report), the contract test's helper functions (job_block, job_mapping, the gate closures), and a repo-wide search for other references to the two affected job names. Commits reviewed: 431c48c3c752048285095e57e8309bcc38ebb25d…87b214e88c3fd05055bac169755ad9ff5fbd906e (merge-base to the assigned head, all 4 commits on the branch at that point).

Did not run: cargo test -p xtask workflow_contract (no build toolchain in this review environment) — verified the new/changed assertions by direct comparison against the YAML instead. CI at 87b214e88c3fd05055bac169755ad9ff5fbd906e: all required checks (CI/build-and-test, CodeQL, Security scan) are green. The self-hosted "E2E self-hosted" pull_request run for this exact commit shows cancelled — superseded by a later push to the branch, which is the workflow's own documented concurrency behavior, not a failure, and none of these self-hosted lanes are required checks.

Not reconciled against any review thread or prior comment on this PR — this was a blind pass, and a stale earlier review from this account exists at a superseded commit that was deliberately not read. Fan-out was not degraded: both workers completed independently and no retry was needed.

No issue-tracker tool is available to this review; the description names several historical ticket IDs as background for pre-existing design choices, not as a ticket this PR closes, so there was nothing to retrieve.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Round 2 addressed in 2f9c9c6 (signed, with sign-off). Both inline threads are replied to and resolved. The other points:

  • e2e-report and skipped lanes: checked on the self-hosted workflow itself, not the ci.yml mock report. On e717523e the E2E consolidated report (self-hosted) job of e2e-selfhosted.yml run 42932854129 concluded success with E2E tests (W7900, …) and E2E tests (Strix Halo, Ubuntu, …) both skipped. Its missing-artifact check also ignores any lane whose needs result is skipped (the script is in e2e-selfhosted.yml, outside this diff).
  • Other per-PR claims: grepped README, docs/*.md and both workflows. Fixed docs/ci-hardware-testing.md ("all three per-PR Strix lanes"), the "only per-PR job with two matrix axes" comment in e2e-selfhosted.yml and the W7900 comment in nightly.yml. The remaining "per-PR" mentions (R9700/MI350P history, MI300X, WSL/Windows, the mock lane) are still true.
  • Overlong lines: reflowed in docs/ci-hardware-testing.md and docs/testing.md.
  • Test wording: the guard now also rejects any $HOME read other than $HOME/.cargo/bin, and its doc comment says it is a textual guard, not proof the job runs. I checked it goes red when the job reads $HOME/e2e-target.
  • Strix/W7900 on merge_group: intended. They are non-required and the queue does not wait for them, and no PR lane depends on W7900 capacity now. If you want them out of the queue too, that is a one-line change per lane but not something I'll do unasked; it is the engineer's call.
  • Nightly package channel: not zero. The MI300X and W7900 nightly lanes both run a channel: [release, nightly] matrix (both legs ran on 11-09); the pool-based Strix nightly lane adds a third once it runs. It is the PR path that has none, as the docs say.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 13:32

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff at 2f9c9c6. I found nothing blocking: the two if: gates, the nightly runs-on switch, the contract-test pins and the README/docs updates agree with each other and with the stated intent, and the diff scan is clean. All checks on this head pass. Self-hosted GPU lanes are not required checks.

Non-blocking points, with inline comments on the first three:

  1. The demoted Strix Ubuntu and W7900 lanes keep merge_group and push, so they become some of the heaviest lanes in the merge queue and on main.
  2. The nightly Strix job's move to the pool is only text-guarded and is continue-on-error, so a first-run failure on a cold host would be silent. A one-off dispatch would confirm it.
  3. The nightly.yml comment says the report crate asserts that nightly lanes publish the same platforms as the per-PR lanes. Worth confirming that assertion is not keyed on a fixed list now that those lanes skip pull_request.
  4. The new guard test is substring-based and only strips full-line comments. It is fine as a tripwire, but it cannot prove the job runs on the pool.

Comment thread .github/workflows/e2e-selfhosted.yml Resolved
Comment thread .github/workflows/nightly.yml Resolved
/// textual guard over the job's non-comment lines, not proof the job runs.
#[test]
fn nightly_strix_ubuntu_job_holds_no_static_host_paths() {
let nightly = read_workflow("nightly.yml");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guard is textual and substring-based (a future HF_TMPDIR would trip the TMPDIR check, and trailing # ... comments are not stripped). Fine as a tripwire, but it cannot show the job actually runs on the pool and the job is continue-on-error. Please run a one-off nightly dispatch to confirm it end to end.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whole-word matching is in 029c48c (HF_TMPDIR no longer reads as TMPDIR). Trailing # ... comments inside a run script are still not stripped, so the guard is a tripwire only, and its doc comment says so.

On the end-to-end run: I did run the same lane on the pool. e2e-selfhosted.yml run 43025895270 (platform=strix-ubuntu, include_nightly=true, serve-2[67]) covered the pool labels, the Rust bootstrap, the GPU preflight, the build and the prewarm; it showed Docker is reachable (serve-26 passed on both channel legs). I have not dispatched nightly.yml itself: it also builds and publishes nightly release assets and cleans up a staging release, and a dispatch from a PR branch is not something I will risk on the shared release tag. The first scheduled run is the real end-to-end confirmation. If you want one before merge, the safe route is a dispatch of nightly.yml from main after this merges, or I can add an inputs-gated lane filter to nightly in a follow-up. Leaving this open for you.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per the earlier discussion I have not dispatched anything new for this. The AIM canaries were already exercised on the pool by e2e-selfhosted.yml run 43025895270 (platform=strix-ubuntu, include_nightly=true, serve-2[67], both channel legs): Docker is reachable and serve-26 passed on both legs, so they do not skip silently there. serve-27 failed at "Local server stop not confirmed"; that is reported in the PR description. A dispatch of nightly.yml itself is not run from this branch because it also builds and publishes release assets; the first scheduled run remains the end-to-end confirmation. Leaving this thread for you to close.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 2f9c9c68

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — DEGRADED (single-context pass) — needs work

Full review of the whole change (7 files: 2 workflow YAML, 1 Rust contract-test file, 1 shell script, 3 docs files), diffed against the merge-base with main.
No ticket named as closed by this PR; the description cites several historical ticket IDs (ROCMAI-125/429/430, EAI-8761/7548/8057) as background for pre-existing design choices this PR builds on, not as something this PR itself closes.

Blocking

  • A newly-added comment in this PR calls the very job this PR removes from pull_request a "per-PR" lane — .github/workflows/nightly.yml:771-772
    This PR's own diff replaces the old e2e-gpu-nightly-strix comment with: "The DevLab Dispatch pool, identical to the per-PR e2e-gpu-strix-ubuntu lane in e2e-selfhosted.yml..." But this same PR's if: change makes e2e-gpu-strix-ubuntu explicitly not run on pull_request anymore (&& github.event_name != 'pull_request', confirmed in the committed YAML). The comment is new text this PR introduces, and it directly contradicts the PR's own headline change in the same file set — it reintroduces exactly the "per-PR" mislabeling that three review rounds on this PR already caught and fixed in four other spots (docs/ci-hardware-testing.md, docs/testing.md, .github/workflows/e2e-selfhosted.yml, tests/e2e-cucumber/README.md).
    Confidence 88 · mechanical · Fix: drop "per-PR" (e.g. "identical to the e2e-gpu-strix-ubuntu lane, which runs on push/merge_group/dispatch but no longer on pull_request").

  • PR description does not mention the two most recent commits' changes — PR description (no single file)
    The description's only changelog section is "Review round 1 (juhovainio)", describing commit 87b214e8. But the current head is a later commit, 2f9c9c68 ("ci: tighten the nightly Strix path guard; fix remaining per-PR Strix/W7900 claims"), pushed in response to a second CHANGES_REQUESTED review from the same reviewer at 87b214e8 (asking to tighten the $HOME guard further, fix remaining stale "per-PR" claims, and reflow two overlong lines). That commit does real, reviewable work — it strengthens nightly_strix_ubuntu_job_holds_no_static_host_paths with a new $HOME-usage scan, fixes two more stale "per-PR" doc sentences, and reflows two lines — none of which is described anywhere in the PR body. A reviewer reading only the description would not know this round of changes exists.
    Confidence 85 · architectural · Fix: add a "Review round 2" section (or fold it into the existing verification notes) describing what 87b214e8→2f9c9c68 changed and why.

Non-blocking

  • Stale "per-PR" wording left in two files outside this PR's own doc sweep — xtask/src/e2e_report.rs:145, tests/e2e-cucumber/src/expectation.rs:1199
    e2e_report.rs:145's doc-comment still reads "...per ROCMAI-125, e2e-gpu-strix-ubuntu's per-PR [release, nightly] leg...", and expectation.rs:1199's test comment still reads "...may each wait out a 1200s readiness cap. Run per-PR on the Strix Ubuntu pool..." — both now false the same way the four already-fixed spots were. Neither file is touched by this PR's diff, and neither affects test behavior (the e2e_report.rs test still compares artifact names correctly regardless of per-PR status), so these are comment-only drift, same root cause as the blocking nightly.yml finding above, just outside the files this PR happened to touch.
    Confidence 80 · mechanical · Fix: same wording fix, next time either file is touched, or in this PR's doc sweep since it already found four siblings of this exact phrase.

Decisions for the author

  • Keeping MI300X as the sole per-PR Linux GPU lane — tradeoff
    It's the only per-PR lane with vLLM coverage, but the PR's own measurement table shows it has the worst queue wait of any lane kept on the PR path (median ~10h, p90 ~22h on 2 runners) and a double-digit per-runner failure rate. Dropping Strix-Ubuntu/W7900 removes their queue problems from the PR path but leaves MI300X's queue problem as the one that remains and is now the sole Linux signal. Worth confirming this is the accepted remaining risk rather than an overlooked one.
  • e2e-gpu-nightly-strix now shares the DevLab Dispatch pool with the per-PR Windows Strix/WSL2 lanes — tradeoff
    Previously isolated on its own (offline) static host, it now draws from the same ephemeral pool (6 Linux + 5 Windows runners per the PR's own capacity numbers) that e2e-gpu-strix-windows and e2e-wsl use on every PR. A single daily scheduled run is unlikely to cause real contention, but a workflow_dispatch re-run during business hours would compete with live PR lanes for pool capacity in a way the old static host never did. Worth a one-line acknowledgment if considered and accepted.

Positive signals

  • The lane-selection choice is backed by a precise, dated 998-run measurement table (pass/fail/timeout/queue-wait per lane) rather than a hunch — unusually rigorous grounding for a CI-scoping change.
  • The gate-pinning test (mi300x_lane_is_skipped_in_the_merge_queue) was iteratively hardened in direct response to review feedback and the author demonstrated it actually catches a regression (documented break-test: moving the pull_request exclusion into the wrong boolean branch, or adding an extra push exclusion, both turned the test red before being reverted).
  • The PR's claim that "e2e-report already tolerates skipped lanes" checks out: I read the untouched e2e-report job in e2e-selfhosted.yml directly — its "Check every lane that ran uploaded its report" step explicitly does [ "$result" = skipped ] && continue before checking for a missing artifact, so the two newly-skipped-on-pull_request lanes will not produce a false "missing artifact" error.

Deployment notes

Two previously-per-PR self-hosted lanes (e2e-gpu-strix-ubuntu, e2e-gpu-w7900) stop running on pull_request but keep running on push/merge_group/workflow_dispatch/nightly. Neither lane's check name is in the repository's required-status-check list, so this does not change what blocks a merge — but anyone relying on their per-PR signal on a PR page needs to know to look at push/nightly runs instead. The repointed e2e-gpu-nightly-strix job has not executed on its new pool hardware at review time (continue-on-error, so a cold-start failure stays non-blocking); its next scheduled run is the real confirmation, as the PR itself discloses.

What this covered

DEGRADED — fan-out could not run. Three attempts to spawn independent review workers (grouped by coupling: the two workflow YAMLs together, the Rust contract-test file alone, and the docs/script group) all failed immediately with "Concurrent subagent limit reached... Do not retry" (confirmed via the agent-listing tool: 20–21 unrelated subagents already occupying the platform-wide slot limit at the time). Per instruction, retries were not attempted. This is therefore a single-context pass, done as thoroughly as that allows — recommend re-running with fan-out available for independent cross-check.

To compensate, this pass went beyond a typical single-context read: full diff at the merge-base (1530f6b1…2f9c9c68) for all 7 changed files; full git show reads of both complete workflow YAML job blocks (changed jobs and their runner-pool siblings); hand-traced boolean evaluation of every changed if: gate against pull_request/push/merge_group/workflow_dispatch; read and hand-traced every new/changed Rust test function (mi300x_lane_is_skipped_in_the_merge_queue, nightly_strix_ubuntu_job_holds_no_static_host_paths, nightly_devlab_dispatch_lanes_share_the_per_pr_pool_labels) together with the helper functions they call (job_block, job_mapping, normalized_whitespace, the gate closure) against the literal YAML; read the untouched e2e-report job in full to verify the PR's own skip-tolerance claim; a repo-wide grep at the PR head for every remaining reference to the two narrowed-out job names, with full-paragraph context read around each hit; CI status (get_check_runs/get_status) — all required checks (build-and-test, clippy, CodeQL, security scans, commit-signatures) green at this exact head, which exercises cargo test -p xtask workflow_contract; and the current PR description cross-checked against the full commit log on the branch.

Did not run: cargo test -p xtask workflow_contract directly in this environment (no local toolchain invoked) — relied on hand-tracing plus the green required CI as corroboration. No separate git log/git blame history pass beyond reconciling the PR body against the branch's commit log for Step 9. No prior-review-comments pass: this PR's earlier review threads were necessarily visible during the mandatory pre-review duplicate-check (confirming no review already existed at this exact head), but no finding, wording, or conclusion from them was carried into this review — everything reported above was independently re-derived from the code and confirmed by direct reading. Not reconciled against any discussion thread on this PR.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 14:29

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: one blocking doc/comment-accuracy issue, otherwise the change looks sound.

Blocking

  • Round 4 dropped e2e-gpu-strix-ubuntu from merge_group, but several claims still say it runs there: the comment in nightly.yml (~L772), the two rows in docs/ci-hardware-testing.md (L39, L41), and the PR body's "still run on push / merge_group" line plus its Verification section. Please grep the claim across all surfaces and bring them in line with the workflow.

Non-blocking

  • The merge queue now has no native Linux GPU lane (only Windows Strix and WSL2) since E2E_MERGE_QUEUE was removed from the Strix Ubuntu job. Worth stating explicitly in the docs and PR body.
  • The nightly Strix job now runs on the pool, where Docker may be unreachable, so the AIM serve canaries (serve-26/27) could skip silently. It has not been exercised end to end; a dispatch run before merge would confirm it fires.
  • A few comment lines are left unwrapped.

CI: the failures seen at the earlier head were apt DNS errors in "Install native build deps" (runner network, not this change). Checks need a rerun at 5abecbc.

Comment thread .github/workflows/nightly.yml Resolved
Comment thread docs/ci-hardware-testing.md Resolved
Comment thread docs/ci-hardware-testing.md Resolved
@tomastola

Copy link
Copy Markdown
Collaborator Author

Addressed in 3c2bb61 (signed, with sign-off):

  • Blocking: every merge_group claim now matches the workflow: the nightly.yml comment, the two docs/ci-hardware-testing.md rows (L39, L41), the PR description (the "Off the PR path" line and the Verification section), plus the Triggers paragraph and the cucumber README. Three threads replied to and resolved.
  • Non-blocking: the docs and PR description now say explicitly that the merge queue has no native Linux GPU lane (MI300X already skipped it), only Windows Strix and WSL2. Overlong comment lines wrapped.
  • serve-26/27 on the pool: not re-dispatched; cited run 43025895270 on the open thread.
  • CI: the failures at 5abecbc were apt DNS errors; the head has moved and there are no red checks at 3c2bb61 (rerunning is not needed, the checks are still pending).

@tomastola
tomastola requested a review from juhovainio November 11, 2026 14:52

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change does what it says: both Strix Ubuntu and W7900 are gated off pull_request and merge_group, the gate test pins the whole if: string, and the nightly Strix job is moved onto the pool labels. I found nothing blocking in the workflow logic. Requesting changes for the stale claims below, plus a few questions.

Stale claims left behind (outside the diff, so no inline anchor)

  • tests/e2e-cucumber/src/expectation.rs, the merge_queue field doc (~line 177), still says the full serve matrix runs "on every lane but MI300X". After this PR the merge queue also skips Strix Ubuntu and W7900, so only the Windows Strix and WSL2 lanes run @merge-queue serves.
  • tests/e2e-cucumber/features/model_serving.feature (~line 364) still describes Strix Ubuntu as a per-PR job.

Questions / smaller points

  • Scope: the title only mentions the per-PR scope, but the merge queue also loses its native Linux GPU lane and vLLM entirely (only Windows Strix and WSL2 remain there). The body discloses this; is the title meant to cover it?
  • The moved e2e-gpu-nightly-strix job isn't exercised end to end. Can you run a workflow_dispatch of the nightly before or after merge? Docker availability on the pool host for the AIM serve scenarios (serve-26/27) is still unconfirmed.
  • The new nightly_strix_ubuntu_job_holds_no_static_host_paths test has no stated mutation check. Could you note that it goes red when a static-host path is reintroduced?
  • Two reworded lines wrap badly (inline).

CI at 3c2bb61

  • The Linux jobs (coverage, affected-crates tests, build-and-test, E2E) all failed in "Install native build deps" with an apt DNS failure on azure.archive.ubuntu.com. That looks like infra; please re-run. Until then cargo test -p xtask workflow_contract has no CI confirmation on this head.
  • Windows debug: therock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaults failed. This PR touches no production Rust, so it is likely flaky, but worth a re-run.

Comment thread tests/e2e-cucumber/src/expectation.rs Resolved
Comment thread docs/testing.md Resolved

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 3c2bb61a84e0

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (9 files: 2 workflow YAML, 1 Rust contract-test file, 1 shell script, 2 docs files, 1 README, 1 test-support Rust file).
No ticket named as closed by this PR; the description cites several historical ticket IDs (ROCMAI-125/429/430, ROCMAI-120/EAI-8761) as background for pre-existing design choices this PR builds on, not as something this PR itself closes.

Blocking

  • Required CI checks are failing at this exact head — build-and-test, build-and-test (ubuntu-26.04), Test (affected crates) (both variants), Windows build and test (debug), windows-build-and-test, Coverage (rocm-dash crates, ratcheted), E2E tests (both variants), all from workflow run 43042579975
    Verified by reading the job logs directly, not assumed. Two distinct causes:
    1. The Linux jobs (build-and-test, Test (affected crates), and apparently Coverage) fail at a sudo apt-get install -y pkg-config libcap-dev dependency-install step with Temporary failure resolving 'azure.archive.ubuntu.com' — a transient package-mirror DNS failure on the hosted runner, before any step touches this PR's changed files.
    2. Windows build and test (debug) fails a genuine unit test, therock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaults, in apps/rocm/src/therock/python/config_props.rs — a file this PR's diff never touches (the diff is CI workflow YAML, docs, a shell script, and xtask/tests/e2e-cucumber Rust only).
      Neither cause traces to anything in this diff, but the rule is objective: a failing check blocks merge regardless of root cause, and the PR's own mergeable_state is currently blocked.
      Confidence 100 (confirmed by direct log inspection) · mechanical · Fix: rerun once the runner network/mirror is healthy; separately confirm the Windows config_props failure is pre-existing/flaky (e.g. via git log on that file or a local rerun) since nothing in this diff could have caused it.

Non-blocking

  • An incomplete "per-PR" terminology sweep leaves stale claims in four places this change touches, after multiple rounds already fixed the same class elsewhere — docs/ci-hardware-testing.md:210, xtask/src/workflow_contract.rs:1609 (doc-comment + per_pr_job/per_pr naming on nightly_devlab_dispatch_lanes_share_the_per_pr_pool_labels, which this PR extends to cover e2e-gpu-strix-ubuntu/e2e-gpu-nightly-strix), xtask/src/e2e_report.rs:276,325,329 (test the_nightly_lanes_publish_the_same_platforms_as_the_per_pr_lanes and its per_pr-named locals/panic text), tests/e2e-cucumber/src/expectation.rs:177 (merge_queue field doc: "the full serve matrix (on every lane but MI300X)")
    docs/ci-hardware-testing.md:210 still reads "e2e-selfhosted.yml's e2e-report covers the per-PR GPU platforms" — eight lines after this same PR's own new text says two of those five platforms no longer run on pull_request. The workflow_contract.rs and e2e_report.rs instances still label e2e-selfhosted.yml's lane set collectively as "per-PR" in test names/doc-comments/panic messages, even though this PR makes that false for two of them; none of these are functional bugs (the underlying assertion logic is correct regardless of the label), but the wording actively misleads. The expectation.rs:177 field doc says the merge queue runs the full serve matrix "on every lane but MI300X" — after this PR it also excludes Strix Ubuntu and W7900, so only the Windows Strix/WSL2 lanes remain. This is the same defect class this PR's own commit history already fixed in at least 6 other spots (nightly.yml's job comment, docs/testing.md, tests/e2e-cucumber/README.md, an earlier e2e_report.rs docstring, etc.) — a new instance keeps surfacing each round.
    Confidence 85 · mechanical · Fix: same wording pattern as the already-fixed siblings — name which lanes still apply and which no longer do.

  • Stale TMPDIR-redirect comment in scripts/reclaim-gpu.sh, sibling of an already-fixed paragraph in the same file — scripts/reclaim-gpu.sh:1395-1397
    Still present tense: "...the nightly Strix lane redirects TMPDIR to the runner's home. A bare /tmp glob removed nothing at all there, so the guard was dead on the one lane that redirects." This PR removed that lane's only TMPDIR override (confirmed in nightly.yml: the job now runs on the DevLab Dispatch pool with no HOME/TMPDIR overrides), so no lane redirects TMPDIR anymore — both the "redirects" and "was dead" claims are now false. A near-identical paragraph earlier in the same file (lines 48-53) was correctly updated by this PR to past tense ("...did, to $HOME/actions-runner/tmp, before it moved to the pool"); this second copy was missed.
    Confidence 85 · mechanical · Fix: update to past tense to match lines 48-53.

  • Garbled sentence introduced by this PR's own edit — tests/e2e-cucumber/src/expectation.rs:1198-1200
    "may each wait out a 1200s readiness cap. Run on every PR on the Strix Ubuntu pool (before that lane left the PR path) they pushed the job (~25 min otherwise) into its 45 min cap on every run." The inserted parenthetical breaks the clause's grammar; it no longer parses. The equivalent fact is expressed cleanly in docs/testing.md:1244-1246 ("...which on the Strix Ubuntu job when it still ran per PR (about 25 minutes without them, 45 minute cap) used up the whole budget."). Comment only, no functional effect.
    Confidence 80 · mechanical · Fix: rephrase along the lines of the docs/testing.md version.

Decisions for the author

  • The same "per-PR"/"every lane but MI300X" staleness has now recurred across at least five review rounds in different files each time — non-blocking-improvement
    Each round's manual grep sweep catches the instances it looks for and misses others (see Non-blocking above). A contract test that greps for the literal phrase "per-PR" or "every lane but MI300X" anywhere in the repo near e2e-gpu-strix-ubuntu/e2e-gpu-w7900's job names, outside an explicit allow-list, would catch future drift mechanically instead of relying on the next person/round to notice.

Positive signals

  • The lane-selection choice is grounded in a precise, dated, two-round measurement table (998 then 944 runs; pass/fail/timeout/queue-wait per lane), including a direct side-by-side "could Strix Ubuntu replace MI300X" analysis — unusually rigorous evidence for a CI-scoping change.
  • The gate-pinning test (mi300x_lane_is_skipped_in_the_merge_queue) asserts the entire if: string rather than a substring, and both it and nightly_strix_ubuntu_job_holds_no_static_host_paths were traced by hand against the literal YAML: each would fail if the production change were reverted, and each fails under a single-clause mutation (an exclusion moved to the wrong branch, a banned env var reintroduced) — the dual verdict a new test should carry, confirmed rather than assumed.
  • e2e-report's skip-tolerance ([ "$result" = skipped ] && continue before the missing-artifact check) was read directly and correctly handles both newly-skipped-on-pull_request lanes without producing a false "missing artifact" error.

Deployment notes

  • Two previously-per-PR self-hosted lanes (e2e-gpu-strix-ubuntu, e2e-gpu-w7900) stop running on pull_request and merge_group, but keep running on push/dispatch/nightly. Neither check name is in the required-status-check list, so this doesn't change what blocks a merge — but the merge queue now has no native-Linux GPU lane at all (MI300X was already excluded); only the Windows Strix/WSL2 lanes run there.
  • The repointed e2e-gpu-nightly-strix job now runs on a shared ephemeral pool instead of a static host and is continue-on-error: true. The PR's own round-2 verification found the pool's AIM-restart canary (serve-27) already fails there on "Local server stop not confirmed" (both channel legs) — pre-existing and unrelated per the author, but it will now surface in the nightly report where the old, permanently-queued static host never reached it.

What this covered

Read all 9 changed files completely via two independent fan-out workers grouped by coupling (the two workflow YAMLs + the Rust contract-test file; the docs/README/script/support-test files), each also running repo-wide greps for every changed identifier/claim. I independently re-verified the key if: gate strings, env-var removals, and runs-on labels myself against the raw YAML/Rust, read the full PR diff directly, and read CI job logs directly for every failing check rather than assuming its cause. Commits reviewed: 1530f6b10b530805a89253a582f104a80bc7d41d (merge-base with origin/main) through 3c2bb61a84e0abce6615651be4e8b1c6e45af956 (8 commits on the branch).

Did not run: cargo test -p xtask workflow_contract directly in this environment (no local Rust toolchain invoked) — the new/changed assertions were hand-traced against the literal YAML/Rust text instead, cross-checked by two independent workers, consistent with CI's own report of that suite passing before the unrelated apt/DNS failures began blocking the run. No separate git log -p/git blame history pass was run beyond git log --oneline to confirm commit structure (all 8 commit messages are descriptive; no "fix"/"wip" noise).

Not reconciled against any discussion thread or prior review on this PR — this was a blind pass. Several earlier automated reviews from this same account exist on this PR at older, superseded commits, and at least one human review now also exists at this exact head; none were read before these findings were reached.

Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola tomastola changed the title ci: run pull requests on one Linux and one Windows GPU node ci: run pull requests on one Linux and one Windows GPU node; merge queue drops its native Linux GPU lanes Nov 11, 2026
@tomastola

Copy link
Copy Markdown
Collaborator Author

Round 6, head 81c0845:

  • Stale claims: expectation.rs (merge_queue doc) and model_serving.feature (AIM comment) fixed; I grepped every surface again for per-PR and merge_group claims and fixed docs/testing.md and the ci-hardware-testing.md merge_group line too.
  • Title now covers the merge queue dropping its native Linux GPU lanes.
  • Mutation check for nightly_strix_ubuntu_job_holds_no_static_host_paths: reintroducing /home/ubuntu/actions-runner/... in the job's reclaim step turned it red, and so did a $HOME/work/... read; both restored, workflow_contract 146 pass.
  • Wrapping fixed (two threads resolved).
  • Nightly dispatch: I am not dispatching nightly.yml from the PR branch, because it builds and publishes nightly release assets. The pool dispatch of e2e-selfhosted.yml (run 43025895270) already ran the same lane steps: serve-26 passed on both channels, so Docker is reachable, and serve-27 fails at stopping the AIM service (STOP FAILED rc=1), with no xfail row yet. The first scheduled nightly after merge is the end-to-end check; I can workflow_dispatch nightly on main after merge if you want.
  • ci: cancel a PR's queued self-hosted E2E runs when it is merged or closed #1032 is still open; I will rebase once it merges.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 15:32

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The PR path now runs MI300X + Windows Strix + WSL2, Strix Ubuntu/W7900 skip pull_request and merge_group, and the nightly Strix lane moves to the pool labels. The docs, README, feature comment and workflow_contract pins all agree with the workflow, and I found no stale per-PR Strix Ubuntu/W7900 claims left in the tree. No leaks in the diff.

Two tradeoffs the PR already documents, noting them only so they are a conscious choice: the merge queue no longer has any native Linux GPU lane, so heavy @merge-queue serves never run on a vLLM-eligible host before merge (Linux-only regressions surface on push to main); and no lane exercises the nightly package channel on a PR any more.

Non-blocking: the nightly Strix lane now relies on the same pool whose per-PR twin hit its cap in roughly 19% of jobs, and Docker on pool hosts is unconfirmed. A workflow_dispatch run of that lane would be worth doing before relying on it (inline comment below). There is also a small wrap nit in the comment around e2e-selfhosted.yml:108-110 ("...skip\n merge_group) and its\n required checks"), cosmetic only.

CI: the three red checks (Coverage rocm-dash crates, build-and-test, build-and-test ubuntu-26.04) all fail at apt-get with DNS resolution errors for azure.archive.ubuntu.com, so they look like infra and need a rerun. This review is a diff read; it does not replace those going green.

# the static `native` host, which is offline: every nightly sat queued until
# the 12h run cap and the lane never ran. Pool hosts are cold,
# so there are no host-specific HOME or cache-dir overrides.
runs-on: [self-hosted, linux, devlab-dispatch, strix-halo]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nightly Strix lane now runs on the same pool whose per-PR twin hangs to its cap in about 19% of jobs, and Docker on pool hosts is still unconfirmed (the step only warns). Please do a workflow_dispatch run of this lane before relying on it, so a hang does not show up as another silent nightly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not dispatching nightly.yml from this branch: it builds and publishes nightly release assets, so a dispatch from a PR branch would touch the shared release tag. What I can show is run 43025895270 of e2e-selfhosted.yml on the same pool (strix-ubuntu, include_nightly=true): Docker was reachable, serve-26 passed on both channel legs, serve-27 fails at stopping the AIM service (no xfail row yet), and nothing hung. The hang risk you name (about 19% of the per-PR twin's jobs) is bounded by the job's timeout-minutes: 120, and the job is continue-on-error, so a hang cannot block the nightly. The first scheduled nightly after merge is the end-to-end check, and I will workflow_dispatch nightly on main right after merge if you want it sooner. Leaving this thread open until that run exists.

A PR now starts the MI300X lane (the only lane where vLLM is the effective
engine) and the Strix Halo Windows pool lanes (native + WSL2). The Strix Halo
Ubuntu and W7900 lanes skip pull_request but keep push/merge_group/dispatch.

nightly.yml's Strix Halo Linux lane targeted the offline static `native`
host and never ran; point it at the DevLab Dispatch pool the per-PR lane uses
and pin that in the workflow contract test.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 12, 2026 09:41

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 0b2d5dc. The workflow if: gates, README, docs and comments agree on the new per-PR/merge-queue lane split, and I found no blocking issues. Requesting changes only for stale claims this PR makes untrue, which this repo expects fixed in the same change.

To fix

  • nightly.yml, e2e-gpu-nightly-strix header comment (inline): still reads as if a pool-based lane might be needed.
  • docs/testing.md (inline): still calls Docker on the nightly Strix host "unconfirmed".
  • e2e-selfhosted.yml ~L475 (outside the diff): still says "per-PR 45min budget" for a job that no longer runs per PR.
  • PR body: say why no Gherkin scenario is needed (CI-only change), per the contribution guidelines.

For the author to decide / confirm

  • The merge queue now has no vLLM-capable or native Linux GPU lane, so heavy @merge-queue serves run only on Strix Windows/WSL2. Is that coverage loss intended?
  • The re-pooled nightly Strix lane is not proven end to end (serve-27 already fails on the pool); it is continue-on-error, so please note the expected state in the PR.
  • The new nightly guard test is textual and rejects ${HOME} forms; fine if intentional.
  • I did not verify the "e2e-report tolerates skipped lanes" claim; please confirm it.

CI: completed checks are green; Strix Ubuntu/W7900 skipped by design; MI300X queued (runner pool issue).

Comment thread .github/workflows/nightly.yml Resolved
Comment thread docs/testing.md Resolved
The nightly Strix lane is on the pool where Docker was reached (serve-26
passed on both legs); say so, and note serve-27's stop failure. The
include_nightly budget comment no longer calls the push-run budget per-PR.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Round 7, head bfde201 (answers the review at 0b2d5dc): fixed the nightly.yml header comment, the docs/testing.md caveat and the e2e-selfhosted.yml 'per-PR 45min budget' wording (L475); the PR body now says why no Gherkin scenario is needed and records the expected state of the nightly Strix lane. Confirmed: e2e-report tolerates skipped lanes (its missing-report check skips any 'skipped' needs result; run 43049162834 had Strix Ubuntu and W7900 skipped and the report job green). The merge-queue coverage loss is intended, per the engineer: MI300X stays the per-PR Linux lane and is not added to the queue (queue wait about 12 h against its timeout). The ${HOME rejection in the nightly guard is intentional. Re-requesting review.

@tomastola
tomastola requested a review from juhovainio November 12, 2026 10:01

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the whole diff. I found no correctness defects in the workflow gating, but a couple of things should be fixed before merge.

To fix

  • model_serving.feature still says it is unconfirmed whether the nightly Strix Halo host has Docker, while the docs/testing.md and nightly.yml changes in this PR say Docker is reachable on the pool. The comment should match (inline).
  • serve-27 is documented as failing ("STOP FAILED rc=1") but has no expectation/xfail row, so every nightly will report an unexpected failure. Please add the row, or link a tracking issue, in this PR (inline).

Non-blocking

  • Dated measurements ("98 of 281", "35%", a specific run id) are copied into several durable docs and comments. The rationale holds without the numbers, and they will go stale.
  • Cosmetic: the comment reflow in e2e-selfhosted.yml leaves a dangling "merge_group) and its" line.

Questions for the author

  • The merge queue now has no native Linux GPU lane, and the heavy @merge-queue serves only run on Windows/WSL2 (lemonade-only). Is it intended that vLLM heavy serves never run before merge, beyond the serve-05 canary on PRs?
  • ROCMAI-125 recommends keeping the strix-ubuntu nightly-channel leg per PR; this PR drops it. The divergence is documented, but the ticket text should probably be updated.
  • The e2e-gpu-nightly-strix pool move has not been run end to end, so the first nightly is its first real run.

Checked and fine: the if: gates match what workflow_contract.rs pins, the nightly label pairing is correct, the "all five lanes" count is right, and I saw no internal-name leaks. This is a diff-only read; CI results are not part of it.

Comment thread .github/workflows/nightly.yml Resolved
…7's stop failure

The model_serving.feature AIM comment still called Docker on the nightly Strix
Halo host unconfirmed; dispatch run 43025895270 reached it (serve-26 passed on
both legs) and serve-27 fails at stopping the service.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 12, 2026 10:22

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking defects found; requesting changes for the items below, which I'd like settled before merge.

Should fix

  1. A run ID and a specific failure string ("STOP FAILED rc=1", "no xfail row yet") are pinned into three durable surfaces: nightly.yml, docs/testing.md and model_serving.feature. They go false the moment serve-27 gets an xfail row or is fixed, and the three copies drift independently. I checked the claim against the run log and it is true today. Keep the evidence in the PR body or a ticket and have the docs reference a tracking ID.
  2. Re-pooling the nightly Strix lane knowingly makes it red every night on serve-27, which has no xfail row and no ticket. It can't block the nightly or the wiki publish, but it is unaccounted-for noise. Please add an xfail row with a tracking ID in this PR, or name a follow-up ticket.
  3. "One Windows GPU node" is imprecise: a PR runs two Windows-pool jobs (e2e-gpu-strix-windows and e2e-wsl) on separate ephemeral runners. "One Windows GPU pool" is accurate. The wording appears in the title, the e2e-selfhosted.yml header, docs/ci-hardware-testing.md and the cucumber README.

Nit

  • The closed pull_request note in e2e-selfhosted.yml (around lines 43-45) still uses W7900 as its "busy runner" example, but W7900 no longer starts on pull_request.

For you to confirm (tradeoffs, disclosed already)

  • Native-Linux Strix and W7900 are no longer exercised before merge, and the merge queue has no native Linux GPU lane.
  • The only pre-merge vLLM serve is the per-PR canary serve-05 on MI300X, whose queue wait is often long.
  • No PR lane exercises the nightly channel.

All merge_group/pull_request claims across docs, README, workflow and xtask comments match the workflows, and the whole-if: contract-test pin looks solid. CI had no failures when I looked; several jobs were still pending.

Comment thread .github/workflows/nightly.yml Resolved
Comment thread docs/testing.md Resolved
Comment thread docs/ci-hardware-testing.md Resolved
…x Halo Ubuntu

Its stop step fails with 'Local server stop not confirmed' (tracked in
rocm-cli#1058), so the nightly lane reports it as xfail instead of an
unexpected regression. Point the nightly.yml, docs and feature comments at it.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 12, 2026 10:46
…lure text; say Windows pool

The serve-27 evidence lives in the tracking issue, so nightly.yml, docs/testing.md,
model_serving.feature and expectations.toml no longer quote a run ID or the
failure string. A PR runs two Windows-pool jobs (Windows and WSL2), so say
one Windows GPU pool, not node.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola tomastola changed the title ci: run pull requests on one Linux and one Windows GPU node; merge queue drops its native Linux GPU lanes ci: run pull requests on one Linux GPU node and one Windows GPU pool; merge queue drops its native Linux GPU lanes Nov 12, 2026

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the full diff and found no blocking defects. The lane if: gates, the E2E_MERGE_QUEUE removal and the nightly Strix re-pooling are consistent with each other, and the contract tests pin them well. Two small fixes before merge:

  1. The new serve-aim-lifecycle-parity expectation row is scoped more widely than its comment says (inline comment below).
  2. docs/ci-hardware-testing.md still carries two claims this PR makes untrue, on lines the diff doesn't touch: around L492, "persistent across runs on the long-lived native-labeled hosts" (this PR removes the last lane that targets native, and the new text elsewhere says nothing targets one), and around L252, "e2e-selfhosted.yml's e2e-report covers the per-PR GPU platforms" (that set now includes lanes that don't run per PR). Please reword both in this PR.

Two tradeoffs I read as deliberate, but please confirm:

  • The merge queue now has no native Linux GPU lane. The only Linux vLLM signal before landing is the per-PR MI300X canary, whose queue wait the PR itself measures at roughly 10-12 h median.
  • The Strix Ubuntu hang class is moved to the nightly rather than fixed, and the re-pooled nightly has not been exercised end to end. The first scheduled nightly after merge is the real proof.

Comment thread tests/e2e-cucumber/expectations.toml Resolved
…i-hardware-testing claims

The row matched os=linux, which WSL2 also reports on gfx1151, so a WSL host that
gained Docker and passed the scenario would XPASS. Add is_wsl = false. Reword the
e2e-report coverage line and the pre-warmed runtime paragraph, which still named
per-PR platforms that no longer all run per PR and the static native hosts.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Round 9, head c91e228 (answers the review at b2f23df):

  • The serve-aim-lifecycle-parity row now carries is_wsl = false, and its comment says why, so a WSL2 host that gains Docker and passes the scenario is not an XPASS.
  • docs/ci-hardware-testing.md: the e2e-report line now says it covers the self-hosted GPU platforms and that a PR runs only the MI300X and Windows Strix Halo pool lanes among them; the pre-warmed runtime paragraph no longer names the native-labeled hosts.

On the two tradeoffs, both are deliberate:

  • No native Linux GPU lane in the merge queue. Deliberate, the engineer's decision. MI300X stays the per-PR Linux lane and is not added to the queue because its queue wait (about 10-12 h median) is far above the queue's timeout. The only Linux vLLM signal before landing is therefore the per-PR MI300X canary (serve-05); the heavy @merge-queue serves run only on Windows Strix and WSL2. Coverage still runs on push, dispatch and nightly.
  • Strix Ubuntu hang class moved to the nightly, not fixed. Deliberate: this PR makes PR and merge-queue lanes reliable and does not claim to fix the hang. The re-pooled nightly lane is not proven end to end; the first scheduled nightly after merge is the real proof, and I can dispatch nightly.yml from main after merge if wanted. The job is continue-on-error, so it cannot block anything in the meantime.

I added nothing that depends on the other self-hosted nightly lanes. Re-requesting review.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the full diff and found nothing that is wrong in the code or the workflow gates. Moving the pull-request lanes to one Linux GPU node plus the Windows pool matches what the description says, and the contract test pins the whole if: of the two skipped lanes rather than a substring. I'm requesting changes for a few things I'd like settled before this merges, none of them a defect in the diff itself:

  1. The nightly Strix Ubuntu lane has never run. Its runs-on changed to the pool labels and the static-host HOME/CARGO_HOME/TMPDIR/PIP_CACHE_DIR overrides were dropped. The test only proves no static paths remain, not that the job works on a cold pool host (cache setup, prewarm, GPU preflight). Please dispatch nightly.yml, or e2e-selfhosted with include_nightly on strix-ubuntu, once and link the run here.
  2. WSL2 and the merge queue are described inconsistently. The docs and the expectation.rs comment say WSL2 runs in the merge queue, while the Triggers paragraph says the heavy serves run only on the Windows Strix Halo lanes. The contract test pins only the e2e-gpu-strix-windows/e2e-wsl gate, not whether e2e-wsl sets E2E_MERGE_QUEUE. Please confirm which is true and reword the other.
  3. Merge-queue coverage loss. With MI300X, Strix Ubuntu and W7900 all skipping merge_group, no Linux lane and no vLLM runs before merge. This is documented and I understand it is deliberate; flagging it so the tradeoff is acknowledged by the maintainers.
  4. The serve-27 xfail row (inline comment) is only evaluated on @nightly lanes, so a stale row only surfaces there.
  5. Docs drift risk. The Triggers paragraph in docs/ci-hardware-testing.md is now very long, and the "all five lanes" count is not pinned by any test. The skipped-lane rationale is also repeated in about five places.

CI is green on everything that has completed so far. Mock E2E, Windows build/test, affected-crate tests and coverage were still running when I looked.

# the scenario is not an XPASS. A host without Docker skips the scenario before
# this row applies.
[["serve-aim-lifecycle-parity"]]
when = { os = "linux", is_wsl = false, therock_family = "gfx1151*", has_amd_gpu = true }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This row is only evaluated on @nightly lanes, so please confirm that a nightly Strix Ubuntu run on the pool reports serve-27 as XFAIL (not skipped) before merge. The reason text ("not confirmed") is also thin; the tracking issue holds the evidence, which is fine, but a one-line symptom here would help.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a one-line symptom to the reason in ee855d7. To confirm XFAIL rather than skip I dispatched e2e-selfhosted.yml with platform=strix-ubuntu and include_nightly=true on this branch (https://github.com/ROCm/rocm-cli/actions/runs/43211119495); I will report what it shows for serve-aim-lifecycle-parity here and leave this thread open until then.

… WSL2 runs the merge-queue serves

The Triggers paragraph said the heavy serves run only on the Windows Strix Halo
lanes while other docs said WSL2 too. The workflow sets E2E_MERGE_QUEUE on both
e2e-gpu-strix-windows and e2e-wsl, so reword the paragraph and pin both in the
contract test. Add a one-line symptom to the serve-27 xfail reason.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Round 10, head ee855d7 (answers the review at c91e228):

  1. Pool run. Dispatched e2e-selfhosted.yml on this branch, platform=strix-ubuntu, include_nightly=true: https://github.com/ROCm/rocm-cli/actions/runs/43211119495 (not nightly.yml, which publishes release assets). It exercises the same pool labels, Rust bootstrap, cache setup, prewarm and GPU preflight as the nightly job's body. It is the dispatch twin, not the e2e-gpu-nightly-strix job itself, so the first scheduled nightly after merge is still the end-to-end proof of that exact job. I will report the result here, including whether serve-aim-lifecycle-parity shows as XFAIL.
  2. WSL2 and the merge queue. The workflow is the source of truth: both e2e-gpu-strix-windows and e2e-wsl set E2E_MERGE_QUEUE for merge_group, so WSL2 does run the heavy serves there. The Triggers paragraph said "only the Windows Strix Halo lanes"; it now says Windows Strix Halo and WSL2. The contract test (mi300x_lane_is_skipped_in_the_merge_queue) now pins that both jobs set E2E_MERGE_QUEUE to the merge_group-only expression; I checked it goes red when e2e-wsl loses it, then restored.
  3. Merge-queue coverage loss. Acknowledged, deliberate (engineer's decision), as described in the PR body.
  4. serve-27 row. I added a one-line symptom to the reason. The row is only evaluated on a lane that includes the nightly scenarios, which is why I dispatched with include_nightly=true; I will confirm XFAIL (not skip) from that run before this merges.
  5. Docs drift. The "five lanes" count is covered indirectly: closing_a_pr_supersedes_its_self_hosted_runs pins an if: for every lane job and fails on an unlisted sixth, so the prose count cannot silently change without that test being touched. Shortening the Triggers paragraph and de-duplicating the skipped-lane rationale is real cleanup, but it would rewrite text unrelated to this change; I would rather do it as a follow-up than widen this PR.

Re-requesting review.

@tomastola
tomastola requested a review from juhovainio November 12, 2026 11:40

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings. Two non-blocking points, one wording and one to check:

  • The W7900 "never started" figure does not match the PR's own table (202 of 507 jobs, about 40%, vs "70%" in the body and "58-70%" in the workflow comment). The comment is durable, so worth fixing.
  • bug = "#1058" in expectations.toml differs from the tracker-ID form the neighbouring rows use; please confirm the xfail report and stale-xfail tooling accept it.

Smaller notes:

  • The re-pooled nightly Strix lane is not proven end to end (the PR says so), and the guard test is textual.
  • mi300x_lane_is_skipped_in_the_merge_queue now pins more than its name says; a rename would help.
  • I did not check other "per-PR Strix" mentions in docs outside the diff.

Tradeoffs the author has presumably weighed: the merge queue ends with no native Linux GPU lane and no vLLM lane, no PR lane exercises the nightly package channel, and Windows Strix + WSL2 are the only runners of the heavy @merge-queue serves. The docs state all of this plainly.

CI at review time: 21 pass, 12 pending, 0 fail.

fail-fast: false
matrix:
sdk_version: ${{ fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix) }}
# Not on pull_request or merge_group: one runner on borrowed quota, 58-70% of

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"58-70% of its PR jobs never started" does not match the PR table (W7900: 202 of 507 jobs never started, about 40%); the PR body also says 70%. Please correct the figure here or state the denominator.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the 58-70% figure does not match the PR table. I will correct it in the workflow comment and the docs in a small follow-up PR from main once this merges, rather than re-spin the queue entry.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I will not correct the W7900 figure. The W7900 lane is being removed entirely by a separate PR (#1063's work), so this comment and the matching docs text go away with it; a number fix here would be churn. The tracker-ID point is a separate small PR: #1065.

# this row applies.
[["serve-aim-lifecycle-parity"]]
when = { os = "linux", is_wsl = false, therock_family = "gfx1151*", has_amd_gpu = true }
bug = "#1058"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other rows use a tracker ID in bug (e.g. ROCMAI-900); please confirm #1058 is an accepted form wherever bug is parsed or rendered (xfail report, stale-xfail hint).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point: the other rows use the tracker-ID form, and #1058 is a GitHub issue. I will switch the row to the convention in the same follow-up PR after this merges.

@tomastola
tomastola added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit b109db1 Nov 12, 2026
47 of 49 checks passed
@tomastola
tomastola deleted the ci/pr-lanes-two-nodes branch November 12, 2026 12:41
tomastola added a commit that referenced this pull request Nov 12, 2026
The R9700, MI350P and W7900 nightly jobs have sat queued for hours and
been cancelled since at least 11-09. Remove them from nightly.yml and
from the consolidated report that waited on them. These lanes now run
on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu,
Windows, WSL2, all on the Strix pool since #1018) are unchanged.

Update the workflow contract tests and the docs to match.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants