Repository navigation
ci: run pull requests on one Linux GPU node and one Windows GPU pool; merge queue drops its native Linux GPU lanes - #1018
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head e717523e
This automation never files an approving review — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (single-context pass; justification below).
No ticket named — the description cites internal tracking IDs (ROCMAI-125, EAI-8761, etc.) inline but does not claim to close any GitHub issue.
Blocking
docs/ci-hardware-testing.md's "Release-candidate regression gate" row is now stale —docs/ci-hardware-testing.md:41
This PR redefines "per-PR lanes" (row above, line 39) to mean the narrowed 3-lane set (MI300X + Strix Windows + WSL), explicitly excludinge2e-gpu-strix-ubuntuande2e-gpu-w7900. The untouched "Release-candidate regression gate" row still reads "The same per-PR lanes," which under the new definition implies the release-candidate gate also narrowed to 3 lanes. It did not: theif:conditions for bothe2e-gpu-strix-ubuntuande2e-gpu-w7900only excludegithub.event_name == 'pull_request', notpush, so a push to arelease/**branch still runs all 5 self-hosted lanes. The same "Triggers" paragraph this PR edited a few sentences later even says so explicitly: "pushing to a release branch runs the full self-hosted matrix." So the doc now contradicts itself within the same section, and a release manager skimming the table would wrongly conclude Strix Ubuntu/W7900 are no longer exercised before cutting a release tag.
Confidence 90 · mechanical · Fix: reword the row (e.g. "The full self-hosted matrix (all platforms below, including the two narrowed out of the per-PR gate)") so it doesn't inherit the new, narrower meaning of "per-PR lanes."
Non-blocking
None
Decisions for the author
None
Positive signals
- The lane-selection choice is backed by a precise, dated 998-run measurement table (pass/fail/timeout/queue-wait per lane) rather than a hunch — unusually rigorous grounding for a CI-scoping change.
xtask/src/workflow_contract.rs's updatedmi300x_lane_is_skipped_in_the_merge_queuetest pins the new per-PR/merge-queue contract as literal substring assertions on each job'sif:gate, so a future accidental revert of thepull_requestexclusion is caught by CI instead of discovered in production.- The offline
native-host nightly-Strix bug (every scheduled nightly queued to the 12h cap since 11-03) is called out with its own evidence and fixed as a clearly-labeled drive-by, including an honest "not exercised end to end" caveat for its first scheduled run — not silently bundled in.
Deployment notes
- After merge, PRs will no longer exercise
e2e-gpu-strix-ubuntuore2e-gpu-w7900; regressions specific to those two lanes surface only on push/merge_group or the nightly run (e2e-gpu-nightly-strix,e2e-gpu-nightly-w7900) — consistent with the author's own "Nightly watch" follow-up item. - The repointed
e2e-gpu-nightly-strixjob (now on the DevLab Dispatch pool) had not executed at review time; its first scheduled run is the real confirmation that the fix works, as the description itself notes.
What this covered
Read all four changed files in full: .github/workflows/e2e-selfhosted.yml (1839 lines), .github/workflows/nightly.yml (~2046 lines — the touched e2e-gpu-nightly-strix job and its per-PR/nightly siblings read closely; the unrelated release/publish jobs skimmed for cross-references), docs/ci-hardware-testing.md (full file), and xtask/src/workflow_contract.rs (the touched tests read in full; the rest of the ~9750-line file searched exhaustively for every reference to the changed job names/gate text rather than read line-by-line, since it is overwhelmingly unrelated pre-existing tests). Reviewed e717523e02a154e8807c0f2c3909367a55e83b07.
Single-context pass, not fanned out: the change is small (4 files, one tightly-coupled workflow pair plus the contract test and doc that describe them) and fits well within what one reviewer can hold completely.
Did not run: cargo test -p xtask workflow_contract was not executed directly (no local checkout in this environment); the new/changed assertions were instead traced by hand against the committed YAML, and the PR's own CI (get_check_runs/get_status, all green, including "E2E consolidated report (self-hosted)") was taken as corroboration that the Rust suite passed. No separate git log/git blame pass was run; the inline comments in both workflow files were read as part of the main pass, which is where the one blocking finding came from.
Deviation to flag: the PR description and title were fetched alongside the diff in the same initial call, so this review was not conducted fully blind to the stated intent as the method prescribes. Findings were nonetheless reached by independently tracing the committed YAML/Rust rather than taken on faith from the description, and the description was deliberately re-checked at the end against the settled findings — it accurately reflects the diff, no scope mismatch found. Not reconciled against any discussion on this PR (one existing comment was left unread, per the method's rule not to read discussion).
Not a human review. This is an automated code review; please verify findings independently before acting on them.
|
Fixed in be3088b.
|
juhovainio
left a comment
There was a problem hiding this comment.
I went through the diff and found nothing that breaks the intent (one Linux and one Windows GPU node per PR), but there are a few things I'd like addressed before merge. Two are inline below.
- The repointed
e2e-gpu-nightly-strixjob loses itsCARGO_HOME/RUSTUP_HOME/TMPDIR/PIP_CACHE_DIRoverrides. Please confirm no later step in that job still reads them, since the job hasn't run since the change. - The new contract assertion only pins the string
github.event_name != 'pull_request'and the absence ofmerge_group. A gate that drops push/merge_group some other way would still pass. Please tighten it and say in the PR text that you broke the gate and saw the test go red. - Please grep README,
docs/testing.mdand the Platforms table for per-PR coverage claims about native-Linux Strix and W7900, which this change makes stale. The "Triggers" paragraph indocs/ci-hardware-testing.mdis also now one very long line; worth reflowing.
Note the tradeoff: PRs no longer get native-Linux Strix (gfx1151) or W7900 coverage, only push/merge_group/dispatch and nightly do. The PR body discloses that, so I'm treating it as an accepted decision. CI shows no failures so far; the build/test and E2E lanes were still pending when I looked.
jussielo-amd
left a comment
There was a problem hiding this comment.
Nothing blocking. I traced the changed if: gates (the added github.event_name != 'pull_request' sits in the non-dispatch arm, so push, merge_group, dispatch and release/** still run both lanes), the e2e-report handling of skipped needs, the repointed nightly Strix labels against the per-PR lane, the removed env overrides, and the contract-test substring assertions against the YAML; they agree. The earlier stale release-candidate row is fixed on this head. This was a static review: nothing was run (no Rust toolchain here), and the self-hosted lanes were still in progress when I read CI; the GitHub-hosted checks are green.
Non-blocking, all stale wording that this change leaves behind in docs/ci-hardware-testing.md and nearby:
- Lines ~88-93: "so the per-PR smoke gate proves the nightly channel boots on at least one platform" is no longer true, since the only lane with the
channelmatrix no longer runs onpull_request; after this change no PR run exercises the nightly channel. Worth saying so explicitly. - Line ~301 still says "all three per-PR Strix lanes (Ubuntu, Windows, WSL2)", and line ~203 says
e2e-reportcovers "the per-PR GPU platforms"; also ~401 ("long-livednative-labeled hosts") and.github/workflows/e2e-selfhosted.yml~485 ("the only per-PR job with two independent matrix axes"). .github/workflows/nightly.yml~765-770: the comment "no nightly lane runs AIM serve and a pool-based one is needed" is stale now that this job is the pool-based one; the open question is Docker on the pool hosts, which the first scheduled run will answer.- Some numbers in the description don't match its own table: W7900 "70% of its jobs never started" is 202 of 507 (about 40%), Strix Ubuntu "19% hang to the cap" is 394 of 1144 (about 34%), and the
mi300x-0/mi300x-1failure counts (129 of 432, 8 of 88) don't add up to the table's 106 test failures plus 2 infra failures. The YAML comment "58-70%" has the same issue. Reconciling or stating the denominators would keep the rationale checkable. - Untouched, but now wrong:
scripts/reclaim-gpu.sh(~49, ~1394) says the nightly Strix lane redirects TMPDIR to$HOME/actions-runner/tmp, anddocs/testing.md(~1245) still calls Ubuntu a per-PR job. The new two-lanes-skip-pull_requestassertion lives insidemi300x_lane_is_skipped_in_the_merge_queue, so a failure there will point at the wrong rule; a dedicated test that the three kept lanes still run onpull_requestwould pin the other half.
🤖 by agenthub on AMD AgentHub
|
Addressed in 87b214e (signed, with sign-off). All three review points hold and are fixed; both inline threads are replied to and resolved:
Not rebased onto #1032; whichever merges second needs it. |
juhovainio
left a comment
There was a problem hiding this comment.
I re-reviewed the current head (87b214e). Nothing here is a hard blocker for the change itself, but a few things need an answer or a fix before this goes in, so I'm requesting changes.
To resolve before merge
- The nightly Strix lane now runs on the pool hosts, and its body hasn't been exercised end to end. Pool hosts are cold; if Docker isn't reachable there, serve-26/27 (AIM, nightly-only) skip silently and the job stays green, so the only place those canaries run stops covering them. A
workflow_dispatchrun on the pool, or a note in the PR on the Docker question, would settle it (inline comment on theruns-online). - The PR body says
e2e-reportalready tolerates skipped lanes, but that file isn't in the diff. The passing "E2E consolidated report" check is theci.ymlmock-lane report, not the self-hosted one. Please confirm from the self-hosted report path. - Please grep the repo for other repeats of the "per-PR Strix Ubuntu / W7900" claim (README.md, docs/nightly-on-the-queue.md, workflow comments) and update any that are now stale. I only saw the diff.
- Two prose lines were left overlong after the reflow:
docs/ci-hardware-testing.mdline 320 (inline) anddocs/testing.mdaround line 1245.
Worth confirming (tradeoffs, not blocking)
- Strix Ubuntu and W7900 still run on
pushandmerge_group. W7900 has a single runner and a very long queue, so the merge queue can still sit on the capacity problem this PR removes from the PR path. They are non-required, so I assume that's intended. - No PR lane now exercises the nightly package channel, and with the nightly pools down, effective coverage of it is zero for now. The docs say so; just flagging it.
nightly_strix_ubuntu_job_holds_no_static_host_pathsonly checks the stringactions-runnerand four env names, so the "no static host paths" wording overstates it. A$HOME/...read other than.cargo/binwould slip through.
What looked good: the gate test compares the whole if: rather than a substring, the nightly lane is pinned to the pool labels by the contract test, and the static-host env is gone. I did not run the contract tests locally; this is a diff read plus CI status.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 87b214e88c3f
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (7 files: 2 workflow YAML, 1 Rust contract-test file, 1 shell script, 3 docs files).
No single ticket named as closed by this PR; the description cites several historical ticket IDs (ROCMAI-125/429/430, EAI-8761/7548/8057) as context for pre-existing design choices this PR builds on, not as something this PR itself closes.
Blocking
None
Non-blocking
- Self-contradictory "per-PR Strix lanes" claims left in
docs/ci-hardware-testing.md—docs/ci-hardware-testing.md:207,:210,:305-306
Line 305-306 reads "Strix Halo isgfx1151on every OS, so all three per-PR Strix lanes (Ubuntu, Windows, WSL2) ... are lemonade-only" — directly contradicting this same PR's own edits a few dozen lines earlier (lines 47-53, 88-90) statinge2e-gpu-strix-ubuntuno longer runs onpull_request. Line 207/210 similarly still calls thee2e-reportjob's 5-lane set "the per-PR GPU platforms," though two of those five (e2e-gpu-strix-ubuntu,e2e-gpu-w7900) are no longer per-PR per this PR's own change. Same root cause (an incomplete "per-PR" sweep in this file), two sites.
Confidence 85 · logic · Fix: say "two" and drop Ubuntu at 305-306; relabel the 207/210 set as "the self-hosted GPU platforms." docs/testing.md's AIM-serve budget rationale anchors on a job that no longer runs per PR —docs/testing.md:1242-1251
The paragraph justifies whyserve-26/serve-27stay@nightlyby citinge2e-gpu-strix-ubuntu's wall-clock budget "when it still ran per PR" — but that job is now off the PR path, so the paragraph's only concrete evidence no longer describes a currently-at-risk lane.e2e-gpu-strix-windowscarries the identical 45-minute cap and still runs per PR today, but goes unmentioned.
Confidence 70 · logic · Fix: anchor the rationale one2e-gpu-strix-windowsinstead (or in addition).- Stale TMPDIR-redirect comment in
scripts/reclaim-gpu.sh'smain()—scripts/reclaim-gpu.sh:1394-1399
This comment still states in the present tense that "the nightly Strix lane redirects TMPDIR to the runner's home" and "the guard was dead on the one lane that redirects." This PR removes the onlyTMPDIRoverride in either workflow file (the nightly Strix lane's env block), so no lane redirects TMPDIR anymore. A near-identical paragraph earlier in the same file (lines 46-54) was correctly updated to past tense; this second occurrence was missed. The script logic itself (scenario_tmp_root()) is unaffected and remains correct.
Confidence 85 · mechanical · Fix: update to past tense, matching the lines 46-54 edit. - Stale "per-PR" comment in
xtask/src/e2e_report.rs(file not touched by this PR) —xtask/src/e2e_report.rs, doc-comment abovewithout_channel_matrix_segment
"...nightly's six GPU lanes and, per ROCMAI-125,e2e-gpu-strix-ubuntu's per-PR[release, nightly]leg..." — this lane's channel matrix is unchanged, but it is no longer a per-PR leg after this PR's gate change. Outside the diff, so missed by the in-PR doc sweep.
Confidence 85 · mechanical · Fix: drop "per-PR" or note the lane now runs on push/merge_group/dispatch/nightly.
Decisions for the author
- Keeping MI300X as the sole per-PR Linux GPU lane — tradeoff
It is the only per-PR lane with vLLM coverage, but the job's own comment already documents its capacity constraint ("limited MI300X runner capacity, the queue entry's job can stay queued long after its PR merged"). Dropping Strix-Ubuntu/W7900 removes their queue problems from the PR path but leaves MI300X's as the one that remains. Worth confirming this is accepted rather than merely the least-bad option by elimination. - No per-PR lane now exercises the nightly package channel — tradeoff
The PR's owndocs/ci-hardware-testing.mdedit states this plainly:e2e-gpu-strix-ubuntuwas the only lane with a[release, nightly]channel matrix, and it's now off the PR path entirely. This is disclosed, not hidden, but worth the author confirming it's an acceptable gap until the next nightly run.
Positive signals
- The new contract test pins the entire
if:gate string fore2e-gpu-strix-ubuntu/e2e-gpu-w7900(not a substring), which the PR's own commit history shows was deliberately hardened after a manual break-test caught a substring check passing on a wrong gate. nightly_strix_ubuntu_job_holds_no_static_host_pathsis a well-targeted regression guard for exactly the failure mode that silently broke the nightly Strix lane before this PR (an unset variable defaulting to empty rather than failing loudly).
Deployment notes
Two previously-per-PR self-hosted lanes (e2e-gpu-strix-ubuntu, e2e-gpu-w7900) stop running on pull_request but keep running on push/merge_group/workflow_dispatch/nightly. Neither lane's check name is in the repository's required-status-check list (per the workflow's own header comment), so this does not change what blocks a merge — but anyone relying on their per-PR signal (e.g. watching their job status on a PR) needs to know to look at push/nightly runs instead.
What this covered
Read completely: .github/workflows/e2e-selfhosted.yml, .github/workflows/nightly.yml, xtask/src/workflow_contract.rs, scripts/reclaim-gpu.sh, docs/ci-hardware-testing.md, docs/testing.md, tests/e2e-cucumber/README.md — via two independent fan-out workers (grouped by coupling: workflow+contract-test, and docs), plus my own direct reads of the key job blocks (e2e-gpu, e2e-gpu-strix-ubuntu, e2e-gpu-w7900, e2e-gpu-nightly-strix, e2e-report), the contract test's helper functions (job_block, job_mapping, the gate closures), and a repo-wide search for other references to the two affected job names. Commits reviewed: 431c48c3c752048285095e57e8309bcc38ebb25d…87b214e88c3fd05055bac169755ad9ff5fbd906e (merge-base to the assigned head, all 4 commits on the branch at that point).
Did not run: cargo test -p xtask workflow_contract (no build toolchain in this review environment) — verified the new/changed assertions by direct comparison against the YAML instead. CI at 87b214e88c3fd05055bac169755ad9ff5fbd906e: all required checks (CI/build-and-test, CodeQL, Security scan) are green. The self-hosted "E2E self-hosted" pull_request run for this exact commit shows cancelled — superseded by a later push to the branch, which is the workflow's own documented concurrency behavior, not a failure, and none of these self-hosted lanes are required checks.
Not reconciled against any review thread or prior comment on this PR — this was a blind pass, and a stale earlier review from this account exists at a superseded commit that was deliberately not read. Fan-out was not degraded: both workers completed independently and no retry was needed.
No issue-tracker tool is available to this review; the description names several historical ticket IDs as background for pre-existing design choices, not as a ticket this PR closes, so there was nothing to retrieve.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
|
Round 2 addressed in 2f9c9c6 (signed, with sign-off). Both inline threads are replied to and resolved. The other points:
|
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed the full diff at 2f9c9c6. I found nothing blocking: the two if: gates, the nightly runs-on switch, the contract-test pins and the README/docs updates agree with each other and with the stated intent, and the diff scan is clean. All checks on this head pass. Self-hosted GPU lanes are not required checks.
Non-blocking points, with inline comments on the first three:
- The demoted Strix Ubuntu and W7900 lanes keep
merge_groupandpush, so they become some of the heaviest lanes in the merge queue and on main. - The nightly Strix job's move to the pool is only text-guarded and is
continue-on-error, so a first-run failure on a cold host would be silent. A one-off dispatch would confirm it. - The nightly.yml comment says the report crate asserts that nightly lanes publish the same platforms as the per-PR lanes. Worth confirming that assertion is not keyed on a fixed list now that those lanes skip
pull_request. - The new guard test is substring-based and only strips full-line comments. It is fine as a tripwire, but it cannot prove the job runs on the pool.
| /// textual guard over the job's non-comment lines, not proof the job runs. | ||
| #[test] | ||
| fn nightly_strix_ubuntu_job_holds_no_static_host_paths() { | ||
| let nightly = read_workflow("nightly.yml"); |
There was a problem hiding this comment.
The guard is textual and substring-based (a future HF_TMPDIR would trip the TMPDIR check, and trailing # ... comments are not stripped). Fine as a tripwire, but it cannot show the job actually runs on the pool and the job is continue-on-error. Please run a one-off nightly dispatch to confirm it end to end.
There was a problem hiding this comment.
Whole-word matching is in 029c48c (HF_TMPDIR no longer reads as TMPDIR). Trailing # ... comments inside a run script are still not stripped, so the guard is a tripwire only, and its doc comment says so.
On the end-to-end run: I did run the same lane on the pool. e2e-selfhosted.yml run 43025895270 (platform=strix-ubuntu, include_nightly=true, serve-2[67]) covered the pool labels, the Rust bootstrap, the GPU preflight, the build and the prewarm; it showed Docker is reachable (serve-26 passed on both channel legs). I have not dispatched nightly.yml itself: it also builds and publishes nightly release assets and cleans up a staging release, and a dispatch from a PR branch is not something I will risk on the shared release tag. The first scheduled run is the real end-to-end confirmation. If you want one before merge, the safe route is a dispatch of nightly.yml from main after this merges, or I can add an inputs-gated lane filter to nightly in a follow-up. Leaving this open for you.
There was a problem hiding this comment.
Per the earlier discussion I have not dispatched anything new for this. The AIM canaries were already exercised on the pool by e2e-selfhosted.yml run 43025895270 (platform=strix-ubuntu, include_nightly=true, serve-2[67], both channel legs): Docker is reachable and serve-26 passed on both legs, so they do not skip silently there. serve-27 failed at "Local server stop not confirmed"; that is reported in the PR description. A dispatch of nightly.yml itself is not run from this branch because it also builds and publishes release assets; the first scheduled run remains the end-to-end confirmation. Leaving this thread for you to close.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 2f9c9c68
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — DEGRADED (single-context pass) — needs work
Full review of the whole change (7 files: 2 workflow YAML, 1 Rust contract-test file, 1 shell script, 3 docs files), diffed against the merge-base with main.
No ticket named as closed by this PR; the description cites several historical ticket IDs (ROCMAI-125/429/430, EAI-8761/7548/8057) as background for pre-existing design choices this PR builds on, not as something this PR itself closes.
Blocking
-
A newly-added comment in this PR calls the very job this PR removes from
pull_requesta "per-PR" lane —.github/workflows/nightly.yml:771-772
This PR's own diff replaces the olde2e-gpu-nightly-strixcomment with: "The DevLab Dispatch pool, identical to the per-PRe2e-gpu-strix-ubuntulane ine2e-selfhosted.yml..." But this same PR'sif:change makese2e-gpu-strix-ubuntuexplicitly not run onpull_requestanymore (&& github.event_name != 'pull_request', confirmed in the committed YAML). The comment is new text this PR introduces, and it directly contradicts the PR's own headline change in the same file set — it reintroduces exactly the "per-PR" mislabeling that three review rounds on this PR already caught and fixed in four other spots (docs/ci-hardware-testing.md,docs/testing.md,.github/workflows/e2e-selfhosted.yml,tests/e2e-cucumber/README.md).
Confidence 88 · mechanical · Fix: drop "per-PR" (e.g. "identical to thee2e-gpu-strix-ubuntulane, which runs on push/merge_group/dispatch but no longer onpull_request"). -
PR description does not mention the two most recent commits' changes — PR description (no single file)
The description's only changelog section is "Review round 1 (juhovainio)", describing commit87b214e8. But the current head is a later commit,2f9c9c68("ci: tighten the nightly Strix path guard; fix remaining per-PR Strix/W7900 claims"), pushed in response to a secondCHANGES_REQUESTEDreview from the same reviewer at87b214e8(asking to tighten the$HOMEguard further, fix remaining stale "per-PR" claims, and reflow two overlong lines). That commit does real, reviewable work — it strengthensnightly_strix_ubuntu_job_holds_no_static_host_pathswith a new$HOME-usage scan, fixes two more stale "per-PR" doc sentences, and reflows two lines — none of which is described anywhere in the PR body. A reviewer reading only the description would not know this round of changes exists.
Confidence 85 · architectural · Fix: add a "Review round 2" section (or fold it into the existing verification notes) describing what87b214e8→2f9c9c68changed and why.
Non-blocking
- Stale "per-PR" wording left in two files outside this PR's own doc sweep —
xtask/src/e2e_report.rs:145,tests/e2e-cucumber/src/expectation.rs:1199
e2e_report.rs:145's doc-comment still reads "...per ROCMAI-125,e2e-gpu-strix-ubuntu's per-PR[release, nightly]leg...", andexpectation.rs:1199's test comment still reads "...may each wait out a 1200s readiness cap. Run per-PR on the Strix Ubuntu pool..." — both now false the same way the four already-fixed spots were. Neither file is touched by this PR's diff, and neither affects test behavior (thee2e_report.rstest still compares artifact names correctly regardless of per-PR status), so these are comment-only drift, same root cause as the blocking nightly.yml finding above, just outside the files this PR happened to touch.
Confidence 80 · mechanical · Fix: same wording fix, next time either file is touched, or in this PR's doc sweep since it already found four siblings of this exact phrase.
Decisions for the author
- Keeping MI300X as the sole per-PR Linux GPU lane — tradeoff
It's the only per-PR lane with vLLM coverage, but the PR's own measurement table shows it has the worst queue wait of any lane kept on the PR path (median ~10h, p90 ~22h on 2 runners) and a double-digit per-runner failure rate. Dropping Strix-Ubuntu/W7900 removes their queue problems from the PR path but leaves MI300X's queue problem as the one that remains and is now the sole Linux signal. Worth confirming this is the accepted remaining risk rather than an overlooked one. e2e-gpu-nightly-strixnow shares the DevLab Dispatch pool with the per-PR Windows Strix/WSL2 lanes — tradeoff
Previously isolated on its own (offline) static host, it now draws from the same ephemeral pool (6 Linux + 5 Windows runners per the PR's own capacity numbers) thate2e-gpu-strix-windowsande2e-wsluse on every PR. A single daily scheduled run is unlikely to cause real contention, but aworkflow_dispatchre-run during business hours would compete with live PR lanes for pool capacity in a way the old static host never did. Worth a one-line acknowledgment if considered and accepted.
Positive signals
- The lane-selection choice is backed by a precise, dated 998-run measurement table (pass/fail/timeout/queue-wait per lane) rather than a hunch — unusually rigorous grounding for a CI-scoping change.
- The gate-pinning test (
mi300x_lane_is_skipped_in_the_merge_queue) was iteratively hardened in direct response to review feedback and the author demonstrated it actually catches a regression (documented break-test: moving thepull_requestexclusion into the wrong boolean branch, or adding an extrapushexclusion, both turned the test red before being reverted). - The PR's claim that "
e2e-reportalready tolerates skipped lanes" checks out: I read the untouchede2e-reportjob ine2e-selfhosted.ymldirectly — its "Check every lane that ran uploaded its report" step explicitly does[ "$result" = skipped ] && continuebefore checking for a missing artifact, so the two newly-skipped-on-pull_requestlanes will not produce a false "missing artifact" error.
Deployment notes
Two previously-per-PR self-hosted lanes (e2e-gpu-strix-ubuntu, e2e-gpu-w7900) stop running on pull_request but keep running on push/merge_group/workflow_dispatch/nightly. Neither lane's check name is in the repository's required-status-check list, so this does not change what blocks a merge — but anyone relying on their per-PR signal on a PR page needs to know to look at push/nightly runs instead. The repointed e2e-gpu-nightly-strix job has not executed on its new pool hardware at review time (continue-on-error, so a cold-start failure stays non-blocking); its next scheduled run is the real confirmation, as the PR itself discloses.
What this covered
DEGRADED — fan-out could not run. Three attempts to spawn independent review workers (grouped by coupling: the two workflow YAMLs together, the Rust contract-test file alone, and the docs/script group) all failed immediately with "Concurrent subagent limit reached... Do not retry" (confirmed via the agent-listing tool: 20–21 unrelated subagents already occupying the platform-wide slot limit at the time). Per instruction, retries were not attempted. This is therefore a single-context pass, done as thoroughly as that allows — recommend re-running with fan-out available for independent cross-check.
To compensate, this pass went beyond a typical single-context read: full diff at the merge-base (1530f6b1…2f9c9c68) for all 7 changed files; full git show reads of both complete workflow YAML job blocks (changed jobs and their runner-pool siblings); hand-traced boolean evaluation of every changed if: gate against pull_request/push/merge_group/workflow_dispatch; read and hand-traced every new/changed Rust test function (mi300x_lane_is_skipped_in_the_merge_queue, nightly_strix_ubuntu_job_holds_no_static_host_paths, nightly_devlab_dispatch_lanes_share_the_per_pr_pool_labels) together with the helper functions they call (job_block, job_mapping, normalized_whitespace, the gate closure) against the literal YAML; read the untouched e2e-report job in full to verify the PR's own skip-tolerance claim; a repo-wide grep at the PR head for every remaining reference to the two narrowed-out job names, with full-paragraph context read around each hit; CI status (get_check_runs/get_status) — all required checks (build-and-test, clippy, CodeQL, security scans, commit-signatures) green at this exact head, which exercises cargo test -p xtask workflow_contract; and the current PR description cross-checked against the full commit log on the branch.
Did not run: cargo test -p xtask workflow_contract directly in this environment (no local toolchain invoked) — relied on hand-tracing plus the green required CI as corroboration. No separate git log/git blame history pass beyond reconciling the PR body against the branch's commit log for Step 9. No prior-review-comments pass: this PR's earlier review threads were necessarily visible during the mandatory pre-review duplicate-check (confirming no review already existed at this exact head), but no finding, wording, or conclusion from them was carried into this review — everything reported above was independently re-derived from the code and confirmed by direct reading. Not reconciled against any discussion thread on this PR.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
Request changes: one blocking doc/comment-accuracy issue, otherwise the change looks sound.
Blocking
- Round 4 dropped
e2e-gpu-strix-ubuntufrommerge_group, but several claims still say it runs there: the comment innightly.yml(~L772), the two rows indocs/ci-hardware-testing.md(L39, L41), and the PR body's "still run on push / merge_group" line plus its Verification section. Please grep the claim across all surfaces and bring them in line with the workflow.
Non-blocking
- The merge queue now has no native Linux GPU lane (only Windows Strix and WSL2) since
E2E_MERGE_QUEUEwas removed from the Strix Ubuntu job. Worth stating explicitly in the docs and PR body. - The nightly Strix job now runs on the pool, where Docker may be unreachable, so the AIM serve canaries (serve-26/27) could skip silently. It has not been exercised end to end; a dispatch run before merge would confirm it fires.
- A few comment lines are left unwrapped.
CI: the failures seen at the earlier head were apt DNS errors in "Install native build deps" (runner network, not this change). Checks need a rerun at 5abecbc.
|
Addressed in 3c2bb61 (signed, with sign-off):
|
juhovainio
left a comment
There was a problem hiding this comment.
The change does what it says: both Strix Ubuntu and W7900 are gated off pull_request and merge_group, the gate test pins the whole if: string, and the nightly Strix job is moved onto the pool labels. I found nothing blocking in the workflow logic. Requesting changes for the stale claims below, plus a few questions.
Stale claims left behind (outside the diff, so no inline anchor)
tests/e2e-cucumber/src/expectation.rs, themerge_queuefield doc (~line 177), still says the full serve matrix runs "on every lane but MI300X". After this PR the merge queue also skips Strix Ubuntu and W7900, so only the Windows Strix and WSL2 lanes run@merge-queueserves.tests/e2e-cucumber/features/model_serving.feature(~line 364) still describes Strix Ubuntu as a per-PR job.
Questions / smaller points
- Scope: the title only mentions the per-PR scope, but the merge queue also loses its native Linux GPU lane and vLLM entirely (only Windows Strix and WSL2 remain there). The body discloses this; is the title meant to cover it?
- The moved
e2e-gpu-nightly-strixjob isn't exercised end to end. Can you run aworkflow_dispatchof the nightly before or after merge? Docker availability on the pool host for the AIM serve scenarios (serve-26/27) is still unconfirmed. - The new
nightly_strix_ubuntu_job_holds_no_static_host_pathstest has no stated mutation check. Could you note that it goes red when a static-host path is reintroduced? - Two reworded lines wrap badly (inline).
CI at 3c2bb61
- The Linux jobs (coverage, affected-crates tests, build-and-test, E2E) all failed in "Install native build deps" with an apt DNS failure on azure.archive.ubuntu.com. That looks like infra; please re-run. Until then
cargo test -p xtask workflow_contracthas no CI confirmation on this head. - Windows debug:
therock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaultsfailed. This PR touches no production Rust, so it is likely flaky, but worth a re-run.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 3c2bb61a84e0
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (9 files: 2 workflow YAML, 1 Rust contract-test file, 1 shell script, 2 docs files, 1 README, 1 test-support Rust file).
No ticket named as closed by this PR; the description cites several historical ticket IDs (ROCMAI-125/429/430, ROCMAI-120/EAI-8761) as background for pre-existing design choices this PR builds on, not as something this PR itself closes.
Blocking
- Required CI checks are failing at this exact head —
build-and-test,build-and-test (ubuntu-26.04),Test (affected crates)(both variants),Windows build and test (debug),windows-build-and-test,Coverage (rocm-dash crates, ratcheted),E2E tests(both variants), all from workflow run43042579975
Verified by reading the job logs directly, not assumed. Two distinct causes:- The Linux jobs (
build-and-test,Test (affected crates), and apparentlyCoverage) fail at asudo apt-get install -y pkg-config libcap-devdependency-install step withTemporary failure resolving 'azure.archive.ubuntu.com'— a transient package-mirror DNS failure on the hosted runner, before any step touches this PR's changed files. Windows build and test (debug)fails a genuine unit test,therock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaults, inapps/rocm/src/therock/python/config_props.rs— a file this PR's diff never touches (the diff is CI workflow YAML, docs, a shell script, andxtask/tests/e2e-cucumberRust only).
Neither cause traces to anything in this diff, but the rule is objective: a failing check blocks merge regardless of root cause, and the PR's ownmergeable_stateis currentlyblocked.
Confidence 100 (confirmed by direct log inspection) · mechanical · Fix: rerun once the runner network/mirror is healthy; separately confirm the Windowsconfig_propsfailure is pre-existing/flaky (e.g. viagit logon that file or a local rerun) since nothing in this diff could have caused it.
- The Linux jobs (
Non-blocking
-
An incomplete "per-PR" terminology sweep leaves stale claims in four places this change touches, after multiple rounds already fixed the same class elsewhere —
docs/ci-hardware-testing.md:210,xtask/src/workflow_contract.rs:1609(doc-comment +per_pr_job/per_prnaming onnightly_devlab_dispatch_lanes_share_the_per_pr_pool_labels, which this PR extends to covere2e-gpu-strix-ubuntu/e2e-gpu-nightly-strix),xtask/src/e2e_report.rs:276,325,329(testthe_nightly_lanes_publish_the_same_platforms_as_the_per_pr_lanesand itsper_pr-named locals/panic text),tests/e2e-cucumber/src/expectation.rs:177(merge_queuefield doc: "the full serve matrix (on every lane but MI300X)")
docs/ci-hardware-testing.md:210still reads "e2e-selfhosted.yml'se2e-reportcovers the per-PR GPU platforms" — eight lines after this same PR's own new text says two of those five platforms no longer run onpull_request. Theworkflow_contract.rsande2e_report.rsinstances still labele2e-selfhosted.yml's lane set collectively as "per-PR" in test names/doc-comments/panic messages, even though this PR makes that false for two of them; none of these are functional bugs (the underlying assertion logic is correct regardless of the label), but the wording actively misleads. Theexpectation.rs:177field doc says the merge queue runs the full serve matrix "on every lane but MI300X" — after this PR it also excludes Strix Ubuntu and W7900, so only the Windows Strix/WSL2 lanes remain. This is the same defect class this PR's own commit history already fixed in at least 6 other spots (nightly.yml's job comment,docs/testing.md,tests/e2e-cucumber/README.md, an earliere2e_report.rsdocstring, etc.) — a new instance keeps surfacing each round.
Confidence 85 · mechanical · Fix: same wording pattern as the already-fixed siblings — name which lanes still apply and which no longer do. -
Stale TMPDIR-redirect comment in
scripts/reclaim-gpu.sh, sibling of an already-fixed paragraph in the same file —scripts/reclaim-gpu.sh:1395-1397
Still present tense: "...the nightly Strix lane redirects TMPDIR to the runner's home. A bare/tmpglob removed nothing at all there, so the guard was dead on the one lane that redirects." This PR removed that lane's onlyTMPDIRoverride (confirmed innightly.yml: the job now runs on the DevLab Dispatch pool with noHOME/TMPDIRoverrides), so no lane redirects TMPDIR anymore — both the "redirects" and "was dead" claims are now false. A near-identical paragraph earlier in the same file (lines 48-53) was correctly updated by this PR to past tense ("...did, to$HOME/actions-runner/tmp, before it moved to the pool"); this second copy was missed.
Confidence 85 · mechanical · Fix: update to past tense to match lines 48-53. -
Garbled sentence introduced by this PR's own edit —
tests/e2e-cucumber/src/expectation.rs:1198-1200
"may each wait out a 1200s readiness cap. Run on every PR on the Strix Ubuntu pool (before that lane left the PR path) they pushed the job (~25 min otherwise) into its 45 min cap on every run." The inserted parenthetical breaks the clause's grammar; it no longer parses. The equivalent fact is expressed cleanly indocs/testing.md:1244-1246("...which on the Strix Ubuntu job when it still ran per PR (about 25 minutes without them, 45 minute cap) used up the whole budget."). Comment only, no functional effect.
Confidence 80 · mechanical · Fix: rephrase along the lines of thedocs/testing.mdversion.
Decisions for the author
- The same "per-PR"/"every lane but MI300X" staleness has now recurred across at least five review rounds in different files each time — non-blocking-improvement
Each round's manual grep sweep catches the instances it looks for and misses others (see Non-blocking above). A contract test that greps for the literal phrase "per-PR" or "every lane but MI300X" anywhere in the repo neare2e-gpu-strix-ubuntu/e2e-gpu-w7900's job names, outside an explicit allow-list, would catch future drift mechanically instead of relying on the next person/round to notice.
Positive signals
- The lane-selection choice is grounded in a precise, dated, two-round measurement table (998 then 944 runs; pass/fail/timeout/queue-wait per lane), including a direct side-by-side "could Strix Ubuntu replace MI300X" analysis — unusually rigorous evidence for a CI-scoping change.
- The gate-pinning test (
mi300x_lane_is_skipped_in_the_merge_queue) asserts the entireif:string rather than a substring, and both it andnightly_strix_ubuntu_job_holds_no_static_host_pathswere traced by hand against the literal YAML: each would fail if the production change were reverted, and each fails under a single-clause mutation (an exclusion moved to the wrong branch, a banned env var reintroduced) — the dual verdict a new test should carry, confirmed rather than assumed. e2e-report's skip-tolerance ([ "$result" = skipped ] && continuebefore the missing-artifact check) was read directly and correctly handles both newly-skipped-on-pull_requestlanes without producing a false "missing artifact" error.
Deployment notes
- Two previously-per-PR self-hosted lanes (
e2e-gpu-strix-ubuntu,e2e-gpu-w7900) stop running onpull_requestandmerge_group, but keep running on push/dispatch/nightly. Neither check name is in the required-status-check list, so this doesn't change what blocks a merge — but the merge queue now has no native-Linux GPU lane at all (MI300X was already excluded); only the Windows Strix/WSL2 lanes run there. - The repointed
e2e-gpu-nightly-strixjob now runs on a shared ephemeral pool instead of a static host and iscontinue-on-error: true. The PR's own round-2 verification found the pool's AIM-restart canary (serve-27) already fails there on "Local server stop not confirmed" (both channel legs) — pre-existing and unrelated per the author, but it will now surface in the nightly report where the old, permanently-queued static host never reached it.
What this covered
Read all 9 changed files completely via two independent fan-out workers grouped by coupling (the two workflow YAMLs + the Rust contract-test file; the docs/README/script/support-test files), each also running repo-wide greps for every changed identifier/claim. I independently re-verified the key if: gate strings, env-var removals, and runs-on labels myself against the raw YAML/Rust, read the full PR diff directly, and read CI job logs directly for every failing check rather than assuming its cause. Commits reviewed: 1530f6b10b530805a89253a582f104a80bc7d41d (merge-base with origin/main) through 3c2bb61a84e0abce6615651be4e8b1c6e45af956 (8 commits on the branch).
Did not run: cargo test -p xtask workflow_contract directly in this environment (no local Rust toolchain invoked) — the new/changed assertions were hand-traced against the literal YAML/Rust text instead, cross-checked by two independent workers, consistent with CI's own report of that suite passing before the unrelated apt/DNS failures began blocking the run. No separate git log -p/git blame history pass was run beyond git log --oneline to confirm commit structure (all 8 commit messages are descriptive; no "fix"/"wip" noise).
Not reconciled against any discussion thread or prior review on this PR — this was a blind pass. Several earlier automated reviews from this same account exist on this PR at older, superseded commits, and at least one human review now also exists at this exact head; none were read before these findings were reached.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
|
Round 6, head 81c0845:
|
juhovainio
left a comment
There was a problem hiding this comment.
Looks good to me. The PR path now runs MI300X + Windows Strix + WSL2, Strix Ubuntu/W7900 skip pull_request and merge_group, and the nightly Strix lane moves to the pool labels. The docs, README, feature comment and workflow_contract pins all agree with the workflow, and I found no stale per-PR Strix Ubuntu/W7900 claims left in the tree. No leaks in the diff.
Two tradeoffs the PR already documents, noting them only so they are a conscious choice: the merge queue no longer has any native Linux GPU lane, so heavy @merge-queue serves never run on a vLLM-eligible host before merge (Linux-only regressions surface on push to main); and no lane exercises the nightly package channel on a PR any more.
Non-blocking: the nightly Strix lane now relies on the same pool whose per-PR twin hit its cap in roughly 19% of jobs, and Docker on pool hosts is unconfirmed. A workflow_dispatch run of that lane would be worth doing before relying on it (inline comment below). There is also a small wrap nit in the comment around e2e-selfhosted.yml:108-110 ("...skip\n merge_group) and its\n required checks"), cosmetic only.
CI: the three red checks (Coverage rocm-dash crates, build-and-test, build-and-test ubuntu-26.04) all fail at apt-get with DNS resolution errors for azure.archive.ubuntu.com, so they look like infra and need a rerun. This review is a diff read; it does not replace those going green.
| # the static `native` host, which is offline: every nightly sat queued until | ||
| # the 12h run cap and the lane never ran. Pool hosts are cold, | ||
| # so there are no host-specific HOME or cache-dir overrides. | ||
| runs-on: [self-hosted, linux, devlab-dispatch, strix-halo] |
There was a problem hiding this comment.
The nightly Strix lane now runs on the same pool whose per-PR twin hangs to its cap in about 19% of jobs, and Docker on pool hosts is still unconfirmed (the step only warns). Please do a workflow_dispatch run of this lane before relying on it, so a hang does not show up as another silent nightly.
There was a problem hiding this comment.
I am not dispatching nightly.yml from this branch: it builds and publishes nightly release assets, so a dispatch from a PR branch would touch the shared release tag. What I can show is run 43025895270 of e2e-selfhosted.yml on the same pool (strix-ubuntu, include_nightly=true): Docker was reachable, serve-26 passed on both channel legs, serve-27 fails at stopping the AIM service (no xfail row yet), and nothing hung. The hang risk you name (about 19% of the per-PR twin's jobs) is bounded by the job's timeout-minutes: 120, and the job is continue-on-error, so a hang cannot block the nightly. The first scheduled nightly after merge is the end-to-end check, and I will workflow_dispatch nightly on main right after merge if you want it sooner. Leaving this thread open until that run exists.
A PR now starts the MI300X lane (the only lane where vLLM is the effective engine) and the Strix Halo Windows pool lanes (native + WSL2). The Strix Halo Ubuntu and W7900 lanes skip pull_request but keep push/merge_group/dispatch. nightly.yml's Strix Halo Linux lane targeted the offline static `native` host and never ran; point it at the DevLab Dispatch pool the per-PR lane uses and pin that in the workflow contract test. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Re-review at 0b2d5dc. The workflow if: gates, README, docs and comments agree on the new per-PR/merge-queue lane split, and I found no blocking issues. Requesting changes only for stale claims this PR makes untrue, which this repo expects fixed in the same change.
To fix
nightly.yml,e2e-gpu-nightly-strixheader comment (inline): still reads as if a pool-based lane might be needed.docs/testing.md(inline): still calls Docker on the nightly Strix host "unconfirmed".e2e-selfhosted.yml~L475 (outside the diff): still says "per-PR 45min budget" for a job that no longer runs per PR.- PR body: say why no Gherkin scenario is needed (CI-only change), per the contribution guidelines.
For the author to decide / confirm
- The merge queue now has no vLLM-capable or native Linux GPU lane, so heavy
@merge-queueserves run only on Strix Windows/WSL2. Is that coverage loss intended? - The re-pooled nightly Strix lane is not proven end to end (
serve-27already fails on the pool); it iscontinue-on-error, so please note the expected state in the PR. - The new nightly guard test is textual and rejects
${HOME}forms; fine if intentional. - I did not verify the "e2e-report tolerates skipped lanes" claim; please confirm it.
CI: completed checks are green; Strix Ubuntu/W7900 skipped by design; MI300X queued (runner pool issue).
The nightly Strix lane is on the pool where Docker was reached (serve-26 passed on both legs); say so, and note serve-27's stop failure. The include_nightly budget comment no longer calls the push-run budget per-PR. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Round 7, head bfde201 (answers the review at 0b2d5dc): fixed the nightly.yml header comment, the docs/testing.md caveat and the e2e-selfhosted.yml 'per-PR 45min budget' wording (L475); the PR body now says why no Gherkin scenario is needed and records the expected state of the nightly Strix lane. Confirmed: e2e-report tolerates skipped lanes (its missing-report check skips any 'skipped' needs result; run 43049162834 had Strix Ubuntu and W7900 skipped and the report job green). The merge-queue coverage loss is intended, per the engineer: MI300X stays the per-PR Linux lane and is not added to the queue (queue wait about 12 h against its timeout). The ${HOME rejection in the nightly guard is intentional. Re-requesting review. |
juhovainio
left a comment
There was a problem hiding this comment.
I went through the whole diff. I found no correctness defects in the workflow gating, but a couple of things should be fixed before merge.
To fix
model_serving.featurestill says it is unconfirmed whether the nightly Strix Halo host has Docker, while thedocs/testing.mdandnightly.ymlchanges in this PR say Docker is reachable on the pool. The comment should match (inline).- serve-27 is documented as failing ("STOP FAILED rc=1") but has no expectation/xfail row, so every nightly will report an unexpected failure. Please add the row, or link a tracking issue, in this PR (inline).
Non-blocking
- Dated measurements ("98 of 281", "35%", a specific run id) are copied into several durable docs and comments. The rationale holds without the numbers, and they will go stale.
- Cosmetic: the comment reflow in
e2e-selfhosted.ymlleaves a dangling "merge_group) and its" line.
Questions for the author
- The merge queue now has no native Linux GPU lane, and the heavy
@merge-queueserves only run on Windows/WSL2 (lemonade-only). Is it intended that vLLM heavy serves never run before merge, beyond the serve-05 canary on PRs? - ROCMAI-125 recommends keeping the strix-ubuntu nightly-channel leg per PR; this PR drops it. The divergence is documented, but the ticket text should probably be updated.
- The
e2e-gpu-nightly-strixpool move has not been run end to end, so the first nightly is its first real run.
Checked and fine: the if: gates match what workflow_contract.rs pins, the nightly label pairing is correct, the "all five lanes" count is right, and I saw no internal-name leaks. This is a diff-only read; CI results are not part of it.
…7's stop failure The model_serving.feature AIM comment still called Docker on the nightly Strix Halo host unconfirmed; dispatch run 43025895270 reached it (serve-26 passed on both legs) and serve-27 fails at stopping the service. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
No blocking defects found; requesting changes for the items below, which I'd like settled before merge.
Should fix
- A run ID and a specific failure string ("STOP FAILED rc=1", "no xfail row yet") are pinned into three durable surfaces:
nightly.yml,docs/testing.mdandmodel_serving.feature. They go false the moment serve-27 gets an xfail row or is fixed, and the three copies drift independently. I checked the claim against the run log and it is true today. Keep the evidence in the PR body or a ticket and have the docs reference a tracking ID. - Re-pooling the nightly Strix lane knowingly makes it red every night on serve-27, which has no xfail row and no ticket. It can't block the nightly or the wiki publish, but it is unaccounted-for noise. Please add an xfail row with a tracking ID in this PR, or name a follow-up ticket.
- "One Windows GPU node" is imprecise: a PR runs two Windows-pool jobs (
e2e-gpu-strix-windowsande2e-wsl) on separate ephemeral runners. "One Windows GPU pool" is accurate. The wording appears in the title, thee2e-selfhosted.ymlheader,docs/ci-hardware-testing.mdand the cucumber README.
Nit
- The
closedpull_request note ine2e-selfhosted.yml(around lines 43-45) still uses W7900 as its "busy runner" example, but W7900 no longer starts on pull_request.
For you to confirm (tradeoffs, disclosed already)
- Native-Linux Strix and W7900 are no longer exercised before merge, and the merge queue has no native Linux GPU lane.
- The only pre-merge vLLM serve is the per-PR canary serve-05 on MI300X, whose queue wait is often long.
- No PR lane exercises the nightly channel.
All merge_group/pull_request claims across docs, README, workflow and xtask comments match the workflows, and the whole-if: contract-test pin looks solid. CI had no failures when I looked; several jobs were still pending.
…x Halo Ubuntu Its stop step fails with 'Local server stop not confirmed' (tracked in rocm-cli#1058), so the nightly lane reports it as xfail instead of an unexpected regression. Point the nightly.yml, docs and feature comments at it. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…lure text; say Windows pool The serve-27 evidence lives in the tracking issue, so nightly.yml, docs/testing.md, model_serving.feature and expectations.toml no longer quote a run ID or the failure string. A PR runs two Windows-pool jobs (Windows and WSL2), so say one Windows GPU pool, not node. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
I went through the full diff and found no blocking defects. The lane if: gates, the E2E_MERGE_QUEUE removal and the nightly Strix re-pooling are consistent with each other, and the contract tests pin them well. Two small fixes before merge:
- The new
serve-aim-lifecycle-parityexpectation row is scoped more widely than its comment says (inline comment below). docs/ci-hardware-testing.mdstill carries two claims this PR makes untrue, on lines the diff doesn't touch: around L492, "persistent across runs on the long-livednative-labeled hosts" (this PR removes the last lane that targetsnative, and the new text elsewhere says nothing targets one), and around L252, "e2e-selfhosted.yml'se2e-reportcovers the per-PR GPU platforms" (that set now includes lanes that don't run per PR). Please reword both in this PR.
Two tradeoffs I read as deliberate, but please confirm:
- The merge queue now has no native Linux GPU lane. The only Linux vLLM signal before landing is the per-PR MI300X canary, whose queue wait the PR itself measures at roughly 10-12 h median.
- The Strix Ubuntu hang class is moved to the nightly rather than fixed, and the re-pooled nightly has not been exercised end to end. The first scheduled nightly after merge is the real proof.
…i-hardware-testing claims The row matched os=linux, which WSL2 also reports on gfx1151, so a WSL host that gained Docker and passed the scenario would XPASS. Add is_wsl = false. Reword the e2e-report coverage line and the pre-warmed runtime paragraph, which still named per-PR platforms that no longer all run per PR and the static native hosts. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Round 9, head c91e228 (answers the review at b2f23df):
On the two tradeoffs, both are deliberate:
I added nothing that depends on the other self-hosted nightly lanes. Re-requesting review. |
juhovainio
left a comment
There was a problem hiding this comment.
I went through the full diff and found nothing that is wrong in the code or the workflow gates. Moving the pull-request lanes to one Linux GPU node plus the Windows pool matches what the description says, and the contract test pins the whole if: of the two skipped lanes rather than a substring. I'm requesting changes for a few things I'd like settled before this merges, none of them a defect in the diff itself:
- The nightly Strix Ubuntu lane has never run. Its
runs-onchanged to the pool labels and the static-hostHOME/CARGO_HOME/TMPDIR/PIP_CACHE_DIRoverrides were dropped. The test only proves no static paths remain, not that the job works on a cold pool host (cache setup, prewarm, GPU preflight). Please dispatchnightly.yml, ore2e-selfhostedwithinclude_nightlyonstrix-ubuntu, once and link the run here. - WSL2 and the merge queue are described inconsistently. The docs and the
expectation.rscomment say WSL2 runs in the merge queue, while the Triggers paragraph says the heavy serves run only on the Windows Strix Halo lanes. The contract test pins only thee2e-gpu-strix-windows/e2e-wslgate, not whethere2e-wslsetsE2E_MERGE_QUEUE. Please confirm which is true and reword the other. - Merge-queue coverage loss. With MI300X, Strix Ubuntu and W7900 all skipping
merge_group, no Linux lane and no vLLM runs before merge. This is documented and I understand it is deliberate; flagging it so the tradeoff is acknowledged by the maintainers. - The serve-27 xfail row (inline comment) is only evaluated on
@nightlylanes, so a stale row only surfaces there. - Docs drift risk. The Triggers paragraph in
docs/ci-hardware-testing.mdis now very long, and the "all five lanes" count is not pinned by any test. The skipped-lane rationale is also repeated in about five places.
CI is green on everything that has completed so far. Mock E2E, Windows build/test, affected-crate tests and coverage were still running when I looked.
| # the scenario is not an XPASS. A host without Docker skips the scenario before | ||
| # this row applies. | ||
| [["serve-aim-lifecycle-parity"]] | ||
| when = { os = "linux", is_wsl = false, therock_family = "gfx1151*", has_amd_gpu = true } |
There was a problem hiding this comment.
This row is only evaluated on @nightly lanes, so please confirm that a nightly Strix Ubuntu run on the pool reports serve-27 as XFAIL (not skipped) before merge. The reason text ("not confirmed") is also thin; the tracking issue holds the evidence, which is fine, but a one-line symptom here would help.
There was a problem hiding this comment.
Added a one-line symptom to the reason in ee855d7. To confirm XFAIL rather than skip I dispatched e2e-selfhosted.yml with platform=strix-ubuntu and include_nightly=true on this branch (https://github.com/ROCm/rocm-cli/actions/runs/43211119495); I will report what it shows for serve-aim-lifecycle-parity here and leave this thread open until then.
… WSL2 runs the merge-queue serves The Triggers paragraph said the heavy serves run only on the Windows Strix Halo lanes while other docs said WSL2 too. The workflow sets E2E_MERGE_QUEUE on both e2e-gpu-strix-windows and e2e-wsl, so reword the paragraph and pin both in the contract test. Add a one-line symptom to the serve-27 xfail reason. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Round 10, head ee855d7 (answers the review at c91e228):
Re-requesting review. |
juhovainio
left a comment
There was a problem hiding this comment.
No blocking findings. Two non-blocking points, one wording and one to check:
- The W7900 "never started" figure does not match the PR's own table (202 of 507 jobs, about 40%, vs "70%" in the body and "58-70%" in the workflow comment). The comment is durable, so worth fixing.
bug = "#1058"inexpectations.tomldiffers from the tracker-ID form the neighbouring rows use; please confirm the xfail report and stale-xfail tooling accept it.
Smaller notes:
- The re-pooled nightly Strix lane is not proven end to end (the PR says so), and the guard test is textual.
mi300x_lane_is_skipped_in_the_merge_queuenow pins more than its name says; a rename would help.- I did not check other "per-PR Strix" mentions in docs outside the diff.
Tradeoffs the author has presumably weighed: the merge queue ends with no native Linux GPU lane and no vLLM lane, no PR lane exercises the nightly package channel, and Windows Strix + WSL2 are the only runners of the heavy @merge-queue serves. The docs state all of this plainly.
CI at review time: 21 pass, 12 pending, 0 fail.
| fail-fast: false | ||
| matrix: | ||
| sdk_version: ${{ fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix) }} | ||
| # Not on pull_request or merge_group: one runner on borrowed quota, 58-70% of |
There was a problem hiding this comment.
"58-70% of its PR jobs never started" does not match the PR table (W7900: 202 of 507 jobs never started, about 40%); the PR body also says 70%. Please correct the figure here or state the denominator.
There was a problem hiding this comment.
Agreed, the 58-70% figure does not match the PR table. I will correct it in the workflow comment and the docs in a small follow-up PR from main once this merges, rather than re-spin the queue entry.
There was a problem hiding this comment.
| # this row applies. | ||
| [["serve-aim-lifecycle-parity"]] | ||
| when = { os = "linux", is_wsl = false, therock_family = "gfx1151*", has_amd_gpu = true } | ||
| bug = "#1058" |
There was a problem hiding this comment.
Other rows use a tracker ID in bug (e.g. ROCMAI-900); please confirm #1058 is an accepted form wherever bug is parsed or rendered (xfail report, stale-xfail hint).
There was a problem hiding this comment.
Fair point: the other rows use the tracker-ID form, and #1058 is a GitHub issue. I will switch the row to the convention in the same follow-up PR after this merges.
The R9700, MI350P and W7900 nightly jobs have sat queued for hours and been cancelled since at least 11-09. Remove them from nightly.yml and from the consolidated report that waited on them. These lanes now run on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu, Windows, WSL2, all on the Strix pool since #1018) are unchanged. Update the workflow contract tests and the docs to match. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
What
A pull request now runs self-hosted GPU E2E on exactly one Linux node and one Windows node:
E2E tests (MI300X, …)(e2e-gpu)mi300x-0/1)E2E tests (Strix Halo, Windows, …)+E2E tests (Strix Halo, WSL2, …)Off the PR path and out of the merge queue (they still run on push to
mainandrelease/**, on dispatch, and nightly):E2E tests (Strix Halo, Ubuntu, release|nightly, …)(e2e-gpu-strix-ubuntu)E2E tests (W7900, …)(e2e-gpu-w7900)Mechanically: those two jobs get
github.event_name != 'pull_request'in theirif:.e2e-reportalready tolerates skipped lanes (its missing-artifact check skipsskippedneeds).Nightly gap found and fixed.
e2e-gpu-nightly-strixtargeted[self-hosted, linux, strix-halo, native], a static host that is offline. Every scheduled nightly (11-03 … 11-10, all cancelled) had that job queued until the 12 h cap, so Strix Linux had no scheduled coverage at all. It now uses the samedevlab-dispatchpool labels as the per-PR lane and drops the static-hostHOME/CARGO_HOME/TMPDIRoverrides;workflow_contract.rspins the labels to the per-PR lane like the Windows pair. Not exercised end to end: the nightly job only runs on schedule/dispatch; its body mirrors the per-PR job that runs on that pool today.Measurements
e2e-selfhosted.yml,pull_requestruns, 2026-10-27 → 2026-11-10, 998 runs (every job's final attempt). "Infra" = failed inSet up job/Reclaim GPUor no steps (runner lost/startup). "Test" = failed in the E2E step. "Timeout" = cancelled after running ≥40 min. "Never started" = cancelled still queued (superseded by a newer push). Strix Ubuntu counts two jobs per run (release + nightly legs).Last lane to finish (runs with ≥1 finished lane, n=668): W7900 557, MI300X 78, Strix Ubuntu 22, Strix Windows 11, WSL2 0. W7900 is the tail of the PR run most of the time, which is the "blocks PR development" symptom.
Runner capacity (repo runners list, 2026-11-10): Strix pool 6 Linux + 5 Windows ephemeral;
mi300x2;w79001;mi350p,r9700, staticstrix-halo-ubuntu-2offline. Per-runner MI300X:mi300x-027% fail,mi300x-110% fail.Why these nodes
*-dcgpu, gfx906/908/90a), which among PR lanes is MI300X alone (docs/ci-hardware-testing.md; Strixgfx1151is Lemonade-only). W7900 has one runner and 70% of its jobs never started. Strix Ubuntu has a pool, but 38% pass: 19% of its jobs hang to the 45 min cap (success p90 is 35 min, so it is a hang, not slowness) and it skips every vLLM scenario. Keeping MI300X keeps vLLM on PRs. Its weakness is queue wait (median ~10 h, p90 ~22 h on 2 runners), which this PR does not fix.Coverage: suites leaving PR gating
All still run on MI300X (Linux) or Strix Halo Windows/WSL2 on PRs, so vLLM, Lemonade and llama.cpp (via Lemonade) keep a PR lane. What leaves PR gating:
e2e-gpu-nightly-strix(see the fix above).e2e-gpu-nightly-w7900(the 11-09 nightly ran it green).Nothing else changes. Not touched: R9700 and MI350P (already nightly-only, ROCMAI-125; both runner sets are currently offline, so those nightly lanes also sit queued, outside this PR).
Required checks
No change needed: none of the self-hosted lane names is in the
mainrequired-checks list (E2E tests,E2E consolidated reportare the GitHub-hosted mock lane / report inci.yml). Verified against the branch-protection API today.Overlap with open drafts
#999, #1009 and #1011 do not touch the same workflows: they edit
scripts/run-e2e-lane.sh,queue-run.ymlanddocs/nightly-on-the-queue.md. All three, like this PR, editxtask/src/workflow_contract.rs, in different tests, so expect only trivial textual conflicts there.Verification
cargo test -p xtask workflow_contractpasses (146 tests). Updatedmi300x_lane_is_skipped_in_the_merge_queue(both lanes now skippull_requestandmerge_group, and the wholeif:of each is pinned; see rounds 3 and 4) and extended the pool-label pin. Data caveat: the Actions runs API stops at 1000 results, so the window is the latest 998 runs (14 days), not 21.Review round 1 (juhovainio)
mi300x_lane_is_skipped_in_the_merge_queue): it now compares the entireif:ofe2e-gpu-strix-ubuntuande2e-gpu-w7900to the expected text, so thepull_requestexclusion must sit in the non-dispatch branch and nothing else may change. I broke the gate and saw it go red: (A) added&& github.event_name != 'push'to the W7900 gate, which the old substring assertion would have passed; (B) moved thepull_requestexclusion into the dispatch branch of the Strix Ubuntu gate. Both failed the test; restored, 146 pass.nightly_strix_ubuntu_job_holds_no_static_host_paths: noactions-runnerpath and no read ofCARGO_HOME/RUSTUP_HOME/TMPDIR/PIP_CACHE_DIRthe job does not set. Broke it by reading$TMPDIRin the run step: red. The onlyHOMEreads left are$HOME/.cargo/bin(Rust bootstrap) andHF_HOME, both valid on a pool host.tests/e2e-cucumber/README.md(note under the job table),docs/testing.md(the AIM-cost paragraph), the channel-matrix paragraph and the Triggers paragraph indocs/ci-hardware-testing.md(reflowed), and aTMPDIRcomment inscripts/reclaim-gpu.sh. The Platforms and README tables are pinned by a contract test, so I added prose under them rather than a column.Review round 2 (juhovainio, 87b214e → 2f9c9c6)
e2e-reportskips lanes whoseneedsresult isskipped(checked on the self-hosted workflow; run 42932854129 succeeded with both lanes skipped).serve-26/27; Docker is reachable,serve-26passed on both channel legs.serve-27failed at "Local server stop not confirmed" on both legs (no expectation row; unrelated to the lane change, but the nightly lane will now show it).$HOMEread other than$HOME/.cargo/bin; remaining stale per-PR wording fixed; two prose lines reflowed.Review round 3 (029c48c)
merge_group(engineer's decision). It still runs on push tomainandrelease/**, dispatch and nightly. The contract test now pins the whole gate of each lane (Strix Ubuntu:pull_requestexcluded; W7900:pull_requestandmerge_groupexcluded) and that the W7900 job has noE2E_MERGE_QUEUE. Broke the gate (removed themerge_groupexclusion): red; restored.nightly.ymlcomment that callede2e-gpu-strix-ubuntua per-PR lane, plus two comment-only leftovers (xtask/src/e2e_report.rs,tests/e2e-cucumber/src/expectation.rs).the_nightly_lanes_publish_the_same_platforms_as_the_per_pr_lanes) is keyed on the artifact names found in each workflow's text, not onif:or events, so skipping lanes onpull_requestdoes not affect it.HF_TMPDIRis notTMPDIR).Strix Ubuntu vs MI300X, last 7 days (2026-11-04 14:22Z → 11-11 14:22Z, 944 runs of
e2e-selfhosted.yml)Queue wait = created → started, over jobs that got a runner. "No runner" = cancelled or expired without ever starting. Run time is over passed or failed jobs. Failure rate = failed / (passed + failed). Strix Ubuntu counts two jobs (release + nightly leg) per run.
Capacity (runner list, 2026-11-11): MI300X 2 runners (both busy); Strix Linux pool 5 registered, 4 online (3 busy), 1 offline (a fresh ephemeral runner per job: 1212 distinct runner names served the 1229 jobs this week); W7900 1 runner.
Could Strix Ubuntu replace MI300X as the per-PR Linux lane?
mainthe same day (see Follow-up 1). The lane also hits its 45 min cap: 35% of merge-queue jobs were cancelled at the cap, 9% overall, which would turn a PR verdict into a timeout.serve-vllm-inference,serve-vllm-default-on-instinct,serve-vllm-low-vram-oom-guidance,runtime-lifecycle-activate-restart-services-succeeds,therock-next-09-…vllm-rocm10x-discovery-pins) skip on Strix;serve-rocr-reindexed-gpu-index-rejected(needs more than one GPU) skips on Strix; the whole vLLM runtime path and Instinct (*-dcgpu, gfx942, 8-GPU) hardware paths. Gained: gfx1151 native-Linux APU paths. Lemonade/llama.cpp scenarios and the diagnose/driver-install/examine bare-metal ones run on both.Review round 4 (5abecbc)
e2e-gpu-strix-ubuntudropped frommerge_group(engineer's decision; 35% of its merge-queue jobs, 98 of 281 over 7 days, hit the 45 min cap; non-required, the queue does not wait for it). It still runs on push (main,release/**), dispatch and nightly. The merge queue now runs only the Windows Strix Halo lanes (MI300X and W7900 were already out).if:gate and that neither job hasE2E_MERGE_QUEUE; mutation-checked three ways (exclusion removed,E2E_MERGE_QUEUErestored, exclusion moved to the dispatch branch), all red, restored.merge_group): it runs only the Windows Strix and WSL2 lanes. Arelease/**push still runs all five lanes.Review round 5
merge_groupnow matches the workflow: this description (the "Off the PR path" line and the Verification section),docs/ci-hardware-testing.md(rows at L39 and L41, the Triggers paragraph), thenightly.ymlcomment one2e-gpu-nightly-strix, and the cucumber README. Grepped formerge_group/merge queueacross the workflows, docs, README and xtask comments.serve-26/27were already exercised on the pool by run 43025895270 (see round 2); nothing new was dispatched.Review round 6
merge_queuefield doc intests/e2e-cucumber/src/expectation.rs(only Windows Strix and WSL2 run@merge-queueserves), themodel_serving.featureAIM comment (Strix Ubuntu no longer a per-PR job),docs/testing.mdand thedocs/ci-hardware-testing.mdmerge_group line (WSL2 named). Two badly wrapped lines re-wrapped.nightly_strix_ubuntu_job_holds_no_static_host_paths: (A) changed the job's reclaim step tobash /home/ubuntu/actions-runner/_work/scripts/reclaim-gpu.sh-> red ("the pool host has no /home/ubuntu/actions-runner tree"); (B) changed it tobash $HOME/work/scripts/reclaim-gpu.sh-> red ("reads $HOME outside the Rust bootstrap"). Both restored;cargo test -p xtask workflow_contract146 pass.nightly.ymlfrom this branch, since it builds and publishes nightly release assets. Dispatch run 43025895270 ofe2e-selfhosted.ymlalready ran the same lane steps on the pool:serve-26passed on both channels (Docker reachable);serve-27fails at stopping the AIM service (STOP FAILED rc=1), now an xfail row, rocm-cli#1058. The first scheduled nightly after merge is the end-to-end check; I canworkflow_dispatchnightly on main after merge if wanted.Review round 7 (head bfde201, rebased onto main with #1032)
workflow_contracttests (146 -> 153 with ci: cancel a PR's queued self-hosted E2E runs when it is merged or closed #1032's), which I mutation-checked.@merge-queueserves run only on Strix Windows and WSL2. MI300X stays the per-PR Linux lane and is not added to the queue (queue wait about 12 h against the queue's timeout). The only vLLM serve before a change lands is the per-PR canary (serve-05).nightly.ymlrun (it publishes release assets, so it is not dispatched from a PR branch). The same lane steps ran on the pool ine2e-selfhosted.ymlrun 43025895270:serve-26passed on both legs,serve-27fails at stopping the AIM service.serve-27is declared an expected failure inexpectations.toml(Linux gfx1151, rocm-cli#1058), so the nightly reports xfail; the job iscontinue-on-errorand cannot block the nightly.${HOMEforms is intentional (a pool host provisions no$HOMEbeyond the Rust bootstrap).e2e-reporttolerates skipped lanes: its "every lane that ran uploaded its report" check skips anyneedsresult ofskipped, and E2E self-hosted run 43049162834 (a pull_request run with Strix Ubuntu and W7900 skipped) finished with the report job green.iftable now pins both PR-skipping lanes with their exclusions.Review round 8 (head b2f23df)
serve-27failure is kept here and in rocm-cli#1058, not in durable comments: run 43025895270 (e2e-selfhosted.yml,platform=strix-ubuntu,include_nightly=true) failedserve-aim-lifecycle-parityon both channel legs at the stop step (STOP FAILED (rc=1) ... Local server stop not confirmed), whileserve-aim-inferencepassed on both.nightly.yml,docs/testing.md,model_serving.featureandexpectations.tomlnow only reference E2E: AIM service stop is not confirmed on Strix Halo Ubuntu (serve-aim-lifecycle-parity, STOP FAILED rc=1) #1058.e2e-gpu-strix-windowsande2e-wsl) on separate runners, so the title, workflow header, cucumber README and docs say "one Windows GPU pool" (and "one Linux GPU node" for MI300X).