Repository navigation
ci: cancel a PR's queued self-hosted E2E runs when it is merged or closed - #1032
Conversation
…osed The E2E self-hosted workflow supersedes a PR's older run only when a newer push starts a run in the PR's concurrency group. A merged or closed PR starts none, so its queued MI300X/W7900 jobs stay in the queue and hold a runner for a head nobody will look at again (about half of the 18 queued MI300X jobs on 2026-11-11 were of this kind). List `closed` among the pull_request types. The run it starts joins the PR's group, so cancel-in-progress cancels the earlier run. `changes` skips on `closed`, which skips every lane, leaving only a no-op `pr-closed` job. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
e453be3 to
201e226
Compare
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 201e22692d2f
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (both changed files, read completely, plus the full surrounding e2e-selfhosted.yml workflow — concurrency group, every downstream job's needs/if gating, and the workflow's own file-header rationale — to judge the change in place).
No ticket named. The description's claims (list closed among pull_request types, skip changes on closed, add a no-op pr-closed job, and a contract test pinning all three) all match the diff; the cited GitHub issue/PR numbers are evidence for the problem statement, not a tracked goal this PR claims to close.
Blocking
None
Non-blocking
docs/ci-hardware-testing.md's "Triggers" section is not updated for the new closed-PR supersession path —docs/ci-hardware-testing.md:358-360
That section is an otherwise exhaustive account of when the self-hosted GPU jobs run and what cancels what ("A new push to a pull request cancels that PR's older run; push,merge_groupand manual runs each get a group of their own, so none is cancelled or replaced by another."). It does not mention that a PR'sclosedevent now also starts a (no-op) run that joins the same concurrency group and supersedes the PR's prior run. The jobs this doc describes are unaffected (they still never run onclosed), so nothing it says is actually wrong, but a reader relying on it for the full cancellation picture would miss this path.
Confidence 70 · architectural · Fix: add a sentence to the "Triggers" section (or a cross-reference to the newon.pull_requestcomment ine2e-selfhosted.yml) noting that closing/merging a PR also joins its group and cancels its queued/running self-hosted run.
Decisions for the author
None
Positive signals
- The fix reuses the workflow's existing concurrency-group/
cancel-in-progressmechanism (a trivial job that only has to start in the same group) instead of adding a new job withactions: writepermission to call the cancellation API directly — no new permissions, minimal blast radius. - The new contract test (
closing_a_pr_supersedes_its_self_hosted_runs) asserts the three sub-changes (thetypes:list, thechangesjob's skip condition, and thepr-closedjob) independently, so a regression in any one of them fails on its own rather than only as a side effect of another assertion.
Deployment notes
None — this is a GitHub Actions workflow/config change with no runtime or release impact.
What this covered
Both changed files in full (.github/workflows/e2e-selfhosted.yml, xtask/src/workflow_contract.rs), diffed e3ad2488d…201e22692d2f (PR base…head, fetched read-only via pull/1032/head), plus the whole surrounding workflow file (concurrency group, every job's needs/if chain down through e2e-report) and docs/ci-hardware-testing.md's trigger/supersession documentation. Traced by hand that on a closed pull_request event: changes is skipped (new if), every GPU lane's pre-existing needs.changes.result == 'success' gate therefore also skips, and e2e-report's pre-existing needs.changes.outputs.serve == 'true' gate evaluates false (a skipped job's output reads as empty), so the whole matrix and the report are skipped — matching the PR description. Confirmed via gh pr view --json statusCheckRollup that all 41 checks on the head commit are green; did not re-run cargo test -p xtask workflow_contract myself, reading the already-green CI result instead per the review method. The diff was small and tightly coupled (one workflow file and its own contract test, 52 lines total), so this was a single-pass review by one reviewer rather than a multi-worker fan-out, per the review method's size-based scaling.
Did not run: history/prior-changes passes — looked at git log/blame context implicitly while reading the file but found nothing germane beyond what's cited above; agent-instruction-file pass — no CLAUDE.md/AGENTS.md rules apply beyond the repository's own documented CI conventions, which the change follows.
Not reconciled against any discussion on the pull request (review threads, if any, were not read).
Note in the Triggers section that a PR's closed event starts a no-op run in the PR's concurrency group, and that a run whose job is still queued for a runner may not finish cancelling. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Thanks for the review of 201e226. The non-blocking note is addressed in 867fb95: the Triggers section of |
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for this. The skip chain looks right to me: changes is skipped on closed, every GPU lane gates on needs.changes.result == 'success', and e2e-report keys off the serve output. I'm requesting changes for the items below; none of them is a regression in the workflow itself.
- The contract test is plain
containsmatching and is not scoped to the job blocks, so it can pass on a comment or on the wrong job. Please scope thechangesskip and thepr-closedjob asserts to their job blocks, and confirm you reverted each of the three edits and saw the test go red. - The cancel behavior is not verified end to end. The PR body says jobs queued on a runner that never picks them up are not cancelled, so this only helps lanes that have capacity. Please say that in the workflow comment and the docs rather than claiming the queue is freed.
strategy.matrixusesfromJSON(needs.changes.outputs.sdk_version_matrix), which is an empty string whenchangesis skipped. I expect the jobifto short-circuit it, but the PR's own CI never produces aclosedevent, so this is untested. Please confirm it on the first merged PR after this lands, or note the check in the PR text.- The PR body should say in one sentence that no scenario is needed because this is CI plumbing.
- Wording nit in the docs: "frees the queue of dead work" reads stronger than the hedge in the next sentence.
Details are in the inline comments.
jussielo-amd
left a comment
There was a problem hiding this comment.
No blockers from me. The closed run lands in the same concurrency group as the PR's earlier runs (the group has no action component, and cancel-in-progress is true for pull_request), changes skips on closed, every lane gates on needs.changes.result == 'success', and pr-closed has no permissions and no checkout. This was a static review: nothing was built or run (no Rust toolchain here).
Non-blocking:
xtask/src/workflow_contract.rs:~7437-7449: the new test checkssh.contains(...)over the whole file, so thechangesskip line would still pass if it moved to another job, and nothing asserts that the lanes ore2e-reportskip onclosed.job_block(&sh, "changes")/job_block(&sh, "pr-closed")would tighten it.e2e-reportdoes not gate onneeds.changes.resultlike the lanes do; it skips becauseneeds.changes.outputs.serveis empty. It works, but the description ("every lane gates onneeds.changes.result == 'success'") and the test do not cover it, so a later edit to itsifcould quietly make it run onclosed.e2e-selfhosted.yml:~34-36,192-194say theclosedrun cancels the older run "and with it any GPU job still queued", while the file header and the new docs paragraph correctly say a job queued for an unavailable runner may not finish cancelling. Reword the comments to the docs' hedge.- Unverified: on
closed,changesis skipped, sofromJSON(needs.changes.outputs.sdk_version_matrix)in each lane'sstrategy.matrixsees an empty output. Skipped jobs should not evaluatestrategy, but this PR never exercises a closed run; closing a scratch PR once would confirm it (or use|| '["current"]').
🤖 by agenthub on AMD AgentHub
Read each assertion from one job's comment-free block, so prose or another job cannot satisfy it, pin the whole `if` of `changes` and `pr-closed` with the latter's runner and empty permissions, and assert that every other job (including e2e-report) skips when `changes` skips. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Reword the workflow comments and the Triggers paragraph to the same hedge: a job queued for a runner that is not picking it up may not finish cancelling, so the closed run frees runners only where the cancel completes. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
@juhovainio all five points are answered: 1 fixed (57227cc, with the mutation check), 2 and 5 fixed (c4ee13d), 4 added to the PR text. Point 3 is not fixed in code; the PR text names the check to make on the first closed PR after this lands, and that thread stays open. jussielo-amd's non-blocking notes ( |
juhovainio
left a comment
There was a problem hiding this comment.
Looks good to me. I found nothing blocking: the closed-PR job only runs on closed, permissions: {} is set, the lanes and e2e-report are all gated so they skip when changes is skipped, and the limitation for already-running W7900 jobs is disclosed in the workflow comment, the docs and the PR body. CI is green on this head.
I left a few non-blocking suggestions inline: a || '["current"]' fallback on the lane matrices so a skipped changes can never trip fromJSON evaluation, a stricter assertion in the contract test for the lane if, and two small tidy-ups (share the comment-stripping helper, re-wrap one long doc line). None of these need to hold up the merge.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head c4ee13df8a9d
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change. Implements the fix described in the PR's own "Why"/"What" sections (cancel a PR's queued self-hosted E2E runs when it is merged or closed); no tracker ticket named.
Blocking
- A closed/merged PR's run likely errors at job scheduling instead of cleanly skipping, because
strategy.matrixis resolved from an emptyfromJSON()input —.github/workflows/e2e-selfhosted.yml:224(e2e-gpu),:488(e2e-gpu-strix-ubuntu),:643(e2e-gpu-strix-windows),:991(e2e-wsl),:1599(e2e-gpu-w7900), gated by thechangesjob's new skip at:104and itssdk_version_matrixoutput at:119.
Each of the five GPU lanes hasstrategy.matrix.sdk_version: ${{ fromJSON(needs.changes.outputs.sdk_version_matrix) }}, and each lane's ownif:includesalways()(to override GitHub's default "skip if a needed job didn't succeed" behavior, so the lane's own, more specific condition governs instead). On aclosedPR event, the newif:onchanges(line 104) causeschangesto be skipped, leavingneeds.changes.outputs.sdk_version_matrixempty. GitHub Actions resolves a job'sstrategy.matrixas part of scheduling the job; the "skip without touching the matrix" shortcut applies only to a job's implicit defaultif: success()— once a job's ownif:explicitly includesalways()(as all five here do), that shortcut does not apply, and the matrix is resolved regardless of what theif:will ultimately evaluate to.fromJSON('')on an empty string throws a job-level "Error from function 'fromJSON': empty input" scheduling error rather than producing a clean skip — a failure mode reproducible in other reports of the samealways()+fromJSON(needs.*.outputs.*)+ skippable-upstream combination (and fixed upstream, for the equivalent Actions-compatible engine, by special-casingalways()/failure()/cancelled()so the matrix is still resolved rather than short-circuited). The PR's own description independently raises this exact risk under "Not verified end to end," expressing the hope that "the jobifshould short-circuit it" — the mechanism above is specifically why that hope likely does not hold oncealways()is present.
Since none of these lanes are required merge checks, this cannot block a merge, but it does mean the change's stated goal (a closed-PR run showing only the no-oppr-closedjob, "with every other job skipped and nostrategyevaluation error," per the PR body) is likely not met, and the new contract test cannot catch this because it only pattern-matches the YAML's text — it cannot execute the workflow and so cannot observe a scheduling-time error.
Confidence 80 · logic · Fix: give eachfromJSON(...)a fallback so it never sees an empty string, e.g.fromJSON(needs.changes.outputs.sdk_version_matrix || '["current"]')on all five lanes — the per-instanceif:already prevents the job from doing anything once instantiated, so a harmless placeholder is sufficient.
Non-blocking
-
Docs: inserted sentences separate a "therefore" from the premise it refers to —
docs/ci-hardware-testing.md:358-366.
The concluding sentence "The heavy@merge-queueserves therefore no longer run on MI300X…" draws on the paragraph's opening parenthetical about MI300X being skipped in the merge queue. The two new sentences about the closed-PR no-op run are inserted directly before it, so "therefore" now sits right after an unrelated sentence about queued-runner cancellation, reading briefly as a non sequitur until the reader reconnects it to the earlier premise.
Confidence 70 · mechanical · Fix: move the two new sentences to the end of the paragraph, after the "therefore" sentence, or start the MI300X conclusion with an explicit backward reference. -
New test re-derives job-id extraction instead of reusing the existing
job_idshelper —xtask/src/workflow_contract.rs(new code inclosing_a_pr_supersedes_its_self_hosted_runs, theidsextraction block) vs. the pre-existingjob_idsfunction atxtask/src/workflow_contract.rs:361-368.
The new test re-implements an equivalent indent/comment filter plus an alnum/-/_allow-list to extract job ids from thejobs:block, duplicating whatjob_ids(text)already does a few hundred lines earlier in the same file (verified the two approaches produce the identical job-id set against the real workflow at this commit). Callingjob_ids(&sh)directly would drop both the duplicated filter and the "extractor sanity check" assertion that exists only to hedge against the newly-written duplicate.
Confidence 85 · mechanical · Fix: call the existingjob_idshelper instead of re-deriving the same filter.
Decisions for the author
None.
Positive signals
- The new contract test derives its job list from the workflow file itself (via the existing
job_ids-equivalent extraction) rather than a hardcoded list, so a newly added lane is automatically required to carry the samechanges-skip guard. - The PR description is unusually candid about what is not verified (the exact
fromJSON/skip risk above) and proposes a concrete post-merge check, rather than overclaiming correctness. pr-closedusespermissions: {}and a GitHub-hosted runner with nouses:/checkout/secrets, avoiding any token or self-hosted-runner exposure for a job that exists purely to occupy a concurrency group.
Deployment notes
If the blocking finding above is correct, every PR merge or close will show a job-level "strategy" evaluation error on the (non-required) E2E self-hosted check rather than the clean single-no-op-job run described in the PR body. This is cosmetic — it does not affect mergeability — but is worth fixing before relying on the author's own planned "check the first real closed PR" verification, since the failure would otherwise appear on every subsequent PR close until patched.
What this covered
Full read of all three changed files (.github/workflows/e2e-selfhosted.yml, docs/ci-hardware-testing.md, xtask/src/workflow_contract.rs) against their pre-existing surrounding code (every job's if:/needs/strategy in the workflow, and the pre-existing helper functions the new test relies on or duplicates), base e3ad2488dea2a408a03e216cefb0c4f67b3c5a90…head c4ee13df8a9d4705a9cd00960ec4abbc37b4ce0c. Two independent fan-out workers each reviewed one coupled file group (workflow+docs; the new Rust test) blind, plus this pass's own direct reading and targeted platform-semantics research to verify the primary finding. All 40 CI checks on the head commit are green; no failing required check. One fan-out worker disclosed that it incidentally viewed the PR title/body despite being asked not to; its stated verdict rests on YAML tracing and general GitHub Actions semantics rather than that content, and the finding is independently corroborated here. Did not run: a git log/git blame history pass, and a prior-review-comments pass (no earlier review visible to reconcile against). Not reconciled against any PR discussion or comments — this review did not read them, per method.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Fall back to ["current"] when changes' matrix output is empty (changes is skipped on closed), so no lane's strategy evaluates fromJSON of an empty string. Require every lane's if to open with always() && needs.changes.result == 'success', share the comment-stripping helper, and re-wrap the docs paragraph. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Reply to nowycondro's review of c4ee13d: the blocking finding (empty |
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. The workflow change, the comment, the docs and the PR body agree, and the diff is clean of internal references. Nothing blocking; two small notes inline.
Not verified from the diff alone:
Test (affected crates)failed on an apt DNS error before reaching tests, soclosing_a_pr_supersedes_its_self_hosted_runshasn't been seen passing in CI yet. Worth a rerun before merge.- That the concurrency group /
cancel-in-progressactually covers theclosedevent, and thate2e-report'sifhas no arm beyond the asserted prefix that is true whenchangesis skipped. A live closed-PR run would confirm both.
The other red checks I looked at were infra flakes (apt DNS, crates.io download EOF) and need a rerun.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 30175ac19a5b
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change. No tracker ticket named; the cited GitHub PR numbers (#998, #959, #694, #818, #991, #997, #1004, #1006, #999, #856, #1018) are evidence for the problem statement, not a tracked goal this PR claims to close.
Blocking
-
The PR description's "Not verified end to end" section describes a risk the diff already fixes, without mentioning the fix — PR description vs.
.github/workflows/e2e-selfhosted.yml:224,488,643,991,1599.
The description says each lane'sstrategy.matrixreadsfromJSON(needs.changes.outputs.sdk_version_matrix), calls this "empty whenchangesis skipped," hopes "the jobifshould short-circuit it," and asks the reader to "check on the first PR merged or closed after this lands" for "nostrategyevaluation error." But the diff itself already adds a|| '["current"]'fallback to all fivefromJSON(...)calls, which deterministically resolves the empty-output case regardless of whether a skip short-circuits strategy evaluation — the description just never says so, and still frames the matter as an open, only-verifiable-live risk. A reviewer or future reader trusting the description would believe this is still unresolved when it is not.
Confidence 90 · mechanical · Fix: update the description to say the|| '["current"]'fallback was added and why it removes the uncertainty, and either drop the "check the first closed PR" ask or narrow it to something the fallback doesn't already cover (e.g. confirmingpr-closedis the only job that runs). -
The new contract test's lane/
e2e-reportif:checks usestarts_with, which a future appended|| …arm would defeat — exactly the gap the test's own comment says it guards against —xtask/src/workflow_contract.rs:7520-7534(assertions), vs. the real conditions at.github/workflows/e2e-selfhosted.yml:235-244(and the other 4 lanes) and:1749-1751(e2e-report).
Each lane's fullif:isalways() && needs.changes.result == 'success' && ... && (...). The test only checkscondition.starts_with("always() && needs.changes.result == 'success'")(same pattern fore2e-report'sserve-output prefix). GitHub Actions expressions bind&&tighter than||— precedence this same test file already relies on elsewhere — so a future edit appending|| true(or any always-true arm) after the condition's existing closing)would make the whole expression evaluatetrueunconditionally, defeating the skip-on-closedguarantee this PR introduces, while still satisfyingstarts_withuntouched. The test's own comment at:7528-7529("an||arm... would make it run whenchangesis skipped") claims exactly the protection astarts_with-only check does not fully deliver against an appended arm (only against one inserted before the asserted prefix).changesandpr-closed's ownif:are checked withassert_eq!just above this block, so the stronger form is already the test's own established pattern — it's just not applied here.
Confidence 70 · logic · Fix: assert the full condition withassert_eq!(matching the pattern already used forchanges/pr-closed), or additionally assert the condition's exact trailing characters so an appended arm can't slip past unexercised.
Non-blocking
-
A dangling "therefore" in the docs —
docs/ci-hardware-testing.md:358-366.
The closing sentence "The heavy@merge-queueserves therefore no longer run on MI300X…" draws on the paragraph's opening parenthetical about MI300X being skipped inmerge_group(a few lines earlier), not on the new closed-PR sentences inserted directly before it. Each new sentence reads fine on its own; the insertion point breaks the "therefore"'s backward reference, reading briefly as a non sequitur.
Confidence 70 · mechanical · Fix: move the two new sentences to the end of the paragraph (after the "therefore" sentence) or elsewhere in the Triggers section. -
New test re-derives job-id extraction instead of calling the file's own existing
job_idshelper —xtask/src/workflow_contract.rs:7498-7509vs.:361-368.
The newidsextraction re-implements the same indent + alnum/-/_filter overtop_level_block(&sh, "jobs")that the pre-existingjob_ids(text)helper already provides and that other tests in this file already share (e.g.self_hosted_e2e_jobs). Verified both produce the identical ordered id list against the real workflow at this commit.
Confidence 85 · mechanical · Fix: replace the block withlet ids = job_ids(&sh);and adjust the handful of comparisons that follow.
Decisions for the author
- The cancellation benefit is weakest for the case the PR's own "Why" cites as motivating it — tradeoff.
W7900 has a single, often-busy runner, and the new comments/docs honestly disclose that a job already queued there may not finish cancelling. The PR's own numbers (18 queued MI300X jobs, 11 dead) suggest MI300X and the Strix pool lanes are where this actually pays off, while W7900 — the hardware explicitly called out for the queued-cancel limitation — benefits least. Worth confirming that's the understood scope rather than an unstated gap.
Positive signals
- Reuses the pre-existing concurrency-group +
cancel-in-progressmechanism via a trivial job that only has to start in the right group, instead of adding a job withactions: writeto call the cancellation API directly — no new permissions, minimal blast radius (consistent with the cleanzizmorsecurity-scan result on this head). pr-closedis deliberately minimal —permissions: {}, GitHub-hosted runner, nouses:/needs:/secrets/checkout — so a job that now fires automatically on every PR close carries no meaningful attack surface.- Every GPU lane's and
e2e-report'sif:opens with thechanges-gate ahead of the rest of its condition, so the skip-on-closedbehavior doesn't depend on how the remainder of a long boolean expression is structured. - The new contract test derives its job-id list from the live workflow file rather than a hardcoded list, so a newly added GPU lane is automatically required to carry the same
changes-skip guard.
Deployment notes
None beyond what's already disclosed: this is a GitHub Actions workflow/doc/test change with no runtime or release impact. The one known limitation (a GPU job already queued on a busy/offline self-hosted runner may not finish cancelling) is disclosed in the workflow's own comments, the docs, and the PR description.
What this covered
Full review of the whole change: all three changed files read completely (.github/workflows/e2e-selfhosted.yml, 1845 lines; docs/ci-hardware-testing.md, 605 lines; xtask/src/workflow_contract.rs, 9985 lines, entirely #[cfg(test)]-gated), diffed e3ad2488dea2a408a03e216cefb0c4f67b3c5a90…30175ac19a5baa2240321bc9a37b55c51947993d, fetched read-only (no checkout). Two independent blind fan-out workers each covered one coupled file group (workflow+docs; the Rust contract-test file in full), plus this pass's own direct tracing of every job's if/needs/strategy chain and a check of the repo's other workflow files (ci.yml, nightly*.yml, queue-run.yml) for whether any other self-hosted, PR-triggered workflow needed the same fix — none does; e2e-selfhosted.yml is the only self-hosted workflow with a pull_request trigger.
CI: 8 checks report FAILURE on this head (build-and-test ×2, Test (affected crates) ×2, Coverage (rocm-dash crates, ratcheted), E2E tests ×2, remote control channel (containerised)). Read the actual job logs for several and confirmed each fails at an apt-get/DNS-resolution step ("Temporary failure resolving 'azure.archive.ubuntu.com'") before any build or test step runs — an environment flake unconnected to this change's content, not something it introduces. One consequence: the new contract test has not been observed passing in CI at this head; its correctness here rests on manual tracing against the real workflow file (by two independent workers and this pass), not an executed run — no cargo/rustc toolchain was available in this environment either.
Agent-instruction file AGENTS.md read in full; nothing in scope violates it (no new third-party uses: to pin, the changed Rust file is entirely test-gated so the 1,500-line production-file cap doesn't apply, no internal/proprietary references in the diff, and the PR's own "no scenario is needed — CI plumbing" line satisfies the one place it requires that explanation).
Did not run: a prior-review-comments reconciliation pass — reading GitHub review threads is out of scope for a blind pass under this method. History pass (git log on the three files) found nothing germane beyond context already incorporated above.
Not reconciled against any PR discussion or comments beyond the description and CI status read at the method's final step.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
|
Reply to nowycondro's review of 30175ac. First blocking finding (the description still framed the empty-matrix risk as open): fixed in the PR description. It now says the |
A lane's if was checked by prefix, so an `|| true` appended to it (`&&` binds tighter than `||`) would still pass while defeating the skip on closed. Pin every lane's and e2e-report's full condition with assert_eq!, and fail on a job with no pinned condition. Take the job ids from job_ids, and move the closed-PR sentences to the end of the Triggers paragraph so the 'therefore' sentence follows the premise it refers to. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Reply to nowycondro's review of 30175ac. Second blocking finding ( |
Read each lane's matrix as
fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix).
On a closed PR changes is skipped, so the placeholder applies; if changes
succeeded but stopped writing the output, fromJSON('') still fails instead of
quietly running current. Pin that expression per entry in the contract test,
and only for jobs pinned as matrix lanes, so a non-matrix helper job is not
forced to carry it.
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
The workflow change itself looks right to me, but this head doesn't build in CI, so I can't sign off yet.
Blocking: the new contract test in xtask/src/workflow_contract.rs declares const MATRIX_EXPR after let statements, which trips clippy items_after_statements (-D warnings). The xtask test target then fails to compile in the clippy and Windows build and test (debug) jobs, so the new test never runs in CI. That also means the PR body's "149 passed" and the mutation-goes-red claims can't hold at this head. build-and-test, build-and-test (ubuntu-26.04) and the rocm-dash coverage check are red too, probably from the same compile error (I only confirmed the clippy and Windows logs).
Non-blocking: the closed-PR run behaviour and fromJSON('') are still untested (you disclose this), and the test is ~125 lines and could be split.
Once the const is moved and CI is green, I'm happy to take another look.
clippy items_after_statements (-D warnings) rejected the const declared after the test's first let, which stopped the xtask test target compiling in the clippy and Windows debug jobs. Declare it at module level. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Looks good to me. This is a diff-only read plus the green checks, so I haven't exercised it on a real merged or closed PR.
Nothing blocking. A few small things, none of which need to hold up the merge:
- Wording: the comment on
changes(and the PR body) says every lane gates onneeds.changes.result == 'success', bute2e-reportgates onneeds.changes.outputs.serveand only skips because a skipped job's output is empty. Worth rewording; inline comment below. - The closed-PR run only cancels the older queued run if it lands in the same concurrency group and
cancel-in-progressevaluates true for it. That part is outside the diff and this PR's own CI never produces aclosedevent, so please confirm it on the first PR that is merged or closed after this lands, as the PR body already suggests. - The contract test pins each lane's whole
if. That is a deliberate choice, but it means any future edit to the gate (#1018 touches the same file) has to update the test too. Fine as long as that is the intent. - Minor: the unnamed
boolin the test's tuple table; inline comment below.
The changes comment said every lane gates on changes' result, but e2e-report skips because a skipped job's output is empty; say both. Replace the table's unnamed bool with a PinnedIf struct whose matrix_lane field says what it is. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
After rebasing onto #1032, closing_a_pr_supersedes_its_self_hosted_runs pinned the strix-ubuntu and w7900 gates without the pull_request and merge_group exclusions this PR adds. Pin them with the exclusions. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
… merge queue drops its native Linux GPU lanes (ROCm#1018) * ci: run pull requests on one Linux and one Windows GPU node A PR now starts the MI300X lane (the only lane where vLLM is the effective engine) and the Strix Halo Windows pool lanes (native + WSL2). The Strix Halo Ubuntu and W7900 lanes skip pull_request but keep push/merge_group/dispatch. nightly.yml's Strix Halo Linux lane targeted the offline static `native` host and never ran; point it at the DevLab Dispatch pool the per-PR lane uses and pin that in the workflow contract test. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs: say which per-PR lanes skip pull_request Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs: release-candidate gate runs the full self-hosted matrix The row said 'the same per-PR lanes', which now means the narrowed three-lane set. A push to release/** still runs all five lanes. Also drop the paragraph that said the Ubuntu nightly lane still pins the offline `native` host. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: pin the whole per-PR gate; guard the nightly Strix job; fix stale coverage docs - Contract test asserts the full if: gate of e2e-gpu-strix-ubuntu and e2e-gpu-w7900, so the pull_request exclusion must sit in the non-dispatch branch and nothing else about the event may change. - New test: the nightly Strix Ubuntu job sets no static-host paths and reads no CARGO_HOME/RUSTUP_HOME/TMPDIR/PIP_CACHE_DIR it does not set. - Docs: the cucumber README, testing.md, the nightly-channel paragraph and the Triggers paragraph no longer claim Strix Ubuntu / W7900 run per PR. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: tighten the nightly Strix path guard; fix remaining per-PR Strix/W7900 claims - nightly_strix_ubuntu_job_holds_no_static_host_paths now also rejects any $HOME read other than the $HOME/.cargo/bin Rust bootstrap. - Docs and workflow comments no longer call the Strix Ubuntu / W7900 lanes per-PR; reflow two overlong prose lines. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: drop W7900 from merge_group; fix the last per-PR Strix claims - e2e-gpu-w7900 no longer runs in the merge queue (one runner on borrowed quota, the queue does not wait for it); it still runs on push, dispatch and nightly. The contract test pins each lane's whole gate and that the W7900 job holds no E2E_MERGE_QUEUE opt-in. - Reword the nightly Strix comment that called e2e-gpu-strix-ubuntu per-PR, and two comment-only leftovers in xtask and the cucumber crate. - The nightly path guard matches whole words, so HF_TMPDIR is not TMPDIR. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: drop e2e-gpu-strix-ubuntu from merge_group Over 7 days 35% of its merge-queue jobs (98 of 281) were cancelled at the 45 min cap. The check is not required and the queue does not wait for it. It still runs on push (main, release/**), dispatch and nightly. The contract test pins both lanes' whole gate and that neither job opts into @merge-queue scenarios; docs updated. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs, ci: make every merge_group claim match the workflow Strix Ubuntu and W7900 skip pull_request and merge_group and run on push, dispatch and nightly. Fix the nightly.yml comment and the ci-hardware-testing.md rows that said otherwise, state that the merge queue has no native Linux GPU lane (Windows Strix and WSL2 only), and wrap the comment lines this PR left long. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs, tests: fix stale merge-queue and per-PR claims, re-wrap two lines The merge_queue field doc said the serve matrix runs on every lane but MI300X; only Windows Strix and WSL2 run @merge-queue serves now. The AIM serve comment in model_serving.feature still called Strix Ubuntu a per-PR job. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: pin the PR-skipping lanes in ROCm#1032's closed-PR if table After rebasing onto ROCm#1032, closing_a_pr_supersedes_its_self_hosted_runs pinned the strix-ubuntu and w7900 gates without the pull_request and merge_group exclusions this PR adds. Pin them with the exclusions. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs, ci: drop the stale Docker 'unconfirmed' and per-PR budget claims The nightly Strix lane is on the pool where Docker was reached (serve-26 passed on both legs); say so, and note serve-27's stop failure. The include_nightly budget comment no longer calls the push-run budget per-PR. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * tests: say Docker is reachable on the nightly Strix pool, and serve-27's stop failure The model_serving.feature AIM comment still called Docker on the nightly Strix Halo host unconfirmed; dispatch run 43025895270 reached it (serve-26 passed on both legs) and serve-27 fails at stopping the service. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * tests: declare serve-aim-lifecycle-parity an expected failure on Strix Halo Ubuntu Its stop step fails with 'Local server stop not confirmed' (tracked in rocm-cli#1058), so the nightly lane reports it as xfail instead of an unexpected regression. Point the nightly.yml, docs and feature comments at it. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs, ci: reference rocm-cli#1058 instead of pinning a run ID and failure text; say Windows pool The serve-27 evidence lives in the tracking issue, so nightly.yml, docs/testing.md, model_serving.feature and expectations.toml no longer quote a run ID or the failure string. A PR runs two Windows-pool jobs (Windows and WSL2), so say one Windows GPU pool, not node. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * tests, docs: keep WSL2 out of the serve-27 xfail row; fix two stale ci-hardware-testing claims The row matched os=linux, which WSL2 also reports on gfx1151, so a WSL host that gained Docker and passed the scenario would XPASS. Add is_wsl = false. Reword the e2e-report coverage line and the pre-warmed runtime paragraph, which still named per-PR platforms that no longer all run per PR and the static native hosts. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci, docs: pin that e2e-wsl and Windows Strix set E2E_MERGE_QUEUE; say WSL2 runs the merge-queue serves The Triggers paragraph said the heavy serves run only on the Windows Strix Halo lanes while other docs said WSL2 too. The workflow sets E2E_MERGE_QUEUE on both e2e-gpu-strix-windows and e2e-wsl, so reword the paragraph and pin both in the contract test. Add a one-line symptom to the serve-27 xfail reason. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> --------- Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Why
On 2026-11-11 the MI300X lane had 18 queued jobs, at least 11 of them dead work: the PR had merged (#998, #959, #694, #818, #991) or the run was for an older head of a still-open PR (#997, #1004, #1006, #999, #856). Two runners take about 3.5 jobs an hour, so dead jobs delay live ones (#1018's job waited 9+ hours; median queue wait was 13 h).
The concurrency group is already right (one workflow file, not reusable, keyed on the PR number,
cancel-in-progresson for PR runs). Nothing in it fires when a PR is merged or closed: no new run starts, so nothing cancels the queued one.What
pull_request: types: [opened, synchronize, reopened, closed]: theclosedrun joins the PR's existing group and supersedes the earlier run.changesskips onclosed. Every lane gates onneeds.changes.result == 'success'ande2e-reportonneeds.changes.outputs.serve(empty for a skipped job), so the whole matrix and the report skip.pr-closedjob (ubuntu, 5 min, no permissions) gives that run something to run.fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix). The placeholder applies only whenchangeswas skipped (a closed PR), so a lane'sstrategynever gets an empty string there; ifchangessucceeded but stopped writingsdk_version_matrix,fromJSON('')still fails loudly instead of quietly runningcurrent. It relies on&&/||returning their operands, which this workflow's concurrency group already depends on.closing_a_pr_supersedes_its_self_hosted_runspins all of it: thetypeslist, thechangesskip, thepr-closedjob, the wholeifof every lane and ofe2e-report, and the guarded matrix expression, which only jobs pinned as matrix lanes must carry.cargo test -p xtask workflow_contract: 149 passed.cargo clippy --locked --workspace --all-targets -- -D warnings(the CI command): clean at 299b2ec; a876036 failed it, which is what 299b2ec fixes.No scenario is needed: this is CI plumbing and changes no product behaviour.
The contract test reads each job's comment-free block (the
changesskip, the wholepr-closedjob, and that every other job skips whenchangesskips). Each of the three edits, the skip moved into a comment or onto another job, and a lane losing both itschangesgates, was applied in turn and turned the test red.Empty matrix on a closed run
On
closed,changesis skipped, soneeds.changes.outputs.sdk_version_matrixis empty. Each of the five lanes (e2e-gpu,e2e-gpu-strix-ubuntu,e2e-gpu-strix-windows,e2e-wsl,e2e-gpu-w7900) therefore uses the guarded expression above, and the contract test requires it, and no other read of the output, in each of them.Still untested, because this PR's own CI never produces a
closedevent: the closed run itself, andfromJSON('')failing on a missing output after a successfulchanges(reasoned from GitHub's behaviour, not run). On the first PR merged or closed after this lands, check that itsE2E self-hostedrun showsSupersede runs of a closed PRas the only job that ran, with every other job skipped and no error.Known limit (not fixed here)
The header of this workflow already notes GitHub cannot cancel a job queued on a runner that is not picking up. Old-head runs whose W7900 job is still queued (one W7900 runner, busy) stayed
queuedfor 14+ hours after a newer push, and the newer run satpendingbehind them. A cancel request on those runs did not complete within minutes either. This PR cannot fix that; it only stops merged/closed PRs from leaving runs behind where the cancel completes; it does not clear runs whose job is queued for a runner that never picks it up. Closing it needs either the W7900 lane not queueing without capacity, orforce-cancel.Conflicts with #1018 (touches the same file) are textual only.