Skip to content

ci: cancel a PR's queued self-hosted E2E runs when it is merged or closed - #1032

Merged
tomastola merged 9 commits into
mainfrom
ci/cancel-gpu-runs-on-pr-close
Nov 12, 2026
Merged

tomastola merged 9 commits into
mainfrom
ci/cancel-gpu-runs-on-pr-close

Conversation

@tomastola

@tomastola tomastola commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Why

On 2026-11-11 the MI300X lane had 18 queued jobs, at least 11 of them dead work: the PR had merged (#998, #959, #694, #818, #991) or the run was for an older head of a still-open PR (#997, #1004, #1006, #999, #856). Two runners take about 3.5 jobs an hour, so dead jobs delay live ones (#1018's job waited 9+ hours; median queue wait was 13 h).

The concurrency group is already right (one workflow file, not reusable, keyed on the PR number, cancel-in-progress on for PR runs). Nothing in it fires when a PR is merged or closed: no new run starts, so nothing cancels the queued one.

What

  • pull_request: types: [opened, synchronize, reopened, closed]: the closed run joins the PR's existing group and supersedes the earlier run.
  • changes skips on closed. Every lane gates on needs.changes.result == 'success' and e2e-report on needs.changes.outputs.serve (empty for a skipped job), so the whole matrix and the report skip.
  • A no-op pr-closed job (ubuntu, 5 min, no permissions) gives that run something to run.
  • Each lane's matrix reads fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix). The placeholder applies only when changes was skipped (a closed PR), so a lane's strategy never gets an empty string there; if changes succeeded but stopped writing sdk_version_matrix, fromJSON('') still fails loudly instead of quietly running current. It relies on &&/|| returning their operands, which this workflow's concurrency group already depends on.
  • Contract test closing_a_pr_supersedes_its_self_hosted_runs pins all of it: the types list, the changes skip, the pr-closed job, the whole if of every lane and of e2e-report, and the guarded matrix expression, which only jobs pinned as matrix lanes must carry.

cargo test -p xtask workflow_contract: 149 passed. cargo clippy --locked --workspace --all-targets -- -D warnings (the CI command): clean at 299b2ec; a876036 failed it, which is what 299b2ec fixes.

No scenario is needed: this is CI plumbing and changes no product behaviour.

The contract test reads each job's comment-free block (the changes skip, the whole pr-closed job, and that every other job skips when changes skips). Each of the three edits, the skip moved into a comment or onto another job, and a lane losing both its changes gates, was applied in turn and turned the test red.

Empty matrix on a closed run

On closed, changes is skipped, so needs.changes.outputs.sdk_version_matrix is empty. Each of the five lanes (e2e-gpu, e2e-gpu-strix-ubuntu, e2e-gpu-strix-windows, e2e-wsl, e2e-gpu-w7900) therefore uses the guarded expression above, and the contract test requires it, and no other read of the output, in each of them.

Still untested, because this PR's own CI never produces a closed event: the closed run itself, and fromJSON('') failing on a missing output after a successful changes (reasoned from GitHub's behaviour, not run). On the first PR merged or closed after this lands, check that its E2E self-hosted run shows Supersede runs of a closed PR as the only job that ran, with every other job skipped and no error.

Known limit (not fixed here)

The header of this workflow already notes GitHub cannot cancel a job queued on a runner that is not picking up. Old-head runs whose W7900 job is still queued (one W7900 runner, busy) stayed queued for 14+ hours after a newer push, and the newer run sat pending behind them. A cancel request on those runs did not complete within minutes either. This PR cannot fix that; it only stops merged/closed PRs from leaving runs behind where the cancel completes; it does not clear runs whose job is queued for a runner that never picks it up. Closing it needs either the W7900 lane not queueing without capacity, or force-cancel.

Conflicts with #1018 (touches the same file) are textual only.

@tomastola
tomastola requested a review from a team as a code owner November 11, 2026 10:43
@tomastola
tomastola requested a review from fredespi November 11, 2026 10:43
…osed

The E2E self-hosted workflow supersedes a PR's older run only when a newer
push starts a run in the PR's concurrency group. A merged or closed PR starts
none, so its queued MI300X/W7900 jobs stay in the queue and hold a runner
for a head nobody will look at again (about half of the 18 queued MI300X jobs
on 2026-11-11 were of this kind).

List `closed` among the pull_request types. The run it starts joins the PR's
group, so cancel-in-progress cancels the earlier run. `changes` skips on
`closed`, which skips every lane, leaving only a no-op `pr-closed` job.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola force-pushed the ci/cancel-gpu-runs-on-pr-close branch from e453be3 to 201e226 Compare November 11, 2026 10:48
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 11:10

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 201e22692d2f

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (both changed files, read completely, plus the full surrounding e2e-selfhosted.yml workflow — concurrency group, every downstream job's needs/if gating, and the workflow's own file-header rationale — to judge the change in place).
No ticket named. The description's claims (list closed among pull_request types, skip changes on closed, add a no-op pr-closed job, and a contract test pinning all three) all match the diff; the cited GitHub issue/PR numbers are evidence for the problem statement, not a tracked goal this PR claims to close.

Blocking

None

Non-blocking

  • docs/ci-hardware-testing.md's "Triggers" section is not updated for the new closed-PR supersession path — docs/ci-hardware-testing.md:358-360
    That section is an otherwise exhaustive account of when the self-hosted GPU jobs run and what cancels what ("A new push to a pull request cancels that PR's older run; push, merge_group and manual runs each get a group of their own, so none is cancelled or replaced by another."). It does not mention that a PR's closed event now also starts a (no-op) run that joins the same concurrency group and supersedes the PR's prior run. The jobs this doc describes are unaffected (they still never run on closed), so nothing it says is actually wrong, but a reader relying on it for the full cancellation picture would miss this path.
    Confidence 70 · architectural · Fix: add a sentence to the "Triggers" section (or a cross-reference to the new on.pull_request comment in e2e-selfhosted.yml) noting that closing/merging a PR also joins its group and cancels its queued/running self-hosted run.

Decisions for the author

None

Positive signals

  • The fix reuses the workflow's existing concurrency-group/cancel-in-progress mechanism (a trivial job that only has to start in the same group) instead of adding a new job with actions: write permission to call the cancellation API directly — no new permissions, minimal blast radius.
  • The new contract test (closing_a_pr_supersedes_its_self_hosted_runs) asserts the three sub-changes (the types: list, the changes job's skip condition, and the pr-closed job) independently, so a regression in any one of them fails on its own rather than only as a side effect of another assertion.

Deployment notes

None — this is a GitHub Actions workflow/config change with no runtime or release impact.

What this covered

Both changed files in full (.github/workflows/e2e-selfhosted.yml, xtask/src/workflow_contract.rs), diffed e3ad2488d…201e22692d2f (PR base…head, fetched read-only via pull/1032/head), plus the whole surrounding workflow file (concurrency group, every job's needs/if chain down through e2e-report) and docs/ci-hardware-testing.md's trigger/supersession documentation. Traced by hand that on a closed pull_request event: changes is skipped (new if), every GPU lane's pre-existing needs.changes.result == 'success' gate therefore also skips, and e2e-report's pre-existing needs.changes.outputs.serve == 'true' gate evaluates false (a skipped job's output reads as empty), so the whole matrix and the report are skipped — matching the PR description. Confirmed via gh pr view --json statusCheckRollup that all 41 checks on the head commit are green; did not re-run cargo test -p xtask workflow_contract myself, reading the already-green CI result instead per the review method. The diff was small and tightly coupled (one workflow file and its own contract test, 52 lines total), so this was a single-pass review by one reviewer rather than a multi-worker fan-out, per the review method's size-based scaling.

Did not run: history/prior-changes passes — looked at git log/blame context implicitly while reading the file but found nothing germane beyond what's cited above; agent-instruction-file pass — no CLAUDE.md/AGENTS.md rules apply beyond the repository's own documented CI conventions, which the change follows.
Not reconciled against any discussion on the pull request (review threads, if any, were not read).

Note in the Triggers section that a PR's closed event starts a no-op run in
the PR's concurrency group, and that a run whose job is still queued for a
runner may not finish cancelling.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Thanks for the review of 201e226. The non-blocking note is addressed in 867fb95: the Triggers section of docs/ci-hardware-testing.md now says that a PR's closed event starts a no-op run in the PR's group that supersedes its earlier run, and that a run whose job is still queued for a runner may not finish cancelling.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The skip chain looks right to me: changes is skipped on closed, every GPU lane gates on needs.changes.result == 'success', and e2e-report keys off the serve output. I'm requesting changes for the items below; none of them is a regression in the workflow itself.

  1. The contract test is plain contains matching and is not scoped to the job blocks, so it can pass on a comment or on the wrong job. Please scope the changes skip and the pr-closed job asserts to their job blocks, and confirm you reverted each of the three edits and saw the test go red.
  2. The cancel behavior is not verified end to end. The PR body says jobs queued on a runner that never picks them up are not cancelled, so this only helps lanes that have capacity. Please say that in the workflow comment and the docs rather than claiming the queue is freed.
  3. strategy.matrix uses fromJSON(needs.changes.outputs.sdk_version_matrix), which is an empty string when changes is skipped. I expect the job if to short-circuit it, but the PR's own CI never produces a closed event, so this is untested. Please confirm it on the first merged PR after this lands, or note the check in the PR text.
  4. The PR body should say in one sentence that no scenario is needed because this is CI plumbing.
  5. Wording nit in the docs: "frees the queue of dead work" reads stronger than the hedge in the next sentence.

Details are in the inline comments.

Comment thread xtask/src/workflow_contract.rs Resolved
Comment thread xtask/src/workflow_contract.rs Resolved
Comment thread .github/workflows/e2e-selfhosted.yml Resolved
Comment thread .github/workflows/e2e-selfhosted.yml Resolved
Comment thread docs/ci-hardware-testing.md Resolved

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers from me. The closed run lands in the same concurrency group as the PR's earlier runs (the group has no action component, and cancel-in-progress is true for pull_request), changes skips on closed, every lane gates on needs.changes.result == 'success', and pr-closed has no permissions and no checkout. This was a static review: nothing was built or run (no Rust toolchain here).

Non-blocking:

  • xtask/src/workflow_contract.rs:~7437-7449: the new test checks sh.contains(...) over the whole file, so the changes skip line would still pass if it moved to another job, and nothing asserts that the lanes or e2e-report skip on closed. job_block(&sh, "changes") / job_block(&sh, "pr-closed") would tighten it.
  • e2e-report does not gate on needs.changes.result like the lanes do; it skips because needs.changes.outputs.serve is empty. It works, but the description ("every lane gates on needs.changes.result == 'success'") and the test do not cover it, so a later edit to its if could quietly make it run on closed.
  • e2e-selfhosted.yml:~34-36,192-194 say the closed run cancels the older run "and with it any GPU job still queued", while the file header and the new docs paragraph correctly say a job queued for an unavailable runner may not finish cancelling. Reword the comments to the docs' hedge.
  • Unverified: on closed, changes is skipped, so fromJSON(needs.changes.outputs.sdk_version_matrix) in each lane's strategy.matrix sees an empty output. Skipped jobs should not evaluate strategy, but this PR never exercises a closed run; closing a scratch PR once would confirm it (or use || '["current"]').

🤖 by agenthub on AMD AgentHub

Read each assertion from one job's comment-free block, so prose or another
job cannot satisfy it, pin the whole `if` of `changes` and `pr-closed`
with the latter's runner and empty permissions, and assert that every other
job (including e2e-report) skips when `changes` skips.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Reword the workflow comments and the Triggers paragraph to the same hedge: a
job queued for a runner that is not picking it up may not finish cancelling,
so the closed run frees runners only where the cancel completes.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

@juhovainio all five points are answered: 1 fixed (57227cc, with the mutation check), 2 and 5 fixed (c4ee13d), 4 added to the PR text. Point 3 is not fixed in code; the PR text names the check to make on the first closed PR after this lands, and that thread stays open. jussielo-amd's non-blocking notes (e2e-report gating, comment wording) are covered by the same two commits.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 12:02

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I found nothing blocking: the closed-PR job only runs on closed, permissions: {} is set, the lanes and e2e-report are all gated so they skip when changes is skipped, and the limitation for already-running W7900 jobs is disclosed in the workflow comment, the docs and the PR body. CI is green on this head.

I left a few non-blocking suggestions inline: a || '["current"]' fallback on the lane matrices so a skipped changes can never trip fromJSON evaluation, a stricter assertion in the contract test for the lane if, and two small tidy-ups (share the comment-stripping helper, re-wrap one long doc line). None of these need to hold up the merge.

Comment thread .github/workflows/e2e-selfhosted.yml Resolved
Comment thread xtask/src/workflow_contract.rs Resolved
Comment thread xtask/src/workflow_contract.rs Resolved
Comment thread docs/ci-hardware-testing.md Resolved

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head c4ee13df8a9d

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change. Implements the fix described in the PR's own "Why"/"What" sections (cancel a PR's queued self-hosted E2E runs when it is merged or closed); no tracker ticket named.

Blocking

  • A closed/merged PR's run likely errors at job scheduling instead of cleanly skipping, because strategy.matrix is resolved from an empty fromJSON() input — .github/workflows/e2e-selfhosted.yml:224 (e2e-gpu), :488 (e2e-gpu-strix-ubuntu), :643 (e2e-gpu-strix-windows), :991 (e2e-wsl), :1599 (e2e-gpu-w7900), gated by the changes job's new skip at :104 and its sdk_version_matrix output at :119.
    Each of the five GPU lanes has strategy.matrix.sdk_version: ${{ fromJSON(needs.changes.outputs.sdk_version_matrix) }}, and each lane's own if: includes always() (to override GitHub's default "skip if a needed job didn't succeed" behavior, so the lane's own, more specific condition governs instead). On a closed PR event, the new if: on changes (line 104) causes changes to be skipped, leaving needs.changes.outputs.sdk_version_matrix empty. GitHub Actions resolves a job's strategy.matrix as part of scheduling the job; the "skip without touching the matrix" shortcut applies only to a job's implicit default if: success() — once a job's own if: explicitly includes always() (as all five here do), that shortcut does not apply, and the matrix is resolved regardless of what the if: will ultimately evaluate to. fromJSON('') on an empty string throws a job-level "Error from function 'fromJSON': empty input" scheduling error rather than producing a clean skip — a failure mode reproducible in other reports of the same always() + fromJSON(needs.*.outputs.*) + skippable-upstream combination (and fixed upstream, for the equivalent Actions-compatible engine, by special-casing always()/failure()/cancelled() so the matrix is still resolved rather than short-circuited). The PR's own description independently raises this exact risk under "Not verified end to end," expressing the hope that "the job if should short-circuit it" — the mechanism above is specifically why that hope likely does not hold once always() is present.
    Since none of these lanes are required merge checks, this cannot block a merge, but it does mean the change's stated goal (a closed-PR run showing only the no-op pr-closed job, "with every other job skipped and no strategy evaluation error," per the PR body) is likely not met, and the new contract test cannot catch this because it only pattern-matches the YAML's text — it cannot execute the workflow and so cannot observe a scheduling-time error.
    Confidence 80 · logic · Fix: give each fromJSON(...) a fallback so it never sees an empty string, e.g. fromJSON(needs.changes.outputs.sdk_version_matrix || '["current"]') on all five lanes — the per-instance if: already prevents the job from doing anything once instantiated, so a harmless placeholder is sufficient.

Non-blocking

  • Docs: inserted sentences separate a "therefore" from the premise it refers to — docs/ci-hardware-testing.md:358-366.
    The concluding sentence "The heavy @merge-queue serves therefore no longer run on MI300X…" draws on the paragraph's opening parenthetical about MI300X being skipped in the merge queue. The two new sentences about the closed-PR no-op run are inserted directly before it, so "therefore" now sits right after an unrelated sentence about queued-runner cancellation, reading briefly as a non sequitur until the reader reconnects it to the earlier premise.
    Confidence 70 · mechanical · Fix: move the two new sentences to the end of the paragraph, after the "therefore" sentence, or start the MI300X conclusion with an explicit backward reference.

  • New test re-derives job-id extraction instead of reusing the existing job_ids helper — xtask/src/workflow_contract.rs (new code in closing_a_pr_supersedes_its_self_hosted_runs, the ids extraction block) vs. the pre-existing job_ids function at xtask/src/workflow_contract.rs:361-368.
    The new test re-implements an equivalent indent/comment filter plus an alnum/-/_ allow-list to extract job ids from the jobs: block, duplicating what job_ids(text) already does a few hundred lines earlier in the same file (verified the two approaches produce the identical job-id set against the real workflow at this commit). Calling job_ids(&sh) directly would drop both the duplicated filter and the "extractor sanity check" assertion that exists only to hedge against the newly-written duplicate.
    Confidence 85 · mechanical · Fix: call the existing job_ids helper instead of re-deriving the same filter.

Decisions for the author

None.

Positive signals

  • The new contract test derives its job list from the workflow file itself (via the existing job_ids-equivalent extraction) rather than a hardcoded list, so a newly added lane is automatically required to carry the same changes-skip guard.
  • The PR description is unusually candid about what is not verified (the exact fromJSON/skip risk above) and proposes a concrete post-merge check, rather than overclaiming correctness.
  • pr-closed uses permissions: {} and a GitHub-hosted runner with no uses:/checkout/secrets, avoiding any token or self-hosted-runner exposure for a job that exists purely to occupy a concurrency group.

Deployment notes

If the blocking finding above is correct, every PR merge or close will show a job-level "strategy" evaluation error on the (non-required) E2E self-hosted check rather than the clean single-no-op-job run described in the PR body. This is cosmetic — it does not affect mergeability — but is worth fixing before relying on the author's own planned "check the first real closed PR" verification, since the failure would otherwise appear on every subsequent PR close until patched.

What this covered

Full read of all three changed files (.github/workflows/e2e-selfhosted.yml, docs/ci-hardware-testing.md, xtask/src/workflow_contract.rs) against their pre-existing surrounding code (every job's if:/needs/strategy in the workflow, and the pre-existing helper functions the new test relies on or duplicates), base e3ad2488dea2a408a03e216cefb0c4f67b3c5a90…head c4ee13df8a9d4705a9cd00960ec4abbc37b4ce0c. Two independent fan-out workers each reviewed one coupled file group (workflow+docs; the new Rust test) blind, plus this pass's own direct reading and targeted platform-semantics research to verify the primary finding. All 40 CI checks on the head commit are green; no failing required check. One fan-out worker disclosed that it incidentally viewed the PR title/body despite being asked not to; its stated verdict rests on YAML tracing and general GitHub Actions semantics rather than that content, and the finding is independently corroborated here. Did not run: a git log/git blame history pass, and a prior-review-comments pass (no earlier review visible to reconcile against). Not reconciled against any PR discussion or comments — this review did not read them, per method.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

Fall back to ["current"] when changes' matrix output is empty (changes is
skipped on closed), so no lane's strategy evaluates fromJSON of an empty
string. Require every lane's if to open with
always() && needs.changes.result == 'success', share the comment-stripping
helper, and re-wrap the docs paragraph.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Reply to nowycondro's review of c4ee13d: the blocking finding (empty fromJSON() in each lane's strategy.matrix on a closed run) is fixed in 30175ac with the || '["current"]' fallback on all five lanes, pinned by the contract test. Not done, both non-blocking: moving the new docs sentences after the "therefore" sentence, and reusing job_ids instead of the test's own id extraction.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 14:23

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The workflow change, the comment, the docs and the PR body agree, and the diff is clean of internal references. Nothing blocking; two small notes inline.

Not verified from the diff alone:

  • Test (affected crates) failed on an apt DNS error before reaching tests, so closing_a_pr_supersedes_its_self_hosted_runs hasn't been seen passing in CI yet. Worth a rerun before merge.
  • That the concurrency group / cancel-in-progress actually covers the closed event, and that e2e-report's if has no arm beyond the asserted prefix that is true when changes is skipped. A live closed-PR run would confirm both.

The other red checks I looked at were infra flakes (apt DNS, crates.io download EOF) and need a rerun.

Comment thread .github/workflows/e2e-selfhosted.yml Resolved
Comment thread xtask/src/workflow_contract.rs Resolved

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 30175ac19a5b

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change. No tracker ticket named; the cited GitHub PR numbers (#998, #959, #694, #818, #991, #997, #1004, #1006, #999, #856, #1018) are evidence for the problem statement, not a tracked goal this PR claims to close.

Blocking

  • The PR description's "Not verified end to end" section describes a risk the diff already fixes, without mentioning the fix — PR description vs. .github/workflows/e2e-selfhosted.yml:224,488,643,991,1599.
    The description says each lane's strategy.matrix reads fromJSON(needs.changes.outputs.sdk_version_matrix), calls this "empty when changes is skipped," hopes "the job if should short-circuit it," and asks the reader to "check on the first PR merged or closed after this lands" for "no strategy evaluation error." But the diff itself already adds a || '["current"]' fallback to all five fromJSON(...) calls, which deterministically resolves the empty-output case regardless of whether a skip short-circuits strategy evaluation — the description just never says so, and still frames the matter as an open, only-verifiable-live risk. A reviewer or future reader trusting the description would believe this is still unresolved when it is not.
    Confidence 90 · mechanical · Fix: update the description to say the || '["current"]' fallback was added and why it removes the uncertainty, and either drop the "check the first closed PR" ask or narrow it to something the fallback doesn't already cover (e.g. confirming pr-closed is the only job that runs).

  • The new contract test's lane/e2e-report if: checks use starts_with, which a future appended || … arm would defeat — exactly the gap the test's own comment says it guards against — xtask/src/workflow_contract.rs:7520-7534 (assertions), vs. the real conditions at .github/workflows/e2e-selfhosted.yml:235-244 (and the other 4 lanes) and :1749-1751 (e2e-report).
    Each lane's full if: is always() && needs.changes.result == 'success' && ... && (...). The test only checks condition.starts_with("always() && needs.changes.result == 'success'") (same pattern for e2e-report's serve-output prefix). GitHub Actions expressions bind && tighter than || — precedence this same test file already relies on elsewhere — so a future edit appending || true (or any always-true arm) after the condition's existing closing ) would make the whole expression evaluate true unconditionally, defeating the skip-on-closed guarantee this PR introduces, while still satisfying starts_with untouched. The test's own comment at :7528-7529 ("an || arm... would make it run when changes is skipped") claims exactly the protection a starts_with-only check does not fully deliver against an appended arm (only against one inserted before the asserted prefix). changes and pr-closed's own if: are checked with assert_eq! just above this block, so the stronger form is already the test's own established pattern — it's just not applied here.
    Confidence 70 · logic · Fix: assert the full condition with assert_eq! (matching the pattern already used for changes/pr-closed), or additionally assert the condition's exact trailing characters so an appended arm can't slip past unexercised.

Non-blocking

  • A dangling "therefore" in the docs — docs/ci-hardware-testing.md:358-366.
    The closing sentence "The heavy @merge-queue serves therefore no longer run on MI300X…" draws on the paragraph's opening parenthetical about MI300X being skipped in merge_group (a few lines earlier), not on the new closed-PR sentences inserted directly before it. Each new sentence reads fine on its own; the insertion point breaks the "therefore"'s backward reference, reading briefly as a non sequitur.
    Confidence 70 · mechanical · Fix: move the two new sentences to the end of the paragraph (after the "therefore" sentence) or elsewhere in the Triggers section.

  • New test re-derives job-id extraction instead of calling the file's own existing job_ids helper — xtask/src/workflow_contract.rs:7498-7509 vs. :361-368.
    The new ids extraction re-implements the same indent + alnum/-/_ filter over top_level_block(&sh, "jobs") that the pre-existing job_ids(text) helper already provides and that other tests in this file already share (e.g. self_hosted_e2e_jobs). Verified both produce the identical ordered id list against the real workflow at this commit.
    Confidence 85 · mechanical · Fix: replace the block with let ids = job_ids(&sh); and adjust the handful of comparisons that follow.

Decisions for the author

  • The cancellation benefit is weakest for the case the PR's own "Why" cites as motivating it — tradeoff.
    W7900 has a single, often-busy runner, and the new comments/docs honestly disclose that a job already queued there may not finish cancelling. The PR's own numbers (18 queued MI300X jobs, 11 dead) suggest MI300X and the Strix pool lanes are where this actually pays off, while W7900 — the hardware explicitly called out for the queued-cancel limitation — benefits least. Worth confirming that's the understood scope rather than an unstated gap.

Positive signals

  • Reuses the pre-existing concurrency-group + cancel-in-progress mechanism via a trivial job that only has to start in the right group, instead of adding a job with actions: write to call the cancellation API directly — no new permissions, minimal blast radius (consistent with the clean zizmor security-scan result on this head).
  • pr-closed is deliberately minimal — permissions: {}, GitHub-hosted runner, no uses:/needs:/secrets/checkout — so a job that now fires automatically on every PR close carries no meaningful attack surface.
  • Every GPU lane's and e2e-report's if: opens with the changes-gate ahead of the rest of its condition, so the skip-on-closed behavior doesn't depend on how the remainder of a long boolean expression is structured.
  • The new contract test derives its job-id list from the live workflow file rather than a hardcoded list, so a newly added GPU lane is automatically required to carry the same changes-skip guard.

Deployment notes

None beyond what's already disclosed: this is a GitHub Actions workflow/doc/test change with no runtime or release impact. The one known limitation (a GPU job already queued on a busy/offline self-hosted runner may not finish cancelling) is disclosed in the workflow's own comments, the docs, and the PR description.

What this covered

Full review of the whole change: all three changed files read completely (.github/workflows/e2e-selfhosted.yml, 1845 lines; docs/ci-hardware-testing.md, 605 lines; xtask/src/workflow_contract.rs, 9985 lines, entirely #[cfg(test)]-gated), diffed e3ad2488dea2a408a03e216cefb0c4f67b3c5a90…30175ac19a5baa2240321bc9a37b55c51947993d, fetched read-only (no checkout). Two independent blind fan-out workers each covered one coupled file group (workflow+docs; the Rust contract-test file in full), plus this pass's own direct tracing of every job's if/needs/strategy chain and a check of the repo's other workflow files (ci.yml, nightly*.yml, queue-run.yml) for whether any other self-hosted, PR-triggered workflow needed the same fix — none does; e2e-selfhosted.yml is the only self-hosted workflow with a pull_request trigger.

CI: 8 checks report FAILURE on this head (build-and-test ×2, Test (affected crates) ×2, Coverage (rocm-dash crates, ratcheted), E2E tests ×2, remote control channel (containerised)). Read the actual job logs for several and confirmed each fails at an apt-get/DNS-resolution step ("Temporary failure resolving 'azure.archive.ubuntu.com'") before any build or test step runs — an environment flake unconnected to this change's content, not something it introduces. One consequence: the new contract test has not been observed passing in CI at this head; its correctness here rests on manual tracing against the real workflow file (by two independent workers and this pass), not an executed run — no cargo/rustc toolchain was available in this environment either.

Agent-instruction file AGENTS.md read in full; nothing in scope violates it (no new third-party uses: to pin, the changed Rust file is entirely test-gated so the 1,500-line production-file cap doesn't apply, no internal/proprietary references in the diff, and the PR's own "no scenario is needed — CI plumbing" line satisfies the one place it requires that explanation).

Did not run: a prior-review-comments reconciliation pass — reading GitHub review threads is out of scope for a blind pass under this method. History pass (git log on the three files) found nothing germane beyond context already incorporated above.

Not reconciled against any PR discussion or comments beyond the description and CI status read at the method's final step.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Reply to nowycondro's review of 30175ac. First blocking finding (the description still framed the empty-matrix risk as open): fixed in the PR description. It now says the || '["current"]' fallback is on all five lanes (e2e-selfhosted.yml:224, 488, 643, 991, 1599) and that the contract test requires it, and it narrows the post-merge check to the closed run itself: Supersede runs of a closed PR should be the only job that ran. Second blocking finding (starts_with on a lane's if would not catch an appended || true): not addressed yet. Agreed that && binds tighter than ||, so an appended arm would defeat the gate while still passing the prefix check. Fixing it needs a new commit, which restarts CI, so I am asking before pushing. The two non-blocking notes (docs "therefore" placement, reuse of job_ids) are also still open.

A lane's if was checked by prefix, so an `|| true` appended to it (`&&` binds
tighter than `||`) would still pass while defeating the skip on closed. Pin
every lane's and e2e-report's full condition with assert_eq!, and fail on a job
with no pinned condition. Take the job ids from job_ids, and move the closed-PR
sentences to the end of the Triggers paragraph so the 'therefore' sentence
follows the premise it refers to.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Reply to nowycondro's review of 30175ac. Second blocking finding (starts_with on a lane's if would not catch an appended || true): fixed in 4cabe65. closing_a_pr_supersedes_its_self_hosted_runs now pins the full if of each of the five lanes and of e2e-report with assert_eq!, and fails on any job that has no pinned condition. Mutation check, each red: || true appended to e2e-gpu-w7900; the same on e2e-gpu; the same on e2e-report; && true appended to e2e-wsl; the merge_group exclusion dropped from e2e-gpu. The workflow was restored after each, and cargo test -p xtask workflow_contract passes (149). Both non-blocking notes are applied in the same commit: the test takes its job ids from job_ids, and the closed-PR sentences now sit at the end of the Triggers paragraph, after the "therefore" sentence. The first blocking finding (the description) was fixed earlier.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 15:41
Read each lane's matrix as
fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix).
On a closed PR changes is skipped, so the placeholder applies; if changes
succeeded but stopped writing the output, fromJSON('') still fails instead of
quietly running current. Pin that expression per entry in the contract test,
and only for jobs pinned as matrix lanes, so a non-matrix helper job is not
forced to carry it.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workflow change itself looks right to me, but this head doesn't build in CI, so I can't sign off yet.

Blocking: the new contract test in xtask/src/workflow_contract.rs declares const MATRIX_EXPR after let statements, which trips clippy items_after_statements (-D warnings). The xtask test target then fails to compile in the clippy and Windows build and test (debug) jobs, so the new test never runs in CI. That also means the PR body's "149 passed" and the mutation-goes-red claims can't hold at this head. build-and-test, build-and-test (ubuntu-26.04) and the rocm-dash coverage check are red too, probably from the same compile error (I only confirmed the clippy and Windows logs).

Non-blocking: the closed-PR run behaviour and fromJSON('') are still untested (you disclose this), and the test is ~125 lines and could be split.

Once the const is moved and CI is green, I'm happy to take another look.

Comment thread xtask/src/workflow_contract.rs Resolved
clippy items_after_statements (-D warnings) rejected the const declared after
the test's first let, which stopped the xtask test target compiling in the
clippy and Windows debug jobs. Declare it at module level.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 11, 2026 17:03

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. This is a diff-only read plus the green checks, so I haven't exercised it on a real merged or closed PR.

Nothing blocking. A few small things, none of which need to hold up the merge:

  • Wording: the comment on changes (and the PR body) says every lane gates on needs.changes.result == 'success', but e2e-report gates on needs.changes.outputs.serve and only skips because a skipped job's output is empty. Worth rewording; inline comment below.
  • The closed-PR run only cancels the older queued run if it lands in the same concurrency group and cancel-in-progress evaluates true for it. That part is outside the diff and this PR's own CI never produces a closed event, so please confirm it on the first PR that is merged or closed after this lands, as the PR body already suggests.
  • The contract test pins each lane's whole if. That is a deliberate choice, but it means any future edit to the gate (#1018 touches the same file) has to update the test too. Fine as long as that is the intent.
  • Minor: the unnamed bool in the test's tuple table; inline comment below.

Comment thread .github/workflows/e2e-selfhosted.yml Resolved
Comment thread xtask/src/workflow_contract.rs Resolved
The changes comment said every lane gates on changes' result, but e2e-report
skips because a skipped job's output is empty; say both. Replace the table's
unnamed bool with a PinnedIf struct whose matrix_lane field says what it is.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 11, 2026 21:55
@tomastola
tomastola added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 10f36cb Nov 12, 2026
41 checks passed
@tomastola
tomastola deleted the ci/cancel-gpu-runs-on-pr-close branch November 12, 2026 09:30
tomastola added a commit that referenced this pull request Nov 12, 2026
After rebasing onto #1032, closing_a_pr_supersedes_its_self_hosted_runs
pinned the strix-ubuntu and w7900 gates without the pull_request and
merge_group exclusions this PR adds. Pin them with the exclusions.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
r0x0r pushed a commit to r0x0r/rocm-cli that referenced this pull request Nov 12, 2026
… merge queue drops its native Linux GPU lanes (ROCm#1018)

* ci: run pull requests on one Linux and one Windows GPU node

A PR now starts the MI300X lane (the only lane where vLLM is the effective
engine) and the Strix Halo Windows pool lanes (native + WSL2). The Strix Halo
Ubuntu and W7900 lanes skip pull_request but keep push/merge_group/dispatch.

nightly.yml's Strix Halo Linux lane targeted the offline static `native`
host and never ran; point it at the DevLab Dispatch pool the per-PR lane uses
and pin that in the workflow contract test.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs: say which per-PR lanes skip pull_request

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs: release-candidate gate runs the full self-hosted matrix

The row said 'the same per-PR lanes', which now means the narrowed
three-lane set. A push to release/** still runs all five lanes. Also drop
the paragraph that said the Ubuntu nightly lane still pins the offline
`native` host.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: pin the whole per-PR gate; guard the nightly Strix job; fix stale coverage docs

- Contract test asserts the full if: gate of e2e-gpu-strix-ubuntu and
  e2e-gpu-w7900, so the pull_request exclusion must sit in the non-dispatch
  branch and nothing else about the event may change.
- New test: the nightly Strix Ubuntu job sets no static-host paths and reads
  no CARGO_HOME/RUSTUP_HOME/TMPDIR/PIP_CACHE_DIR it does not set.
- Docs: the cucumber README, testing.md, the nightly-channel paragraph and the
  Triggers paragraph no longer claim Strix Ubuntu / W7900 run per PR.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: tighten the nightly Strix path guard; fix remaining per-PR Strix/W7900 claims

- nightly_strix_ubuntu_job_holds_no_static_host_paths now also rejects any
  $HOME read other than the $HOME/.cargo/bin Rust bootstrap.
- Docs and workflow comments no longer call the Strix Ubuntu / W7900 lanes
  per-PR; reflow two overlong prose lines.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: drop W7900 from merge_group; fix the last per-PR Strix claims

- e2e-gpu-w7900 no longer runs in the merge queue (one runner on borrowed
  quota, the queue does not wait for it); it still runs on push, dispatch
  and nightly. The contract test pins each lane's whole gate and that the
  W7900 job holds no E2E_MERGE_QUEUE opt-in.
- Reword the nightly Strix comment that called e2e-gpu-strix-ubuntu per-PR,
  and two comment-only leftovers in xtask and the cucumber crate.
- The nightly path guard matches whole words, so HF_TMPDIR is not TMPDIR.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: drop e2e-gpu-strix-ubuntu from merge_group

Over 7 days 35% of its merge-queue jobs (98 of 281) were cancelled at the
45 min cap. The check is not required and the queue does not wait for it.
It still runs on push (main, release/**), dispatch and nightly. The contract
test pins both lanes' whole gate and that neither job opts into
@merge-queue scenarios; docs updated.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs, ci: make every merge_group claim match the workflow

Strix Ubuntu and W7900 skip pull_request and merge_group and run on push,
dispatch and nightly. Fix the nightly.yml comment and the
ci-hardware-testing.md rows that said otherwise, state that the merge queue
has no native Linux GPU lane (Windows Strix and WSL2 only), and wrap the
comment lines this PR left long.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs, tests: fix stale merge-queue and per-PR claims, re-wrap two lines

The merge_queue field doc said the serve matrix runs on every lane but
MI300X; only Windows Strix and WSL2 run @merge-queue serves now. The AIM
serve comment in model_serving.feature still called Strix Ubuntu a per-PR
job.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: pin the PR-skipping lanes in ROCm#1032's closed-PR if table

After rebasing onto ROCm#1032, closing_a_pr_supersedes_its_self_hosted_runs
pinned the strix-ubuntu and w7900 gates without the pull_request and
merge_group exclusions this PR adds. Pin them with the exclusions.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs, ci: drop the stale Docker 'unconfirmed' and per-PR budget claims

The nightly Strix lane is on the pool where Docker was reached (serve-26
passed on both legs); say so, and note serve-27's stop failure. The
include_nightly budget comment no longer calls the push-run budget per-PR.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* tests: say Docker is reachable on the nightly Strix pool, and serve-27's stop failure

The model_serving.feature AIM comment still called Docker on the nightly Strix
Halo host unconfirmed; dispatch run 43025895270 reached it (serve-26 passed on
both legs) and serve-27 fails at stopping the service.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* tests: declare serve-aim-lifecycle-parity an expected failure on Strix Halo Ubuntu

Its stop step fails with 'Local server stop not confirmed' (tracked in
rocm-cli#1058), so the nightly lane reports it as xfail instead of an
unexpected regression. Point the nightly.yml, docs and feature comments at it.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs, ci: reference rocm-cli#1058 instead of pinning a run ID and failure text; say Windows pool

The serve-27 evidence lives in the tracking issue, so nightly.yml, docs/testing.md,
model_serving.feature and expectations.toml no longer quote a run ID or the
failure string. A PR runs two Windows-pool jobs (Windows and WSL2), so say
one Windows GPU pool, not node.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* tests, docs: keep WSL2 out of the serve-27 xfail row; fix two stale ci-hardware-testing claims

The row matched os=linux, which WSL2 also reports on gfx1151, so a WSL host that
gained Docker and passed the scenario would XPASS. Add is_wsl = false. Reword the
e2e-report coverage line and the pre-warmed runtime paragraph, which still named
per-PR platforms that no longer all run per PR and the static native hosts.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci, docs: pin that e2e-wsl and Windows Strix set E2E_MERGE_QUEUE; say WSL2 runs the merge-queue serves

The Triggers paragraph said the heavy serves run only on the Windows Strix Halo
lanes while other docs said WSL2 too. The workflow sets E2E_MERGE_QUEUE on both
e2e-gpu-strix-windows and e2e-wsl, so reword the paragraph and pin both in the
contract test. Add a one-line symptom to the serve-27 xfail reason.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

---------

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants