Repository navigation
ci: remove the W7900 lane from e2e-selfhosted - #1066
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
The W7900 job targets a runner label nothing else uses, and it has not run on pull_request or merge_group since the one-Linux-node change. Its push, release and dispatch runs sat queued behind the single runner. Remove the job, its dispatch choice, its report artifacts and the consolidated-report entries, and update the workflow contract and docs. W7900 hardware is covered by the queue. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
1548ebd to
f487d20
Compare
|
CI status on the earlier head (1548ebd):
|
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for the cleanup. The workflow change itself looks fine to me and matches the PR description; I found nothing blocking in the YAML. I'm requesting changes only because the docs overstate the result.
- The header comment in
e2e-selfhosted.ymlanddocs/ci-hardware-testing.mdsay W7900 is "covered by the queue". The queue docs andqueue-run.ymldon't mention W7900, and the description itself says thew7900-duallane hasn't run yet. Until it does, W7900 has no coverage on push,release/**or dispatch. Please reword both (inline comments below). docs/testing.md(around line 3069) still describes W7900 as a PR lane. It's outside this diff so I can't comment inline, but it should be updated here too.
Leftovers (the e2e-gpu-w7900-report artifact name, descriptor rows, release_readiness.rs, e2e_report.rs) are deferred to #1063 as the description says; whichever of the two merges second will need a rebase.
CI: the two Security scan checks were red when I looked, with cancelled jobs beside their reruns, so possibly an artefact; worth a rerun. Several other jobs were still running.
| # Ubuntu and W7900 lanes keep their push / dispatch triggers and run nightly | ||
| # (nightly.yml); neither starts on pull_request or merge_group. | ||
| # Ubuntu lane keeps its push / dispatch triggers; it does not start on | ||
| # pull_request or merge_group. There is no W7900 lane: that hardware is covered |
There was a problem hiding this comment.
"that hardware is covered by the queue" is stronger than what's true today: docs/nightly-on-the-queue.md and queue-run.yml don't mention W7900 at this head, and the PR description says the queue's w7900-dual lane hasn't run yet. Suggest wording that matches the description, e.g. "intended to be covered by the queue's w7900-dual lane once it has run".
There was a problem hiding this comment.
Reworded in d5233a6: the header now says W7900 is covered only by the scheduled queue nightly's w7900-dual lane, not on push, release/** or dispatch. One fact changed since the description: the 11-13 queue nightly (https://github.com/ROCm/rocm-cli/actions/runs/43351651832) did run w7900-dual: it got capacity at once and finished Done, exit 0 (advisory lane, 13 files collected), so I no longer say it has not run.
| DevLab Dispatch pool labels, `e2e-gpu-nightly-w7900`). | ||
| Ubuntu lane (`e2e-gpu-strix-ubuntu`) no longer starts on `pull_request` or in | ||
| the merge queue; it keeps its `push`/dispatch triggers. There is no W7900 lane | ||
| in `e2e-selfhosted.yml` any more: that hardware is covered by the queue (below). |
There was a problem hiding this comment.
Same claim here: "covered by the queue (below)", but nothing below names W7900. Please reword to say W7900 has no coverage until the queue lane runs, so the doc doesn't promise coverage that doesn't exist yet.
There was a problem hiding this comment.
Reworded in d5233a6: W7900 is covered only by the scheduled queue nightly's w7900-dual lane (advisory), not on push, release/** or dispatch, and the first run is cited (https://github.com/ROCm/rocm-cli/actions/runs/43351651832, Done, exit 0). docs/nightly-on-the-queue.md omitted the lane, so it now names w7900-dual and says it is the only scheduled W7900 coverage.
nowycondro
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (base 4221d665…head f487d208). No ticket named in the description beyond cross-references to sibling/predecessor PRs (#1018, #1063); this change matches its stated scope of removing the W7900 lane from e2e-selfhosted.yml only.
Blocking
- Doc paragraph still attributes a W7900 artifact name to a workflow that no longer produces it —
docs/ci-hardware-testing.md:324-332
The "lane artifacts are named canonically (...)" paragraph listse2e-gpu-w7900-reportamong names produced "inci.ymlande2e-selfhosted.yml". Verified at the base commit that this was true (the now-deleted job uploaded exactly that name); after this PR removes the wholee2e-gpu-w7900job frome2e-selfhosted.yml, that literal artifact name is only reachable vianightly.yml's channel-matrix upload (e2e-gpu-w7900-release-report/-nightly-report, which the project's own artifact-naming test strips to this base name for comparison). The sentence needs to either drop this entry or correctly attribute it tonightly.yml. The paragraph's next sentence also explains whyrad3/mi350phave no-n-1/-n-2entries (nightly-only lanes) but doesn't mention thate2e-gpu-w7900-reportis now in the exact same position, despite losing its own-n-1/-n-2entries in this very diff.
Confidence 85 · mechanical · Fix: reword to scopee2e-gpu-w7900-reporttonightly.yml(or explain it's kept only for the cross-workflow comparison test), and fold it into the nightly-only explanation alongside rad3/mi350p.
Non-blocking
- Narrative comments in the contract-test file still say "eight" after the assertions were updated to seven —
xtask/src/workflow_contract.rs:2836-2837, 2888, 2897-2898, 2920, 2936, 2969, 2981, 4315
Theassert_eq!counts inevery_native_linux_preflight_delegates_to_the_shared_scriptand its sibling tests were correctly bumped from 8 to 7, but several nearby prose comments explaining the same invariant ("the eight native Linux preflights are now one script plus a floor", "the eight are one gate in name only", etc.) were left saying "eight". These counts were accurate pre-PR and simply weren't touched here.
Confidence 80 · mechanical · Fix: update the remaining "eight" mentions to "seven" for consistency with the assertions right next to them.
Decisions for the author
- Stale "mirrors the e2e-gpu-w7900 lane... in e2e-selfhosted.yml" comment in
nightly.yml— answered-by-intent
The still-existinge2e-gpu-nightly-w7900job's header comment says it "Mirrors the e2e-gpu-w7900 lane (which no longer runs on pull_request) in e2e-selfhosted.yml" and "Keep the two in step" — both go stale the moment this PR merges, since that lane is now gone entirely rather than merely restricted. The PR description states the linked follow-up "removes the W7900 nightly job" outright, which would delete this comment along with it. Would otherwise be a blocking doc-accuracy finding; worth confirming that follow-up lands soon so the staleness window stays short. - Whether the static nightly W7900 job becomes redundant with the shared GPU queue — tradeoff
The new comment ine2e-selfhosted.ymlframes the hardware as now "covered by the queue," whilenightly.ymlstill runs its own dedicated, differently-triggerede2e-gpu-nightly-w7900job. If the queue's lane is meant to be the long-term coverage, worth confirming whether the static nightly job is also being retired, or intentionally kept running in parallel for a while.
Positive signals
- Unusually thorough propagation for a lane removal: the
workflow_dispatchchoice list, the consolidated-reportneedslist andLANE_REPORTSmap, the pinned event-gate test, and the preflight-lane count were all updated together in one pass, and the one genuinely cross-cutting case was handled correctly (theCANONICAL_REPORT_ARTIFACTSlist kept the bare artifact name specifically becausenightly.ymlstill produces it).
Deployment notes
None.
What this covered
Read the full diff (6 files) completely, plus the current full text of .github/workflows/e2e-selfhosted.yml, .github/workflows/nightly.yml, docs/ci-hardware-testing.md, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs and xtask/src/e2e_report.rs at the head commit, and a repository-wide search for remaining w7900 references to check blast radius beyond the touched files. Two independent passes covered (a) the workflow file plus its contract-test, and (b) the docs/README/report-artifact files, and converged on the same doc findings reported above. CI on the PR is green (an earlier security-scan run failure was superseded by a later successful run on the same commit). Not reconciled against any prior review or PR discussion thread.
e2e-selfhosted.yml has no W7900 lane any more, so e2e-gpu-w7900-report is uploaded only by nightly.yml, like the rad3 and mi350p reports. Say so next to them, and bring the lane-count comments in line with the seven lanes the assertions now pin. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Thanks. Taken in 1c99c06:
|
The comment and docs claimed the queue covers W7900, which neither the queue docs nor queue-run.yml said. Name the w7900-dual lane in the queue doc, cite the 11-13 queue nightly where it ran and finished Done, and state that W7900 has no coverage on push, release/** or dispatch. Fix the testing guide, which still listed W7900 as a pull-request lane. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Thanks @juhovainio. All three points are in d5233a6 (together with the earlier bot-review fixes in 1c99c06):
|
juhovainio
left a comment
There was a problem hiding this comment.
Looks good to me. The deleted job, its needs entry, the LANE_REPORTS row and the dispatch option all come out together, and the workflow-contract counts (8 to 7, "all four lanes") match what's left. No action pins changed and nothing internal leaked into the diff. There's no user-visible CLI change, so no scenario is needed.
A few non-blocking things:
- The docs state that the 11-13 queue nightly's
w7900-duallane "finishedDone, exit 0". That run's overall conclusion is failure (exit code 1 in "Run and publish"), and I couldn't confirm the per-lane result. Please verify it or drop the run-specific claim, since it lands in durable docs (also repeated innightly-on-the-queue.mdand the PR body). nightly.yml(around lines 674-676) still says it mirrors thee2e-gpu-w7900lane ine2e-selfhosted.yml. I assume #1063 clears that; if it lands second or not at all, the comment goes stale.- The release-candidate gate loses its W7900 signal before tagging. The PR says so; just confirming that is intended.
- Nit in the
docs/testing.mdhunk: "Of those" after the Lemonade lane list is slightly misleading, since MI300X is the vLLM lane.
CI: everything that had finished at the time of review passes; the affected-crates, Windows, mock E2E and coverage jobs were still running.
| Ubuntu lane (`e2e-gpu-strix-ubuntu`) no longer starts on `pull_request` or in | ||
| the merge queue; it keeps its `push`/dispatch triggers. There is no W7900 lane | ||
| in `e2e-selfhosted.yml` any more: W7900 is covered only by the scheduled queue | ||
| nightly's `w7900-dual` lane (below; advisory), not on `push`, `release/**` or |
There was a problem hiding this comment.
"(below; advisory)" - nothing below describes w7900-dual; it is only described in nightly-on-the-queue.md. Link that doc instead.
There was a problem hiding this comment.
| in `e2e-selfhosted.yml` any more: W7900 is covered only by the scheduled queue | ||
| nightly's `w7900-dual` lane (below; advisory), not on `push`, `release/**` or | ||
| dispatch. Its first run, the 11-13 queue nightly | ||
| ([run 43351651832](https://github.com/ROCm/rocm-cli/actions/runs/43351651832)), got capacity at once and finished `Done`, exit 0. |
There was a problem hiding this comment.
Run 43351651832 shows an overall conclusion of failure (exit code 1 in "Run and publish"), and I couldn't confirm the w7900-dual lane itself "finished Done, exit 0". Please verify from the per-lane results, or soften/drop the run-specific claim, since it lands in durable docs.
There was a problem hiding this comment.
Checked against the run's per-lane results rather than its overall conclusion. The run does conclude failure, from the blocking lanes: the lane matrix in the Run and publish job log (and the queue-run-results artifact) lists mi300x Failed, exit 1, and r9700s Failed, with no files collected, and both are in blocking: "mi300x,r9700s". In the same matrix w7900-dual is phase Done, exit 0, Exported, advisory, and its lane log collected 13 files. (mi325x, mi350p, mi350x and mi355x also failed, as advisory lanes.) So the claim in the docs is only that the lane ran and finished with exit 0, not that its scenarios were all green. https://github.com/ROCm/rocm-cli/actions/runs/43351651832
…same lines Preparation for merging main into this branch without conflicts; the next commits restore this change's own wording on top of the merge. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…ry ones out of the verdict The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X lane is not the self-hosted one) and marks one the queue does not block on as advisory. release-readiness leaves advisory rows out of its failure count and still counts a blocking lane that came back with no result. xtask gains the `e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly platform check goes, since #1066 removed the per-PR W7900 lane it exempted. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X, MI350X and MI355X, which only the queue runs, would never appear. e2e-report-nightly now fetches the newest queue-run-results artifact of a main run that finished in the last 24 hours (and its job log, for the tested branch and each lane's gate), imports it, and opens the matrix with a Queue lanes table. A missing, late or malformed queue run never fails the report: the page says so instead. The workflow contract pins the new steps and the native Linux preflight lane counts for this change plus #1066 (four). The docs describe the queue import, link the queue page instead of "(below; advisory)", and fix the "Of those" wording in testing.md. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…same lines Preparation for merging main into this branch without conflicts; the next commits restore this change's own wording on top of the merge. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…ry ones out of the verdict The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X lane is not the self-hosted one) and marks one the queue does not block on as advisory. release-readiness leaves advisory rows out of its failure count and still counts a blocking lane that came back with no result. xtask gains the `e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly platform check goes, since #1066 removed the per-PR W7900 lane it exempted. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X, MI350X and MI355X, which only the queue runs, would never appear. e2e-report-nightly now fetches the newest queue-run-results artifact of a main run that finished in the last 24 hours (and its job log, for the tested branch and each lane's gate), imports it, and opens the matrix with a Queue lanes table. A missing, late or malformed queue run never fails the report: the page says so instead. The workflow contract pins the new steps and the native Linux preflight lane counts for this change plus #1066 (four). The docs describe the queue import, link the queue page instead of "(below; advisory)", and fix the "Of those" wording in testing.md. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
One conflict: `apps/rocm/src/advised_commands.rs`. Both sides appended entries to the same `NOT_INVOCATIONS` table, so the hunks collided on the trailing `),` they share rather than on any disagreement. #1067 added the `rocm fix --no-such-option; echo $?` entry, which belongs to the "Deliberate negative examples" group already above it; this branch added the ROCMAI-550 `device:`-value entries for README.md and docs/manual-testing.md under their own comment. Both kept, with #1067's entry restored to its group and this branch's block following it. All three entries still name text that exists in those docs. #1066's removal of the W7900 lane from e2e-selfhosted.yml is untouched by this branch and carried through as main left it. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ROCMAI-444) #1066 removed e2e-gpu-w7900 from e2e-selfhosted.yml, the only lane the contract test allowed to stay masked. With no masked lane left, the per-workflow allow-list and its stale-entry check have nothing to hold, so drop them: the contract test now fails on any line in a self-hosted E2E lane of nightly.yml or e2e-selfhosted.yml that may set continue-on-error to something other than false, and names each such line with its lane. The line scan, its pinned limits, the non-vacuity guard and the e2e-report-nightly always() pin stay. A unit test drives the failure message through a synthetic masked lane. The docs and the nightly W7900 lane's comments no longer describe a masked e2e-selfhosted.yml lane or a per-PR W7900 counterpart. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
One conflict: `apps/rocm/src/advised_commands.rs`. Both sides appended entries to the same `NOT_INVOCATIONS` table, so the hunks collided on the trailing `),` they share rather than on any disagreement. #1067 added the `rocm fix --no-such-option; echo $?` entry, which belongs to the "Deliberate negative examples" group already above it; this branch added the ROCMAI-550 `device:`-value entries for README.md and docs/manual-testing.md under their own comment. Both kept, with #1067's entry restored to its group and this branch's block following it. All three entries still name text that exists in those docs. #1066's removal of the W7900 lane from e2e-selfhosted.yml is untouched by this branch and carried through as main left it. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
What
Removes the
e2e-gpu-w7900lane frome2e-selfhosted.yml, so that together with #1063 (which removes the W7900 nightly job) no workflow targets thew7900runner label. Follow-up to #1018, which already took the lane offpull_requestandmerge_group; it still ran on push,release/**and dispatch.Changes
e2e-selfhosted.yml: delete the job, itsw7900dispatch choice, and its entries ine2e-report(needsandLANE_REPORTS); update the header comments.xtaskworkflow_contract: drop the lane from the pinned event gates and the preflight lane counts (8 → 7).xtaske2e_report: drop the-n-1/-n-2W7900 artifact names (the lane was the only uploader).docs/ci-hardware-testing.md,tests/e2e-cucumber/README.mdand a doc comment.Coverage
W7900 is covered only by the scheduled queue nightly's
w7900-duallane (advisory), not on push,release/**or dispatch. That lane has run: the 11-13 queue nightly (https://github.com/ROCm/rocm-cli/actions/runs/43351651832) got capacity at once and finished Done, exit 0.Relationship to #1063
#1063 edits some of the same docs lines and the same preflight lane counts in
workflow_contract.rs(8 → 4 there, 7 here; 3 once both land), so the second to merge needs a small rebase. Thee2e-gpu-w7900-reportartifact name and the descriptor rows are kept here because the nightly W7900 job still uploads them until #1063 merges; no workflow-file hunks overlap.Testing
cargo test -p xtask(857 passed),cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --check.