Repository navigation
Conversation
Contributor
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
rominf
force-pushed
the
ROCMAI-444-ci-nightly-unmask-gpu-lanes
branch
from
November 13, 2026 13:32
d248034 to
77730c9
Compare
Every nightly self-hosted E2E lane set job-level continue-on-error, so a failed leg made needs.<lane>.result and the run's conclusion read success: a broken lane was indistinguishable from a passing one outside its own log. The lanes are separate jobs with fail-fast: false and the consolidated report job runs on always(), so the masking bought nothing. Drop it from every nightly lane. The report, wiki and release-readiness jobs still run on a red night, and Nightly gates no merge or tag. A new workflow contract test pins the set of lanes allowed to mask failures: none in nightly.yml, only e2e-gpu-w7900 per PR, and that the nightly report job keeps running on always(). Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…444) The masked-lane contract test only read a job-level continue-on-error, so a step-level one could still let a failing step pass a GPU lane and turn the Nightly run green. It now does a fail-closed line scan of every self-hosted E2E lane, job and step level: any non-comment line naming the key that is not exactly `continue-on-error: false` marks the lane masked. The scan's documented limits are pinned as test rows. The allow-list policy is split out and unit-tested both ways: a mask the list does not name fails, and so does a list entry that is no longer masked. The nightly, release-readiness and wiki comments and docs/ci-hardware-testing.md now match the workflow guards: the E2E wiki page is updated only when e2e-report-nightly succeeded, and readiness reads the report job's result and failure total, not the lane jobs' own results. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The W7900 lane in e2e-selfhosted.yml no longer runs on pull requests, so the nightly preflight comment now names it by job and workflow rather than calling it the per-PR lane. The hardware-testing doc points at the continue_on_error_lines doc comment as the single list of what the mask scan cannot see, instead of restating part of it. The mask-policy failure message now lists only the offending lanes' continue-on-error lines. An allowed lane's lines are permitted, and listing them only obscured which line to fix. A test pins this with two lanes. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ROCMAI-444) #1066 removed e2e-gpu-w7900 from e2e-selfhosted.yml, the only lane the contract test allowed to stay masked. With no masked lane left, the per-workflow allow-list and its stale-entry check have nothing to hold, so drop them: the contract test now fails on any line in a self-hosted E2E lane of nightly.yml or e2e-selfhosted.yml that may set continue-on-error to something other than false, and names each such line with its lane. The line scan, its pinned limits, the non-vacuity guard and the e2e-report-nightly always() pin stay. A unit test drives the failure message through a synthetic masked lane. The docs and the nightly W7900 lane's comments no longer describe a masked e2e-selfhosted.yml lane or a per-PR W7900 counterpart. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
No self-hosted E2E lane runs with continue-on-error any more, but the Strix Ubuntu and Strix Windows lane comments still called the lanes non-blocking "while this hardware is proven out" and "so it never gates the PR", and the missing-report check explained itself in terms of a continue-on-error lane. Say instead that the lanes are non-blocking because their checks are not required (as the header already states), and that a failing lane is red but the grid would not say which report is absent. Correct the mask_failure doc comment too: it named only one place that pins a report job to always(); both report jobs are pinned, each by a named contract test. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf
force-pushed
the
ROCMAI-444-ci-nightly-unmask-gpu-lanes
branch
from
November 16, 2026 12:19
77730c9 to
699935f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The scheduled Nightly run reported success on nights when most of its GPU E2E lanes failed. Every self-hosted E2E lane in
nightly.ymlset job-levelcontinue-on-error: true, and at job level that does more than let the run carry on: a failed lane readssuccessinneeds.<lane>.resultdownstream and in the run's own conclusion. Outside the lane's own log, a broken lane looked exactly like a passing one.Evidence from this repository's Actions history:
successwhile 12 of its 14 E2E lane legs concludedfailureon real unexpected failures. The MI300X release leg, for example, ends withReconciliation: 5 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 2 unexpected failure(s).anderror: E2E suite failed. The consolidated report and wiki-publish jobs both succeeded.needscontext:e2e-gpu-nightly,e2e-gpu-nightly-mi350pande2e-gpu-nightly-strix-windowsall read"result": "success", although both legs of each had failed.This change removes
continue-on-errorfrom all seven nightly E2E lanes, so a lane with an unexpected failure fails its job and the Nightly run, and adds a contract test that keeps every self-hosted E2E lane unmasked, at job and step level, in bothnightly.ymlande2e-selfhosted.yml.Why removing it is safe
The masking bought nothing the workflow did not already have:
fail-fast: false, so one failing lane or leg does not cancel the others; all lanes still run to completion.e2e-report-nightlyruns onif: always()andrelease-readiness-nightlyon!cancelled(), so the consolidated report and the release-readiness jobs still run on a red night.workflow_runtrigger on it and no "latest successful nightly" lookup. Nightly gates no merge and no tag, so a red run blocks nothing; it reports.tests/e2e-cucumber/expectations.tomldo not fail a lane.Alternative considered
The original suggestion was to make
xtask e2e-reportexit non-zero on failures and leave the lanes masked. That was rejected: it would block the support-matrix wiki publish on exactly the nights the matrix is most useful, it would change a binary shared withci.ymlande2e-selfhosted.yml, and it would still miss harness and preflight failures that never produce a report, which the lane's own exit code catches. The case of a lane that ran but uploaded no report is already caught by the report job's "Check every lane that ran uploaded its report" step.What changes
nightly.yml:continue-on-error: trueis dropped frome2e-gpu-nightly,-rad3,-mi350p,-w7900,-strix,-strix-windowsande2e-wsl-nightly. Comments that justifiedalways(), the missing-report check and the WSL copy guard in terms of masked lanes are reworded.e2e_lanes_do_not_mask_failures_with_continue_on_errorinxtask/src/workflow_contract.rs. It line-scans the job block of every self-hosted E2E lane innightly.ymlande2e-selfhosted.yml, and fails on any line, at job or step level, that mentionscontinue-on-errorand is not exactlycontinue-on-error: false(or- continue-on-error: false). That is fail-closed for the spellings it reads: a true value, an expression, a quoted key or value, a flow mapping, and even text inside arun:script are all reported. There is no allow-list; no self-hosted E2E lane in either workflow may be masked. The scan does not parse YAML, so it has known limits (a quoted#cuts the rest of a line, the match is case-sensitive, a merge key from outside the job block is not followed, a comment indented like a job id ends the block); these are listed in thecontinue_on_error_linesdoc comment and pinned as rows of a unit test, so they are documented behaviour rather than silent holes. The test also pinse2e-report-nightlytoif: always(), since with unmasked lanes the defaultsuccess()condition would skip the report on the night it matters, and it requires a known lane in each workflow to be found so it cannot pass vacuously.e2e-report-nightlysucceeded andpublish-e2e-wiki's other guards (run not cancelled, repository,mainref) hold. That includes nights when a lane failed but every lane that ran uploaded its report; when a lane uploaded no report the report job fails and the previous page stays in place.e2e-report-nightly's result and the consolidated report's failure total, not the lane jobs' own results, so it can read GO while a lane is red for a reason the report does not record as a failure. That is now stated indocs/ci-hardware-testing.md,xtask/src/release_readiness.rsand the--e2e-report-job-resulthelp text, replacing the claim that the lane results are alwayssuccessand so cannot be read.continue-on-error" inci.yml,e2e-selfhosted.yml,xtask/src/main.rsand existing contract-test messages; "non-blocking while this hardware is proven out" on the e2e-selfhosted Strix lanes, which run withcontinue-on-error: falseand are non-blocking only because their checks are not required; and nightly lane comments that still described per-PR R9700, MI350P and W7900 counterparts. The R9700 and MI350P lanes were made nightly-only earlier, and ci: remove the W7900 lane from e2e-selfhosted #1066 removede2e-gpu-w7900frome2e-selfhosted.yml.docs/ci-hardware-testing.mdnow states that no nightly lane setscontinue-on-error, what job- and step-level masking would do, and how the contract test enforces it;tests/e2e-cucumber/README.mdno longer calls the nightly lanes non-blocking, and its existing contract test follows the new wording.Expected consequence: Nightly will read red whenever any lane has an unexpected failure, until each lane's current unexpected failures are fixed or recorded as known failures; some of those are infrastructure rather than product failures. That red is the intended signal.
Verification
Mutation evidence for the contract test; each mutation turned it red, and it passed again once restored:
continue-on-error: trueon a nightly lane;continue-on-error: trueon a nightly lane;continue-on-error: trueon ane2e-selfhosted.ymllane;e2e-report-nightlychanged away fromif: always();Results on the head of this branch:
cargo test -p xtask: 862 passed.cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --all --check,cargo xtask check-module-size,cargo xtask check-architecture-docand the repository's prek hooks: clean.Not verified:
cargo test --workspace --all-targetson the local host has failures in therocmcrate that fail identically on unmodifiedmainon that host (host-specific); this change touches no Rust outsidextask.cargo xtask smokestops at a step that needs a nested bubblewrap sandbox, which the local host cannot provide.actionlint,lycheeand the license-header check were not run locally; CI covers them.Scope
continue-on-error(release-readiness-narrativeand two optional steps ofpublish-release-readiness-wiki) are unchanged; they are meant to be allowed to fail.e2e-selfhosted.ymlwith no allow-list. Please squash-merge.Follow-ups
self-hostedrunner label, and the vacuity guard checks only one known lane per workflow. A stronger check would assert that the discovered lanes equal the report job'sneeds.always()pin one2e-report-nightlyaccepts onlyalways()and${{ always() }}, so harmless spellings such as one with a trailing comment are rejected.ci.ymlcomment and in thee2e-reportcrate's HTML and Markdown output.Related: #381 (gates the per-PR consolidated report on its lane) and #383 (a lane where every GPU scenario resolves to skip is still reported ok) are the same family of "a lane can look green when it is not".
Checklist
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No rows reference ROCMAI-444.)docs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation). (N/A).rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry (cargo xtask check-architecture-docfails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (N/A: no files added, renamed or removed.)