Skip to content

ci(nightly): let a failing GPU lane fail the Nightly run (ROCMAI-444) - #1046

Draft
rominf wants to merge 5 commits into
mainfrom
ROCMAI-444-ci-nightly-unmask-gpu-lanes
Draft

rominf wants to merge 5 commits into
mainfrom
ROCMAI-444-ci-nightly-unmask-gpu-lanes

Conversation

@rominf

@rominf rominf commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The scheduled Nightly run reported success on nights when most of its GPU E2E lanes failed. Every self-hosted E2E lane in nightly.yml set job-level continue-on-error: true, and at job level that does more than let the run carry on: a failed lane reads success in needs.<lane>.result downstream and in the run's own conclusion. Outside the lane's own log, a broken lane looked exactly like a passing one.

Evidence from this repository's Actions history:

  • Nightly run 40642561487 concluded success while 12 of its 14 E2E lane legs concluded failure on real unexpected failures. The MI300X release leg, for example, ends with Reconciliation: 5 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 2 unexpected failure(s). and error: E2E suite failed. The consolidated report and wiki-publish jobs both succeeded.
  • In run 40827922228, the consolidated report job printed its needs context: e2e-gpu-nightly, e2e-gpu-nightly-mi350p and e2e-gpu-nightly-strix-windows all read "result": "success", although both legs of each had failed.

This change removes continue-on-error from all seven nightly E2E lanes, so a lane with an unexpected failure fails its job and the Nightly run, and adds a contract test that keeps every self-hosted E2E lane unmasked, at job and step level, in both nightly.yml and e2e-selfhosted.yml.

Why removing it is safe

The masking bought nothing the workflow did not already have:

  • Each lane is its own job with fail-fast: false, so one failing lane or leg does not cancel the others; all lanes still run to completion.
  • e2e-report-nightly runs on if: always() and release-readiness-nightly on !cancelled(), so the consolidated report and the release-readiness jobs still run on a red night.
  • Nothing consumes the Nightly run's conclusion: there is no workflow_run trigger on it and no "latest successful nightly" lookup. Nightly gates no merge and no tag, so a red run blocks nothing; it reports.
  • The suite exits non-zero only on unexpected failures. Known failures recorded in tests/e2e-cucumber/expectations.toml do not fail a lane.

Alternative considered

The original suggestion was to make xtask e2e-report exit non-zero on failures and leave the lanes masked. That was rejected: it would block the support-matrix wiki publish on exactly the nights the matrix is most useful, it would change a binary shared with ci.yml and e2e-selfhosted.yml, and it would still miss harness and preflight failures that never produce a report, which the lane's own exit code catches. The case of a lane that ran but uploaded no report is already caught by the report job's "Check every lane that ran uploaded its report" step.

What changes

  • nightly.yml: continue-on-error: true is dropped from e2e-gpu-nightly, -rad3, -mi350p, -w7900, -strix, -strix-windows and e2e-wsl-nightly. Comments that justified always(), the missing-report check and the WSL copy guard in terms of masked lanes are reworded.
  • Contract test e2e_lanes_do_not_mask_failures_with_continue_on_error in xtask/src/workflow_contract.rs. It line-scans the job block of every self-hosted E2E lane in nightly.yml and e2e-selfhosted.yml, and fails on any line, at job or step level, that mentions continue-on-error and is not exactly continue-on-error: false (or - continue-on-error: false). That is fail-closed for the spellings it reads: a true value, an expression, a quoted key or value, a flow mapping, and even text inside a run: script are all reported. There is no allow-list; no self-hosted E2E lane in either workflow may be masked. The scan does not parse YAML, so it has known limits (a quoted # cuts the rest of a line, the match is case-sensitive, a merge key from outside the job block is not followed, a comment indented like a job id ends the block); these are listed in the continue_on_error_lines doc comment and pinned as rows of a unit test, so they are documented behaviour rather than silent holes. The test also pins e2e-report-nightly to if: always(), since with unmasked lanes the default success() condition would skip the report on the night it matters, and it requires a known lane in each workflow to be found so it cannot pass vacuously.
  • What a red night does, stated where it is described: the E2E wiki page is updated only when e2e-report-nightly succeeded and publish-e2e-wiki's other guards (run not cancelled, repository, main ref) hold. That includes nights when a lane failed but every lane that ran uploaded its report; when a lane uploaded no report the report job fails and the previous page stays in place.
  • Release readiness, comments and docs only, no logic change: the gate reads e2e-report-nightly's result and the consolidated report's failure total, not the lane jobs' own results, so it can read GO while a lane is red for a reason the report does not record as a failure. That is now stated in docs/ci-hardware-testing.md, xtask/src/release_readiness.rs and the --e2e-report-job-result help text, replacing the claim that the lane results are always success and so cannot be read.
  • Stale comments corrected: "the lanes are continue-on-error" in ci.yml, e2e-selfhosted.yml, xtask/src/main.rs and existing contract-test messages; "non-blocking while this hardware is proven out" on the e2e-selfhosted Strix lanes, which run with continue-on-error: false and are non-blocking only because their checks are not required; and nightly lane comments that still described per-PR R9700, MI350P and W7900 counterparts. The R9700 and MI350P lanes were made nightly-only earlier, and ci: remove the W7900 lane from e2e-selfhosted #1066 removed e2e-gpu-w7900 from e2e-selfhosted.yml. docs/ci-hardware-testing.md now states that no nightly lane sets continue-on-error, what job- and step-level masking would do, and how the contract test enforces it; tests/e2e-cucumber/README.md no longer calls the nightly lanes non-blocking, and its existing contract test follows the new wording.

Expected consequence: Nightly will read red whenever any lane has an unexpected failure, until each lane's current unexpected failures are fixed or recorded as known failures; some of those are infrastructure rather than product failures. That red is the intended signal.

Verification

Mutation evidence for the contract test; each mutation turned it red, and it passed again once restored:

  • a job-level continue-on-error: true on a nightly lane;
  • a step-level continue-on-error: true on a nightly lane;
  • continue-on-error: true on an e2e-selfhosted.yml lane;
  • e2e-report-nightly changed away from if: always();
  • the scan itself disabled, which turns the scan's unit tests red.

Results on the head of this branch:

  • cargo test -p xtask: 862 passed.
  • cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo xtask check-module-size, cargo xtask check-architecture-doc and the repository's prek hooks: clean.

Not verified:

  • The end-to-end effect on a real Nightly run. Nightly was not dispatched from this branch, because it builds and stages nightly releases and occupies the GPU fleet; the first scheduled Nightly after merge is where a red conclusion on a failing lane shows.
  • cargo test --workspace --all-targets on the local host has failures in the rocm crate that fail identically on unmodified main on that host (host-specific); this change touches no Rust outside xtask.
  • cargo xtask smoke stops at a step that needs a nested bubblewrap sandbox, which the local host cannot provide.
  • actionlint, lychee and the license-header check were not run locally; CI covers them.

Scope

  • No Gherkin scenario: the change is CI workflows, xtask tests and docs, and alters nothing a user of the CLI can observe (AGENTS.md §3).
  • Non-lane jobs that use continue-on-error (release-readiness-narrative and two optional steps of publish-release-readiness-wiki) are unchanged; they are meant to be allowed to fail.
  • Merge note: the first three commits fail one contract test on their own, until the fourth widens it to e2e-selfhosted.yml with no allow-list. Please squash-merge.

Follow-ups

  • Release readiness does not read the lane jobs' own results, so it can read GO with a red lane (documented above, not changed here).
  • Lane discovery relies on a literal self-hosted runner label, and the vacuity guard checks only one known lane per workflow. A stronger check would assert that the discovered lanes equal the report job's needs.
  • The always() pin on e2e-report-nightly accepts only always() and ${{ always() }}, so harmless spellings such as one with a trailing comment are rejected.
  • "Non-blocking while proven out" wording remains in a ci.yml comment and in the e2e-report crate's HTML and Markdown output.

Related: #381 (gates the per-PR consolidated report on its lane) and #383 (a lane where every GPU scenario resolves to skip is still reported ok) are the same family of "a lane can look green when it is not".

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No rows reference ROCMAI-444.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation). (N/A)
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (N/A: no files added, renamed or removed.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen. (N/A: no CLI messages changed. The changed contract-test failure messages were read against the checks they accompany.)

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@rominf
rominf force-pushed the ROCMAI-444-ci-nightly-unmask-gpu-lanes branch from d248034 to 77730c9 Compare November 13, 2026 13:32
Every nightly self-hosted E2E lane set job-level continue-on-error, so a
failed leg made needs.<lane>.result and the run's conclusion read
success: a broken lane was indistinguishable from a passing one outside
its own log. The lanes are separate jobs with fail-fast: false and the
consolidated report job runs on always(), so the masking bought nothing.

Drop it from every nightly lane. The report, wiki and release-readiness
jobs still run on a red night, and Nightly gates no merge or tag. A new
workflow contract test pins the set of lanes allowed to mask failures:
none in nightly.yml, only e2e-gpu-w7900 per PR, and that the nightly
report job keeps running on always().

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…444)

The masked-lane contract test only read a job-level continue-on-error,
so a step-level one could still let a failing step pass a GPU lane and
turn the Nightly run green. It now does a fail-closed line scan of every
self-hosted E2E lane, job and step level: any non-comment line naming
the key that is not exactly `continue-on-error: false` marks the lane
masked. The scan's documented limits are pinned as test rows.

The allow-list policy is split out and unit-tested both ways: a mask
the list does not name fails, and so does a list entry that is no
longer masked.

The nightly, release-readiness and wiki comments and
docs/ci-hardware-testing.md now match the workflow guards: the E2E wiki
page is updated only when e2e-report-nightly succeeded, and readiness
reads the report job's result and failure total, not the lane jobs'
own results.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The W7900 lane in e2e-selfhosted.yml no longer runs on pull requests, so
the nightly preflight comment now names it by job and workflow rather
than calling it the per-PR lane. The hardware-testing doc points at the
continue_on_error_lines doc comment as the single list of what the mask
scan cannot see, instead of restating part of it.

The mask-policy failure message now lists only the offending lanes'
continue-on-error lines. An allowed lane's lines are permitted, and
listing them only obscured which line to fix. A test pins this with two
lanes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ROCMAI-444)

#1066 removed e2e-gpu-w7900 from e2e-selfhosted.yml, the only lane the
contract test allowed to stay masked. With no masked lane left, the
per-workflow allow-list and its stale-entry check have nothing to hold,
so drop them: the contract test now fails on any line in a self-hosted
E2E lane of nightly.yml or e2e-selfhosted.yml that may set
continue-on-error to something other than false, and names each such
line with its lane.

The line scan, its pinned limits, the non-vacuity guard and the
e2e-report-nightly always() pin stay. A unit test drives the failure
message through a synthetic masked lane.

The docs and the nightly W7900 lane's comments no longer describe a
masked e2e-selfhosted.yml lane or a per-PR W7900 counterpart.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
No self-hosted E2E lane runs with continue-on-error any more, but the
Strix Ubuntu and Strix Windows lane comments still called the lanes
non-blocking "while this hardware is proven out" and "so it never gates
the PR", and the missing-report check explained itself in terms of a
continue-on-error lane. Say instead that the lanes are non-blocking
because their checks are not required (as the header already states),
and that a failing lane is red but the grid would not say which report
is absent.

Correct the mask_failure doc comment too: it named only one place that
pins a report job to always(); both report jobs are pinned, each by a
named contract test.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-444-ci-nightly-unmask-gpu-lanes branch from 77730c9 to 699935f Compare November 16, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant