Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -787,7 +787,8 @@ jobs:
# CARGO_HOME override (pool hosts have room under the default paths, unlike
# the full-partition static host this replaced), though CARGO_TARGET_DIR
# still points at RUNNER_WORKSPACE to dodge checkout's `git clean -ffdx`
# mid-job. Non-blocking while this hardware is proven out.
# mid-job. Non-blocking: its check is not a required status check (see the
# header), but a failure still turns it red.
e2e-gpu-strix-ubuntu:
name: E2E tests (Strix Halo, Ubuntu, ${{ matrix.channel }}, ${{ matrix.sdk_version }})
# 45min: see e2e-gpu — one collapsed job runs all serves + per-scenario
Expand Down Expand Up @@ -972,7 +973,8 @@ jobs:

# First real Windows GPU coverage: the Strix Halo Windows 11 runner. The
# existing windows-debug/windows-release jobs use GitHub-hosted windows-latest,
# which has no GPU. Non-blocking so it never gates the PR.
# which has no GPU. Non-blocking: its check is not a required status check
# (see the header), but a failure still turns it red.
e2e-gpu-strix-windows:
name: E2E tests (Strix Halo, Windows, ${{ matrix.sdk_version }})
# 45min: see e2e-gpu — one collapsed job runs all serves + per-scenario
Expand Down Expand Up @@ -2007,10 +2009,10 @@ jobs:
# A lane that dies before writing report.json (preflight, build, a crash
# before results exist) uploads nothing, and `discover()` can only render
# the reports that arrived, so that lane would drop out of the grid with
# no row and no FAIL. On a `continue-on-error` lane nothing else is red
# either. So check it here: every lane that ran (any `needs` result but
# `skipped`, and not self-skipped as `stale`) must have uploaded its report
# artifact. This reads the run's
# no row and no FAIL. The lane's own job is red, but nothing in the grid
# would say which report is absent. So check it here: every lane that ran
# (any `needs` result but `skipped`, and not self-skipped as `stale`) must
# have uploaded its report artifact. This reads the run's
# artifact list rather than e2e-artifacts/, because download-artifact
# flattens a single match into the root and loses its name. The list also
# holds earlier attempts' artifacts, so a re-run that uploads nothing can
Expand All @@ -2022,9 +2024,9 @@ jobs:
# stand-in for the whole lane. LANE_REPORTS, and this script's
# behaviour,
# are pinned by xtask's workflow contract tests. The annotation quotes the
# lane's `needs` result verbatim; on a `continue-on-error` lane it may read
# `success` even though the lane failed, so the absent artifact is the
# signal, not that value. `!cancelled()`, not `always()`: a run
# lane's `needs` result verbatim, but the absent artifact is the signal,
# not that value: a lane whose other leg failed reads `failure` whether or
# not this leg uploaded. `!cancelled()`, not `always()`: a run
# superseded mid-flight cuts lanes off before upload, which is not a gap.
- name: Check every lane that ran uploaded its report
if: ${{ !cancelled() }}
Expand Down
40 changes: 22 additions & 18 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -404,14 +404,20 @@ jobs:
# to stay fast.
# Runs unconditionally on the schedule/dispatch (independent of the nightly
# BUILD skip above — a regression run is worth doing even with no new commits).
# Non-blocking, mirrors e2e-selfhosted.yml's e2e-gpu job (minus its PR-only
# stale-run guard) but sets E2E_INCLUDE_NIGHTLY.
# Mirrors e2e-selfhosted.yml's e2e-gpu job (minus its PR-only stale-run
# guard) but sets E2E_INCLUDE_NIGHTLY.
# Like every nightly lane, it sets no `continue-on-error`: a failing leg fails
# this job and the Nightly run, and `needs` downstream reads `failure`. The
# report and release-readiness jobs below still run on such a night. The E2E
# wiki page is updated only when e2e-report-nightly succeeded (it fails, for
# example, when a lane that ran uploaded no report) and the other guards in
# publish-e2e-wiki's `if:` hold; otherwise the previous page stays in place.
# Nightly gates no merge or tag, so a red run blocks nothing; it only reports.
e2e-gpu-nightly:
name: E2E tests (MI300X, incl. nightly-only)
timeout-minutes: 90
# `mi300x`, not `amd-gpu` — see the same pin on e2e-selfhosted.yml's e2e-gpu.
runs-on: [self-hosted, linux, mi300x]
continue-on-error: true
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -530,7 +536,6 @@ jobs:
# the 12h run cap and the lane never ran. Pool hosts are cold,
# so there are no host-specific HOME or cache-dir overrides.
runs-on: [self-hosted, linux, devlab-dispatch, strix-halo]
continue-on-error: true
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -620,7 +625,6 @@ jobs:
name: E2E tests (Strix Halo, Windows, incl. nightly-only)
timeout-minutes: 90
runs-on: [self-hosted, windows, devlab-dispatch, strix-halo]
continue-on-error: true
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -890,7 +894,6 @@ jobs:
# conservative starting budget, tune down after a measured run.
timeout-minutes: 120
runs-on: [self-hosted, windows, devlab-dispatch, strix-halo]
continue-on-error: true
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -1341,8 +1344,8 @@ jobs:
# `error` makes that a failed step here, in the lane that owns the
# copy, where default `warn` would upload nothing without a word.
# The consolidated report job's missing-report check flags the absent
# artifact too, and is the only signal on a `continue-on-error` lane:
# a failed step there does not turn the lane's check red. Only the
# artifact too, so the gap is named in the consolidated report job as
# well as here, where the failed step turns this lane red. Only the
# WSL lanes need this guard: every other lane writes report.json
# directly in-job, with no guest-to-host copy step that can fail on
# its own. A harness preflight refusal is not one of those:
Expand Down Expand Up @@ -1375,9 +1378,9 @@ jobs:
- e2e-gpu-nightly-strix-windows
- e2e-wsl-nightly
# Unconditional `always()`, unlike the per-PR workflow: there is no paths
# filter to consult on a schedule, and the lanes are `continue-on-error`, so
# a failed lane must still be reported. A nightly that broke is precisely
# when the grid is worth reading.
# filter to consult on a schedule, and a failed lane fails its job, which
# would skip a job with the default `success()` condition. A nightly that
# broke is precisely when the grid is worth reading.
if: always()
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down Expand Up @@ -1534,10 +1537,11 @@ jobs:
# step publishes it, so it shows as an EMPTY row (a post-run lane failure
# as a FAIL row with lane_failure), not as missing. `discover()` can only
# render the reports that arrived, so a lane that uploads nothing would
# drop out of the grid with no row and no FAIL. On a `continue-on-error`
# lane nothing else is red either. So check it here: every lane that ran
# (any `needs` result but `skipped`, and not self-skipped as `stale`)
# must have uploaded its report artifact. This reads the run's
# drop out of the grid with no row and no FAIL. The lane's own job is red,
# but nothing in the grid would say which report is absent. So check it
# here: every lane that ran (any `needs` result but `skipped`, and not
# self-skipped as `stale`) must have uploaded its report artifact. This
# reads the run's
# artifact list rather than e2e-artifacts/, because download-artifact
# flattens a single match into the root and loses its name. The list also
# holds earlier attempts' artifacts, so a re-run that uploads nothing can
Expand All @@ -1548,9 +1552,9 @@ jobs:
# checked explicitly, comma-separated, rather than reading one leg as a
# stand-in for the whole lane. LANE_REPORTS, and this script's behaviour,
# are pinned by xtask's workflow contract tests. The annotation quotes the
# lane's `needs` result verbatim; on a `continue-on-error` lane it may read
# `success` even though the lane failed, so the absent artifact is the
# signal, not that value. `!cancelled()`, not `always()`: a run
# lane's `needs` result verbatim, but the absent artifact is the signal,
# not that value: a lane whose other leg failed reads `failure` whether or
# not this leg uploaded. `!cancelled()`, not `always()`: a run
# superseded mid-flight cuts lanes off before upload, which is not a gap.
- name: Check every lane that ran uploaded its report
if: ${{ !cancelled() }}
Expand Down
45 changes: 36 additions & 9 deletions docs/ci-hardware-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,7 +417,10 @@ via `cargo xtask release-readiness`: it gates on the nightly Linux/Windows
build results, the consolidated E2E failure total, and the `e2e-report-nightly`
job's own result (so a lane that dies before uploading a report still fails
the gate), and buckets commits since the last release tag into
features/fixes/breaking changes. Unlike the wiki-publish jobs above, the
features/fixes/breaking changes. It reads `e2e-report-nightly`'s result and
the report's failure total, not the lane jobs' own results, so it can read GO
while a lane is red for a reason the report does not record as a failure.
Unlike the wiki-publish jobs above, the
`release-readiness-nightly` and `release-readiness-narrative` jobs have no
`main`/repo guard — they run on any branch this workflow is dispatched on.
The narrative job drafts prose from those facts via `actions/ai-inference`
Expand Down Expand Up @@ -706,14 +709,38 @@ The self-hosted jobs — `e2e-gpu`, `e2e-gpu-strix-ubuntu`,
`e2e-gpu-strix-windows`, and `e2e-wsl` — all run with
check names that are absent from branch protection's required-status-check
list (see below), so a hardware failure never gates a PR merge no matter how
it reports. They run with `continue-on-error: false`, so a real
regression shows red instead of always green. Their results also surface in
the self-hosted consolidated report for visibility. A lane that ran but uploaded
no report (it died before writing its results) fails the consolidated report
job, naming the lane, rather than silently dropping out of the grid (a lane that
skipped itself as stale is the exception); on a
`continue-on-error: true` lane that is the only red signal for such a failure.
The nightly report job applies the same check to every nightly lane.
it reports. Each sets `continue-on-error: false`, so a real regression shows
red instead of always green. Their results also surface in the self-hosted
consolidated report for visibility. A lane that ran but uploaded no report (it
died before writing its results) fails the consolidated report job, naming the
lane, rather than silently dropping out of the grid (a lane that skipped itself
as stale is the exception). The nightly report job applies the same check to
every nightly lane.

No E2E lane in [`nightly.yml`](../.github/workflows/nightly.yml) sets
`continue-on-error`, on the job or on any step, so a leg that fails turns its
lane and the whole Nightly run red. (Outside the lanes it is still used: by
the `release-readiness-narrative` job, by steps of
`publish-release-readiness-wiki`, and by the queue-import steps of
`e2e-report-nightly`, which must never stop the report.) Job-level
`continue-on-error: true` would instead make the failed lane read `success`,
both in `needs.<lane>.result` and in the run's conclusion, and a step-level one
would let the failing step pass the job. Nightly gates no merge or tag either
way. On a red
night the consolidated report and release-readiness jobs still run, and the E2E
wiki page is updated only when `e2e-report-nightly` succeeded and
`publish-e2e-wiki`'s other guards hold; the report job fails, for example, when
a lane that ran uploaded no report, and then the previous page stays in place.
`e2e_lanes_do_not_mask_failures_with_continue_on_error` in
[`workflow_contract.rs`](../xtask/src/workflow_contract.rs) enforces this with a
line scan of every self-hosted lane in both workflows that is fail-closed for
the spellings it reads: outside comments, any line mentioning
`continue-on-error` that is not exactly `continue-on-error: false` (or
`- continue-on-error: false`) fails the test, even inside a `run:` script, and
the failure names each such line with its lane. There is no exception list: no
self-hosted E2E lane in either workflow may carry such a line. What the scan
cannot see, including spellings it misses, is listed in the
`continue_on_error_lines` doc comment.

A lane the harness failed after its run (no scenario executed, or no GPU
scenario, `@requires-gpu` or `@requires-multi-gpu`, on an `E2E_EXPECT_GPU=1`
Expand Down
5 changes: 3 additions & 2 deletions tests/e2e-cucumber/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,10 +246,11 @@ exercises `@requires-wsl` scenarios; `@requires-bare-metal` scenarios resolve to
skip there.

The nightly workflow runs the MI300X and the Strix Halo hardware from the
table above as non-blocking lanes (`e2e-gpu-nightly`, `e2e-gpu-nightly-strix`,
table above, in its own lanes (`e2e-gpu-nightly`, `e2e-gpu-nightly-strix`,
`e2e-gpu-nightly-strix-windows`, `e2e-wsl-nightly`) with
`E2E_INCLUDE_NIGHTLY=1`, then consolidates them into the same cross-platform
grid. R9700, MI350P and W7900 no longer have a self-hosted nightly lane in
grid. They gate no merge, but a failing lane turns the Nightly run red.
R9700, MI350P and W7900 no longer have a self-hosted nightly lane in
`nightly.yml`: they run on the queue ([nightly-on-the-queue.md](../../docs/nightly-on-the-queue.md)),
and the nightly report adds the queue's lanes to that grid. The shared large-model scenario follows each lane's serve engine: it
serves `Qwen/Qwen3.6-27B` through vLLM on the Instinct lanes (MI300X, MI350P),
Expand Down
6 changes: 3 additions & 3 deletions xtask/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -378,9 +378,9 @@ pub(crate) enum Command {
/// That job already fails if any E2E lane that ran didn't upload its
/// `report.json` (preflight failure, timeout, runner loss), which would
/// otherwise just vanish from the consolidated E2E report's totals,
/// reading as zero failures. Gating on this job's result (rather than
/// each lane job's own `continue-on-error`-masked result, which is
/// always `success`) covers every lane, present and future, for free.
/// reading as zero failures. Gating on this job's result covers every
/// lane, present and future, with no second copy of the lane list. The
/// lane jobs' own results are not read (see docs/ci-hardware-testing.md).
/// Caveat: a re-run can pass on an earlier attempt's artifact, so this
/// can miss a lane.
#[arg(long)]
Expand Down
35 changes: 18 additions & 17 deletions xtask/src/release_readiness.rs
Original file line number Diff line number Diff line change
Expand Up @@ -142,18 +142,17 @@ fn assemble_facts(
Some(_) => {}
}
// The Total row above only counts platforms whose `report.json` actually
// got uploaded — a GPU lane that dies before uploading (preflight
// failure, timeout, runner loss) just drops out of it, silently reading
// as zero failures. Every lane is `continue-on-error`, so the lane job's
// own `needs.<job>.result` reads `success` even when it failed; gating on
// that (as an earlier version of this code did) can never fire. Gate on
// `e2e-report-nightly`'s result instead: its "every lane that ran
// uploaded its report" check (`nightly.yml`, `LANE_REPORTS`) already
// fails that job when a lane is silently missing, covering every lane in
// `LANE_REPORTS` — including ones added later — for free (a re-run can
// pass on an earlier attempt's artifact, so it can still miss a lane). Unlike the
// build gates, this job is `if: always()` and has no legitimate
// no-op-night skip, so `skipped` isn't accepted here.
// got uploaded — a GPU lane that dies before uploading (preflight failure,
// timeout, runner loss) just drops out of it, silently reading as zero
// failures. `e2e-report-nightly`'s "every lane that ran uploaded its
// report" check (`nightly.yml`, `LANE_REPORTS`) fails that job when a lane
// is silently missing, covering every lane in `LANE_REPORTS` — including
// ones added later — without a second copy of the lane list here (a re-run
// can pass on an earlier attempt's artifact, so it can still miss a lane),
// so gate on that job's result. The lane jobs' own results are not read;
// docs/ci-hardware-testing.md's readiness paragraph says what that leaves
// out. Unlike the build gates, this job is `if: always()` and has no
// legitimate no-op-night skip, so `skipped` isn't accepted here.
if e2e_report_job_result != "success" {
failing_gates.push(format!("E2E (lane report check: {e2e_report_job_result})"));
}
Expand Down Expand Up @@ -430,11 +429,13 @@ fn queue_absent_reason(markdown: &str) -> Option<String> {
///
/// The second part exists because two kinds of failed row contribute `0` to
/// the `Fail` column, so the Total row's own Fail cell has no record of them:
/// an `EMPTY` platform (no `report.json`, or one that's empty/unparseable),
/// and a `FAIL` platform whose only problem is the harness's post-run lane
/// check (`lane_failure` in its `platform.json`, e.g. "0 scenarios executed"),
/// which fails no scenario cell. A `FAIL` row with a nonzero own `Fail` cell
/// is already folded into the Total row and is not counted again.
/// an `EMPTY` platform (a `report.json` with no scenarios, or one that is empty
/// or unparseable, with no `platform.json` beside it; a missing `report.json`
/// gives no row at all), and a `FAIL` platform whose only problem is the
/// harness's post-run lane check (`lane_failure` in its `platform.json`, e.g.
/// "0 scenarios executed"), which fails no scenario cell. A `FAIL` row with a
/// nonzero own `Fail` cell is already folded into the Total row and is not
/// counted again.
///
/// A row for a shared-queue lane the queue does not block on (its platform
/// cell carries [`e2e_report::ADVISORY_QUEUE_MARKER`]) counts for neither: the
Expand Down
Loading
Loading