Skip to content

test(e2e): fail lanes that test nothing (ROCMAI-388, ROCMAI-450) - #1068

Queued
rominf wants to merge 6 commits into
mainfrom
ROCMAI-388-fix-e2e-zero-scenario-green
Queued

rominf wants to merge 6 commits into
mainfrom
ROCMAI-388-fix-e2e-zero-scenario-green

Conversation

@rominf

@rominf rominf commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator
  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No rows reference ROCMAI-388, ROCMAI-450 or E2E: a lane where every @requires-gpu scenario resolves to skip should be flagged, not reported ok #383.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation). (n/a: no subcommand or subsystem.)
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (New tests/e2e-cucumber/src/lane_verdict.rs added; the capability.rs entry now lists the ProbeError kinds and the consolidated/mod.rs entry names platform_status.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen. (The new messages are harness and report output, not rocm output. Each lane_verdict test asserts the fatal/non-fatal outcome together with the message; the report's new Status wording is pinned beside a comment naming the tests that prove a lane failure turns the row FAIL.)

Why

A self-hosted GPU lane could go green without testing its GPU, and nothing in the result distinguished that from a lane that exercised everything. Three causes:

  1. A failed capability probe read as "no GPU". The harness probes the host by running rocm examine and rocm engines list, but ignored their exit status. A probe that failed (wrong binary, a crash, empty output) produced the same facts as a host without a GPU, so every @requires-gpu, @requires-multi-gpu and gfx-target-gated scenario resolved to skip. A probe that never returned hung the whole run instead.
  2. A run that executed nothing exited 0. Skipped scenarios never reach report.json, and the harness's only outcome-based exit iterated the results in it, looking for unexpected failures and stale XPASSes. With nothing in it there was nothing to fail. A cucumber --name/--tags filter on the command line also replaces the harness's own scenario filter, so a filter matching nothing gave the same silent green.
  3. The consolidated report could not show such a lane as failed. Its Status column is computed from scenario results, so a lane whose resolved scenarios were all skips read PASS whatever the harness's exit status.

What changes

Lanes declare whether they have a GPU, in E2E_EXPECT_GPU:

Value Where Meaning
1 every self-hosted GPU lane in e2e-selfhosted.yml and nightly.yml, and every lane of scripts/run-e2e-lane.sh (default 1) the lane has a GPU; a run that found or used none is broken
0 e2e-wsl, e2e-wsl-nightly (forwarded into the guest via WSLENV) deliberately no usable GPU: the distro is provisioned without ROCm's WSL driver bridge, which rocm install driver installs under test
unset GitHub-hosted mock lane, local runs no GPU claim

Any other value fails the run rather than being read as "undeclared", so a typo cannot switch the check off. The variable name is one shared constant, e2e_report::EXPECT_GPU_ENV, read by the harness and by the workflow contract test.

Each probe command runs under a deadline, E2E_PROBE_TIMEOUT_SECS (default 120 s; any whole number ≥ 1, anything else uses the default). A probe failure names the command and says which way it failed: it could not be started, exited non-zero (with its exit status and stderr tail), printed nothing, timed out (it is killed and reaped, and the message gives the deadline and the stderr read so far), could not be waited on, or exited while a process it started still held its stdout open. The post-run version probe for the report heading uses the same deadline but fails quietly: it only blanks the version columns.

The harness checks the run before and after the scenarios (new tests/e2e-cucumber/src/lane_verdict.rs):

  • Before any scenario, on every lane: fail on an invalid declaration or a failed probe.
  • Before any scenario, on a 1 lane: fail when the probe found no usable AMD GPU, naming the detected gfx target, WSL flag, driver status and visibility mask the decision was made from.
  • After the run, on every lane: fail when 0 scenarios executed.
  • After the run, on a 1 lane: fail when no @requires-gpu/@requires-multi-gpu scenario executed. GPU scenarios the run's own configuration leaves out (@nightly, @merge-queue, anything outside an E2E_ONLY_LIFECYCLE selection) count as resolved but never as executed, so E2E_EXPECT_GPU=1 with E2E_ONLY_LIFECYCLE=1 and no GPU scenario selected fails. Skipped when a cucumber CLI filter replaced the harness's filter, since which scenarios need a GPU is then unknown.
  • On a 0 lane, notes only: that GPU scenarios are not applicable, how many did not run, and that the declaration is stale if the probe finds a GPU anyway.

Failures print a GitHub ::error:: annotation followed by the plain message, and exit 1; notes print as ::notice::. Annotation data now escapes %, CR and LF, so a probe's multi-line stderr tail cannot truncate or corrupt the workflow command.

The decisions live in pure, unit-tested functions in lane_verdict.rs: preflight (declaration, probe result, GPU presence), the GPU scenario counts, run_failed, finish_run (post-run verdict plus the platform.json manifest), failure_lines/note_line and workflow_command. What stays in tests/e2e.rs is glue: printing, exiting, and writing the file.

A post-run failure reaches the consolidated report. Its reason is written to platform.json as lane_failure. The report then shows that lane as FAIL even though its Fail cell is 0, and gives the reason in the Status cell's tooltip, in a paragraph in the platform's section of the HTML report, and in a "Needs attention: platforms" list (in both the step summary and the HTML report). The scenario-level list that used the same "Needs attention" heading is now "Needs attention: scenarios", so neither renderer shows two identical headings. release-readiness already counts a non-PASS row whose Fail cell is 0 (it did so for EMPTY rows), so such a lane makes the nightly verdict NO-GO; a new test pins that, and that a row which also has failed scenarios is counted once. A shared-queue lane reaches the report through cargo xtask e2e-queue-import, which keeps lane_failure when it copies the lane's platform.json; one the queue marks advisory reads FAIL too but, like its failed scenarios, stays out of the verdict.

A pre-run refusal writes no platform.json, so its row shows EMPTY (red), with the reason only in the job log. The asymmetry with the post-run failure is deliberate in this change, and carrying the pre-run reason into the report is listed under Not in this change.

e2e_report::platform_status / PlatformStatus (one platform's ok flag, Status text and lane failure) is new public API, used by the harness's tests to read back the platform.json that finish_run builds through the report's own loader.

A workflow contract test (every_self_hosted_lane_declares_whether_it_has_a_gpu in xtask/src/workflow_contract.rs) fails when a self-hosted E2E lane does not set the variable in its job-level env:, sets 0 without a recorded reason, or sets a WSLENV that does not name it. It also finds the step that runs cargo xtask e2e and fails when there is none, when that step sets E2E_EXPECT_GPU itself (overriding the job's declaration), and when that step runs the suite in WSL without a WSLENV of its own naming the variable, since a WSLENV set in another step never reaches the guest. It also checks the run-e2e-lane.sh default. The hardware-lanes table in docs/ci-hardware-testing.md gains a "GPU declared" column, held to the workflows by the existing docs-table test.

Docs: docs/testing.md (new "E2E runs that test nothing fail"), docs/ci-hardware-testing.md, docs/nightly-on-the-queue.md, docs/architecture.md and tests/e2e-cucumber/README.md (new "Lane verdict" section and the E2E_PROBE_TIMEOUT_SECS row; it also states that -n/--tags bypasses capability gating).

User-visible and CI-visible changes

No rocm CLI behaviour changes. What changes is which CI runs can go red, and what the report says about them:

  • Every 1 lane (MI300X, Strix Halo Ubuntu and Strix Halo Windows in e2e-selfhosted.yml; the MI300X, Strix Halo Ubuntu and Strix Halo Windows lanes in nightly.yml; the queue lanes) now fails when its probe finds no usable GPU or when no GPU scenario executes. Previously these were green.
  • The Strix Halo Windows lanes now depend on rocm examine reporting the GPU on native Windows. Their GPU preflight already tolerates an absent rocm-smi, so rocm examine is the only thing between those lanes and red.
  • Any lane, including the hosted mock lane, fails when a probe fails (including one running past 120 s, or one leaving its stdout open), and when a run executes no scenario.
  • Locally, cargo xtask e2e -- -n <regex> with a regex matching nothing exits 1 instead of 0.
  • A lane failed by the post-run check reads FAIL in the consolidated report, with its reason, and turns the nightly release-readiness verdict NO-GO (unless it is a shared-queue lane the queue marks advisory).
  • The report's "Needs attention" heading is split into "Needs attention: platforms" and "Needs attention: scenarios".
  • The WSL lanes stay green with GPU scenarios not applicable, and say so in a notice.

Verification

Unit and state-matrix tests, each shown to fail with its target broken (guard reverted or inverted):

  • capability.rs: the probe classifier for every failure kind, the deadline parsing (unset, padded, very large, and empty, zero, negative, fractional, non-numeric or past u64 falling back to 120), the timeout reaching the probe through the same env lookup, kill-and-reap on timeout, and the stderr kept so far. A probe that reads stdin gets EOF at once instead of blocking until its deadline (removing the runner's stdin(Stdio::null()) makes it fail); like its sibling in rocm-core's process/bounded.rs, it catches a missing stdin setting but does not pin null against inherit, which passes whenever the test process's own stdin is already at EOF. Exit statuses are built per platform, so the classification tests also run on the Windows CI job.
  • lane_verdict.rs: one test per row of the state matrix: invalid declaration; probe failed × each declaration; probe OK × GPU present/absent × 1/0/unset; post-run with 0 executed, with GPU scenarios resolved but not executed, none resolved, executed, and with a CLI filter; the 0-lane note's count when some GPU scenarios ran ("4 of 7", which fails when the note reports the resolved count instead); run_failed on each of its three reasons; the escaping of workflow_command and the exact failure and notice lines; and finish_run's manifest read back through e2e_report::platform_status as FAIL with its reason.
  • xtask/src/workflow_contract.rs: the per-lane declaration and WSLENV contract, plus the docs-table column. Adding env: E2E_EXPECT_GPU: "0" to the MI300X lane's suite step fails it ("step Run E2E tests on GPU hardware sets E2E_EXPECT_GPU itself, overriding the job-level declaration"), and so does moving the e2e-wsl lane's WSLENV line from its suite step to the step before it, which keeps a WSLENV naming the variable in the job ("step Run E2E tests on Strix Halo WSL2 runs the suite in WSL but sets no WSLENV naming E2E_EXPECT_GPU").
  • e2e-report: an all-skip lane with lane_failure reads FAIL with its reason in the summary list, Status tooltip and platform paragraph (HTML-escaped); the same lane without one still reads PASS with none of them; a zero-expectation manifest reads FAIL only with a recorded failure; a lane stopped before any scenario reads EMPTY and not ok; both attention headings render.
  • xtask/src/release_readiness.rs: a lane-failure row with a Fail cell of 0 counts once, and one with a failed scenario is not counted twice.

Local runs of the real harness on a Strix Halo Linux host with a GPU:

Setup Result
ROCM_CLI_BINARY pointing at a wrapper that exits non-zero exit 1, probe-failed message with the exit status and stderr tail
ROCM_CLI_BINARY pointing at a nonexistent path exit 1, "could not start" message
-n with a regex matching no scenario exit 1, "0 scenarios executed", lane_failure in platform.json
E2E_EXPECT_GPU=1 HIP_VISIBLE_DEVICES= exit 1, declared-GPU-missing message naming the visibility mask
E2E_EXPECT_GPU=1 E2E_ONLY_LIFECYCLE=1 E2E_ALSO_RUN_IDS=examine-version (the harness's own env filter, so the GPU check applies) exit 1, "this lane declares a GPU (E2E_EXPECT_GPU=1) but no @requires-gpu (or @requires-multi-gpu) scenario executed (52 resolved, none ran)", lane_failure in platform.json
the real binary, E2E_EXPECT_GPU=1 exit 0
E2E_EXPECT_GPU=0 notes printed, exit 0

Not verified:

  • The try_wait error arm in the probe's bounded runner (output_within) has no test: it is unreachable without fault injection, and injecting one would need a new process abstraction.
  • The open-pipe, drain and reap tests are Unix-only; there is no Windows end-to-end equivalent. The classification of each failure kind is tested on both.
  • The cfg(windows) paths were not run locally (no Windows host); the Windows CI job runs their unit tests, and the Strix Halo Windows lanes' GPU detection through rocm examine and the WSL lanes' WSLENV forwarding are exercised only by those lanes.
  • The tests/e2e.rs glue (printing, exiting, writing platform.json) has no test of its own; the local runs above exercised it.
  • cargo test --workspace --all-targets had local failures in apps/rocm tests this PR does not touch, caused by the host: a podman-backed docker that ignores DOCKER_HOST, and scratch repositories that break under the host's SSH commit-signing config.
  • python scripts/smoke_local.py: the sandbox step could not run because bubblewrap is blocked on the host.
  • cargo xtask coverage was not run locally.

Scenario coverage

No Gherkin scenario is added: this changes the test harness, the E2E report and CI lane configuration, not anything a rocm user can observe (AGENTS.md §3). Every self-hosted E2E lane exercises it on push and nightly; the hosted mock lane (e2e in ci.yml) exercises the failed-probe and zero-executed checks on every PR.

Not in this change

  • One bounded subprocess runner instead of three (rocm_core::process::run_with_timeout, device_memory::bounded_stdout, and the probe's output_within, which repeat each other to avoid a new crate edge).
  • Bounding the docker info and lemond --version probes.
  • Carrying a pre-run refusal's reason into the report instead of EMPTY.
  • Making a failed platform.json write fatal (it prints a warning).
  • Making the probe's pipe reader retry Interrupted and report other read errors; it treats any read error as EOF.
  • A warning when E2E_PROBE_TIMEOUT_SECS is invalid (it silently uses 120).
  • Indenting the continuation lines of the plain failure message.
  • Making PlatformStatus.status an enum, or hiding the type from docs.
  • Counting only the GPU scenarios the run is configured to run, and naming E2E_ONLY_LIFECYCLE/E2E_INCLUDE_NIGHTLY/E2E_MERGE_QUEUE in the failure as a hint.
  • Moving the Resolution → ResolvedRun mapping in tests/e2e.rs into the tested lane_verdict module.
  • Making the contract test's step helpers skip commented-out lines; a comment naming cargo xtask e2e or Invoke-WslBash.ps1 counts as a match.
  • Giving the wsl and wsl-no-passthrough capability fixtures the driver status a real host of that shape reports.
  • Escaping the lane reason in the markdown summary, needed once a pre-run reason (which quotes probe stderr) is carried into the report.
  • Surfacing the stale-declaration notice (E2E_EXPECT_GPU=0 on a host whose probe finds a GPU) in the report, not only as a job-log notice.

Related

Closes #383. The issue asked that a lane on which every @requires-gpu scenario resolved to skip be flagged rather than reported ok, keyed on the probe reporting a GPU. This keys on the lane's declaration instead, because one of the failures was the probe itself breaking, and a broken probe reports no GPU. Every self-hosted lane is held to declaring one by the contract test.

Interaction with #517 and #518

Both change the same lane configuration. They compose with this change rather than replace it, and whichever lands later carries the adaptations below.

@rominf
rominf force-pushed the ROCMAI-388-fix-e2e-zero-scenario-green branch from fd63f77 to 8840c8c Compare November 16, 2026 18:16
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

A self-hosted GPU lane could pass without testing its GPU. The capability
probe ignored `rocm examine`'s exit status, so a probe that failed read as
a host with no GPU and every @requires-gpu scenario resolved to skip.
Skipped scenarios never reach report.json, and the only outcome-based exit
iterated its results, so a run in which everything was filtered out exited
0.

Lanes now declare E2E_EXPECT_GPU: 1 on the self-hosted GPU lanes and the
queue lane script, 0 on the WSL lanes, whose distro is deliberately left
without ROCm's WSL driver bridge. The harness fails before any scenario
when the probe could not read the binary (spawn failure, non-zero exit,
empty output) or a declared GPU is missing, and after the run when nothing
executed or, on a GPU-declaring lane, no @requires-gpu scenario executed.
A workflow contract test holds every self-hosted lane to the declaration.

Refs: #383
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The lane checks added in the previous commit could still be defeated
in two ways, and this closes both.

A capability probe that never returned hung the whole run instead of
refusing it. Each probe command (`rocm examine`, `rocm engines list`)
now runs under a deadline, E2E_PROBE_TIMEOUT_SECS (whole seconds, 120
by default), and is killed and reaped when it passes. ProbeError gains
Timeout, Wait (the child could not be waited on) and OutputNotClosed
(the command exited but a process it started still holds its output),
each naming the command and keeping the stderr read so far. The
post-run version probe shares the deadline but only blanks the version
columns.

A lane failed by the post-run check (no scenario executed, or no
@requires-gpu scenario on an E2E_EXPECT_GPU=1 lane) exited non-zero but
still read green in the consolidated report when every scenario it
resolved was a skip. The reason is now written to platform.json as
`lane_failure`, and the report renders that lane as FAIL with the
reason, in both the step summary and the HTML report. The single
"Needs attention" list is split into a platform heading (lane-check
failures) and a scenario heading (the expectation grid), so neither
renderer shows two identical headings. Release readiness counts such a
row, whose Fail cell is 0 and so never reaches the Total row, and does
not count a row twice when it also has failed scenarios.

Annotation and notice lines now escape `%`, CR and LF as the runner
expects, so a probe's stderr tail can no longer truncate or corrupt a
workflow command.

The decisions the harness takes around these checks (reading the
declaration, counting GPU scenarios, building the manifest, the final
exit and the exact annotation lines) move out of tests/e2e.rs into pure
functions in lane_verdict.rs, each unit-tested; the harness keeps only
the glue that runs the probe and the scenarios and writes the files.

Refs: #383
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The GPU-lane check already counts @requires-multi-gpu scenarios as GPU
scenarios, but its messages, doc comments and docs named @requires-gpu
alone. The fatal reason and the E2E_EXPECT_GPU=0 note now name both
tags, and the docs say that GPU scenarios the run's own configuration
leaves out (@nightly, @merge-queue, outside an E2E_ONLY_LIFECYCLE
selection) count as resolved but never as executed.

New tests: the E2E_EXPECT_GPU=0 note's "N of M" arithmetic when some GPU
scenarios ran (4 of 7), and a probe that reads stdin getting EOF at once
rather than blocking until its deadline. Like its rocm-core sibling, the
stdin test catches a missing stdin setting but does not pin null against
inherit. The constant-equals-itself test is dropped; the workflow
contract test already ties the name to the workflows.

The workflow contract test now also fails when the step that runs the
suite sets E2E_EXPECT_GPU itself, overriding the job-level declaration,
when a lane has no such step, and when that step runs in WSL without a
WSLENV of its own naming the variable.

Stale comments corrected: the workflows now say the Windows lane's
E2E_EXPECT_GPU=1 check backstops its best-effort GPU preflight, and that
a harness preflight refusal shows as an EMPTY row rather than a missing
report; run-e2e-lane.sh says a set-but-empty E2E_EXPECT_GPU leaves the
lane undeclared. The wsl fixture gets the driver status a passthrough
host reports, so tests built from it no longer reset the field.

The markdown report folds any whitespace run in a lane-failure reason
to one space, CRLF included, not only LF.

Refs: #383
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-388-fix-e2e-zero-scenario-green branch from d5a6a79 to dcf2e21 Compare November 17, 2026 10:31
The late writer in the post-kill drain test inherited stdout as well as
stderr, so the stdout wait already held the probe open until `late` was
written, and the test stayed green with the stderr wait removed. Send the
writer's stdout to /dev/null so only the stderr wait can collect it.

Also correct two notes in the testing docs: passing an @id to -n selects
nothing, and the run then fails with "0 scenarios executed" rather than
passing silently.

Refs: #383
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf marked this pull request as ready for review November 17, 2026 11:29
@rominf
rominf requested a review from a team as a code owner November 17, 2026 11:29
@rominf
rominf requested a review from tomastola November 17, 2026 11:29

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The change delivers the chosen fixes for ROCMAI-388 and ROCMAI-450, the lane_verdict state matrix is well covered, and CI is green at this head. Nothing blocking; a few non-blocking notes:

  • Counts in comments. output_within in capability.rs (and the Cargo.toml comment) justifies the 5 s grace with "three times per suite" / "three probes". An invariant would survive adding a probe.
  • Unverified claim. nightly.yml, the README, the consolidated/mod.rs test doc and the PR body say a pre-run refusal leaves an empty report.json that uploads and reads as EMPTY. The code that creates it isn't in this diff and the new tests only read a hand-written empty file. Worth naming where it is created and that the upload step is if: always().
  • Comment wrapping is off in nightly.yml, lane_verdict.rs and e2e-report lib.rs (inline).
  • Probe wrapper chain (run_probe -> run_probe_using -> run_probe_with -> run_probe_bounded) has three test-only seams, and platform_status / PlatformStatus is public API only tests use.
  • Duplicated spawn/kill/drain logic across device_memory::bounded_stdout, capability::output_within and rocm-core run_with_timeout, held together by cross-reference comments; the deferred shared runner would fix it.
  • Scope. The probe timeout and report-layer lane_failure rendering go beyond the ticket comments (the latter was left to ROCMAI-337/444); both are disclosed in the PR body.

Not run on my side: the build, tests and mutation checks. The GPU/nightly lanes have no results because the runner pools are down.

/// - it drains after a kill too, where `bounded_stdout` gives up at once,
/// because the stderr written before the deadline is the only evidence of
/// why `rocm examine` hung;
/// - its grace is 5 s (`PROBE_DRAIN_GRACE`) rather than 2 s, because it runs

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rationale rests on a count ("three times per suite"). Please state the invariant rather than a number, so it stays true when a probe is added. The three probes line a few lines below and the Cargo.toml comment have the same shape. Also worth confirming collect_versions calls the probe exactly once.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2819c33. The output_within doc now says the probes run "a handful of times per suite: each probe runs once", not once per sample. It names the call sites instead of giving a count. The leaked-reader line and the tests/e2e-cucumber/Cargo.toml comment no longer give a count either. On collect_versions: it calls run_probe(…, &["examine"]) exactly once, and its only caller is run_suite in tests/e2e.rs. The capability probes are examine and engines list, called once each in probe_host_capability, which host_capability caches in a OnceLock.

assert_eq!(r.status_text(), "EMPTY");
}

/// The artifact a lane uploads when the harness's PRE-run check refuses to

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc (and the nightly.yml/README text) says the harness creates an empty report.json before probing, and that it gets uploaded. That code isn't in this diff and no test drives the harness exit path. Could you name where report.json is created relative to lane_verdict::preflight, and confirm the upload step runs with if: always()?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2819c33: the docs now name the place. The empty report.json is created by run_suite in tests/e2e-cucumber/tests/e2e.rs (std::fs::File::create(dir.join("report.json"))). That happens right after results_dir(), before host_capability() and before lane_verdict::preflight. A refusal exits 1 through lane_failure, and platform.json is only written after the run. Upload steps: in e2e-selfhosted.yml, e2e-gpu uploads under if: always() && steps.stale.outputs.stale != 'true' && …, and a stale-skipped run never reaches the suite. e2e-gpu-strix-ubuntu, e2e-gpu-strix-windows and e2e-wsl use if: always() && matrix.sdk_version == …. Every nightly lane uploads with if: always(), and both WSL copy-out steps are if: always() too. The docs now say this holds "on every self-hosted lane that ran its suite", because of the e2e-gpu stale condition.

Comment thread .github/workflows/nightly.yml Outdated
# lane failure as a FAIL row with lane_failure), not as missing.
# `discover()` can only render the reports that arrived, so a lane that
# uploads nothing would drop out of the grid with no row and no FAIL. On
# a `continue-on-error` lane nothing else is red either. So check it here: every lane that ran (any `needs` result but

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment line is ~124 chars after the re-flow; could you rewrap it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrapped in 2819c33, together with the rest of that comment block (it was also reworded for the thread on consolidated/mod.rs).

Comment thread tests/e2e-cucumber/src/lane_verdict.rs Outdated

/// Read [`EXPECT_GPU_ENV`]'s value.
///
/// Anything other than `1`, `0`, empty or unset is an error rather than a guess, since a misspelt declaration that read

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wrap of this doc is broken mid-sentence (over-long line); could you rewrap it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrapped in 2819c33 (the parse_gpu_declaration doc).

Comment thread crates/e2e-report/src/lib.rs Outdated
/// before any scenario runs when its capability probe finds no usable AMD GPU,
/// and after the run when no `@requires-gpu` (or `@requires-multi-gpu`)
/// scenario executed. `0` means the lane deliberately has no usable GPU (the
/// WSL lanes, whose distro is left without ROCm's WSL driver bridge). Unset or empty leaves the lane undeclared,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Over-long doc line after the WSL parenthetical; could you rewrap it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrapped in 2819c33 (the EXPECT_GPU_ENV doc).

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change. Implements ROCMAI-388 (a run that executes zero scenarios now fails instead of exiting 0) and ROCMAI-450 (the capability probe's exit status is now checked, so a failed probe is distinguished from "no GPU") in full, and closes GitHub issue #383 — deliberately keying the "every GPU scenario skipped" check on the lane's own E2E_EXPECT_GPU declaration rather than on the probe's reported GPU presence as the issue originally suggested, because a broken probe (the ROCMAI-450 failure mode) also reports no GPU, so a probe-keyed check can't tell the two apart.

Blocking

None

Non-blocking

None

Decisions for the author

  • Keying the post-run GPU check on the lane's declaration rather than on the probe's live finding — tradeoff. Issue #383 suggested keying on the probe reporting a GPU; this change keys on E2E_EXPECT_GPU instead, because a probe that itself fails also reports "no GPU," so a probe-keyed check can't distinguish "no GPU" from "broken probe" the way this design does (the latter is caught separately at preflight). The PR body already argues this under "Related" — flagging it here so it's confirmed as the intended final design rather than only a post-hoc explanation.
  • The pre-run/post-run asymmetry in what reaches the report — tradeoff. A pre-run refusal (invalid declaration, failed probe) writes no platform.json, so its row reads EMPTY with the reason only in the job log; a post-run failure writes lane_failure into platform.json and surfaces the reason in the report itself. "Not in this change" explicitly defers unifying these, since carrying the pre-run reason (which can quote probe stderr) into the report needs escaping that doesn't exist yet. Confirm this sequencing (fix now vs. a disclosed follow-up) is still intended.

Positive signals

  • lane_verdict.rs's preflight_verdict/post_run_verdict state matrices are exhaustively enumerated (every combination of declaration × GPU-found, and of executed/resolved GPU-scenario counts) with one test per row, each asserting the exact Verdict value rather than a loose substring.
  • workflow_command's %-then-CR/LF escaping order is actually load-bearing (a literal %0A substring in the input is correctly re-escaped rather than swallowed into a real newline-escape) and is exercised by a test built to catch exactly that.
  • The platform.json round trip (finish_run builds it → real serde_json serialization → e2e_report::platform_status reads it back) is tested end-to-end through both real serialization and the report crate's own loader, rather than mocked on either side.
  • scripts/run-e2e-lane.sh's "${E2E_EXPECT_GPU-1}" (unset-only default, not :-) deliberately lets a caller pass an explicitly empty value through as "undeclared" rather than silently defaulting it to 1 — a subtlety that's easy to get wrong and is documented inline.
  • The new workflow_contract.rs check closes the exact drift loophole this kind of hand-maintained CI convention usually suffers (a new lane added without the declaration, or one overridden at the step level) with an enforced test rather than only a documented convention.

Deployment notes

  • Per the PR's own "User-visible and CI-visible changes": several previously-green self-hosted lanes can now go red, most notably the Strix Halo Windows lanes, which now depend on rocm examine correctly reporting the GPU on native Windows (previously only the advisory rocm-smi preflight gated them). The PR discloses the cfg(windows) live-lane paths were not exercised against real hardware locally; CI's Windows job does run their unit tests. Worth confirming on the actual Windows runners before/soon after merge.
  • GitHub currently reports this PR's mergeable_state as dirty against main as of this review — confirm it merges cleanly; unrelated to this review's content.

What this covered

Full diff origin/main...pr-1068 (23 files, +3056/-127) at head 41d48d181bf05f3444dccc13f6f387b88074c43b, split into coupled groups and read completely: capability.rs+device_memory.rs+Cargo.toml; lane_verdict.rs (new file)+tests/e2e.rs+expectation.rs+lib.rs; crates/e2e-report's consolidated/*+lib.rs+components.rs; xtask's release_readiness.rs+workflow_contract.rs; both workflow YAMLs+run-e2e-lane.sh; and the five docs files — plus a dedicated pass checking the whole diff against this repo's AGENTS.md (read in full) and comment/doc-vs-code accuracy.

Four of the seven passes ran as independent fan-out workers (capability/probe logic, lane_verdict/harness glue, CI workflow YAML, AGENTS.md-compliance). The other three (e2e-report crate, xtask, docs) hit a concurrent-subagent limit shared with other review jobs running on this account, on both the initial attempt and one re-spawn; rather than drop them I reviewed those three myself directly against the same standard, so — noted per method — those three passes are not independent of the rest of this review's judgment, though still complete reads.

Two candidate findings from the capability/probe worker (a doc comment allegedly describing a nonexistent lane_verdict module, and probe_facts_summary allegedly unused) were verified false positives: that worker's repo-wide grep ran against the on-disk working tree, which sits on main, not the pr-1068 tree where that file and its one caller (tests/e2e.rs) actually live; git grep ... pr-1068 confirms both exist and are wired up correctly. Not included above.

Impact analysis: grepped every new identifier (EXPECT_GPU_ENV, lane_failure, platform_status/PlatformStatus) across the pr-1068 tree for consumers and found none missing or stale; confirmed no leftover reference to the old singular "Needs attention" heading anywhere. Both named Jira tickets (ROCMAI-388, ROCMAI-450) and GitHub issue #383 were retrieved and read in full; the change matches their described root causes and suggested direction (with the one disclosed, justified deviation noted above). CI was read via get_check_runs at the current head: every check completed green (one neutral CodeQL result, not a failure).

Not reconciled against the existing discussion on this PR (a standing APPROVE review from this same account with no prior marker) — per this review's instructions that was treated as superseded by this fresh pass rather than read beforehand.


🤖 Automated by an AI agent

Conflicts: nightly.yml's missing-report comment keeps this branch's
preflight-refusal text and adds main's stale-lane exemption;
workflow_contract.rs keeps both sides' new tests (the GPU-declaration
contract and main's stale-run gate tests). No lanes were added or removed.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The probe-grace rationale in capability.rs's output_within doc, and the
leaked-thread note under it, rested on "three" probes. They now state the
invariant: each probe runs once per suite (the capability probes cached
by host_capability, and collect_versions' version probe), not once per
sample. The Cargo.toml note on the copied spawn-and-reap helpers likewise
says "these copies" rather than counting them.

The claim that a harness preflight refusal leaves an empty report.json
that is still uploaded now names where that holds: run_suite in
tests/e2e-cucumber/tests/e2e.rs creates report.json before it calls
lane_verdict::preflight, and on every self-hosted lane that ran its
suite the results upload (and the WSL copy-out) is gated on always().
This is said in the e2e-report test doc, the e2e README and both
nightly.yml comments.

Rewrapped the over-long lines in nightly.yml's missing-report comment,
lane_verdict.rs's parse_gpu_declaration doc and e2e-report's
EXPECT_GPU_ENV doc.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf added this pull request to the merge queue Nov 17, 2026
Any commits made after this event will not be merged.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

atlassian ignore — placeholder, replaced below

let mut chunk = [0u8; 8192];
loop {
match pipe.read(&mut chunk) {
Ok(0) | Err(_) => break,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision for the author (answered-by-intent, not blocking): Ok(0) | Err(_) => break treats every pipe-read error, including a transient Interrupted, as EOF. The sibling Capture::draining in crates/rocm-core/src/process/bounded.rs explicitly retries ErrorKind::Interrupted and surfaces other errors instead of treating either as EOF ("Treating either as EOF would silently shorten the output instead"), with a dedicated test for exactly that. PipeReader here has neither the distinction nor a test, so a transient EINTR during a probe read would silently truncate captured output and could misclassify host capability.

This is disclosed in the PR body's "Not in this change" list ("Making the probe's pipe reader retry Interrupted and report other read errors; it treats any read error as EOF"), so I'm not blocking on it — just confirming it's still the intended scope, since the cross-reference comments here and in device_memory.rs ("a change to one should look at the other") don't currently call out this divergence.

// Refuse to run when the result could not mean anything: a probe that could
// not read the binary, or a lane that declares a GPU the probe did not find,
// would resolve every GPU scenario to skip and pass. See `lane_verdict`.
let preflight = lane_verdict::preflight(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision for the author (answered-by-intent, not blocking): report.json is created empty a few lines above this, before preflight runs, and a preflight refusal exits via lane_failure()/process::exit(1) without ever constructing a platform.json. So this failure class most likely renders as an EMPTY row in the consolidated report rather than a FAIL row with a reason, unlike a post-run failure (which does get lane_failure recorded). The CI job itself still goes red either way.

The PR body already states this asymmetry is deliberate ("A pre-run refusal writes no platform.json... carrying the pre-run reason into the report is listed under Not in this change"), so flagging only so it's confirmed the dashboard-reading audience knows EMPTY can mean this too, not just "never ran."

@juhovainio

Copy link
Copy Markdown
Collaborator

Correction: the review I just submitted (APPROVE) was posted with a placeholder body by mistake instead of the actual report. Here is the report that review was supposed to carry:

Review — no blocking findings

Full review of the whole change (base bfff5216b50ba997e380a90daf3d356dfb44e3f0, head 2819c3304f81322119a748d517d4cb8dea8d5a11; 23 files, +3076/-129), reviewed blind across 9 independent fan-out passes plus direct verification against source, before reading the PR's own description or existing review threads.

Implements the two fixes named in the title (ROCMAI-388, ROCMAI-450) and closes GitHub issue #383, deliberately keying the "every GPU scenario skipped" check on the lane's own E2E_EXPECT_GPU declaration rather than on the probe's live GPU finding — because a broken probe also reports no GPU, which is exactly the other failure mode this change fixes, so a probe-keyed check couldn't tell the two apart.

Blocking

None

Non-blocking

None

Decisions for the author

  • PipeReader (capability.rs) treats every pipe-read error, including a transient Interrupted, as EOF — answered-by-intent. Verified by direct comparison: the sibling Capture::draining in crates/rocm-core/src/process/bounded.rs explicitly retries ErrorKind::Interrupted and surfaces other errors rather than treating either as EOF ("Treating either as EOF would silently shorten the output instead"), with a dedicated test proving it. The new PipeReader has neither the distinction nor a test, so a transient EINTR during a probe read would silently truncate captured output and could misclassify host capability. The PR body's own "Not in this change" list names this precisely: "Making the probe's pipe reader retry Interrupted and report other read errors; it treats any read error as EOF." Would otherwise be a blocking logic/silent-failure finding — confirm it's still the intended scope, since the cross-reference comments in capability.rs/device_memory.rs ("a change to one should look at the other") don't currently mention this divergence. (See inline comment on capability.rs.)
  • Three independent bounded-subprocess-executor implementations now exist (rocm-core::process::bounded, device_memory::bounded_stdout, capability::output_within) — answered-by-intent. The PR body lists "One bounded subprocess runner instead of three... which repeat each other to avoid a new crate edge" under "Not in this change." The finding above is concrete evidence the "look at the other" convention has already drifted once; worth weighing whether that raises the priority of the deferred unification.
  • A pre-run preflight refusal never writes platform.json, so it most likely renders as an EMPTY row rather than a FAIL row in the consolidated report, unlike a post-run failure (which does get lane_failure recorded) — answered-by-intent. The PR body states this asymmetry is deliberate: "A pre-run refusal writes no platform.json... The asymmetry with the post-run failure is deliberate in this change, and carrying the pre-run reason into the report is listed under Not in this change." Confirm the dashboard-reading audience knows EMPTY can mean this too, not only "never ran." (See inline comment on tests/e2e.rs.)
  • Keying the post-run "no GPU scenario executed" check on the lane's own E2E_EXPECT_GPU declaration rather than on the probe's live GPU finding — tradeoff, already argued in the PR's "Related" section (a probe-keyed check can't distinguish a legitimately GPU-less host from one whose probe itself broke). Sound reasoning; flagging only so it's confirmed as the intended final design rather than only a post-hoc explanation.
  • E2E_ONLY_LIFECYCLE-filtered-out GPU scenarios still count toward "resolved" but can never reach "executed" — non-blocking-improvement. Not live today (E2E_ONLY_LIFECYCLE is only set in ci.yml, which never sets E2E_EXPECT_GPU=1), but if a future lane ever combines both, post_run_verdict would wrongly fail it for executing zero GPU scenarios it was deliberately filtered away from. Worth a guard or comment before that combination is ever introduced.

Positive signals

  • preflight_verdict/post_run_verdict in the new lane_verdict.rs are tested with full Verdict-equality assertions per state-matrix row, not substring matches, and the matrices are exhaustive over declaration × GPU-found and resolved/executed × declaration respectively.
  • workflow_command's percent-then-CR/LF escaping order is correct, hand-traced against a literal %0A-looking input; the timeout path is proven by a real subprocess test (kill -0 on the actual pid), not only a type-level assertion.
  • The new PlatformStatus cannot represent an illegal state: ok: true and a populated lane_failure are mutually exclusive by construction, confirmed by tracing every path that builds one.
  • release_readiness.rs's counting has a real end-to-end test proving a lane that is both lane_failure-FAIL and has failed scenario cells counts exactly once, not twice.
  • The CI-contract test's lane-declaration staleness check is symmetric (catches a lane marked 0 that's missing from the docs table, and one listed that no longer matches), and the doc table's values are derived from the workflow source rather than hand-copied.

Deployment notes

  • The Strix Halo Windows self-hosted lanes now depend on rocm examine correctly reporting the GPU on native Windows for the first time (previously only the advisory rocm-smi preflight gated them), and the PR discloses this cfg(windows) path wasn't exercised against real hardware locally. Worth watching those specific lanes closely right after merge.
  • Every currently-green self-hosted GPU lane that has in fact been testing nothing will go red the first time this runs for real; that's the intended effect, but worth expecting new red lanes right after this merges rather than reading them as a regression.

What this covered

Full diff bfff5216b50ba997e380a90daf3d356dfb44e3f0...2819c3304f81322119a748d517d4cb8dea8d5a11 (23 files, +3076/-129), read blind (no PR description, no review threads) across 9 independent passes grouped by coupling: capability/probe timeout logic (plus cross-referenced rocm-core::process::bounded); lane_verdict.rs + harness glue (tests/e2e.rs, expectation.rs, lib.rs); the e2e-report consolidated renderer + release_readiness.rs; xtask::workflow_contract (the full ~11,400-line file); the two CI workflow YAMLs + run-e2e-lane.sh; docs/testing.md; the remaining docs (architecture.md, ci-hardware-testing.md, nightly-on-the-queue.md, the e2e README); a dedicated AGENTS.md-compliance sweep; and a git-history/prior-changes pass. Every file in scope was read in full by at least one pass.

One pass (the CI workflow YAMLs + run-e2e-lane.sh) hit a shared concurrent-subagent limit on both the initial attempt and one respawn; rather than drop it I ran that pass myself directly, so it is not independent of the rest of this review's judgment. It found nothing at reportable confidence, and independently confirmed several claims: the empty-report.json-before-preflight ordering, the unset-only ${VAR-1} expansion in run-e2e-lane.sh, and that every self-hosted lane in both workflow files declares E2E_EXPECT_GPU consistently with its actual hardware.

CI was read via the checks API at the current head: every completed required check is green (one neutral CodeQL result, not a failure); one unrelated job, "Coverage (rocm-dash crates, ratcheted)," was still in progress at review time and touches no file in this diff.

Reconciled only after blind findings were locked: the PR's own description discloses, under "Not verified" and "Not in this change," the exact three items this review's blind passes independently found (the Interrupted-as-EOF pipe reader, the triple-duplicated bounded-subprocess runner, and the pre-run/post-run report asymmetry) — all retired above as answered-by-intent rather than left as findings. The existing review threads (an earlier, now-superseded automated review at a stale head, and a human reviewer's four inline comments) are already resolved by commits on the current head; nothing in them remains open.


🤖 Automated by an AI agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

E2E: a lane where every @requires-gpu scenario resolves to skip should be flagged, not reported ok

2 participants