Repository navigation
Conversation
fd63f77 to
8840c8c
Compare
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
A self-hosted GPU lane could pass without testing its GPU. The capability probe ignored `rocm examine`'s exit status, so a probe that failed read as a host with no GPU and every @requires-gpu scenario resolved to skip. Skipped scenarios never reach report.json, and the only outcome-based exit iterated its results, so a run in which everything was filtered out exited 0. Lanes now declare E2E_EXPECT_GPU: 1 on the self-hosted GPU lanes and the queue lane script, 0 on the WSL lanes, whose distro is deliberately left without ROCm's WSL driver bridge. The harness fails before any scenario when the probe could not read the binary (spawn failure, non-zero exit, empty output) or a declared GPU is missing, and after the run when nothing executed or, on a GPU-declaring lane, no @requires-gpu scenario executed. A workflow contract test holds every self-hosted lane to the declaration. Refs: #383 Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The lane checks added in the previous commit could still be defeated in two ways, and this closes both. A capability probe that never returned hung the whole run instead of refusing it. Each probe command (`rocm examine`, `rocm engines list`) now runs under a deadline, E2E_PROBE_TIMEOUT_SECS (whole seconds, 120 by default), and is killed and reaped when it passes. ProbeError gains Timeout, Wait (the child could not be waited on) and OutputNotClosed (the command exited but a process it started still holds its output), each naming the command and keeping the stderr read so far. The post-run version probe shares the deadline but only blanks the version columns. A lane failed by the post-run check (no scenario executed, or no @requires-gpu scenario on an E2E_EXPECT_GPU=1 lane) exited non-zero but still read green in the consolidated report when every scenario it resolved was a skip. The reason is now written to platform.json as `lane_failure`, and the report renders that lane as FAIL with the reason, in both the step summary and the HTML report. The single "Needs attention" list is split into a platform heading (lane-check failures) and a scenario heading (the expectation grid), so neither renderer shows two identical headings. Release readiness counts such a row, whose Fail cell is 0 and so never reaches the Total row, and does not count a row twice when it also has failed scenarios. Annotation and notice lines now escape `%`, CR and LF as the runner expects, so a probe's stderr tail can no longer truncate or corrupt a workflow command. The decisions the harness takes around these checks (reading the declaration, counting GPU scenarios, building the manifest, the final exit and the exact annotation lines) move out of tests/e2e.rs into pure functions in lane_verdict.rs, each unit-tested; the harness keeps only the glue that runs the probe and the scenarios and writes the files. Refs: #383 Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The GPU-lane check already counts @requires-multi-gpu scenarios as GPU scenarios, but its messages, doc comments and docs named @requires-gpu alone. The fatal reason and the E2E_EXPECT_GPU=0 note now name both tags, and the docs say that GPU scenarios the run's own configuration leaves out (@nightly, @merge-queue, outside an E2E_ONLY_LIFECYCLE selection) count as resolved but never as executed. New tests: the E2E_EXPECT_GPU=0 note's "N of M" arithmetic when some GPU scenarios ran (4 of 7), and a probe that reads stdin getting EOF at once rather than blocking until its deadline. Like its rocm-core sibling, the stdin test catches a missing stdin setting but does not pin null against inherit. The constant-equals-itself test is dropped; the workflow contract test already ties the name to the workflows. The workflow contract test now also fails when the step that runs the suite sets E2E_EXPECT_GPU itself, overriding the job-level declaration, when a lane has no such step, and when that step runs in WSL without a WSLENV of its own naming the variable. Stale comments corrected: the workflows now say the Windows lane's E2E_EXPECT_GPU=1 check backstops its best-effort GPU preflight, and that a harness preflight refusal shows as an EMPTY row rather than a missing report; run-e2e-lane.sh says a set-but-empty E2E_EXPECT_GPU leaves the lane undeclared. The wsl fixture gets the driver status a passthrough host reports, so tests built from it no longer reset the field. The markdown report folds any whitespace run in a lane-failure reason to one space, CRLF included, not only LF. Refs: #383 Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
d5a6a79 to
dcf2e21
Compare
The late writer in the post-kill drain test inherited stdout as well as stderr, so the stdout wait already held the probe open until `late` was written, and the test stayed green with the stderr wait removed. Send the writer's stdout to /dev/null so only the stderr wait can collect it. Also correct two notes in the testing docs: passing an @id to -n selects nothing, and the run then fails with "0 scenarios executed" rather than passing silently. Refs: #383 Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. The change delivers the chosen fixes for ROCMAI-388 and ROCMAI-450, the lane_verdict state matrix is well covered, and CI is green at this head. Nothing blocking; a few non-blocking notes:
- Counts in comments.
output_withinin capability.rs (and the Cargo.toml comment) justifies the 5 s grace with "three times per suite" / "three probes". An invariant would survive adding a probe. - Unverified claim. nightly.yml, the README, the
consolidated/mod.rstest doc and the PR body say a pre-run refusal leaves an emptyreport.jsonthat uploads and reads as EMPTY. The code that creates it isn't in this diff and the new tests only read a hand-written empty file. Worth naming where it is created and that the upload step isif: always(). - Comment wrapping is off in nightly.yml,
lane_verdict.rsand e2e-reportlib.rs(inline). - Probe wrapper chain (
run_probe->run_probe_using->run_probe_with->run_probe_bounded) has three test-only seams, andplatform_status/PlatformStatusis public API only tests use. - Duplicated spawn/kill/drain logic across
device_memory::bounded_stdout,capability::output_withinand rocm-corerun_with_timeout, held together by cross-reference comments; the deferred shared runner would fix it. - Scope. The probe timeout and report-layer
lane_failurerendering go beyond the ticket comments (the latter was left to ROCMAI-337/444); both are disclosed in the PR body.
Not run on my side: the build, tests and mutation checks. The GPU/nightly lanes have no results because the runner pools are down.
| /// - it drains after a kill too, where `bounded_stdout` gives up at once, | ||
| /// because the stderr written before the deadline is the only evidence of | ||
| /// why `rocm examine` hung; | ||
| /// - its grace is 5 s (`PROBE_DRAIN_GRACE`) rather than 2 s, because it runs |
There was a problem hiding this comment.
This rationale rests on a count ("three times per suite"). Please state the invariant rather than a number, so it stays true when a probe is added. The three probes line a few lines below and the Cargo.toml comment have the same shape. Also worth confirming collect_versions calls the probe exactly once.
There was a problem hiding this comment.
Fixed in 2819c33. The output_within doc now says the probes run "a handful of times per suite: each probe runs once", not once per sample. It names the call sites instead of giving a count. The leaked-reader line and the tests/e2e-cucumber/Cargo.toml comment no longer give a count either. On collect_versions: it calls run_probe(…, &["examine"]) exactly once, and its only caller is run_suite in tests/e2e.rs. The capability probes are examine and engines list, called once each in probe_host_capability, which host_capability caches in a OnceLock.
| assert_eq!(r.status_text(), "EMPTY"); | ||
| } | ||
|
|
||
| /// The artifact a lane uploads when the harness's PRE-run check refuses to |
There was a problem hiding this comment.
This doc (and the nightly.yml/README text) says the harness creates an empty report.json before probing, and that it gets uploaded. That code isn't in this diff and no test drives the harness exit path. Could you name where report.json is created relative to lane_verdict::preflight, and confirm the upload step runs with if: always()?
There was a problem hiding this comment.
Fixed in 2819c33: the docs now name the place. The empty report.json is created by run_suite in tests/e2e-cucumber/tests/e2e.rs (std::fs::File::create(dir.join("report.json"))). That happens right after results_dir(), before host_capability() and before lane_verdict::preflight. A refusal exits 1 through lane_failure, and platform.json is only written after the run. Upload steps: in e2e-selfhosted.yml, e2e-gpu uploads under if: always() && steps.stale.outputs.stale != 'true' && …, and a stale-skipped run never reaches the suite. e2e-gpu-strix-ubuntu, e2e-gpu-strix-windows and e2e-wsl use if: always() && matrix.sdk_version == …. Every nightly lane uploads with if: always(), and both WSL copy-out steps are if: always() too. The docs now say this holds "on every self-hosted lane that ran its suite", because of the e2e-gpu stale condition.
| # lane failure as a FAIL row with lane_failure), not as missing. | ||
| # `discover()` can only render the reports that arrived, so a lane that | ||
| # uploads nothing would drop out of the grid with no row and no FAIL. On | ||
| # a `continue-on-error` lane nothing else is red either. So check it here: every lane that ran (any `needs` result but |
There was a problem hiding this comment.
This comment line is ~124 chars after the re-flow; could you rewrap it?
There was a problem hiding this comment.
Rewrapped in 2819c33, together with the rest of that comment block (it was also reworded for the thread on consolidated/mod.rs).
|
|
||
| /// Read [`EXPECT_GPU_ENV`]'s value. | ||
| /// | ||
| /// Anything other than `1`, `0`, empty or unset is an error rather than a guess, since a misspelt declaration that read |
There was a problem hiding this comment.
The wrap of this doc is broken mid-sentence (over-long line); could you rewrap it?
There was a problem hiding this comment.
Rewrapped in 2819c33 (the parse_gpu_declaration doc).
| /// before any scenario runs when its capability probe finds no usable AMD GPU, | ||
| /// and after the run when no `@requires-gpu` (or `@requires-multi-gpu`) | ||
| /// scenario executed. `0` means the lane deliberately has no usable GPU (the | ||
| /// WSL lanes, whose distro is left without ROCm's WSL driver bridge). Unset or empty leaves the lane undeclared, |
There was a problem hiding this comment.
Over-long doc line after the WSL parenthetical; could you rewrap it?
There was a problem hiding this comment.
Rewrapped in 2819c33 (the EXPECT_GPU_ENV doc).
juhovainio
left a comment
There was a problem hiding this comment.
Review — no blocking findings
Full review of the whole change. Implements ROCMAI-388 (a run that executes zero scenarios now fails instead of exiting 0) and ROCMAI-450 (the capability probe's exit status is now checked, so a failed probe is distinguished from "no GPU") in full, and closes GitHub issue #383 — deliberately keying the "every GPU scenario skipped" check on the lane's own E2E_EXPECT_GPU declaration rather than on the probe's reported GPU presence as the issue originally suggested, because a broken probe (the ROCMAI-450 failure mode) also reports no GPU, so a probe-keyed check can't tell the two apart.
Blocking
None
Non-blocking
None
Decisions for the author
- Keying the post-run GPU check on the lane's declaration rather than on the probe's live finding — tradeoff. Issue #383 suggested keying on the probe reporting a GPU; this change keys on
E2E_EXPECT_GPUinstead, because a probe that itself fails also reports "no GPU," so a probe-keyed check can't distinguish "no GPU" from "broken probe" the way this design does (the latter is caught separately at preflight). The PR body already argues this under "Related" — flagging it here so it's confirmed as the intended final design rather than only a post-hoc explanation. - The pre-run/post-run asymmetry in what reaches the report — tradeoff. A pre-run refusal (invalid declaration, failed probe) writes no
platform.json, so its row reads EMPTY with the reason only in the job log; a post-run failure writeslane_failureintoplatform.jsonand surfaces the reason in the report itself. "Not in this change" explicitly defers unifying these, since carrying the pre-run reason (which can quote probe stderr) into the report needs escaping that doesn't exist yet. Confirm this sequencing (fix now vs. a disclosed follow-up) is still intended.
Positive signals
lane_verdict.rs'spreflight_verdict/post_run_verdictstate matrices are exhaustively enumerated (every combination of declaration × GPU-found, and of executed/resolved GPU-scenario counts) with one test per row, each asserting the exactVerdictvalue rather than a loose substring.workflow_command's%-then-CR/LF escaping order is actually load-bearing (a literal%0Asubstring in the input is correctly re-escaped rather than swallowed into a real newline-escape) and is exercised by a test built to catch exactly that.- The
platform.jsonround trip (finish_runbuilds it → realserde_jsonserialization →e2e_report::platform_statusreads it back) is tested end-to-end through both real serialization and the report crate's own loader, rather than mocked on either side. scripts/run-e2e-lane.sh's"${E2E_EXPECT_GPU-1}"(unset-only default, not:-) deliberately lets a caller pass an explicitly empty value through as "undeclared" rather than silently defaulting it to1— a subtlety that's easy to get wrong and is documented inline.- The new
workflow_contract.rscheck closes the exact drift loophole this kind of hand-maintained CI convention usually suffers (a new lane added without the declaration, or one overridden at the step level) with an enforced test rather than only a documented convention.
Deployment notes
- Per the PR's own "User-visible and CI-visible changes": several previously-green self-hosted lanes can now go red, most notably the Strix Halo Windows lanes, which now depend on
rocm examinecorrectly reporting the GPU on native Windows (previously only the advisoryrocm-smipreflight gated them). The PR discloses thecfg(windows)live-lane paths were not exercised against real hardware locally; CI's Windows job does run their unit tests. Worth confirming on the actual Windows runners before/soon after merge. - GitHub currently reports this PR's
mergeable_stateasdirtyagainstmainas of this review — confirm it merges cleanly; unrelated to this review's content.
What this covered
Full diff origin/main...pr-1068 (23 files, +3056/-127) at head 41d48d181bf05f3444dccc13f6f387b88074c43b, split into coupled groups and read completely: capability.rs+device_memory.rs+Cargo.toml; lane_verdict.rs (new file)+tests/e2e.rs+expectation.rs+lib.rs; crates/e2e-report's consolidated/*+lib.rs+components.rs; xtask's release_readiness.rs+workflow_contract.rs; both workflow YAMLs+run-e2e-lane.sh; and the five docs files — plus a dedicated pass checking the whole diff against this repo's AGENTS.md (read in full) and comment/doc-vs-code accuracy.
Four of the seven passes ran as independent fan-out workers (capability/probe logic, lane_verdict/harness glue, CI workflow YAML, AGENTS.md-compliance). The other three (e2e-report crate, xtask, docs) hit a concurrent-subagent limit shared with other review jobs running on this account, on both the initial attempt and one re-spawn; rather than drop them I reviewed those three myself directly against the same standard, so — noted per method — those three passes are not independent of the rest of this review's judgment, though still complete reads.
Two candidate findings from the capability/probe worker (a doc comment allegedly describing a nonexistent lane_verdict module, and probe_facts_summary allegedly unused) were verified false positives: that worker's repo-wide grep ran against the on-disk working tree, which sits on main, not the pr-1068 tree where that file and its one caller (tests/e2e.rs) actually live; git grep ... pr-1068 confirms both exist and are wired up correctly. Not included above.
Impact analysis: grepped every new identifier (EXPECT_GPU_ENV, lane_failure, platform_status/PlatformStatus) across the pr-1068 tree for consumers and found none missing or stale; confirmed no leftover reference to the old singular "Needs attention" heading anywhere. Both named Jira tickets (ROCMAI-388, ROCMAI-450) and GitHub issue #383 were retrieved and read in full; the change matches their described root causes and suggested direction (with the one disclosed, justified deviation noted above). CI was read via get_check_runs at the current head: every check completed green (one neutral CodeQL result, not a failure).
Not reconciled against the existing discussion on this PR (a standing APPROVE review from this same account with no prior marker) — per this review's instructions that was treated as superseded by this fresh pass rather than read beforehand.
🤖 Automated by an AI agent
Conflicts: nightly.yml's missing-report comment keeps this branch's preflight-refusal text and adds main's stale-lane exemption; workflow_contract.rs keeps both sides' new tests (the GPU-declaration contract and main's stale-run gate tests). No lanes were added or removed. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The probe-grace rationale in capability.rs's output_within doc, and the leaked-thread note under it, rested on "three" probes. They now state the invariant: each probe runs once per suite (the capability probes cached by host_capability, and collect_versions' version probe), not once per sample. The Cargo.toml note on the copied spawn-and-reap helpers likewise says "these copies" rather than counting them. The claim that a harness preflight refusal leaves an empty report.json that is still uploaded now names where that holds: run_suite in tests/e2e-cucumber/tests/e2e.rs creates report.json before it calls lane_verdict::preflight, and on every self-hosted lane that ran its suite the results upload (and the WSL copy-out) is gated on always(). This is said in the e2e-report test doc, the e2e README and both nightly.yml comments. Rewrapped the over-long lines in nightly.yml's missing-report comment, lane_verdict.rs's parse_gpu_declaration doc and e2e-report's EXPECT_GPU_ENV doc. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
atlassian ignore — placeholder, replaced below
| let mut chunk = [0u8; 8192]; | ||
| loop { | ||
| match pipe.read(&mut chunk) { | ||
| Ok(0) | Err(_) => break, |
There was a problem hiding this comment.
Decision for the author (answered-by-intent, not blocking): Ok(0) | Err(_) => break treats every pipe-read error, including a transient Interrupted, as EOF. The sibling Capture::draining in crates/rocm-core/src/process/bounded.rs explicitly retries ErrorKind::Interrupted and surfaces other errors instead of treating either as EOF ("Treating either as EOF would silently shorten the output instead"), with a dedicated test for exactly that. PipeReader here has neither the distinction nor a test, so a transient EINTR during a probe read would silently truncate captured output and could misclassify host capability.
This is disclosed in the PR body's "Not in this change" list ("Making the probe's pipe reader retry Interrupted and report other read errors; it treats any read error as EOF"), so I'm not blocking on it — just confirming it's still the intended scope, since the cross-reference comments here and in device_memory.rs ("a change to one should look at the other") don't currently call out this divergence.
| // Refuse to run when the result could not mean anything: a probe that could | ||
| // not read the binary, or a lane that declares a GPU the probe did not find, | ||
| // would resolve every GPU scenario to skip and pass. See `lane_verdict`. | ||
| let preflight = lane_verdict::preflight( |
There was a problem hiding this comment.
Decision for the author (answered-by-intent, not blocking): report.json is created empty a few lines above this, before preflight runs, and a preflight refusal exits via lane_failure()/process::exit(1) without ever constructing a platform.json. So this failure class most likely renders as an EMPTY row in the consolidated report rather than a FAIL row with a reason, unlike a post-run failure (which does get lane_failure recorded). The CI job itself still goes red either way.
The PR body already states this asymmetry is deliberate ("A pre-run refusal writes no platform.json... carrying the pre-run reason into the report is listed under Not in this change"), so flagging only so it's confirmed the dashboard-reading audience knows EMPTY can mean this too, not just "never ran."
|
Correction: the review I just submitted (APPROVE) was posted with a placeholder body by mistake instead of the actual report. Here is the report that review was supposed to carry: Review — no blocking findingsFull review of the whole change (base Implements the two fixes named in the title (ROCMAI-388, ROCMAI-450) and closes GitHub issue #383, deliberately keying the "every GPU scenario skipped" check on the lane's own BlockingNone Non-blockingNone Decisions for the author
Positive signals
Deployment notes
What this coveredFull diff One pass (the CI workflow YAMLs + CI was read via the checks API at the current head: every completed required check is green (one Reconciled only after blind findings were locked: the PR's own description discloses, under "Not verified" and "Not in this change," the exact three items this review's blind passes independently found (the 🤖 Automated by an AI agent |
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No rows reference ROCMAI-388, ROCMAI-450 or E2E: a lane where every @requires-gpu scenario resolves to skip should be flagged, not reported ok #383.)docs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation). (n/a: no subcommand or subsystem.).rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry (cargo xtask check-architecture-docfails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (Newtests/e2e-cucumber/src/lane_verdict.rsadded; thecapability.rsentry now lists theProbeErrorkinds and theconsolidated/mod.rsentry namesplatform_status.)rocmoutput. Eachlane_verdicttest asserts the fatal/non-fatal outcome together with the message; the report's new Status wording is pinned beside a comment naming the tests that prove a lane failure turns the row FAIL.)Why
A self-hosted GPU lane could go green without testing its GPU, and nothing in the result distinguished that from a lane that exercised everything. Three causes:
rocm examineandrocm engines list, but ignored their exit status. A probe that failed (wrong binary, a crash, empty output) produced the same facts as a host without a GPU, so every@requires-gpu,@requires-multi-gpuand gfx-target-gated scenario resolved to skip. A probe that never returned hung the whole run instead.report.json, and the harness's only outcome-based exit iterated the results in it, looking for unexpected failures and stale XPASSes. With nothing in it there was nothing to fail. A cucumber--name/--tagsfilter on the command line also replaces the harness's own scenario filter, so a filter matching nothing gave the same silent green.What changes
Lanes declare whether they have a GPU, in
E2E_EXPECT_GPU:1e2e-selfhosted.ymlandnightly.yml, and every lane ofscripts/run-e2e-lane.sh(default1)0e2e-wsl,e2e-wsl-nightly(forwarded into the guest viaWSLENV)rocm install driverinstalls under testAny other value fails the run rather than being read as "undeclared", so a typo cannot switch the check off. The variable name is one shared constant,
e2e_report::EXPECT_GPU_ENV, read by the harness and by the workflow contract test.Each probe command runs under a deadline,
E2E_PROBE_TIMEOUT_SECS(default 120 s; any whole number ≥ 1, anything else uses the default). A probe failure names the command and says which way it failed: it could not be started, exited non-zero (with its exit status and stderr tail), printed nothing, timed out (it is killed and reaped, and the message gives the deadline and the stderr read so far), could not be waited on, or exited while a process it started still held its stdout open. The post-run version probe for the report heading uses the same deadline but fails quietly: it only blanks the version columns.The harness checks the run before and after the scenarios (new
tests/e2e-cucumber/src/lane_verdict.rs):1lane: fail when the probe found no usable AMD GPU, naming the detected gfx target, WSL flag, driver status and visibility mask the decision was made from.1lane: fail when no@requires-gpu/@requires-multi-gpuscenario executed. GPU scenarios the run's own configuration leaves out (@nightly,@merge-queue, anything outside anE2E_ONLY_LIFECYCLEselection) count as resolved but never as executed, soE2E_EXPECT_GPU=1withE2E_ONLY_LIFECYCLE=1and no GPU scenario selected fails. Skipped when a cucumber CLI filter replaced the harness's filter, since which scenarios need a GPU is then unknown.0lane, notes only: that GPU scenarios are not applicable, how many did not run, and that the declaration is stale if the probe finds a GPU anyway.Failures print a GitHub
::error::annotation followed by the plain message, and exit 1; notes print as::notice::. Annotation data now escapes%, CR and LF, so a probe's multi-line stderr tail cannot truncate or corrupt the workflow command.The decisions live in pure, unit-tested functions in
lane_verdict.rs:preflight(declaration, probe result, GPU presence), the GPU scenario counts,run_failed,finish_run(post-run verdict plus theplatform.jsonmanifest),failure_lines/note_lineandworkflow_command. What stays intests/e2e.rsis glue: printing, exiting, and writing the file.A post-run failure reaches the consolidated report. Its reason is written to
platform.jsonaslane_failure. The report then shows that lane as FAIL even though its Fail cell is 0, and gives the reason in the Status cell's tooltip, in a paragraph in the platform's section of the HTML report, and in a "Needs attention: platforms" list (in both the step summary and the HTML report). The scenario-level list that used the same "Needs attention" heading is now "Needs attention: scenarios", so neither renderer shows two identical headings.release-readinessalready counts a non-PASS row whose Fail cell is 0 (it did so for EMPTY rows), so such a lane makes the nightly verdict NO-GO; a new test pins that, and that a row which also has failed scenarios is counted once. A shared-queue lane reaches the report throughcargo xtask e2e-queue-import, which keepslane_failurewhen it copies the lane'splatform.json; one the queue marks advisory reads FAIL too but, like its failed scenarios, stays out of the verdict.A pre-run refusal writes no
platform.json, so its row shows EMPTY (red), with the reason only in the job log. The asymmetry with the post-run failure is deliberate in this change, and carrying the pre-run reason into the report is listed under Not in this change.e2e_report::platform_status/PlatformStatus(one platform's ok flag, Status text and lane failure) is new public API, used by the harness's tests to read back theplatform.jsonthatfinish_runbuilds through the report's own loader.A workflow contract test (
every_self_hosted_lane_declares_whether_it_has_a_gpuinxtask/src/workflow_contract.rs) fails when a self-hosted E2E lane does not set the variable in its job-levelenv:, sets0without a recorded reason, or sets aWSLENVthat does not name it. It also finds the step that runscargo xtask e2eand fails when there is none, when that step setsE2E_EXPECT_GPUitself (overriding the job's declaration), and when that step runs the suite in WSL without aWSLENVof its own naming the variable, since aWSLENVset in another step never reaches the guest. It also checks therun-e2e-lane.shdefault. The hardware-lanes table indocs/ci-hardware-testing.mdgains a "GPU declared" column, held to the workflows by the existing docs-table test.Docs:
docs/testing.md(new "E2E runs that test nothing fail"),docs/ci-hardware-testing.md,docs/nightly-on-the-queue.md,docs/architecture.mdandtests/e2e-cucumber/README.md(new "Lane verdict" section and theE2E_PROBE_TIMEOUT_SECSrow; it also states that-n/--tagsbypasses capability gating).User-visible and CI-visible changes
No
rocmCLI behaviour changes. What changes is which CI runs can go red, and what the report says about them:1lane (MI300X, Strix Halo Ubuntu and Strix Halo Windows ine2e-selfhosted.yml; the MI300X, Strix Halo Ubuntu and Strix Halo Windows lanes innightly.yml; the queue lanes) now fails when its probe finds no usable GPU or when no GPU scenario executes. Previously these were green.rocm examinereporting the GPU on native Windows. Their GPU preflight already tolerates an absentrocm-smi, sorocm examineis the only thing between those lanes and red.cargo xtask e2e -- -n <regex>with a regex matching nothing exits 1 instead of 0.Verification
Unit and state-matrix tests, each shown to fail with its target broken (guard reverted or inverted):
capability.rs: the probe classifier for every failure kind, the deadline parsing (unset, padded, very large, and empty, zero, negative, fractional, non-numeric or pastu64falling back to 120), the timeout reaching the probe through the same env lookup, kill-and-reap on timeout, and the stderr kept so far. A probe that reads stdin gets EOF at once instead of blocking until its deadline (removing the runner'sstdin(Stdio::null())makes it fail); like its sibling in rocm-core'sprocess/bounded.rs, it catches a missing stdin setting but does not pinnullagainstinherit, which passes whenever the test process's own stdin is already at EOF. Exit statuses are built per platform, so the classification tests also run on the Windows CI job.lane_verdict.rs: one test per row of the state matrix: invalid declaration; probe failed × each declaration; probe OK × GPU present/absent ×1/0/unset; post-run with 0 executed, with GPU scenarios resolved but not executed, none resolved, executed, and with a CLI filter; the0-lane note's count when some GPU scenarios ran ("4 of 7", which fails when the note reports the resolved count instead);run_failedon each of its three reasons; the escaping ofworkflow_commandand the exact failure and notice lines; andfinish_run's manifest read back throughe2e_report::platform_statusas FAIL with its reason.xtask/src/workflow_contract.rs: the per-lane declaration andWSLENVcontract, plus the docs-table column. Addingenv: E2E_EXPECT_GPU: "0"to the MI300X lane's suite step fails it ("stepRun E2E tests on GPU hardwaresets E2E_EXPECT_GPU itself, overriding the job-level declaration"), and so does moving thee2e-wsllane'sWSLENVline from its suite step to the step before it, which keeps aWSLENVnaming the variable in the job ("stepRun E2E tests on Strix Halo WSL2runs the suite in WSL but sets no WSLENV naming E2E_EXPECT_GPU").e2e-report: an all-skip lane withlane_failurereads FAIL with its reason in the summary list, Status tooltip and platform paragraph (HTML-escaped); the same lane without one still reads PASS with none of them; a zero-expectation manifest reads FAIL only with a recorded failure; a lane stopped before any scenario reads EMPTY and not ok; both attention headings render.xtask/src/release_readiness.rs: a lane-failure row with a Fail cell of 0 counts once, and one with a failed scenario is not counted twice.Local runs of the real harness on a Strix Halo Linux host with a GPU:
ROCM_CLI_BINARYpointing at a wrapper that exits non-zeroROCM_CLI_BINARYpointing at a nonexistent path-nwith a regex matching no scenariolane_failureinplatform.jsonE2E_EXPECT_GPU=1 HIP_VISIBLE_DEVICES=E2E_EXPECT_GPU=1 E2E_ONLY_LIFECYCLE=1 E2E_ALSO_RUN_IDS=examine-version(the harness's own env filter, so the GPU check applies)lane_failureinplatform.jsonE2E_EXPECT_GPU=1E2E_EXPECT_GPU=0Not verified:
try_waiterror arm in the probe's bounded runner (output_within) has no test: it is unreachable without fault injection, and injecting one would need a new process abstraction.cfg(windows)paths were not run locally (no Windows host); the Windows CI job runs their unit tests, and the Strix Halo Windows lanes' GPU detection throughrocm examineand the WSL lanes'WSLENVforwarding are exercised only by those lanes.tests/e2e.rsglue (printing, exiting, writingplatform.json) has no test of its own; the local runs above exercised it.cargo test --workspace --all-targetshad local failures inapps/rocmtests this PR does not touch, caused by the host: a podman-backeddockerthat ignoresDOCKER_HOST, and scratch repositories that break under the host's SSH commit-signing config.python scripts/smoke_local.py: the sandbox step could not run because bubblewrap is blocked on the host.cargo xtask coveragewas not run locally.Scenario coverage
No Gherkin scenario is added: this changes the test harness, the E2E report and CI lane configuration, not anything a
rocmuser can observe (AGENTS.md §3). Every self-hosted E2E lane exercises it on push and nightly; the hosted mock lane (e2einci.yml) exercises the failed-probe and zero-executed checks on every PR.Not in this change
rocm_core::process::run_with_timeout,device_memory::bounded_stdout, and the probe'soutput_within, which repeat each other to avoid a new crate edge).docker infoandlemond --versionprobes.platform.jsonwrite fatal (it prints a warning).Interruptedand report other read errors; it treats any read error as EOF.E2E_PROBE_TIMEOUT_SECSis invalid (it silently uses 120).PlatformStatus.statusan enum, or hiding the type from docs.E2E_ONLY_LIFECYCLE/E2E_INCLUDE_NIGHTLY/E2E_MERGE_QUEUEin the failure as a hint.Resolution→ResolvedRunmapping intests/e2e.rsinto the testedlane_verdictmodule.cargo xtask e2eorInvoke-WslBash.ps1counts as a match.wslandwsl-no-passthroughcapability fixtures the driver status a real host of that shape reports.E2E_EXPECT_GPU=0on a host whose probe finds a GPU) in the report, not only as a job-log notice.Related
Closes #383. The issue asked that a lane on which every
@requires-gpuscenario resolved to skip be flagged rather than reported ok, keyed on the probe reporting a GPU. This keys on the lane's declaration instead, because one of the failures was the probe itself breaking, and a broken probe reports no GPU. Every self-hosted lane is held to declaring one by the contract test.Interaction with #517 and #518
Both change the same lane configuration. They compose with this change rather than replace it, and whichever lands later carries the adaptations below.
E2E_EXPECT_GPUstays separate from test(e2e): describe GPU hosts with simulated machines (ROCMAI-677) #517'sE2E_HARDWARE.E2E_HARDWARE=realsays the scenarios read this machine, not that it has a GPU: test(e2e): describe GPU hosts with simulated machines (ROCMAI-677) #517 sets it on every self-hosted lane, the WSL lanes included, and those are real hardware with no usable GPU (E2E_EXPECT_GPU=0). A lane's verdict needs both facts: which hardware the scenarios read, and whether a GPU must be present.@requires-real-gpuinstead of plain@requires-gpu. The post-run "no GPU scenario executed" check has to count those as GPU scenarios (it reads the parsedrequires_gpuflag, which test(e2e): describe GPU hosts with simulated machines (ROCMAI-677) #517 sets for that tag), and its@requires-multi-gputerm can go once no scenario carries that tag.xtask/src/workflow_contract.rsand theenv:blocks of both workflow files conflict textually.E2E_GPU_ONLY=1on the GPU lanes andE2E_GPU_SMOKE_ONLY=1on pull requests and in the merge queue. Those lanes still execute scenarios, all of them GPU scenarios, so neither post-run check fires on them; a GPU-only lane that loses its GPU fails before any scenario runs (declared GPU not found) instead of passing with nothing executed. ci(e2e): run only GPU scenarios on GPU lanes, smoke-only on PRs (ROCMAI-535) #518 leaves the WSL lanes out of GPU-only.