Repository navigation
refactor(rocm-core): run the host probes' subprocesses through one bounded executor with a typed outcome - #1004
Conversation
… same stand-in programs Characterisation for the subprocess-runners pair of docs/host-probing.md, green against the unchanged runners. run (over run_raw), capture_optional_command_with_timeout and run_with_timeout are each run against the same fake programs: success, a non-zero exit with output, a program that cannot be started, a timeout with partial output, output written after the child exits, a descendant holding the pipe (with and without a timeout, each with a fixed short life so a runner that waits for it is slow and cannot hang the suite), output that is not UTF-8, UTF-16LE output and a signal death. Where the runners differ the difference is an assertion, so the convergence that follows flips exactly the ones that are meant to change. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…e bounded executor run_with_timeout stays the bounded executor. Its spawn and wait are shared with run_bounded, which returns the run as a CommandCapture instead of an error: the raw bytes of both streams, a typed CommandOutcome (not found, spawn failed, timed out, wait failed, exited with a code, killed by a signal), the status and the error text, and the partial output of a child killed at its deadline. A capture whose readers were still going when the drain grace ended says so in its detail. CommandCapture gains those fields; the hand-built captures in rocm and rocmd move to CommandCapture::from_text. run_raw and capture_optional_command_candidate_with_timeout become thin adapters over it and keep their return shapes. capture_diagnostic_command (Windows) moves behind it too, with the candidate loop and the translation of each outcome behind an injectable runner so a unit test runs it on Linux and on the Windows job. None of them joins a reader thread any more, so a descendant that keeps a pipe open no longer outlasts the call or its timeout, and output written more than the 200 ms drain grace after the child exits is no longer kept. The wait loop backs off from 5 ms to 50 ms instead of sleeping a flat 50 ms, so a probe that exits at once is not charged the full interval; both loops take their sleep as a parameter so a test sees the schedule they follow. The characterisation cases from the previous commit are flipped where the answer is meant to change; scenario examine-39 drives a probe tool that leaves a background process behind through rocm examine. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Marks the first step delivered: run_with_timeout stays the one bounded executor, CommandCapture is the shared carrier with the raw bytes and a typed outcome, and run, capture_optional_command* and the Windows diagnostic capture are adapters over it. Lists every difference that moved (late descendant output within and beyond the drain grace, a descendant holding the pipe with and without a timeout, a descendant holding only stderr, a pipe read error, a failure to wait, spawn failure and encoding, signal death, the poll interval, the diagnostic capture on a timeout) with the test that pins it and whether the adapter keeps the old answer, and keeps the later steps open. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Native Windows runtime validation is still needed for the shared executor’s adapters, timeout handling, and inherited-pipe behavior, which Linux execution, cross-compilation, and injected tests do not verify.
No open issues 🎉
What changed in this PR
Consolidates the host-probe subprocess runners onto a shared bounded executor, addressing part of #623 without changing helper signatures. Descendants holding output pipes no longer prolong captures indefinitely; output arriving after the drain grace may be omitted.
Changes:
- Extends
CommandCapturewith raw output, typed outcomes, process status and diagnostic details. - Converts three probe runners into adapters over shared execution logic and adds adaptive polling.
- Adds regression tests and a Linux inspection scenario, updates capture constructors, and documents behavior changes and remaining work.
| File | Description |
|---|---|
tests/e2e-cucumber/tests/e2e/examine_steps.rs |
Adds a pipe-holding lspci fixture and timing/output assertions. |
tests/e2e-cucumber/tests/e2e.rs |
Documents reuse of elapsed-time state for inspection. |
tests/e2e-cucumber/features/examine.feature |
Adds the Linux bounded-inspection scenario. |
docs/testing.md |
Documents the scenario and invocation. |
docs/host-probing.md |
Records convergence decisions, behavior differences and follow-ups. |
docs/architecture.md |
Updates executor, adapter and capture descriptions. |
crates/rocm-core/src/process.rs |
Shares bounded execution, adds typed captures and adaptive polling tests. |
crates/rocm-core/src/probe_exec.rs |
Adapts probe execution while preserving return-code and decoding behavior. |
crates/rocm-core/src/lib.rs |
Re-exports CommandOutcome. |
crates/rocm-core/src/host_gpu/diagnostics.rs |
Migrates Windows diagnostic capture and tests outcome translation. |
crates/rocm-core/src/host_gpu/capture.rs |
Migrates optional capture and adds runner regression tests. |
crates/rocm-core/src/command_capture.rs |
Adds outcome metadata, raw streams and capture constructors. |
apps/rocmd/src/watchers/therock_update.rs |
Migrates test captures to from_text. |
apps/rocmd/src/watchers/driver_upgrade.rs |
Migrates a test capture to from_text. |
apps/rocmd/src/sandbox.rs |
Migrates test captures to from_text. |
apps/rocm/src/chat/mcp_bridge.rs |
Uses from_text for in-process command results. |
Review effort: Balanced
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
juhovainio
left a comment
There was a problem hiding this comment.
The refactor matches ROCMAI-925's first step: one bounded executor, a typed outcome, adapters keeping their return shapes, and the capture behind an injectable runner. I found no correctness defects in the diff. Two things need resolving before this can merge, and I left a few smaller notes.
Before merge
- The
E2E tests (Strix Halo, Ubuntu, nightly, current)check fails at the "Run E2E tests on Strix Halo" step after about 8 minutes. I couldn't read the log, so I can't say whether this change caused it or whether main already fails there. Please confirm either way. - The PR conflicts with main. The body notes that #990, #998 and #694 also add
examine-39, so that scenario id needs renumbering when you rebase.
Smaller notes (non-blocking)
- A tool that exits 0 but writes after the 200 ms drain grace now returns a short answer as a full success (see the inline comment on
capture.rs). - The optional capture now pipes stderr where it used to send it to null. A noisy tool costs memory and up to the 200 ms drain wait.
runreturns 124 for a wait failure, the same code as a timeout. That makes the two indistinguishable to callers.- The ticket says "touches no call site", but the struct literals in
rocmandrocmdare rewritten toCommandCapture::from_text. The PR body discloses this, so this is only a note for the ticket. - Copilot's note about validating the executor on native Windows still stands. The Windows build and e2e lanes are green.
| /// before the capture moved onto the shared executor. | ||
| fn optional_stdout(capture: CommandCapture) -> Result<Option<String>, (&'static str, String)> { | ||
| match capture.outcome { | ||
| CommandOutcome::Exited(0) => Ok(String::from_utf8(capture.stdout_raw).ok()), |
There was a problem hiding this comment.
An Exited(0) capture whose detail is STILL_ARRIVING is returned as a full success here, so a tool that exits 0 but writes after the 200 ms grace silently yields a short answer. Consider at least a ROCM_CLI_DEBUG_COMMAND_CAPTURE debug line when detail is set, so the truncation is observable before the adapters are retired.
There was a problem hiding this comment.
Done in ce1fef8. When ROCM_CLI_DEBUG_COMMAND_CAPTURE is set, an optional capture that exited 0 now prints rocm debug: command capture output from <program> may be incomplete: <detail> whenever the capture carries a detail, which is a descendant still writing when the 200 ms grace ended or a pipe that failed to read. The answer the capture returns is unchanged, and a run that failed gets only its existing failure line, not a second one. The line and the answer come from one call in the tests: a_short_answer_from_the_optional_capture_comes_with_a_debug_line_and_a_whole_one_does_not runs a program whose helper writes after the grace (short answer plus the line) and one whose helper writes inside it (whole answer, no line), a_failed_run_gets_its_failure_line_and_no_incomplete_line covers the failed run, and only_a_run_that_exited_0_with_something_cutting_its_output_short_is_called_incomplete covers the decision on every outcome. I broke the code three ways (note never reported, note reported for every outcome, note computed but never handed to the debug sink) and each turned at least one of them red. The branch is also merged with current main (98e7e6a, 36cb66e and b3e062c), and docs/host-probing.md now says which adapter surfaces the note and which do not.
| let notes: Vec<String> = [ | ||
| stdout_failure.map(|error| format!("failed to read stdout: {error}")), | ||
| stderr_failure.map(|error| format!("failed to read stderr: {error}")), | ||
| (!drained).then(|| STILL_ARRIVING.to_owned()), |
There was a problem hiding this comment.
drained == false becomes only a detail note, and no adapter reads it today (per the PR text). Worth a tracking item so this silent-truncation case does not outlive the adapters.
There was a problem hiding this comment.
drained == false still only becomes a detail note on the capture, and that is deliberate for this step: the adapters keep their return shapes. What changed in ce1fef8 is that the optional capture no longer ignores it, so the truncation is observable there with ROCM_CLI_DEBUG_COMMAND_CAPTURE set. run, run_with_timeout and the Windows diagnostic capture still do not read it, and the PR body and the table in docs/host-probing.md now say that per adapter instead of saying no adapter reads it. Surfacing it where a probe can act on it belongs with retiring the adapters site by site, because a probe that calls run_bounded directly sees the typed outcome and the note without an adapter in between. I am keeping the follow-up as a separate tracking item so the case does not outlive the adapters; the text for it is below.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head b65d5b89f5e
Review — needs work
Full review of the whole change. Implements the first step of ROCMAI-925 (converging the host-probe subprocess runners onto one bounded executor), by its own account not the rest of that convergence.
Blocking
- A required-lane E2E check fails on the exact reviewed commit, flagged by the suite's own reconciliation as a regression, not a known flake — CI check
E2E tests (Strix Halo, Ubuntu, nightly, current)(run 42790014203, job 129740876880). Scenarioautomations-11("The background helper stops on its own once no watcher is enabled") fails its first assertion: immediately after the CLI reportshelper: started background automation daemon,rocm daemon --statusreportsStatus: not running. The suite's own summary is explicit:FAIL: 'automations-helper-stops-when-no-watcher-is-enabled' was expected to pass on this host but FAILED — a regression— distinct from the run's 8 toleratedXPASS (flaky, tolerated)entries, i.e. this scenario is not in the project's accepted-flake registry.
I traced the failing path (apps/rocm/src/automations.rs::render_daemon_text, which reads arunningflag from a state file the spawnedrocmddaemon writes, andapps/rocm/src/service_launch/helper_autostart.rs, which spawns that daemon with no readiness wait) and neither file is touched by this diff, so I could not establish from the code that this change causes it — it may be a pre-existing daemon-startup race that this run happened to surface. Regardless, it is a failing check on the commit under review and is not already known-flaky here.
Confidence 100 · architectural · Fix: re-run the job to rule out a transient flake; if it reproduces onmainat the same base commit, it's pre-existing and orthogonal to this PR — otherwise bisect for what in this diff could affect daemon-spawn timing.
Non-blocking
CommandCapture::from_textclones both strings before moving them —crates/rocm-core/src/command_capture.rs:141-153. It doesstdout.clone().into_bytes()/stderr.clone().into_bytes()then moves the originals into the text fields, paying two allocations where one would do. Only a hand-built, no-child capture path, so the cost is trivial. Confidence 70 · mechanical · Fix: derive the raw bytes without cloning theStrings (e.g.stdout.as_bytes().to_vec()).- The "timed out" + "still draining" combination has no dedicated test —
crates/rocm-core/src/process.rs:1264-1282(capture_of).outcome(forced toTimedOut) and the!drained→STILL_ARRIVINGdetail are independent booleans; each is tested alone but never together. Low risk since there's no interaction logic between them, so this is a coverage suggestion, not a correctness gap. Confidence 70 · logic · Fix: add (or note as deliberately omitted) a case withtimed_out: true, drained: false.
Decisions for the author
- A short/truncated capture now silently reads as an ordinary success — answered-by-intent. Traced independently in all three adapters:
host_gpu/capture.rs'soptional_stdout(CommandOutcome::Exited(0)arm never inspectscapture.detail),probe_exec.rs'srun_result(ignoresdetailentirely), andhost_gpu/diagnostics.rs'scapture_diagnostic_with(itsExited/Signaledarm also ignoresdetail). Before this PR,runand the optional capture always joined their reader to EOF, so a late-but-real write was never silently dropped; now a program whose meaningful output arrives just past the 200 ms drain grace (e.g. behind a lingering descendant) returns as an indistinguishable success with truncated output. This would otherwise be blocking ("behavior the change introduces that no test exercises" at the adapter level). The PR body owns it explicitly: "The adapters do not yet surface the newdetail, so a short capture still reads as a success torunand the optional capture, which is the silent truncation the second row above accepts and the note flags. A probe that wants to react to it is the next step (retiring the adapters site by site)." Confirm the blast radius (any CLI-invoked tool whose meaningful output could legitimately arrive from a backgrounded helper after the parent exits) is acceptable until that follow-up lands. - Unbounded reader-thread accumulation for a long-running daemon — tradeoff.
process.rs's own doc comment onCaptureacknowledges that each abandoned capture (a descendant still holding the pipe when the drain grace ends) leaves a reader thread parked inreaduntil the grandchild lets go, "so every probe that meets one costs two until it does" — bounded memory, unbounded thread count, called a deliberate trade. Worth confirming this still holds forrocmd's periodic watchers (a neighboring comment mentions a 60 s poll cadence) and not only the short-lived CLI invocations the tradeoff is framed around. - Ticket ROCMAI-925 named as closed by this PR's first step; not retrievable from this environment (internal AMD Jira tracker, no access from a public review) — named here per this repo's convention, not fetched or verified.
Positive signals
- The wait-loop backoff and drain-grace loop take
sleep: impl FnMut(Duration)as a parameter purely so a test can assert the actual production schedule (confirmed the 5/10/20/40/50/50 ms sequence) rather than inferring it from the constants in isolation. Settledcleanly separates "what happened to the child" from "what to do with that" (→Result<Output>orCommandCapture), which is why the dangerous paths (wait failure, timeout, read failure) are each unit-testable without a misbehaving kernel — one test explicitly pins that stdout/stderr bytes are never swapped in the read-failure case.docs/host-probing.md/docs/testing.md's citations of the new constants, function names and line-level claims were checked against the actual source (both subagents did this independently) and match exactly.
Deployment notes
None beyond the truncation behavior covered above — no new external dependency, config key, or migration.
What this covered
Diffed d027ff8e3ad9220ebb622667c97ed227d4b6c3e1...b65d5b89f5ec6e495b2c2f325749bbfa17ebac67 (16 files, +1935/-361) in a detached, read-only worktree fetched via plain git over https (never checked out in a shared clone). All 16 changed files read in full at head. Split into 5 coupled-file groups per the method: (1) process.rs + command_capture.rs (the executor/carrier pair) and (2) examine.feature + e2e.rs + examine_steps.rs + the three touched docs each ran as an independent, blind fan-out worker; (3) host_gpu/capture.rs + probe_exec.rs + lib.rs, (4) host_gpu/diagnostics.rs, and (5) the four CommandCapture::from_text call-site migrations (mcp_bridge.rs, sandbox.rs, driver_upgrade.rs, therock_update.rs) were read and verified by me directly rather than by independent workers, after repeated attempts to launch them hit this shared environment's concurrency ceiling — disclosed as a reduced-independence pass for those three groups, not a blind fan-out pass. Ran the agent-instruction-adherence pass myself (docs/architecture.md entries present for every touched file; process.rs's 1,500-production-line module-size guideline was already exceeded before this PR and not newly crossed by it, so not flagged; no tokio/spawn_blocking concerns apply since this code is synchronous std::process plus threads). Ran a light history pass (git log on the touched files); found nothing beyond this PR's own three commits. Did not run: a prior-changes pass (no earlier review round on these exact files to compare against), and no local build/test execution (no Rust toolchain in this environment — all verification is by reading, including manually re-deriving the old vs. new probe_exec.rs wait-failure handling from the base commit to confirm the 124 overlap between timeout and wait-failure is pre-existing, not newly introduced).
One deviation from a fully blind review, disclosed for transparency: this task's own pre-review duplicate-check step required fetching the PR's existing reviews before findings were reached, which incidentally surfaced the PR description and another human reviewer's published review (noting the same CI failure, the same truncation gap, and the same 124 overlap this report also raises). None of those were used as the source of a finding — every item above was independently re-derived and verified by me or by a blind fan-out worker against the code or the live CI run itself; the overlap reflects both reviews checking the same primary evidence, not reuse of the other's text. The PR description was read in full only after findings were otherwise settled, per Step 9, to check it against what was found; no mismatch between its claims and the diff was found beyond the item already retired above as answered-by-intent.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Signed-off-by: Jussi Elo <jussi.elo@amd.com> # Conflicts: # docs/architecture.md # docs/host-probing.md # tests/e2e-cucumber/features/examine.feature # tests/e2e-cucumber/tests/e2e/examine_steps.rs
…output may be short An optional capture that exited 0 returned its stdout as a complete answer even when the executor had stopped waiting for a descendant that was still writing, or a pipe had failed to read, so a short answer could not be told from a whole one. With ROCM_CLI_DEBUG_COMMAND_CAPTURE set it now prints a line naming the program and why the output may be incomplete. The answer itself is unchanged. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com> # Conflicts: # docs/architecture.md # docs/host-probing.md # tests/e2e-cucumber/features/examine.feature # tests/e2e-cucumber/tests/e2e/examine_steps.rs
There was a problem hiding this comment.
🔵 Needs a closer look
Native Windows runtime validation is needed because Linux fixtures, injected adapter outcomes, and cross-compilation do not verify inherited-pipe, late-output, and timeout behavior in the shared subprocess capture code.
No open issues 🎉
Review effort: Balanced
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…as not an advised command The advised-command scanner reads any string literal beginning with rocm as an invocation to parse, so the new debug line next to the existing capture failure line, which is already listed in NOT_INVOCATIONS as a debug log line, failed every_advised_command_parses. List it beside its sibling with the same reason, which also keeps the two lines sharing one greppable prefix. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head db6a5c14dc4
Review — no blocking findings
Full review of the whole change. Implements the first step of ROCMAI-925 (converging the host-probe subprocess runners onto one bounded executor), by its own account not the rest of that convergence.
Blocking
None.
Non-blocking
None.
Decisions for the author
- A fixed, universal 200 ms "drain grace" replaces three runners' previously different blocking tolerances — tradeoff.
runused to join its reader threads to EOF (unbounded), the optional capture likewise, whilerun_with_timeoutalready bounded itself; all three now share one 200 ms grace after the child's own exit/timeout/wait-failure. The code's own doc comment onprocess.rs'sCaptureand the PR text both call a per-call grace "undecided" and leave it as open follow-up — this is a live design question, not a defect, but worth the author confirming no current caller actually needed a longer grace (e.g. a probe whose own helper legitimately writes slightly after exit). - Reader-thread count is explicitly unbounded for an abandoned capture, memory is not — tradeoff, and deliberately so.
process.rs's doc comment states it plainly: "the number of these threads is not bounded here: each abandoned capture leaves one parked inreaduntil the grandchild lets go, so every probe that meets one costs two until it does," reasoned as acceptable because "a parked reader holds a stack and no CPU." This directly concernsrocmd's periodic watchers (confirmedapps/rocmd/src/host_snapshot.rsruns on a 60 s cadence through this same executor) — the comment addresses that daemon by name ("a buffer nobody will read again cannot grow without bound in a daemon that probes every 60 seconds"), so this is a reasoned, disclosed trade rather than a silent gap. Worth re-confirming it still holds once the later convergence steps (e.g.apps/rocm/src/therock/process.rs's still-unconvergedwait_with_output_bounded) land, since those have their own long-lived callers. - No adapter yet surfaces the new
detail/truncation to its caller, so a short-but-exited-0 answer reads identical to a complete one everywhere — tradeoff, confirmed self-consistent rather than a silent regression. Verified independently inhost_gpu/capture.rs(optional_stdout, which only emits a debug-channel note, opt-in viaROCM_CLI_DEBUG_COMMAND_CAPTURE),probe_exec.rs(run_result, ignoresdetailentirely, matching its pre-change behavior of discarding the same kind of error), andhost_gpu/diagnostics.rs(ignoresdetailin every outcome arm). Both the code's own comments and the PR text own this explicitly as the next step ("retiring the adapters site by site"), and no caller of any of the three currently branches on the missing information, so nothing currently silently misbehaves — just a known, disclosed gap to track.
Positive signals
- Commit
bea7cfffflands characterisation tests pinning the old behaviour of all three runners against shared fake-program fixtures beforedde4db8bdoes the actual convergence, and the convergence commit flips only the assertions meant to change — a clean, verifiable refactor-safety sequence, confirmed directly from the commit history rather than taken on faith. Capture's abandoned-buffer synchronization (process.rs, the reader thread andtake()both gating on the same mutex-guardedabandonedflag) was traced by hand and holds: a buffer nobody will read again stops growing the moment it's abandoned, closing the race the doc comment describes.docs/host-probing.md's "Decided and delivered, first step" table names, for every behavioural difference, the exact test that pins it and whether the adapter keeps the old answer, flags Windows coverage as explicitly "UNCONFIRMED," and states plainly what was not run. Unusually falsifiable documentation for a refactor of this shape — every function, constant and test name it cites was checked against the real source and matches.- The Windows-only diagnostic-capture translation logic (
capture_diagnostic_with) takes its runner as an injectable argument specifically so its full outcome-to-text mapping, including the retry-boundary logic, is unit-tested on Linux CI (and the Windows job), consistent with this repo's platform-gated-code testing rule — confirmed no branch of that logic is Windows-only and therefore silently untested on the lanes most contributors run. - The advised-command scanner's new
NOT_INVOCATIONSentry (apps/rocm/src/advised_commands.rs) was traced to a genuine internal debug-log string incapture.rs(never user-facing advice), sitting beside its pre-existing sibling with the same justification — a scoped fix, not a cover-up, and the scanner's own self-checking test (exclusions_still_match_something) guards against it going stale.
Deployment notes
mainhas moved inprocess.rs,lib.rs,docs/architecture.md,docs/testing.mdandtests/e2e-cucumber/tests/e2e.rssince this PR's base commit (confirmed by diffing the PR's stated base against currentorigin/main), consistent with GitHub reporting this PR'smergeable_stateasblocked. Not a code defect in the diff itself, but a rebase/re-merge will be needed, andprocess.rsis exactly the file most likely to conflict given the shared-executor work.- No new external dependency, migration, or config key.
What this covered
Diffed 62ac2f3137ff6b6406581432d7790738b0e882d1...db6a5c14dc48a688c26ed11427086317a93d0dd5 (17 files, +2054/-366; confirmed the merge-base of that base commit against the PR head equals the base commit itself) in a dedicated scratch clone, head fetched read-only via plain git over HTTPS and never checked out as a branch switch. All 17 changed files read in full at head by five independent, blind fan-out workers grouped by coupling: (1) process.rs + command_capture.rs (the executor/carrier pair), (2) host_gpu/capture.rs + probe_exec.rs + lib.rs (the probe-adapter layer), (3) host_gpu/diagnostics.rs (the Windows diagnostic capture), (4) the four CommandCapture::from_text call-site migrations plus advised_commands.rs, and (5) the e2e scenario/steps files plus docs/testing.md, docs/host-probing.md, docs/architecture.md. One worker in group 5 hit a concurrency limit on first launch and was re-spawned once, per the method's recovery rule; all five ultimately returned genuine, independently-reached results with zero findings scoring ≥70. I additionally ran the agent-instruction-adherence pass myself: module-size guideline (process.rs production code is ~1420 lines, under the 1,500 limit, both before and after this PR), docs/architecture.md entry accuracy for the touched modules, and the Gherkin-scenario requirement for user-observable behaviour (examine-48, confirmed non-vacuous against the real DRAIN_GRACE/timeout constants, and confirmed not colliding with any scenario id present on current origin/main, which still tops out at examine-47). Live CI was re-checked directly against this exact head commit (confirmed via the workflow run's own head_sha) immediately before writing this report: every completed check is green, including the E2E tests (Strix Halo, Ubuntu, nightly, current) lane whose equivalent run failed on an earlier head of this PR — it now passes at db6a5c14d. Two GPU-hardware lanes (W7900, MI300X) are still queued (pending self-hosted runner availability, not failing).
Did not run: a dedicated "prior changes" pass (no earlier review of these exact files on a different PR was sought out); a full git blame-level history pass beyond the commit-ordering check in group 1 and the base-vs-head size diff I ran myself; any local build or test execution (no Rust toolchain in this environment — all verification is by reading and by live CI's own results). Ticket ROCMAI-925, named by the PR as closed by this change, is not retrievable from this environment (internal tracker, no access here) — named per this repo's convention, not fetched or verified; the PR's own text, read after findings were settled, is otherwise consistent with the diff and discloses every open gap found above before this review reached it independently. Not reconciled against prior review discussion on this PR; this report was reached blind from the code and CI state only.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
I read through the diff and found nothing blocking. A few non-blocking notes:
CommandCapturegained fields, so the rocm chat bridge and the rocmd tests now build it throughfrom_text. The ticket says no call site is touched, but the PR body discloses this and I don't see a problem with it.- Output a descendant writes more than 200 ms after exit 0 is now dropped by
runand the optional capture; before, it was kept. It is disclosed and pinned by tests. Please confirm that's acceptable for slow-host probes such aswsl.exeorlspci. - The optional capture now pipes stderr instead of sending it to null, so a descendant that holds stderr open can cost up to the 200 ms grace. Same question: fine as a tradeoff?
- Nothing was run on a Windows host (disclosed), though the Windows CI jobs pass.
- Minor smell: the
Settled(Vec<u8>, Option<io::Error>)pair could be a small struct.
CI is green at this head apart from the GPU E2E lanes, which are pending on the runner outage. This reflects a read of the diff, not a run of the GPU lanes.
…d trade The drain grace is one value for every caller and a descendant that writes later than it loses output; say so beside the constant. The reader-thread count of an abandoned capture is unbounded by design and was documented only on the Capture type; list it among the open items of the host-probing note so it is tracked next to the per-call grace. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…d-outcome Signed-off-by: Jussi Elo <jussi.elo@amd.com>
After merging main, crates/rocm-core/src/process.rs was 1,601 production lines, over the 1,500-line limit cargo xtask check-module-size enforces. Move the executor (Capture, Settled, the wait and drain loops, run_with_timeout, run_bounded, MAX_QUOTED_OUTPUT_CHARS) and its tests into a private child module, process/bounded.rs, unchanged. process.rs re-exports the three public items so no caller's path changes. Add the new file's architecture.md row and point the docs that cite the executor and its tests at it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 70c00a73
Review — no blocking findings
Full review of the whole change (DEGRADED — see "What this covered"). Implements the first step of ROCMAI-925 (converging the host-probe subprocess runners onto one bounded executor), by its own account not the rest of that convergence; ROCMAI-925 is an internal tracker identifier, not retrievable from this environment, named here per this repo's own convention and not independently checked against its content.
Blocking
None.
Non-blocking
- The real production wiring for the new "output may be incomplete" debug line has no direct test —
crates/rocm-core/src/host_gpu/capture.rs:95-172. Every test of this feature (a_short_answer_from_the_optional_capture_comes_with_a_debug_line_and_a_whole_one_does_not,a_failed_run_gets_its_failure_line_and_no_incomplete_line, insubprocess_runners_pair) callscandidate_with_debugwith an injected sink closure, never the realcapture_optional_command_candidate_with_timeout→debug_command_capturepath that actually readsROCM_CLI_DEBUG_COMMAND_CAPTUREviacrate::runtime::env_flagand writes to stderr witheprintln!.env_flagitself (crates/rocm-core/src/runtime.rs:232) has no test anywhere in the crate (confirmed by grepping its only two call sites). A bug in the real gate — say, an inverted check that always suppresses the line, or one that always prints it — would pass every test this PR adds. This mirrors a gap a GitHub Copilot review flagged on an earlier head of this PR; re-derived independently here by reading the test bodies and the gate's call sites, not by reading that review's text. It does not regress anything: the pre-PRdebug_command_capture_failurehad the identical untested env-gate/eprintln!shape, and the decision logic this PR adds (when and what to print) is now thoroughly unit-tested, which it wasn't before. Confidence 70 · test-coverage · Fix: a test that sets the env var and asserts on captured stderr (needs care around process-wide env mutation and parallel tests), or a thinner seam that lets a unit test confirm the real gate calls into the already-tested decision logic.
Decisions for the author
- A single, universal 200 ms drain grace replaces three runners' different historical blocking tolerances — tradeoff, already disclosed in
process/bounded.rs'sCapturedoc comment and indocs/host-probing.md's open-items list ("whether the adapters... take a per-call drain grace"). Worth confirming no current probe needs a longer grace. - Reader-thread count for an abandoned capture is explicitly unbounded (memory is bounded, thread count is not) — tradeoff, documented in the same doc comment and named there against
rocmd's 60-second probe cadence. Reasoned and disclosed, not hidden. - None of the three adapters (
run, the optional capture, the diagnostic capture) yet surfaces the newdetail/truncation to its own caller, so a short-but-exited-0 answer still reads as an ordinary success everywhere except the opt-in debug channel — named explicitly in the PR body and indocs/host-probing.mdas the next step ("retiring the adapters site by site"). Not a silent regression: none of the three adapters told a short answer from a complete one before this PR either. - Ticket ROCMAI-925 named as closed by this PR's "first step" — the PR body itself repeatedly frames this as only the first of several planned steps (adapter retirement,
apps/rocm/src/therock/process.rs's still-unconvergedwait_with_output_bounded, the candidate-retry-on-timeout question), so the claim is about this step, not the whole ticket. Not independently checkable against the ticket's own content from here.
Positive signals
Capture's abandoned-buffer bookkeeping (process/bounded.rs) gates the abandon flag and the buffer emptying under the same mutex a concurrent reader thread's append-or-discard check also takes, closing exactly the race its own doc comment describes (a reader appending between "marked abandoned" and "buffer emptied"); traced by hand against the reader thread's loop and held up.- The wait-loop and drain-grace back-off schedule (5 ms doubling to 50 ms) is tested by capturing the actual
sleepcalls a fake clock receives (the_wait_loop_sleeps_the_back_off_schedule_between_polls,the_drain_wait_sleeps_the_back_off_schedule_until_both_readers_finish), not only the purenext_poll_intervalformula in isolation, so what a real call does is what's pinned. docs/host-probing.mdanddocs/architecture.md's new/updated entries cite specific function and test names; every one checked (acrosscapture.rs,bounded.rs,diagnostics.rs,probe_exec.rs,command_capture.rs, and thecommand_failure_detailcross-reference) matches the source at head.- The split of
process.rsintoprocess.rs+process/bounded.rs(to stay under this repo's 1,500-production-line module-size limit) reads as a genuine pure move on inspection: the moved functions' doc comments, logic and tests match what they replaced at the pre-split commit, modulo the new file's own module doc comment andusestatements.
Deployment notes
User-observable behavior change, covered by scenario examine-48 (new; no id collision with current main, which tops out at examine-47): a tool whose output-holding background process survives past the tool's own exit no longer holds up a host probe — the probe now returns after a 200 ms drain grace with whatever was printed before the tool exited, where it previously either waited for that background process too (unbounded) or, on a timeout, was not actually bounded by the timeout at all (tracked separately by issue #623). No new external dependency, migration, or config key.
What this covered
Diffed eb2a8194a2aa4323904b01ab15a573c5b9cb0fa3...70c00a7340aaaaf1d97ef7948f28f18589187fe2 (18 files, +3407/-1682) in a dedicated scratch clone, head fetched read-only via plain git over HTTPS and never checked out as a branch switch (one accidental git checkout <sha> -- . into the clone's own working tree was immediately reset with git reset --hard; nothing was pushed, committed, or modified in the actual PR). All 18 changed files read in full at head, plus the pre-split version of process.rs to check the final commit's "pure move" claim function-by-function.
DEGRADED: this ran single-context, not as a blind fan-out. All 6 parallel subagent spawns I attempted failed immediately on launch ("Concurrent subagent limit reached... Do not retry" — the sandbox's concurrency cap was already fully occupied by unrelated agents running in this shared environment), and the tool's own guidance says not to retry, so every file above was read and judged directly by me rather than by independent workers. Per the review-code method this is reduced coverage relative to a real fan-out; re-running with fan-out available is recommended, particularly a second independent pass over process/bounded.rs (1938 lines, the largest and most safety-critical file in the diff).
Ran: the agent-instruction-adherence pass against AGENTS.md (module-size limit — process.rs is 1405 total lines, comfortably under the 1,500-production-line limit; docs/architecture.md entries present and accurate for both process.rs and the new process/bounded.rs; the Gherkin-scenario requirement for user-observable behavior, examine-48; the NOT_INVOCATIONS self-check entry for the new debug-line string in advised_commands.rs, confirmed to match exactly what capture.rs emits). Read what CI already reports for this exact head commit (statusCheckRollup): every completed check is SUCCESS, including Windows build/test, E2E tests, coverage, clippy, CodeQL and the commit-signature gate; two checks (Skill checks (skillscope), Sphinx docs build) are SKIPPED, not failures. No Rust toolchain is available in this sandbox (cargo/rustc: command not found), so nothing was actually executed by me — every finding above is capped at confidence 70-85 (static reasoning, or a grep/diff that directly confirmed a specific claim), none at 100.
Did not run: a dedicated "prior changes" pass on this PR's own earlier review rounds. One disclosed deviation: this task's own pre-review duplicate-check step required fetching this PR's existing reviews (to check reviewer login and commit SHA) before my findings were reached, which incidentally returned the full text of earlier rounds, including a prior round of my own and a human reviewer's, both reaching materially the same "no blocking defects" conclusion and both independently naming several of the same tradeoffs (the drain grace, the unbounded reader-thread count, the un-surfaced detail). Every item in this report was re-derived and verified by me against the code directly (file:line evidence above), not copied from that text; the overlap reflects both passes reading the same primary evidence, not reuse. Not otherwise reconciled against any discussion on this PR. No local build, test, or cargo xtask run.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Closes ROCMAI-925.
This is the first step of converging the host probes' subprocess runners (pair 6 of 6 in docs/host-probing.md), and only that step: the runners keep their return shapes and no call site of
run,run_with_env,run_utf16leorcapture_optional_command*changes.run_with_timeoutstays the one bounded executor. Its spawn and wait are shared with a new crate-privaterun_bounded, which returns the run as aCommandCaptureinstead of turning a timeout or a failure into anErr.CommandCaptureis extended rather than duplicated: besides the lossy text and the legacyexit_statusit now holds the raw bytes of both streams, a typedCommandOutcome(not found, spawn failed with an error kind, timed out, wait failed, exited with a code, killed by a signal), theExitStatusof a reaped child and the error text the executor saw, and a capture whose readers were still going when the 200 ms drain grace ended says so in itsdetail; withROCM_CLI_DEBUG_COMMAND_CAPTUREset, the optional capture prints that note (or a pipe's read failure) as a debug line when the program exited 0, so a short answer can be told from a whole one.run_raw(underrun,run_with_envandrun_utf16le) andcapture_optional_command_candidate_with_timeoutbecome thin adapters overrun_bounded, and the Windows-onlycapture_diagnostic_commandruns on it too, with its candidate loop and the translation of each outcome incapture_diagnostic_with, which takes the runner as an argument so its unit tests run on Linux and on the Windows job. None of the three joins a reader thread any more, which is the unbounded join that issue #623 tracks. The wait loop starts at 5 ms and doubles to the 50 ms it used to sleep flat, so a probe that exits at once is not charged 50 ms where the older helpers polled every 20 to 25 ms.Because
CommandCapturegained fields, the hand-built captures inrocm(the chat bridge) androcmd(two watchers and the sandbox tests) moved toCommandCapture::from_text; nothing else changed in those files, and I list it because the ticket said no call site is touched and adding fields to a struct with public fields cannot honour that for its struct literals. The commits are in the order the note asks for: a characterisation commit that is green against the unchanged runners, the convergence commit that flips exactly the assertions that are meant to change, and the note updated with the decision.docs/architecture.mdanddocs/testing.mdare updated for the moved entries and the new scenario.Merging current main into this branch pushed
crates/rocm-core/src/process.rsover the 1,500-production-line limit thatcargo xtask check-module-sizeenforces (1,601 lines; this branch alone was under it), so the executor now lives in a new private child module,crates/rocm-core/src/process/bounded.rs: the timeout machinery (Capture,Settled, the wait and drain loops,run_with_timeout,run_bounded,MAX_QUOTED_OUTPUT_CHARS) with its tests. It is a pure move with no change to any moved line, andprocess.rsre-exportsrun_with_timeout,run_boundedandMAX_QUOTED_OUTPUT_CHARS, so no caller's path changes.docs/architecture.mdhas a row for the new file, anddocs/host-probing.mdlinks to it where it cites the executor and its tests.Nothing here is output-preserving without qualification. Each difference that moved, the pinning test, and whether the adapters keep the old answer (unit tests are in
crates/rocm-core/src/host_gpu/capture.rsmodulesubprocess_runners_pairunless a file is named):runand the optional capture kept it (they join their readers);run_with_timeoutlost itoutput_written_more_than_the_drain_grace_after_the_child_exits_is_lost_to_all_three, controloutput_written_within_the_drain_grace_after_the_child_exits_is_kept_by_all_three; thedetailthat marks a short capture isoutput_written_after_the_grace_is_missing_and_the_capture_says_soanda_capture_says_when_a_reader_was_still_going_at_the_end_of_the_grace(bounded.rs); the optional capture's debug line for it isa_short_answer_from_the_optional_capture_comes_with_a_debug_line_and_a_whole_one_does_not,a_failed_run_gets_its_failure_line_and_no_incomplete_lineandonly_a_run_that_exited_0_with_something_cutting_its_output_short_is_called_incompleterunand the optional capture lasted as long as the descendanta_descendant_holding_the_pipe_no_longer_extends_run_or_the_optional_capture, scenarioexamine-48runand the optional capture waited for the descendant, so the timeout bounded nothingrunstill returns124with the partial output that arrived, the optional capture still returnsNonea_descendant_holding_the_pipe_no_longer_outlasts_the_timeout_of_run_or_the_optional_capture; the plain timeout with no descendant, unchanged, isa_timeout_keeps_partial_output_for_run_and_quotes_it_for_the_bounded_executorrunwaited for the descendantrunreturns after the grace without the late stderra_descendant_holding_only_stderr_costs_the_optional_capture_at_most_the_drain_grace(bounds the call by the descendant's life; the extra 200 ms is below what a timing assertion can show)run_with_timeoutturns it into an errorrun_with_timeoutunchanged; the capture keeps the bytes and the real outcome and records the error (per stream) indetail; adapters ignore it as before, except that the optional capture also prints it on its debug channel when the run exited 0a_read_failure_on_either_stream_keeps_the_bytes_and_the_outcome_the_child_had(stdout, stderr and both),a_pipe_that_failed_to_read_does_not_change_what_run_returns(probe_exec.rs),a_pipe_that_failed_to_read_does_not_change_the_answer, and forrun_with_timeouta_read_failure_without_a_timeout_is_reported_rather_than_returned_as_output(bounded.rs). The older helpers' behaviour needs the kernel to fail a pipe read, so that column is derived from the code and not runrun_raw:124without killing the child, then an unbounded join; the optional capture: kills,None; the diagnostic capture: kills,failed to wait: <os error>run:124with the bytes that had arrived; the optional capture:None; the diagnostic capture:failed to wait:followed by the executor's own wording (failed to poll process: ...orfailed to reap timed-out process: ...)runno longer leaves the child running and the diagnostic text gains a prefixa_failure_to_wait_is_a_capture_that_keeps_the_bytes_that_had_arrived(poll and reap failure, bounded.rs),a_failure_to_wait_is_final_and_reports_no_output(diagnostics.rs),a_run_reads_as_the_rc_run_raw_has_always_returned(probe_exec.rs)run:127with empty output; the optional capture:Noneand a debug line with the OS error; the diagnostic capture: next candidate;run_with_timeout: an errora_program_that_cannot_be_started_is_127_empty_none_or_an_error,a_spawn_failure_reports_the_error_the_os_gave,a_program_that_never_starts_reports_the_last_launch_error(diagnostics.rs),a_program_that_cannot_be_started_is_a_capture_not_an_error(bounded.rs),a_program_that_exists_but_cannot_be_executed_is_a_spawn_failure_not_not_found(bounded.rs),only_a_missing_program_is_not_found(command_capture.rs)runlossy, the optional capture strict UTF-8,run_utf16lefrom the raw bytesoutput_that_is_not_utf8_is_lossy_for_run_none_for_the_optional_capture_and_raw_for_the_executor,utf16le_output_is_decoded_by_declaration_from_the_raw_bytes,a_program_that_exited_0_with_output_that_is_not_utf8_gives_no_answer_and_no_debug_linerun:-1; the optional capture:None;exit_status:1outcomenow saysSignaledwith the signal number on Unixa_program_killed_by_a_signal_has_no_exit_code,a_child_killed_by_a_signal_reads_as_exit_status_1(command_capture.rs),a_run_is_a_capture_with_its_exit_code_or_its_signal_and_both_streams, and on every hostan_exit_code_outranks_a_signal_and_no_code_is_a_signal,a_status_is_an_exit_code_or_a_signalrun_raw, 25 ms in the two capture loops, 50 ms flat inrun_with_timeoutrun_with_timeoutcallers get the same schedulethe_poll_interval_backs_off_from_a_few_milliseconds_to_the_flat_one, and the sleeps each loop really takes are recorded bythe_wait_loop_sleeps_the_back_off_schedule_between_pollsandthe_drain_wait_sleeps_the_back_off_schedule_until_both_readers_finish; the latency itself is not asserted because a timing assertion on a few milliseconds is flakytimed_out, bounded; next spelling only when a program could not be started, as before; the status text is still the platform'sExitStatusa_timeout_is_final_keeps_the_output_that_arrived_and_says_so,the_next_spelling_is_tried_only_when_a_program_could_not_be_started(diagnostics.rs, run on every host)Not changed and still open in the note: the candidate loop of
capture_optional_command_with_timeoutstill starts the next spelling after a timeout;wait_with_output_boundedinapps/rocmstill joins its readers with no limit; the runners still start a bare program name through the operating system, which resolves an emptyPATHentry as the current directory wherewhichdoes not; and a per-call drain grace is undecided; the number of reader threads an abandoned capture leaves parked until a descendant lets go is not bounded either (the memory is), whichdocs/host-probing.mdnow lists with the grace andprocess/bounded.rsdocuments onCapture. Only the optional capture surfaces the newdetail, and only as a line on its opt-in debug channel (ROCM_CLI_DEBUG_COMMAND_CAPTURE):run,run_with_timeoutand the Windows diagnostic capture ignore it, and no adapter changes what it returns, so a short capture still reads as a success to the caller, which is the silent truncation the first row above accepts and the note flags. A probe that wants to react to it is the next step (retiring the adapters site by site).User-observable behaviour does change, so there is a scenario:
examine-48inexamine.featureplants a native Linux root (the PCI probe is skipped on WSL) whoselspcistand-in onPATHprints one AMD GPU, exits 0 and leaves asleepholding its output for 30 seconds;rocm examine --jsonmust finish in under 15 seconds and still list the GPU. It runs on the Linux lanes and needs no GPU. With the executor's drain loop made unbounded it fails at 30.4 seconds, and with the captured stdout emptied it fails on the listing. It does not pin the length of the grace, which the unit cases do, and the loss of late output (first row) has no scenario because no tool the CLI runs is known to write more than 200 ms after it exits. The id follows the last one on main: #990 and #694 landed with 39 to 44 and #998 with 45 to 47, so this scenario is 48. The debug line has no scenario of its own: it is an opt-in stderr diagnostic that no existing scenario covers, so it is pinned by unit tests that run the real executor and read the answer and the line from one call.Mutation checks, each by breaking the code the test covers and watching it go red, then restoring and confirming the diff was byte-identical: wait-loop backoff replaced by the flat interval, drain wait replaced by the flat interval, the drain bound removed, the wait-failure bytes dropped, the timed-out outcome ignored, the read-failure detail dropped (stdout and stderr separately), the still-arriving flag dropped, not-found collapsed into spawn-failed, the signal ignored in the status mapping, the exit-code-versus-signal precedence swapped, the legacy
exit_statusdefault changed, the spawn error text dropped, the argv arguments dropped, stdout and stderr swapped in the capture, therunreturn code for a signal and for a timeout changed, the optional capture treating every exit as success and using lossy UTF-8, and the diagnostic capture retrying after a timeout or dropping the status. The debug line was checked three ways (the note never reported, the note reported for every outcome, the note computed but never handed to the sink), and each turned at least the test named for it red before the file was restored byte for byte. Every one turned at least the test named for it red. The scenario was checked with the unbounded drain loop and with the stdout emptied. An independent scored review of the branch also found a flaky readiness wait in a new test (fixed with a per-stream counter, 0 failures in 60 runs under CPU load), a back-off schedule that no test drove through the loops, a stderr read-failure branch with no test, and an overclaiming scenario comment; all four are fixed.Windows: the translation logic is behind injectable runners (
capture_diagnostic_with,run_result,optional_stdout,CommandOutcome::from_exit) with unit tests that run on both OSes, plus onecmdcase for the executor itself. Nothing was run on a Windows host:cargo clippy --workspace --all-targets --target x86_64-pc-windows-msvc -- -D warningswas run on Linux (C compiler and archiver stubbed, since nothing links), so the Windows job is the first run of those tests. The real-process timeout, grandchild and late-output cases use shell fixtures and are Linux only; it is unconfirmed whether a Windows child can be arranged to leave a late-writing descendant in the harness, so those rows rest on the executor's existing Windows test.Run locally before the second merge of main (main as of 62ac2f3):
cargo fmt --check;cargo clippy --all-targets -- -D warningsforrocm-core,rocm,rocmdande2e-cucumber; the Windows-target clippy above; the whole ofcargo test --no-fail-fast -p rocm --bins(1785 passed, 8 ignored) and-p rocm-core --lib(1445 passed, 6 ignored);cargo test -q --locked -p xtask prepush(98); the e2efeature_namingtest (4);cargo xtask check-architecture-docandcheck-module-size; and the examine scenarios 35 to 48 throughcargo xtask e2e -- -n 'examine-(3[5-9]|4[0-8]) '(14 of 14 passed), which includes this PR'sexamine-48. The debug line is listed inNOT_INVOCATIONSinapps/rocm/src/advised_commands.rsnext to the existing capture failure line, because the advised-command scanner reads any string that begins withrocmas an invocation to parse; the wholerocmsuite is what catches that, and an earlier head of this PR missed it by running only filtered tests.rocmd's own suite was last run on an earlier head. Not run:cargo xtask coverage,scripts/smoke_local.py, the full e2e suite (on this host without a GPU the five scenarios that need an AMD GPU fail at their first step, which is the host and not the change), anything on a GPU or Windows host, and CI.After that merge (main as of eb2a819) the module-size check failed on
process.rs(about 100 production lines over the limit with the managed-service stop routine merged in), so the executor was moved out as a pure move; on the final headcargo fmt --check,cargo clippy --all-targets -- -D warningsforrocm-core,rocmandrocmd,cargo xtask check-module-size,check-architecture-docandcheck-crate-edges, and the whole ofcargo test --no-fail-fast -p rocm-core(1496 lib tests passed, 6 ignored) all pass. Therocm --binsrun and the examine scenarios above were not repeated after the split, since the split changes no behavior; CI runs them on this head.The
E2E tests (Strix Halo, Ubuntu, nightly, current)check onb65d5b89ffailed onautomations-11(run 42790014203). It does not look caused by this change: that scenario starts the background helper, waits for the helper's state file to say it is running, then asksrocm daemon --statusand gotStatus: not running. The status text reads the same state file throughAutomationRuntimeState::load, which turns a read or parse error into "not running", and the helper rewrites that file with a plain truncate-and-write three times while starting, so a read that lands inside one of those writes reports not running. None of that code is in this diff, the helper's start-up path does not call the changed helpers, and the release-channel Ubuntu lane, the MI300X lane and the WSL2 and Windows lanes passed on the same commit (the release-channel lane ranautomations-11in the same minute). I could not reproduce it:automations-10andautomations-11passed 40 of 40 runs through the e2e harness under CPU load, and a probe that polledrocm daemon --statusafter enabling the watcher saw no running-then-not-running flip in 50 starts, so the torn read is the explanation the code points to and not one I have seen happen.