Skip to content

feat(vllm): Tackle out of memory errors (EAI-8058) - #251

Merged
r0x0r merged 57 commits into
mainfrom
gpu-out-of-memory
Oct 6, 2026
Merged

r0x0r merged 57 commits into
mainfrom
gpu-out-of-memory

Conversation

@r0x0r

@r0x0r r0x0r commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator

This pull request improves GPU selection and user guidance for ROCm and vLLM, especially in shared or containerized environments where the standard amd-smi tool may not be available. It adds a fallback for GPU VRAM telemetry, enhances user warnings and hints for out-of-memory (OOM) conditions, adds the vLLM startup OOM to the diagnose/fix catalog, and ensures consistent messaging across CLI and engine surfaces.

GPU selection and VRAM telemetry improvements:

  • Added a fallback to read per-GPU VRAM usage from the amdgpu DRM sysfs counters (/sys/class/drm/card*/device/mem_info_vram_{total,used}) when amd-smi is not available, so --gpu auto ranking and low-VRAM warnings still have telemetry on a single-AMD-GPU host without amd-smi. The fallback deliberately withholds telemetry on a host with more than one AMD DRM card, because card<N> numbering is not guaranteed to match HIP's device ordinal there — so it is a single-GPU convenience, not a multi-GPU replacement.
  • Explicit --gpu <index> validation was reworked on this branch to check membership in the KFD/DRM usable set rather than a device count. That is superseded by fix(serve): close serve races and translate GPU selection through visibility masks (EAI-7194) #267, which landed a validator consulting the visibility-resolved set directly (and with mask-aware wording), and the merge below drops this branch's version in favour of it — the two answered the same question, and fix(serve): close serve races and translate GPU selection through visibility masks (EAI-7194) #267's answer is the stronger one. The DRM sysfs VRAM rows are never used as a --gpu <index> bound; they stay scoped to --gpu auto ranking.
  • --gpu auto ranks the actual reported VRAM row indices, which can be non-contiguous under a visibility mask, rather than a synthetic 0..count range that would look up absent rows and fall through to a bogus GPU 0.

User guidance and warnings for vLLM:

  • Introduced a shared constant VLLM_GPU_MEMORY_UTILIZATION_HINT for the recommended workaround when running out of memory on a shared/busy GPU, ensuring CLI and engine logs use consistent wording.
  • The serve summary prints a note about the --gpu-memory-utilization workaround when vLLM is selected and the GPU is busy, both interactively and in the deployment summary.
  • vLLM engine startup logs append the same utilization hint if an OOM is detected.

vLLM startup OOM in the diagnose/fix catalog (EAI-8060):

  • Added a keyword-scored OOM signature (check_16_vllm_oom) plus a print-only fix-16-vllm-oom recipe. The remediation stays conditional — it distinguishes a tenancy collision with vLLM's fixed ~90% VRAM reservation from a model that genuinely does not fit — rather than prescribing a lower --gpu-memory-utilization unconditionally.
  • Only the lines carrying a vLLM anchor (vllm / gpu_memory_utilization) are scored — never the whole pasted log — so an incidental vllm mention cannot misattribute another framework's OOM. The anchor is not a boolean gate: because regex matching ignores line boundaries, gating on the anchor and then scoring the whole symptom still counted keyword hits from unanchored lines at full weight (a benign gpu_memory_utilization=0.9 config echo plus an unrelated torch.OutOfMemoryError: CUDA out of memory on the next line reached 95, above HIGH_CONFIDENCE). The same-line requirement now falls out of the scoring itself: an anchored line with no OOM token contributes nothing. Pinned by only_the_anchored_lines_are_scored_not_the_whole_paste.
  • gpu_memory_utilization remains an anchor even though it is also a 20-point scoring entry: with only anchored lines scored, a bare flag echo (a config dump, or a paste of rocm-cli's own low-VRAM hint, which prints --gpu-memory-utilization) is worth 20 — far below MIN_SCORE_FOR_MATCH — so it can surface as a weak signal but can never carry a verdict, nor lend its anchor to an OOM elsewhere in the paste (rocm_cli_s_own_low_vram_hint_is_never_a_verdict_on_its_own).
  • The vLLM OOM table alone is scored through keyword_score_collapsing_overlaps, which de-duplicates overlapping match spans, so one phrase yields one evidence bullet and one weight (HIP out of memory is not also counted as the nested out of memory). The rest of the catalog keeps the default "every matching entry is an independent signal" scoring: several older tables pair a greedy .* pattern with a second, independent keyword that the greedy span swallows, and collapsing there would push real diagnoses below MIN_SCORE_FOR_MATCH.
  • The engine's printed rocm diagnose --symptom '...' is decoupled from its own coarse detector: it consults vllm_oom_symptom_is_diagnosable and falls back to VLLM_OOM_CANONICAL_SYMPTOM when the user's real line would score sub-threshold, so the command it hands the user always reports a cause. The user's actual failing line stays visible in the human-readable message.
  • The diagnosis-embedded Fix and the rocm fix catalog are cross-checked for shared fix ids, so the two surfaces cannot silently drift.
  • On WSL, suppression of the routing note is gated on has_match (a checker actually cleared MIN_SCORE_FOR_MATCH), not on matched being non-empty, so a weak sub-threshold signal cannot bury the WSL setup guidance.

Documentation:

  • Updated docs/vllm.md, README.md, and skills/rocm-cli-assistant/SKILL.md to explain the behavior on shared/busy GPUs, the single-GPU-only telemetry fallback, the rocm diagnose --symptom pointer, and the recommended OOM workaround.

Behavior coverage (e2e) — per AGENTS.md §3, with the lane named for each:

@id: Behavior Tags / lane
diagnose-vllm-oom-is-conditional (diagnose-20) A vLLM startup OOM is diagnosed, its remedy distinguishes a busy GPU from a model that does not fit, and the rocm fix <id> the report names actually runs on the host that produced it @requires-os:linux — runs on every Linux lane including the GitHub-hosted mock lane; WSL2 matches this tag, so the self-hosted e2e-wsl lane is the one that can observe the platform-gate half
serve-vllm-low-vram-oom-guidance (serve-22) The serve plan warns the pinned GPU is low on VRAM and explains vLLM's total-VRAM reservation, not merely the flag name @requires-gpu @requires-engine:vllm @requires-os:linux — vLLM GPU hardware lane only
serve-absent-gpu-index-rejected (serve-16) An explicit --gpu <index> that is not usable is rejected outright, never silently remapped — under either validation authority @requires-gpu @requires-os:linux — GPU hardware lanes only

Why the two GPU-gated ones cannot run on the mock lane: on a no-GPU host the GPU-required pre-flight refuses with "no usable AMD GPU" before an index is ever validated or a plan is ever built, so neither the index-specific rejection nor the pre-launch VRAM note is observable there. serve-22 additionally injects its near-full reading through the e2e-test-hooks-gated ROCM_E2E_FORCE_LOW_VRAM=<ordinal> seam, because the GPU lane's real cards are comfortably free; the engine still launches against the real device. The low-VRAM seam is unit-tested on every lane by forced_low_vram_hook_trips_the_serve_plan_warning.

A coverage gap this branch had stated is now closed by #267. The masked sub-case of --gpu validation (a visibility mask leaving a non-contiguous usable set) had no scenario of its own here, because it was reachable only where amd-smi is absent. #267 revalidates against the visible set unconditionally and ships serve-19 (serve-masked-gpu-index-rejected) and serve-20 (serve-rocr-reindexed-gpu-index-rejected) for it, so the gap and the serve-16 comment that recorded it are both gone.

serve-22's two Then steps divide the work deliberately: the warning step is the scenario's premise (the low-VRAM warning predates this change) and pins only that the warning names the GPU the serve was pinned to; the memory-knob step is the verdict and requires the note's own reservation rationale, because matching the bare flag name would also be satisfied by a recipe echo of a user-supplied --gpu-memory-utilization. Both literals are additionally pinned in the always-run unit test serve_notes_pair_low_vram_with_the_vllm_utilization_hint, so a rewording fails on every lane rather than silently on the one lane that can observe the scenario.

Review follow-ups (2026-09-11).

  • fix-16-vllm-oom now applies on WSL2. check_16_vllm_oom is registered for ["linux", "wsl"], but the recipe was LINUX_ONLY, and rocm fix re-gates a fix-id on the recipe's list against the running OS — where WSL2 is its own family. So rocm diagnose printed apply with: rocm fix fix-16-vllm-oom and that command then printed the plan and exited 3 with "This fix only applies on: linux. Running OS is: wsl.", leaving the supported way to serve from a Windows host with a diagnosis and no fix path. Nothing about the remediation is bare-metal — it is vLLM CLI flags. The invariant is now asserted for the whole catalog by every_checker_platform_is_covered_by_its_recipe, which requires every platform family a checker answers on to be one its recipe will act on (containment, not equality: rocm fix <id> is reachable without a diagnosis). It surfaced no other mismatch across the remaining 24 entries (25 in the catalog, recounted after the merge).
  • The shared --gpu-memory-utilization hint said 0.1 while every command said 0.5. VLLM_GPU_MEMORY_UTILIZATION_HINT is printed verbatim by the serve low-VRAM note and the vLLM post-failure hint, and interpolated into the fix-16-vllm-oom diagnosis summary — so a single rocm diagnose printed "e.g. 0.1 for a small model" directly above rocm serve <model> --gpu-memory-utilization 0.5. The const is now 0.5, and the_utilization_hint_example_matches_the_recipe_command pins the hint's worked value to the value the recipe commands. Neither existing guard covered it: the diagnose/fix cross-check exempts the summary as prose framing, and a line-based grep misses the const because the flag and the value sit on different continuation lines.

Review follow-ups (2026-09-14).

  • The rocm diagnose --symptom '...' command the OOM hint prints is no longer built from unescaped subprocess output. The vLLM engine routed the user's actual failing line — taken verbatim from the engine's own startup-log tail — into a single-quoted shell argument in a message that invites the user to paste it. A single apostrophe, routine in Python error text (can't allocate, model 'foo'), closed the quote, and a line only has to mention running out of memory to be selected; the neighbouring human-readable echo passed the same bytes through including ANSI. A line that cannot be rendered as one intact single-quoted word is now treated as unusable and takes the VLLM_OOM_CANONICAL_SYMPTOM fallback that already existed for exactly that case, rather than being escaped: escaping keeps the exact bytes but yields a command a reader cannot check by eye, and a wrong escape is runnable and misleading instead of obviously broken, while control bytes would still reach the terminal. The user's own line stays visible in the human-readable sentence (and in the log tail printed with it), now stripped of ANSI sequences and control bytes. Pinned by an_apostrophe_in_the_failing_line_cannot_break_out_of_the_printed_command and control_bytes_from_the_log_are_stripped_from_the_echoed_line, both of which fail on the previous code.
  • DiagnoseReport::out_of_scope's doc comment matched the code again. It had been changed to assert that matched may hold sub-threshold entries alongside the field and that the field is set when nothing clears MIN_SCORE_FOR_MATCH. Neither is true: diagnose clears matched whenever out_of_scope is set, and the field is decided purely by catalog_covers — platform coverage, never a score. The (e.g. WSL2) example can no longer occur at all now that WSL2 is a covered platform. Since this is the serialized JSON contract, a consumer trusting it would wait for rows that can never arrive. an_out_of_scope_report_never_carries_matched_entries now pins the relationship across all four platform families.

Coverage note for the first item. The engine's post-failure OOM hint is only emitted when a real vLLM server process fails to start on a GPU host with an out-of-memory error, which no CI lane can produce (the GPU lanes' cards are free, and serve-22 forces only a low-VRAM reading, not a failed launch). The behaviour is therefore covered by the two unit tests above, which run on every lane, and the gap is stated here per AGENTS.md §3 rather than papered over with a scenario that cannot run. diagnose-20 continues to cover the rocm diagnose side, which is unchanged.

These improvements make GPU selection more robust in diverse environments and provide clear, actionable guidance to users encountering memory issues with vLLM.

Merge with main + a blocking fix (2026-09-15).

main moved under this branch (#331, #327, #267), leaving six conflict hunks — five in apps/rocm/src/main.rs, one in model_serving.feature. Merged (never rebased).

  • --gpu validation. This branch's resolve_pinned_gpu_index / validate_pinned_gpu_index_against are dropped in favour of fix(serve): close serve races and translate GPU selection through visibility masks (EAI-7194) #267's validate_pinned_gpu_index, which consults the visibility-resolved set and words its refusal differently depending on whether a mask is actually set. Keeping both would have been two validators for one question. effective_gpu_count survives: it is what lets --gpu auto rank devices when amd-smi is absent but the DRM sysfs fallback produced rows.
  • select_auto_gpu_index is a genuine union. Candidate ordinals come from the actual VRAM rows (this branch — a mask makes reported ordinals non-contiguous), and are then narrowed to the visible set (fix(serve): close serve races and translate GPU selection through visibility masks (EAI-7194) #267 — never target a masked-out device). Neither side alone is correct after the merge.
  • Scenario index collision. main published serve-19/20/21 and they are cross-referenced from src/expectation.rs and from each other, so they keep their numbers; this branch's low-VRAM OOM scenario shifts to serve-22. Its @id: is unchanged, so expectations.toml is unaffected.
  • Three defects the textual merge produced with no conflict markers, all caught by compiling: a validate_pinned_gpu_index tail that kept this branch's usable bail inside fix(serve): close serve races and translate GPU selection through visibility masks (EAI-7194) #267's function body; a vec![all[0]] fallback whose all binding lived on the other side of the hunk; and three select_auto_gpu_index call sites still on the old arity. A fourth was marker-free and compiled: a serve-16 comment naming a unit test the merge had deleted.

Blocking fix: the all-busy auto-selection fallback returned a GPU that was never reported. Once --gpu auto candidates come from the VRAM rows, the terminal "every candidate is busy, pick one anyway" fallback could no longer return a hardcoded 0. On the base branch that was safe by construction (candidates were 0..count, so ordinal 0 was always real). With rows reported at [2, 3] and both pinned by a managed service the candidate list empties, all three passes fall through, and the function returned [0] — an ordinal absent from the reported set, exported verbatim as HIP_VISIBLE_DEVICES. Under the strict GPU-required / no-silent-fallback policy that is a silently wrong pin, not a fail-fast. The fallback now returns the lowest ordinal that was actually reported, and the adjacent comment (which claimed "Every detected GPU is pinned... We still return GPU 0") is corrected. This is a fabrication removed, not a policy change: "all busy, pick one anyway" still holds, and rocm serve refuses in exactly the same situations as before.

auto_selection_all_reported_busy_falls_back_to_a_reported_ordinal covers it. The pre-existing auto_selection_ranks_the_actual_reported_row_indices claimed pass 3 "never returns a fabricated index 0" but only exercised the partially busy case, which never reaches the fallback. Falsified: reverting the fallback to vec![0] fails the new test with left: [0], right: [2]; restoring it passes. Equality assertion, not fragment absence.

Why no new scenario for the fallback (AGENTS.md §3). Reaching it needs every GPU the host reports to be simultaneously pinned by a live rocm-cli managed service, on a host whose reported ordinals are sparse. No lane can arrange that without a seam that would make the scenario assert against the seam rather than the product, and a serve under those conditions is a deliberate over-commit that the low-VRAM warning already reports. The selection function is pure and injectable, so the unit test covers the decision exactly; what is not covered live is the sparse-ordinal host itself, which is the same hardware gap #267 records on serve-20.

Review follow-ups (2026-09-17).

  • rendered_lines's never-merge guarantee is qualified where a consumer reads it. The doc stated it as an unqualified absolute, while the one counterexample was written down in a body comment inside the private next_token — so it reached no generated documentation — and the public doc pointed at it with See [next_token], a link to a private item that rustdoc will not resolve. The exception now has a single authoritative statement in the module documentation, which is public and rendered: a line break inside a well-formed string-argument body (OSC, DCS, SOS, PM, APC) is consumed with that body, worked example included. rendered_lines carries the qualifier inline so the only absolute a consumer can read is true, and links to the module docs instead. The behaviour is unchanged; following the grammar rather than guessing where an unterminated body ends is still the right call. a_line_break_inside_a_well_formed_string_body_is_consumed_with_it pins the worked example, plus the two shapes that bound the hole (the same break outside a string body still splits; an unterminated body still stops at the following ESC). cargo doc -p rocm-core --no-deps warned on the dangling link and goes from 6 warnings to 3 with this change — the remaining three are pre-existing on main.

Docs scope for this PR. docs/testing.md and docs/manual-testing.md are deliberately untouched. Neither file itemises fix IDs or diagnose signatures today — docs/testing.md documents how to run the suites and docs/manual-testing.md walks the install/serve/dash flows — so the three user-observable additions here (the fix-16-vllm-oom catalog entry, the OOM diagnosis, and the pre-launch low-VRAM serve warning) have no row to update in either. The user-facing surfaces that do describe them — README.md, docs/vllm.md, and skills/rocm-cli-assistant/SKILL.md — are updated in this PR, and each behaviour is covered by the @id:-tagged scenarios tabulated above. (Worth noting precisely: this branch's AGENTS.md carries no unconditional same-change docs rule. Its only such requirement is the scenario rule — "User-observable behavior needs a scenario, not only a unit test" — which this PR discharges in the table above. main's copy has since grown a stronger clause, "when a command's flags, defaults, arguments, or observable behavior change, update README.md, its --help/doc comment, docs/testing.md, and docs/manual-testing.md in the same change"; that text is not in this branch's AGENTS.md, and it will apply on the next merge from main, at which point the reasoning above is the answer: no flag, default or argument changed, and neither file states a behaviour claim this PR falsifies.)

On the three effective_gpu_count assertions (apps/rocm/src/main.rs): they are characterisation tests. They are revert-sensitive to the filter(|&count| count > 0) guard, but that guard predates this branch and this PR's own change to the function is doc-only — so they close a pre-existing coverage gap on a function the merge kept, rather than pinning behaviour this PR introduces.

@r0x0r
r0x0r requested a review from a team as a code owner August 13, 2026 13:06
@r0x0r
r0x0r force-pushed the gpu-out-of-memory branch 5 times, most recently from 6190726 to ee202f0 Compare August 19, 2026 08:35
@volen-silo

Copy link
Copy Markdown
Collaborator

A few observations from a read of this change:

  • No Gherkin scenario covers the new user-visible behavior (the extra serve note, --gpu auto picking from the sysfs fallback); AGENTS.md §3 asks for one or a stated reason. The unit tests cover helpers rather than behavior.
  • validate_pinned_gpu_index still sees only the amd-smi count, so on the exact environment this targets (no amd-smi, sysfs works) an out-of-range --gpu <n> is accepted and fails later in the engine.
  • read_sysfs_u64(...mem_info_vram_used).unwrap_or(0) makes a card with an unreadable used counter look 100% free — which is what --gpu auto prefers first.
  • read_drm_vram_usage orders by ascending card<N>; an AMD APU passes the same vendor + mem_info_vram_total filter, so on APU + dGPU the ordinal can diverge from HIP's, and it feeds HIP_VISIBLE_DEVICES directly.
  • docs/vllm.md: the fallback probes amd-smi only, never rocm-smi.

On the stack: #284's diff against this branch removes the sysfs fallback, its tests and the docs bullet — looks unintended.

r0x0r added a commit that referenced this pull request Aug 21, 2026
…back

Address review feedback on PR #251:

- resolve_gpu_indices now validates an explicit --gpu <index> against
  the DRM sysfs fallback's device count when amd-smi is unavailable,
  instead of only the amd-smi count. Previously an out-of-range index
  was silently accepted on a host with no amd-smi (sysfs works) and
  only failed later inside the engine.
- read_drm_vram_usage no longer treats an unreadable
  mem_info_vram_used counter as 0 bytes used (which made the card look
  100% free -- exactly what --gpu auto prefers first); it now skips
  that card instead.
- read_drm_vram_usage withholds telemetry entirely when more than one
  AMD DRM card is present, since ascending card<N> order is only
  guaranteed to match HIP's compute-topology ordinal on a single-GPU
  host (an APU passes the same vendor + mem_info_vram_total filter as
  a discrete GPU, so an APU+dGPU host could previously feed a
  diverged ordinal into HIP_VISIBLE_DEVICES).
- docs/vllm.md: corrected the fallback description, which only ever
  probes amd-smi (never rocm-smi) before falling back to DRM sysfs.

Extracted the count-fallback logic into a new effective_gpu_count
helper shared by --gpu auto ranking and --gpu <index> validation, and
added unit tests for all of the above.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r force-pushed the gpu-out-of-memory branch from ee202f0 to a97a39f Compare August 21, 2026 10:14
@r0x0r

r0x0r commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the detailed review — pushed a fix commit (a97a39f) addressing the concrete bugs:

  • validate_pinned_gpu_index only saw the amd-smi count: resolve_gpu_indices now derives an effective_gpu_count that falls back to the DRM sysfs row count when amd-smi is unavailable, so an out-of-range --gpu <n> is rejected up front on exactly the environment this PR targets (no amd-smi, sysfs works), instead of failing later inside the engine.
  • read_sysfs_u64(...mem_info_vram_used).unwrap_or(0): a card whose used counter can't be read is now skipped entirely rather than treated as 0 bytes used (100% free).
  • read_drm_vram_usage ordinal divergence on APU + dGPU: ascending card<N> order only mirrors HIP's KFD-topology ordinal when there's exactly one AMD DRM card. The function now returns no rows at all when more than one AMD card is found, instead of guessing an ordinal that could feed the wrong device into HIP_VISIBLE_DEVICES.
  • docs/vllm.md: fixed — the fallback only ever probes amd-smi, never rocm-smi; wording corrected.

Added unit tests for all four (effective_gpu_count_*, resolve_gpu_indices_rejects_out_of_range_index_from_sysfs_fallback_count, read_drm_vram_usage_skips_a_card_with_an_unreadable_used_counter, read_drm_vram_usage_withholds_telemetry_when_multiple_amd_cards_are_present).

On the Gherkin scenario: I didn't add one for the "no amd-smi, sysfs fallback" path — there's no runner in the current fleet shaped like that (real GPU hardware with amd-smi absent), so I couldn't author or verify a @requires-gpu scenario for it. The corrected behavior is covered at the unit level instead (pure functions, planted sysfs fixtures). Happy to add e2e coverage if/when a suitable lane exists.

On #284: worth double-checking before merging that stack — its diff against this branch appears to drop the sysfs fallback, its tests, and the docs bullet, which does look unintended given this PR is what introduces them.

r0x0r added a commit that referenced this pull request Aug 25, 2026
…back

Address review feedback on PR #251:

- resolve_gpu_indices now validates an explicit --gpu <index> against
  the DRM sysfs fallback's device count when amd-smi is unavailable,
  instead of only the amd-smi count. Previously an out-of-range index
  was silently accepted on a host with no amd-smi (sysfs works) and
  only failed later inside the engine.
- read_drm_vram_usage no longer treats an unreadable
  mem_info_vram_used counter as 0 bytes used (which made the card look
  100% free -- exactly what --gpu auto prefers first); it now skips
  that card instead.
- read_drm_vram_usage withholds telemetry entirely when more than one
  AMD DRM card is present, since ascending card<N> order is only
  guaranteed to match HIP's compute-topology ordinal on a single-GPU
  host (an APU passes the same vendor + mem_info_vram_total filter as
  a discrete GPU, so an APU+dGPU host could previously feed a
  diverged ordinal into HIP_VISIBLE_DEVICES).
- docs/vllm.md: corrected the fallback description, which only ever
  probes amd-smi (never rocm-smi) before falling back to DRM sysfs.

Extracted the count-fallback logic into a new effective_gpu_count
helper shared by --gpu auto ranking and --gpu <index> validation, and
added unit tests for all of the above.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Aug 25, 2026
…back

Address review feedback on PR #251:

- resolve_gpu_indices now validates an explicit --gpu <index> against
  the DRM sysfs fallback's device count when amd-smi is unavailable,
  instead of only the amd-smi count. Previously an out-of-range index
  was silently accepted on a host with no amd-smi (sysfs works) and
  only failed later inside the engine.
- read_drm_vram_usage no longer treats an unreadable
  mem_info_vram_used counter as 0 bytes used (which made the card look
  100% free -- exactly what --gpu auto prefers first); it now skips
  that card instead.
- read_drm_vram_usage withholds telemetry entirely when more than one
  AMD DRM card is present, since ascending card<N> order is only
  guaranteed to match HIP's compute-topology ordinal on a single-GPU
  host (an APU passes the same vendor + mem_info_vram_total filter as
  a discrete GPU, so an APU+dGPU host could previously feed a
  diverged ordinal into HIP_VISIBLE_DEVICES).
- docs/vllm.md: corrected the fallback description, which only ever
  probes amd-smi (never rocm-smi) before falling back to DRM sysfs.

Extracted the count-fallback logic into a new effective_gpu_count
helper shared by --gpu auto ranking and --gpu <index> validation, and
added unit tests for all of the above.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@volen-silo

Copy link
Copy Markdown
Collaborator

Round 2. Items 3 and 5 are genuinely fixed. Items 2 and 4 are fixed in intent, but the implementations leave a hole and add a new regression.

  • The multi-card guard counts surviving rows, not AMD cards found. In read_drm_vram_usage, three continues drop a card before cards.push(...) — vendor mismatch, missing mem_info_vram_total, missing mem_info_vram_used (the new item-3 skip). if cards.len() > 1 { return Vec::new() } then runs on the already-shrunk vector. On a genuine 2-AMD-card host where one card is filtered out, the guard doesn't fire and the survivor is assigned ordinal 0 regardless of its real card<N> — which feeds HIP_VISIBLE_DEVICES. That is the APU+dGPU misattribution the guard exists to prevent, and it's worst in exactly the case the doc comment reasons about (an APU whose mem_info_vram_total is absent gets dropped, the dGPU becomes "ordinal 0"). read_drm_vram_usage_skips_a_card_with_an_unreadable_used_counter uses a single-card tree, so it can't catch it. Count AMD cards in a separate counter incremented before the telemetry filters.

  • effective_gpu_count turns best-effort telemetry into a hard input-validation bound. A sysfs row count of 1 now hard-rejects --gpu 1..n at the CLI. Two ways that refuses a legitimate index: via the bug above (transient used read failures shrink the count), and via DRM-vs-KFD divergence — combine_amd_gpu_counts exists for precisely this and its own test asserts combine_amd_gpu_counts(Some(3), Some(1)) == Some(3) ("KFD larger than DRM, e.g. multi-partition compute nodes"). usable_amd_gpu_indices() is the amd-smi-independent authority here and is already used for this class of check in main.rs:4858, engines/vllm/src/lib.rs:1122, engines/lemonade/src/lib.rs:3784,3826. Deriving the count from it would also keep read_drm_vram_usage's withholding scoped to VRAM ranking rather than disabling the feature outright on multi-GPU hosts.

  • resolve_gpu_indices_rejects_out_of_range_index_from_sysfs_fallback_count doesn't call resolve_gpu_indices. It hand-composes validate_pinned_gpu_index(1, effective_gpu_count(None, Some(&single_gpu))). Reverting the production fix leaves it green. resolve_gpu_indices has no test caller at all.

  • On the Gherkin decline: agreed for the low-VRAM note: line — the harness has only tag-based host selection plus HIP_VISIBLE_DEVICES masking, no amd-smi shim or fixture sysfs tree. But AGENTS.md §3 wants the reason in the PR text, and @id:serve-absent-gpu-index-rejected (model_serving.feature:152) already covers the out-of-range rejection this PR changes behavior for — §3 asks that be named rather than silently relied on. Body edit, no code change.

Smaller things:

  • After the guard, cards.sort_by_key(...) + .enumerate() can only ever see 0 or 1 elements — the sort, the tuple sort key and the "Placeholder ordinal" comment are dead, and the doc comment's "assigns sequential 0-based ordinals in ascending card order" is not what the code does.
  • drm_device_is_amd checks vendor == 0x1002 only; rocm_core::is_amdgpu_device checks vendor or uevent containing DRIVER=amdgpu. The weaker one feeds the guard. The card walk also duplicates linux_drm_amdgpu_card_count line for line — both rocm-core helpers are private, so a shared exported enumerator would be better than a third, weaker predicate.
  • README.md:353 and skills/rocm-cli-assistant/SKILL.md:30 both still describe amd-smi as the only VRAM source.
  • docs/vllm.md:132 says the note prints "before launch" — true for the plain path, but in summary mode collect_serve_notes runs after start_managed_service and the smoke test. And the --gpu 3 example at :145 sits under the paragraph explaining the fallback only works single-GPU, where --gpu 3 is now rejected.
  • PR body says the fallback works "in stripped-down containers and shared nodes", but it yields nothing on any host with >1 AMD DRM card; docs/vllm.md:129 states the real scope correctly.
  • #[allow(clippy::too_many_arguments)] on collect_serve_notes at 8 positional args — a params struct would read better at the 4 call sites.

Untested: the gpu_vram_usage() amd-smi→sysfs chaining itself, the plain-path note: print, non-numeric sysfs contents.

@r0x0r

r0x0r commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the Round-2 review — pushed 3dcaa7b addressing it.

Multi-card guard counted surviving rows, not AMD cards found. Fixed. read_drm_vram_usage now increments a dedicated amd_cards_found counter the moment drm_device_is_amd returns true — before any telemetry (readable-counter) filtering — and withholds all rows when amd_cards_found > 1, so a second AMD card that yields no counter still suppresses the potentially-misnumbered telemetry. New test: read_drm_vram_usage_withholds_telemetry_when_a_second_amd_card_has_no_counter.

effective_gpu_count hard-bounded --gpu from best-effort telemetry. Fixed. --gpu validation now goes through pinned_index_bound, which prefers the amd-smi count and falls back to rocm_core::usable_amd_gpu_indices().map(|i| i.len()), never the VRAM-row count. effective_gpu_count is now used only for auto-ranking (doc updated). New tests: resolve_pinned_gpu_index_validates_against_the_detected_count, pinned_index_bound_prefers_the_amd_smi_count.

The "rejects out-of-range index" test didn't call resolve_gpu_indices. Fixed by extracting resolve_pinned_gpu_index(index, detected) (the Index arm) and testing it directly, replacing the old test that never exercised the path.

Body should name the declined scenario + its Gherkin reason. Done in the body: @id:serve-absent-gpu-index-rejected's premise (an index-specific rejection) can only be observed on a real GPU host, since a no-GPU host refuses at the GPU-required pre-flight first.

Smaller items: removed the dead sort/enumerate tail after the guard; drm_device_is_amd now matches rocm_core::is_amdgpu_device (vendor 0x1002 or uevent DRIVER=amdgpu); README and the assistant SKILL now mention the DRM sysfs VRAM fallback.

@volen-silo

Copy link
Copy Markdown
Collaborator

Round 3. The multi-card guard (A), the dead sort/enumerate tail, drm_device_is_amd, and the README/SKILL updates are all genuinely fixed — verified against source, not just the reply. Two things still block.

Blockers

  1. apps/rocm/src/main.rs:16751-16756 — pinned_index_bound collapses usable_amd_gpu_indices() to .len() and uses it as an upper bound. That function returns absolute, unrenumbered ordinals after applying the visibility mask — crates/rocm-core/src/lib.rs:11548 asserts usable_amd_gpu_indices_from(4, Some("2,0")) == Some(vec![2, 0]). So on a 4-GPU host with HIP_VISIBLE_DEVICES=2 and no amd-smi (detected == None — exactly this PR's target environment), pinned_index_bound yields Some(1) and --gpu 2 is hard-rejected as "out of range", even though 2 is the only usable device. The mirror case also holds: with usable == [2, 3], --gpu 0 passes CLI validation for a device that isn't visible at all.

    This is a regression: pre-PR, detected == None meant no CLI-side validation, and the request reached the engine gate, which does it correctly — engines/vllm/src/lib.rs:1139-1150 checks usable.contains(index), and engines/lemonade/src/lib.rs:3784 does the same. main.rs:16754 is the only call site in the repo that reduces that API to a length. The check wants membership, not a count.

  2. apps/rocm/src/main.rs:23962-23967 — pinned_index_bound_prefers_the_amd_smi_count only exercises Some(4) / Some(0); nothing calls pinned_index_bound(None). The .or_else fallback is the entire behavior this round added, and it's the branch carrying the bug above. As written it isn't injectable (it calls the real probe), so testing it means splitting a pinned_index_bound_from(detected, usable) seam — the same shape resolve_gpu_indices_against already uses in the engines. Same gap as round 2's item 3, one layer down.

  3. AGENTS.md §3 — the reply says the Gherkin decline reason and @id:serve-absent-gpu-index-rejected went into the PR body, but the body is unchanged from the original. §3 wants the named scenario and the gated-lane reason in the PR text. Body edit, no code change.

Nits (non-blocking)

  • docs/vllm.md:133 says the note prints "before launch". True on the plain path; in the default interactive summary mode collect_serve_notes runs at main.rs:5058, after start_managed_service and the smoke test.
  • docs/vllm.md:145 — the --gpu 3 example still sits under the paragraph explaining the fallback withholds telemetry on multi-GPU hosts, so the reader gets a 4-GPU example in the section about the case where none of this telemetry exists. Both new examples also drop --engine vllm, unlike every other bash block in the file.
  • PR body still claims the fallback makes things work "in stripped-down containers and shared nodes" and that "the device count for selection now also derives from these sysfs rows" — the first yields nothing on any host with >1 AMD DRM card, and the second is now auto-ranking only.
  • read_drm_vram_usage's card walk is still a line-for-line third copy of the enumeration in rocm_core::linux_drm_amdgpu_card_count + is_amdgpu_device. A single exported enumerator would keep the two from drifting again.
  • log_tail_shows_oom matches "out of memory" anywhere in the 80-line tail, so an unrelated failure whose tail happens to mention memory gets the hint appended. Low impact — the hint is additive — but it isn't guarded.

cargo fmt and cargo clippy --workspace --all-targets are clean; -p rocm --bins and -p rocm-engine-vllm --lib pass (one unrelated flake in providers::tests under parallel run, passes in isolation, untouched by this PR).

@r0x0r

r0x0r commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the Round-3 review — pushed 59c7f31 (rebased onto the current branch tip, which now carries #290's squash; one docs/vllm.md example-block conflict resolved to keep both your diagnose line and the --engine vllm fix).

Blocker 1 — the fallback reduced the usable set to a length. Fixed at the layer you named. The count bound is gone; --gpu <index> now validates membership. When amd-smi is present its renumbered 0..count is still the authority (index >= count → out of range); when it's absent, the index must be a member of usable_amd_gpu_indices() — the absolute, unrenumbered ordinals the engine gate already trusts. On your 4-GPU / HIP_VISIBLE_DEVICES=2 case, --gpu 2 (usable [2]) now passes and the mirror case — a hidden --gpu 0 with usable == [2, 3] — is rejected, with a message that names the usable set and the visibility vars. An empty usable set is deliberately not authoritative here (the no-usable-GPU fail-fast owns that message), so it doesn't reject.

Blocker 2 — the fallback branch wasn't injectable. Split the seam you asked for: validate_pinned_gpu_index_against(index, detected, usable). resolve_pinned_gpu_index supplies the real usable_amd_gpu_indices().as_deref(), and the new validate_pinned_gpu_index_falls_back_to_usable_set_membership test drives the detected == None branch directly (sole-usable passes, hidden rejected, empty set allowed through, amd-smi count still wins). The pre-existing tests now go through the same seam.

Blocker 3 — scenario + gated lane in the body. Done in the PR body (not just this comment). The user-observable "explicit --gpu <index> is rejected outright, never remapped" behavior is @id:serve-absent-gpu-index-rejected (Scenario 13, tests/e2e-cucumber/features/model_serving.feature), tagged @requires-gpu @requires-os:linux: on a no-GPU host the GPU-required pre-flight refuses with "no usable AMD GPU" before the index is ever validated, so the index-specific rejection is only observable on the GPU hardware lanes. The membership refinement is unit-covered on every lane.

Nits. Fixed the two docs/vllm.md inaccuracies (the "before launch" timing now also names the interactive post-readiness path; both workaround examples gained --engine vllm, and the stray 4-GPU --gpu 3 example is now --gpu 1). Corrected the PR-body overclaims you flagged — "stripped-down containers and shared nodes" is now scoped to single-GPU, and the "device count for selection derives from sysfs rows" line is gone (that path is auto-ranking only). Left read_drm_vram_usage's card-walk dedup and the broad log_tail_shows_oom match as noted-but-deferred; happy to fold the DRM enumerator into rocm_core in a follow-up if you'd rather have it here.

Verification note: the full cargo test/clippy workspace build can't run on my macOS box — an unrelated st_mode u16/u32 libc mismatch in engines/lemonade, and macOS is unsupported per AGENTS §6 — so the Linux CI lanes are the authoritative gate for the compiled and e2e checks. cargo fmt and rust-analyzer are clean locally.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a well-tested change and the core GPU-selection/validation logic (the amd-smi-independent membership check, the DRM sysfs fallback and its multi-card guard, the conditional OOM diagnosis) looks sound after the several rounds of review already visible on the thread. I only found one new issue, introduced in the latest commit while resolving a rebase conflict in the docs.

In docs/vllm.md, the "Explicitly, the workaround for an OOM on a shared card is:" example has a duplicated ```bash fence opener right before the example commands. As written, the block opens, immediately closes/reopens on the literal text, and the commands after it are no longer inside a fenced code block for the rest of that section (until the next ```` ) closes it) - so the example won't render as a code block. Please drop the duplicate line.

Comment thread docs/vllm.md Resolved
@r0x0r
r0x0r force-pushed the gpu-out-of-memory branch from 59c7f31 to e94ca5f Compare August 28, 2026 11:11
@r0x0r

r0x0r commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — good catch, the merge-conflict resolution had left two consecutive bash fence openers before the shared-card workaround example, so the block reopened on the literal text and the commands rendered outside the fence. Fixed in e94ca5f (now the PR head): dropped the duplicate opener so it renders as a single fenced block. No logic touched — docs-only.

@r0x0r
r0x0r requested a review from rominf August 31, 2026 08:52
@r0x0r

r0x0r commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

@rominf the duplicated ```bash fence opener in docs/vllm.md is removed, so the "workaround for an OOM on a shared card" example is a single fenced code block again (the commands render inside it rather than as literal text). Ready for another look — thanks.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewing at e94ca5f1 — the docs fence I flagged earlier is fixed, thanks. The four layers here are individually well built: the keyword table's own comment explains why a bare "out of memory" is sub-threshold, VLLM_ANCHOR_PATTERN exists precisely because the torch OOM shape isn't vLLM-specific, and the WSL rework is careful to keep sub-threshold hits in matched rather than dropping them. --gpu auto ranking on free VRAM with the multi-card ordinal guard in read_drm_vram_usage is a good piece of work.

My concerns are all at the seams between the layers, where each side is individually reasonable and the pair isn't.

The biggest: the engine detects an OOM with a bare substring scan, then tells the user to run rocm diagnose --symptom 'vllm: <the failing line>' — but the diagnose table scores several of the lines the engine accepts below MIN_SCORE_FOR_MATCH. I replicated keyword_score against the table in this diff rather than eyeballing it:

DET score=  0 NONE   vllm: RuntimeError: hipErrorOutOfMemory
DET score=  0 NONE   vllm: torch.cuda.OutOfMemoryError
DET score= 25 WEAK   vllm: HIP error: out of memory
DET score= 25 WEAK   vllm: the process was killed: out of memory
DET score= 90 LIKELY vllm: torch.OutOfMemoryError: HIP out of memory. Tried to allocate 7.21 GiB.

DET means log_tail_shows_oom accepted it, so the engine printed the command. The first two produce nothing at all. Details inline.

Second: hip out of memory (45) and out of memory (25) both match the same token span, and keyword_score sums the top two, so one phrase yields 70 and two "independent" evidence bullets. That's the exact overlap torch_oom_class_alone_is_not_a_vllm_match guards against for the PyTorch pattern — the guard just isn't applied here.

Third: making out_of_scope depend on the symptom turns it from a host property into a host+symptom property, and diagnose-states-whether-the-platform-is-covered asserts it agrees with examine's status — which is unconditionally "wsl" on WSL. There's no WSL lane in CI (the scenario's own comment says so), so this lands on a developer.

For what it's worth I checked and disproved several things, so they're not worth your time: applies_on: LINUX_ONLY does not block fix-16 on WSL; \boutofmemory\b and torch\.outofmemoryerror don't double-count; wsl_out_of_scope_message is byte-identical to main; fix-16 is the only checker registered for "wsl".

On CI — the red E2E tests (GPU) lane looks like a stale base rather than this PR. All four failures share device_policy: gpu_required; no active ROCm runtime is configured, which 283811f (#322) fixed on main; main's latest self-hosted run is green. A rebase should clear it, and is worth doing before merge so the lane is actually informative.

Comment thread engines/vllm/src/lib.rs Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
Comment thread apps/rocm/src/main.rs Resolved
Comment thread apps/rocm/src/main.rs Resolved
Comment thread crates/rocm-core/src/fix.rs Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
Comment thread apps/rocm/src/main.rs Resolved
r0x0r added a commit that referenced this pull request Sep 3, 2026
…back

Address review feedback on PR #251:

- resolve_gpu_indices now validates an explicit --gpu <index> against
  the DRM sysfs fallback's device count when amd-smi is unavailable,
  instead of only the amd-smi count. Previously an out-of-range index
  was silently accepted on a host with no amd-smi (sysfs works) and
  only failed later inside the engine.
- read_drm_vram_usage no longer treats an unreadable
  mem_info_vram_used counter as 0 bytes used (which made the card look
  100% free -- exactly what --gpu auto prefers first); it now skips
  that card instead.
- read_drm_vram_usage withholds telemetry entirely when more than one
  AMD DRM card is present, since ascending card<N> order is only
  guaranteed to match HIP's compute-topology ordinal on a single-GPU
  host (an APU passes the same vendor + mem_info_vram_total filter as
  a discrete GPU, so an APU+dGPU host could previously feed a
  diverged ordinal into HIP_VISIBLE_DEVICES).
- docs/vllm.md: corrected the fallback description, which only ever
  probes amd-smi (never rocm-smi) before falling back to DRM sysfs.

Extracted the count-fallback logic into a new effective_gpu_count
helper shared by --gpu auto ranking and --gpu <index> validation, and
added unit tests for all of the above.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r force-pushed the gpu-out-of-memory branch from e94ca5f to e1adff1 Compare September 3, 2026 08:38
@r0x0r

r0x0r commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @rominf — pushed e1adff1 (rebased onto latest main) addressing each point. Summary of what changed:

Engine hint vs. scorer coupling (the printed --symptom could report nothing).
The vLLM engine no longer hard-codes vllm: <failing line> into the rocm diagnose --symptom '…' it prints. oom_utilization_hint now consults rocm_core::vllm_oom_symptom_is_diagnosable(&candidate) and, when the user's actual line would not itself clear the match threshold, falls back to rocm_core::VLLM_OOM_CANONICAL_SYMPTOM (which carries two independent signals and is guaranteed to score). The user's real failing line still shows in the human-readable message; only the copy-pasteable command is normalized. New loop-invariant test every_emitted_oom_symptom_is_diagnosable asserts that whatever the engine emits is diagnosable — including bare lines that trigger the fallback.

Double-counted keyword spans / "one bullet per phrase".
keyword_score now collects each match's span and drops a hit whose span overlaps an already-kept stronger hit before taking the top two, so out of memory nested inside HIP out of memory is not counted twice. The KEYWORDS_VLLM_OOM table now deliberately splits allocator messages (hip out of memory, cuda out of memory, hipErrorOutOfMemory → 50, clear alone) from the exception class name (torch.OutOfMemoryError → 45, sub-threshold on its own), which keeps torch_oom_class_alone_is_not_a_vllm_match and a_bare_out_of_memory_stays_below_the_match_threshold honest.

Same-line anchor.
The vllm / gpu_memory_utilization anchor must now appear on the same line as an OOM token (vllm_oom_line_is_anchored), not merely somewhere in a pasted log — closing the misattribution where a stray vllm mention re-enabled attribution of an unrelated framework's OOM. New test vllm_anchor_must_be_on_the_same_line_as_the_oom_token.

WSL out-of-scope routing.
The WSL branch now suppresses on any_cleared_threshold(&wsl_matches) rather than "matches is non-empty", so a genuine vLLM OOM on WSL is surfaced while a sub-threshold signal is preserved and still routed out of scope. Tests: a_genuine_vllm_oom_match_on_wsl_is_surfaced_not_routed_out_of_scope, wsl_sub_threshold_vllm_signal_is_preserved_but_still_routed_out_of_scope.

serve --gpu <index> rejection wording.
The usable-set membership rejection now reads --gpu index N is not available; the usable AMD GPU(s) are […], matching the engine wording and the e2e serve-absent-gpu-index-rejected contract ("not available"). Test updated.

--gpu auto on non-contiguous ordinals.
select_auto_gpu_index now ranks the actual reported VRAM row indices (which can be non-contiguous when devices are hidden by a visibility mask) instead of a synthetic 0..count, so it can no longer look up absent rows and fall through to a bogus GPU 0. New test auto_selection_ranks_the_actual_reported_row_indices constructs the real amd-smi non-contiguous-index state.

Duplicated fix-16 remediation text.
Reconciled the verify/notes/executable commands between the rocm fix catalog (fix::RECIPES) and the diagnosis-embedded Fix for fix-15/fix-16 (they had diverged on a two-vs-three-space verify and a reworded note), and added a cross-check test diagnosis_remediation_matches_the_fix_catalog_for_shared_fix_ids so they cannot silently drift again. The prose framing (catalog rationale vs. diagnosis summary) legitimately differs by surface, so that is intentionally not asserted equal.

Pre-launch low-VRAM note scenario.
The pre-launch low-VRAM guidance is still unit-covered here; its user-observable Gherkin scenario (@id:serve-oom-memory-guidance) lands in the stacked follow-up #284, which depends on this branch. Called out so the coverage gap is explicit rather than silent.

Full workspace cargo test --all-targets, clippy -D warnings, and scripts/smoke_local.py are green locally; the pre-push gate (fmt/clippy/test) passed on push.

@r0x0r
r0x0r requested a review from rominf September 3, 2026 08:39
r0x0r added a commit that referenced this pull request Sep 3, 2026
…back

Address review feedback on PR #251:

- resolve_gpu_indices now validates an explicit --gpu <index> against
  the DRM sysfs fallback's device count when amd-smi is unavailable,
  instead of only the amd-smi count. Previously an out-of-range index
  was silently accepted on a host with no amd-smi (sysfs works) and
  only failed later inside the engine.
- read_drm_vram_usage no longer treats an unreadable
  mem_info_vram_used counter as 0 bytes used (which made the card look
  100% free -- exactly what --gpu auto prefers first); it now skips
  that card instead.
- read_drm_vram_usage withholds telemetry entirely when more than one
  AMD DRM card is present, since ascending card<N> order is only
  guaranteed to match HIP's compute-topology ordinal on a single-GPU
  host (an APU passes the same vendor + mem_info_vram_total filter as
  a discrete GPU, so an APU+dGPU host could previously feed a
  diverged ordinal into HIP_VISIBLE_DEVICES).
- docs/vllm.md: corrected the fallback description, which only ever
  probes amd-smi (never rocm-smi) before falling back to DRM sysfs.

Extracted the count-fallback logic into a new effective_gpu_count
helper shared by --gpu auto ranking and --gpu <index> validation, and
added unit tests for all of the above.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r added this pull request to the merge queue Oct 1, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 1, 2026
@r0x0r
r0x0r added this pull request to the merge queue Oct 1, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 1, 2026
r0x0r added 4 commits October 2, 2026 10:13
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	tests/e2e-cucumber/features/model_serving.feature
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	crates/rocm-core/src/fix.rs
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	crates/rocm-core/src/lib.rs
`fix-16-vllm-oom` is added to the CLI's closed catalog by this branch, but
`skills/rocm-doctor/reference.md` still described `fix-16` as "a reserved
handle, not a missing row". The skill doc is asserted against the CLI
catalog by three e2e scenarios, and all three failed on the merge:
`rocm fix` offers fix-16-vllm-oom, which reference.md does not document.

Add the catalog row (linux/wsl, not auto-applicable, matching the recipe's
LINUX_AND_WSL and auto_applicable: false), raise the catalog count from 24
to 25, and move the reserved-handle note to `fix-18`, which is still the
only gap in the numeric series.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r enabled auto-merge October 2, 2026 12:45
@r0x0r
r0x0r added this pull request to the merge queue Oct 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 2, 2026
r0x0r added 5 commits October 5, 2026 09:46
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	docs/vllm.md
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	crates/rocm-core/src/lib.rs
#	tests/e2e-cucumber/features/diagnose.feature
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	crates/rocm-core/src/diagnose.rs
#	crates/rocm-core/src/fix.rs
#	skills/rocm-doctor/reference.md
#	tests/e2e-cucumber/features/diagnose.feature
`check_16_vllm_oom` built its `Fix` with an inline `auto_applicable: false`.
`no_checker_hand_sets_auto_applicable` forbids that: only
`take_applicability_from_the_catalog` may set the field, because the catalog
pass overwrites whatever a checker puts there. The output was right either
way -- which is the point. A checker that disagreed with the catalog would be
silently masked rather than caught, the way `fix-9-igpu-dgpu` once drifted.

The field is now left at `Fix::default()` and filled from the recipe, whose
`applies_on: PRINT_ON_LINUX_AND_WSL` already carries `FixClass::PrintOnly`.

This was latent rather than new: the guard never ran on the previous head
because the suite aborted at an unrelated flaky test first, so the offending
line reached `gpu-out-of-memory` unnoticed.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	tests/e2e-cucumber/features/diagnose.feature
@r0x0r
r0x0r added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit 5ed9365 Oct 6, 2026
24 checks passed
@r0x0r
r0x0r deleted the gpu-out-of-memory branch October 6, 2026 09:01
r0x0r added a commit that referenced this pull request Oct 6, 2026
Resolving the `gpu_vram_usage` conflict after #251 was squash-merged, I kept
this branch's pre-refactor body and then restored main's dispatcher -- but not
`simulated_low_vram_usage()` or its `#[cfg(feature = "e2e-test-hooks")]` call,
both of which arrived with #251. The symbol count fell from main's 4 to 2,
leaving only the two dangling references in the test module.

Nothing in the default build reaches the hook, which is why 21 CI checks
stayed green over it. It breaks only under the feature:

    cargo check -p rocm --features rocm/e2e-test-hooks --tests
    error[E0425]: cannot find function `simulated_low_vram_usage`

and the step at serving_steps.rs:865 still set ROCM_E2E_FORCE_LOW_VRAM with
nothing left to read it -- a scenario quietly asserting nothing.

Restored verbatim from main: both the hook and its gated call are now
byte-identical to origin/main, and the symbol count matches again.

Verification note: the Linux devbox is unavailable (its pod has been Pending
~90m behind a 502 from the node proxy) and macOS cannot build this workspace
-- `cargo check` stops in rocm-engine-lemonade on `u32 & u16` long before
reaching this crate -- so the feature-gated compile has NOT been re-run here.
It is pending, and the restored text is identical to the known-good tree.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
rominf added a commit that referenced this pull request Oct 7, 2026
…ntract

The merge from main brought #251's selection: the device count falls back
to the VRAM telemetry rows when amd-smi reports none, candidates come from
the rows' actual (possibly sparse) ordinals rather than `0..count`, the
listed `0..n` range is unioned back in when every row falls inside it, and
the result is narrowed to the visible set. The old oracle bounded choices
by the amd-smi count alone, so it rejected a pick the contract requires
(no amd-smi count, one telemetry row at ordinal 0, GPU 0 chosen).

The oracle is now an independent statement of that contract: which
ordinals are present and visible, busy GPUs avoided while a free one is
choosable, a pin whenever something is choosable, and none when there is
neither a count nor a telemetry row. The generator reaches a missing and a
zero amd-smi count with and without rows, sparse row ordinals, rows at or
beyond the listed count, and masks hiding some rows; a fixed-seed reach
test asserts a floor for each shape.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
juhovainio pushed a commit that referenced this pull request Oct 7, 2026
* feat(vllm): add GPU memory utilization hint for OOM errors and improve documentation

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): validate --gpu against sysfs GPU count and harden DRM fallback

Address review feedback on PR #251:

- resolve_gpu_indices now validates an explicit --gpu <index> against
  the DRM sysfs fallback's device count when amd-smi is unavailable,
  instead of only the amd-smi count. Previously an out-of-range index
  was silently accepted on a host with no amd-smi (sysfs works) and
  only failed later inside the engine.
- read_drm_vram_usage no longer treats an unreadable
  mem_info_vram_used counter as 0 bytes used (which made the card look
  100% free -- exactly what --gpu auto prefers first); it now skips
  that card instead.
- read_drm_vram_usage withholds telemetry entirely when more than one
  AMD DRM card is present, since ascending card<N> order is only
  guaranteed to match HIP's compute-topology ordinal on a single-GPU
  host (an APU passes the same vendor + mem_info_vram_total filter as
  a discrete GPU, so an APU+dGPU host could previously feed a
  diverged ordinal into HIP_VISIBLE_DEVICES).
- docs/vllm.md: corrected the fallback description, which only ever
  probes amd-smi (never rocm-smi) before falling back to DRM sysfs.

Extracted the count-fallback logic into a new effective_gpu_count
helper shared by --gpu auto ranking and --gpu <index> validation, and
added unit tests for all of the above.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): validate --gpu against amd-smi-independent count; harden DRM guard

Round-2 review follow-ups on the DRM sysfs GPU-selection fallback.

- Validate an explicit --gpu <index> against the amd-smi list count or, when
  amd-smi is absent, the KFD/DRM authority behind usable_amd_gpu_indices()
  (pinned_index_bound) instead of the DRM VRAM-fallback row count. The row
  count withholds telemetry on multi-GPU hosts and shrinks on a transient
  counter-read failure, so borrowing it as a validation bound hard-rejected
  legitimate indices. VRAM rows stay scoped to --gpu auto ranking.
- The multi-card ordinal-ambiguity guard now counts AMD cards *found* (before
  the telemetry filters) rather than surviving rows, so a second AMD card with
  an unreadable counter still trips the guard instead of mislabelling the
  survivor ordinal 0 (the APU+dGPU misattribution the guard exists to prevent).
- drm_device_is_amd now matches on vendor id OR an amdgpu uevent DRIVER line,
  the same two-signal test as rocm_core::is_amdgpu_device, so the fallback probe
  and the count authority agree on what an AMD card is.
- Drop the now-dead sort/enumerate tail in read_drm_vram_usage (at most one card
  survives the guard) and fix the stale doc comment.
- Tests: exercise the real Index-arm delegate (resolve_pinned_gpu_index) and add
  a two-AMD-card-with-unreadable-counter regression; add pinned_index_bound
  coverage.
- Docs: README and the assistant SKILL note the DRM sysfs VRAM fallback rather
  than presenting amd-smi as the only source.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060) (#290)

* feat(diagnose): add vLLM out-of-memory failure mode to the catalog (EAI-8060)

Add a keyword-scored OOM signature to the rocm-core diagnosis catalog and a
matching print-only remediation recipe. The error covers two distinct faults
(a tenancy collision with vLLM's fixed ~90% VRAM reservation, versus a model
that genuinely does not fit), so the wording stays conditional: it never
prescribes lowering --gpu-memory-utilization as the unconditional answer, and
the verify step avoids the tenancy knob. Because an Examination carries no
per-GPU VRAM or tenancy fields, the match is keyword-only and the checker is
gated to Linux (vLLM is Linux/WSL-only).

- diagnose.rs: KEYWORDS_VLLM_OOM table + check_16_vllm_oom (linux-only)
- fix.rs: print-only fix-16-vllm-oom recipe; catalog count 15 -> 16 and
  AUTO-set assertion strengthened
- engines/vllm: serve OOM hint points users at 'rocm diagnose --symptom'
- e2e-cucumber: conditional-remediation scenario + step defs and catalog
  contract updated

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): require a vLLM anchor before matching the OOM checker

Review feedback on #290 flagged that KEYWORDS_VLLM_OOM never required
anything vLLM-specific: 'torch.OutOfMemoryError: CUDA out of memory'
scores 45+45=90 (high confidence) under the keyword table alone, so
any ROCm PyTorch job's OOM would be misreported as a vLLM startup OOM
with rocm-serve-only remediation. gpu_memory_utilization -- the one
vLLM-specific token -- was weighted lowest and never required.

check_16_vllm_oom now requires an explicit vLLM anchor (the word
'vllm', or one of its distinctive flags: gpu[-_]memory[-_]utilization,
tensor[-_]parallel) before the keyword table is scored at all. Update
the serve OOM hint's suggested --symptom text and the matching
e2e-cucumber step to carry that anchor so the self-referential
'rocm serve' -> 'rocm diagnose' flow keeps working, and add a
regression test for the reported false positive.

Also fix a second issue from the same review: the diagnose() WSL
branch dropped sub-threshold hits from matched entirely when nothing
cleared MIN_SCORE_FOR_MATCH, which the DiagnoseReport::matched doc
says should never happen. It now keeps whatever run_all_checks
returns (empty only when no wsl-applicable checker fired at all) and
adds a regression test.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* diagnose: drop the non-existent multi-GPU sharding remedy and tighten the vLLM OOM anchor

Round-2 review (EAI-8060):
- Remove the `--tensor-parallel-size` / multi-GPU sharding remediation from
  both the fix catalog (fix-16-vllm-oom) and the diagnose summary: the flag does
  not exist and rocm-cli serves one model on a single GPU (docs/vllm.md). The
  'model does not fit' branch now points only at a smaller/quantized model.
- Drop `tensor[-_]parallel` from VLLM_ANCHOR_PATTERN: it is a Megatron/DeepSpeed
  term, so anchoring on it would misattribute those frameworks' OOMs to vLLM.
  The anchor is now just `vllm|gpu[-_]memory[-_]utilization`.
- Route the user's *actual* failing log line into the `rocm diagnose --symptom`
  hint (vllm-anchored) instead of a canned literal.
- Escape the dot in the `torch\.outofmemoryerror` keyword regex and let the
  `gpu[-_]memory[-_]utilization` keyword accept a hyphen like the anchor.
- Tests: e2e reads high_confidence_threshold from the report instead of a
  hardcoded 75, locks out `--tensor-parallel-size`, and shares a find_vllm_oom
  helper; the sub-threshold unit test uses `.expect()` instead of a vacuous
  `if let`.
- docs/vllm.md and the @requires-bare-metal doc note the `rocm diagnose
  --symptom` pointer and the WSL keyword-only exception.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* style: rustfmt the new OOM --symptom routing test assertion

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): validate --gpu against the usable-device set, not its length (EAI-8058)

When amd-smi is unavailable, `--gpu <index>` validation fell back to the *length* of usable_amd_gpu_indices(). That set holds absolute, unrenumbered ordinals after the visibility mask, so on a masked multi-GPU host (e.g. HIP_VISIBLE_DEVICES=2 on a 4-GPU box -> usable [2]) the length-1 bound rejected --gpu 2 (the sole usable device) and accepted a hidden --gpu 0.

Replace the count bound with an injectable validate_pinned_gpu_index_against that prefers the amd-smi count and otherwise checks index membership in the usable set -- the same authority the engine device gate uses. The detected/usable seam makes both branches unit-testable without hardware.

Also correct two docs/vllm.md inaccuracies (note timing; missing --engine vllm in the workaround examples).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(vllm): remove duplicate code-fence opener in OOM workaround example

A merge-conflict resolution left two consecutive ```bash fence openers before
the shared-card OOM workaround example, so the block opened and immediately
reopened on the literal text and the commands rendered outside a code block.
Drop the duplicate opener so the example renders as a single fenced block.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* serve: address OOM diagnosis review — decouple engine hint from scorer, dedup keyword spans, anchor per-line

Reviewer follow-ups on the vLLM OOM diagnosis path:

- Engine no longer hard-codes `vllm: <line>` into the printed
  `rocm diagnose --symptom`. It now consults
  rocm_core::vllm_oom_symptom_is_diagnosable and falls back to
  VLLM_OOM_CANONICAL_SYMPTOM when the user's actual line would not score,
  so the command it prints always reports a cause. The user's real line
  stays visible in the human-readable message. Adds a loop-invariant test
  that every emitted --symptom value is diagnosable.

- keyword_score now de-duplicates overlapping regex spans before taking
  the top two, so a phrase nested inside a stronger one is not
  double-counted. The VLLM_OOM table splits allocator *messages* (score
  50, clear alone) from the exception *class* name (45, sub-threshold).

- vLLM OOM anchor must appear on the same line as an OOM token, not merely
  somewhere in a pasted log, closing a misattribution gap.

- WSL out-of-scope routing is gated on has_match rather than a non-empty
  match list, so a sub-threshold signal is preserved and still routed out.

- serve --gpu <index> membership rejection now says "not available" and
  lists the usable set, matching the engine wording and the e2e contract.

- --gpu auto ranks the actual reported VRAM row indices (which can be
  non-contiguous under a visibility mask) instead of a synthetic 0..count.

- diagnosis Fix for shared fix ids (fix-15, fix-16) is cross-checked
  against the fix catalog so verify/notes/commands cannot silently diverge.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(vllm): cover serve-plan low-VRAM OOM guidance on the GPU lane (EAI-8058)

The serve plan's pre-launch low-VRAM warning and its vLLM `--gpu-memory-utilization` note had unit coverage but no end-to-end scenario, so the user-observable behavior on a real serve went unverified. The plan note is reached only when a *selected* GPU is below the free-VRAM bar, which the GPU lane's genuinely-free cards never trip.

Add an e2e-test-hooks-gated seam in gpu_vram_usage() (ROCM_E2E_FORCE_LOW_VRAM=<ordinal>) that reports a single near-full synthetic device, making the warning deterministic on the non-APU vLLM lane without touching the device the engine actually launches on. Add serve-19 (@requires-gpu @requires-engine:vllm @requires-os:linux) asserting the plan warns about low VRAM and names the memory-utilization knob, with its step definitions and a unit test for the seam.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): give the low-VRAM serve scenario its runtime precondition (EAI-8058)

serve-19 (@id:serve-vllm-low-vram-oom-guidance) is the only real-serve
scenario on the GPU lane that does not open with `a managed runtime is
active`. It runs `rocm serve ... --managed` in a fresh isolated root, so
the GPU-required device policy refuses before a plan is ever built:

    Error: device_policy: gpu_required; no active ROCm runtime is
    configured; run `rocm runtimes list` and `rocm runtimes activate
    <runtime_key>`, or pass --runtime-id/--env-id

That is the single unexpected failure in the E2E tests (GPU) lane on this
branch (the other five are declared xfail), so the scenario added to cover
the pre-launch OOM note never actually exercised it. Add the same
precondition serve-05/06/07 use; the `ROCM_E2E_FORCE_LOW_VRAM` Given stays
second so the runtime setup's own `rocm` calls do not drain command_env.

Also correct auto_selection_uses_vram_row_count_when_amd_smi_count_is_unknown,
which attributed a three-row fixture to the DRM sysfs fallback. That
fallback withholds telemetry once a second AMD card is found and so emits
at most one row; a multi-row `detected == None` state is amd-smi `metric`
answering while `list` does not. Relabel it and add the single-row case the
sysfs fallback can actually reach.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): scope keyword overlap de-duplication to the vLLM OOM table

The overlapping-span de-duplication added for `KEYWORDS_VLLM_OOM` was
applied by `keyword_score` to all 16 keyword tables. Several older tables
pair a greedy `.*` pattern with a second, genuinely independent keyword,
and the greedy match span swallows it: `api-ms-win-crt-.*\.dll` runs to
the last `.dll` on the line and covers an independent `msvcp140.dll`,
`dkms .*failed` covers an independent `dpkg: error`, `amdhip64.*not
found` covers `could not find hip`, and `404.*repo\.radeon\.com` covers
`unable to locate package rocm`.

Collapsing can only lower a score, so those pairs lost their second
signal and dropped below MIN_SCORE_FOR_MATCH or HIGH_CONFIDENCE. On a
Windows host with no PATH probe result, the symptom
"api-ms-win-crt-runtime-l1-1-0.dll missing and msvcp140.dll not found"
scored 35 instead of 65 and fix-15-msvc-redist disappeared from the
report entirely.

Keep the default "every matching entry is an independent signal" scoring
for the catalog and route only `KEYWORDS_VLLM_OOM` — whose overlaps are
true nestings of literal phrases rather than wildcard artifacts —
through the collapsing variant. Pin the four pre-change scores plus the
user-visible fix-15 outcome so the dedup cannot be re-generalized.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): score only the vLLM-anchored lines, not the whole symptom

The same-line anchor was a boolean gate: `vllm_oom_line_is_anchored`
returned true if any one line carried the anchor together with an OOM
token, and `check_16_vllm_oom` then scored the *entire* symptom. Regex
matching ignores line boundaries, so keyword hits on unanchored lines
still counted at full weight, which is exactly the misattribution the
anchor exists to prevent.

A paste of

    vllm config: gpu_memory_utilization=0.9
    llama.cpp: torch.OutOfMemoryError: CUDA out of memory

passed the gate on the benign config echo, then scored `cuda out of
memory` (50) plus `torch.outofmemoryerror` (45) off the llama.cpp line —
95, above HIGH_CONFIDENCE — and reported another framework's failure as a
confident "vLLM ran the GPU out of memory".

Replace the gate with `vllm_anchored_lines`, which returns the anchored
lines joined by newlines, and score that. The joins are by `\n` and the
table's patterns are literals or `.*` (which does not cross a newline),
so dropping unanchored lines cannot fabricate a match across the seam.
The same-line requirement now falls out of the scoring itself: an
anchored line with no OOM token contributes nothing.

`gpu[-_]memory[-_]utilization` stays an anchor despite also being a
20-point scoring entry: with only anchored lines scored, a bare flag echo
(a config dump, or rocm-cli's own low-VRAM hint) is worth 20 — far below
MIN_SCORE_FOR_MATCH — so it can surface as a weak signal but can never
carry a verdict, nor lend its anchor to an OOM elsewhere in the paste.

Scoring is otherwise unchanged: only `check_16_vllm_oom` is touched, so
no other catalog table moves.

Refs: EAI-8058
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(vllm): align the OOM reservation example with the fix recipe

The catalog recipe and the --gpu-memory-utilization parse error both use 0.5;
the docs said 0.1 for the same shared-GPU case. A 0.1 budget is below the
weights of most models people serve, so it trades one startup failure for
another.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): make the low-VRAM serve scenario assert the new guidance (EAI-8058)

The scenario's warning step matched "serving may fail on VRAM", which predates
this change, and its sibling matched only the bare flag name --- which a recipe
echo of a user-supplied --gpu-memory-utilization would also satisfy. Neither
fragment is specific to the pre-launch hint this scenario exists to prove.

Require the warning to name the pinned GPU and the note to carry vLLM's
total-VRAM reservation rationale, and pin both literals in the always-run unit
test so a rewording fails on every lane rather than only on the GPU lane that
can observe the scenario.

Also record on serve-16 why the masked, non-contiguous usable-set case behind
the --gpu membership rule has no scenario of its own: it is reachable only where
amd-smi is absent, so a scenario would assert against an injected seam rather
than the product. It stays unit-covered.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(diagnose): retire the WSL out-of-scope routing assertions

The WSL2 catalog (#340) makes WSL a covered platform, so `diagnose` no
longer emits an out-of-scope routing note there. Two tests written against
the old behaviour were asserting that note, and one of them also assumed a
default WSL examination produces no match -- it now legitimately does, since
a host with no /dev/dxg really is a `fix-wsl-1-gpu-not-exposed` finding.

Both now build on `wsl_base()`, the healthy-WSL fixture, so the vLLM OOM
signal is the only thing under test. The load-bearing half is unchanged and
still asserted: a score-25 `out of memory` mention stays visible in
`matched` as a WEAK row and does not promote itself into a match.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(fix): let fix-16-vllm-oom run on WSL2, where its checker answers

`check_16_vllm_oom` is registered for `["linux", "wsl"]` -- deliberately, per
the comment at diagnose.rs -- but the matching recipe was `LINUX_ONLY`.
`rocm fix` re-gates a fix-id on the *recipe's* platform list against the
running OS, where WSL2 is its own family, so on WSL2 `rocm diagnose` printed
`apply with: rocm fix fix-16-vllm-oom` and that command then printed the plan
and exited 3 with "This fix only applies on: linux. Running OS is: wsl."

Nothing about the remediation is bare-metal: it is vLLM CLI flags. WSL2 is the
supported way to serve from a Windows host and no `fix-wsl-*` entry covers the
failure mode, so the platform had a diagnosis and no working fix path.

The class, not just the instance: the two platform lists are written in
separate modules and each looks right alone. Add
`every_checker_platform_is_covered_by_its_recipe`, which walks `CHECKERS` and
requires every family a checker answers on to be one the recipe will act on.
Containment rather than equality -- a recipe may legitimately apply more widely
than its checker fires, since `rocm fix <id>` is reachable without a diagnosis.
It found no other mismatch across the remaining 23 entries.

The loop is user-observable, so diagnose-20 gains a step that runs the fix-id
the report named and requires the success rc (the wrong-OS refusal prints the
recipe first, so output alone cannot tell the outcomes apart). It is
`@requires-os:linux`, which WSL2 matches, so the `e2e-wsl` self-hosted lane is
the one that can observe the difference; every Linux lane runs the step.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): the shared OOM hint recommended 0.1 while every command said 0.5

`VLLM_GPU_MEMORY_UTILIZATION_HINT` still carried "(e.g. 0.1 for a small
model)". It is printed verbatim by `rocm serve`'s low-VRAM note and the vLLM
engine's post-failure OOM hint, and interpolated into the `fix-16-vllm-oom`
diagnosis summary -- so one `rocm diagnose` printed "e.g. 0.1 for a small
model" directly above `rocm serve <model> --gpu-memory-utilization 0.5`.

0.5 is the settled value; every command surface already used it (the recipe,
the diagnosis Fix, docs/vllm.md). 2c3a84e moved the docs and left the const
behind, and that commit's own message says why 0.1 is wrong: a 0.1 budget is
below the weights of most models people serve, so it trades one startup failure
for another. The harmful advice was what the product actually printed.

Pin the two together with `the_utilization_hint_example_matches_the_recipe_command`,
which compares the hint's worked value against the value the recipe commands.
Neither existing guard covers this: the diagnose/fix cross-check exempts the
summary as prose framing, and a line-based grep misses the const because the
flag and the value sit on different continuation lines -- which is exactly how
it was missed the first time.

Also correct the const's own doc claim, which named two sharing surfaces when
there are three, and record that the catalog rationale is deliberately separate
prose whose worked value is nonetheless pinned to this one.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): drop the duplicate LINUX_AND_WSL const from the merge

Both branches added an identical const — this branch for the fix-16 WSL gate,
main for #386 — at different offsets, so git merged both copies without a
conflict marker and the lib test failed to compile (E0428).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): keep untrusted log text out of the suggested diagnose command

On a failed vLLM serve the post-failure hint printed the user's own failing
log line inside a single-quoted `rocm diagnose --symptom '...'` command it
invites them to paste into a shell. The line is the vLLM subprocess's own
startup-log tail and nothing escaped it, so a routine apostrophe in Python
error text ("can't allocate", "model 'foo'") closed the quote: at best the
command the feature exists to provide was syntactically broken, at worst text
the user does not control landed outside the quotes. The neighbouring
human-readable echo passed the same bytes through verbatim, ANSI and all.

A line that cannot be rendered as one intact single-quoted word is now treated
as unusable and takes the fallback to VLLM_OOM_CANONICAL_SYMPTOM that already
existed for exactly that case, rather than being escaped: escaping preserves
the bytes but yields a command a reader cannot check by eye, and a wrong escape
is runnable and misleading instead of obviously broken. The echoed line is
stripped of ANSI sequences and control bytes.

Also restores the `out_of_scope` doc comment to what `diagnose` does. It had
been changed to claim `matched` may hold sub-threshold entries alongside it and
that the field is set when nothing clears MIN_SCORE_FOR_MATCH; neither holds —
`matched` is cleared whenever the field is set, and the field is decided purely
by platform coverage. The WSL2 example can no longer occur now that WSL2 is a
covered platform. That is the documented JSON contract, so a consumer trusting
it waits for rows that can never arrive. The relationship is now asserted.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(vllm): pin the exact rendering instead of fragment absence

The control-byte test carried the message "the ANSI sequence must be
removed whole, not just its escape byte" but only asserted that "[31m"
and "[0m" were absent. A stripper that drops the escape byte and the "["
it introduces and nothing further leaves "31m"/"0m" behind, which
satisfies both absence checks and carries no control byte, so the test
passed under exactly the defect it named. Verified by running that
mutant: green before, failing now.

Assert the stripper's output and the full rendered hint by equality
instead. The neighbouring quote-rejection test had the same shape -- "no
apostrophe in the symptom" also holds if the apostrophes were silently
deleted, the escaping-style rewrite the canonical fallback exists to
avoid -- so pin the fallback symptom by equality too. That mutant was
also green before and fails now.

Tests only; no behaviour change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(serve): pin the all-busy auto-GPU fallback to a reported ordinal

`select_auto_gpu_index` now sources its candidates from the actual VRAM
rows, because a visibility mask can make reported ordinals
non-contiguous. That removed the invariant the terminal
all-candidates-busy fallback quietly relied on: with candidates drawn
from a dense `0..count`, ordinal 0 was always a real device, so
returning a literal `0` was safe by construction. With rows reported at
[2, 3] it is not — the busy filter empties the candidate list, every
pass falls through, and a fabricated 0 goes straight into
`HIP_VISIBLE_DEVICES`, pinning a device nothing ever reported. Under the
strict GPU-required / no-silent-fallback policy that is a silently wrong
pin, not a fail-fast.

The merge resolution already routes the fallback through the reported,
mask-filtered set, so the behaviour is correct. What was missing is
coverage: `auto_selection_ranks_the_actual_reported_row_indices` claims
pass 3 "never returns a fabricated index 0", but it only exercises the
*partially* busy case (busy = [3]), which never reaches the fallback at
all. This adds the all-busy case that does.

Falsified: with the fallback reverted to `vec![0]` this test fails with
left: [0], right: [2] (and takes main's
`auto_selection_all_visible_busy_falls_back_to_lowest_visible_not_zero`
down with it); both pass with the fallback restored. Equality assertion
rather than fragment absence, so a future regression cannot pass it
vacuously.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(e2e): drop a serve-16 note the merge left dangling

The comment on `serve-16` described `--gpu` validation as falling back to
membership in the KFD/DRM usable set and pointed at
`validate_pinned_gpu_index_falls_back_to_usable_set_membership` as the
unit coverage for the masked sub-case. Merging #267 replaced that
validator with one that consults the visibility-resolved set directly,
so the named test no longer exists, and the masked sub-case it called
uncovered now has two scenarios of its own (serve-19, serve-20).

Nothing here changes behaviour or which lane runs what; the paragraph is
rewritten to say what serve-16 still covers and to point at the
scenarios that own the rest. The conflict was textual-merge-clean — this
block came wholly from one side — so only reading it caught the dangling
reference.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): treat a bare CR as a line boundary in the vLLM anchor filter

`vllm_anchored_lines` split with `str::lines()`, which splits only on `\n`
and `\r\n` -- a lone `\r` is ordinary text to it. Every progress bar in this
ecosystem repaints with a bare CR (tqdm, pip, huggingface), and pasting a
captured terminal scrollback is the expected way to use
`rocm diagnose --symptom '...'`, so a real capture collapsed into one giant
"line". The anchor then matched anywhere in it and every keyword in the paste
scored at full weight: a llama.cpp OOM in a capture that also mentions vLLM
came back at 95, well above HIGH_CONFIDENCE, as "vLLM ran the GPU out of
memory at startup".

That is the whole-paste scoring `only_the_anchored_lines_are_scored_not_the_
whole_paste` exists to prevent, reinstated by one byte. Neither existing guard
saw it: both use `\n`-only fixtures.

Split on `\r` as well, so the line boundary is the one the terminal actually
renders, and add a regression case built from a bare-CR capture. It also pins
the `\n` form of the same content and an anchored-across-CRs control, so the
case stays a boundary question rather than a scoring one.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): parse the escape grammar and reject format characters

The OOM hint quotes a line of untrusted subprocess output into a
`rocm diagnose --symptom '...'` command and echoes it in a `bail!`-ed
message, so the sanitizer in front of it is load-bearing. Three holes:

The escape scan ran to the next byte in `0x40..=0x7E` with no notion of the
CSI grammar's parameter, intermediate and final classes. On well-formed
colour codes that is indistinguishable from correct; on the malformed and
truncated sequences a *killed* process emits -- which is the only case this
code path exists for -- every one of them corrupted the message. A truncated
CSI swallowed the next sequence's introducer (`ESC [ 1 ; 2 ESC [ 0 m Killed`
rendered as `0mKilled`); a multi-byte scalar can never be in that ASCII
range, so the scan ran past it and ate to the next byte that happened to land
in it (`ESC [ 1 2 é out of memory` rendered as `ut of memory`); an OSC title
leaked its body as text; a stray `ESC ESC` left `[0m` printed literally.
Follow the grammar, and on an invalid byte stop *without consuming it*, so
the damage is bounded to the malformed sequence and a following well-formed
one is still recognised.

`char::is_control` is Unicode category `Cc` only, so `Cf` format characters
survived both the stripper and `quotable_in_single_quotes`. U+202E in a log
line cannot escape the single quotes, but it does reverse how the printed
command renders -- the same "untrusted output must not control what the
terminal shows" concern, so it gets the same answer.

The stripper's doc comment promised control bytes could not repaint the
terminal "from inside rocm-cli's own error message", while the log tail a few
lines above it in the same message went out raw. Sanitize the tail too, per
line so it keeps its shape.

Also states why `.rev()` is deliberate (the last OOM line is the proximate
failure) and what it costs, since nothing said so; behaviour is unchanged.

Tests: the existing control-byte test only exercised an `'m'`-terminated CSI,
so narrowing the final-byte range to `'m'` left it green. The new case table
pins the range, the byte classes and every introducer. The diagnosability
test could not fail for the feature it is named for either -- a mutant that
always returns VLLM_OOM_CANONICAL_SYMPTOM satisfies it, because that symptom
is diagnosable by construction -- so it now pins the exact symptom per line,
over a table that deliberately straddles the routing and fallback branches.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(diagnose): pin the fix-16 plan to its catalog copy

The command block is maintained twice -- once on the `Fix` that `diagnose`
attaches, once on the `rocm fix` catalog recipe -- and the two had already
drifted. The cross-check that runs over every matched finding could not see
it: `recipe_verify_notes_and_run_commands` filters `#` lines by construction,
and only the comments differed.

That prose is the point of this entry. Two different faults produce the same
OOM and need opposite responses, so the comments are what tell the user which
of the two each step addresses -- a user who meets the `diagnose` copy and a
user who meets the `rocm fix` copy must be told the same thing. Align the
`diagnose` copy to the catalog's Case 1/Case 2 wording and wire fix-16 into
`assert_plan_matches_the_catalog_copy`, the byte-identical guard that already
existed for fix-17-torch-dlpack.

Also restores the general rule to `LINUX_AND_WSL`'s doc. Deduplicating the
const in 1e23717d kept the fix-16-specific comment and dropped main's general
one, leaving a shared const documented as if fix-16 were its only user;
fix-19-shm-too-small uses it too.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* refactor(serve): drop a vacuous guard and the duplicated AMD-device test

Three leftovers from the auto-selection rewrite and the merge around it:

`select_auto_gpu_index`'s pass-2 `max_by` still carried
`&& usage_for(index).is_some()`. Inside `if vram.is_some()` every candidate
came from a reported VRAM row, so that is vacuously true -- it implies a
failure mode (a selected ordinal telemetry never reported) that only existed
when candidates were a synthetic `0..count` range. Replace it with a comment
saying why no guard is needed.

`drm_device_is_amd` re-implemented the private `rocm_core::is_amdgpu_device`,
with a doc comment asserting the two were "the same two-signal test" and that
divergence would let the multi-card ordinal guard slip -- a promise nothing
enforced. They must agree because both count AMD cards and this one feeds
that guard, so make the rocm-core function public and delegate. One copy is
better than two copies plus a cross-check.

The second assertion in `auto_selection_all_reported_busy_falls_back_to_a_
reported_ordinal` varied the order of the `busy` slice, which is only ever
read through `.contains()` -- so it was a duplicate of the first while its
comment claimed an order-independence property nobody was testing. Vary the
order of the VRAM *rows* instead: `amd-smi` orders by enumeration, not by
index, and that is what `reported.sort_unstable()` is there for. Dropping the
sort now turns it red with `left: [3], right: [2]`; it could not before.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): split the vLLM anchor filter on rendered lines, not two bytes

The previous round fixed the whole-paste collapse for a bare `\r` and left it
open for every other line advance. `.split(['\n', '\r'])` is an allowlist of two
characters under a doc comment claiming the result was "the line boundary the
one the terminal actually renders", and it was not: with the `vllm` anchor and a
llama.cpp OOM on genuinely separate rendered lines, `ESC E` (NEL), `ESC D`
(IND), `CSI 1 B` (CUD), `\x0b`, `\x0c`, `\x85`, U+2028/U+2029 and any stray
control byte each scored 95 against a `high_confidence_threshold` of 75 --
citing the llama.cpp line's tokens as the evidence for a vLLM verdict. `ESC E`,
`ESC D` and `CSI n B` are terminfo's `nel`/`ind`/`cud1`, i.e. ordinary output
from curses- and `rich`-style progress UIs, and `rocm serve`'s own failure path
invites exactly this paste.

The one-line `char::is_control()` split that suggests itself does close every
one of those, but it trades the false positive for a false negative on the most
ordinary input there is: a colourised logger emits SGR *inside* a line, so
splitting at `ESC` cuts `ESC[31m vllm: ESC[1m torch.OutOfMemoryError` between
its anchor and its error token. Measured on that candidate, that line went from
95 to 0.

Getting both halves right needs the escape grammar, and this crate's dependents
already have one -- `strip_terminal_control_sequences`, which took three rounds
to get right in the vLLM engine. Rather than write a second parser, move the
ECMA-48 walk down into `rocm-core::terminal` as a classifier over three tokens
(`Text`, `LineBreak`, `Ignorable`) and give it two consumers: the engine's
stripper, whose behaviour is unchanged, and a new `rendered_lines` the anchor
filter splits on.

The classification is deliberately lopsided rather than terminal-accurate, and
the doc says so instead of claiming fidelity it does not deliver. Only SGR and
the string-argument sequences (OSC/DCS/SOS/PM/APC) are `Ignorable`, because
neither can move the cursor; `\t` stays intra-line text; everything else is a
boundary. That is a one-sided guarantee -- two things drawn on different rows
are never scored as one line, while one row may come back split -- and it is the
right asymmetry here: merging rows produces a confident wrong cause, splitting
one loses a diagnosis the canonical-symptom fallback already covers. A real
terminal emulator is not decidable on a byte stream with no screen model
(`CSI H` addresses a row that depends on everything before it), and
over-splitting needs none of that.

After: every separator above scores 0, `\n` and `\r` stay 0, and both the plain
and the colourised genuine vLLM OOM stay at 95.

Tests: the CR regression becomes a table over the separators rather than one
test per byte, plus the SGR case in the other direction, plus a property test
over the whole C0/C1 range pinned to `char::is_control` rather than to this
module's own classifier. Reverting the boundary set turns three of them red
(`nel esc-e`, `U+0000`, `a ESC E (NEL)-separated llama.cpp OOM ...`); making SGR
a boundary instead turns the colour test red on `SGR inside a line must not
change the verdict`.

A confidently wrong cause is user-observable, so it is also pinned as
`@id:diagnose-vllm-oom-not-attributed-across-rendered-lines` (diagnose-21),
which fails on the pre-fix boundary set and passes after.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* chore(core): add the licence header the new terminal module was missing

`fcf99156` created `crates/rocm-core/src/terminal.rs` without the SPDX header
every other source file carries, so `License header check (hawkeye)` failed —
its `licenserc.toml` includes `**/*.rs`. Header only; no code change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): scope the consumption rule and name the one stripper change

Two prose defects in the module the previous commit added, both of which a
reviewer would rely on.

`next_token`'s doc asserted that "the contract here is 'render what the terminal
would have rendered', which is the only rule that stays decidable on a byte
stream with no framing". That was true of the stripper the paragraph came from,
where the only question was how much of a well-formed sequence to consume. It is
false of `next_token`, which also decides line boundaries and deliberately
over-splits -- twenty-eight lines further down the same comment says a terminal
emulator "is not decidable on a byte stream with no screen model", and the
section between explains the classification is lopsided rather than
terminal-accurate. Left standing, that sentence is a documented licence to
"fix" the over-splitting toward terminal accuracy, which reopens the row-merging
misattribution the previous commit closed. It is now scoped to the consumption
decision it is actually about: a well-formed sequence is consumed whole because
a real terminal consumes it too, and which token it is then labelled with is the
separate, deliberately non-terminal-like question below.

The same commit said the engine stripper's "behaviour is unchanged" by the move.
It is unchanged except for U+2028/U+2029. The engine-local stripper classified
non-escape characters with `is_control_or_format` alone, which is false for both
(neither is `Cc`, neither is in the enumerated `Cf` set), so both survived into
the stripped message; `classify_char` puts them in the boundary set, so they are
now dropped. Measured on this head: the old code returns "a\u{2028}b" for input
"a\u{2028}b", the new one returns "ab". The widening is the better behaviour --
a mandatory Unicode line break is exactly what this stripper exists to keep
untrusted output from injecting -- but nothing stated it and no test pinned
either scalar, so a silent drift back would have gone unnoticed. `classify_char`
now says so, and the vLLM stripper table gains both cases: reverting the pair
out of the boundary set turns `the_stripper_follows_the_escape_grammar_not_just_
the_colour_case` red on "a line separator must not survive into the message".

Also from the same review, three prose corrections with no behaviour change:
`rendered_lines`'s lead sentence now states the over-approximation up front
instead of stating the strong property and retracting it a paragraph later; the
string-argument arm documents the one crafted-input hole in the never-merge
guarantee (a properly terminated OSC body containing a line feed is swallowed
whole, so `"vllm ESC ] 0 ; ti \n tle BEL llama.cpp OOM"` comes back as one line,
where common terminals abort the control string and draw two rows); and the
`fix-16` catalog test's comment cites the cross-check that actually exists,
`diagnosis_remediation_matches_the_fix_catalog_for_shared_fix_ids`, rather than
a name that is in neither the tree nor its history.

Last, `effective_gpu_count`'s doc claimed the VRAM-row fallback applies when the
amd-smi count is absent, while the code also falls through when it is present
and zero -- a host amd-smi cannot enumerate but sysfs can. The doc now names
both triggers and the second one gains the test case it never had; dropping the
`count > 0` filter on `detected` turns it red.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): qualify the never-merge guarantee where a consumer reads it

`rendered_lines` promised, unqualified, that "two pieces of text the
terminal drew on different rows never end up in the same returned
segment". That is the invariant the module exists for -- it is what stops
a `llama.cpp` OOM being scored against a `vllm` anchor -- and it is not
what the code holds. The counterexample was already written down in the
same commit, 140 lines above, but in a body comment inside the private
`next_token`, so it reached no reader of the generated documentation, and
the public doc's pointer to it (`See [next_token]`) was a link to a
private item that rustdoc will not resolve. `cargo doc -p rocm-core`
warned on exactly that line.

This is the third finding of the same class on this module, so the fix is
structural rather than a reworded sentence. The exception now has one
authoritative statement, in the module documentation, which is public and
therefore rendered: a line break inside a well-formed string-argument
body (`OSC`, `DCS`, `SOS`, `PM`, `APC`) is consumed with that body, with
the worked example spelled out. `rendered_lines`'s own doc carries the
qualifier inline -- so the only absolute a consumer can read is true --
and links to the module docs rather than to a private item. The
`next_token` arm that creates the exception no longer restates it; it
points at the module paragraph and names what has to change with it.

The behaviour is unchanged. Following the grammar rather than guessing
where an unterminated body ends is the right call: the alternative
reintroduces the leak of OSC bodies as text.

`a_line_break_inside_a_well_formed_string_body_is_consumed_with_it` pins
the worked example so prose and behaviour move together; flipping its
expectation to the two-segment reading fails with
`left: ["vllmllama.cpp OOM"]`. It also pins that the same break outside a
string body still splits, and that an unterminated body still stops at
the following `ESC`, so the hole cannot widen unnoticed.

Also, both non-blocking wording items and one more instance of the same
dangling-link defect the PR introduced:

- `the_split_never_merges_two_rendered_lines` overstated what the module
  guarantees; its body already scoped itself to single non-drawing
  scalars, so it is renamed to
  `a_lone_non_drawing_scalar_never_merges_two_rendered_lines`.
- the citation in `the_oom_plan_matches_the_catalog_copy` called
  `diagnosis_remediation_matches_the_fix_catalog_for_shared_fix_ids` "the
  cross-check that runs over every matched finding"; it iterates a fixed
  two-element array of hand-built diagnoses, so it is now "the cross-check
  over the fixes shared with the catalog".
- `DiagnoseReport::out_of_scope`'s doc linked `[catalog_covers]`, also a
  private item, also warned on by `cargo doc`, and also added by this
  branch. It now names the rule in prose instead.

`cargo doc -p rocm-core --no-deps` goes from 6 warnings to 3; the
remaining three are pre-existing on `main` and are not touched here.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): drop the terminator from the merge exception; it is not required

`rendered_lines`'s doc said two rendered rows share a segment "only when the
break between them sits inside a well-formed string-argument sequence ...
whose body ECMA-48 defines as running to its terminator", and the module
section said the shape "needs a well-formed introducer and terminator around
the line break". Both make a terminator necessary for the merge. It is not.

`skip_string_sequence_body` ends the body scan on `BEL`, on `ST`, on any other
bare `ESC` (left in place), or on end of input, and `next_token` returns
`Token::Ignorable` for all of them. The break is swallowed either way. The
test added in this PR is itself the counterexample and passes: its input
carries no `BEL` and no `ST` anywhere, and the two rows still merge. The
private helper's own doc comment already said this, so the public doc was
contradicting the private one.

State what the code does instead, and add the unterminated input as a second
worked example so the prose has to move with it.

That also invalidates the reassurance built on top of the terminator being
required. An unterminated body followed by more output is reachable from
ordinary output by two routes: a process killed part-way through writing an
`OSC` title, which is the case this walk exists for, and the interleaved
capture itself, since the serve log hands the subprocess's stdout and stderr
the same file handle and either stream can cut the other's title. Say so, and
say why following the grammar is still right -- guessing where an
unterminated body ends reintroduces the leak of `OSC` bodies as text.

`vllm_anchored_lines` still carried the original unqualified "two things drawn
on different rows are never scored as one line", which is the sentence the
module section claims to be the single authoritative statement of. Replace the
restatement with a pointer rather than a second qualified copy; a second copy
is a stale copy waiting to happen, and this is the call site where a stale one
does the damage.

Docs, comments and one test rename only; no behaviour change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): delete the restated reachability claim, and name the third behaviour

`rendered_lines`'s own doc claimed a process killed mid-title was enough to
reach the merge shape. It is not. The merge needs drawable text on both sides
of the swallowed break landing in one segment, and a kill leaves the body
running to the end of the input, where there is no far side to join to:

    rendered_lines("vllm\u{1b}]0;ti\ntle") == ["vllm"]

The `tle` a terminal drew on its second row comes back in no segment at all.
That is a third behaviour -- neither the split nor the merge the one-sided
framing describes -- and it was undocumented, so the prose reached for it as an
example of the merge instead.

This is the fifth round on one sentence, and the sentence is always in the same
place: `rendered_lines` restating the exception rather than pointing at the
module section that owns it. So the restatement goes rather than gets reworded.
The qualifier that binds callers stays; the terminator detail and the
reachability claim are deleted and the `[the module documentation](self)`
pointer now carries the routes, which makes the "single authoritative statement"
claim true rather than aspirational.

The drop is documented rather than filed as a defect: it is the direct,
already-argued consequence of not guessing where an unterminated body ends, and
it errs in the losing direction the contract leaves unpromised -- it costs a
diagnosis, it cannot invent one. Both one-sided framings are widened to admit
it, so neither now claims two directions while a third exists.

Also corrected: the interleaved capture reaches `rendered_lines` through a user
paste into `rocm diagnose --symptom`, not through the engine's own path, which
pre-splits its log tail with `str::lines`; and the contract is not "stated
once" in `vllm_anchored_lines`'s doc, which names two places in the same breath.

The end-of-input case is now pinned by a test. Nothing covered it, which is why
the prose could cite it wrongly twice.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): the merge trigger is text following the sequence, not end of input

The module section named end of input as the reason a swallowed break loses a
row rather than merging. It is not the reason: a properly BEL-terminated body
with nothing after it loses the row just the same, and so does one with text
after it that sits past a break. What decides is whether drawable text follows
the sequence in the same segment, however the body ended.

Rather than restate that correctly in each of the three places the section said
it, keep one statement where the third behaviour is introduced and delete the
other two -- the divergence between copies is what produced this round and the
five before it.

Pin the two shapes that were not covered, so no future wording can name a
narrower trigger than the code has without a test failing.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): qualify the closing summary's loss condition to match the rule

The section states the trigger precisely as drawable text following the
sequence *in that same segment*, then the closing summary restated it
without the qualifier. rendered_lines("vllm\u{1b}]0;ti\ntle\u{7}\nx")
is the case that splits them: text does follow the sequence, the row is
lost anyway, because the text is past a break and so in another segment.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): retitle the residual-behaviour section and stop restating its rule

The section documents two residual behaviours, not one exception, and its
own claim about the file's structure was false: the code arm and the test
do restate the rule, accurately and usefully, as local commentary. Say
instead that this section is the normative statement.

Three further corrections, all the same recurring defect of the same rule
living in more than one place and drifting:

- a break inside a string body can lose more than one rendered row
  (`rendered_lines("vllm\u{1b}]0;ti\ntle\nmore") == ["vllm"]`), so "one or
  more";
- the "what decides" clause named only the far side, but a row is lost
  with no near side too (`rendered_lines("\u{1b}]0;ti\ntle\u{7}x") ==
  ["x"]`) — dropped in favour of the both-sides sentence before it;
- the closing summary repeated the same far-side-only qualifier, so it is
  dropped there as well.

Net two lines shorter.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): keep a detected GPU with no VRAM row as an auto-select candidate

`select_auto_gpu_index` builds its candidate ordinals from the VRAM rows
alone whenever telemetry is present. That was introduced so a visibility
mask reporting non-contiguous ordinals is scanned by its real indices
instead of a synthetic `0..count` range, which is right — but a short row
set is not always a mask. `parse_gpu_vram_usage` drops any device entry
missing `/mem_usage/used_vram/value` or `/mem_usage/total_vram/value`, so
a device the lighter `list` enumeration counts can simply have no row.

With device 1 absent from the rows and device 0 pinned by a managed
service, `reported` was `[0]`, the busy filter emptied the candidate list,
all three passes iterated nothing and the terminal fallback returned
device 0 — `serve` pinned to an already-occupied GPU, then failing on
VRAM, which is the fault this selection exists to reduce.

Union the detected `0..count` range back in only when `visible` is known.
The existing retain then validates every ordinal against the mask, so no
unconfirmed index is ever synthesised: under a `[2, 3]` mask with
`count == 2` the synthetic `[0, 1]` is dropped wholesale and the rows
still stand alone. Without `visible` there is no second source to confirm
an ordinal against, so the rows remain authoritative.

Pins the case with a test: a two-GPU count, one row, an all-busy reported
set, which the suite never exercised (every all-busy case passed
`detected: None`).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): discriminate a masked row set by its indices, not by `visible`

The previous commit unioned the detected `0..count` range back into the
auto-select candidates only `if visible.is_some()`, so the fix reached
only hosts whose visibility set could be probed.

`probe_usable_amd_gpu_indices` returns `None` unconditionally for
`#[cfg(not(target_os = "linux"))]`, and `serve` threads that straight
into `visible` through `select_gpu_indices_under_launch_lock`. So
`visible` is always `None` on Windows — a supported platform — and on
any Linux host whose KFD topology and DRM cards are both unreadable.
With `count == 2`, one VRAM row for GPU 0 and GPU 0 pinned by a managed
service, those hosts still returned the busy GPU 0 where the base branch
returned the idle GPU 1: the exact fault this selection exists to reduce.

`visible` was never the right discriminator. The rows are a re-indexed or
sparse set precisely when some row index falls at or above `count`
(`[2, 3]` against `count == 2`), which is the shape the rows-only
construction was introduced for; below the count they are a subset of a
dense range the count already asserts exists, and unioning `0..count`
there is no stronger an assumption than the telemetry-less arm already
makes. Require `detected` to be `Some(n > 0)` as well, so the count comes
from the `list` enumeration rather than from `rows.len()` restating the
rows' own size — `parse_gpu_vram_usage` only falls back to the array
position when an entry has no `gpu` field, so a payload naming `"gpu": 0`
twice yields duplicate ordinals that would otherwise conjure a GPU 1.

The visible retain still runs on top whenever a mask is known, so nothing
synthesised survives where the mask contradicts it.

Pins both halves of the condition: forcing either to `true` turns an
assertion of the new test red, and the masked assertion now holds with
`visible: None` so it pins the row-index rule itself rather than the
retain.

Also replaces the Pass 2 rationale, which the union falsified. `usage_for`
is no longer total over `candidate_indices`, but no row guard is needed
for a different reason: the comparator keys are `Option<u64>` and `None`
sorts below every `Some`, so a rowless ordinal can win only when no
candidate has a row, and then the index tie-break returns the lowest —
exactly what Pass 3 would return. Re-adding the guard leaves all 771 unit
tests green, so the guard is behaviourally inert and stays out.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): reject a mandatory line break from the quoted symptom too

`quotable_in_single_quotes` gates what is embedded in the
`rocm diagnose --symptom '...'` command the OOM hint prints for the user
to paste, and it tested each character with `is_control_or_format` alone.
That predicate is `char::is_control` plus the enumerated `Cf` set, and
`U+2028 LINE SEPARATOR` / `U+2029 PARAGRAPH SEPARATOR` are `Zl`/`Zp`, so
both passed the gate.

The value reaching it is raw: `candidate` is built from `raw_line`, not
from the stripped `symptom_line` beside it, and `str::lines` does not
split on `U+2028`, so the scalar survives inside a selected line. A tail
line like `RuntimeError: HIP out of memory<U+2028>...` cleared the OOM
scan, cleared this gate, scored above threshold and was printed verbatim
inside the quotes — the bytes stay inside them, but the rest of the
command renders on the next row. That is the same class of display
control the guard's own documentation says it closes.

`classify_char` already widened past `is_control_or_format` for exactly
this pair, so the module stated one contract and the gate implemented a
narrower one. Rather than repeat the `matches!` at the gate, export
`is_control_or_line_separator` from `rocm_core::terminal` and use it at
both sites: `is_control_or_format`'s own documentation advertises itself
as the answer to this concern, so the next call site would reach for the
narrower test again. One predicate removes the trap, and the rationale
moves onto it instead of being restated at each user.

The stripper table test pinned both scalars against
`strip_terminal_control_sequences` only; it now pins them against the
quoting guard as well, which is where they got through.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(skill): document fix-16-vllm-oom in the rocm-doctor reference

`fix-16-vllm-oom` is added to the CLI's closed catalog by this branch, but
`skills/rocm-doctor/reference.md` still described `fix-16` as "a reserved
handle, not a missing row". The skill doc is asserted against the CLI
catalog by three e2e scenarios, and all three failed on the merge:
`rocm fix` offers fix-16-vllm-oom, which reference.md does not document.

Add the catalog row (linux/wsl, not auto-applicable, matching the recipe's
LINUX_AND_WSL and auto_applicable: false), raise the catalog count from 24
to 25, and move the reserved-handle note to `fix-18`, which is still the
only gap in the numeric series.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): let the catalog fill fix-16's auto_applicable

`check_16_vllm_oom` built its `Fix` with an inline `auto_applicable: false`.
`no_checker_hand_sets_auto_applicable` forbids that: only
`take_applicability_from_the_catalog` may set the field, because the catalog
pass overwrites whatever a checker puts there. The output was right either
way -- which is the point. A checker that disagreed with the catalog would be
silently masked rather than caught, the way `fix-9-igpu-dgpu` once drifted.

The field is now left at `Fix::default()` and filled from the recipe, whose
`applies_on: PRINT_ON_LINUX_AND_WSL` already carries `FixClass::PrintOnly`.

This was latent rather than new: the guard never ran on the previous head
because the suite aborted at an unrelated flaky test first, so the offending
line reached `gpu-out-of-memory` unnoticed.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Juho Vainio <juho.vainio@amd.com>
mikeroySoft pushed a commit to mikeroySoft/rocm-cli that referenced this pull request Nov 6, 2026
…i-8059) (ROCm#284)

* feat(vllm): add GPU memory utilization hint for OOM errors and improve documentation

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): validate --gpu against sysfs GPU count and harden DRM fallback

Address review feedback on PR #251:

- resolve_gpu_indices now validates an explicit --gpu <index> against
  the DRM sysfs fallback's device count when amd-smi is unavailable,
  instead of only the amd-smi count. Previously an out-of-range index
  was silently accepted on a host with no amd-smi (sysfs works) and
  only failed later inside the engine.
- read_drm_vram_usage no longer treats an unreadable
  mem_info_vram_used counter as 0 bytes used (which made the card look
  100% free -- exactly what --gpu auto prefers first); it now skips
  that card instead.
- read_drm_vram_usage withholds telemetry entirely when more than one
  AMD DRM card is present, since ascending card<N> order is only
  guaranteed to match HIP's compute-topology ordinal on a single-GPU
  host (an APU passes the same vendor + mem_info_vram_total filter as
  a discrete GPU, so an APU+dGPU host could previously feed a
  diverged ordinal into HIP_VISIBLE_DEVICES).
- docs/vllm.md: corrected the fallback description, which only ever
  probes amd-smi (never rocm-smi) before falling back to DRM sysfs.

Extracted the count-fallback logic into a new effective_gpu_count
helper shared by --gpu auto ranking and --gpu <index> validation, and
added unit tests for all of the above.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): validate --gpu against amd-smi-independent count; harden DRM guard

Round-2 review follow-ups on the DRM sysfs GPU-selection fallback.

- Validate an explicit --gpu <index> against the amd-smi list count or, when
  amd-smi is absent, the KFD/DRM authority behind usable_amd_gpu_indices()
  (pinned_index_bound) instead of the DRM VRAM-fallback row count. The row
  count withholds telemetry on multi-GPU hosts and shrinks on a transient
  counter-read failure, so borrowing it as a validation bound hard-rejected
  legitimate indices. VRAM rows stay scoped to --gpu auto ranking.
- The multi-card ordinal-ambiguity guard now counts AMD cards *found* (before
  the telemetry filters) rather than surviving rows, so a second AMD card with
  an unreadable counter still trips the guard instead of mislabelling the
  survivor ordinal 0 (the APU+dGPU misattribution the guard exists to prevent).
- drm_device_is_amd now matches on vendor id OR an amdgpu uevent DRIVER line,
  the same two-signal test as rocm_core::is_amdgpu_device, so the fallback probe
  and the count authority agree on what an AMD card is.
- Drop the now-dead sort/enumerate tail in read_drm_vram_usage (at most one card
  survives the guard) and fix the stale doc comment.
- Tests: exercise the real Index-arm delegate (resolve_pinned_gpu_index) and add
  a two-AMD-card-with-unreadable-counter regression; add pinned_index_bound
  coverage.
- Docs: README and the assistant SKILL note the DRM sysfs VRAM fallback rather
  than presenting amd-smi as the only source.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060) (#290)

* feat(diagnose): add vLLM out-of-memory failure mode to the catalog (EAI-8060)

Add a keyword-scored OOM signature to the rocm-core diagnosis catalog and a
matching print-only remediation recipe. The error covers two distinct faults
(a tenancy collision with vLLM's fixed ~90% VRAM reservation, versus a model
that genuinely does not fit), so the wording stays conditional: it never
prescribes lowering --gpu-memory-utilization as the unconditional answer, and
the verify step avoids the tenancy knob. Because an Examination carries no
per-GPU VRAM or tenancy fields, the match is keyword-only and the checker is
gated to Linux (vLLM is Linux/WSL-only).

- diagnose.rs: KEYWORDS_VLLM_OOM table + check_16_vllm_oom (linux-only)
- fix.rs: print-only fix-16-vllm-oom recipe; catalog count 15 -> 16 and
  AUTO-set assertion strengthened
- engines/vllm: serve OOM hint points users at 'rocm diagnose --symptom'
- e2e-cucumber: conditional-remediation scenario + step defs and catalog
  contract updated

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): require a vLLM anchor before matching the OOM checker

Review feedback on #290 flagged that KEYWORDS_VLLM_OOM never required
anything vLLM-specific: 'torch.OutOfMemoryError: CUDA out of memory'
scores 45+45=90 (high confidence) under the keyword table alone, so
any ROCm PyTorch job's OOM would be misreported as a vLLM startup OOM
with rocm-serve-only remediation. gpu_memory_utilization -- the one
vLLM-specific token -- was weighted lowest and never required.

check_16_vllm_oom now requires an explicit vLLM anchor (the word
'vllm', or one of its distinctive flags: gpu[-_]memory[-_]utilization,
tensor[-_]parallel) before the keyword table is scored at all. Update
the serve OOM hint's suggested --symptom text and the matching
e2e-cucumber step to carry that anchor so the self-referential
'rocm serve' -> 'rocm diagnose' flow keeps working, and add a
regression test for the reported false positive.

Also fix a second issue from the same review: the diagnose() WSL
branch dropped sub-threshold hits from matched entirely when nothing
cleared MIN_SCORE_FOR_MATCH, which the DiagnoseReport::matched doc
says should never happen. It now keeps whatever run_all_checks
returns (empty only when no wsl-applicable checker fired at all) and
adds a regression test.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* diagnose: drop the non-existent multi-GPU sharding remedy and tighten the vLLM OOM anchor

Round-2 review (EAI-8060):
- Remove the `--tensor-parallel-size` / multi-GPU sharding remediation from
  both the fix catalog (fix-16-vllm-oom) and the diagnose summary: the flag does
  not exist and rocm-cli serves one model on a single GPU (docs/vllm.md). The
  'model does not fit' branch now points only at a smaller/quantized model.
- Drop `tensor[-_]parallel` from VLLM_ANCHOR_PATTERN: it is a Megatron/DeepSpeed
  term, so anchoring on it would misattribute those frameworks' OOMs to vLLM.
  The anchor is now just `vllm|gpu[-_]memory[-_]utilization`.
- Route the user's *actual* failing log line into the `rocm diagnose --symptom`
  hint (vllm-anchored) instead of a canned literal.
- Escape the dot in the `torch\.outofmemoryerror` keyword regex and let the
  `gpu[-_]memory[-_]utilization` keyword accept a hyphen like the anchor.
- Tests: e2e reads high_confidence_threshold from the report instead of a
  hardcoded 75, locks out `--tensor-parallel-size`, and shares a find_vllm_oom
  helper; the sub-threshold unit test uses `.expect()` instead of a vacuous
  `if let`.
- docs/vllm.md and the @requires-bare-metal doc note the `rocm diagnose
  --symptom` pointer and the WSL keyword-only exception.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* style: rustfmt the new OOM --symptom routing test assertion

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): validate --gpu against the usable-device set, not its length (EAI-8058)

When amd-smi is unavailable, `--gpu <index>` validation fell back to the *length* of usable_amd_gpu_indices(). That set holds absolute, unrenumbered ordinals after the visibility mask, so on a masked multi-GPU host (e.g. HIP_VISIBLE_DEVICES=2 on a 4-GPU box -> usable [2]) the length-1 bound rejected --gpu 2 (the sole usable device) and accepted a hidden --gpu 0.

Replace the count bound with an injectable validate_pinned_gpu_index_against that prefers the amd-smi count and otherwise checks index membership in the usable set -- the same authority the engine device gate uses. The detected/usable seam makes both branches unit-testable without hardware.

Also correct two docs/vllm.md inaccuracies (note timing; missing --engine vllm in the workaround examples).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(vllm): remove duplicate code-fence opener in OOM workaround example

A merge-conflict resolution left two consecutive ```bash fence openers before
the shared-card OOM workaround example, so the block opened and immediately
reopened on the literal text and the commands rendered outside a code block.
Drop the duplicate opener so the example renders as a single fenced block.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* serve: address OOM diagnosis review — decouple engine hint from scorer, dedup keyword spans, anchor per-line

Reviewer follow-ups on the vLLM OOM diagnosis path:

- Engine no longer hard-codes `vllm: <line>` into the printed
  `rocm diagnose --symptom`. It now consults
  rocm_core::vllm_oom_symptom_is_diagnosable and falls back to
  VLLM_OOM_CANONICAL_SYMPTOM when the user's actual line would not score,
  so the command it prints always reports a cause. The user's real line
  stays visible in the human-readable message. Adds a loop-invariant test
  that every emitted --symptom value is diagnosable.

- keyword_score now de-duplicates overlapping regex spans before taking
  the top two, so a phrase nested inside a stronger one is not
  double-counted. The VLLM_OOM table splits allocator *messages* (score
  50, clear alone) from the exception *class* name (45, sub-threshold).

- vLLM OOM anchor must appear on the same line as an OOM token, not merely
  somewhere in a pasted log, closing a misattribution gap.

- WSL out-of-scope routing is gated on has_match rather than a non-empty
  match list, so a sub-threshold signal is preserved and still routed out.

- serve --gpu <index> membership rejection now says "not available" and
  lists the usable set, matching the engine wording and the e2e contract.

- --gpu auto ranks the actual reported VRAM row indices (which can be
  non-contiguous under a visibility mask) instead of a synthetic 0..count.

- diagnosis Fix for shared fix ids (fix-15, fix-16) is cross-checked
  against the fix catalog so verify/notes/commands cannot silently diverge.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(vllm): cover serve-plan low-VRAM OOM guidance on the GPU lane (EAI-8058)

The serve plan's pre-launch low-VRAM warning and its vLLM `--gpu-memory-utilization` note had unit coverage but no end-to-end scenario, so the user-observable behavior on a real serve went unverified. The plan note is reached only when a *selected* GPU is below the free-VRAM bar, which the GPU lane's genuinely-free cards never trip.

Add an e2e-test-hooks-gated seam in gpu_vram_usage() (ROCM_E2E_FORCE_LOW_VRAM=<ordinal>) that reports a single near-full synthetic device, making the warning deterministic on the non-APU vLLM lane without touching the device the engine actually launches on. Add serve-19 (@requires-gpu @requires-engine:vllm @requires-os:linux) asserting the plan warns about low VRAM and names the memory-utilization knob, with its step definitions and a unit test for the seam.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* feat(vllm): OOM detection, serve-summary memory guidance, and diagnose routing (EAI-8059)

Squashed the EAI-8059 series into one logical change: narrow vLLM OOM
detection, gate the serve-summary memory guidance on a real launch
failure (not a reused already-running service), tie the note's log-tail
budget to the shared engine constant, and cover it with a feature-gated
e2e scenario that reuses a planted managed service so it allocates no GPU
memory.

Original commits:
- feat(vllm): enhance OOM diagnostics and guidance in serve summary
- fix(vllm): address review feedback on OOM diagnostics
- serve: narrow OOM detection and gate the note on real launch failure
- serve: detect reusable running service before GPU pre-flight
- e2e: cover positive OOM-launch memory guidance via fault injection
- e2e: gate serve-oom scenario on the fault-injection build
- serve: tie OOM note tail budget to the shared engine constant

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* serve: address EAI-8059 re-review — validate --gpu before engine work, tighten OOM negative assertion

Three review fixes on top of the squashed EAI-8059 change:

- serve flow: move the pinned --gpu index validation (resolve_gpu_indices)
  back ahead of the no-runtime bail and ensure_self_managed_engine_ready,
  so a nonexistent --gpu index is rejected before any engine preparation
  begins. This restores the 'refused before any engine starts' contract
  that serve-absent-gpu-index-rejected asserts on the GPU lane; the reorder
  had pushed index validation after engine-ready work. It still sits after
  the no-usable-GPU pre-flight so a GPU-less host refuses first.

- e2e: the negative OOM assertion used a literal contains("ran out of GPU
  memory") while the note wraps across the 80-column PTY grid, so it passed
  even if the note WERE wrongly rendered. Match whitespace-insensitively,
  exactly as the positive assertions do, so a mis-rendered note is caught.

- vllm: fix a stale reference to the pre-rename log_tail_shows_oom in the
  emitted-symptom test; the detector now lives in rocm-core as
  vllm_log_shows_oom.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): reconcile OOM detector with diagnose threshold and de-dup env const (eai-8059)

Address PR review on the OOM serve-summary work.

rocm-core: classify a vLLM startup-log tail as OOM with the same rule as
the `rocm diagnose` vLLM-OOM checker (anchored line clearing
MIN_SCORE_FOR_MATCH) instead of a second, looser substring scan. A bare
"out of memory" (kernel OOM-killer, a dependency's log) or a bare
`torch.OutOfMemoryError` class with no allocator message no longer
drives the memory note. Add `vllm_oom_diagnose_symptom` so the engine
hint and the CLI serve summary select the `--symptom` line one way.
Correct the utilization-hint bound (the parser rejects 0, so `<0-1>`
was wrong).

serve summary: the OOM note now carries the model-too-large caveat
(lowering the reservation will not help; serve a smaller/quantized
model) and routes to `rocm diagnose --symptom`, matching the diagnose
entry's split remediation.

e2e: hoist OOM_FAULT_INJECTION_ENV into the shared e2e-report crate so
xtask (producer) and the e2e-cucumber harness (consumer) reference one
literal and cannot drift.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): give the low-VRAM serve scenario its runtime precondition (EAI-8058)

serve-19 (@id:serve-vllm-low-vram-oom-guidance) is the only real-serve
scenario on the GPU lane that does not open with `a managed runtime is
active`. It runs `rocm serve ... --managed` in a fresh isolated root, so
the GPU-required device policy refuses before a plan is ever built:

    Error: device_policy: gpu_required; no active ROCm runtime is
    configured; run `rocm runtimes list` and `rocm runtimes activate
    <runtime_key>`, or pass --runtime-id/--env-id

That is the single unexpected failure in the E2E tests (GPU) lane on this
branch (the other five are declared xfail), so the scenario added to cover
the pre-launch OOM note never actually exercised it. Add the same
precondition serve-05/06/07 use; the `ROCM_E2E_FORCE_LOW_VRAM` Given stays
second so the runtime setup's own `rocm` calls do not drain command_env.

Also correct auto_selection_uses_vram_row_count_when_amd_smi_count_is_unknown,
which attributed a three-row fixture to the DRM sysfs fallback. That
fallback withholds telemetry once a second AMD card is found and so emits
at most one row; a multi-row `detected == None` state is amd-smi `metric`
answering while `list` does not. Relabel it and add the single-row case the
sysfs fallback can actually reach.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* serve: make the GPU fail-fast contract true and share the vLLM tail budget (eai-8059)

Review follow-up on three points that the earlier response commits did not
actually close in the code:

- The reuse pre-gate keyed on the engine alone, so a live managed service for an
  unrelated model pulled the reuse probe's `ResolveModel` round-trip — and, for a
  self-managing engine, an `ensure_self_managed_engine_ready` install that prints
  "Preparing <engine> for GPU serving..." — ahead of the no-usable-GPU bail. The
  comment above the bail claimed that work could not precede it. Gate the probe on
  the model as well (`any_live_managed_service_for_model`), matching with the same
  lenient relation the service surfaces already use so short-vs-canonical
  spellings still reach the probe, and state the real contract in the comment.

- `engines/vllm`'s `STARTUP_FAILURE_LOG_TAIL_LINES` was an independent `80`
  literal while the serve summary's comment claimed the two surfaces were coupled
  to `rocm_engine_protocol::DEFAULT_LOG_TAIL_LINES`. Derive it from that constant
  so the claim holds.

- `assert_prebuilt_e2e_lanes_enable_test_hooks` said it asserted the exact feature
  list `cargo xtask e2e` builds; xtask now builds a superset. Describe it as the
  shared hook floor and note why the fault-injection hook is deliberately absent
  from prebuilt lanes.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): scope keyword overlap de-duplication to the vLLM OOM table

The overlapping-span de-duplication added for `KEYWORDS_VLLM_OOM` was
applied by `keyword_score` to all 16 keyword tables. Several older tables
pair a greedy `.*` pattern with a second, genuinely independent keyword,
and the greedy match span swallows it: `api-ms-win-crt-.*\.dll` runs to
the last `.dll` on the line and covers an independent `msvcp140.dll`,
`dkms .*failed` covers an independent `dpkg: error`, `amdhip64.*not
found` covers `could not find hip`, and `404.*repo\.radeon\.com` covers
`unable to locate package rocm`.

Collapsing can only lower a score, so those pairs lost their second
signal and dropped below MIN_SCORE_FOR_MATCH or HIGH_CONFIDENCE. On a
Windows host with no PATH probe result, the symptom
"api-ms-win-crt-runtime-l1-1-0.dll missing and msvcp140.dll not found"
scored 35 instead of 65 and fix-15-msvc-redist disappeared from the
report entirely.

Keep the default "every matching entry is an independent signal" scoring
for the catalog and route only `KEYWORDS_VLLM_OOM` — whose overlaps are
true nestings of literal phrases rather than wildcard artifacts —
through the collapsing variant. Pin the four pre-change scores plus the
user-visible fix-15 outcome so the dedup cannot be re-generalized.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): score only the vLLM-anchored lines, not the whole symptom

The same-line anchor was a boolean gate: `vllm_oom_line_is_anchored`
returned true if any one line carried the anchor together with an OOM
token, and `check_16_vllm_oom` then scored the *entire* symptom. Regex
matching ignores line boundaries, so keyword hits on unanchored lines
still counted at full weight, which is exactly the misattribution the
anchor exists to prevent.

A paste of

    vllm config: gpu_memory_utilization=0.9
    llama.cpp: torch.OutOfMemoryError: CUDA out of memory

passed the gate on the benign config echo, then scored `cuda out of
memory` (50) plus `torch.outofmemoryerror` (45) off the llama.cpp line —
95, above HIGH_CONFIDENCE — and reported another framework's failure as a
confident "vLLM ran the GPU out of memory".

Replace the gate with `vllm_anchored_lines`, which returns the anchored
lines joined by newlines, and score that. The joins are by `\n` and the
table's patterns are literals or `.*` (which does not cross a newline),
so dropping unanchored lines cannot fabricate a match across the seam.
The same-line requirement now falls out of the scoring itself: an
anchored line with no OOM token contributes nothing.

`gpu[-_]memory[-_]utilization` stays an anchor despite also being a
20-point scoring entry: with only anchored lines scored, a bare flag echo
(a config dump, or rocm-cli's own low-VRAM hint) is worth 20 — far below
MIN_SCORE_FOR_MATCH — so it can surface as a weak signal but can never
carry a verdict, nor lend its anchor to an OOM elsewhere in the paste.

Scoring is otherwise unchanged: only `check_16_vllm_oom` is touched, so
no other catalog table moves.

Refs: EAI-8058
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(vllm): align the OOM reservation example with the fix recipe

The catalog recipe and the --gpu-memory-utilization parse error both use 0.5;
the docs said 0.1 for the same shared-GPU case. A 0.1 budget is below the
weights of most models people serve, so it trades one startup failure for
another.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): make the low-VRAM serve scenario assert the new guidance (EAI-8058)

The scenario's warning step matched "serving may fail on VRAM", which predates
this change, and its sibling matched only the bare flag name --- which a recipe
echo of a user-supplied --gpu-memory-utilization would also satisfy. Neither
fragment is specific to the pre-launch hint this scenario exists to prove.

Require the warning to name the pinned GPU and the note to carry vLLM's
total-VRAM reservation rationale, and pin both literals in the always-run unit
test so a rewording fails on every lane rather than only on the GPU lane that
can observe the scenario.

Also record on serve-16 why the masked, non-contiguous usable-set case behind
the --gpu membership rule has no scenario of its own: it is reachable only where
amd-smi is absent, so a scenario would assert against an injected seam rather
than the product. It stays unit-covered.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(diagnose): retire the WSL out-of-scope routing assertions

The WSL2 catalog (#340) makes WSL a covered platform, so `diagnose` no
longer emits an out-of-scope routing note there. Two tests written against
the old behaviour were asserting that note, and one of them also assumed a
default WSL examination produces no match -- it now legitimately does, since
a host with no /dev/dxg really is a `fix-wsl-1-gpu-not-exposed` finding.

Both now build on `wsl_base()`, the healthy-WSL fixture, so the vLLM OOM
signal is the only thing under test. The load-bearing half is unchanged and
still asserted: a score-25 `out of memory` mention stays visible in
`matched` as a WEAK row and does not promote itself into a match.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(fix): let fix-16-vllm-oom run on WSL2, where its checker answers

`check_16_vllm_oom` is registered for `["linux", "wsl"]` -- deliberately, per
the comment at diagnose.rs -- but the matching recipe was `LINUX_ONLY`.
`rocm fix` re-gates a fix-id on the *recipe's* platform list against the
running OS, where WSL2 is its own family, so on WSL2 `rocm diagnose` printed
`apply with: rocm fix fix-16-vllm-oom` and that command then printed the plan
and exited 3 with "This fix only applies on: linux. Running OS is: wsl."

Nothing about the remediation is bare-metal: it is vLLM CLI flags. WSL2 is the
supported way to serve from a Windows host and no `fix-wsl-*` entry covers the
failure mode, so the platform had a diagnosis and no working fix path.

The class, not just the instance: the two platform lists are written in
separate modules and each looks right alone. Add
`every_checker_platform_is_covered_by_its_recipe`, which walks `CHECKERS` and
requires every family a checker answers on to be one the recipe will act on.
Containment rather than equality -- a recipe may legitimately apply more widely
than its checker fires, since `rocm fix <id>` is reachable without a diagnosis.
It found no other mismatch across the remaining 23 entries.

The loop is user-observable, so diagnose-20 gains a step that runs the fix-id
the report named and requires the success rc (the wrong-OS refusal prints the
recipe first, so output alone cannot tell the outcomes apart). It is
`@requires-os:linux`, which WSL2 matches, so the `e2e-wsl` self-hosted lane is
the one that can observe the difference; every Linux lane runs the step.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): the shared OOM hint recommended 0.1 while every command said 0.5

`VLLM_GPU_MEMORY_UTILIZATION_HINT` still carried "(e.g. 0.1 for a small
model)". It is printed verbatim by `rocm serve`'s low-VRAM note and the vLLM
engine's post-failure OOM hint, and interpolated into the `fix-16-vllm-oom`
diagnosis summary -- so one `rocm diagnose` printed "e.g. 0.1 for a small
model" directly above `rocm serve <model> --gpu-memory-utilization 0.5`.

0.5 is the settled value; every command surface already used it (the recipe,
the diagnosis Fix, docs/vllm.md). 2c3a84e moved the docs and left the const
behind, and that commit's own message says why 0.1 is wrong: a 0.1 budget is
below the weights of most models people serve, so it trades one startup failure
for another. The harmful advice was what the product actually printed.

Pin the two together with `the_utilization_hint_example_matches_the_recipe_command`,
which compares the hint's worked value against the value the recipe commands.
Neither existing guard covers this: the diagnose/fix cross-check exempts the
summary as prose framing, and a line-based grep misses the const because the
flag and the value sit on different continuation lines -- which is exactly how
it was missed the first time.

Also correct the const's own doc claim, which named two sharing surfaces when
there are three, and record that the catalog rationale is deliberately separate
prose whose worked value is nonetheless pinned to this one.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): drop the duplicate LINUX_AND_WSL const from the merge

Both branches added an identical const — this branch for the fix-16 WSL gate,
main for #386 — at different offsets, so git merged both copies without a
conflict marker and the lib test failed to compile (E0428).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): keep untrusted log text out of the suggested diagnose command

On a failed vLLM serve the post-failure hint printed the user's own failing
log line inside a single-quoted `rocm diagnose --symptom '...'` command it
invites them to paste into a shell. The line is the vLLM subprocess's own
startup-log tail and nothing escaped it, so a routine apostrophe in Python
error text ("can't allocate", "model 'foo'") closed the quote: at best the
command the feature exists to provide was syntactically broken, at worst text
the user does not control landed outside the quotes. The neighbouring
human-readable echo passed the same bytes through verbatim, ANSI and all.

A line that cannot be rendered as one intact single-quoted word is now treated
as unusable and takes the fallback to VLLM_OOM_CANONICAL_SYMPTOM that already
existed for exactly that case, rather than being escaped: escaping preserves
the bytes but yields a command a reader cannot check by eye, and a wrong escape
is runnable and misleading instead of obviously broken. The echoed line is
stripped of ANSI sequences and control bytes.

Also restores the `out_of_scope` doc comment to what `diagnose` does. It had
been changed to claim `matched` may hold sub-threshold entries alongside it and
that the field is set when nothing clears MIN_SCORE_FOR_MATCH; neither holds —
`matched` is cleared whenever the field is set, and the field is decided purely
by platform coverage. The WSL2 example can no longer occur now that WSL2 is a
covered platform. That is the documented JSON contract, so a consumer trusting
it waits for rows that can never arrive. The relationship is now asserted.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(vllm): pin the exact rendering instead of fragment absence

The control-byte test carried the message "the ANSI sequence must be
removed whole, not just its escape byte" but only asserted that "[31m"
and "[0m" were absent. A stripper that drops the escape byte and the "["
it introduces and nothing further leaves "31m"/"0m" behind, which
satisfies both absence checks and carries no control byte, so the test
passed under exactly the defect it named. Verified by running that
mutant: green before, failing now.

Assert the stripper's output and the full rendered hint by equality
instead. The neighbouring quote-rejection test had the same shape -- "no
apostrophe in the symptom" also holds if the apostrophes were silently
deleted, the escaping-style rewrite the canonical fallback exists to
avoid -- so pin the fallback symptom by equality too. That mutant was
also green before and fails now.

Tests only; no behaviour change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): the serve summary's OOM note let log text break out of the command it prints

`oom_memory_note` interpolated `rocm_core::vllm_oom_diagnose_symptom`
straight into a single-quoted shell word:

    run `rocm diagnose --symptom '{symptom}'`

That value is a line of vLLM's own subprocess output, and the sentence
around it invites the user to paste the command. An apostrophe -- routine
in Python error text (`GPU 0 can't allocate the model's weights`,
`model 'foo'`) -- closes the quote, and the ANSI/BEL bytes vLLM's
colourised logger emits reach the terminal verbatim. The vLLM engine's
startup-failure hint had exactly this defect and was fixed on
`gpu-out-of-memory`; this is the same defect on a second surface.

Promote the two guards written for the engine into `rocm-core`, next to
the helpers they guard, and use them from both command-building sites:

- `quotable_in_single_quotes` -- used by the engine hint and now by the
  serve summary note. A quote- or control-bearing symptom is rejected in
  favour of `VLLM_OOM_CANONICAL_SYMPTOM`, which is guaranteed to report a
  cause.
- `strip_terminal_control_sequences` -- used by the engine, which echoes
  the raw failing line. The serve summary echoes no raw line, so it needs
  only the quotability guard.

Rejecting rather than escaping (`'` -> `'\''`) stays deliberate: a human
pastes the result, an escaped form cannot be eye-checked, a subtly wrong
escape is runnable-and-misleading rather than obviously broken, and
control bytes would still reach the terminal.

`vllm_oom_diagnose_symptom` keeps its contract -- it returns log text,
not a shell token -- because some callers only display it; guaranteeing
quotability there would withhold the user's real error from text that is
never pasted anywhere. Its doc now says so, and says what a
command-building caller owes.

Regression tests on the serve-summary surface, both falsified against the
pre-fix code: an apostrophe-bearing traceback tail produced four quotes
in the rendered command, and a colourised line put raw escape bytes in
it. Both assert the exact symptom that is printed rather than the absence
of a fragment -- "contains no `'`" also holds if the apostrophes were
silently deleted, which is the escaping behaviour the fallback exists to
avoid. A third test pins that a clean line is still quoted verbatim, so
the guard does not cost the common case its own error text.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(serve): pin the all-busy auto-GPU fallback to a reported ordinal

`select_auto_gpu_index` now sources its candidates from the actual VRAM
rows, because a visibility mask can make reported ordinals
non-contiguous. That removed the invariant the terminal
all-candidates-busy fallback quietly relied on: with candidates drawn
from a dense `0..count`, ordinal 0 was always a real device, so
returning a literal `0` was safe by construction. With rows reported at
[2, 3] it is not — the busy filter empties the candidate list, every
pass falls through, and a fabricated 0 goes straight into
`HIP_VISIBLE_DEVICES`, pinning a device nothing ever reported. Under the
strict GPU-required / no-silent-fallback policy that is a silently wrong
pin, not a fail-fast.

The merge resolution already routes the fallback through the reported,
mask-filtered set, so the behaviour is correct. What was missing is
coverage: `auto_selection_ranks_the_actual_reported_row_indices` claims
pass 3 "never returns a fabricated index 0", but it only exercises the
*partially* busy case (busy = [3]), which never reaches the fallback at
all. This adds the all-busy case that does.

Falsified: with the fallback reverted to `vec![0]` this test fails with
left: [0], right: [2] (and takes main's
`auto_selection_all_visible_busy_falls_back_to_lowest_visible_not_zero`
down with it); both pass with the fallback restored. Equality assertion
rather than fragment absence, so a future regression cannot pass it
vacuously.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(e2e): drop a serve-16 note the merge left dangling

The comment on `serve-16` described `--gpu` validation as falling back to
membership in the KFD/DRM usable set and pointed at
`validate_pinned_gpu_index_falls_back_to_usable_set_membership` as the
unit coverage for the masked sub-case. Merging #267 replaced that
validator with one that consults the visibility-resolved set directly,
so the named test no longer exists, and the masked sub-case it called
uncovered now has two scenarios of its own (serve-19, serve-20).

Nothing here changes behaviour or which lane runs what; the paragraph is
rewritten to say what serve-16 still covers and to point at the
scenarios that own the rest. The conflict was textual-merge-clean — this
block came wholly from one side — so only reading it caught the dangling
reference.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): treat a bare CR as a line boundary in the vLLM anchor filter

`vllm_anchored_lines` split with `str::lines()`, which splits only on `\n`
and `\r\n` -- a lone `\r` is ordinary text to it. Every progress bar in this
ecosystem repaints with a bare CR (tqdm, pip, huggingface), and pasting a
captured terminal scrollback is the expected way to use
`rocm diagnose --symptom '...'`, so a real capture collapsed into one giant
"line". The anchor then matched anywhere in it and every keyword in the paste
scored at full weight: a llama.cpp OOM in a capture that also mentions vLLM
came back at 95, well above HIGH_CONFIDENCE, as "vLLM ran the GPU out of
memory at startup".

That is the whole-paste scoring `only_the_anchored_lines_are_scored_not_the_
whole_paste` exists to prevent, reinstated by one byte. Neither existing guard
saw it: both use `\n`-only fixtures.

Split on `\r` as well, so the line boundary is the one the terminal actually
renders, and add a regression case built from a bare-CR capture. It also pins
the `\n` form of the same content and an anchored-across-CRs control, so the
case stays a boundary question rather than a scoring one.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): parse the escape grammar and reject format characters

The OOM hint quotes a line of untrusted subprocess output into a
`rocm diagnose --symptom '...'` command and echoes it in a `bail!`-ed
message, so the sanitizer in front of it is load-bearing. Three holes:

The escape scan ran to the next byte in `0x40..=0x7E` with no notion of the
CSI grammar's parameter, intermediate and final classes. On well-formed
colour codes that is indistinguishable from correct; on the malformed and
truncated sequences a *killed* process emits -- which is the only case this
code path exists for -- every one of them corrupted the message. A truncated
CSI swallowed the next sequence's introducer (`ESC [ 1 ; 2 ESC [ 0 m Killed`
rendered as `0mKilled`); a multi-byte scalar can never be in that ASCII
range, so the scan ran past it and ate to the next byte that happened to land
in it (`ESC [ 1 2 é out of memory` rendered as `ut of memory`); an OSC title
leaked its body as text; a stray `ESC ESC` left `[0m` printed literally.
Follow the grammar, and on an invalid byte stop *without consuming it*, so
the damage is bounded to the malformed sequence and a following well-formed
one is still recognised.

`char::is_control` is Unicode category `Cc` only, so `Cf` format characters
survived both the stripper and `quotable_in_single_quotes`. U+202E in a log
line cannot escape the single quotes, but it does reverse how the printed
command renders -- the same "untrusted output must not control what the
terminal shows" concern, so it gets the same answer.

The stripper's doc comment promised control bytes could not repaint the
terminal "from inside rocm-cli's own error message", while the log tail a few
lines above it in the same message went out raw. Sanitize the tail too, per
line so it keeps its shape.

Also states why `.rev()` is deliberate (the last OOM line is the proximate
failure) and what it costs, since nothing said so; behaviour is unchanged.

Tests: the existing control-byte test only exercised an `'m'`-terminated CSI,
so narrowing the final-byte range to `'m'` left it green. The new case table
pins the range, the byte classes and every introducer. The diagnosability
test could not fail for the feature it is named for either -- a mutant that
always returns VLLM_OOM_CANONICAL_SYMPTOM satisfies it, because that symptom
is diagnosable by construction -- so it now pins the exact symptom per line,
over a table that deliberately straddles the routing and fallback branches.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(diagnose): pin the fix-16 plan to its catalog copy

The command block is maintained twice -- once on the `Fix` that `diagnose`
attaches, once on the `rocm fix` catalog recipe -- and the two had already
drifted. The cross-check that runs over every matched finding could not see
it: `recipe_verify_notes_and_run_commands` filters `#` lines by construction,
and only the comments differed.

That prose is the point of this entry. Two different faults produce the same
OOM and need opposite responses, so the comments are what tell the user which
of the two each step addresses -- a user who meets the `diagnose` copy and a
user who meets the `rocm fix` copy must be told the same thing. Align the
`diagnose` copy to the catalog's Case 1/Case 2 wording and wire fix-16 into
`assert_plan_matches_the_catalog_copy`, the byte-identical guard that already
existed for fix-17-torch-dlpack.

Also restores the general rule to `LINUX_AND_WSL`'s doc. Deduplicating the
const in 1e23717d kept the fix-16-specific comment and dropped main's general
one, leaving a shared const documented as if fix-16 were its only user;
fix-19-shm-too-small uses it too.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* refactor(serve): drop a vacuous guard and the duplicated AMD-device test

Three leftovers from the auto-selection rewrite and the merge around it:

`select_auto_gpu_index`'s pass-2 `max_by` still carried
`&& usage_for(index).is_some()`. Inside `if vram.is_some()` every candidate
came from a reported VRAM row, so that is vacuously true -- it implies a
failure mode (a selected ordinal telemetry never reported) that only existed
when candidates were a synthetic `0..count` range. Replace it with a comment
saying why no guard is needed.

`drm_device_is_amd` re-implemented the private `rocm_core::is_amdgpu_device`,
with a doc comment asserting the two were "the same two-signal test" and that
divergence would let the multi-card ordinal guard slip -- a promise nothing
enforced. They must agree because both count AMD cards and this one feeds
that guard, so make the rocm-core function public and delegate. One copy is
better than two copies plus a cross-check.

The second assertion in `auto_selection_all_reported_busy_falls_back_to_a_
reported_ordinal` varied the order of the `busy` slice, which is only ever
read through `.contains()` -- so it was a duplicate of the first while its
comment claimed an order-independence property nobody was testing. Vary the
order of the VRAM *rows* instead: `amd-smi` orders by enumeration, not by
index, and that is what `reported.sort_unstable()` is there for. Dropping the
sort now turns it red with `left: [3], right: [2]`; it could not before.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): stop the pre-launch hint suppressing the whole OOM note (eai-8059)

Address the review round on PR #284.

serve summary: `append_oom_serve_note` returned early whenever the shared
`--gpu-memory-utilization` hint was already in the notes, which discarded
the entire OOM note on the feature's most likely trigger — low VRAM
warned about pre-launch, user proceeds, vLLM OOMs. What was lost is not a
duplicated fragment: the confirmation that this attempt really did run
out of GPU memory, the "if the model does not fit, lowering the
reservation will not help" branch, and `rocm diagnose --symptom` with the
user's own failing line. De-duplication now happens at composition time
in `oom_memory_note`, which omits the hint sentence only. The unit test
that asserted `notes == pre_launch_notes` cemented the suppression as the
specification; it now pins "the hint appears exactly once, and the rest
of the note is still appended".

rocm-core: restore the `--gpu-memory-utilization` range wording a merge
from a pre-fix tree silently reverted. `parse_gpu_memory_utilization`
rejects `<= 0` and accepts `1`, so `<0-1>` advertised a value the same
tool refuses. The existing pin only checked the worked example (`0.5`),
so extend it to the range text.

rocm-core: drop the unreachable `VLLM_OOM_CANONICAL_SYMPTOM` fallback in
`vllm_oom_diagnose_symptom`. Both callers pre-gated on
`vllm_log_shows_oom` and the `.find()` re-evaluated the byte-identical
predicate, so it could never fire. The function now returns
`Option<String>` and its `None` *is* the "no OOM in this tail" answer, so
the callers use it as the gate instead of running the same predicate over
the same string twice. The quotability fallback beside it is genuinely
reachable and is unchanged.

e2e: delete the duplicated `extra_env` loop in `tui_driver`. The
surviving loop runs before the provider-credential `env_remove` calls; the
duplicate ran after them, so an `extra_env` entry naming a credential key
could have defeated the strip and selected a cloud backend for a journey
meant to be deterministic local chat. Both current callers pass
non-credential keys, so behaviour is unchanged.

Also non-blocking review items: correct the two comments that called the
`any_live_managed_service_for_model` pre-gate cheap (its condition goes
through `load_managed_services`, which probes and rewrites every record
before the model filter); document that the OOM guidance is interactive-
only; route the interactive-summary wait through
`tui_driver::default_timeout()` so `E2E_TUI_TIMEOUT_SECS` applies; assert
both knobs in the serve-24 step that promises "knobs"; document
`@requires-oom-fault-injection` in the e2e README; and cross-reference the
`ROCM_E2E_SIMULATE_OOM_LAUNCH` producer and consumer.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(diagnose): split the vLLM anchor filter on rendered lines, not two bytes

The previous round fixed the whole-paste collapse for a bare `\r` and left it
open for every other line advance. `.split(['\n', '\r'])` is an allowlist of two
characters under a doc comment claiming the result was "the line boundary the
one the terminal actually renders", and it was not: with the `vllm` anchor and a
llama.cpp OOM on genuinely separate rendered lines, `ESC E` (NEL), `ESC D`
(IND), `CSI 1 B` (CUD), `\x0b`, `\x0c`, `\x85`, U+2028/U+2029 and any stray
control byte each scored 95 against a `high_confidence_threshold` of 75 --
citing the llama.cpp line's tokens as the evidence for a vLLM verdict. `ESC E`,
`ESC D` and `CSI n B` are terminfo's `nel`/`ind`/`cud1`, i.e. ordinary output
from curses- and `rich`-style progress UIs, and `rocm serve`'s own failure path
invites exactly this paste.

The one-line `char::is_control()` split that suggests itself does close every
one of those, but it trades the false positive for a false negative on the most
ordinary input there is: a colourised logger emits SGR *inside* a line, so
splitting at `ESC` cuts `ESC[31m vllm: ESC[1m torch.OutOfMemoryError` between
its anchor and its error token. Measured on that candidate, that line went from
95 to 0.

Getting both halves right needs the escape grammar, and this crate's dependents
already have one -- `strip_terminal_control_sequences`, which took three rounds
to get right in the vLLM engine. Rather than write a second parser, move the
ECMA-48 walk down into `rocm-core::terminal` as a classifier over three tokens
(`Text`, `LineBreak`, `Ignorable`) and give it two consumers: the engine's
stripper, whose behaviour is unchanged, and a new `rendered_lines` the anchor
filter splits on.

The classification is deliberately lopsided rather than terminal-accurate, and
the doc says so instead of claiming fidelity it does not deliver. Only SGR and
the string-argument sequences (OSC/DCS/SOS/PM/APC) are `Ignorable`, because
neither can move the cursor; `\t` stays intra-line text; everything else is a
boundary. That is a one-sided guarantee -- two things drawn on different rows
are never scored as one line, while one row may come back split -- and it is the
right asymmetry here: merging rows produces a confident wrong cause, splitting
one loses a diagnosis the canonical-symptom fallback already covers. A real
terminal emulator is not decidable on a byte stream with no screen model
(`CSI H` addresses a row that depends on everything before it), and
over-splitting needs none of that.

After: every separator above scores 0, `\n` and `\r` stay 0, and both the plain
and the colourised genuine vLLM OOM stay at 95.

Tests: the CR regression becomes a table over the separators rather than one
test per byte, plus the SGR case in the other direction, plus a property test
over the whole C0/C1 range pinned to `char::is_control` rather than to this
module's own classifier. Reverting the boundary set turns three of them red
(`nel esc-e`, `U+0000`, `a ESC E (NEL)-separated llama.cpp OOM ...`); making SGR
a boundary instead turns the colour test red on `SGR inside a line must not
change the verdict`.

A confidently wrong cause is user-observable, so it is also pinned as
`@id:diagnose-vllm-oom-not-attributed-across-rendered-lines` (diagnose-21),
which fails on the pre-fix boundary set and passes after.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* chore(rocm-core): give the new terminal module the license header CI requires

`crates/rocm-core/src/terminal.rs` arrived from the base branch (fcf99156)
without the `Copyright`/`SPDX-License-Identifier: MIT` header that
`licenserc.toml` requires of every `**/*.rs`, so `License header check
(hawkeye)` is red. It is red on PR #251 at the same commit, for the same file,
so this is the base's omission surfacing here rather than anything the merge
introduced -- fixing it at the source would be better, and this can be dropped
from the merge once it lands there.

The header is the exact text `licenserc.toml` declares, copied from the
sibling modules, and it goes above the `//!` module doc the way `diagnose.rs`,
`runtime.rs` and `uv.rs` place theirs.

EAI-8059

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* chore(core): add the licence header the new terminal module was missing

`fcf99156` created `crates/rocm-core/src/terminal.rs` without the SPDX header
every other source file carries, so `License header check (hawkeye)` failed —
its `licenserc.toml` includes `**/*.rs`. Header only; no code change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): scope the consumption rule and name the one stripper change

Two prose defects in the module the previous commit added, both of which a
reviewer would rely on.

`next_token`'s doc asserted that "the contract here is 'render what the terminal
would have rendered', which is the only rule that stays decidable on a byte
stream with no framing". That was true of the stripper the paragraph came from,
where the only question was how much of a well-formed sequence to consume. It is
false of `next_token`, which also decides line boundaries and deliberately
over-splits -- twenty-eight lines further down the same comment says a terminal
emulator "is not decidable on a byte stream with no screen model", and the
section between explains the classification is lopsided rather than
terminal-accurate. Left standing, that sentence is a documented licence to
"fix" the over-splitting toward terminal accuracy, which reopens the row-merging
misattribution the previous commit closed. It is now scoped to the consumption
decision it is actually about: a well-formed sequence is consumed whole because
a real terminal consumes it too, and which token it is then labelled with is the
separate, deliberately non-terminal-like question below.

The same commit said the engine stripper's "behaviour is unchanged" by the move.
It is unchanged except for U+2028/U+2029. The engine-local stripper classified
non-escape characters with `is_control_or_format` alone, which is false for both
(neither is `Cc`, neither is in the enumerated `Cf` set), so both survived into
the stripped message; `classify_char` puts them in the boundary set, so they are
now dropped. Measured on this head: the old code returns "a\u{2028}b" for input
"a\u{2028}b", the new one returns "ab". The widening is the better behaviour --
a mandatory Unicode line break is exactly what this stripper exists to keep
untrusted output from injecting -- but nothing stated it and no test pinned
either scalar, so a silent drift back would have gone unnoticed. `classify_char`
now says so, and the vLLM stripper table gains both cases: reverting the pair
out of the boundary set turns `the_stripper_follows_the_escape_grammar_not_just_
the_colour_case` red on "a line separator must not survive into the message".

Also from the same review, three prose corrections with no behaviour change:
`rendered_lines`'s lead sentence now states the over-approximation up front
instead of stating the strong property and retracting it a paragraph later; the
string-argument arm documents the one crafted-input hole in the never-merge
guarantee (a properly terminated OSC body containing a line feed is swallowed
whole, so `"vllm ESC ] 0 ; ti \n tle BEL llama.cpp OOM"` comes back as one line,
where common terminals abort the control string and draw two rows); and the
`fix-16` catalog test's comment cites the cross-check that actually exists,
`diagnosis_remediation_matches_the_fix_catalog_for_shared_fix_ids`, rather than
a name that is in neither the tree nor its history.

Last, `effective_gpu_count`'s doc claimed the VRAM-row fallback applies when the
amd-smi count is absent, while the code also falls through when it is present
and zero -- a host amd-smi cannot enumerate but sysfs can. The doc now names
both triggers and the second one gains the test case it never had; dropping the
`count > 0` filter on `detected` turns it red.

EAI-8058

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): qualify the never-merge guarantee where a consumer reads it

`rendered_lines` promised, unqualified, that "two pieces of text the
terminal drew on different rows never end up in the same returned
segment". That is the invariant the module exists for -- it is what stops
a `llama.cpp` OOM being scored against a `vllm` anchor -- and it is not
what the code holds. The counterexample was already written down in the
same commit, 140 lines above, but in a body comment inside the private
`next_token`, so it reached no reader of the generated documentation, and
the public doc's pointer to it (`See [next_token]`) was a link to a
private item that rustdoc will not resolve. `cargo doc -p rocm-core`
warned on exactly that line.

This is the third finding of the same class on this module, so the fix is
structural rather than a reworded sentence. The exception now has one
authoritative statement, in the module documentation, which is public and
therefore rendered: a line break inside a well-formed string-argument
body (`OSC`, `DCS`, `SOS`, `PM`, `APC`) is consumed with that body, with
the worked example spelled out. `rendered_lines`'s own doc carries the
qualifier inline -- so the only absolute a consumer can read is true --
and links to the module docs rather than to a private item. The
`next_token` arm that creates the exception no longer restates it; it
points at the module paragraph and names what has to change with it.

The behaviour is unchanged. Following the grammar rather than guessing
where an unterminated body ends is the right call: the alternative
reintroduces the leak of OSC bodies as text.

`a_line_break_inside_a_well_formed_string_body_is_consumed_with_it` pins
the worked example so prose and behaviour move together; flipping its
expectation to the two-segment reading fails with
`left: ["vllmllama.cpp OOM"]`. It also pins that the same break outside a
string body still splits, and that an unterminated body still stops at
the following `ESC`, so the hole cannot widen unnoticed.

Also, both non-blocking wording items and one more instance of the same
dangling-link defect the PR introduced:

- `the_split_never_merges_two_rendered_lines` overstated what the module
  guarantees; its body already scoped itself to single non-drawing
  scalars, so it is renamed to
  `a_lone_non_drawing_scalar_never_merges_two_rendered_lines`.
- the citation in `the_oom_plan_matches_the_catalog_copy` called
  `diagnosis_remediation_matches_the_fix_catalog_for_shared_fix_ids` "the
  cross-check that runs over every matched finding"; it iterates a fixed
  two-element array of hand-built diagnoses, so it is now "the cross-check
  over the fixes shared with the catalog".
- `DiagnoseReport::out_of_scope`'s doc linked `[catalog_covers]`, also a
  private item, also warned on by `cargo doc`, and also added by this
  branch. It now names the rule in prose instead.

`cargo doc -p rocm-core --no-deps` goes from 6 warnings to 3; the
remaining three are pre-existing on `main` and are not touched here.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): drop the terminator from the merge exception; it is not required

`rendered_lines`'s doc said two rendered rows share a segment "only when the
break between them sits inside a well-formed string-argument sequence ...
whose body ECMA-48 defines as running to its terminator", and the module
section said the shape "needs a well-formed introducer and terminator around
the line break". Both make a terminator necessary for the merge. It is not.

`skip_string_sequence_body` ends the body scan on `BEL`, on `ST`, on any other
bare `ESC` (left in place), or on end of input, and `next_token` returns
`Token::Ignorable` for all of them. The break is swallowed either way. The
test added in this PR is itself the counterexample and passes: its input
carries no `BEL` and no `ST` anywhere, and the two rows still merge. The
private helper's own doc comment already said this, so the public doc was
contradicting the private one.

State what the code does instead, and add the unterminated input as a second
worked example so the prose has to move with it.

That also invalidates the reassurance built on top of the terminator being
required. An unterminated body followed by more output is reachable from
ordinary output by two routes: a process killed part-way through writing an
`OSC` title, which is the case this walk exists for, and the interleaved
capture itself, since the serve log hands the subprocess's stdout and stderr
the same file handle and either stream can cut the other's title. Say so, and
say why following the grammar is still right -- guessing where an
unterminated body ends reintroduces the leak of `OSC` bodies as text.

`vllm_anchored_lines` still carried the original unqualified "two things drawn
on different rows are never scored as one line", which is the sentence the
module section claims to be the single authoritative statement of. Replace the
restatement with a pointer rather than a second qualified copy; a second copy
is a stale copy waiting to happen, and this is the call site where a stale one
does the damage.

Docs, comments and one test rename only; no behaviour change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): delete the restated reachability claim, and name the third behaviour

`rendered_lines`'s own doc claimed a process killed mid-title was enough to
reach the merge shape. It is not. The merge needs drawable text on both sides
of the swallowed break landing in one segment, and a kill leaves the body
running to the end of the input, where there is no far side to join to:

    rendered_lines("vllm\u{1b}]0;ti\ntle") == ["vllm"]

The `tle` a terminal drew on its second row comes back in no segment at all.
That is a third behaviour -- neither the split nor the merge the one-sided
framing describes -- and it was undocumented, so the prose reached for it as an
example of the merge instead.

This is the fifth round on one sentence, and the sentence is always in the same
place: `rendered_lines` restating the exception rather than pointing at the
module section that owns it. So the restatement goes rather than gets reworded.
The qualifier that binds callers stays; the terminator detail and the
reachability claim are deleted and the `[the module documentation](self)`
pointer now carries the routes, which makes the "single authoritative statement"
claim true rather than aspirational.

The drop is documented rather than filed as a defect: it is the direct,
already-argued consequence of not guessing where an unterminated body ends, and
it errs in the losing direction the contract leaves unpromised -- it costs a
diagnosis, it cannot invent one. Both one-sided framings are widened to admit
it, so neither now claims two directions while a third exists.

Also corrected: the interleaved capture reaches `rendered_lines` through a user
paste into `rocm diagnose --symptom`, not through the engine's own path, which
pre-splits its log tail with `str::lines`; and the contract is not "stated
once" in `vllm_anchored_lines`'s doc, which names two places in the same breath.

The end-of-input case is now pinned by a test. Nothing covered it, which is why
the prose could cite it wrongly twice.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): the merge trigger is text following the sequence, not end of input

The module section named end of input as the reason a swallowed break loses a
row rather than merging. It is not the reason: a properly BEL-terminated body
with nothing after it loses the row just the same, and so does one with text
after it that sits past a break. What decides is whether drawable text follows
the sequence in the same segment, however the body ended.

Rather than restate that correctly in each of the three places the section said
it, keep one statement where the third behaviour is introduced and delete the
other two -- the divergence between copies is what produced this round and the
five before it.

Pin the two shapes that were not covered, so no future wording can name a
narrower trigger than the code has without a test failing.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): qualify the closing summary's loss condition to match the rule

The section states the trigger precisely as drawable text following the
sequence *in that same segment*, then the closing summary restated it
without the qualifier. rendered_lines("vllm\u{1b}]0;ti\ntle\u{7}\nx")
is the case that splits them: text does follow the sequence, the row is
lost anyway, because the text is past a break and so in another segment.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core): retitle the residual-behaviour section and stop restating its rule

The section documents two residual behaviours, not one exception, and its
own claim about the file's structure was false: the code arm and the test
do restate the rule, accurately and usefully, as local commentary. Say
instead that this section is the normative statement.

Three further corrections, all the same recurring defect of the same rule
living in more than one place and drifting:

- a break inside a string body can lose more than one rendered row
  (`rendered_lines("vllm\u{1b}]0;ti\ntle\nmore") == ["vllm"]`), so "one or
  more";
- the "what decides" clause named only the far side, but a row is lost
  with no near side too (`rendered_lines("\u{1b}]0;ti\ntle\u{7}x") ==
  ["x"]`) — dropped in favour of the both-sides sentence before it;
- the closing summary repeated the same far-side-only qualifier, so it is
  dropped there as well.

Net two lines shorter.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): keep a detected GPU with no VRAM row as an auto-select candidate

`select_auto_gpu_index` builds its candidate ordinals from the VRAM rows
alone whenever telemetry is present. That was introduced so a visibility
mask reporting non-contiguous ordinals is scanned by its real indices
instead of a synthetic `0..count` range, which is right — but a short row
set is not always a mask. `parse_gpu_vram_usage` drops any device entry
missing `/mem_usage/used_vram/value` or `/mem_usage/total_vram/value`, so
a device the lighter `list` enumeration counts can simply have no row.

With device 1 absent from the rows and device 0 pinned by a managed
service, `reported` was `[0]`, the busy filter emptied the candidate list,
all three passes iterated nothing and the terminal fallback returned
device 0 — `serve` pinned to an already-occupied GPU, then failing on
VRAM, which is the fault this selection exists to reduce.

Union the detected `0..count` range back in only when `visible` is known.
The existing retain then validates every ordinal against the mask, so no
unconfirmed index is ever synthesised: under a `[2, 3]` mask with
`count == 2` the synthetic `[0, 1]` is dropped wholesale and the rows
still stand alone. Without `visible` there is no second source to confirm
an ordinal against, so the rows remain authoritative.

Pins the case with a test: a two-GPU count, one row, an all-busy reported
set, which the suite never exercised (every all-busy case passed
`detected: None`).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): discriminate a masked row set by its indices, not by `visible`

The previous commit unioned the detected `0..count` range back into the
auto-select candidates only `if visible.is_some()`, so the fix reached
only hosts whose visibility set could be probed.

`probe_usable_amd_gpu_indices` returns `None` unconditionally for
`#[cfg(not(target_os = "linux"))]`, and `serve` threads that straight
into `visible` through `select_gpu_indices_under_launch_lock`. So
`visible` is always `None` on Windows — a supported platform — and on
any Linux host whose KFD topology and DRM cards are both unreadable.
With `count == 2`, one VRAM row for GPU 0 and GPU 0 pinned by a managed
service, those hosts still returned the busy GPU 0 where the base branch
returned the idle GPU 1: the exact fault this selection exists to reduce.

`visible` was never the right discriminator. The rows are a re-indexed or
sparse set precisely when some row index falls at or above `count`
(`[2, 3]` against `count == 2`), which is the shape the rows-only
construction was introduced for; below the count they are a subset of a
dense range the count already asserts exists, and unioning `0..count`
there is no stronger an assumption than the telemetry-less arm already
makes. Require `detected` to be `Some(n > 0)` as well, so the count comes
from the `list` enumeration rather than from `rows.len()` restating the
rows' own size — `parse_gpu_vram_usage` only falls back to the array
position when an entry has no `gpu` field, so a payload naming `"gpu": 0`
twice yields duplicate ordinals that would otherwise conjure a GPU 1.

The visible retain still runs on top whenever a mask is known, so nothing
synthesised survives where the mask contradicts it.

Pins both halves of the condition: forcing either to `true` turns an
assertion of the new test red, and the masked assertion now holds with
`visible: None` so it pins the row-index rule itself rather than the
retain.

Also replaces the Pass 2 rationale, which the union falsified. `usage_for`
is no longer total over `candidate_indices`, but no row guard is needed
for a different reason: the comparator keys are `Option<u64>` and `None`
sorts below every `Some`, so a rowless ordinal can win only when no
candidate has a row, and then the index tie-break returns the lowest —
exactly what Pass 3 would return. Re-adding the guard leaves all 771 unit
tests green, so the guard is behaviourally inert and stays out.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(vllm): reject a mandatory line break from the quoted symptom too

`quotable_in_single_quotes` gates what is embedded in the
`rocm diagnose --symptom '...'` command the OOM hint prints for the user
to paste, and it tested each character with `is_control_or_format` alone.
That predicate is `char::is_control` plus the enumerated `Cf` set, and
`U+2028 LINE SEPARATOR` / `U+2029 PARAGRAPH SEPARATOR` are `Zl`/`Zp`, so
both passed the gate.

The value reaching it…
mikeroySoft pushed a commit to mikeroySoft/rocm-cli that referenced this pull request Nov 6, 2026
…8457) (ROCm#447)

* fix(serve): run the KFD pre-flight at every amd-smi launch site (EAI-8457)

`amd-smi` launched against a kernel fusion driver in a bad state blocks in
uninterruptible sleep, which no signal escapes — so `rocm serve` hangs and
survives both Ctrl-C and SIGKILL. The repo documents that and calls the
`/dev/kfd` pre-flight mandatory, but three of the four production launch
sites skipped it. Two of those, `gpu_vram_usage` and `detect_gpu_count`,
sit on the normal `rocm serve` path for any run that is not CPU-only, and
had no timeout either.

Give the pre-flight one definition in `rocm-core` and route the three
unguarded sites through a single guarded launcher, so a future launch site
inherits the guard rather than having to remember it.

The gate is Linux-only on purpose. `/dev/kfd` exists nowhere else, and
`amd-smi.exe` is a supported Windows binary with a live Strix Halo lane, so
a check that merely failed to find the device would have disabled GPU
detection on every Windows host.

Two consequences worth naming:

- On Linux the two `serve` probes now decline to launch when `/dev/kfd`
  cannot be opened, where before they launched and usually failed anyway.
  GPU auto-selection degrades to service state alone, which is what it
  already does whenever amd-smi is unavailable.
- `run_command_with_timeout` now drains both pipes on their own threads.
  Polling `try_wait` without draining deadlocks once the child fills the
  64 KiB pipe buffer — which `amd-smi metric --json` clears easily on a
  multi-GPU host — and that deadlock surfaced as a spurious timeout.
  Routing the serve path through the helper without this would have traded
  the hang for silent loss of VRAM telemetry on exactly the machines that
  matter. Its timeout message no longer quotes partial output, because
  collecting it means waiting on a child that may be the unkillable one.

The dash collector keeps its own pre-flight: it already runs one, it needs
an async launcher, and an edge from `rocm-dash-collectors` to `rocm-core`
is what the `crate_edges` layering guard exists to stop. Its check is not
cfg-gated, so the dash detects no GPU off Linux — a separate pre-existing
bug, untouched here.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* style: apply rustfmt to the amd-smi pre-flight change

The hand-wrapped import list and the amd_smi_json call did not match what rustfmt produces; the prek hygiene lane caught it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): wave the KFD pre-flight through on WSL (EAI-8457)

WSL is `target_os = "linux"`, so it compiled the real-open branch of
`kfd_readable` -- but it has no `/dev/kfd` at all. It reaches the GPU
through `/dev/dxg` and the Windows host driver, which is exactly why the
WSL diagnosis catalog skips every bare-metal driver check (docs/wsl.md,
examine::WslFacts).

So the open always failed there and `amd_smi_json` declined every launch:
`gpu_vram_usage` and `detect_gpu_count` on the ordinary serve path, and
the daemon bridge snapshot. Both serve probes discard the error with
`.ok()?`, so this was a silent loss of GPU detection on a supported
platform -- one with its own diagnosis catalog and its own GPU E2E lane.
The D-state hazard the gate exists for is an amdgpu/KFD one, which is
precisely the stack WSL never loads.

Wave WSL through at the top of the Linux `kfd_readable`, with the
determination injected via `kfd_readable_with` so the behaviour can be
pinned from an ordinary Linux host. Correct the `KFD_DEVICE` comment,
which asserted no other platform exposes one without excluding WSL.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): reap the child the amd-smi timeout path kills (EAI-8457)

`run_command_with_timeout` killed a timed-out child and then dropped the
handle. `std::process::Child` has no reaping `Drop`, so the corpse stayed
in state `Z` for the lifetime of the parent. Both per-app copies this
function replaced called `wait_with_output()` after the `kill`, so the
reap used to happen; consolidating them dropped it.

That is a real leak rather than a tidiness point because one caller is a
daemon. `rocmd` probes `amd-smi` with a 2s timeout from its watcher tick,
twice per GPU-telemetry collection, on a 60s cadence. A host whose
`/dev/kfd` is readable but whose `amd-smi` simply overruns 2s — ordinary
on a busy multi-GPU box — therefore accrues zombies indefinitely. The KFD
pre-flight added here does not cover it: that only prevents the spawn when
the device is unreadable.

Restoring the `wait` inline would reintroduce the hang the timeout exists
to bound, since an uninterruptible-sleep child blocks the `wait` exactly
as it ignores the `kill`. Hand the child to a detached reaper thread
instead, so the calling thread still returns promptly. The reap runs
whether or not `kill` succeeded, because the likeliest failure is a child
that exited between the poll and the kill — already a zombie, and already
in need of reaping.

The doc comment is explicit that this is best-effort: an unkillable child
parks its reaper instead of leaving a zombie, trading a stranded PID for a
stranded thread. That is better than blocking the caller and no worse than
the leak it replaces, but it is still a leak, and the pre-flight remains
the only actual mitigation. Reapers do not accumulate otherwise — one per
timeout, each exiting as soon as its child does.

Covered by a Linux-only test that reads the child's recorded pid straight
out of `/proc`: without the reap it observes state `Z` for the full ten-
second poll budget, with it the entry is gone in under a second.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(core): assert the timed-out child is gone, not merely not-a-zombie (EAI-8457)

`assert_ne!(state, Some('Z'))` was satisfied by two different outcomes: a
child the timeout path killed and reaped, and a child it never killed at
all, which sits in `S` while it sleeps on. Only one of those is the
contract, so the assertion could not tell a working timeout path from a
broken one. Assert the `/proc` entry is gone instead, and retry on any
surviving state rather than only on `Z` so neither the post-`SIGKILL`
teardown transient nor the detached reaper reads as a pass.

Mutation-checked: unmutated passes; dropping `child.kill()` fails with
`Some('S')` where the old assertion passed; dropping
`reap_in_background` still fails with `Some('Z')`.

Also record why the declined-launch behaviour has no Gherkin scenario, as
AGENTS.md section 3 requires when one cannot be written. The premise needs a
host that is GPU-present by the KFD/DRM sysfs rule yet cannot open
`/dev/kfd`; no lane is that host and the harness cannot manufacture it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): restore the low-VRAM test hook my merge dropped

Resolving the `gpu_vram_usage` conflict after ROCm#251 was squash-merged, I kept
this branch's pre-refactor body and then restored main's dispatcher -- but not
`simulated_low_vram_usage()` or its `#[cfg(feature = "e2e-test-hooks")]` call,
both of which arrived with ROCm#251. The symbol count fell from main's 4 to 2,
leaving only the two dangling references in the test module.

Nothing in the default build reaches the hook, which is why 21 CI checks
stayed green over it. It breaks only under the feature:

    cargo check -p rocm --features rocm/e2e-test-hooks --tests
    error[E0425]: cannot find function `simulated_low_vram_usage`

and the step at serving_steps.rs:865 still set ROCM_E2E_FORCE_LOW_VRAM with
nothing left to read it -- a scenario quietly asserting nothing.

Restored verbatim from main: both the hook and its gated call are now
byte-identical to origin/main, and the symbol count matches again.

Verification note: the Linux devbox is unavailable (its pod has been Pending
~90m behind a 502 from the node proxy) and macOS cannot build this workspace
-- `cargo check` stops in rocm-engine-lemonade on `u32 & u16` long before
reaching this crate -- so the feature-gated compile has NOT been re-run here.
It is pending, and the restored text is identical to the known-good tree.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(serve): put the VRAM doc comments back on their own functions

My restore of the low-VRAM hook in a3085af spliced the function in
between `gpu_vram_usage`'s doc comment and `gpu_vram_usage` itself. The
result compiles, so CI stayed green, but the amd-smi/sysfs fallback
prose ended up describing `simulated_low_vram_usage`, `gpu_vram_usage`
was left undocumented, and the hook's own twelve-line comment --
explaining why the seam exists and that production builds never see it
-- was dropped entirely.

Restores both comments to the functions they describe, matching 70d6aa1
(the last commit before the merge that dropped the hook) byte for byte.
No executable code changes: `gpu_vram_usage`'s body is moved verbatim,
and moving a top-level item is inert.

Verified without a build host: the region is byte-identical to
70d6aa1, the diff touches no statement, `rustfmt --check --edition
2024` parses the file and reports no formatting change, and the three
intra-doc link targets (`AUTO_FREE_VRAM_FRACTION`,
`vram_capacity_is_meaningful`, `read_drm_vram_usage`) all still resolve.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): address review on the amd-smi KFD pre-flight

Blocking items from juhovainio's review and the automated one, on top of
this round's merge of main.

Unbounded output (the one that reinstates a defect ROCm#526 already fixed):
`amd_smi_json_with` pasted a failing `amd-smi`'s whole stdout/stderr into
the error. `rocmd` renders that into its snapshot note and the GPU-metrics
watcher copies the note into rewrite-the-whole-file JSONL logs on every
poll -- the path that produced a 151K-character log line. Bounded with
`MAX_QUOTED_OUTPUT_CHARS = 2000` through `examine::truncate_to_chars`,
deliberately the same name, value and helper `fix/rocmai-448-drain-child-
pipes` uses one layer down, so converging the two is a deletion rather
than a behaviour change. No new abstraction is introduced here.

WSL wave-through: `is_wsl_host()` alone repeated the mistake `b4f98bc2`
corrected in the dashboard pre-flight -- a WSL host whose ROCDXG plumbing
is not ready is one where amd-smi still hangs. Now
`is_wsl_host() && has_usable_amd_gpu()`, the same conjunction
`dash.rs::gpu_reachable_for_preflight` feeds the dashboard, so the
cross-surface claim at `rocm-dash-collectors/src/amd_smi.rs` stays true.
`&&` short-circuits, so bare metal never pays for the extra probe.

Hardware-probe routing: main's `hardware_root` contract says every
`/dev/kfd` read goes through `host_path`. The gate now does, with the real
node keeping a veto exactly as the dash pre-flight does, and the module
doc lists it.

rocmd kept its own unprotected `run_command_with_timeout` for
`run_rocm_capture_for_paths`, reachable from the MCP bridge: the same
pipe-fill deadlock fixed everywhere else, and its timeout error quoted the
child's whole output. Deleted for the shared helper.

Also: `amd_smi_preflight_ok` was dead public API describing a call graph
the code did not have -- deleted, docs repointed; three doc comments that
contradicted the rewritten `run_command_with_timeout` corrected; the gate
doc and decline message narrowed to what a read-open actually detects
(inaccessible, not unhealthy); `drain_on_thread` uses `thread::Builder` so
a thread-creation failure cannot panic out of the daemon; the e2e KNOWN
GAP note corrected where its premises no longer hold.

New tests, each confirmed to fail without its fix:
- a_failing_amd_smi_cannot_paste_its_whole_output_into_the_error
- the_public_amd_smi_entry_point_gates_on_the_real_kfd_device
- the_kfd_gate_honours_a_simulated_root_but_keeps_the_real_device_veto
- the_kfd_preflight_waves_through_wsl_hosts_only_when_the_gpu_is_reachable

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(core): make the gate-wiring test non-vacuous under a simulated host root

On a machine that has a readable /dev/kfd the comparison alone is
vacuous: both the gate and a wrapper passing some other readable path
say "allow". Under an e2e-test-hooks build an empty simulated root has
no dev/kfd, so the gate declines on every host while the obvious mutant
(Path::new("/")) re-roots to the root directory and opens.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(core): serialise the device-gate tests on the simulated-root lock

The gate now resolves its device through host_path, so an e2e-test-hooks
build in which one test sets ROCM_CLI_TEST_HOST_ROOT re-roots another
test's tempdir device out from under it. Same lock fix.rs already holds
for the same reason.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): hoist a test const above the statements clippy objects to

`clippy::items_after_statements` rejects `const PAYLOAD_CHARS` declared
part-way through the test body: items exist from the start of the scope
regardless, so declaring one mid-function reads as if it did not.

Caught by the full workspace clippy run, which the agent that wrote this
test did not reach before stalling.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(core): spell the host-root lock so the env-mutation guard sees it

`xtask`'s env-mutation contract failed on this test:

  crates/rocm-core/src/lib.rs:6757: RestoredEnvVar::set

The lock was already held -- `host_root_guard()` on the line above takes
HOST_ROOT_TEST_LOCK -- but that guard matches TEXT, recognising a
serializer by the `_TEST_LOCK` suffix or the name `ScopedTestEnv`. The
helper's name carries neither, so a correctly serialized test read as an
unguarded mutation. This is the only test here that both takes the lock
and calls `RestoredEnvVar::set`, which is why the other three using the
helper pass.

Acquire the same lock by name instead, with a comment so the next person
does not "simplify" it back to the helper.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): make the off-Linux KFD wave-through a const fn

The Windows lane builds with `-D warnings` and runs clippy, so
`clippy::missing_const_for_fn` turned the `cfg(not(target_os = "linux"))`
stub into a hard compile error there. The Linux body opens a device and
cannot be const, so the lint only ever fires on the other branch of the
pair -- which no Linux-side gate can catch.

error: this could be a `const fn`
  --> crates\rocm-core\src\lib.rs:4619:1
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): restore the io::Read import the lib.rs extraction took with it

Merging main's `process.rs`/`net.rs`/`paths.rs` extraction (ROCm#599) into this
branch is a textbook silent semantic conflict: main deleted `use std::io::Read`
from `lib.rs` along with the code that needed it, this branch added
`drain_on_thread<R: Read + ...>` to the same file, and the two edits never
touched adjacent lines -- so git merged them without a marker and left a
module that cannot compile.

    error[E0405]: cannot find trait `Read` in this scope
        --> crates/rocm-core/src/lib.rs:2869:23

`run_command_with_timeout` and its drain threads are not `cfg`-gated, so the
import is live on every target and cannot trip the Windows lane's
`-D warnings` unused-import check.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(core): plant the fake amd-smi through the ETXTBSY-safe writer

Both new amd-smi tests wrote their stand-in binary with `fs::write` plus
`set_permissions` and then executed it. Main's ROCm#649 landed
`fake_executable::write_fake_executable` precisely because that pattern races
every other test thread in the same process: a parallel `fork` inherits the
writer's fd and the kernel refuses to `exec` a file anyone still holds open
for writing.

The merge brought the helper in but could not retrofit these call sites, and
the race duly fired on a 112-core run:

    ---- tests::the_kfd_gate_stops_amd_smi_before_it_is_spawned stdout ----
    a readable device must let the launch through:
      0: failed to spawn child process
      1: Text file busy (os error 26)

Routing both plants through the helper makes the gate test deterministic
again; `cargo test -p rocm-core --lib` now passes three runs in a row where
it failed one in two before.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(rocmd): route the sandbox timeout through the consolidated helper

Addresses review on PR ROCm#447.

`apps/rocmd/src/sandbox.rs::run_process_with_timeout` was a third copy of the
poll-and-kill loop from the same split (e760d9a) that produced the amd-smi
ones, and carried the same defect: it polled `try_wait` with both pipes
undrained, so a sandbox child that filled the 64 KiB pipe buffer blocked
mid-write and was then reported as having exceeded its timeout. Route it
through `rocm_core::run_command_with_timeout`, which drains both pipes on their
own threads and reaps a killed child on a detached thread.

Bound the gate's own `/dev/kfd` open so `kfd_readable`'s safety no longer rests
on the unverifiable assumption that `kfd_open` always returns promptly. An open
that overruns `KFD_OPEN_TIMEOUT` reads as "not readable", which is the declining
answer, so the unanticipated case fails in the safe direction. Pinned by a test
against a FIFO with no writer, whose `open` really does block in the kernel.

Deduplicate the stderr-then-stdout precedence `amd_smi_json_with` had inlined
into `command_failure_detail_or_else`, which `command_failure_detail` now
delegates to; only the both-blank fallback differs between the two callers.

Document the `serve` worst-case probe latency (two sequential 10s bounds,
replacing two unbounded `.output()` calls) and why `rocmd`'s 2s bound on the
same helper is deliberately shorter.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): create the blocking-open fixture without unsafe

The workspace lints deny `unsafe_code`, so `libc::mkfifo` cannot be called
directly; shell out to `mkfifo(1)` and skip when it is absent. Also narrow
sandbox.rs's `std::thread` import to `cfg(test)`: the Linux timeout no longer
runs its own sleep-poll loop.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(core): watch the blocking-open probe so an unbounded mutant fails, not hangs

Calling the probe inline meant a mutant that drops the timeout from
`device_readable_within` would block forever on the FIFO and hang the test
binary — a CI timeout rather than a readable failure. Run it on a watched
thread with a much looser deadline so that mutant reports the cause instead.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(core): move the fifo path into the probe thread instead of cloning it

`clippy::redundant_clone`: the fixture path has no use after the spawn,
so the clone was dead. The thread can take it by move.

Caught by CI, not locally: the devbox run that cleared this branch was
made against the merge commit, before this test existed.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): reach both sides of the KFD pre-flight from a scenario

The `/dev/kfd` pre-flight refuses to spawn `amd-smi` when the device
cannot be opened. The per-PR E2E lane has no `/dev/kfd`, so every
scenario that plants a stub `amd-smi` was refused before the stub ran:
examine-24/25/26 (the pipe-drain regression tests from ROCm#526) started
asserting against a decline instead of the behaviour they are about.

`ROCM_CLI_TEST_HOST_ROOT` cannot open the gate. `kfd_readable_with`
keeps the real node's veto on purpose, so a simulated root can make the
gate decline on a machine that has a GPU but can never make it allow on
one that does not -- the invariant that stops a simulated root launching
`amd-smi` where there is nothing to talk to.

So add `ROCM_E2E_KFD_DEVICE`, honoured only by an `e2e-test-hooks`
build, which moves the device the gate opens. It is an override, not a
bypass: the probe still runs, against the named path, and a shipped
build has no override at all. Tagging the three scenarios `@requires-gpu`
was the alternative and was rejected -- examine.feature says in terms
that they are built to need no GPU so they run per PR, and a GPU tag
would strand ROCm#526's regression tests on a nightly lane.

The seam also makes the declined branch observable, which AGENTS.md §3
asks for and the PR had only unit tests for. examine-29 points the
pre-flight at an absent device and asserts the bridge snapshot explains
itself, names that device, and -- the load-bearing half -- that the stub
`amd-smi`, which would have answered, never ran.

Signed-off-by: Roman <roman.sirokov@amd.com>

* test(e2e): declare the KFD pre-flight scenario last in examine.feature

`feature_naming` requires the per-feature scenario index to follow
declaration order, so a new examine-29 cannot sit beside examine-26.

Signed-off-by: Roman <roman.sirokov@amd.com>

* docs(serve): stop AMD_SMI_SERVE_PROBE_TIMEOUT calling itself serve-only

`gpu_vram_usage_amd_smi` is also reached from `host_accelerator_memory`,
and so from the `--model` readiness path, not only from `serve`. The
budget is still the right one there -- same question, same one-shot
cost -- but the doc said something untrue about where it is spent.

Signed-off-by: Roman <roman.sirokov@amd.com>

* fix(dash): bound the dashboard collector's /dev/kfd probe too

`rocm-dash-collectors` runs its own pre-flight before spawning `amd-smi`
for the dashboard's GPU metrics, on the same `/dev/kfd` and for the same
reason as the CLI launch sites. It used a plain
`OpenOptions::open(..).is_ok()`, so it stayed the one launch site that a
wedged device could still block indefinitely — the exact hazard the rest
of EAI-8457 removes.

It now shares `rocm_core::device_readable_bounded`, which carries the 2s
bound and the in-flight probe budget with it, instead of restating them.
A second copy of that primitive would be free to drift from the gate it
belongs to.

That needs a new first-party edge, `rocm-dash-collectors -> rocm-core`,
so `ALLOWLIST` gains it and the module doc records why. The invariant
that forbids this edge binds `rocm-dash-tui`, the presentation layer,
which makes no such call and keeps the stricter rule.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(amd-smi): document the KFD seam and examine-31, and state the gate's real scope

docs/testing.md had no entry for the `ROCM_E2E_KFD_DEVICE` seam or scenario
examine-31, which the repo guidelines ask for in the same change.

The gate declines on a failed `open` as well as on one that never returns, and
the first class cannot cause the D-state hang the rationale described. Keep the
fail-safe behaviour -- narrowing it would mean deciding per errno which kernel
states are safe to launch into, an unverifiable list of exactly the kind the
bounded open exists to avoid resting on -- and say so, including what the
broader refusal costs `detect_gpu_count` and what answers in its place.

Prune the doc comments to the invariants: drop the review-round narration, the
"this used to be X" archaeology and the two citations of a commit hash that does
not exist in this repository.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(dash,core): keep the bounded KFD probe off the async worker

Review round 3 on ROCm#447.

- `AmdSmiCollector::new` reached `device_accessible` -> the bounded open
  from an `async fn`, parking a tokio worker for up to the 2s
  `KFD_OPEN_TIMEOUT` per probe — ~4s under a simulated host root, where
  there are two. The pre-flight now runs through
  `preflight_passes_off_thread`, which short-circuits the two non-device
  branches before the thread hop and puts the rest on
  `spawn_blocking`. A `JoinError` reads as "not readable", matching
  `device_readable_within_budget`: every branch that cannot get an
  answer declines rather than opening on the caller's thread.

- The timeout and budget tests returned `Ok(())` when `mkfifo(1)` was
  missing, and on WSL, which is indistinguishable from a pass for a test
  that asserted nothing. Both now announce the skip through a shared
  `skip_notice`.

- `command_failure_detail_or_else` is `pub(crate)`; its only caller
  outside its module is `amd_smi_json`, in this same crate.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(core,dash): stop claiming the dash crates have no rocm-core edge

This PR added `rocm-dash-collectors -> rocm-core` so the dashboard's
`amd-smi` launch could share the bounded `/dev/kfd` probe, which made six
doc comments false as written. The invariant that survives binds only the
presentation layer: `rocm-dash-tui` has no edge to `rocm-core` and
`crate_edges.rs` still forbids it one. The collectors crate does have one,
and says what for.

`runner.rs` and the collector's own docs additionally explained the
threading of `gpu_reachable`/`host_kfd_device` by the missing dependency,
which is no longer the reason: the verdict is threaded through because
`serve`, `examine` and the dashboard must not reach it independently, not
because the collector could not compute it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(amd-smi): fail the build when a launch site skips the KFD pre-flight

The pre-flight protects nothing unless every `amd-smi` launch goes through
it, and nothing in the tree failed when one stopped: reverting either
`gpu_policy.rs` probe to `Command::new(resolve_amd_smi_binary())` left the
whole suite green, because the regression is the absence of a call and no
behavioural test can observe that.

So guard the ingredient. The managed SDK ships `amd-smi` inside the runtime
wheel's bin directory rather than on `PATH`, so `resolve_amd_smi_binary` is
the only way to find a binary to launch; a static contract test now fails
any file outside a two-entry, justified allowlist that calls it.
Bidirectional like the crate-edge allowlist, so an exemption cannot outlive
its reason. Verified by mutation: a raw spawn in `detect_gpu_count` fails
`only_preflighted_sites_resolve_the_amd_smi_binary` by name.

It is a source-text scan, not a call-graph analysis, and the module doc says
so rather than overclaiming.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): say why the amd-smi probe declined, and pin the quoted latencies

Both serve-path probes return `Option`, so a decline had nowhere to go and
was dropped with `.ok()?`. That silenced the two cases a user wants: an
`amd-smi` that ran and failed, and a false decline when the 2s open bound
trips on a slow but healthy host. `amd_smi_probe_json` now logs the full
error chain at `debug` -- not `warn`, because on a GPU-less host it is the
expected answer on every `serve` -- so "why did `--gpu auto` not rank my
GPUs?" is answerable from `RUST_LOG=debug`.

The gate's 2s bound and 8-slot budget are restated as literal numbers in
five places, two of them as derived figures (caller timeout + 2s), none of
which can see the constant. A tripwire test now fails on a change to either
and names every restatement to update.

Also state plainly, in `kfd_readable`'s docs and `docs/testing.md`, what the
gate is not: it proves only that `open(/dev/kfd)` returned, while the hang
is attributed to amd-smi's later ioctls, and no unhealthy-driver run backs
it up. It is a mitigation heuristic. And the decline is broader than the
ticket -- an absent node and EACCES decline too, and `detect_gpu_count` has
no sysfs fallback.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(dash): cfg-split device_accessible so the Windows lane builds

`device_readable_bounded` waves through with a literal off Linux, which
makes this wrapper `const`-able there, and `clippy::missing_const_for_fn`
is a hard error on the Windows lane's `-D warnings` build. Split into a
Linux body that opens the device and a non-Linux `const fn`, the same
shape `kfd_readable` and `device_readable_bounded` already use.

Caught by CI's Windows job, not locally: the lint only fires on the
branch the Mac and the Linux box never compile. Verified this time by
running clippy over every crate that does build on Darwin.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(amd-smi): keep the WSL short-circuit, and stop the docs saying debug

Review round on ROCm#447.

- Extracting the WSL condition into `wsl_wave_through` turned a
  short-circuit into two eager argument evaluations, so
  `has_usable_amd_gpu` ran on every Linux launch instead of on WSL only.
  The probe is passed unevaluated now, which restores the original
  behaviour and keeps the condition testable. The comment that still
  claimed `&&` short-circuited was describing code that no longer did.

- `docs/architecture.md` and `docs/testing.md` still said the discarded
  decline is logged at `debug`; it is `info!`, deliberately, because the
  default filter is `warn,rocm=info`. Both now say so and why.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(amd-smi): stop device_readable_bounded waving through off Linux

The previous round made the off-Linux branch return `true`, on the
reasoning that it should mirror `kfd_readable`'s wave-through as its doc
claimed. That was the wrong half to fix. The two answer different
questions: `kfd_readable` gates this crate's own `amd-smi` launch, which
off Linux has no `/dev/kfd` to reason about; `device_readable_bounded` is
the shared probe `rocm-dash-collectors` calls for a pre-flight that is
expected to decline a missing device on every platform.

Windows CI caught it: `missing_kfd_and_no_reachability_fails_preflight`
and `real_kfd_keeps_a_veto_over_the_host_root_view` both went green while
losing the veto they exist to protect. The off-Linux branch is a direct
open again and the doc now says why it does not wave through.

That also removes the `const`-able body, so the `missing_const_for_fn`
split added for it is gone from both crates. Both tests verified passing
on Darwin, which compiles the same non-Linux branch Windows does.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(amd-smi): put the probe sentence in the right bullet, and stop two claims overreaching

Review round on ROCm#447.

- The `AMD_SMI_SERVE_PROBE_TIMEOUT` sentence had drifted into
  `architecture.md`'s `freeform_planner.rs` bullet across several merge
  resolutions. Both probes live in `gpu_policy.rs`; it sits there now,
  next to `detect_gpu_count`.

- `amd_smi.rs` said that losing the amd-smi count leaves `--gpu auto`
  with no selection. It does not: `effective_gpu_count` falls through to
  the DRM sysfs VRAM rows, which is what `docs/testing.md` already said.
  The code was right and this doc was wrong, so the doc now matches and
  names the function.

- `docs/testing.md` claimed unit tests cover the rewritten call sites.
  They cover what those sites do with a *missing count*
  (`effective_gpu_count_prefers_amd_smi_then_falls_back_to_vram_rows`);
  that they route through the chokepoint at all is guarded only by the
  xtask contract scan. Both halves are now stated separately.

- examine-31 was `@requires-os:linux`, but WSL2 reports that os_family
  and `kfd_readable` waves through there on a reachable GPU, so the
  decline it asserts never happens. Now `@requires-bare-metal` too.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman <roman.sirokov@amd.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants