Skip to content

feat(examine): probe Vulkan instead of inferring it from the gfx table - #1036

Open
r0x0r wants to merge 100 commits into
ROCMAI-550-vulkan-support-messagingfrom
split/vulkan-probe
Open

r0x0r wants to merge 100 commits into
ROCMAI-550-vulkan-support-messagingfrom
split/vulkan-probe

Conversation

@r0x0r

@r0x0r r0x0r commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Split out of #678 on review. Stacked on it — the base is ROCMAI-550-vulkan-support-messaging, because the GpuSupportLevel classifier this extends is that PR's work. Review #678 first; this diff is only what Vulkan adds on top (~1.4k lines).

Symptom

rocm examine told a gfx90c APU it could run models through Lemonade's Vulkan backend on the strength of the gfx table alone. Nothing had looked for a driver. On a host with none, that is an instruction the user cannot carry out — the same class of problem as the static ROCm claim #678 fixes, one level down.

Change

Probe, rather than infer:

  • ICD manifests (Linux directories, the Windows Khronos registry key)
  • vulkaninfo --summary where it exists

A fourth support level, NoGpuBackend, carries the confirmed answer, and a vulkan: line prints what was found.

Where the probe refuses to guess

Absent is reserved for a probe that found something and ruled it out, and it always takes vulkaninfo's own answer: a vulkaninfo that ran and enumerated nothing, an enumeration of only software devices with no GPU ICD beside it, or a software-only ICD list that vulkaninfo did not contradict. Neither manifest source is exhaustive enough to deny a driver on its own:

  • the Linux ICD search is a fixed directory list and ignores VK_ICD_FILENAMES / VK_DRIVER_FILES
  • Windows reads only the legacy Khronos Drivers key, which current AMD drivers need not populate
  • vulkaninfo is frequently not installed on either

Anything short of that yields Inconclusive — which reaches the user as a distinct VulkanUnconfirmed support level, not as a confirmed absence. Inferring absence from an unread source would tell a host with a working driver to install one it already has. NoGpuBackend renders as a confirmed fact, so it has to be earned.

A software rasteriser never counts as a GPU path — lavapipe/llvmpipe never reach Available. But neither does it deny a GPU path a manifest asserts: a host carrying radeon_icd.x86_64.json whose vulkaninfo enumerates only llvmpipe (the ordinary shape of a missing render-group membership) is DriverPresentDevicesUnverified, since "install a Vulkan driver" would contradict the icds field of the same report.

Scenarios

examine-56 through examine-60 cover the answers on the ungated mock lane, which needs neither a Renoir APU nor a Vulkan driver — only the KFD node, ICD state and vulkaninfo stub the harness can plant.

scenario host it plants status reached
examine-56 a GPU ICD and an enumerated AMD device Available
examine-57 a vulkaninfo that exits 0 and reports nothing Absent
examine-58 only a software rasteriser (lavapipe ICD + llvmpipe device) Absent
examine-59 no ICD, no device, and no vulkaninfo on PATH Inconclusive
examine-60 a GPU ICD whose vulkaninfo enumerates only llvmpipe DriverPresentDevicesUnverified

examine-58 and examine-60 plant the same enumeration and differ only in which manifest sits beside it, so an implementation that reads the devices before the manifests fails exactly one of the pair.

Falsified: both negative scenarios fail when gpu_support_level is made to ignore the probe; examine-60 fails when the software-only early return is made to ignore the ICD evidence.

Known limitations

1. A non-AMD-only enumeration still routes to a Vulkan path. A successful enumeration that returns only non-AMD hardware (an NVIDIA box, say) still falls through to "driver present, devices unverified" rather than counting as evidence against an AMD Vulkan path. Raised in review on #678; not fixed here, and pinned by a test so it is a known state rather than an accident.

2. rocm install sdk, rocm engines and doctor info-20 still promise Vulkan without consulting the probe. rocm examine is the only surface this PR routes through the probe. therock::family::unsupported_host_gpu_message — the error rocm install sdk raises for a detected GPU with no ROCm build — still ends unconditionally with "the Vulkan backend can still be used. Try rocm serve <model> --engine lemonade". That is the same sentence, said on the same trigger condition (an AMD GPU detected, no ROCm build), that this PR's module doc names as the motivating defect and that render.rs's examine_render_does_not_promise_vulkan_on_a_host_that_has_none now pins out of examine. rocm engines and the MCP engines tool pass VulkanStatus::Unknown, which maps to VulkanOnly, so the same host can get two different answers in one session.

Scoped out rather than fixed, deliberately: family.rs is a pure-function error builder reached before any host probe runs, so routing it through the probe means giving it a VulkanStatus parameter and threading one down from resolve_family's callers — a change to the install path's call graph rather than to a string, and one that belongs with the rocm engines / doctor info-20 surfaces that also still speak from the gfx table. rocm engines additionally has a cost argument against it: spawning vulkaninfo (and PowerShell on Windows) for a plain listing command is a per-invocation subprocess the listing does not otherwise need. No follow-up ticket is filed for this yet — it is a disclosed scope-out, not tracked work. rocm examine remains the surface that answers from evidence, and this PR does not claim otherwise.

3. Confirmed absence is narrower than "no driver is installed". VulkanStatus::Absent (and so NoGpuBackend) always needs vulkaninfo to have answered. On a real driverless Linux host vulkaninfo --summary prints Found no drivers! and exits 1, and capture_optional_command collapses non-zero exit, a missing binary and a timeout into one None — so that host classifies as Inconclusive, not Absent. examine-57 plants a vulkaninfo that exits 0 with an empty device list, which is the shape that does earn the confirmed absence, so that scenario mirrors the implementation rather than the most common real tool behaviour.

The other route to Absent is not affected and is genuinely reachable: a host whose evidence contains only a software rasteriser is ruled out on the evidence it did produce, which is what examine-58 covers.

Treating exit 1 plus the loader message as an answer was considered and rejected for now. It means threading exit status and stderr out of capture_optional_command, which is shared well beyond the Vulkan probe, and it would make a confirmed "no GPU backend" depend on matching Found no drivers! — loader text that is not a stable contract. A vulkaninfo that exits 1 for any other reason would then be promoted into exactly the unearned absence claim this PR exists to stop making. Inconclusive is the conservative classification, and the line it renders already names vulkaninfo as the step that settles it.

Cost

The probe runs once per full ExamineSummary::gather(), so every caller pays one vulkaninfo (and on Windows one extra PowerShell start), each bounded by the shared 1.5 s OPTIONAL_COMMAND_TIMEOUT. It is not on any periodic path: rocmd's watcher tick calls gather_gpu_snapshot_for_config only, which takes the amd-smi route and never reaches the probe, and build_bridge_snapshot (which does gather) is reached only from rocmd mcp and rocmd CLI request handling. rocm mcp-call examine|doctor gathers twice and so pays up to two probes — a pre-existing double gather, now with a larger constant.

Verification

Linux MI300X: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, cargo test -p rocm-core, cargo test -p rocm --bin rocm, cargo test -p e2e-cucumber --test feature_naming, cargo xtask check-architecture-doc / check-crate-edges / check-module-size, and cargo xtask e2e -- -n '(examine-56|examine-57|examine-58|examine-59|examine-60)'.

Not verified anywhere: the Windows registry branch of the probe end to end. Neither this Mac nor the Linux build box can build the MSVC target (ring/aws-lc-sys need a real Windows toolchain), so CI's windows-build-and-test is the only check on the live registry read.

The pure seam around it is covered on every platform: windows_icds_from_capture turns the capture's Option<String> into VulkanEvidence<String>, and its tests drive the None case (missing powershell, non-zero exit, or timeout — capture_optional_command collapses all three) alongside the empty-key and populated-key cases. What remains unexercised is the registry script itself against a real registry, and the PowerShell spawn's own cold-start behaviour; closing that needs an injectable runner on the Windows job, deferred to a follow-up with a Windows dev environment.

Support levels this adds

level evidence what the user is told
VulkanOnly a GPU ICD, or an AMD device enumerated Vulkan runs here
VulkanUnconfirmed probed, nothing found, and at least one of the two sources never answered what was looked for, and how to settle it
NoGpuBackend vulkaninfo answered and a GPU path was ruled out confirmed: install a driver (unless the host's own driver_status already shows the AMD display driver, in which case repairing it is the step)

VulkanUnconfirmed exists because NoGpuBackend renders as a confirmed fact and
is serialized. On Windows the Khronos Drivers key is legacy and vulkaninfo is
rarely on PATH, so a working Adrenalin laptop can leave both sources empty —
telling it "no GPU backend" would be worse than the static claim this change
replaces.

examine-59 covers that row; examine-57 cannot, since it plants a
vulkaninfo that answers (exit 0, empty list), which is a confirmed absence.

r0x0r and others added 30 commits October 14, 2026 14:53
…tic ROCm claim

ROCMAI-550: on a GPU with no ROCm/HIP build (e.g. gfx90c), `rocm examine` and
`rocm engines` kept claiming the embedded Lemonade engine runs "ROCm llama.cpp
backend" even though it actually falls back to Vulkan. `rocm install sdk`'s
refusal for such a GPU didn't name the actual next step, and neither `rocm
serve`'s deployment summary nor `rocm services list` showed which backend/
device an engine actually selected.

- add `GpuSupportLevel` (RocmHip/VulkanOnly/Unknown) in rocm-core, derived from
  the existing `normalize_therock_family`/`gfx_target_has_no_rocm_build`
  primitives
- make the Lemonade engine note in `rocm examine` and `rocm engines`
  GPU-aware instead of a hardcoded ROCm claim
- `rocm install sdk`'s unsupported-GPU errors now suggest `rocm serve <model>`
- surface the engine's actual backend/device (read live via the existing
  `Healthcheck` RPC, never guessed) in `rocm serve`'s deployment summary and
  `rocm services list`

Most of the ticket's `rocm examine`/`install sdk` complaints for gfx90c were
already fixed by #648 (ROCMAI-542); this covers what was left.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
CodeQL flagged two assertions as cleartext logging of `api_key`. Those
two use `base_summary()`, whose key is `None`, so the literal instances
are false positives — but the pattern is not. `render_summary` prints
the key by design, and `api_key_client_config_shown_only_when_present`
sets a real one, so a failure there copied `endpoint-secret` into CI
logs, which are retained and widely readable.

Assert on the real text, print a redacted copy. Covers all twelve sites
that interpolated a rendered summary, not just the two CodeQL picked.
Production output is unchanged: the key is still shown once, and the
test pinning that still asserts the exact string.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…l is produced

The gfx90c refusals in `therock.rs` gained a `rocm serve <model>` remediation
sentence, a second user-observable output change in this PR with no scenario
behind it. AGENTS.md §3 wants remediation advice asserted where the message is
really printed, not only against the formatter.

runtime-19 already opens the one door of the two a lane can reach -- naming the
target with `--family` -- so the assertion goes there. The detected-host door
needs a Vega APU no runner has and stays unit-covered, as its existing comment
already records.

Also names the codex-bridge engine inventory at both the static table and the
bridge helper as a deliberate scope boundary, instead of only in
`lemonade_engine_note`'s doc comment where a reader of the table would miss it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`unsupported_requested_target_message` is raised from the `--family` and
`ROCM_CLI_THEROCK_FAMILY` branches at the top of `resolve_family`, before
any host probe runs, so the rejected target says nothing about the GPU in
this machine. It had borrowed its sibling's closing sentence promising
that `rocm serve <model>` runs on Lemonade's Vulkan backend -- true in
`unsupported_host_gpu_message`, which is only reached once host detection
itself returns a target with no ROCm build, but false here: a stale env
value naming gfx90c refuses just the same on a host whose own GPU is
fully ROCm-capable, and there serving would run on ROCm/HIP.

Point at `rocm examine` instead. It is true on every host, it is still a
next step, and it is the command this PR just taught to report the real
support level.

runtime-19 was asserting the wrong half of this: it drives `--family
gfx90c`, so it was pinning the Vulkan-serving claim green on exactly the
ROCm-capable runners where it was false. It now asserts the examine
pointer and the negative.

Also route `ExamineSummary::rocm_support_line` through `gpu_support_level`
rather than re-deriving the same classification, and document the serve
summary's `device:` row in README and the manual-testing GPU check.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…tory

The codex-bridge engine inventory kept the unconditional "with ROCm
llama.cpp backend" wording that ROCMAI-550 removed from `rocm examine`
and `rocm engines`, so a bridge client still rendered a ROCm claim to
the operator on a host that cannot use it.

Both call sites can answer the question: the snapshot already gathers
the ExamineSummary carrying the two inputs, and the `engines` tool takes
the same cheap host probe `rocm engines` uses. Only the `summary` string
changes; the field set and the bridge's JSON shape are untouched.

The dashboard TUI pane and rocmd's own catalog keep the static wording
deliberately -- the pane has no probe behind it, and rocmd answers for
whichever host a client asks about, not the one it runs on.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Resolves the one conflict, apps/rocm/src/main.rs, by taking main's
post-#667 file (chat/MCP-bridge extraction removed 185 functions) and
re-applying this branch's 23 hunks onto it. One hunk follows its
function: `run_internal_mcp_call` now lives in apps/rocm/src/chat.rs, so
the `engines` tool's call to `builtin_codex_bridge_engine_inventory`
is updated there and `host_gpu_support_level` is imported from the crate
root.

Verified by construction: the `fn` name sets of main, this branch and the
resolution comm out to resolved == main's set plus exactly the five
functions this branch adds (`fetch_service_device`,
`host_gpu_support_level`, `lemonade_engine_note` and the two new examine
tests), with nothing of main's dropped.

Also converts this branch's one new non-ASCII production literal to ASCII
for the output contract #683 added, which this branch predates: the em
dash in `unsupported_host_gpu_message`'s "Try `rocm serve <model>`"
sentence becomes a hyphen, matching how #683 converted the rest of the
tree. `xtask`'s `every_literal_that_can_reach_a_redirectable_sink_is_ascii`
named that literal and only that one; everything else this branch adds
with non-ASCII is a comment or lives under the exempt `tests/` prefix.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
… and the bridge

`append_examine_engine_inventory` got tests for both support levels when
ROCMAI-550 landed; the other two surfaces carrying the same correction did
not. `rocm engines` and the codex-bridge inventory (including its `engines`
MCP tool) could have gone back to asserting ROCm support on a Vulkan-only
host with nothing red.

Both were untestable for the same reason: the branch's input is a live sysfs
probe, and no CI runner has a GPU that is detected *and* confirmed to have no
ROCm build. So each surface now takes the level the way
`append_examine_engine_inventory` already did -- the caller probes once, the
renderer is given the answer. `render_engine_inventory_text_for_host` and
`internal_mcp_engines_tool_result` are those renderers; neither changes what
is printed.

The three copies of the lemonade substitution collapse into
`engine_inventory_note`, so the surfaces cannot drift. The new tests drive
the renderers, not that helper: asserting on it would restate the condition
rather than prove a surface prints it.

`gpu_support_level_for_host_gpu` reads the two `HostGpuSummary` fields once
instead of at each call site. They are same-typed positional arguments, so a
transposed call site would have classified a Vulkan-only host as `RocmHip`
and still compiled; it now has a test.

Also drops the dangling `GFX_TARGETS_WITHOUT_ROCM_SUPPORT` reference from the
`GpuSupportLevel` doc comment -- no such constant exists in `diagnose.rs` or
anywhere else, so the doc pointed readers at nothing.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Resolves the conflicts with main's `main.rs` modularization (Phase 6f,
ROCMAI-208/213/214/215) and addresses the outstanding review findings in
the same pass, because several of the fixes land in code this merge had
to relocate.

Conflict resolution. Every large hunk had an empty `origin/main` side:
main moved the code out of `main.rs`, it did not delete it. The HEAD
block was dropped and this branch's additions re-applied in the module
main moved them to.

- `examine_human_report`, `render_engine_inventory_text_with_paths` /
  `_for_host` and `append_examine_engine_inventory` ->
  `examine_report_text.rs`, with the seven `append_examine_engine_inventory`
  test call sites and the five relocated inventory tests.
- `render_services_text`'s `device:` row -> `services_text.rs`, with the
  `!rendered.contains("device:")` assertion that pins the `"unknown"` case.
- `fetch_service_device` -> `engine_ipc.rs`, beside the `engine_request`
  it calls; `serve_cmd.rs` imports it from there.
- The codex-bridge inventory stayed in `main.rs`, so its three tests
  went back into `main.rs`'s test module.

All 21 test functions this branch adds are present and run; `engine_inventory()`
now carries a third engine (`aim`) from main and the renderers handle it
unchanged. No scenario ids were added by this branch (it only appends
`Then` steps to serve-05, serve-07 and runtime-19), so nothing needed
renumbering.

Review findings.

- `unsupported_host_gpu_message` pins `--engine lemonade` instead of
  suggesting a bare `rocm serve <model>`: a configured `default_engine`
  out-ranks the host preference in `select_serve_engine`, so the bare
  command could attempt vLLM on a GPU with no ROCm build.
- `lemonade_engine_note` now matches `GpuSupportLevel` exhaustively.
  `Unknown` no longer inherits the ROCm claim through a catch-all; it
  asserts no backend and points at `rocm examine`.
- `fetch_service_device` hands the engine the service's 0600 key file via
  `ROCM_SERVE_API_KEY_FILE`, so a public / `--require-api-key` service
  does not 401 its own healthcheck and lose the row.
- `rocm services list` probes under one aggregate budget
  (`fetch_service_devices`), so the row cannot turn a local-only render
  into an unbounded wait.
- `engines/vllm` and `engines/aim` report the device in the same
  `<backend> gpu [<n>]` shape `engines/lemonade` does, instead of the
  literal `"rocm_gpu"` that README.md and `docs/manual-testing.md` never
  described.
- The `services list` e2e step parses the device value, checks it against
  the engine that served, and cross-checks it with the `rocm serve`
  deployment summary.

Verified on the Linux MI300X box: `cargo fmt --all --check`, `cargo
clippy --workspace --all-targets -- -D warnings`, `cargo clippy --locked
-p e2e-cucumber --test e2e -- -D warnings` and `cargo test --workspace
--all-targets --no-fail-fast` all rc 0, 4823 tests passed / 0 failed.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…y tier

Review round 3 on #678.

- `rocm examine` now prints a `rocm_support` line for RocmHip and Unknown as
  well as VulkanOnly. The Unknown-tier lemonade note sends the user to
  `rocm examine`, and examine printed nothing for exactly those hosts, so the
  remediation named a command that could not clear the condition. The note is
  also reworded to promise the support level, which is what examine delivers,
  rather than the backend, which only a launch can settle.
- The `device:` row carries the GPU name. ROCMAI-550 asks for the backend
  "and the device name"; on the ticket's own Windows gfx90c host no ordinal is
  recorded, so the row rendered a bare `vulkan gpu` - a backend and a noun
  naming nothing. All three engines now record `gpu_name` at launch and format
  through the shared `rocm_engine_protocol::format_reported_device`, which
  drops the `gpu` noun when there is no ordinal behind it.
- `fetch_service_devices` splits its budget loop into `collect_service_devices`
  with an injected clock and probe, so the budget-exhausted exit is tested -
  including that later services are not probed at all. The ~18s worst case
  (budget plus one in-flight probe) is documented on
  `SERVICE_DEVICE_PROBE_BUDGET`, on `render_services_text` and in README.
- The e2e `device:` step polls for the row instead of panicking on the first
  look: lemonade reports `unknown` until its own inference probe returns, so a
  correct implementation legitimately has no row for a short window.
- AIM's `rocm gpu` change is pinned by a test that drives the real
  write-state/read-back/report path.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The `device:` row now renders `<backend> [gpu <n>] [(<name>)]`, so the strings
the advised-command scan has to ignore moved with it: vLLM no longer builds
`rocm gpu {index}`, AIM no longer carries a bare `rocm gpu`, and the two docs
quote the named-device example instead. Also drops a redundant initialiser the
e2e `device:` poll never reads.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…er removed

Neither vLLM nor AIM holds a `"rocm gpu ..."` string literal any more - both
build the `device:` value through
`rocm_engine_protocol::format_reported_device` - so the entries that excused
those literals no longer match anything and `exclusions_still_match_something`
rejects them. Also drops a stale `"{backend} gpu {index}"` shape note from
vLLM's doc comment.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main split `engines/lemonade/src/state.rs` into `process.rs` and
`device_policy.rs`, so this branch's `reported_device` change moves to
`device_policy.rs`: the shared `format_reported_device` renderer plus
`gpu_name_from_state`, which reads the name recorded at launch.

main's `reported_device_reflects_backend_and_pinned_ordinal` asserted the
old `<backend> gpu` shape, which this branch deliberately replaced — a
row naming no device. It is superseded by this branch's two tests,
relocated with the function, which pin both the named-ordinal row and the
ordinal-less one (including a legacy state file with no `gpu_name`).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Import-only conflict in serve_cmd.rs: #703 added `EngineSelection` to the
`engine_runtime_select` import while this branch added
`fetch_service_device` to the `engine_ipc` one. Both kept.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
… step failing a correct run

Review round 3 on #678.

- The `device:` e2e step failed a legitimately correct run. Lemonade
  reports `unknown` until its own inference probe returns, and the serve
  summary reads the device once right after `ready`, so the row can be
  absent on a correct implementation. Only serve-07 being xfail'd on
  lemonade+linux hid it. The step now tolerates the absence and the
  window is documented rather than asserted away.

- `rocm examine` grew a `rocm_support` line for RocmHip and Unknown with
  no scenario asserting it. examine-31 covers it, deliberately untagged
  so each lane's own hardware is the coverage: the claim is cross-checked
  against `detected_gfx_target` and `compatible_therock_family` from the
  same report, so "full ROCm/HIP support" on the mock lane or "not
  supported by ROCm" on an MI300X both fail.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Two conflicts, both from main's ongoing module splits.

`examine_report_text.rs`: main replaced the hand-written marker/`writeln!`
pair with `write_marked_row`. Took main's call shape and kept this branch's
`engine_inventory_note(..., gpu_support_level)`, which is the whole point of
the row here.

`therock.rs`: main moved the family-resolution cluster into
`therock/family.rs`, so both conflict hunks had an empty main side — the code
moved rather than went away. This branch's three contributions moved with it:
the `--engine lemonade` pin on `unsupported_host_gpu_message`, the
`rocm examine` pointer on `unsupported_requested_target_message`, and the two
tests pinning the asymmetry between them (plus the extra assertion on
`unsupported_gpu_error_names_the_gpu_and_does_not_suggest_families`).

Note on numbering: this branch's new `examine-31` is correct for its own
file position, and so is #447's. They collide only when the second of the two
merges, which is when the loser renumbers to examine-32 — renumbering either
now would fail `feature_naming` on its own branch.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main split `host_gpu.rs` into a directory module plus `gfx_target.rs`, so
every conflict hunk had an empty main side — the code moved rather than
went away. This branch's additions move with it:

- `GpuSupportLevel`, `gpu_support_level` and
  `gpu_support_level_for_host_gpu` join the family tables in
  `gfx_target.rs`, which is where the predicate they build on
  (`gfx_target_has_no_rocm_build`) now lives.
- `pinned_gpu_name` and its naming helpers stay in `host_gpu.rs`, next to
  the probes they guard.
- `WindowsExamineInventory::amd_display_count` moves to
  `host_gpu/windows_inventory.rs`.
- The exhaustive `rocm_support_line` replaces the VulkanOnly-only version
  main carried in `host_gpu/render.rs`. That is the point of the change:
  every support level gets a line, because `lemonade_engine_note` sends
  Unknown hosts to `rocm examine` and silence there is not an answer.

All 13 tests this branch added to `host_gpu.rs` were relocated with their
subjects and are verified present and passing, not dropped with the moved
region.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main moved the codex-bridge snapshot cluster into `bridge_snapshot.rs`,
so that hunk had an empty main side again. This branch's contributions
move with it: `builtin_codex_bridge_engine_inventory` takes the host's
`GpuSupportLevel` and routes lemonade's row through
`engine_inventory_note`, and `build_codex_bridge_snapshot` hoists the
`ExamineSummary` out of the struct literal to derive it. The
`host_gpu_support_level` helper stays at the crate root, where
`chat.rs`'s two MCP `engines` call sites reach it.

Three smaller ones: the `examine_report_text.rs` import takes both sides;
main changed the serve helper to return its stdout while this branch
records it on the world, so it now does both; and runtime-19's new
`Then` step sits alongside main's new runtime-20/21.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
#284 landed, so `serve_summary.rs` and `docs/testing.md` now carry its
OOM tests and prose alongside this branch's device-row ones; both kept.
main also added the stop-confirmation contract to the README and the
serve `--help` text, which sits beside this branch's device-line
paragraph rather than replacing it. `engine_manages_own_runtime` moved to
`engine_install_env`, and `rocm-core` gained `windows_amd_gpu_listing`
next to this branch's `pinned_gpu_name` re-export.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ge cut

Review round on #678.

- The `rocm serve` `--help` doc lost the tail of a sentence when I
  combined this branch's device-line paragraph with main's
  stop-confirmation text: it read "...signals the server process directly
  and exits `rocm services`." and the documented exit-0 behaviour was
  gone. Restored from main's wording.

- `GpuSupportLevel` and the two `gpu_support_level*` functions were
  appended after `gfx_target.rs`'s `mod tests`. Moved above it.

- main added `freeform_prefix_with_quote` and the runtime-22 steps to
  `runtime_steps.rs` next to this branch's examine-pointing assertion;
  both kept.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…he row

Review round on #678. The note promised that
`rocm serve <model> --engine lemonade` reports the backend on its
`device:` row, while this PR's own docs say that row can be absent: the
serve summary reads the device once, right after `ready`, and lemonade
answers `unknown` until its own inference probe returns.

The advice now names `rocm services list`, which shows the row once it is
known, so the command can actually clear the condition it is printed for.
The reasoning is recorded beside the arm rather than left implied.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…efuse

Review round on #678.

The `Unknown` tier covers two different hosts and was giving both the
same advice. A detected-but-unrecognised gfx target has a GPU, so
`rocm serve` runs and the engine's own `device:` row settles what this
catalog could not. A host with nothing detected is refused outright by
the no-usable-GPU pre-flight under the default `gpu_required` policy, so
naming that command was advice that could not clear the condition it was
printed for (AGENTS.md section 3).

The no-detection arm now points at `rocm diagnose`, whose
`fix-3-rocm-kernel`, `fix-4-render-group` and `fix-5-amdgpu-load` entries
are exactly the reasons a present GPU goes undetected. The sibling arm
keeps the serve advice.

`no_gpu_advice_names_a_command_that_host_can_run` asserts the behaviour
rather than the wording, as asked: it pins the tier through
`gpu_support_level` and then checks the rendered line does not name
`rocm serve`. Falsified by reverting the arm — the test fails with the
offending line quoted.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main split chat.rs into chat/{intent,mcp_bridge,summaries,turn}.rs and moved
the clap definitions out of main.rs into cli.rs, so an empty origin/main side
in these conflicts meant "relocated", not "deleted". The `engines` MCP tool
change and its tests now live in chat/mcp_bridge.rs, and the `serve` and
`services list` help text now lives in cli.rs.

main also added examine-31..34, so this PR's scenario is renumbered to
examine-35 (its docs/testing.md reference follows it); the kfd pre-flight
examine-31 that docs and model_serving.feature name is main's, unchanged.

Dropped `missing_packaged_engine_reason` from examine_report_text.rs: main
deleted that function and its call sites outright.

While porting, restored the `host_gpu_support_level` doc comment an earlier
merge on this branch had welded onto `engine_inventory_note`, and the
`rocm serve` help sentence the same merge truncated mid-clause.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…t refused

`rocm install sdk` and `rocm examine` tell a gfx90c APU to run
`rocm serve <model> --engine lemonade`, but nothing showed serve's
no-usable-GPU pre-flight accepts that host. It does: a zero KFD compute-node
count falls back to the DRM amdgpu card count, so the host the refusal was
printed on resolves ordinal 0. Pin that composition off one planted /sys, in a
unit test and in an e2e scenario that runs the real binary down the route the
CLI sends the user.

Also close the test gaps review 5826143385 named: the positive `device:` row
and the `unknown` filter now have assertions that run in default CI, the
endpoint key file's scoping is pinned, and the examine support-line check
asserts the specific command each tier owes instead of any `rocm ` word.
Windows reads the display inventory once per `pinned_gpu_name` instead of
twice, and the docs say the engine *reports* a device rather than naming the
one it observed.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	apps/rocm/src/serve_summary.rs
#	apps/rocm/src/services_text.rs
#	tests/e2e-cucumber/features/examine.feature
#	tests/e2e-cucumber/features/model_serving.feature
…ildren

The comment beside the `env_remove` cited
`an_external_plugin_child_is_not_handed_the_endpoint_key_file` as the thing
keeping it honest. That test did not exist, so the claim was false and the
behaviour change — which applies to every external-plugin spawn, not only
the device probe — was unguarded.

Write it, and give it something to hold onto: the env setup moves into
`apply_external_plugin_env`, so the exclusion is assertable without
spawning a plugin binary. The test distinguishes CLEARED from never-set,
since `Command` records `env_remove` as an explicit `(key, None)` and a
bare absence check would pass with the `env_remove` deleted. Verified:
deleting it turns the test red.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	tests/e2e-cucumber/features/model_serving.feature
Three conflicts, all where both sides had appended to the same place:

- apps/rocm/src/main.rs: main's side was EMPTY, which here means the
  sigpipe guard and its `engine_stdin_write_under_sig_dfl_is_guarded_not_fatal`
  test MOVED to crates/rocm-core/src/sigpipe.rs rather than being deleted.
  Took main's side and dropped the block; keeping ours would have duplicated
  the test and left it calling a `with_sigpipe_ignored` that main.rs no longer
  defines. `reset_sigpipe` stays, since main.rs still defines and calls its own.

- crates/rocm-core/src/host_gpu.rs: both sides added different tests (ours the
  single-GPU naming cases, main's the kernel-release reads). Kept both.

- tests/e2e-cucumber/features/examine.feature: both sides added scenarios
  numbered from examine-39. Kept main's 39-47 as canonical and renumbered this
  branch's ROCm-support-level scenario to examine-48, so one scenario moves
  instead of nine and the file stays sequential.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`GpuSupportLevel::VulkanOnly` was derived from the gfx-target table alone:
"no ROCm build" was read as "Vulkan then", on a machine nobody had looked at.
Every surface that renders that level tells the user to go and serve a model
on it, so a host with no Vulkan driver installed -- or with only a software
rasteriser -- was sent to a backend that cannot start, with no CPU fallback
to catch it.

Adds a real probe (host_gpu/vulkan.rs): ICD manifests through `host_path` on
Linux, the Khronos registry on Windows, and `vulkaninfo --summary` for device
enumeration. Filesystem and subprocess evidence rather than dlopening
libvulkan, because the loader reads the real ICD list and so could never be
faked on the no-GPU lane -- and because running vendor driver code inside
`examine` can hang a headless runner.

`gpu_support_level` now takes that evidence. An installed driver and an
enumerated device are kept apart (`DriverPresentDevicesUnverified`), since
collapsing them would replace one unverified promise with another; a platform
that cannot be probed keeps `VulkanOnly`, because withholding the only path
such a host has, on the strength of a probe that never ran, would be a worse
answer than the assumption being replaced.

The negative case gets its own level. `NoGpuBackend` is not folded into
`Unknown`: both halves there are established (the catalog confirms no ROCm
build, the probe finds no driver), whereas `Unknown` means neither could be.
Collapsing them commits, in the other direction, the conflation `Unknown`
exists to prevent.

A software rasteriser is never a GPU path. Mesa lavapipe is a complete Vulkan
implementation running on the CPU, so counting it would reintroduce the
silent CPU fallback through a backend rather than through a policy change.

`rocm engines` and the codex bridge deliberately do not probe: they pass
`VulkanStatus::Unknown` and answer exactly as before, so a listing command
does not grow a subprocess. `rocm examine` already gathers the probe and is
the surface those messages send people to.

examine-49/50/51 cover the three answers on the ungated mock lane, which
needs neither a Renoir APU nor a Vulkan driver -- only the KFD node and ICD
state the harness can plant. Falsified: both negative scenarios fail when
`gpu_support_level` is made to ignore the probe.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`engine_stdio_command` set `ROCM_SERVE_API_KEY_FILE` when the caller
passed one and otherwise left the variable alone. `rocm` passes `None`
for every external engine plugin — a third-party binary — so whenever
this process had inherited the variable, the path to a mode-0600 file
holding a live endpoint credential was handed straight through.

Clear it on the `None` branch. Living in the shared stdio client rather
than at the `rocm` call site means `rocmd`'s plugin calls get the same
treatment.

The test asserts CLEARED rather than absent: `Command` records
`env_remove` as an explicit `(key, None)`, so a plain "is it there" check
would pass with the clearing deleted. Verified by deleting it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…e gate

`apps/rocm/src/services_text.rs` crossed the 1500-line module-size limit
that ROCMAI-889 made blocking — main is at 1458 and this change's 164
added lines took it to 1512, so it is this PR's overflow, not an
inherited one.

Trimmed the two doc blocks the change itself added, which a reviewer had
already called out as heavy. The load-bearing facts stay: the ~18s bound
and where its arithmetic lives, why the chat tool pays it rather than
rendering a device-less view, and what the probe seam does and does not
cover. What went was restatement.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocking finding, plus a few smaller points.

Blocking

  • classify_vulkan_support returns Absent in the software_only branch before it looks at gpu_icds. A host with radeon_icd.x86_64.json installed whose vulkaninfo lists only llvmpipe (for example no render-group access) is told "no Vulkan GPU driver was found - install one". That advice cannot clear the condition, and it contradicts the icds field. No test covers a GPU ICD together with a software-only enumeration. See the inline comment.

Non-blocking

  • A software-only ICD list yields a confirmed Absent even when vulkaninfo never answered. LINUX_ICD_DIRS also omits loader paths, which matters now that this earns a confirmed result. The doc on Absent (both sources must answer) disagrees with this branch.
  • When the enumeration is non-AMD only, the report still says "a Vulkan driver is installed" while the detail says none of the devices is AMD (known limitation 1, pinned by a test).
  • Windows now spawns a second PowerShell per full gather. Please confirm rocmd does not gather on a periodic tick. The registry failure and timeout paths have no injectable-runner test (acknowledged in the PR text).
  • Nit: the comment block above the inner match in gpu_support_level reads as two comments merged.

Decisions to confirm

  • rocm engines, the MCP engines tool, doctor info-20 and rocm install sdk still make the unconditional Vulkan promise while examine does not. Documented as a scope-out; is there a follow-up ticket?

Comment thread crates/rocm-core/src/host_gpu/vulkan.rs Outdated
}

let software_only = !devices.is_empty() && devices.iter().all(|device| !device.is_hardware());
if software_only {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

software_only returns Absent before gpu_icds is consulted. With radeon_icd.x86_64.json installed and vulkaninfo listing only llvmpipe (e.g. no render-group access) this yields NoGpuBackend and "no Vulkan GPU driver was found - install one" while icds holds a GPU driver. Route to the driver-present-but-unverified path when a GPU ICD exists, naming the software-only enumeration in the detail, and add a test for GPU ICD + software-only device (the lavapipe tests only use lvp_icd).

// registry key produces. Both sources must have answered before an empty
// result may be called an absence.
if icds.is_empty() && devices.is_empty() && !(icds_listed && enumerated) {
// Which source went unread decides the sentence: a host is owed the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Inconclusive guard only fires when icds is empty, so a lavapipe-only manifest list reaches the confirmed-Absent software-only path with enumerated == false. That gives a confirmed absence without vulkaninfo having answered, against the "both sources must answer" rule documented on Absent. Consider requiring both sources to have answered there too.

/// what lets the no-GPU CI lane cover this probe at all.
#[cfg(target_os = "linux")]
const LINUX_ICD_DIRS: [&str; 4] = [
"/usr/share/vulkan/icd.d",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The directory list omits paths the loader reads (/usr/local/etc/vulkan/icd.d, ~/.local/share/vulkan/icd.d, XDG_DATA_DIRS entries, /opt/amdgpu*), which matters now that a software-only manifest list can produce a confirmed Absent. A proprietary AMD ICD in an unsearched directory next to Mesa's lavapipe would be reported as absent.

Main landed ROCMAI-703 (#1050), which rewrites the same deployment-summary
prose and adds its own serve scenario, so all three conflicts are with it.

tests/e2e-cucumber/features/model_serving.feature: main took `serve-60` for
its smoke-test-failure scenario. Main's keeps 60; this branch's APU scenario
becomes `serve-61` and moves after it, so each number still equals its
position. `apps/rocm/src/therock/family.rs`'s prose reference follows. The two
`-n "serve-60"` repro filters in docs are main's own and are left alone.

docs/testing.md: two independent sections appended at the same place, kept
whole. They are unnumbered `##` headings, so nothing renumbers.

README.md: both sides rewrite the one paragraph describing the deployment
summary, so this is a real merge rather than a choice between them. The
summary sentence keeps this branch's `device:` row and main's endpoint/model/
smoke-test list; main's smoke-test-failure paragraph and this branch's
"device line is omitted when the engine cannot say yet" paragraph both
survive; and the shared closing sentence is kept once. Taking either side
whole would have silently dropped the other's user-facing contract.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change: base 3d59dbcb883f0c73df389f6599a7ec3191e55db4 (tip of the stacked branch ROCMAI-550-vulkan-support-messaging) against head a6fb45c976195fe95aaab8d92bb23a652e0aed32, 18 files (+2968/-124). Run as five independent file-group passes (Vulkan probe core; the classification/rendering layer; the CLI/app-layer consumers; the e2e cucumber tests; the docs), plus direct tracing of the two most central functions.
Implements the Vulkan-probing half split out of PR #678 on this same repository; the GpuSupportLevel classifier it extends is #678's own work and is out of scope here. No other tracker ticket is named.

Blocking

  • classify_vulkan_support's device-based "software only" check short-circuits before it looks at the ICD evidence, so a host with a real GPU driver installed can be told none was found — crates/rocm-core/src/host_gpu/vulkan.rs:565-574 (the early return), contrast vulkan.rs:577-667 (the ICD-aware branch it bypasses)
    The software_only check fires purely on what vulkaninfo enumerated and returns a confirmed Absent immediately, before gpu_icds (the filtered list of real GPU driver manifests) is even computed. A host with radeon_icd.x86_64.json installed whose vulkaninfo enumerates only llvmpipe (a common shape for a render-group permission problem) is told the confirmed-fact sentence "no Vulkan GPU driver was found - install one (Mesa RADV on Linux...)" — advice that cannot be followed, since the driver is already installed and named in icds. No test in this file pairs a real GPU ICD with a software-only enumeration; the e2e scenario that plants a software-only ICD (examine-52) also plants a software-only enumeration, so this exact combination has never been exercised in either layer.
    Confidence 85 · logic · Fix: compute gpu_icds before the software_only check (or fold the check into the branch below), and route a software-only enumeration alongside a real GPU ICD to DriverPresentDevicesUnverified rather than Absent.

  • The same function's ICD-based "software only" fallback can confirm Absent without vulkaninfo ever having run, contradicting its own documented invariant — vulkan.rs:657-667, invariant stated at vulkan.rs:63-76 ("[Absent] reachable only when both sources answer and answer empty")
    When icds holds only a software-rasteriser manifest (e.g. lvp_icd.x86_64.json) and the device evidence is unanswered() (vulkaninfo missing, failed, or timed out), gpu_icds is empty and the function falls through to the final Absent return — the icds.is_empty() branch is false here, so the comment's own precondition ("both sources answered") holds only for that sibling branch, not this one. The one source that could have found a driver whose manifest sits outside the fixed search directories (vulkaninfo) never ran. No test exercises icds=[lvp_icd] with an unanswered device source.
    Confidence 85 · logic · Fix: require enumerated before this fallback may return Absent; otherwise route to Inconclusive, consistent with the gpu_icds.is_empty() branch immediately above it.

  • The NoGpuBackend rendering arm gives unconditional "install the driver" advice with no cross-check against the already-detected Windows driver status, unlike its sibling VulkanUnconfirmed arm — crates/rocm-core/src/host_gpu/render.rs:159-164 (missing the check), contrast render.rs:43-57 (vulkan_unconfirmed_next_step, which has it)
    vulkan_unconfirmed_next_step checks self.driver.status == AMD_DISPLAY_DRIVER_DETECTED before telling a Windows host to install a driver, because on Windows that status means the Vulkan ICD is already present. The NoGpuBackend arm has no equivalent check and always emits "...no Vulkan GPU driver was found - install one (Mesa RADV on Linux, the AMD display driver on Windows)...". The report's own driver_status: line would print driver_status: amd_display_driver_detected in the same output — this PR's own shared test fixture (examine_summary_with_gpu, reused by examine_render_does_not_promise_vulkan_on_a_host_that_has_none) defaults to exactly that driver status on a Windows host, but the test asserts only that "install one" appears, not that it is consistent with driver_status.
    Confidence 85 · logic · Fix: apply the same cross-check (ideally factored into a shared helper, see Decisions for the author) inside the NoGpuBackend arm, and add an assertion that a Windows host already showing the AMD display driver is not told to install it under NoGpuBackend.

  • The new banner text distinguishing VulkanOnly from the three other "no usable backend" levels has no test asserting the distinction, so a mutation merging the arms would pass every existing test — apps/rocm/src/examine_report_text.rs:293-349 (gpu_execution_line)
    Neither of the two banner sentences ("...that is the Vulkan backend, not ROCm." / "...CPU execution is not supported.") appears anywhere outside this one production function. The existing tests that touch these levels only check the per-engine note row and ROCm-claim parity, neither of which contains this banner's wording, so they would stay green even if VulkanOnly were folded into the shared arm.
    Confidence 85 · logic (test coverage) · Fix: add a direct assertion that the VulkanOnly banner sentence is present only for VulkanOnly and absent for the other three levels, mirroring the differential style already used for the note-row tests.

Non-blocking

  • A load-bearing comment on VulkanSupport::driver_label() states a narrower precondition than the code and its own test exhibit — vulkan.rs:256-258
    The comment claims the ICD-name fallback "only ever runs once an AMD hardware device has already been confirmed", but it also runs whenever no AMD device was found at all — a test calls it with an empty device list and still gets a result from this fallback. The actual, well-tested behaviour looks intentional; only the comment is wrong.
    Confidence 85 · mechanical (comment accuracy) · Fix: reword to state the actual precondition.

  • VulkanStatus::DriverPresentDevicesUnverified is never produced by any e2e scenario, and the ICD-only software-rasteriser-exclusion branch inside it is dead at the e2e level — tests/e2e-cucumber/features/examine.feature:766-818, tests/e2e-cucumber/tests/e2e/examine_steps.rs:3421-3686
    The four new scenarios reach Available, Absent (via two different routes) and Inconclusive, but never DriverPresentDevicesUnverified. examine-52 plants an lvp_icd manifest, but because its vulkaninfo stub enumerates a CPU device, the device-based software_only early return (first Blocking item above) fires first, so the planted ICD manifest never reaches gpu_icds and is inert to the scenario's outcome.
    Confidence 70 · logic (test coverage) · Fix: add a scenario planting a GPU ICD with no vulkaninfo on PATH, or a vulkaninfo summary naming only non-AMD hardware.

  • README's "Vulkan-unconfirmed" row names the wrong source as the one that can fail to answer — README.md:612
    The row reads "...no ICD, no enumerated device, and no vulkaninfo to ask", but classify_vulkan_support also reaches this status when vulkaninfo did answer and the ICD/registry listing is what failed (realistic on Windows). The VulkanStatus::Inconclusive doc comment itself says "at least one of the two sources" — an "or", not specifically vulkaninfo.
    Confidence 85 · documentation · Fix: generalize the row to name either source.

Decisions for the author

  • Other GPU-support surfaces still make the unconditional Vulkan promise this PR removes from rocm examine — answered-by-intent
    rocm engines, the MCP engines tool, and rocm install sdk's refusal message all still read the gfx table alone, so the same host can get two different answers about its Vulkan path depending which command is run. The PR description answers this directly: "rocm examine is the only surface this PR routes through the probe... Scoped out rather than fixed, deliberately: family.rs is a pure-function error builder reached before any host probe runs... and one that belongs with the rocm engines / doctor info-20 surfaces that also still speak from the gfx table." Would otherwise be an architectural finding. Worth confirming whether a follow-up is tracked, since the inconsistency is real and currently untested anywhere.

  • The Windows driver_status cross-check is implemented ad hoc inside one match arm rather than as a shared predicate — non-blocking-improvement
    vulkan_unconfirmed_next_step is the only place that currently consults driver.status before giving install advice; the missing NoGpuBackend cross-check (Blocking, above) is exactly the failure mode this duplication invites. Worth extracting into one helper every advice-giving arm calls.

Positive signals

  • VulkanEvidence<T> makes "did this source answer" and "what did it find" impossible to conflate — both fields are private, constructible only via answered()/unanswered(), and the two monomorphizations make passing one evidence type where the other belongs a compile error rather than a silent status flip.
  • Every match on GpuSupportLevel/VulkanStatus in the changed files is exhaustive with no wildcard arm, so a future variant is a compile error rather than a silent fallthrough.
  • Several tests deliberately assert what a message must not say alongside what it must say — exactly the style that would have caught some of the findings above had it been applied to every arm.
  • The PR description and in-code doc comments disclose several real known limitations up front, with reasoning, rather than leaving them for a reviewer to discover.

Deployment notes

  • The cross-surface inconsistency described under Decisions for the author widens in practice with this change: examine now gives a materially different, evidence-based answer than rocm engines/install sdk on the same host.
  • CI for the head commit was green on every completed check at review time; one job (E2E tests (MI300X, current)) was still queued and had not completed.

What this covered

Every file in the diff (18 files, +2968/-124) between 3d59dbcb883f0c73df389f6599a7ec3191e55db4 and a6fb45c976195fe95aaab8d92bb23a652e0aed32, read completely, split into five file-group passes (Vulkan probe core: vulkan.rs/host_gpu.rs/lib.rs; classification+rendering: gfx_target.rs/render.rs; CLI/app layer: examine_report_text.rs/main.rs/cli.rs/family.rs/bridge_snapshot.rs/driver_plan.rs; e2e cucumber tests; docs), plus direct tracing of classify_vulkan_support and the render.rs rendering arms. Agent-instruction adherence checked against this repository's own contribution guidance: the new vulkan.rs file has an accurate architecture-doc entry, no file approaches the production module-size limit, and the specific internal-guidance citations in the code's own comments were verified accurate. CI was read from the pull request's check runs rather than assumed.

Did not run: a full history pass beyond a light log skim of the classification file's pre-PR commits (nothing of note surfaced). No test suite was executed (no toolchain available in this environment); every finding above is scored on direct code tracing, not execution.

A process note on independence: checking this PR for an existing claimed review (required before starting) necessarily surfaced the text of a prior review already posted at this exact head commit before any blind pass began. That prior review's sole blocking item covers the same code path as the first Blocking finding above; that finding should be read as corroborated rather than purely blind, though it was independently re-derived here by direct tracing of classify_vulkan_support rather than taken on the earlier review's word. The other three Blocking findings, and all Non-blocking findings, were derived by fan-out passes that were not given that review's text and had no access to this PR's discussion thread.


🤖 Automated by an AI agent

Comment thread crates/rocm-core/src/host_gpu/vulkan.rs Outdated
}

let software_only = !devices.is_empty() && devices.iter().all(|device| !device.is_hardware());
if software_only {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. This software_only check fires purely on what vulkaninfo enumerated, and returns a confirmed Absent here before gpu_icds (computed below, line 577) is ever consulted. A host with radeon_icd.x86_64.json installed whose vulkaninfo enumerates only llvmpipe (e.g. a render-group permission issue) is told the confirmed-fact sentence "no Vulkan GPU driver was found - install one (Mesa RADV on Linux...)" — advice that cannot be followed, since the driver is already installed and present in icds. No test in this file pairs a real GPU ICD with a software-only enumeration, and the e2e scenario that plants a software-only ICD (examine-52) also plants a software-only enumeration, so this combination is untested in either layer.

Confidence 85 · logic · Fix: compute gpu_icds before this check (or fold it into the branch below), and route "software-only enumeration + a real GPU ICD present" to DriverPresentDevicesUnverified rather than Absent.

"only a software (CPU) Vulkan driver is installed; that is not a GPU path".to_owned()
};
return VulkanSupport {
status: VulkanStatus::Absent,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. This fallback can confirm Absent without vulkaninfo ever having run, contradicting the invariant documented on VulkanStatus::Absent ("reachable only when both sources answer and answer empty", lines 63-76). When icds holds only a software-rasteriser manifest (e.g. lvp_icd.x86_64.json) and the device evidence is unanswered() (vulkaninfo missing/failed/timed out), gpu_icds is empty, so this branch is reached with icds.is_empty() false — the comment two lines up ("both sources answered... this is the only place entitled to deny the driver outright") only actually holds for the icds.is_empty() sibling, not this one. No test exercises icds=[lvp_icd] paired with an unanswered device source.

Confidence 85 · logic · Fix: require enumerated before this fallback may return Absent; otherwise route to Inconclusive, consistent with the gpu_icds.is_empty() branch above it.

Comment thread crates/rocm-core/src/host_gpu/render.rs Outdated
// precisely so that this sentence is never said to one.
GpuSupportLevel::NoGpuBackend => format!(
" rocm_support: GPU detected ({}) but not supported by ROCm, and no Vulkan GPU \
driver was found - install one (Mesa RADV on Linux, the AMD display driver on \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. This arm gives unconditional "install one (Mesa RADV on Linux, the AMD display driver on Windows)" advice with no cross-check against self.driver.status, unlike its sibling VulkanUnconfirmed arm (vulkan_unconfirmed_next_step, lines 43-57), which checks driver.status == AMD_DISPLAY_DRIVER_DETECTED before giving the same kind of advice, specifically because on Windows that status means the Vulkan ICD is already installed. The report's own driver_status: line would print driver_status: amd_display_driver_detected in the same output for such a host — this PR's own shared test fixture (examine_summary_with_gpu, reused by examine_render_does_not_promise_vulkan_on_a_host_that_has_none) defaults to exactly that driver status on a Windows host (per the comment at line ~741 describing this fixture), but the test only asserts "install one" appears, not that it's consistent with driver_status.

Confidence 85 · logic · Fix: apply the same driver.status == AMD_DISPLAY_DRIVER_DETECTED cross-check here (ideally factored into a shared helper both arms call), and add an assertion that a Windows host already showing the AMD display driver is not told to install it under NoGpuBackend.

/// three variants and the Vulkan probe added two more, and the compile error
/// is what forced the decision below rather than letting either new level
/// fall through to a sentence about a backend nothing had established.
const fn gpu_execution_line(level: rocm_core::GpuSupportLevel) -> &'static str {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. Neither of the two banner sentences this function introduces ("...that is the Vulkan backend, not ROCm." / "...CPU execution is not supported.") appears anywhere outside this production function — confirmed by grepping both substrings across apps/rocm/src and tests/. The existing tests touching these levels only check the per-engine note row and ROCm-claim parity, neither of which contains this banner's wording, so a mutation merging the VulkanOnly arm into the shared arm below it would leave every existing test green.

Confidence 85 · logic (test coverage) · Fix: add a direct assertion that the VulkanOnly banner sentence is present only for VulkanOnly and absent for the other three levels, mirroring the differential style already used for the note-row tests.

Comment thread crates/rocm-core/src/host_gpu/vulkan.rs Outdated
.and_then(|device| device.driver_name.clone())
.or_else(|| {
// Restricted to AMD-branded manifests, because this fallback
// only ever runs once an AMD hardware device has *already* been

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. This comment claims the ICD-name fallback "only ever runs once an AMD hardware device has already been confirmed", but the fallback also runs whenever no AMD device was found at all — an_installed_driver_with_no_enumerated_device_stays_unverified calls driver_label() with an empty device list and still gets Some("Mesa RADV") from this fallback. The actual, well-tested behaviour looks intentional; only the comment overstates its own precondition, which risks a future change "fixing" this believing the case unreachable.

Confidence 85 · mechanical (comment accuracy) · Fix: reword to state the actual precondition (no AMD+hardware device's own name is available, whether none was confirmed or one was confirmed without a name).

# gfx table reaches only one: "no ROCm build" does not mean "Vulkan then" on a
# machine nobody has looked at, and the three scenarios are what keep those
# answers distinguishable.
@id:examine-names-the-vulkan-driver-an-unsupported-gpu-can-use @requires-os:linux

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. These four scenarios (examine-50..53) reach Available, Absent (via two different routes) and Inconclusive, but never VulkanStatus::DriverPresentDevicesUnverified — a status covering three real host shapes (missing vulkaninfo, an empty enumeration with a GPU ICD present, or a non-AMD-only enumeration). examine-52 plants an lvp_icd manifest, but because its vulkaninfo stub enumerates a CPU device, the device-based software_only early return in classify_vulkan_support fires first, so the planted ICD manifest never reaches gpu_icds and is inert to this scenario's outcome.

Confidence 70 · logic (test coverage) · Fix: add a scenario planting a GPU ICD with no vulkaninfo on PATH, or a vulkaninfo summary naming only non-AMD hardware, asserting DriverPresentDevicesUnverified's distinct wording.

Comment thread README.md Outdated
| --- | --- |
| ROCm/HIP-supported | A TheRock package family covers this GPU. |
| Vulkan-only | No ROCm build, **and** either a Vulkan GPU driver was found here, or the platform cannot be probed for one at all (anything other than Linux and Windows, or a summary from an older `rocm remote doctor` that predates the probe). The `rocm_support` line says which of the two it is, and only names a driver when one was actually found. |
| Vulkan-unconfirmed | No ROCm build, and the probe could not reach an answer — no ICD, no enumerated device, and no `vulkaninfo` to ask. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. This row names vulkaninfo specifically as the one that could not be asked, but classify_vulkan_support's (icds_listed, enumerated) match has a (false, true) arm reachable when vulkaninfo did answer and the ICD/registry listing is what failed instead (realistic on Windows, where the registry read is a separate PowerShell call independent of vulkaninfo's presence). The VulkanStatus::Inconclusive doc comment says "at least one of the two sources" — an "or", not specifically vulkaninfo.

Confidence 85 · documentation · Fix: generalize to "...and at least one of the two sources (the ICD/registry listing or vulkaninfo) never answered."

Tracks #678 past its merge of ROCMAI-703 (#1050), which renumbered that
branch's APU scenario to serve-61 and reworked the README's
deployment-summary paragraph. Merged clean -- no conflicts here.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main landed the WSL librocdxg work (#1091), which appends to the same two
examine files this branch does.

tests/e2e-cucumber/features/examine.feature: main took `examine-49` through
`examine-53`. This branch's support-level scenario becomes `examine-54` and
moves after them, so each number still equals its position. Three references
follow it: `host_gpu/render.rs`'s doc comment, and the prose and repro filter
in `docs/testing.md`. The `examine-49`..`examine-53` references further down
that file are main's own librocdxg paragraphs and are left alone.

tests/e2e-cucumber/tests/e2e/examine_steps.rs: two independent step blocks
appended at the same place, both kept. The split left each side's final
function unterminated while sharing a single closing brace, so the brace is
reproduced for each rather than closing only the block that came last.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
#678 merged #1091, which took `examine-49` through `examine-53` and moved
#678's own support-level scenario to `examine-54`. This branch's four Vulkan
scenarios collided with that block, so they become `examine-55` through
`examine-58`. They already sat last in the file, so only the numbers moved.

Nine references follow them: three comments in `examine_steps.rs`, the
`host_gpu/vulkan.rs` row in `docs/architecture.md`, and the four support-level
table rows plus the repro filter in `docs/testing.md`. Two neighbouring
`examine-51` references are main's librocdxg ones -- the `render.rs` test
comment and the librocdxg paragraphs in `docs/testing.md` -- and are left
alone.

crates/rocm-core/src/host_gpu/render.rs: both sides added a field to the same
`format!` template. Resolved as the union -- this branch's `vulkan: {}` and
main's extra placeholder before `managed_runtimes` -- because the argument
list git had already merged carries both `self.vulkan_line()` and
`self.runtime_librocdxg_line()`. Taking either template alone would have left
the placeholders and the arguments out of step.

docs/testing.md: this branch raises the support levels from three to five,
which is its own change, so its wording wins over the base's.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main landed runtime provenance (#1090), which appends its own scenario to
examine.feature and took `examine-54`. This branch's support-level scenario
becomes `examine-55` and moves after it, so each number still equals its
position.

Two references follow it: `host_gpu/render.rs`'s doc comment and the prose
and repro filter in `docs/testing.md`. The other `examine-54` references in
that file belong to main's provenance scenario
(`@id:examine-reports-where-the-active-runtime-came-from`) and are left alone.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main's #1090 took `examine-54`, so #678's support-level scenario moved to
`examine-55` and this branch's four Vulkan scenarios shift again, to
`examine-56` through `examine-59`.

Nine references follow them: three comments in `examine_steps.rs`, the
`host_gpu/vulkan.rs` row in `docs/architecture.md`, and the four
support-level table rows plus their repro filter in `docs/testing.md`. The
`examine-55` repro filter immediately above that table is #678's and stays,
as do the `examine-54` references to main's provenance scenario.

docs/testing.md: this branch raises the support levels from three to five,
which is its own change, so its wording wins over the base's.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent fresh review of the full diff against this PR's actual base (ROCMAI-550-vulkan-support-messaging @ 30f91a09, not main), 18 files / +2968 -124, head 1cd9609c.

This PR replaces a gfx-table-only Vulkan inference with a real probe (ICD manifests, Windows Khronos registry, vulkaninfo --summary) and is, on the whole, carefully engineered: the evidence-vs-found split (VulkanEvidence<T>), the render text for every status/evidence combination, the five-variant GpuSupportLevel match with no wildcard arm, the Windows display-driver cross-check in vulkan_unconfirmed_next_step, and the new examine-56..59 e2e scenarios are solid and well tested. (The PR body's description of them as examine-50..53 is stale — presumably from rebasing onto the stacked base — but the four cases described are the ones actually present.)

However, the PR currently carries a standing, unresolved CHANGES_REQUESTED review from a maintainer, most recently at 2026-11-17T10:33:03Z, with roughly thirty open, non-outdated inline comments, two marked Blocking. I independently re-derived and confirmed the central one by reading the code and test suite fresh, before reading any review thread:

In classify_vulkan_support (crates/rocm-core/src/host_gpu/vulkan.rs:565-575):

let software_only = !devices.is_empty() && devices.iter().all(|device| !device.is_hardware());
if software_only {
    return VulkanSupport { status: VulkanStatus::Absent, /* "only a software (CPU) Vulkan device was found" */ .. };
}
let gpu_icds: Vec<&String> = icds.iter().filter(|icd| icd_is_gpu_driver(icd)).collect();

This returns a confirmed VulkanStatus::Absent from devices alone, before gpu_icds (computed on the very next line) is ever consulted. A host with a real AMD GPU ICD manifest installed (e.g. radeon_icd.x86_64.json) whose vulkaninfo enumerates only a software rasterizer — the common case of a user not in the Linux render group — is told the confirmed-fact sentence "no Vulkan GPU driver was found - install one", directly contradicting the ICD evidence already in hand. gpu_support_level then routes this straight to GpuSupportLevel::NoGpuBackend, so the false claim reaches the user verbatim. I confirmed no test exercises this combination: a_software_rasteriser_is_never_a_usable_vulkan_path (vulkan.rs:827) only plants a software-only ICD (lvp_icd.x86_64.json) together with the lavapipe device, never a real GPU ICD alongside a software-only enumeration, and the e2e examine-58 scenario likewise only plants lvp_icd.x86_64.json, not a GPU ICD plus a software-only vulkaninfo.

I verified via git diff <the commit that review was posted against>...HEAD -- crates/rocm-core/src/host_gpu/vulkan.rs that this file is byte-identical to the state that review round was posted against, so the finding stands unaddressed at the current head and that standing review is accurate as of now.

I did not re-post the other open inline comments (several other Absent/Inconclusive edge cases in the same function, a stale "same registry read" line in docs/host-probing.md, missing coverage for the new NoGpuBackend/VulkanUnconfirmed engine-note arms in main.rs, etc.) since they are already on the thread, anchored to the current diff, and I didn't find anything beyond what's already there; I spot-checked a sample of them against the current code and they hold.

CI is green (37/37 checks on the current head).

Given a maintainer's own standing change request already covers what I independently found, and the underlying defect is real, reproducible from the code, and user-visible, I'm requesting changes as well rather than adding a duplicate inline thread.

🤖 Automated by an AI agent

… GPU driver

`classify_vulkan_support` returned a confirmed `Absent` from `devices` alone,
before `gpu_icds` was computed on the next line. A host with
`radeon_icd.x86_64.json` installed whose `vulkaninfo` lists only `llvmpipe` --
the ordinary shape of a missing `render`-group membership -- was told "no
Vulkan GPU driver was found - install one", contradicting the `icds` field of
the same report with advice it cannot act on. The ICD evidence is now consulted
first, and such a host is `DriverPresentDevicesUnverified` with the
software-only enumeration named as the reason nothing was confirmed.

The ICD-based software-only fallback could likewise confirm `Absent` without
`vulkaninfo` ever running: a lavapipe-only manifest list skips the
`icds.is_empty()` guard, so the documented "both sources answered"
precondition held for its sibling branch only. That fallback now requires
`enumerated`, which makes `vulkaninfo`'s answer an invariant of every `Absent`
path rather than a property of most of them.

`absent_always_has_vulkaninfos_answer_behind_it` sweeps every reachable
combination of the two sources to hold that invariant mechanically.
`examine-60` is the e2e half: it plants `examine-58`'s enumeration with a GPU
manifest beside it, so reading the devices before the manifests fails exactly
one of the pair.

Signed-off-by: Roman Sirokov <roman@flowrl.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…tate under the lock

AIM's `reported_device` hardcoded `rocm` and probed the host for a GPU name
whatever the launch's `device_policy` was, so a `cpu_only` container — which
`resolve_use_gpu`/`docker::RunSpec` correctly give no /dev/kfd, no /dev/dri and
no HIP_VISIBLE_DEVICES — still rendered e.g. `rocm (AMD Instinct MI300X)`. That
is the invented device claim this branch exists to remove, shipped in compiled
code even though `parse_device_policy` currently refuses `cpu` at the CLI. The
policy the launch recorded now decides, and the probe is not made at all when
no GPU was attached.

AIM also recorded no `gpu_indices`, so on a multi-GPU host it answered a bare
`rocm` where lemonade and vLLM answer `rocm gpu <n> (<name>)` for the same
hardware — the engine does pin an ordinal, `plan_gpus` resolves it and
`HIP_VISIBLE_DEVICES` narrows the container to it. `run_container` now reports
the ordinals it attached so the state file and the container cannot disagree.

Both engines' `write_state` were a lock-free `fs::write` that bypassed
`carry_published_anthropic_endpoint`. The unlocked write predates this branch,
but the slow host GPU probe that makes the race practically reachable is added
to these two state writes by this branch itself, so the exposure is this
branch's to close: routing them through `rocm_core::write_json_state_file`
orders them against every merge and carries a sibling thread's publication
through the rewrite, as lemonade's already does. The status writers patch
through `merge_json_state_file` instead of rewriting the document whole.

`healthcheck_service_at` is split out of both handlers, the way lemonade's
already was, so a test can drive the real RPC entry point: every device test in
the three engines called the private formatter, so reverting a handler's wiring
to a hardcoded literal kept the suite green.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ld not

`gpu_support_level`'s `(_, Some(_)) => RocmHip` arm ignores the detected
target on purpose — `resolve_family` asks it about a `--family` the user
named, where there may be no detected target — but every call in the suite
passed Some/Some, None/None or Some/None, so narrowing the arm to
`(Some(_), Some(_))` was invisible.

`gpu_name_for_single_gpu_host`'s ordinal guard was in the same position: the
only test with a non-zero ordinal also supplied a host count of 2, which the
check two lines below already declines on, so the guard could be deleted
outright with the suite still green. Both mutations now turn it red.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…river_status

The `NoGpuBackend` arm emitted "install one (Mesa RADV on Linux, the AMD
display driver on Windows)" unconditionally, while its sibling
`VulkanUnconfirmed` arm already consulted `driver.status` first -- because on
Windows the AMD display driver *is* the Vulkan ICD, so its presence contradicts
that advice. The same report printed `driver_status: amd_display_driver_detected`
a few lines above the sentence telling the reader to go and install it.

The check is now one predicate both arms call, which is the shape that would
have kept the second arm from shipping without it. The confirmed absence is
unchanged -- the probe did look and found no Vulkan GPU device -- only the step
offered differs, and `vulkaninfo` is not offered there the way it is under
`VulkanUnconfirmed`, since reaching `Absent` already takes its answer.

`examine_render_does_not_promise_vulkan_on_a_host_that_has_none` asserted
"install one" against the shared Windows fixture, which defaults to exactly the
contradictory driver status; it now plants the host it is actually about.

Signed-off-by: Roman Sirokov <roman@flowrl.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…et it read both renderers

`assert_services_list_reports_device`'s comparison against `rocm serve`'s
summary was dead twice over. `world.serve_summary` comes from
`run_rocm_bounded`, which redirects stdout to a file, so `serve_cmd.rs`'s
`summary_mode` is false and the CLI prints `print_managed_launch_plain`, which
has no device row in any state; and `device_row_value` looked for `"device:"`
while `render_summary` writes a padded label with no colon, so it could not
have matched a summary even if one carried the row.

The parser now reads both renderers' shapes (and still refuses the
`device_policy:` row the serve plan prints), and the step says in full that on
today's lanes the comparison does not run and the backend assertion above it is
what discriminates — rather than letting the step's name imply coverage it has
not got. Making it live means driving the vLLM and Lemonade per-PR serve
canaries through a `TuiSession`, which is a change to those canaries' own
harness.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…end levels

`gpu_execution_line` tells a `VulkanOnly` host which backend it has and tells
the other three that the CPU is not an alternative. Neither sentence appeared
anywhere outside that function, so merging the `VulkanOnly` arm into the shared
one below it left every test green -- including the ROCm-parity sweep, which
only reads "ROCm GPU execution is required" and is satisfied by either.

Asserted as the discrimination rather than as either sentence: each level
requires its own banner present and the other's absent, and the three
no-usable-backend levels are swept rather than sampled.

Signed-off-by: Roman Sirokov <roman@flowrl.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…onade serve flow

`device_worth_reporting` claimed the `"unknown"`-until-ready gate was uniform
and cited vLLM and Lemonade as sharing it. They do not: vLLM and AIM answer as
soon as a state file exists, while Lemonade keeps saying `"unknown"` until its
own inference probe returns. That difference is the whole reason
`SERVE_DEVICE_PROBE_BUDGET` exists, so the comment now names it. The probe
bound alongside it also cites AIM's own `HEALTHCHECK_TIMEOUT_MS`, which it
relied on without naming.

`docs/architecture.md`'s `gfx_target.rs` entry still described pure lookups
after that file gained the whole `GpuSupportLevel` classifier; the gate checks
presence, not accuracy, so nothing caught it.

`docs/manual-testing.md` dropped `--foreground` from the Lemonade serve
example but still called the result "that foreground server" 34 lines later,
with no way to stop it. The prose now matches a backgrounded service and gives
the stop command, as the parallel flow below it already does.

Also drops review-thread references from source comments and a note to
reviewers from docs/testing.md; each says the reason instead.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`docs/host-probing.md` said Windows devices come from "the same registry read"
-- they come from `vulkaninfo` there as on Linux, the registry is the ICD
source only -- and both it and the `vulkan.rs` architecture row still described
an `enumerated` flag that `vulkan_devices_from_capture` no longer returns; it
returns a `VulkanEvidence<T>` whose `did_answer()` carries that question. Both
now state the invariant as implemented: `vulkaninfo` must have answered before
anything may be reported as a confirmed absence, and a GPU manifest outranks a
software-only enumeration.

README's Vulkan-unconfirmed row named `vulkaninfo` as the source that could not
be asked; `Inconclusive` is reachable when `vulkaninfo` answered and the
ICD/registry listing is what failed, which is realistic on Windows. Generalized
to either source, as the `VulkanUnconfirmed` doc comment now is too.

Also splits the two comments that had merged above the inner `match` in
`gpu_support_level`.

Signed-off-by: Roman Sirokov <roman@flowrl.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — three rounds piled up and I only just caught them. Everything below is at 0cce20990, four commits on top of 1cd9609c3. I agreed with all four blocking findings; where I adopted a suggested fix I checked it against the evidence model first rather than applying it mechanically, and one of them I widened.

Blocking

1. software_only short-circuits before the ICD evidence — fixed (c97969d7c)

Confirmed and fixed as suggested. gpu_icds is now computed before the early return, which is guarded on it:

let software_only = !devices.is_empty() && devices.iter().all(|d| !d.is_hardware());
let gpu_icds: Vec<&String> = icds.iter().filter(|icd| icd_is_gpu_driver(icd)).collect();
if software_only && gpu_icds.is_empty() { /* Absent */ }

A host with radeon_icd.x86_64.json and an llvmpipe-only enumeration now falls through to the arm at the bottom and earns DriverPresentDevicesUnverified. I checked that against the status docs before adopting it: a GPU manifest is installed and printed in icds, no device confirmed it, and "a driver is installed, no AMD device confirmed behind it" is exactly what that variant documents. It also gets its own detail rather than borrowing the vendor-mismatch one — "none of the devices it enumerated is an AMD GPU" is true of an llvmpipe-only host but reads as "your other vendor's card answered", so the arm names the software-only enumeration and the render group instead.

Pinned at both layers:

  • a_software_only_enumeration_does_not_deny_an_installed_gpu_driver — the GPU-ICD host against the lavapipe-ICD host, same enumeration, so the pair is a discrimination rather than two assertions.
  • examine-60, new e2e scenario: plant_unsupported_gpu(radeon_icd.x86_64.json, LAVAPIPE_VULKANINFO_SUMMARY) — deliberately examine-58's enumeration with one thing changed. It is also the only scenario that reaches DriverPresentDevicesUnverified at all, which closes the non-blocking coverage finding on examine.feature.

Mutation evidence: reverting the guard to if software_only { fails a_software_only_enumeration_does_not_deny_an_installed_gpu_driver and examine-60 (1 unexpected failure), and leaves examine-58 and the other 21 unit tests in that module green.

2. ICD-based fallback can confirm Absent without vulkaninfo running — fixed (c97969d7c)

Also confirmed. The icds=[lvp_icd] + unanswered-devices path reached the final Absent because the icds.is_empty() guard is false there. It now requires enumerated, routing to Inconclusive otherwise, as suggested.

I went one step further than the inline comment: with both fixes in, every path to Absent has vulkaninfo's answer behind it, so I turned that into a stated invariant on VulkanStatus::Absent and a mechanical test rather than leaving it a property of three separate arms. absent_always_has_vulkaninfos_answer_behind_it sweeps the reachable cross-product of {no icds, lvp only, radeon} × {no devices, llvmpipe} × {answered, unanswered} and asserts status != Absent || enumerated. A future arm that denies a driver on the ICD listing alone fails there even if it is added somewhere none of the hand-written cases look.

I did reword the Absent doc rather than keep "both sources answer and answer empty" verbatim: that is still exactly true of the empty-everything path, but the software-only paths are "answered and ruled out", not "answered empty", so the doc states the two halves separately now.

Mutation evidence: disabling the new guard fails both a_lavapipe_only_install_is_not_a_confirmed_absence_until_vulkaninfo_answers and the sweep.

Not fixed: LINUX_ICD_DIRS omitting /usr/local/etc/vulkan/icd.d, ~/.local/share/vulkan/icd.d, XDG_DATA_DIRS, /opt/amdgpu*. The stated argument for it was that a software-only manifest list earns a confirmed Absent; after this fix it no longer can without vulkaninfo having answered, and vulkaninfo reads the full loader search path. So the risk the finding was about is closed by a different route. Widening the list is still a real improvement and I would rather do it as its own change with its own tests than fold it in here.

3. NoGpuBackend gives install advice with no Windows cross-check — fixed (b7e77e811)

Confirmed, and the fixture point was the sharp end of it: examine_summary_with_gpu defaults to a Windows host with amd_display_driver_detected, so the existing test was asserting "install one" against a summary whose own driver_status line contradicted it.

Factored into one predicate as suggested — install_advice_is_contradicted_by_the_driver() — which both arms now call. That is the shape that would have stopped the second arm shipping without the check.

What the arm says on that host needed a decision, and it is not VulkanUnconfirmed's sentence: reaching Absent takes vulkaninfo's answer (see #2), so "run vulkaninfo yourself" is not a step left here the way it is one level up. The confirmed absence stands; what changes is that the host is told the display driver is installed and is exposing no Vulkan GPU device, so repairing or updating it is the step that could open one.

examine_render_does_not_tell_a_no_backend_windows_host_to_install_the_driver_it_has asserts the Windows and Linux halves against two summaries differing only in driver_status; asserts the report really does print driver_status: amd_display_driver_detected, so the contradiction being guarded is between two lines of one report rather than a hypothetical; and asserts the fixture still is the display-driver host, since otherwise the test quietly stops being a discrimination. examine_render_does_not_promise_vulkan_on_a_host_that_has_none now plants the Linux host it was actually about.

Mutation evidence: making no_gpu_backend_next_step unconditional fails the new test and nothing else.

4. The VulkanOnly banner has no test asserting the distinction — fixed (16ccdf90f)

Confirmed, including that the ROCm-parity sweep does not catch it: that test only reads the substring "ROCm GPU execution is required", which both banner sentences satisfy by absence.

engines_banner_claims_a_vulkan_backend_only_where_one_was_established asserts each level's own banner present and the other's absent, sweeps the three no-usable-backend levels rather than sampling, and checks RocmHip too, so that losing that arm would not let the sweep pass vacuously.

Mutation evidence: replacing the VulkanOnly arm's string with the shared one fails this test and only this test (31 other examine_report_text tests stay green).

Non-blocking

  • driver_label() comment overstates its precondition — fixed in c97969d7c. Correct: an_installed_driver_with_no_enumerated_device_stays_unverified exercises the no-AMD-device case. Reworded to the actual precondition (no AMD hardware device supplied a driverName of its own, whether because none was confirmed or because the one that was went unnamed). The AMD-only restriction and its dual-vendor reasoning stand.
  • README's Vulkan-unconfirmed row names the wrong source — fixed in 0cce20990, generalized to either source with the Windows case called out. Also applied to the GpuSupportLevel::VulkanUnconfirmed doc comment, which had the same wording.
  • docs/host-probing.md "same registry read" for devices — fixed. Devices come from vulkaninfo on Windows as on Linux; the registry is the ICD source only. The row now also says why that source usually does not answer there.
  • The stale enumerated flag in host-probing.md and the architecture.md vulkan.rs row — fixed; both now describe VulkanEvidence<T> / did_answer(), and both state the revised Absent precondition.
  • Nit: two comments merged above the inner match in gpu_support_level — fixed.
  • Missing coverage for the new NoGpuBackend/VulkanUnconfirmed engine-note arms — already addressed at the reviewed head, not newly fixed: examine_engine_inventory_separates_an_unconfirmed_vulkan_from_no_backend_at_all and engines_inventory_separates_an_unconfirmed_vulkan_from_no_backend_at_all tie each note to its level on both renderers, and each asserts the other level's wording absent. What was genuinely missing was the banner above those rows, which is finding 4.
  • Stale PR body — rewritten. examine-51→examine-57 and friends throughout (the scenarios are examine-56..60 now, including the new one); the "plants a failing vulkaninfo" claim corrected to exit 0 with an empty list; and the Windows-coverage paragraph now says what is and is not covered — windows_icds_from_capture's tests do drive the None case that a missing powershell, a non-zero exit and a timeout all collapse into, so "the failure and timeout paths have no unit coverage" was wrong. What remains uncovered is the registry script against a real registry and the PowerShell spawn's own cold-start behaviour. The stale numbering also appeared in examine.feature's own comments and in docs/testing.md; both fixed.

Decisions you asked about

  • Is there a follow-up ticket for rocm engines / MCP engines / doctor info-20 / rocm install sdk? No — there is not one, and I would rather say so than imply otherwise. It is a disclosed scope-out, not tracked work. The PR body now says that in those words rather than leaving it to inference. The inconsistency is real and it widens with this change, since examine now answers from evidence and the others still read the gfx table. Happy to file one if you want it tracked before merge rather than after.
  • rocm mcp-call examine|doctor gathers twice. Confirmed, acknowledged, not changed here. Pre-existing double gather; this PR enlarges the constant (up to two 1.5 s probes, plus a PowerShell start each on Windows) without introducing the second gather. Now stated in the PR body's Cost section.
  • Does rocmd gather on a periodic tick? No — re-verified. The only repeatedly-polled path is the watcher tick, watchers.rs:72 → host_snapshot::gather_gpu_snapshot_for_config, which takes the amd-smi route and never reaches ExamineSummary::gather(). The two build_bridge_snapshot call sites (mcp.rs:583, cli.rs:127) and every other gather() site are request-scoped. So Windows pays the second PowerShell per request, not per tick.
  • Non-AMD-only enumeration says "a Vulkan driver is installed" while the detail says none is AMD. Unchanged, still known limitation 1, still pinned by a test.

Verification

Linux box, all green on 0cce20990: cargo fmt --all --check; cargo clippy --workspace --all-targets -- -D warnings; cargo clippy -p e2e-cucumber --test e2e -- -D warnings; cargo test -p rocm-core (1657 passed, 1 known shared-box flake — loopback_lease_blocks_a_second_bind_until_released, passes alone); cargo test -p rocm --bin rocm (1951 passed); cargo test -p e2e-cucumber --test feature_naming; cargo xtask check-architecture-doc / check-crate-edges / check-module-size; and cargo xtask e2e -- -n '(examine-56|examine-57|examine-58|examine-59|examine-60)' — 5 scenarios, 24 steps, all passed.

Every new test was mutation-checked by breaking the production fix it guards and confirming it goes red, as noted per finding above.

I have left the standing reviews as they are rather than re-requesting; your call on the next round.

One conflict, in `apps/rocm/src/chat/mcp_bridge.rs`, and it is the
code-was-removed shape rather than two additions colliding.

Main's #980 deletes `internal_mcp_arguments` outright and replaces it with
`internal_mcp_tool_call` plus `mcp_call_cli_arguments`, so that a pasted MCP
envelope is unwrapped in one named step instead of silently inside the
dispatcher -- the point of that change being that the approval card and the
executed call cannot disagree. This branch still carried the old function.

Resolved by taking main's design whole and keeping only this branch's own
addition from the other side, the `engines` MCP tool-result helper. Keeping
this branch's `internal_mcp_arguments` would have reinstated the unwrapping
main had just removed, which is the opposite of what #980 is for; grep
confirms no caller of it survives.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
docs/testing.md: the base still describes Vulkan-only as inferred from the
gfx table with three support levels; this branch replaces that inference with
a probe and has five. This branch's wording wins, as before.

The merge also went clean in `gfx_target.rs` and did not compile. #678 added
two `gpu_support_level` cases for its `(_, Some(_))` arm, written against its
own two-argument signature; this branch adds a third `vulkan` parameter, so
both arrived calling a function that no longer exists in that shape. Both now
pass `VulkanStatus::Absent`, matching the assertions immediately above them,
which deliberately vary the Vulkan status to show a covering family wins
outright -- so the merged cases keep #678's intent (a detected target cannot
outvote a resolved family) and gain this branch's stronger premise: it holds
even when the probe found nothing.

Caught by `cargo clippy --workspace --all-targets`, not by the merge.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review: PR #1036 Vulkan GPU Probing for rocm-cli

Repository: ROCm/rocm-cli
PR: #1036 (adds Vulkan GPU probe logic + classification state machine)
Review Method: Blind multi-pass review per review-code skill methodology
Result: REQUEST_CHANGES (3 blocking findings ≥70 confidence)


Summary

This PR adds comprehensive Vulkan GPU detection to rocm-cli, including ICD manifest discovery (Linux/Windows), vulkaninfo parsing, and classification of Vulkan capability (Available/Absent/Inconclusive/DriverPresentDevicesUnverified). The core algorithm is well-tested and correct, with exhaustive unit coverage and matching e2e scenarios.

Three blocking findings prevent approval:

  1. Doc-comment contradiction in VulkanStatus enum (lines 87-90 vs implementation)
  2. Untested Windows registry integration path (violates AGENTS.md §3)
  3. Contradictory error message in rocm install sdk vs rocm examine (violates remediation contract)

Blocking Findings

Finding 1: VulkanStatus::Inconclusive doc-comment contradiction (confidence 75)

File: crates/rocm-core/src/host_gpu/vulkan.rs:87-90
Category: Logic/mechanical
Severity: Medium

The enum documentation claims Inconclusive's precondition includes "no ICD manifest," but the classifier (lines 695-707) correctly reaches this state with non-empty ICD lists. The test a_lavapipe_only_install_is_not_a_confirmed_absence_until_vulkaninfo_answers confirms this: it passes icds = ["lvp_icd.x86_64.json"] and asserts status == Inconclusive.

The behavior is correct (software-only ICD without vulkaninfo answer should be Inconclusive). The documentation is wrong.

Required fix: Update the enum doc comment to reflect actual logic. Suggested: "The probe came away with nothing either way: only software-rasterizer ICDs, no enumerated device, and at least one of the two sources that never answered..."


Finding 2: Untested Windows registry PowerShell integration (confidence 72)

File: crates/rocm-core/src/host_gpu/vulkan.rs:450-469
Category: Architectural (platform-gated code)
Severity: Medium

The real #[cfg(windows)] fn detect_icds() performs a subprocess call to PowerShell against the Khronos registry key. While unit tests cover the pure parsing halves (windows_icds_from_capture, parse_windows_vulkan_icd_listing) and inspect the PowerShell script string, the full integration path—capturing live registry output through the subprocess—has zero test coverage.

The file's own doc comment and docs/host-probing.md:213 claim this is "covered by CI's windows-build-and-test job," but repo-wide grep for HKLM/Khronos returns only code references, not test references. The only scenario tagged @requires-os:windows in examine.feature is unrelated (AMD display-driver version/date).

Per AGENTS.md §3: "platform-gated subprocess logic must be tested on its own gate."

Required fix: Add a unit test for the Windows registry integration path. Options:

  • Mock the subprocess capture to inject realistic PowerShell output and assert end-to-end behavior
  • Explicitly document why this path is untested and update code comments to match (declare only the parsing seam as cross-platform tested)

Finding 3: Contradictory Vulkan remediation across same session (confidence 80)

File: apps/rocm/src/therock/family.rs:244-250 (unsupported_host_gpu_message)
Category: Correctness/consistency bug
Severity: High

The unsupported_host_gpu_message function unconditionally claims "the Vulkan backend can still be used" and recommends rocm serve <model> --engine lemonade, without consulting the new Vulkan probe result.

Unlike its sibling unsupported_requested_target_message (which handles user-named targets that may not exist on this host), this call site:

  1. Only executes after the host's own detected GPU architecture is known
  2. Runs on an already-failing, non-hot path (extra probe cost is negligible)
  3. Directly contradicts rocm examine output in the same session when VulkanStatus is actually NoGpuBackend

Example conflict: On a host with no AMD Vulkan drivers:

  • rocm examine → "no Vulkan driver was found, so no GPU backend can run here"
  • rocm install sdk → "the Vulkan backend can still be used; try rocm serve <model> --engine lemonade"

The remediation command will not clear the condition it was printed for.

This violates AGENTS.md §3: "remediation advice naming a command — the named command must exist, accept those flags, and actually clear the condition that printed it."

Required fix: Either:

  • Thread the Vulkan probe result into this message and condition the Lemonade recommendation on a successful probe, OR
  • Defer to rocm examine for Vulkan status (consistent with how other CLI commands separate concerns)

Verified Passes (No Issues >=70 Confidence)

  • GPU classification state machine: All 25 (family, VulkanStatus) state combinations fully tested and logically correct
  • E2E Vulkan probe scenarios (examine-56 through examine-60): All 5 scenarios correctly specify and exercise the expected outcomes
  • Documentation/architecture consistency: All file references properly linked, stale claims removed, test descriptions updated
  • AGENTS.md compliance: No internal keywords, DCO/commit signing enforced, FFI safety checklist validated, platform-gated tests on correct gates (only Windows registry path is gap)

CI Status & Merge Readiness

This PR requires fixes to the three blocking findings before merge. The underlying architecture and algorithm are sound; the issues are localized:

  1. One documentation error (10 minutes to fix)
  2. One missing test (1-2 hours to implement)
  3. One message logic contradiction (1-2 hours to wire probe or refactor)

🤖 Automated by an AI agent

Comment thread crates/rocm-core/src/host_gpu/vulkan.rs Outdated
/// The probe came away with nothing either way: no ICD manifest, no
/// enumerated device, and at least one of the two sources that never
/// answered — `vulkaninfo` missing or failing, or (on Windows) a registry
/// listing that could not be run.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc-comment contradiction: Inconclusive precondition claim doesn't match implementation

The enum doc (lines 87-90) claims Inconclusive's precondition includes "no ICD manifest," but the classifier reaches this state with non-empty ICD lists. This is confirmed by the test a_lavapipe_only_install_is_not_a_confirmed_absence_until_vulkaninfo_answers, which passes icds = ["lvp_icd.x86_64.json"] and asserts status == Inconclusive.

The behavior is correct (software-only ICD without probe answer should be Inconclusive). The doc needs updating to reflect actual logic: change "no ICD manifest" to something like "no ICD manifest that confirms a GPU driver" or "only software-rasterizer ICDs found."

WINDOWS_VULKAN_ICD_SCRIPT,
],
))
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing test coverage: Real Windows registry PowerShell integration untested

The #[cfg(windows)] fn detect_icds() at lines 450-469 performs a real subprocess call to PowerShell against the Khronos registry key. While unit tests cover the pure parsing half (windows_icds_from_capture, parse_windows_vulkan_icd_listing) and inspect the PowerShell script string itself, the full integration path—capturing live registry output through the subprocess—has zero coverage.

The file's own doc comment and docs/host-probing.md:213 claim this is "covered by CI's windows-build-and-test job," but repo-wide grep for HKLM/Khronos returns only code references, not test references. Per AGENTS.md §3, platform-gated subprocess code must be tested on its gate.

Add a unit test that either:

  • Mocks the subprocess capture to test integration with realistic PowerShell output, or
  • Documents why full-path testing is deferred and updates the code comment to reflect that only the parsing seam is tested cross-platform

`VulkanStatus::Inconclusive`'s doc gave "no ICD manifest" as a precondition,
and the classifier reaches that status with a non-empty `icds`: a host whose
only manifest is a software rasteriser, with no `vulkaninfo` answer behind it,
lands here rather than in `Absent`, because the ICD search is not exhaustive
and so lavapipe-is-all-I-can-see does not rule out a GPU driver the probe
never reads.

The behaviour is right and already pinned by
`a_lavapipe_only_install_is_not_a_confirmed_absence_until_vulkaninfo_answers`.
Only the doc was wrong, so only the doc changes, and it now names that test so
the next reader can check the claim instead of taking it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The file and docs/host-probing.md both claimed the Khronos registry read was
"covered by CI's windows-build-and-test job". Nothing calls `detect_icds` on
Windows, so that job compiles the call and never runs it: the program name,
the flags and the script handed to PowerShell had no test at all. Dropping
`-Command`, or losing `-NonInteractive` so a prompting profile turns the probe
into a 1.5s timeout, would have shipped and shown up only on real hardware as
a permanently inconclusive Vulkan line.

`windows_icds_through` takes the subprocess as an argument, so everything the
Windows path does except starting the process now runs on every lane, argv
included, through to `classify_vulkan_support`. What is left uncovered is
PowerShell's own behaviour on a real host, and both docs now say exactly that
instead of naming a job that does not test it. That residue is bounded: every
such failure arrives as a `None`, which the seam routes to `Inconclusive`
rather than to a claim about the machine.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`unsupported_host_gpu_message` closed, unconditionally, with "the Vulkan
backend can still be used - try `rocm serve <model> --engine lemonade`",
without ever consulting the probe this branch exists to add. On a host with no
AMD Vulkan driver one session could print "no Vulkan driver was found, so no
GPU backend can run here" from `rocm examine` and the opposite from `rocm
install sdk`, and the command it named could not clear the condition it was
printed for, which AGENTS.md §3 requires of remediation advice.

The message now classifies with this host's real `VulkanStatus` through the
same `gpu_support_level` the examine line uses, and takes its wording from the
same shared constants: only `VulkanOnly` keeps the Lemonade recommendation,
`VulkanUnconfirmed` makes it conditional on what `vulkaninfo` reports, and
`NoGpuBackend` offers the driver install instead.

The refusal itself is untouched. Its gate is `target_is_vulkan_only`, which
still classifies with `Unknown`, because whether a ROCm build exists is a fact
about the catalog and must not turn on an unrelated Vulkan probe; the probe is
paid for inside the already-failing branch. The sibling
`unsupported_requested_target_message` keeps the unconditional claim for the
same reason it keeps its own closing sentence: it is about a target the user
named, which may not be the GPU in this machine, so no probe of this host
bears on it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

All three blocking findings are addressed in bf773728e (branch split/vulkan-probe, three commits: ab87a9033, f3a33ca3c, bf773728e). Each new test was mutation-checked — the production change broken, the test confirmed red, then restored.


Finding 1 — VulkanStatus::Inconclusive's doc contradicted the classifier — fixed, doc only (ab87a9033)

You were right that the behaviour is correct and the doc was wrong. The precondition now reads "no ICD manifest that names a GPU driver", with an explicit paragraph that icds is therefore not necessarily empty here: a lavapipe-only host has found no GPU driver but has not ruled one out, because the ICD search is not exhaustive, so without vulkaninfo's answer it lands in Inconclusive and not Absent. The doc now names a_lavapipe_only_install_is_not_a_confirmed_absence_until_vulkaninfo_answers so the next reader can check the claim rather than take it.

No test, because nothing in the behaviour changed — the test that would have caught this was already there and already passing.

Finding 2 — untested Windows registry PowerShell path — I took the first option: injected a fake capture, and corrected the doc as well (f3a33ca3c)

Both, in fact, because the honest statement turned out to be narrower than either option alone.

I looked for an existing seam first, as you suggest. There wasn't one: capture_optional_command is called directly inside #[cfg(windows)] fn detect_icds(), and windows_icds_from_capture is only the pure half after the capture. So the argv — the program name, the flags, the script — was read by nobody but a human. Dropping -Command, or losing -NonInteractive so a prompting profile turns the probe into a 1.5 s timeout, would have compiled and shipped and surfaced only on real hardware as a permanently inconclusive Vulkan line.

windows_icds_through now takes the subprocess as an argument. detect_icds on Windows is windows_icds_through(capture_optional_command); the new cross-platform test the_windows_registry_lookup_runs_the_khronos_script_through_powershell injects a fake capture and asserts the program, the flags, that the script is the same WINDOWS_VULKAN_ICD_SCRIPT the parser's test reads, and then end-to-end that a realistic two-row Adrenalin capture becomes DriverPresentDevicesUnverified while a None becomes Inconclusive.

And the doc claim is still corrected, because the test does not make it true. Nothing calls detect_icds on Windows, so windows-build-and-test compiles that call and never runs it — your grep was right. Both the code comment and docs/host-probing.md now state exactly that: everything except starting the process is covered on every lane; what is left uncovered is PowerShell's own behaviour on a real host (output encoding, an execution policy that blocks the script, a cold start past the 1.5 s timeout). That residue is bounded rather than hand-waved — all three arrive as a None, which the seam routes to Inconclusive, so the untested step can make the answer less useful but not false.

Mutation evidence, two mutants, each killed only by the new test:

  • removed "-NonInteractive" from the argv → red (a profile that prompts turns this probe into a 1.5s timeout…); every other test still green.
  • replaced the windows_icds_from_capture call with VulkanEvidence::answered(parse(…unwrap_or_default())), i.e. the exact None/empty-key conflation the evidence type exists to prevent → red. The existing a_powershell_that_never_answered_is_not_an_empty_registry_key does not catch this one: it calls the pure half directly, so it never sees the wiring.

Finding 3 — unsupported_host_gpu_message promised Vulkan without consulting the probe — fixed (bf773728e)

Agreed, and this was the strongest of the three. The message now classifies with this host's real VulkanStatus through the same gpu_support_level that rocm examine's rocm_support line uses, and takes its wording from the same shared constants in rocm-core, so the two surfaces cannot describe one machine two ways:

  • VulkanOnly (device enumerated, driver installed, or platform unprobeable) — unchanged: VULKAN_ONLY_SUPPORT_CLAIM and rocm serve <model> --engine lemonade.
  • VulkanUnconfirmed — VULKAN_UNCONFIRMED_CLAIM + VULKAN_UNCONFIRMED_NEXT_STEP, and the Lemonade suggestion survives only as a consequence of what vulkaninfo reports, not as a promise. A working Adrenalin laptop lands here routinely, so withholding it entirely would be as wrong as asserting it.
  • NoGpuBackend — no Vulkan claim and no rocm serve at all, since that command cannot clear what printed it; it offers the driver install instead.

On your note about VULKAN_ONLY_SUPPORT_CLAIM: the fix does not belong in the constant. The constant is the claim; who may make it is the question, and the two refusals answer it differently. unsupported_requested_target_message is about a target the user named, which may not be the GPU in this machine, so no probe of this host bears on it and it keeps the unconditional claim — I was careful not to give it a probe. The constant's doc now spells out the three surfaces and why each says what it says.

The refusal itself is unchanged: its gate is still target_is_vulkan_only, which still classifies with VulkanStatus::Unknown, because whether a ROCm build exists is a fact about the catalog and must not turn on an unrelated Vulkan probe. The probe is paid for inside the already-failing branch, next to detect_host_gpu_diagnostics, which costs more.

New test the_detected_gpu_refusal_grades_its_vulkan_claim_by_the_probe covers all five statuses. Mutation: forcing the match to GpuSupportLevel::VulkanOnly — exactly the pre-fix behaviour — turns it red with the old sentence in the failure message, and no other test notices.

One gap I should name rather than let you find. The call site wiring — that resolve_family passes detect_vulkan_support().status and not Unknown — has no automated test. detect_host_gfx_target reads real sysfs/WMI with no simulated-host override, so no lane can reach that branch; runtime-19's comment already records that the detected-host door is unit-covered only, and that is still true. The graded formatter is fully covered; the one line that feeds it is covered by review.

Gates

Run on a Linux devbox (this Mac cannot build the workspace): cargo fmt --all --check, cargo clippy --workspace --all-targets -D warnings, cargo clippy -p e2e-cucumber --test e2e -D warnings, cargo test -p e2e-cucumber --test feature_naming, cargo xtask check-architecture-doc, check-crate-edges, check-module-size, cargo test -p rocm-core and cargo test -p rocm --bin rocm — all green. The one failure in the rocm-core sweep was the known shared-box uv install-path flake, which passes alone. No e2e scenario was added or renumbered.

@r0x0r
r0x0r force-pushed the ROCMAI-550-vulkan-support-messaging branch from 20c48b4 to f04357f Compare November 17, 2026 18:13
r0x0r added a commit that referenced this pull request Nov 17, 2026
`GpuSupportLevel::VulkanOnly` was derived from the gfx-target table alone:
"no ROCm build" was read as "Vulkan then", on a machine nobody had looked at.
Every surface that renders that level tells the user to go and serve a model
on it, so a host with no Vulkan driver installed -- or with only a software
rasteriser -- was sent to a backend that cannot start, with no CPU fallback
to catch it.

Adds a real probe (host_gpu/vulkan.rs): ICD manifests through `host_path` on
Linux, the Khronos registry on Windows, and `vulkaninfo --summary` for device
enumeration. Filesystem and subprocess evidence rather than dlopening
libvulkan, because the loader reads the real ICD list and so could never be
faked on the no-GPU lane -- and because running vendor driver code inside
`examine` can hang a headless runner.

`gpu_support_level` now takes that evidence. An installed driver and an
enumerated device are kept apart (`DriverPresentDevicesUnverified`), since
collapsing them would replace one unverified promise with another; a platform
that cannot be probed keeps `VulkanOnly`, because withholding the only path
such a host has, on the strength of a probe that never ran, would be a worse
answer than the assumption being replaced.

The negative case gets its own level. `NoGpuBackend` is not folded into
`Unknown`: both halves there are established (the catalog confirms no ROCm
build, the probe finds no driver), whereas `Unknown` means neither could be.
Collapsing them commits, in the other direction, the conflation `Unknown`
exists to prevent.

A software rasteriser is never a GPU path. Mesa lavapipe is a complete Vulkan
implementation running on the CPU, so counting it would reintroduce the
silent CPU fallback through a backend rather than through a policy change.

`rocm engines` and the codex bridge deliberately do not probe: they pass
`VulkanStatus::Unknown` and answer exactly as before, so a listing command
does not grow a subprocess. `rocm examine` already gathers the probe and is
the surface those messages send people to.

examine-49/50/51 cover the three answers on the ungated mock lane, which
needs neither a Renoir APU nor a Vulkan driver -- only the KFD node and ICD
state the harness can plant. Falsified: both negative scenarios fail when
`gpu_support_level` is made to ignore the probe.

Later superseded: The Vulkan probe this added was later split out of this branch into PR #1036; what remains here is the support-level classifier it was built on.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
…ulkan is absent

Three review findings, all correctness rather than wording.

The endpoint-key clearing was added to the SHARED stdio builder, so it
reached `rocmd` too — and `rocmd` passes `endpoint_key_file: None` for a
loopback service that simply needs no key, which is not the same statement
as "this child may not see one". Its own test expects a keyless child to
inherit the ambient value, and only passed because the variable happened
to be unset. The withholding is now an explicit option the caller asks
for: `rocm` sets it for external plugins, `rocmd` does not. Both
directions are asserted.

The Vulkan probe reported `Absent` when it found no ICD manifest and
enumerated no device — but neither source is exhaustive (the Linux search
ignores `VK_ICD_FILENAMES`, Windows reads only the legacy Khronos key, and
`vulkaninfo` is often not installed), and `Absent` renders as a confirmed
"no Vulkan GPU driver was found - install one". A host with a working
driver could be told to install one it already has. No evidence either way
is now `Unknown`; `Absent` is kept for a probe that found something and
ruled it out.

The support-level scenario required `rocm serve --engine lemonade` for any
detected GPU with no covering family, but a `NoGpuBackend` host is given
no serve command at all. Since the scenario is untagged it would have
failed on exactly that hardware. It branches on the rendered tier now, as
the renderer does.

Scenario ids shift again after merging main (which took examine-48):
examine-49..52 and the references that cite them.

Later superseded: The key-file scoping here was later reset to main's version, and the Vulkan half moved to PR #1036; neither is in this branch's final diff.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
Three leftovers from the split, all of which made this diff claim work it
no longer ships.

`stdio_client.rs` still carried an older, UNCONDITIONAL version of the
endpoint key-file clearing — the one that broke `rocmd`'s contract before
#1034 made it opt-in. Reset to main's. The guard lives in #1034 alone now,
and `rocmd`'s own
`the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key`
is green again here.

`docs/testing.md` still described four support levels and pointed at
examine-50/51/52. This change ships three levels and examine-49; the
fourth level and those scenarios are #1036's, and the text now says so.

The `NoGpuBackend` arm in `assert_support_level_advice_fits` was dead
here — nothing can produce that level without the probe — so it goes back
to #1036 with the level it tests.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
#678's commit messages were rewritten (trailers, review chatter, and notes on
work later moved to this PR or reset to main), which gave every commit on that
branch a new SHA. This branch's own commits were written on the old SHAs, so
after that force-push GitHub's merge-base fell back to main and the PR diff
showed all of #678's changes as this branch's own.

This commit re-parents onto the reworded base. Its tree is byte-identical to
this branch's previous tip -- the rewrite changed only commit messages, and
#678's tree object is unchanged either side of it -- so nothing in this PR's
content moves. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants