Repository navigation
Conversation
…tic ROCm claim ROCMAI-550: on a GPU with no ROCm/HIP build (e.g. gfx90c), `rocm examine` and `rocm engines` kept claiming the embedded Lemonade engine runs "ROCm llama.cpp backend" even though it actually falls back to Vulkan. `rocm install sdk`'s refusal for such a GPU didn't name the actual next step, and neither `rocm serve`'s deployment summary nor `rocm services list` showed which backend/ device an engine actually selected. - add `GpuSupportLevel` (RocmHip/VulkanOnly/Unknown) in rocm-core, derived from the existing `normalize_therock_family`/`gfx_target_has_no_rocm_build` primitives - make the Lemonade engine note in `rocm examine` and `rocm engines` GPU-aware instead of a hardcoded ROCm claim - `rocm install sdk`'s unsupported-GPU errors now suggest `rocm serve <model>` - surface the engine's actual backend/device (read live via the existing `Healthcheck` RPC, never guessed) in `rocm serve`'s deployment summary and `rocm services list` Most of the ticket's `rocm examine`/`install sdk` complaints for gfx90c were already fixed by #648 (ROCMAI-542); this covers what was left. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
CodeQL flagged two assertions as cleartext logging of `api_key`. Those two use `base_summary()`, whose key is `None`, so the literal instances are false positives — but the pattern is not. `render_summary` prints the key by design, and `api_key_client_config_shown_only_when_present` sets a real one, so a failure there copied `endpoint-secret` into CI logs, which are retained and widely readable. Assert on the real text, print a redacted copy. Covers all twelve sites that interpolated a rendered summary, not just the two CodeQL picked. Production output is unchanged: the key is still shown once, and the test pinning that still asserts the exact string. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…l is produced The gfx90c refusals in `therock.rs` gained a `rocm serve <model>` remediation sentence, a second user-observable output change in this PR with no scenario behind it. AGENTS.md §3 wants remediation advice asserted where the message is really printed, not only against the formatter. runtime-19 already opens the one door of the two a lane can reach -- naming the target with `--family` -- so the assertion goes there. The detected-host door needs a Vega APU no runner has and stays unit-covered, as its existing comment already records. Also names the codex-bridge engine inventory at both the static table and the bridge helper as a deliberate scope boundary, instead of only in `lemonade_engine_note`'s doc comment where a reader of the table would miss it. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`unsupported_requested_target_message` is raised from the `--family` and `ROCM_CLI_THEROCK_FAMILY` branches at the top of `resolve_family`, before any host probe runs, so the rejected target says nothing about the GPU in this machine. It had borrowed its sibling's closing sentence promising that `rocm serve <model>` runs on Lemonade's Vulkan backend -- true in `unsupported_host_gpu_message`, which is only reached once host detection itself returns a target with no ROCm build, but false here: a stale env value naming gfx90c refuses just the same on a host whose own GPU is fully ROCm-capable, and there serving would run on ROCm/HIP. Point at `rocm examine` instead. It is true on every host, it is still a next step, and it is the command this PR just taught to report the real support level. runtime-19 was asserting the wrong half of this: it drives `--family gfx90c`, so it was pinning the Vulkan-serving claim green on exactly the ROCm-capable runners where it was false. It now asserts the examine pointer and the negative. Also route `ExamineSummary::rocm_support_line` through `gpu_support_level` rather than re-deriving the same classification, and document the serve summary's `device:` row in README and the manual-testing GPU check. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…tory The codex-bridge engine inventory kept the unconditional "with ROCm llama.cpp backend" wording that ROCMAI-550 removed from `rocm examine` and `rocm engines`, so a bridge client still rendered a ROCm claim to the operator on a host that cannot use it. Both call sites can answer the question: the snapshot already gathers the ExamineSummary carrying the two inputs, and the `engines` tool takes the same cheap host probe `rocm engines` uses. Only the `summary` string changes; the field set and the bridge's JSON shape are untouched. The dashboard TUI pane and rocmd's own catalog keep the static wording deliberately -- the pane has no probe behind it, and rocmd answers for whichever host a client asks about, not the one it runs on. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Resolves the one conflict, apps/rocm/src/main.rs, by taking main's post-#667 file (chat/MCP-bridge extraction removed 185 functions) and re-applying this branch's 23 hunks onto it. One hunk follows its function: `run_internal_mcp_call` now lives in apps/rocm/src/chat.rs, so the `engines` tool's call to `builtin_codex_bridge_engine_inventory` is updated there and `host_gpu_support_level` is imported from the crate root. Verified by construction: the `fn` name sets of main, this branch and the resolution comm out to resolved == main's set plus exactly the five functions this branch adds (`fetch_service_device`, `host_gpu_support_level`, `lemonade_engine_note` and the two new examine tests), with nothing of main's dropped. Also converts this branch's one new non-ASCII production literal to ASCII for the output contract #683 added, which this branch predates: the em dash in `unsupported_host_gpu_message`'s "Try `rocm serve <model>`" sentence becomes a hyphen, matching how #683 converted the rest of the tree. `xtask`'s `every_literal_that_can_reach_a_redirectable_sink_is_ascii` named that literal and only that one; everything else this branch adds with non-ASCII is a comment or lives under the exempt `tests/` prefix. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
… and the bridge `append_examine_engine_inventory` got tests for both support levels when ROCMAI-550 landed; the other two surfaces carrying the same correction did not. `rocm engines` and the codex-bridge inventory (including its `engines` MCP tool) could have gone back to asserting ROCm support on a Vulkan-only host with nothing red. Both were untestable for the same reason: the branch's input is a live sysfs probe, and no CI runner has a GPU that is detected *and* confirmed to have no ROCm build. So each surface now takes the level the way `append_examine_engine_inventory` already did -- the caller probes once, the renderer is given the answer. `render_engine_inventory_text_for_host` and `internal_mcp_engines_tool_result` are those renderers; neither changes what is printed. The three copies of the lemonade substitution collapse into `engine_inventory_note`, so the surfaces cannot drift. The new tests drive the renderers, not that helper: asserting on it would restate the condition rather than prove a surface prints it. `gpu_support_level_for_host_gpu` reads the two `HostGpuSummary` fields once instead of at each call site. They are same-typed positional arguments, so a transposed call site would have classified a Vulkan-only host as `RocmHip` and still compiled; it now has a test. Also drops the dangling `GFX_TARGETS_WITHOUT_ROCM_SUPPORT` reference from the `GpuSupportLevel` doc comment -- no such constant exists in `diagnose.rs` or anywhere else, so the doc pointed readers at nothing. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Resolves the conflicts with main's `main.rs` modularization (Phase 6f,
ROCMAI-208/213/214/215) and addresses the outstanding review findings in
the same pass, because several of the fixes land in code this merge had
to relocate.
Conflict resolution. Every large hunk had an empty `origin/main` side:
main moved the code out of `main.rs`, it did not delete it. The HEAD
block was dropped and this branch's additions re-applied in the module
main moved them to.
- `examine_human_report`, `render_engine_inventory_text_with_paths` /
`_for_host` and `append_examine_engine_inventory` ->
`examine_report_text.rs`, with the seven `append_examine_engine_inventory`
test call sites and the five relocated inventory tests.
- `render_services_text`'s `device:` row -> `services_text.rs`, with the
`!rendered.contains("device:")` assertion that pins the `"unknown"` case.
- `fetch_service_device` -> `engine_ipc.rs`, beside the `engine_request`
it calls; `serve_cmd.rs` imports it from there.
- The codex-bridge inventory stayed in `main.rs`, so its three tests
went back into `main.rs`'s test module.
All 21 test functions this branch adds are present and run; `engine_inventory()`
now carries a third engine (`aim`) from main and the renderers handle it
unchanged. No scenario ids were added by this branch (it only appends
`Then` steps to serve-05, serve-07 and runtime-19), so nothing needed
renumbering.
Review findings.
- `unsupported_host_gpu_message` pins `--engine lemonade` instead of
suggesting a bare `rocm serve <model>`: a configured `default_engine`
out-ranks the host preference in `select_serve_engine`, so the bare
command could attempt vLLM on a GPU with no ROCm build.
- `lemonade_engine_note` now matches `GpuSupportLevel` exhaustively.
`Unknown` no longer inherits the ROCm claim through a catch-all; it
asserts no backend and points at `rocm examine`.
- `fetch_service_device` hands the engine the service's 0600 key file via
`ROCM_SERVE_API_KEY_FILE`, so a public / `--require-api-key` service
does not 401 its own healthcheck and lose the row.
- `rocm services list` probes under one aggregate budget
(`fetch_service_devices`), so the row cannot turn a local-only render
into an unbounded wait.
- `engines/vllm` and `engines/aim` report the device in the same
`<backend> gpu [<n>]` shape `engines/lemonade` does, instead of the
literal `"rocm_gpu"` that README.md and `docs/manual-testing.md` never
described.
- The `services list` e2e step parses the device value, checks it against
the engine that served, and cross-checks it with the `rocm serve`
deployment summary.
Verified on the Linux MI300X box: `cargo fmt --all --check`, `cargo
clippy --workspace --all-targets -- -D warnings`, `cargo clippy --locked
-p e2e-cucumber --test e2e -- -D warnings` and `cargo test --workspace
--all-targets --no-fail-fast` all rc 0, 4823 tests passed / 0 failed.
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…y tier Review round 3 on #678. - `rocm examine` now prints a `rocm_support` line for RocmHip and Unknown as well as VulkanOnly. The Unknown-tier lemonade note sends the user to `rocm examine`, and examine printed nothing for exactly those hosts, so the remediation named a command that could not clear the condition. The note is also reworded to promise the support level, which is what examine delivers, rather than the backend, which only a launch can settle. - The `device:` row carries the GPU name. ROCMAI-550 asks for the backend "and the device name"; on the ticket's own Windows gfx90c host no ordinal is recorded, so the row rendered a bare `vulkan gpu` - a backend and a noun naming nothing. All three engines now record `gpu_name` at launch and format through the shared `rocm_engine_protocol::format_reported_device`, which drops the `gpu` noun when there is no ordinal behind it. - `fetch_service_devices` splits its budget loop into `collect_service_devices` with an injected clock and probe, so the budget-exhausted exit is tested - including that later services are not probed at all. The ~18s worst case (budget plus one in-flight probe) is documented on `SERVICE_DEVICE_PROBE_BUDGET`, on `render_services_text` and in README. - The e2e `device:` step polls for the row instead of panicking on the first look: lemonade reports `unknown` until its own inference probe returns, so a correct implementation legitimately has no row for a short window. - AIM's `rocm gpu` change is pinned by a test that drives the real write-state/read-back/report path. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The `device:` row now renders `<backend> [gpu <n>] [(<name>)]`, so the strings
the advised-command scan has to ignore moved with it: vLLM no longer builds
`rocm gpu {index}`, AIM no longer carries a bare `rocm gpu`, and the two docs
quote the named-device example instead. Also drops a redundant initialiser the
e2e `device:` poll never reads.
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…er removed
Neither vLLM nor AIM holds a `"rocm gpu ..."` string literal any more - both
build the `device:` value through
`rocm_engine_protocol::format_reported_device` - so the entries that excused
those literals no longer match anything and `exclusions_still_match_something`
rejects them. Also drops a stale `"{backend} gpu {index}"` shape note from
vLLM's doc comment.
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main split `engines/lemonade/src/state.rs` into `process.rs` and `device_policy.rs`, so this branch's `reported_device` change moves to `device_policy.rs`: the shared `format_reported_device` renderer plus `gpu_name_from_state`, which reads the name recorded at launch. main's `reported_device_reflects_backend_and_pinned_ordinal` asserted the old `<backend> gpu` shape, which this branch deliberately replaced — a row naming no device. It is superseded by this branch's two tests, relocated with the function, which pin both the named-ordinal row and the ordinal-less one (including a legacy state file with no `gpu_name`). Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Import-only conflict in serve_cmd.rs: #703 added `EngineSelection` to the `engine_runtime_select` import while this branch added `fetch_service_device` to the `engine_ipc` one. Both kept. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
… step failing a correct run Review round 3 on #678. - The `device:` e2e step failed a legitimately correct run. Lemonade reports `unknown` until its own inference probe returns, and the serve summary reads the device once right after `ready`, so the row can be absent on a correct implementation. Only serve-07 being xfail'd on lemonade+linux hid it. The step now tolerates the absence and the window is documented rather than asserted away. - `rocm examine` grew a `rocm_support` line for RocmHip and Unknown with no scenario asserting it. examine-31 covers it, deliberately untagged so each lane's own hardware is the coverage: the claim is cross-checked against `detected_gfx_target` and `compatible_therock_family` from the same report, so "full ROCm/HIP support" on the mock lane or "not supported by ROCm" on an MI300X both fail. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Two conflicts, both from main's ongoing module splits. `examine_report_text.rs`: main replaced the hand-written marker/`writeln!` pair with `write_marked_row`. Took main's call shape and kept this branch's `engine_inventory_note(..., gpu_support_level)`, which is the whole point of the row here. `therock.rs`: main moved the family-resolution cluster into `therock/family.rs`, so both conflict hunks had an empty main side — the code moved rather than went away. This branch's three contributions moved with it: the `--engine lemonade` pin on `unsupported_host_gpu_message`, the `rocm examine` pointer on `unsupported_requested_target_message`, and the two tests pinning the asymmetry between them (plus the extra assertion on `unsupported_gpu_error_names_the_gpu_and_does_not_suggest_families`). Note on numbering: this branch's new `examine-31` is correct for its own file position, and so is #447's. They collide only when the second of the two merges, which is when the loser renumbers to examine-32 — renumbering either now would fail `feature_naming` on its own branch. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main split `host_gpu.rs` into a directory module plus `gfx_target.rs`, so every conflict hunk had an empty main side — the code moved rather than went away. This branch's additions move with it: - `GpuSupportLevel`, `gpu_support_level` and `gpu_support_level_for_host_gpu` join the family tables in `gfx_target.rs`, which is where the predicate they build on (`gfx_target_has_no_rocm_build`) now lives. - `pinned_gpu_name` and its naming helpers stay in `host_gpu.rs`, next to the probes they guard. - `WindowsExamineInventory::amd_display_count` moves to `host_gpu/windows_inventory.rs`. - The exhaustive `rocm_support_line` replaces the VulkanOnly-only version main carried in `host_gpu/render.rs`. That is the point of the change: every support level gets a line, because `lemonade_engine_note` sends Unknown hosts to `rocm examine` and silence there is not an answer. All 13 tests this branch added to `host_gpu.rs` were relocated with their subjects and are verified present and passing, not dropped with the moved region. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main moved the codex-bridge snapshot cluster into `bridge_snapshot.rs`, so that hunk had an empty main side again. This branch's contributions move with it: `builtin_codex_bridge_engine_inventory` takes the host's `GpuSupportLevel` and routes lemonade's row through `engine_inventory_note`, and `build_codex_bridge_snapshot` hoists the `ExamineSummary` out of the struct literal to derive it. The `host_gpu_support_level` helper stays at the crate root, where `chat.rs`'s two MCP `engines` call sites reach it. Three smaller ones: the `examine_report_text.rs` import takes both sides; main changed the serve helper to return its stdout while this branch records it on the world, so it now does both; and runtime-19's new `Then` step sits alongside main's new runtime-20/21. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
#284 landed, so `serve_summary.rs` and `docs/testing.md` now carry its OOM tests and prose alongside this branch's device-row ones; both kept. main also added the stop-confirmation contract to the README and the serve `--help` text, which sits beside this branch's device-line paragraph rather than replacing it. `engine_manages_own_runtime` moved to `engine_install_env`, and `rocm-core` gained `windows_amd_gpu_listing` next to this branch's `pinned_gpu_name` re-export. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ge cut Review round on #678. - The `rocm serve` `--help` doc lost the tail of a sentence when I combined this branch's device-line paragraph with main's stop-confirmation text: it read "...signals the server process directly and exits `rocm services`." and the documented exit-0 behaviour was gone. Restored from main's wording. - `GpuSupportLevel` and the two `gpu_support_level*` functions were appended after `gfx_target.rs`'s `mod tests`. Moved above it. - main added `freeform_prefix_with_quote` and the runtime-22 steps to `runtime_steps.rs` next to this branch's examine-pointing assertion; both kept. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…he row Review round on #678. The note promised that `rocm serve <model> --engine lemonade` reports the backend on its `device:` row, while this PR's own docs say that row can be absent: the serve summary reads the device once, right after `ready`, and lemonade answers `unknown` until its own inference probe returns. The advice now names `rocm services list`, which shows the row once it is known, so the command can actually clear the condition it is printed for. The reasoning is recorded beside the arm rather than left implied. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…efuse Review round on #678. The `Unknown` tier covers two different hosts and was giving both the same advice. A detected-but-unrecognised gfx target has a GPU, so `rocm serve` runs and the engine's own `device:` row settles what this catalog could not. A host with nothing detected is refused outright by the no-usable-GPU pre-flight under the default `gpu_required` policy, so naming that command was advice that could not clear the condition it was printed for (AGENTS.md section 3). The no-detection arm now points at `rocm diagnose`, whose `fix-3-rocm-kernel`, `fix-4-render-group` and `fix-5-amdgpu-load` entries are exactly the reasons a present GPU goes undetected. The sibling arm keeps the serve advice. `no_gpu_advice_names_a_command_that_host_can_run` asserts the behaviour rather than the wording, as asked: it pins the tier through `gpu_support_level` and then checks the rendered line does not name `rocm serve`. Falsified by reverting the arm — the test fails with the offending line quoted. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main split chat.rs into chat/{intent,mcp_bridge,summaries,turn}.rs and moved
the clap definitions out of main.rs into cli.rs, so an empty origin/main side
in these conflicts meant "relocated", not "deleted". The `engines` MCP tool
change and its tests now live in chat/mcp_bridge.rs, and the `serve` and
`services list` help text now lives in cli.rs.
main also added examine-31..34, so this PR's scenario is renumbered to
examine-35 (its docs/testing.md reference follows it); the kfd pre-flight
examine-31 that docs and model_serving.feature name is main's, unchanged.
Dropped `missing_packaged_engine_reason` from examine_report_text.rs: main
deleted that function and its call sites outright.
While porting, restored the `host_gpu_support_level` doc comment an earlier
merge on this branch had welded onto `engine_inventory_note`, and the
`rocm serve` help sentence the same merge truncated mid-clause.
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…t refused `rocm install sdk` and `rocm examine` tell a gfx90c APU to run `rocm serve <model> --engine lemonade`, but nothing showed serve's no-usable-GPU pre-flight accepts that host. It does: a zero KFD compute-node count falls back to the DRM amdgpu card count, so the host the refusal was printed on resolves ordinal 0. Pin that composition off one planted /sys, in a unit test and in an e2e scenario that runs the real binary down the route the CLI sends the user. Also close the test gaps review 5826143385 named: the positive `device:` row and the `unknown` filter now have assertions that run in default CI, the endpoint key file's scoping is pinned, and the examine support-line check asserts the specific command each tier owes instead of any `rocm ` word. Windows reads the display inventory once per `pinned_gpu_name` instead of twice, and the docs say the engine *reports* a device rather than naming the one it observed. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com> # Conflicts: # apps/rocm/src/serve_summary.rs # apps/rocm/src/services_text.rs # tests/e2e-cucumber/features/examine.feature # tests/e2e-cucumber/features/model_serving.feature
…ildren The comment beside the `env_remove` cited `an_external_plugin_child_is_not_handed_the_endpoint_key_file` as the thing keeping it honest. That test did not exist, so the claim was false and the behaviour change — which applies to every external-plugin spawn, not only the device probe — was unguarded. Write it, and give it something to hold onto: the env setup moves into `apply_external_plugin_env`, so the exclusion is assertable without spawning a plugin binary. The test distinguishes CLEARED from never-set, since `Command` records `env_remove` as an explicit `(key, None)` and a bare absence check would pass with the `env_remove` deleted. Verified: deleting it turns the test red. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com> # Conflicts: # tests/e2e-cucumber/features/model_serving.feature
Three conflicts, all where both sides had appended to the same place: - apps/rocm/src/main.rs: main's side was EMPTY, which here means the sigpipe guard and its `engine_stdin_write_under_sig_dfl_is_guarded_not_fatal` test MOVED to crates/rocm-core/src/sigpipe.rs rather than being deleted. Took main's side and dropped the block; keeping ours would have duplicated the test and left it calling a `with_sigpipe_ignored` that main.rs no longer defines. `reset_sigpipe` stays, since main.rs still defines and calls its own. - crates/rocm-core/src/host_gpu.rs: both sides added different tests (ours the single-GPU naming cases, main's the kernel-release reads). Kept both. - tests/e2e-cucumber/features/examine.feature: both sides added scenarios numbered from examine-39. Kept main's 39-47 as canonical and renumbered this branch's ROCm-support-level scenario to examine-48, so one scenario moves instead of nine and the file stays sequential. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`GpuSupportLevel::VulkanOnly` was derived from the gfx-target table alone: "no ROCm build" was read as "Vulkan then", on a machine nobody had looked at. Every surface that renders that level tells the user to go and serve a model on it, so a host with no Vulkan driver installed -- or with only a software rasteriser -- was sent to a backend that cannot start, with no CPU fallback to catch it. Adds a real probe (host_gpu/vulkan.rs): ICD manifests through `host_path` on Linux, the Khronos registry on Windows, and `vulkaninfo --summary` for device enumeration. Filesystem and subprocess evidence rather than dlopening libvulkan, because the loader reads the real ICD list and so could never be faked on the no-GPU lane -- and because running vendor driver code inside `examine` can hang a headless runner. `gpu_support_level` now takes that evidence. An installed driver and an enumerated device are kept apart (`DriverPresentDevicesUnverified`), since collapsing them would replace one unverified promise with another; a platform that cannot be probed keeps `VulkanOnly`, because withholding the only path such a host has, on the strength of a probe that never ran, would be a worse answer than the assumption being replaced. The negative case gets its own level. `NoGpuBackend` is not folded into `Unknown`: both halves there are established (the catalog confirms no ROCm build, the probe finds no driver), whereas `Unknown` means neither could be. Collapsing them commits, in the other direction, the conflation `Unknown` exists to prevent. A software rasteriser is never a GPU path. Mesa lavapipe is a complete Vulkan implementation running on the CPU, so counting it would reintroduce the silent CPU fallback through a backend rather than through a policy change. `rocm engines` and the codex bridge deliberately do not probe: they pass `VulkanStatus::Unknown` and answer exactly as before, so a listing command does not grow a subprocess. `rocm examine` already gathers the probe and is the surface those messages send people to. examine-49/50/51 cover the three answers on the ungated mock lane, which needs neither a Renoir APU nor a Vulkan driver -- only the KFD node and ICD state the harness can plant. Falsified: both negative scenarios fail when `gpu_support_level` is made to ignore the probe. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`engine_stdio_command` set `ROCM_SERVE_API_KEY_FILE` when the caller passed one and otherwise left the variable alone. `rocm` passes `None` for every external engine plugin — a third-party binary — so whenever this process had inherited the variable, the path to a mode-0600 file holding a live endpoint credential was handed straight through. Clear it on the `None` branch. Living in the shared stdio client rather than at the `rocm` call site means `rocmd`'s plugin calls get the same treatment. The test asserts CLEARED rather than absent: `Command` records `env_remove` as an explicit `(key, None)`, so a plain "is it there" check would pass with the clearing deleted. Verified by deleting it. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…e gate `apps/rocm/src/services_text.rs` crossed the 1500-line module-size limit that ROCMAI-889 made blocking — main is at 1458 and this change's 164 added lines took it to 1512, so it is this PR's overflow, not an inherited one. Trimmed the two doc blocks the change itself added, which a reviewer had already called out as heavy. The load-bearing facts stay: the ~18s bound and where its arithmetic lives, why the chat tool pays it rather than rendering a device-less view, and what the probe seam does and does not cover. What went was restatement. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
One blocking finding, plus a few smaller points.
Blocking
classify_vulkan_supportreturnsAbsentin thesoftware_onlybranch before it looks atgpu_icds. A host withradeon_icd.x86_64.jsoninstalled whosevulkaninfolists only llvmpipe (for example no render-group access) is told "no Vulkan GPU driver was found - install one". That advice cannot clear the condition, and it contradicts theicdsfield. No test covers a GPU ICD together with a software-only enumeration. See the inline comment.
Non-blocking
- A software-only ICD list yields a confirmed
Absenteven whenvulkaninfonever answered.LINUX_ICD_DIRSalso omits loader paths, which matters now that this earns a confirmed result. The doc onAbsent(both sources must answer) disagrees with this branch. - When the enumeration is non-AMD only, the report still says "a Vulkan driver is installed" while the detail says none of the devices is AMD (known limitation 1, pinned by a test).
- Windows now spawns a second PowerShell per full gather. Please confirm
rocmddoes not gather on a periodic tick. The registry failure and timeout paths have no injectable-runner test (acknowledged in the PR text). - Nit: the comment block above the inner
matchingpu_support_levelreads as two comments merged.
Decisions to confirm
rocm engines, the MCP engines tool, doctorinfo-20androcm install sdkstill make the unconditional Vulkan promise whileexaminedoes not. Documented as a scope-out; is there a follow-up ticket?
| } | ||
|
|
||
| let software_only = !devices.is_empty() && devices.iter().all(|device| !device.is_hardware()); | ||
| if software_only { |
There was a problem hiding this comment.
software_only returns Absent before gpu_icds is consulted. With radeon_icd.x86_64.json installed and vulkaninfo listing only llvmpipe (e.g. no render-group access) this yields NoGpuBackend and "no Vulkan GPU driver was found - install one" while icds holds a GPU driver. Route to the driver-present-but-unverified path when a GPU ICD exists, naming the software-only enumeration in the detail, and add a test for GPU ICD + software-only device (the lavapipe tests only use lvp_icd).
| // registry key produces. Both sources must have answered before an empty | ||
| // result may be called an absence. | ||
| if icds.is_empty() && devices.is_empty() && !(icds_listed && enumerated) { | ||
| // Which source went unread decides the sentence: a host is owed the |
There was a problem hiding this comment.
This Inconclusive guard only fires when icds is empty, so a lavapipe-only manifest list reaches the confirmed-Absent software-only path with enumerated == false. That gives a confirmed absence without vulkaninfo having answered, against the "both sources must answer" rule documented on Absent. Consider requiring both sources to have answered there too.
| /// what lets the no-GPU CI lane cover this probe at all. | ||
| #[cfg(target_os = "linux")] | ||
| const LINUX_ICD_DIRS: [&str; 4] = [ | ||
| "/usr/share/vulkan/icd.d", |
There was a problem hiding this comment.
The directory list omits paths the loader reads (/usr/local/etc/vulkan/icd.d, ~/.local/share/vulkan/icd.d, XDG_DATA_DIRS entries, /opt/amdgpu*), which matters now that a software-only manifest list can produce a confirmed Absent. A proprietary AMD ICD in an unsearched directory next to Mesa's lavapipe would be reported as absent.
Main landed ROCMAI-703 (#1050), which rewrites the same deployment-summary prose and adds its own serve scenario, so all three conflicts are with it. tests/e2e-cucumber/features/model_serving.feature: main took `serve-60` for its smoke-test-failure scenario. Main's keeps 60; this branch's APU scenario becomes `serve-61` and moves after it, so each number still equals its position. `apps/rocm/src/therock/family.rs`'s prose reference follows. The two `-n "serve-60"` repro filters in docs are main's own and are left alone. docs/testing.md: two independent sections appended at the same place, kept whole. They are unnumbered `##` headings, so nothing renumbers. README.md: both sides rewrite the one paragraph describing the deployment summary, so this is a real merge rather than a choice between them. The summary sentence keeps this branch's `device:` row and main's endpoint/model/ smoke-test list; main's smoke-test-failure paragraph and this branch's "device line is omitted when the engine cannot say yet" paragraph both survive; and the shared closing sentence is kept once. Taking either side whole would have silently dropped the other's user-facing contract. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change: base 3d59dbcb883f0c73df389f6599a7ec3191e55db4 (tip of the stacked branch ROCMAI-550-vulkan-support-messaging) against head a6fb45c976195fe95aaab8d92bb23a652e0aed32, 18 files (+2968/-124). Run as five independent file-group passes (Vulkan probe core; the classification/rendering layer; the CLI/app-layer consumers; the e2e cucumber tests; the docs), plus direct tracing of the two most central functions.
Implements the Vulkan-probing half split out of PR #678 on this same repository; the GpuSupportLevel classifier it extends is #678's own work and is out of scope here. No other tracker ticket is named.
Blocking
-
classify_vulkan_support's device-based "software only" check short-circuits before it looks at the ICD evidence, so a host with a real GPU driver installed can be told none was found —crates/rocm-core/src/host_gpu/vulkan.rs:565-574(the early return), contrastvulkan.rs:577-667(the ICD-aware branch it bypasses)
Thesoftware_onlycheck fires purely on whatvulkaninfoenumerated and returns a confirmedAbsentimmediately, beforegpu_icds(the filtered list of real GPU driver manifests) is even computed. A host withradeon_icd.x86_64.jsoninstalled whosevulkaninfoenumerates onlyllvmpipe(a common shape for arender-group permission problem) is told the confirmed-fact sentence "no Vulkan GPU driver was found - install one (Mesa RADV on Linux...)" — advice that cannot be followed, since the driver is already installed and named inicds. No test in this file pairs a real GPU ICD with a software-only enumeration; the e2e scenario that plants a software-only ICD (examine-52) also plants a software-only enumeration, so this exact combination has never been exercised in either layer.
Confidence 85 · logic · Fix: computegpu_icdsbefore thesoftware_onlycheck (or fold the check into the branch below), and route a software-only enumeration alongside a real GPU ICD toDriverPresentDevicesUnverifiedrather thanAbsent. -
The same function's ICD-based "software only" fallback can confirm
Absentwithoutvulkaninfoever having run, contradicting its own documented invariant —vulkan.rs:657-667, invariant stated atvulkan.rs:63-76("[Absent] reachable only when both sources answer and answer empty")
Whenicdsholds only a software-rasteriser manifest (e.g.lvp_icd.x86_64.json) and the device evidence isunanswered()(vulkaninfo missing, failed, or timed out),gpu_icdsis empty and the function falls through to the finalAbsentreturn — theicds.is_empty()branch is false here, so the comment's own precondition ("both sources answered") holds only for that sibling branch, not this one. The one source that could have found a driver whose manifest sits outside the fixed search directories (vulkaninfo) never ran. No test exercisesicds=[lvp_icd]with an unanswered device source.
Confidence 85 · logic · Fix: requireenumeratedbefore this fallback may returnAbsent; otherwise route toInconclusive, consistent with thegpu_icds.is_empty()branch immediately above it. -
The
NoGpuBackendrendering arm gives unconditional "install the driver" advice with no cross-check against the already-detected Windows driver status, unlike its siblingVulkanUnconfirmedarm —crates/rocm-core/src/host_gpu/render.rs:159-164(missing the check), contrastrender.rs:43-57(vulkan_unconfirmed_next_step, which has it)
vulkan_unconfirmed_next_stepchecksself.driver.status == AMD_DISPLAY_DRIVER_DETECTEDbefore telling a Windows host to install a driver, because on Windows that status means the Vulkan ICD is already present. TheNoGpuBackendarm has no equivalent check and always emits "...no Vulkan GPU driver was found - install one (Mesa RADV on Linux, the AMD display driver on Windows)...". The report's owndriver_status:line would printdriver_status: amd_display_driver_detectedin the same output — this PR's own shared test fixture (examine_summary_with_gpu, reused byexamine_render_does_not_promise_vulkan_on_a_host_that_has_none) defaults to exactly that driver status on a Windows host, but the test asserts only that "install one" appears, not that it is consistent withdriver_status.
Confidence 85 · logic · Fix: apply the same cross-check (ideally factored into a shared helper, see Decisions for the author) inside theNoGpuBackendarm, and add an assertion that a Windows host already showing the AMD display driver is not told to install it underNoGpuBackend. -
The new banner text distinguishing
VulkanOnlyfrom the three other "no usable backend" levels has no test asserting the distinction, so a mutation merging the arms would pass every existing test —apps/rocm/src/examine_report_text.rs:293-349(gpu_execution_line)
Neither of the two banner sentences ("...that is the Vulkan backend, not ROCm." / "...CPU execution is not supported.") appears anywhere outside this one production function. The existing tests that touch these levels only check the per-engine note row and ROCm-claim parity, neither of which contains this banner's wording, so they would stay green even ifVulkanOnlywere folded into the shared arm.
Confidence 85 · logic (test coverage) · Fix: add a direct assertion that theVulkanOnlybanner sentence is present only forVulkanOnlyand absent for the other three levels, mirroring the differential style already used for the note-row tests.
Non-blocking
-
A load-bearing comment on
VulkanSupport::driver_label()states a narrower precondition than the code and its own test exhibit —vulkan.rs:256-258
The comment claims the ICD-name fallback "only ever runs once an AMD hardware device has already been confirmed", but it also runs whenever no AMD device was found at all — a test calls it with an empty device list and still gets a result from this fallback. The actual, well-tested behaviour looks intentional; only the comment is wrong.
Confidence 85 · mechanical (comment accuracy) · Fix: reword to state the actual precondition. -
VulkanStatus::DriverPresentDevicesUnverifiedis never produced by any e2e scenario, and the ICD-only software-rasteriser-exclusion branch inside it is dead at the e2e level —tests/e2e-cucumber/features/examine.feature:766-818,tests/e2e-cucumber/tests/e2e/examine_steps.rs:3421-3686
The four new scenarios reachAvailable,Absent(via two different routes) andInconclusive, but neverDriverPresentDevicesUnverified.examine-52plants anlvp_icdmanifest, but because itsvulkaninfostub enumerates a CPU device, the device-basedsoftware_onlyearly return (first Blocking item above) fires first, so the planted ICD manifest never reachesgpu_icdsand is inert to the scenario's outcome.
Confidence 70 · logic (test coverage) · Fix: add a scenario planting a GPU ICD with novulkaninfoon PATH, or avulkaninfosummary naming only non-AMD hardware. -
README's "Vulkan-unconfirmed" row names the wrong source as the one that can fail to answer —
README.md:612
The row reads "...no ICD, no enumerated device, and novulkaninfoto ask", butclassify_vulkan_supportalso reaches this status whenvulkaninfodid answer and the ICD/registry listing is what failed (realistic on Windows). TheVulkanStatus::Inconclusivedoc comment itself says "at least one of the two sources" — an "or", not specificallyvulkaninfo.
Confidence 85 · documentation · Fix: generalize the row to name either source.
Decisions for the author
-
Other GPU-support surfaces still make the unconditional Vulkan promise this PR removes from
rocm examine— answered-by-intent
rocm engines, the MCPenginestool, androcm install sdk's refusal message all still read the gfx table alone, so the same host can get two different answers about its Vulkan path depending which command is run. The PR description answers this directly: "rocm examineis the only surface this PR routes through the probe... Scoped out rather than fixed, deliberately:family.rsis a pure-function error builder reached before any host probe runs... and one that belongs with therocm engines/ doctorinfo-20surfaces that also still speak from the gfx table." Would otherwise be an architectural finding. Worth confirming whether a follow-up is tracked, since the inconsistency is real and currently untested anywhere. -
The Windows
driver_statuscross-check is implemented ad hoc inside one match arm rather than as a shared predicate — non-blocking-improvement
vulkan_unconfirmed_next_stepis the only place that currently consultsdriver.statusbefore giving install advice; the missingNoGpuBackendcross-check (Blocking, above) is exactly the failure mode this duplication invites. Worth extracting into one helper every advice-giving arm calls.
Positive signals
VulkanEvidence<T>makes "did this source answer" and "what did it find" impossible to conflate — both fields are private, constructible only viaanswered()/unanswered(), and the two monomorphizations make passing one evidence type where the other belongs a compile error rather than a silent status flip.- Every match on
GpuSupportLevel/VulkanStatusin the changed files is exhaustive with no wildcard arm, so a future variant is a compile error rather than a silent fallthrough. - Several tests deliberately assert what a message must not say alongside what it must say — exactly the style that would have caught some of the findings above had it been applied to every arm.
- The PR description and in-code doc comments disclose several real known limitations up front, with reasoning, rather than leaving them for a reviewer to discover.
Deployment notes
- The cross-surface inconsistency described under Decisions for the author widens in practice with this change:
examinenow gives a materially different, evidence-based answer thanrocm engines/install sdkon the same host. - CI for the head commit was green on every completed check at review time; one job (
E2E tests (MI300X, current)) was still queued and had not completed.
What this covered
Every file in the diff (18 files, +2968/-124) between 3d59dbcb883f0c73df389f6599a7ec3191e55db4 and a6fb45c976195fe95aaab8d92bb23a652e0aed32, read completely, split into five file-group passes (Vulkan probe core: vulkan.rs/host_gpu.rs/lib.rs; classification+rendering: gfx_target.rs/render.rs; CLI/app layer: examine_report_text.rs/main.rs/cli.rs/family.rs/bridge_snapshot.rs/driver_plan.rs; e2e cucumber tests; docs), plus direct tracing of classify_vulkan_support and the render.rs rendering arms. Agent-instruction adherence checked against this repository's own contribution guidance: the new vulkan.rs file has an accurate architecture-doc entry, no file approaches the production module-size limit, and the specific internal-guidance citations in the code's own comments were verified accurate. CI was read from the pull request's check runs rather than assumed.
Did not run: a full history pass beyond a light log skim of the classification file's pre-PR commits (nothing of note surfaced). No test suite was executed (no toolchain available in this environment); every finding above is scored on direct code tracing, not execution.
A process note on independence: checking this PR for an existing claimed review (required before starting) necessarily surfaced the text of a prior review already posted at this exact head commit before any blind pass began. That prior review's sole blocking item covers the same code path as the first Blocking finding above; that finding should be read as corroborated rather than purely blind, though it was independently re-derived here by direct tracing of classify_vulkan_support rather than taken on the earlier review's word. The other three Blocking findings, and all Non-blocking findings, were derived by fan-out passes that were not given that review's text and had no access to this PR's discussion thread.
🤖 Automated by an AI agent
| } | ||
|
|
||
| let software_only = !devices.is_empty() && devices.iter().all(|device| !device.is_hardware()); | ||
| if software_only { |
There was a problem hiding this comment.
Blocking. This software_only check fires purely on what vulkaninfo enumerated, and returns a confirmed Absent here before gpu_icds (computed below, line 577) is ever consulted. A host with radeon_icd.x86_64.json installed whose vulkaninfo enumerates only llvmpipe (e.g. a render-group permission issue) is told the confirmed-fact sentence "no Vulkan GPU driver was found - install one (Mesa RADV on Linux...)" — advice that cannot be followed, since the driver is already installed and present in icds. No test in this file pairs a real GPU ICD with a software-only enumeration, and the e2e scenario that plants a software-only ICD (examine-52) also plants a software-only enumeration, so this combination is untested in either layer.
Confidence 85 · logic · Fix: compute gpu_icds before this check (or fold it into the branch below), and route "software-only enumeration + a real GPU ICD present" to DriverPresentDevicesUnverified rather than Absent.
| "only a software (CPU) Vulkan driver is installed; that is not a GPU path".to_owned() | ||
| }; | ||
| return VulkanSupport { | ||
| status: VulkanStatus::Absent, |
There was a problem hiding this comment.
Blocking. This fallback can confirm Absent without vulkaninfo ever having run, contradicting the invariant documented on VulkanStatus::Absent ("reachable only when both sources answer and answer empty", lines 63-76). When icds holds only a software-rasteriser manifest (e.g. lvp_icd.x86_64.json) and the device evidence is unanswered() (vulkaninfo missing/failed/timed out), gpu_icds is empty, so this branch is reached with icds.is_empty() false — the comment two lines up ("both sources answered... this is the only place entitled to deny the driver outright") only actually holds for the icds.is_empty() sibling, not this one. No test exercises icds=[lvp_icd] paired with an unanswered device source.
Confidence 85 · logic · Fix: require enumerated before this fallback may return Absent; otherwise route to Inconclusive, consistent with the gpu_icds.is_empty() branch above it.
| // precisely so that this sentence is never said to one. | ||
| GpuSupportLevel::NoGpuBackend => format!( | ||
| " rocm_support: GPU detected ({}) but not supported by ROCm, and no Vulkan GPU \ | ||
| driver was found - install one (Mesa RADV on Linux, the AMD display driver on \ |
There was a problem hiding this comment.
Blocking. This arm gives unconditional "install one (Mesa RADV on Linux, the AMD display driver on Windows)" advice with no cross-check against self.driver.status, unlike its sibling VulkanUnconfirmed arm (vulkan_unconfirmed_next_step, lines 43-57), which checks driver.status == AMD_DISPLAY_DRIVER_DETECTED before giving the same kind of advice, specifically because on Windows that status means the Vulkan ICD is already installed. The report's own driver_status: line would print driver_status: amd_display_driver_detected in the same output for such a host — this PR's own shared test fixture (examine_summary_with_gpu, reused by examine_render_does_not_promise_vulkan_on_a_host_that_has_none) defaults to exactly that driver status on a Windows host (per the comment at line ~741 describing this fixture), but the test only asserts "install one" appears, not that it's consistent with driver_status.
Confidence 85 · logic · Fix: apply the same driver.status == AMD_DISPLAY_DRIVER_DETECTED cross-check here (ideally factored into a shared helper both arms call), and add an assertion that a Windows host already showing the AMD display driver is not told to install it under NoGpuBackend.
| /// three variants and the Vulkan probe added two more, and the compile error | ||
| /// is what forced the decision below rather than letting either new level | ||
| /// fall through to a sentence about a backend nothing had established. | ||
| const fn gpu_execution_line(level: rocm_core::GpuSupportLevel) -> &'static str { |
There was a problem hiding this comment.
Blocking. Neither of the two banner sentences this function introduces ("...that is the Vulkan backend, not ROCm." / "...CPU execution is not supported.") appears anywhere outside this production function — confirmed by grepping both substrings across apps/rocm/src and tests/. The existing tests touching these levels only check the per-engine note row and ROCm-claim parity, neither of which contains this banner's wording, so a mutation merging the VulkanOnly arm into the shared arm below it would leave every existing test green.
Confidence 85 · logic (test coverage) · Fix: add a direct assertion that the VulkanOnly banner sentence is present only for VulkanOnly and absent for the other three levels, mirroring the differential style already used for the note-row tests.
| .and_then(|device| device.driver_name.clone()) | ||
| .or_else(|| { | ||
| // Restricted to AMD-branded manifests, because this fallback | ||
| // only ever runs once an AMD hardware device has *already* been |
There was a problem hiding this comment.
Non-blocking. This comment claims the ICD-name fallback "only ever runs once an AMD hardware device has already been confirmed", but the fallback also runs whenever no AMD device was found at all — an_installed_driver_with_no_enumerated_device_stays_unverified calls driver_label() with an empty device list and still gets Some("Mesa RADV") from this fallback. The actual, well-tested behaviour looks intentional; only the comment overstates its own precondition, which risks a future change "fixing" this believing the case unreachable.
Confidence 85 · mechanical (comment accuracy) · Fix: reword to state the actual precondition (no AMD+hardware device's own name is available, whether none was confirmed or one was confirmed without a name).
| # gfx table reaches only one: "no ROCm build" does not mean "Vulkan then" on a | ||
| # machine nobody has looked at, and the three scenarios are what keep those | ||
| # answers distinguishable. | ||
| @id:examine-names-the-vulkan-driver-an-unsupported-gpu-can-use @requires-os:linux |
There was a problem hiding this comment.
Non-blocking. These four scenarios (examine-50..53) reach Available, Absent (via two different routes) and Inconclusive, but never VulkanStatus::DriverPresentDevicesUnverified — a status covering three real host shapes (missing vulkaninfo, an empty enumeration with a GPU ICD present, or a non-AMD-only enumeration). examine-52 plants an lvp_icd manifest, but because its vulkaninfo stub enumerates a CPU device, the device-based software_only early return in classify_vulkan_support fires first, so the planted ICD manifest never reaches gpu_icds and is inert to this scenario's outcome.
Confidence 70 · logic (test coverage) · Fix: add a scenario planting a GPU ICD with no vulkaninfo on PATH, or a vulkaninfo summary naming only non-AMD hardware, asserting DriverPresentDevicesUnverified's distinct wording.
| | --- | --- | | ||
| | ROCm/HIP-supported | A TheRock package family covers this GPU. | | ||
| | Vulkan-only | No ROCm build, **and** either a Vulkan GPU driver was found here, or the platform cannot be probed for one at all (anything other than Linux and Windows, or a summary from an older `rocm remote doctor` that predates the probe). The `rocm_support` line says which of the two it is, and only names a driver when one was actually found. | | ||
| | Vulkan-unconfirmed | No ROCm build, and the probe could not reach an answer — no ICD, no enumerated device, and no `vulkaninfo` to ask. | |
There was a problem hiding this comment.
Non-blocking. This row names vulkaninfo specifically as the one that could not be asked, but classify_vulkan_support's (icds_listed, enumerated) match has a (false, true) arm reachable when vulkaninfo did answer and the ICD/registry listing is what failed instead (realistic on Windows, where the registry read is a separate PowerShell call independent of vulkaninfo's presence). The VulkanStatus::Inconclusive doc comment says "at least one of the two sources" — an "or", not specifically vulkaninfo.
Confidence 85 · documentation · Fix: generalize to "...and at least one of the two sources (the ICD/registry listing or vulkaninfo) never answered."
Main landed the WSL librocdxg work (#1091), which appends to the same two examine files this branch does. tests/e2e-cucumber/features/examine.feature: main took `examine-49` through `examine-53`. This branch's support-level scenario becomes `examine-54` and moves after them, so each number still equals its position. Three references follow it: `host_gpu/render.rs`'s doc comment, and the prose and repro filter in `docs/testing.md`. The `examine-49`..`examine-53` references further down that file are main's own librocdxg paragraphs and are left alone. tests/e2e-cucumber/tests/e2e/examine_steps.rs: two independent step blocks appended at the same place, both kept. The split left each side's final function unterminated while sharing a single closing brace, so the brace is reproduced for each rather than closing only the block that came last. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
#678 merged #1091, which took `examine-49` through `examine-53` and moved #678's own support-level scenario to `examine-54`. This branch's four Vulkan scenarios collided with that block, so they become `examine-55` through `examine-58`. They already sat last in the file, so only the numbers moved. Nine references follow them: three comments in `examine_steps.rs`, the `host_gpu/vulkan.rs` row in `docs/architecture.md`, and the four support-level table rows plus the repro filter in `docs/testing.md`. Two neighbouring `examine-51` references are main's librocdxg ones -- the `render.rs` test comment and the librocdxg paragraphs in `docs/testing.md` -- and are left alone. crates/rocm-core/src/host_gpu/render.rs: both sides added a field to the same `format!` template. Resolved as the union -- this branch's `vulkan: {}` and main's extra placeholder before `managed_runtimes` -- because the argument list git had already merged carries both `self.vulkan_line()` and `self.runtime_librocdxg_line()`. Taking either template alone would have left the placeholders and the arguments out of step. docs/testing.md: this branch raises the support levels from three to five, which is its own change, so its wording wins over the base's. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main landed runtime provenance (#1090), which appends its own scenario to examine.feature and took `examine-54`. This branch's support-level scenario becomes `examine-55` and moves after it, so each number still equals its position. Two references follow it: `host_gpu/render.rs`'s doc comment and the prose and repro filter in `docs/testing.md`. The other `examine-54` references in that file belong to main's provenance scenario (`@id:examine-reports-where-the-active-runtime-came-from`) and are left alone. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main's #1090 took `examine-54`, so #678's support-level scenario moved to `examine-55` and this branch's four Vulkan scenarios shift again, to `examine-56` through `examine-59`. Nine references follow them: three comments in `examine_steps.rs`, the `host_gpu/vulkan.rs` row in `docs/architecture.md`, and the four support-level table rows plus their repro filter in `docs/testing.md`. The `examine-55` repro filter immediately above that table is #678's and stays, as do the `examine-54` references to main's provenance scenario. docs/testing.md: this branch raises the support levels from three to five, which is its own change, so its wording wins over the base's. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Independent fresh review of the full diff against this PR's actual base (ROCMAI-550-vulkan-support-messaging @ 30f91a09, not main), 18 files / +2968 -124, head 1cd9609c.
This PR replaces a gfx-table-only Vulkan inference with a real probe (ICD manifests, Windows Khronos registry, vulkaninfo --summary) and is, on the whole, carefully engineered: the evidence-vs-found split (VulkanEvidence<T>), the render text for every status/evidence combination, the five-variant GpuSupportLevel match with no wildcard arm, the Windows display-driver cross-check in vulkan_unconfirmed_next_step, and the new examine-56..59 e2e scenarios are solid and well tested. (The PR body's description of them as examine-50..53 is stale — presumably from rebasing onto the stacked base — but the four cases described are the ones actually present.)
However, the PR currently carries a standing, unresolved CHANGES_REQUESTED review from a maintainer, most recently at 2026-11-17T10:33:03Z, with roughly thirty open, non-outdated inline comments, two marked Blocking. I independently re-derived and confirmed the central one by reading the code and test suite fresh, before reading any review thread:
In classify_vulkan_support (crates/rocm-core/src/host_gpu/vulkan.rs:565-575):
let software_only = !devices.is_empty() && devices.iter().all(|device| !device.is_hardware());
if software_only {
return VulkanSupport { status: VulkanStatus::Absent, /* "only a software (CPU) Vulkan device was found" */ .. };
}
let gpu_icds: Vec<&String> = icds.iter().filter(|icd| icd_is_gpu_driver(icd)).collect();This returns a confirmed VulkanStatus::Absent from devices alone, before gpu_icds (computed on the very next line) is ever consulted. A host with a real AMD GPU ICD manifest installed (e.g. radeon_icd.x86_64.json) whose vulkaninfo enumerates only a software rasterizer — the common case of a user not in the Linux render group — is told the confirmed-fact sentence "no Vulkan GPU driver was found - install one", directly contradicting the ICD evidence already in hand. gpu_support_level then routes this straight to GpuSupportLevel::NoGpuBackend, so the false claim reaches the user verbatim. I confirmed no test exercises this combination: a_software_rasteriser_is_never_a_usable_vulkan_path (vulkan.rs:827) only plants a software-only ICD (lvp_icd.x86_64.json) together with the lavapipe device, never a real GPU ICD alongside a software-only enumeration, and the e2e examine-58 scenario likewise only plants lvp_icd.x86_64.json, not a GPU ICD plus a software-only vulkaninfo.
I verified via git diff <the commit that review was posted against>...HEAD -- crates/rocm-core/src/host_gpu/vulkan.rs that this file is byte-identical to the state that review round was posted against, so the finding stands unaddressed at the current head and that standing review is accurate as of now.
I did not re-post the other open inline comments (several other Absent/Inconclusive edge cases in the same function, a stale "same registry read" line in docs/host-probing.md, missing coverage for the new NoGpuBackend/VulkanUnconfirmed engine-note arms in main.rs, etc.) since they are already on the thread, anchored to the current diff, and I didn't find anything beyond what's already there; I spot-checked a sample of them against the current code and they hold.
CI is green (37/37 checks on the current head).
Given a maintainer's own standing change request already covers what I independently found, and the underlying defect is real, reproducible from the code, and user-visible, I'm requesting changes as well rather than adding a duplicate inline thread.
🤖 Automated by an AI agent
… GPU driver `classify_vulkan_support` returned a confirmed `Absent` from `devices` alone, before `gpu_icds` was computed on the next line. A host with `radeon_icd.x86_64.json` installed whose `vulkaninfo` lists only `llvmpipe` -- the ordinary shape of a missing `render`-group membership -- was told "no Vulkan GPU driver was found - install one", contradicting the `icds` field of the same report with advice it cannot act on. The ICD evidence is now consulted first, and such a host is `DriverPresentDevicesUnverified` with the software-only enumeration named as the reason nothing was confirmed. The ICD-based software-only fallback could likewise confirm `Absent` without `vulkaninfo` ever running: a lavapipe-only manifest list skips the `icds.is_empty()` guard, so the documented "both sources answered" precondition held for its sibling branch only. That fallback now requires `enumerated`, which makes `vulkaninfo`'s answer an invariant of every `Absent` path rather than a property of most of them. `absent_always_has_vulkaninfos_answer_behind_it` sweeps every reachable combination of the two sources to hold that invariant mechanically. `examine-60` is the e2e half: it plants `examine-58`'s enumeration with a GPU manifest beside it, so reading the devices before the manifests fails exactly one of the pair. Signed-off-by: Roman Sirokov <roman@flowrl.com> Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…tate under the lock AIM's `reported_device` hardcoded `rocm` and probed the host for a GPU name whatever the launch's `device_policy` was, so a `cpu_only` container — which `resolve_use_gpu`/`docker::RunSpec` correctly give no /dev/kfd, no /dev/dri and no HIP_VISIBLE_DEVICES — still rendered e.g. `rocm (AMD Instinct MI300X)`. That is the invented device claim this branch exists to remove, shipped in compiled code even though `parse_device_policy` currently refuses `cpu` at the CLI. The policy the launch recorded now decides, and the probe is not made at all when no GPU was attached. AIM also recorded no `gpu_indices`, so on a multi-GPU host it answered a bare `rocm` where lemonade and vLLM answer `rocm gpu <n> (<name>)` for the same hardware — the engine does pin an ordinal, `plan_gpus` resolves it and `HIP_VISIBLE_DEVICES` narrows the container to it. `run_container` now reports the ordinals it attached so the state file and the container cannot disagree. Both engines' `write_state` were a lock-free `fs::write` that bypassed `carry_published_anthropic_endpoint`. The unlocked write predates this branch, but the slow host GPU probe that makes the race practically reachable is added to these two state writes by this branch itself, so the exposure is this branch's to close: routing them through `rocm_core::write_json_state_file` orders them against every merge and carries a sibling thread's publication through the rewrite, as lemonade's already does. The status writers patch through `merge_json_state_file` instead of rewriting the document whole. `healthcheck_service_at` is split out of both handlers, the way lemonade's already was, so a test can drive the real RPC entry point: every device test in the three engines called the private formatter, so reverting a handler's wiring to a hardcoded literal kept the suite green. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ld not `gpu_support_level`'s `(_, Some(_)) => RocmHip` arm ignores the detected target on purpose — `resolve_family` asks it about a `--family` the user named, where there may be no detected target — but every call in the suite passed Some/Some, None/None or Some/None, so narrowing the arm to `(Some(_), Some(_))` was invisible. `gpu_name_for_single_gpu_host`'s ordinal guard was in the same position: the only test with a non-zero ordinal also supplied a host count of 2, which the check two lines below already declines on, so the guard could be deleted outright with the suite still green. Both mutations now turn it red. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…river_status The `NoGpuBackend` arm emitted "install one (Mesa RADV on Linux, the AMD display driver on Windows)" unconditionally, while its sibling `VulkanUnconfirmed` arm already consulted `driver.status` first -- because on Windows the AMD display driver *is* the Vulkan ICD, so its presence contradicts that advice. The same report printed `driver_status: amd_display_driver_detected` a few lines above the sentence telling the reader to go and install it. The check is now one predicate both arms call, which is the shape that would have kept the second arm from shipping without it. The confirmed absence is unchanged -- the probe did look and found no Vulkan GPU device -- only the step offered differs, and `vulkaninfo` is not offered there the way it is under `VulkanUnconfirmed`, since reaching `Absent` already takes its answer. `examine_render_does_not_promise_vulkan_on_a_host_that_has_none` asserted "install one" against the shared Windows fixture, which defaults to exactly the contradictory driver status; it now plants the host it is actually about. Signed-off-by: Roman Sirokov <roman@flowrl.com> Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…et it read both renderers `assert_services_list_reports_device`'s comparison against `rocm serve`'s summary was dead twice over. `world.serve_summary` comes from `run_rocm_bounded`, which redirects stdout to a file, so `serve_cmd.rs`'s `summary_mode` is false and the CLI prints `print_managed_launch_plain`, which has no device row in any state; and `device_row_value` looked for `"device:"` while `render_summary` writes a padded label with no colon, so it could not have matched a summary even if one carried the row. The parser now reads both renderers' shapes (and still refuses the `device_policy:` row the serve plan prints), and the step says in full that on today's lanes the comparison does not run and the backend assertion above it is what discriminates — rather than letting the step's name imply coverage it has not got. Making it live means driving the vLLM and Lemonade per-PR serve canaries through a `TuiSession`, which is a change to those canaries' own harness. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…end levels `gpu_execution_line` tells a `VulkanOnly` host which backend it has and tells the other three that the CPU is not an alternative. Neither sentence appeared anywhere outside that function, so merging the `VulkanOnly` arm into the shared one below it left every test green -- including the ROCm-parity sweep, which only reads "ROCm GPU execution is required" and is satisfied by either. Asserted as the discrimination rather than as either sentence: each level requires its own banner present and the other's absent, and the three no-usable-backend levels are swept rather than sampled. Signed-off-by: Roman Sirokov <roman@flowrl.com> Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…onade serve flow `device_worth_reporting` claimed the `"unknown"`-until-ready gate was uniform and cited vLLM and Lemonade as sharing it. They do not: vLLM and AIM answer as soon as a state file exists, while Lemonade keeps saying `"unknown"` until its own inference probe returns. That difference is the whole reason `SERVE_DEVICE_PROBE_BUDGET` exists, so the comment now names it. The probe bound alongside it also cites AIM's own `HEALTHCHECK_TIMEOUT_MS`, which it relied on without naming. `docs/architecture.md`'s `gfx_target.rs` entry still described pure lookups after that file gained the whole `GpuSupportLevel` classifier; the gate checks presence, not accuracy, so nothing caught it. `docs/manual-testing.md` dropped `--foreground` from the Lemonade serve example but still called the result "that foreground server" 34 lines later, with no way to stop it. The prose now matches a backgrounded service and gives the stop command, as the parallel flow below it already does. Also drops review-thread references from source comments and a note to reviewers from docs/testing.md; each says the reason instead. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`docs/host-probing.md` said Windows devices come from "the same registry read" -- they come from `vulkaninfo` there as on Linux, the registry is the ICD source only -- and both it and the `vulkan.rs` architecture row still described an `enumerated` flag that `vulkan_devices_from_capture` no longer returns; it returns a `VulkanEvidence<T>` whose `did_answer()` carries that question. Both now state the invariant as implemented: `vulkaninfo` must have answered before anything may be reported as a confirmed absence, and a GPU manifest outranks a software-only enumeration. README's Vulkan-unconfirmed row named `vulkaninfo` as the source that could not be asked; `Inconclusive` is reachable when `vulkaninfo` answered and the ICD/registry listing is what failed, which is realistic on Windows. Generalized to either source, as the `VulkanUnconfirmed` doc comment now is too. Also splits the two comments that had merged above the inner `match` in `gpu_support_level`. Signed-off-by: Roman Sirokov <roman@flowrl.com> Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Thanks — three rounds piled up and I only just caught them. Everything below is at Blocking1.
|
One conflict, in `apps/rocm/src/chat/mcp_bridge.rs`, and it is the code-was-removed shape rather than two additions colliding. Main's #980 deletes `internal_mcp_arguments` outright and replaces it with `internal_mcp_tool_call` plus `mcp_call_cli_arguments`, so that a pasted MCP envelope is unwrapped in one named step instead of silently inside the dispatcher -- the point of that change being that the approval card and the executed call cannot disagree. This branch still carried the old function. Resolved by taking main's design whole and keeping only this branch's own addition from the other side, the `engines` MCP tool-result helper. Keeping this branch's `internal_mcp_arguments` would have reinstated the unwrapping main had just removed, which is the opposite of what #980 is for; grep confirms no caller of it survives. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
docs/testing.md: the base still describes Vulkan-only as inferred from the gfx table with three support levels; this branch replaces that inference with a probe and has five. This branch's wording wins, as before. The merge also went clean in `gfx_target.rs` and did not compile. #678 added two `gpu_support_level` cases for its `(_, Some(_))` arm, written against its own two-argument signature; this branch adds a third `vulkan` parameter, so both arrived calling a function that no longer exists in that shape. Both now pass `VulkanStatus::Absent`, matching the assertions immediately above them, which deliberately vary the Vulkan status to show a covering family wins outright -- so the merged cases keep #678's intent (a detected target cannot outvote a resolved family) and gain this branch's stronger premise: it holds even when the probe found nothing. Caught by `cargo clippy --workspace --all-targets`, not by the merge. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Code Review: PR #1036 Vulkan GPU Probing for rocm-cli
Repository: ROCm/rocm-cli
PR: #1036 (adds Vulkan GPU probe logic + classification state machine)
Review Method: Blind multi-pass review per review-code skill methodology
Result: REQUEST_CHANGES (3 blocking findings ≥70 confidence)
Summary
This PR adds comprehensive Vulkan GPU detection to rocm-cli, including ICD manifest discovery (Linux/Windows), vulkaninfo parsing, and classification of Vulkan capability (Available/Absent/Inconclusive/DriverPresentDevicesUnverified). The core algorithm is well-tested and correct, with exhaustive unit coverage and matching e2e scenarios.
Three blocking findings prevent approval:
- Doc-comment contradiction in VulkanStatus enum (lines 87-90 vs implementation)
- Untested Windows registry integration path (violates AGENTS.md §3)
- Contradictory error message in rocm install sdk vs rocm examine (violates remediation contract)
Blocking Findings
Finding 1: VulkanStatus::Inconclusive doc-comment contradiction (confidence 75)
File: crates/rocm-core/src/host_gpu/vulkan.rs:87-90
Category: Logic/mechanical
Severity: Medium
The enum documentation claims Inconclusive's precondition includes "no ICD manifest," but the classifier (lines 695-707) correctly reaches this state with non-empty ICD lists. The test a_lavapipe_only_install_is_not_a_confirmed_absence_until_vulkaninfo_answers confirms this: it passes icds = ["lvp_icd.x86_64.json"] and asserts status == Inconclusive.
The behavior is correct (software-only ICD without vulkaninfo answer should be Inconclusive). The documentation is wrong.
Required fix: Update the enum doc comment to reflect actual logic. Suggested: "The probe came away with nothing either way: only software-rasterizer ICDs, no enumerated device, and at least one of the two sources that never answered..."
Finding 2: Untested Windows registry PowerShell integration (confidence 72)
File: crates/rocm-core/src/host_gpu/vulkan.rs:450-469
Category: Architectural (platform-gated code)
Severity: Medium
The real #[cfg(windows)] fn detect_icds() performs a subprocess call to PowerShell against the Khronos registry key. While unit tests cover the pure parsing halves (windows_icds_from_capture, parse_windows_vulkan_icd_listing) and inspect the PowerShell script string, the full integration path—capturing live registry output through the subprocess—has zero test coverage.
The file's own doc comment and docs/host-probing.md:213 claim this is "covered by CI's windows-build-and-test job," but repo-wide grep for HKLM/Khronos returns only code references, not test references. The only scenario tagged @requires-os:windows in examine.feature is unrelated (AMD display-driver version/date).
Per AGENTS.md §3: "platform-gated subprocess logic must be tested on its own gate."
Required fix: Add a unit test for the Windows registry integration path. Options:
- Mock the subprocess capture to inject realistic PowerShell output and assert end-to-end behavior
- Explicitly document why this path is untested and update code comments to match (declare only the parsing seam as cross-platform tested)
Finding 3: Contradictory Vulkan remediation across same session (confidence 80)
File: apps/rocm/src/therock/family.rs:244-250 (unsupported_host_gpu_message)
Category: Correctness/consistency bug
Severity: High
The unsupported_host_gpu_message function unconditionally claims "the Vulkan backend can still be used" and recommends rocm serve <model> --engine lemonade, without consulting the new Vulkan probe result.
Unlike its sibling unsupported_requested_target_message (which handles user-named targets that may not exist on this host), this call site:
- Only executes after the host's own detected GPU architecture is known
- Runs on an already-failing, non-hot path (extra probe cost is negligible)
- Directly contradicts
rocm examineoutput in the same session when VulkanStatus is actually NoGpuBackend
Example conflict: On a host with no AMD Vulkan drivers:
rocm examine→ "no Vulkan driver was found, so no GPU backend can run here"rocm install sdk→ "the Vulkan backend can still be used; tryrocm serve <model> --engine lemonade"
The remediation command will not clear the condition it was printed for.
This violates AGENTS.md §3: "remediation advice naming a command — the named command must exist, accept those flags, and actually clear the condition that printed it."
Required fix: Either:
- Thread the Vulkan probe result into this message and condition the Lemonade recommendation on a successful probe, OR
- Defer to
rocm examinefor Vulkan status (consistent with how other CLI commands separate concerns)
Verified Passes (No Issues >=70 Confidence)
- GPU classification state machine: All 25 (family, VulkanStatus) state combinations fully tested and logically correct
- E2E Vulkan probe scenarios (examine-56 through examine-60): All 5 scenarios correctly specify and exercise the expected outcomes
- Documentation/architecture consistency: All file references properly linked, stale claims removed, test descriptions updated
- AGENTS.md compliance: No internal keywords, DCO/commit signing enforced, FFI safety checklist validated, platform-gated tests on correct gates (only Windows registry path is gap)
CI Status & Merge Readiness
This PR requires fixes to the three blocking findings before merge. The underlying architecture and algorithm are sound; the issues are localized:
- One documentation error (10 minutes to fix)
- One missing test (1-2 hours to implement)
- One message logic contradiction (1-2 hours to wire probe or refactor)
🤖 Automated by an AI agent
| /// The probe came away with nothing either way: no ICD manifest, no | ||
| /// enumerated device, and at least one of the two sources that never | ||
| /// answered — `vulkaninfo` missing or failing, or (on Windows) a registry | ||
| /// listing that could not be run. |
There was a problem hiding this comment.
Doc-comment contradiction: Inconclusive precondition claim doesn't match implementation
The enum doc (lines 87-90) claims Inconclusive's precondition includes "no ICD manifest," but the classifier reaches this state with non-empty ICD lists. This is confirmed by the test a_lavapipe_only_install_is_not_a_confirmed_absence_until_vulkaninfo_answers, which passes icds = ["lvp_icd.x86_64.json"] and asserts status == Inconclusive.
The behavior is correct (software-only ICD without probe answer should be Inconclusive). The doc needs updating to reflect actual logic: change "no ICD manifest" to something like "no ICD manifest that confirms a GPU driver" or "only software-rasterizer ICDs found."
| WINDOWS_VULKAN_ICD_SCRIPT, | ||
| ], | ||
| )) | ||
| } |
There was a problem hiding this comment.
Missing test coverage: Real Windows registry PowerShell integration untested
The #[cfg(windows)] fn detect_icds() at lines 450-469 performs a real subprocess call to PowerShell against the Khronos registry key. While unit tests cover the pure parsing half (windows_icds_from_capture, parse_windows_vulkan_icd_listing) and inspect the PowerShell script string itself, the full integration path—capturing live registry output through the subprocess—has zero coverage.
The file's own doc comment and docs/host-probing.md:213 claim this is "covered by CI's windows-build-and-test job," but repo-wide grep for HKLM/Khronos returns only code references, not test references. Per AGENTS.md §3, platform-gated subprocess code must be tested on its gate.
Add a unit test that either:
- Mocks the subprocess capture to test integration with realistic PowerShell output, or
- Documents why full-path testing is deferred and updates the code comment to reflect that only the parsing seam is tested cross-platform
`VulkanStatus::Inconclusive`'s doc gave "no ICD manifest" as a precondition, and the classifier reaches that status with a non-empty `icds`: a host whose only manifest is a software rasteriser, with no `vulkaninfo` answer behind it, lands here rather than in `Absent`, because the ICD search is not exhaustive and so lavapipe-is-all-I-can-see does not rule out a GPU driver the probe never reads. The behaviour is right and already pinned by `a_lavapipe_only_install_is_not_a_confirmed_absence_until_vulkaninfo_answers`. Only the doc was wrong, so only the doc changes, and it now names that test so the next reader can check the claim instead of taking it. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The file and docs/host-probing.md both claimed the Khronos registry read was "covered by CI's windows-build-and-test job". Nothing calls `detect_icds` on Windows, so that job compiles the call and never runs it: the program name, the flags and the script handed to PowerShell had no test at all. Dropping `-Command`, or losing `-NonInteractive` so a prompting profile turns the probe into a 1.5s timeout, would have shipped and shown up only on real hardware as a permanently inconclusive Vulkan line. `windows_icds_through` takes the subprocess as an argument, so everything the Windows path does except starting the process now runs on every lane, argv included, through to `classify_vulkan_support`. What is left uncovered is PowerShell's own behaviour on a real host, and both docs now say exactly that instead of naming a job that does not test it. That residue is bounded: every such failure arrives as a `None`, which the seam routes to `Inconclusive` rather than to a claim about the machine. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
`unsupported_host_gpu_message` closed, unconditionally, with "the Vulkan backend can still be used - try `rocm serve <model> --engine lemonade`", without ever consulting the probe this branch exists to add. On a host with no AMD Vulkan driver one session could print "no Vulkan driver was found, so no GPU backend can run here" from `rocm examine` and the opposite from `rocm install sdk`, and the command it named could not clear the condition it was printed for, which AGENTS.md §3 requires of remediation advice. The message now classifies with this host's real `VulkanStatus` through the same `gpu_support_level` the examine line uses, and takes its wording from the same shared constants: only `VulkanOnly` keeps the Lemonade recommendation, `VulkanUnconfirmed` makes it conditional on what `vulkaninfo` reports, and `NoGpuBackend` offers the driver install instead. The refusal itself is untouched. Its gate is `target_is_vulkan_only`, which still classifies with `Unknown`, because whether a ROCm build exists is a fact about the catalog and must not turn on an unrelated Vulkan probe; the probe is paid for inside the already-failing branch. The sibling `unsupported_requested_target_message` keeps the unconditional claim for the same reason it keeps its own closing sentence: it is about a target the user named, which may not be the GPU in this machine, so no probe of this host bears on it. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
All three blocking findings are addressed in Finding 1 —
|
20c48b4 to
f04357f
Compare
`GpuSupportLevel::VulkanOnly` was derived from the gfx-target table alone: "no ROCm build" was read as "Vulkan then", on a machine nobody had looked at. Every surface that renders that level tells the user to go and serve a model on it, so a host with no Vulkan driver installed -- or with only a software rasteriser -- was sent to a backend that cannot start, with no CPU fallback to catch it. Adds a real probe (host_gpu/vulkan.rs): ICD manifests through `host_path` on Linux, the Khronos registry on Windows, and `vulkaninfo --summary` for device enumeration. Filesystem and subprocess evidence rather than dlopening libvulkan, because the loader reads the real ICD list and so could never be faked on the no-GPU lane -- and because running vendor driver code inside `examine` can hang a headless runner. `gpu_support_level` now takes that evidence. An installed driver and an enumerated device are kept apart (`DriverPresentDevicesUnverified`), since collapsing them would replace one unverified promise with another; a platform that cannot be probed keeps `VulkanOnly`, because withholding the only path such a host has, on the strength of a probe that never ran, would be a worse answer than the assumption being replaced. The negative case gets its own level. `NoGpuBackend` is not folded into `Unknown`: both halves there are established (the catalog confirms no ROCm build, the probe finds no driver), whereas `Unknown` means neither could be. Collapsing them commits, in the other direction, the conflation `Unknown` exists to prevent. A software rasteriser is never a GPU path. Mesa lavapipe is a complete Vulkan implementation running on the CPU, so counting it would reintroduce the silent CPU fallback through a backend rather than through a policy change. `rocm engines` and the codex bridge deliberately do not probe: they pass `VulkanStatus::Unknown` and answer exactly as before, so a listing command does not grow a subprocess. `rocm examine` already gathers the probe and is the surface those messages send people to. examine-49/50/51 cover the three answers on the ungated mock lane, which needs neither a Renoir APU nor a Vulkan driver -- only the KFD node and ICD state the harness can plant. Falsified: both negative scenarios fail when `gpu_support_level` is made to ignore the probe. Later superseded: The Vulkan probe this added was later split out of this branch into PR #1036; what remains here is the support-level classifier it was built on. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ulkan is absent Three review findings, all correctness rather than wording. The endpoint-key clearing was added to the SHARED stdio builder, so it reached `rocmd` too — and `rocmd` passes `endpoint_key_file: None` for a loopback service that simply needs no key, which is not the same statement as "this child may not see one". Its own test expects a keyless child to inherit the ambient value, and only passed because the variable happened to be unset. The withholding is now an explicit option the caller asks for: `rocm` sets it for external plugins, `rocmd` does not. Both directions are asserted. The Vulkan probe reported `Absent` when it found no ICD manifest and enumerated no device — but neither source is exhaustive (the Linux search ignores `VK_ICD_FILENAMES`, Windows reads only the legacy Khronos key, and `vulkaninfo` is often not installed), and `Absent` renders as a confirmed "no Vulkan GPU driver was found - install one". A host with a working driver could be told to install one it already has. No evidence either way is now `Unknown`; `Absent` is kept for a probe that found something and ruled it out. The support-level scenario required `rocm serve --engine lemonade` for any detected GPU with no covering family, but a `NoGpuBackend` host is given no serve command at all. Since the scenario is untagged it would have failed on exactly that hardware. It branches on the rendered tier now, as the renderer does. Scenario ids shift again after merging main (which took examine-48): examine-49..52 and the references that cite them. Later superseded: The key-file scoping here was later reset to main's version, and the Vulkan half moved to PR #1036; neither is in this branch's final diff. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Three leftovers from the split, all of which made this diff claim work it no longer ships. `stdio_client.rs` still carried an older, UNCONDITIONAL version of the endpoint key-file clearing — the one that broke `rocmd`'s contract before #1034 made it opt-in. Reset to main's. The guard lives in #1034 alone now, and `rocmd`'s own `the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key` is green again here. `docs/testing.md` still described four support levels and pointed at examine-50/51/52. This change ships three levels and examine-49; the fourth level and those scenarios are #1036's, and the text now says so. The `NoGpuBackend` arm in `assert_support_level_advice_fits` was dead here — nothing can produce that level without the probe — so it goes back to #1036 with the level it tests. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
#678's commit messages were rewritten (trailers, review chatter, and notes on work later moved to this PR or reset to main), which gave every commit on that branch a new SHA. This branch's own commits were written on the old SHAs, so after that force-push GitHub's merge-base fell back to main and the PR diff showed all of #678's changes as this branch's own. This commit re-parents onto the reworded base. Its tree is byte-identical to this branch's previous tip -- the rewrite changed only commit messages, and #678's tree object is unchanged either side of it -- so nothing in this PR's content moves. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Split out of #678 on review. Stacked on it — the base is
ROCMAI-550-vulkan-support-messaging, because theGpuSupportLevelclassifier this extends is that PR's work. Review #678 first; this diff is only what Vulkan adds on top (~1.4k lines).Symptom
rocm examinetold a gfx90c APU it could run models through Lemonade's Vulkan backend on the strength of the gfx table alone. Nothing had looked for a driver. On a host with none, that is an instruction the user cannot carry out — the same class of problem as the static ROCm claim #678 fixes, one level down.Change
Probe, rather than infer:
vulkaninfo --summarywhere it existsA fourth support level,
NoGpuBackend, carries the confirmed answer, and avulkan:line prints what was found.Where the probe refuses to guess
Absentis reserved for a probe that found something and ruled it out, and it always takesvulkaninfo's own answer: avulkaninfothat ran and enumerated nothing, an enumeration of only software devices with no GPU ICD beside it, or a software-only ICD list thatvulkaninfodid not contradict. Neither manifest source is exhaustive enough to deny a driver on its own:VK_ICD_FILENAMES/VK_DRIVER_FILESDriverskey, which current AMD drivers need not populatevulkaninfois frequently not installed on eitherAnything short of that yields
Inconclusive— which reaches the user as a distinctVulkanUnconfirmedsupport level, not as a confirmed absence. Inferring absence from an unread source would tell a host with a working driver to install one it already has.NoGpuBackendrenders as a confirmed fact, so it has to be earned.A software rasteriser never counts as a GPU path — lavapipe/llvmpipe never reach
Available. But neither does it deny a GPU path a manifest asserts: a host carryingradeon_icd.x86_64.jsonwhosevulkaninfoenumerates onlyllvmpipe(the ordinary shape of a missingrender-group membership) isDriverPresentDevicesUnverified, since "install a Vulkan driver" would contradict theicdsfield of the same report.Scenarios
examine-56throughexamine-60cover the answers on the ungated mock lane, which needs neither a Renoir APU nor a Vulkan driver — only the KFD node, ICD state andvulkaninfostub the harness can plant.examine-56Availableexamine-57vulkaninfothat exits 0 and reports nothingAbsentexamine-58Absentexamine-59vulkaninfoon PATHInconclusiveexamine-60vulkaninfoenumerates only llvmpipeDriverPresentDevicesUnverifiedexamine-58andexamine-60plant the same enumeration and differ only in which manifest sits beside it, so an implementation that reads the devices before the manifests fails exactly one of the pair.Falsified: both negative scenarios fail when
gpu_support_levelis made to ignore the probe;examine-60fails when the software-only early return is made to ignore the ICD evidence.Known limitations
1. A non-AMD-only enumeration still routes to a Vulkan path. A successful enumeration that returns only non-AMD hardware (an NVIDIA box, say) still falls through to "driver present, devices unverified" rather than counting as evidence against an AMD Vulkan path. Raised in review on #678; not fixed here, and pinned by a test so it is a known state rather than an accident.
2.
rocm install sdk,rocm enginesand doctorinfo-20still promise Vulkan without consulting the probe.rocm examineis the only surface this PR routes through the probe.therock::family::unsupported_host_gpu_message— the errorrocm install sdkraises for a detected GPU with no ROCm build — still ends unconditionally with "the Vulkan backend can still be used. Tryrocm serve <model> --engine lemonade". That is the same sentence, said on the same trigger condition (an AMD GPU detected, no ROCm build), that this PR's module doc names as the motivating defect and thatrender.rs'sexamine_render_does_not_promise_vulkan_on_a_host_that_has_nonenow pins out ofexamine.rocm enginesand the MCPenginestool passVulkanStatus::Unknown, which maps toVulkanOnly, so the same host can get two different answers in one session.Scoped out rather than fixed, deliberately:
family.rsis a pure-function error builder reached before any host probe runs, so routing it through the probe means giving it aVulkanStatusparameter and threading one down fromresolve_family's callers — a change to the install path's call graph rather than to a string, and one that belongs with therocm engines/ doctorinfo-20surfaces that also still speak from the gfx table.rocm enginesadditionally has a cost argument against it: spawningvulkaninfo(and PowerShell on Windows) for a plain listing command is a per-invocation subprocess the listing does not otherwise need. No follow-up ticket is filed for this yet — it is a disclosed scope-out, not tracked work.rocm examineremains the surface that answers from evidence, and this PR does not claim otherwise.3. Confirmed absence is narrower than "no driver is installed".
VulkanStatus::Absent(and soNoGpuBackend) always needsvulkaninfoto have answered. On a real driverless Linux hostvulkaninfo --summaryprintsFound no drivers!and exits 1, andcapture_optional_commandcollapses non-zero exit, a missing binary and a timeout into oneNone— so that host classifies asInconclusive, notAbsent.examine-57plants avulkaninfothat exits 0 with an empty device list, which is the shape that does earn the confirmed absence, so that scenario mirrors the implementation rather than the most common real tool behaviour.The other route to
Absentis not affected and is genuinely reachable: a host whose evidence contains only a software rasteriser is ruled out on the evidence it did produce, which is whatexamine-58covers.Treating
exit 1plus the loader message as an answer was considered and rejected for now. It means threading exit status and stderr out ofcapture_optional_command, which is shared well beyond the Vulkan probe, and it would make a confirmed "no GPU backend" depend on matchingFound no drivers!— loader text that is not a stable contract. Avulkaninfothat exits 1 for any other reason would then be promoted into exactly the unearned absence claim this PR exists to stop making.Inconclusiveis the conservative classification, and the line it renders already namesvulkaninfoas the step that settles it.Cost
The probe runs once per full
ExamineSummary::gather(), so every caller pays onevulkaninfo(and on Windows one extra PowerShell start), each bounded by the shared 1.5 sOPTIONAL_COMMAND_TIMEOUT. It is not on any periodic path:rocmd's watcher tick callsgather_gpu_snapshot_for_configonly, which takes theamd-smiroute and never reaches the probe, andbuild_bridge_snapshot(which does gather) is reached only fromrocmd mcpandrocmdCLI request handling.rocm mcp-call examine|doctorgathers twice and so pays up to two probes — a pre-existing double gather, now with a larger constant.Verification
Linux MI300X:
cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo clippy -p e2e-cucumber --test e2e -- -D warnings,cargo test -p rocm-core,cargo test -p rocm --bin rocm,cargo test -p e2e-cucumber --test feature_naming,cargo xtask check-architecture-doc/check-crate-edges/check-module-size, andcargo xtask e2e -- -n '(examine-56|examine-57|examine-58|examine-59|examine-60)'.Not verified anywhere: the Windows registry branch of the probe end to end. Neither this Mac nor the Linux build box can build the MSVC target (
ring/aws-lc-sysneed a real Windows toolchain), so CI'swindows-build-and-testis the only check on the live registry read.The pure seam around it is covered on every platform:
windows_icds_from_captureturns the capture'sOption<String>intoVulkanEvidence<String>, and its tests drive theNonecase (missingpowershell, non-zero exit, or timeout —capture_optional_commandcollapses all three) alongside the empty-key and populated-key cases. What remains unexercised is the registry script itself against a real registry, and the PowerShell spawn's own cold-start behaviour; closing that needs an injectable runner on the Windows job, deferred to a follow-up with a Windows dev environment.Support levels this adds
VulkanOnlyVulkanUnconfirmedNoGpuBackendvulkaninfoanswered and a GPU path was ruled outdriver_statusalready shows the AMD display driver, in which case repairing it is the step)VulkanUnconfirmedexists becauseNoGpuBackendrenders as a confirmed fact andis serialized. On Windows the Khronos
Driverskey is legacy andvulkaninfoisrarely on PATH, so a working Adrenalin laptop can leave both sources empty —
telling it "no GPU backend" would be worse than the static claim this change
replaces.
examine-59covers that row;examine-57cannot, since it plants avulkaninfothat answers (exit 0, empty list), which is a confirmed absence.