Skip to content

feat(examine,serve): report real GPU/backend support instead of a static ROCm claim - #678

Open
r0x0r wants to merge 65 commits into
mainfrom
ROCMAI-550-vulkan-support-messaging
Open

r0x0r wants to merge 65 commits into
mainfrom
ROCMAI-550-vulkan-support-messaging

Conversation

@r0x0r

@r0x0r r0x0r commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

ROCMAI-550: report the GPU/backend support this host actually has

rocm examine and the engine notes asserted "default embedded Lemonade server with ROCm llama.cpp backend" on every host, and install sdk offered package families for GPUs ROCm has never shipped a build for. On a Ryzen 4000/5000 laptop whose Vega iGPU is gfx90c — the machine this ticket came from — that is a claim about hardware nobody looked at.

This PR makes the support claim come from one place and say only what the host supports. Body regenerated from the final diff at @juhovainio's request; the previous version omitted two changes and listed no deferrals.

The one classifier

rocm_core::gpu_support_level derives the level from the two facts ExamineSummary already computes — the detected gfx target and whichever TheRock family covers it:

Level Meaning
ROCm/HIP-supported A TheRock package family covers this GPU.
Vulkan-only Recognized, confirmed to have no ROCm build.
unknown No GPU detected, or a target this build's catalog does not recognize.

unknown deliberately covers both "nothing detected" and "unrecognized target". Conflating the latter with "confirmed no ROCm build" would turn a gap in this build's catalog into a claim about the card, so an empty compatible_therock_family never reads as a failed probe.

A rocm_support line is now printed on every host, not only the unsupported one — silence could not answer the question lemonade_engine_note sends users to rocm examine to ask.

Every surface that makes a support claim renders from this one function, with exhaustive matches, so adding a level is a compile error rather than a silently stale sentence.

What each level advises

The advice has to be a command that can actually clear the condition it is printed for:

  • ROCm/HIP-supported — rocm install sdk, or, when a managed runtime is already installed, says so instead of recommending an install that is already done.
  • Vulkan-only — rocm serve <model> --engine lemonade.
  • unknown, GPU detected — plain rocm serve <model>, then read the device: row of rocm services list. Deliberately not --engine lemonade: an unrecognized target may be a newer ROCm-capable GPU, and naming the Vulkan backend would be the same unestablished claim in the other direction. The engine chooses, and device: reports what it used.
  • unknown, nothing detected — rocm diagnose. rocm serve is refused by the no-usable-GPU pre-flight on such a host, so recommending it would be useless.

The reported backend device

rocm serve and rocm services list both grew a device: row, read from the engine over one Healthcheck RPC so the two surfaces cannot disagree. The row is optional by design: an engine still loading answers "unknown", and asserting that a ready service must have a row is a false failure (serve-05/serve-07 assert the row's shape and backend, not its presence — see Tests for what their cross-surface comparison can and cannot reach).

rocm_core::pinned_gpu_name names the GPU only on a host with exactly one AMD GPU and no visibility mask, because every name probe answers for the primary adapter rather than a HIP ordinal. Multi-GPU hosts get the backend and ordinal with no name — correct and unspecific rather than specific and wrong.

Also in this diff

Listed because the previous body omitted them:

  • runtime-19: the --family / ROCM_CLI_THEROCK_FAMILY refusal is reworded. Asking for a target ROCm does not build is refused outright, and no other family is substituted.
  • redacted() in serve_summary.rs tests: a helper that masks api key: values so the summary snapshots can be asserted without pinning a secret.

The Anthropic-endpoint publish race

Omitted from every earlier version of this body, which is very plausibly why it took several review rounds to be looked at. apps/rocmd/src/anthropic_proxy.rs, crates/rocm-core/src/anthropic_endpoint.rs and crates/rocm-core/src/endpoint_readiness.rs are a real subsystem of this diff.

The cause is this PR's own change. rocm_core::pinned_gpu_name is now called inside every engine's post-spawn whole-file state write, and on Windows that probe takes seconds. The Anthropic proxy is a thread of the same process (run_builtin_engine_serve_http starts it, then runs the adapter), and it publishes its endpoint into the same state file — inside the window the probe opened. The whole-file rewrite then replaced it.

publish_anthropic_endpoint notes the publication in a process-local map before touching the file, and write_json_state_file carries that note back into any full rewrite of the same path, under the state-directory lock.

All three built-in adapters now go through it. The first revision wired only Lemonade and argued the vLLM/AIM gap "predates this PR"; that was half right at best — the unlocked fs::write predates it, but the slow probe that makes the race practically reachable is added to those two engines by this diff, so closing it is this diff's job. Their write_state now routes through rocm_core::write_json_state_file and their status writers through merge_json_state_file, and both engines have a test that fails if the raw fs::write comes back.

The carry note is process-wide and never evicted. Resolving the GPU name before the spawn would remove the need for it; that is a larger restructuring of the launch path and is not attempted here.

Tests

examine-55 holds the printed level against the detected_gfx_target and compatible_therock_family in the same report, so every lane checks the level its own hardware implies rather than hard-coding one. serve-61 drives detection and the serve pre-flight from the same planted /sys, so the advised command is checked against the host that was shown it — it shows the pre-flight accepts that host, stopping at an out-of-range --gpu, not that a Lemonade launch then succeeds there. no_gpu_advice_names_a_command_that_host_can_run pins the advice against behaviour rather than wording.

Which lane covers the device: row. End to end, only the GPU lanes: serve-05 (vLLM, the per-PR canary) and serve-07 (Lemonade, xfail'd on lemonade+linux). The blocking mock job has no live device: row at all; there it is covered by unit tests — the per-engine reported_device formatters, the serve_summary.rs/services_text.rs row matrices, pinned_gpu_name's decision logic, and (new in the latest round) tests that drive each engine's real healthcheck_service_at RPC handler and assert the .device it answers with, which is the one thing a reverted handler wiring would have slipped past.

assert_services_list_reports_device's cross-surface comparison against rocm serve's own summary does not run on any current lane, and the step now says so: the summary those scenarios capture is pipe-redirected, so the CLI prints the row-less plain form rather than the interactive deployment summary. What discriminates there is the backend/shape assertion above it. Making the comparison live means driving those two canaries through a PTY, which is a change to the canaries' harness and is not bundled here.

Known costs, recorded

  • rocm services list can block ~18s (10s budget + one in-flight 8s probe; the budget is a start gate, tested before each probe, never inside one), and rocm chat's natural-language services command pays the same, holding the process-wide engine env lock. (Not the structured MCP tool that shares the name: it renders through render_services_tool_result_text, which has no device: row and makes no probe.) Moving the probe off the render path would stop the row reflecting the host at the moment it prints, which is the point of it; probing once and printing nothing after makes output depend on service ordering. Reversible: it is a constant plus a start gate, and a hard 10s cap needs a cancellable probe, not a smaller number.
  • On Windows, pinned_gpu_name reads the display inventory per call — deduplicated within a call, not cached across calls, and write_running_state runs more than once per launch. Launch path only; a cache would need hotplug invalidation.

Deferred, explicitly

None of these are done, and none should be read as done:

  • docs/rocm-docs/, the doctor skill files, and docs/ux-guidelines.md are untouched. ROCMAI-550 asks for them, and the ticket is already marked Done — so this needs a follow-up ticket rather than a note here.
  • The static "ROCm llama.cpp backend" claim survives in crates/rocm-dash-tui/src/ui/engine_manager.rs and apps/rocmd/src/host_snapshot.rs. Both mirror apps/rocm's own static engine_inventory() catalog, where the dynamic note is printed alongside it; in the TUI pane and the rocmd snapshot there is no note, so the static claim stands alone. Making them dynamic means giving the TUI layer a rocm-core dependency it deliberately avoids, and threading a support level into rocmd — a design change rather than a reword, which is why it is not bundled here.
  • Windows is unverified. Nothing available to this change can build the MSVC target; CI's Windows lanes are the only check on the Windows paths.
  • The VulkanOnly level has no hardware-grounded e2e scenario. No CI lane carries a gfx90c-class GPU, so examine-55 can only check that the printed level is self-consistent with the detected_gfx_target and compatible_therock_family of whatever hardware that lane has — on every current lane that is the RocmHip or Unknown branch, and one of its step assertions derives its expected value from the line it is checking. The VulkanOnly wording itself is asserted literally only by Rust unit tests (host_gpu/render.rs, examine_report_text.rs, mcp_bridge.rs, therock/family.rs) against planted inputs. Closing this needs either such a lane or a planted gfx90c host driven through an examine scenario (the repo does this elsewhere, e.g. host_with_a_kfd_less_gfx90c); it is a known, accepted gap here, not an oversight.

Verification

Linux devbox (this Mac cannot build rocm-engine-lemonade): cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, cargo test -p rocm-core (1627 passed), cargo test -p rocm --bin rocm (1947 passed), cargo test -p rocm-engine-aim (95) / -p rocm-engine-vllm (172) / -p rocm-engine-protocol, cargo test -p e2e-cucumber --test feature_naming, cargo xtask check-crate-edges / check-module-size (431 files) / check-architecture-doc.

Every test added in the latest round was mutation-checked: breaking the production change it covers turns it red, restoring it turns it green. That includes the AIM cpu_only gate, AIM's recorded ordinal, both engines' RPC-handler device wiring, both engines' locked state write, gpu_support_level's (_, Some(_)) arm and gpu_name_for_single_gpu_host's ordinal guard.

serve-05/serve-07 cannot run on that devbox — their Given step (rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed) fails there with rc=1. Confirmed pre-existing and environmental by running them on this PR's own previous head with no local changes: identical failure. They pass in CI.

…tic ROCm claim

ROCMAI-550: on a GPU with no ROCm/HIP build (e.g. gfx90c), `rocm examine` and
`rocm engines` kept claiming the embedded Lemonade engine runs "ROCm llama.cpp
backend" even though it actually falls back to Vulkan. `rocm install sdk`'s
refusal for such a GPU didn't name the actual next step, and neither `rocm
serve`'s deployment summary nor `rocm services list` showed which backend/
device an engine actually selected.

- add `GpuSupportLevel` (RocmHip/VulkanOnly/Unknown) in rocm-core, derived from
  the existing `normalize_therock_family`/`gfx_target_has_no_rocm_build`
  primitives
- make the Lemonade engine note in `rocm examine` and `rocm engines`
  GPU-aware instead of a hardcoded ROCm claim
- `rocm install sdk`'s unsupported-GPU errors now suggest `rocm serve <model>`
- surface the engine's actual backend/device (read live via the existing
  `Healthcheck` RPC, never guessed) in `rocm serve`'s deployment summary and
  `rocm services list`

Most of the ticket's `rocm examine`/`install sdk` complaints for gfx90c were
already fixed by #648 (ROCMAI-542); this covers what was left.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r requested a review from a team as a code owner October 14, 2026 12:01
@r0x0r
r0x0r requested a review from siloteemu October 14, 2026 12:01
Comment thread apps/rocm/src/serve_summary.rs Fixed
Comment thread apps/rocm/src/serve_summary.rs Fixed
CodeQL flagged two assertions as cleartext logging of `api_key`. Those
two use `base_summary()`, whose key is `None`, so the literal instances
are false positives — but the pattern is not. `render_summary` prints
the key by design, and `api_key_client_config_shown_only_when_present`
sets a real one, so a failure there copied `endpoint-secret` into CI
logs, which are retained and widely readable.

Assert on the real text, print a redacted copy. Covers all twelve sites
that interpolated a rendered summary, not just the two CodeQL picked.
Production output is unchanged: the key is still shown once, and the
test pinning that still asserts the exact string.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

CodeQL's two rust/cleartext-logging alerts are cleared in 65a24f8a — but the two lines it picked were not the dangerous ones, so the fix is wider than the alerts.

What CodeQL flagged (serve_summary.rs:452, :462): assertions in device_row_shown_only_when_known that interpolate a whole rendered summary into the panic message. Both use base_summary(), whose api_key is None, so no key could ever reach a log there — CodeQL is flow-insensitive about the Option.

What it was actually pointing at. render_summary prints api key: {api_key} by design (that is the feature, pinned by api_key_client_config_shown_only_when_present). That same test sets summary.api_key = Some("endpoint-secret") and then asserted with "{rendered}" — so a failure there would have copied the key into CI logs, which are retained and broadly readable. The rule found the right hazard class and the wrong two instances.

Fix: assert on the real text, print a redacted copy. A small test-only redacted() helper masks any api key: value, and all twelve sites that interpolated a rendered summary now go through it — not just the two flagged.

Production output is untouched: the key is still shown once, and the test that pins the exact string still asserts rendered.contains("api key: endpoint-secret") against the unredacted render.

Verified on Linux at 65a24f8a: serve_summary tests 16 passed / 0 failed (including the api-key feature test), cargo clippy --workspace --all-targets -- -D warnings exit 0, cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings exit 0, and cargo test --workspace --all-targets --no-fail-fast exit 0 — 33 binaries, 3852 passed, 0 failed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unknown GPUs remain misreported, authenticated Lemonade devices are omitted, and required observable-output coverage is incomplete.

7 open findings
What changed in this PR

Updates GPU/backend reporting to reflect detected support and live engine state.

Changes:

  • Adds GPU support classification and dynamic Lemonade messaging.
  • Reports live backend/device details in serving output.
  • Improves unsupported-SDK guidance and test coverage.
File Description
crates/​rocm-core/​src/​host_gpu.rs Adds GPU support tiers.
crates/​rocm-core/​src/​lib.rs Exports support APIs.
apps/​rocm/​src/​main.rs Adds dynamic notes and live device reporting.
apps/​rocm/​src/​serve_cmd.rs Fetches devices for deployment summaries.
apps/​rocm/​src/​serve_summary.rs Renders the device row.
apps/​rocm/​src/​therock.rs Adds serving remediation guidance.
apps/​rocm/​src/​providers.rs Updates summary construction.
tests/​e2e-cucumber/​features/​model_serving.feature Extends GPU serving scenarios.
tests/​e2e-cucumber/​tests/​e2e/​serving_steps.rs Adds device-reporting assertion.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apps/rocm/src/main.rs
Comment on lines +2751 to +2756
match level {
rocm_core::GpuSupportLevel::VulkanOnly => {
"default embedded Lemonade server; this GPU has no ROCm/HIP support, so it runs on the Vulkan llama.cpp backend"
}
_ => "default embedded Lemonade server with ROCm llama.cpp backend",
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right and this is now fixed in f54a8d2f. The earlier round added engines_inventory_does_not_claim_vulkan_for_an_unrecognized_gpu and its two siblings, which pinned that Unknown must not claim Vulkan — but they said nothing about the ROCm side, and the _ => catch-all kept printing "with ROCm llama.cpp backend". That is exactly the stale claim this PR exists to remove, surviving on precisely the hosts where the CLI knows least.

lemonade_engine_note (apps/rocm/src/main.rs:2748) now matches GpuSupportLevel exhaustively, with no catch-all, so a future variant has to be classified rather than silently inheriting the ROCm wording:

  • VulkanOnly — confirmed no ROCm build, names the Vulkan backend.
  • RocmHip — keeps the ROCm wording.
  • Unknown — claims no backend: "default embedded Lemonade server; run rocm examine to see which llama.cpp backend this GPU uses".

Not the Vulkan wording for Unknown either, for the reason the type exists: a detection gap is not a confirmed absence. Pointing at rocm examine keeps the row actionable, and matches what unsupported_requested_target_message already does on the other branch where the host is unknown.

The two Unknown tests now assert all three facts — no Vulkan claim, no ROCm claim, and a next step — at apps/rocm/src/examine_report_text.rs:2608 and apps/rocm/src/main.rs (codex_bridge_engine_inventory_does_not_claim_vulkan_for_an_unrecognized_gpu). Falsified on the Linux box by restoring the catch-all: both go red.

Comment thread apps/rocm/src/main.rs Outdated
Comment on lines +14263 to +14270
let response: HealthcheckResponse = engine_request(
Some(paths),
engine,
EngineMethod::Healthcheck,
&HealthcheckRequest {
service_id: service_id.to_owned(),
},
)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified against the current code and fixed in f54a8d2f — this was still live, not stale.

Confirmed the mechanism you described: engines/lemonade/src/state.rs::inference_verified passes rocm_engine_protocol::resolve_endpoint_api_key() to its probe, which reads ROCM_SERVE_API_KEY_FILE from the environment. fetch_service_device supplied only AppPaths, so for a public or --require-api-key service the engine's own /v1/models probe answered 401, listed/ready came back false, the device degraded to "unknown", and the row vanished for exactly the services that are protected.

Fix: the key file is threaded per call, since the key is per service id.

  • engine_request_with_scoped_env (apps/rocm/src/engine_ipc.rs:152) takes an endpoint_key_file beside the existing env_root. For the built-in path it is scoped by with_scoped_builtin_engine_env inside the same BUILTIN_ENGINE_ENV_LOCK critical section as the other vars, so the env-mutation contract (xtask/src/env_mutation_contract.rs) still holds; for an external plugin it is set on the child Command.
  • fetch_service_device (:305) passes endpoint_keys::endpoint_key_file_if_present(paths, service_id) — None for a loopback service, which has no key file and needs none.
  • engine_request_with_env_root delegates, so no existing call site changed.

Your pointer to the restart path was the right precedent: services_text.rs already re-threads the same file onto the engine child it spawns.

Not verified against a live protected server — the dev box has no usable GPU telemetry, so there is no managed service to probe there. This is wiring verified by compilation and the existing gates, not an observed 401 turning into a device row.

Comment thread apps/rocm/src/therock.rs Outdated
Comment on lines +3462 to +3463
the Vulkan backend can still be used. \
Try `rocm serve <model>` — it runs on Lemonade's Vulkan backend instead."

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in f54a8d2f. @juhovainio and @nowycondro reached the same defect from the configured-default_engine side; your recipe-selection angle is the other half of it, and the fix covers both.

unsupported_host_gpu_message (apps/rocm/src/therock.rs:3607) now pins the engine, which is the first of your two suggestions:

Try rocm serve <model> --engine lemonade - it runs on Lemonade's Vulkan backend instead.

I did not take the second (describe backend selection conditionally). Conditioning means resolve_family reading RocmCliConfig and the model recipe to say which engine would win — a config load and a recipe resolution added to an install-path error message, to produce a sentence that is still only correct for the inputs it happened to check. --engine out-ranks every source in select_serve_engine, so pinning makes the claim unconditional instead of conditional, which is the stronger guarantee for less machinery.

One correction to your description, for the record: a recipe's preferred_engines is ranked below the host-GPU preference in select_serve_engine, not above it — the override that genuinely out-ranks the host is the configured default_engine. Either way the bare command could not be promised, so the conclusion stands.

Test: unsupported_gpu_error_pins_the_engine_a_configured_default_could_override (therock.rs:8399), which asserts the pinned form and the absence of the bare one still carrying the promise — necessary because the pinned form contains the bare one as a prefix.

Your note about line 3479 refers to unsupported_requested_target_message; that one was already fixed in 16ccfb3a and now points at rocm examine instead of predicting a backend.

Comment thread apps/rocm/src/main.rs Outdated
Comment on lines +2744 to +2746
/// The lemonade engine note shown in `rocm examine` and `rocm engines`, adjusted
/// for whether this host's detected GPU can actually use the ROCm llama.cpp
/// backend. Both call sites have a live `GpuSupportLevel` in scope; other

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Judging this one partly answered and partly not reachable, with the reasons below rather than a silent skip.

Answered. The two notes are no longer unit-only in the sense you meant: engines_inventory_* (apps/rocm/src/examine_report_text.rs:2550-2628), examine_engine_inventory_* and codex_bridge_engine_inventory_* / engines_mcp_tool_* now drive the real renderers for all three support levels, including the Unknown expectation you asked for — which this push strengthened to assert that Unknown claims no backend rather than merely not claiming Vulkan (see the main.rs:2837 thread).

Not reachable as a scenario, and this is the honest part. A Gherkin assertion on the ROCm-versus-Vulkan wording needs a host whose GPU is detected and confirmed to have no ROCm build — a Renoir/Cezanne Vega APU. detect_host_gpu_summary reads real sysfs/WMI with no test override, and every lane reports a ROCm-capable GPU (platform=mi300x, Strix Halo, W7900). A scenario written for the Vulkan branch would be permanently skipped on every lane, which reads greener than it is.

What a lane can assert is the negative, and that is now pinned: runtime-19 checks that the --family gfx90c refusal points at rocm examine and does not contain rocm serve, on a runner that is ROCm-capable. Proving the absence of a false prediction is the half a lane can actually reach.

The device: row — the other observable this PR adds — is now asserted end-to-end, including its value and agreement between rocm serve and rocm services list; see the model_serving.feature and serving_steps.rs threads.

Comment thread apps/rocm/src/therock.rs Outdated
assert!(message.contains("no ROCm SDK build"), "{message}");
assert!(message.contains("`gfx90c`"), "{message}");
assert!(message.contains("Vulkan"), "{message}");
assert!(message.contains("rocm serve <model>"), "{message}");

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Standing by the existing answer here rather than adding a scenario, and stating the reason explicitly rather than leaving it in another thread: the full write-up is in the PR description section "Why the rocm install sdk remediation sentence is unit-covered and not a scenario", and in the reply to @juhovainio's thread on the same hunk.

Short version: unsupported_host_gpu_message is only reachable from resolve_family after the --family, ROCM_CLI_THEROCK_FAMILY and managed-runtime doors are all closed, and then only when detect_host_gfx_target() returns a target normalize_therock_family does not cover. That needs a real Renoir/Cezanne Vega APU on the runner, and detect_host_gfx_target reads sysfs/WMI with no test override. runtime-19 drives the --family branch precisely because that is the one door a lane can open — you are right that it does not prove the detected-host path, and nothing in CI can.

What did change in f54a8d2f is the remediation itself: it now names the engine, rocm serve <model> --engine lemonade, because a configured default_engine could otherwise make the sentence false (see @nowycondro's and @juhovainio's threads). The two unit tests you point at are joined by a third, unsupported_gpu_error_pins_the_engine_a_configured_default_could_override (apps/rocm/src/therock.rs:8399), which pins the engine-pinned form and the absence of the bare one — the bare command is a prefix of the pinned one, so a presence-only assertion could not tell them apart.

When the user sends a chat completion request
Then the response contains a model reply
And the response identifies the correct model
And the services list reports the engine's actual backend device

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and fixed in f54a8d2f. The scenarios ran rocm serve in a Given and then asserted only on a later services list, so the deployment summary's own device: row — half of what this PR adds — had no live assertion anywhere.

Two changes:

  • E2eWorld gains serve_summary (tests/e2e-cucumber/tests/e2e.rs), set by serve_and_wait and serve_gpu_model on the successful attempt. Deliberately not cli_output: that field belongs to each scenario's When step, and overwriting it from a Given would change what every existing Then on these scenarios reads.
  • assert_services_list_reports_device (serving_steps.rs:1491) now cross-checks the two surfaces: if the summary printed a device: row it must equal the one services list reports (they read the same Healthcheck RPC), and if it did not, the summary must not have said ready — so the row's absence has to be explained by a launch that was still starting, not by the row being dropped.

The assertion is two-sided rather than unconditional on purpose. rocm serve only asks the engine once the launch already reported ready, and a managed launch can legitimately return while the weights are still loading — which is exactly why the Given then polls /v1/models. An unconditional "summary must contain device:" would have been flaky on a slow cold start rather than a real check.

Caveat on where this actually runs: serve-07 is xfail'd on lemonade+linux (EAI-7423), so on that lane no step of it is a live assertion. serve-05 (the vLLM canary) is what exercises this on every PR.

Comment on lines +1361 to +1364
assert!(
block
.lines()
.any(|line| line.trim_start().starts_with("device:")),

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, and fixed in f54a8d2f. The step checked only that a line starts with device:, so an empty, hardcoded or wrong value all passed.

assert_services_list_reports_device (tests/e2e-cucumber/tests/e2e/serving_steps.rs:1491) now parses the value and holds it to three independent facts:

  1. Shape. parse_device_row (:1453) requires <backend> gpu or <backend> gpu <n> with nothing trailing and a numeric ordinal — the shape every engine reports. An empty or malformed value fails here.
  2. Backend against the engine that served. The step reads the engine: row from the same services-list block and checks the backend against it: lemonade may report rocm or vulkan; vllm and aim are ROCm-only, so vulkan from either means the value did not come from the engine that served. unknown fails for all three — it is the engines' own "no state for this", which fetch_service_device is supposed to drop rather than print. That is your rocm gpu … vs vulkan gpu … distinction, enforced rather than assumed.
  3. Agreement with the serve summary — see the thread on model_serving.feature.

Note the shape check only became meaningful with the engine fix in this same push: engines/vllm used to report the literal "rocm_gpu", which parse_device_row rejects. It now reports rocm gpu <n> like Lemonade.

What I could not do is assert the expected backend for a given host — that needs a Vulkan-only GPU, and no lane has one. The check is "consistent with what this engine can select", not "equal to what this host must select".

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this with a review-code fan-out pass plus a separate Standards/Spec pass against Jira ROCMAI-550. The core design is good: GpuSupportLevel is derived from existing detection primitives rather than adding new probing, the device: row is read live from each engine's own Healthcheck RPC rather than guessed from the launch-time GPU plan, and the PR description is refreshingly honest about what it couldn't verify end-to-end (both GPU canaries fail on this box for pre-existing, unrelated environment reasons, clearly disclosed) and what it checked by hand instead.

Requesting changes on one gap: the rocm install sdk remediation text added in therock.rs (unsupported_host_gpu_message/unsupported_requested_target_message now suggest rocm serve <model> as the next step) is a second user-observable CLI text change in this same PR, but only the device: row change got a Gherkin scenario. AGENTS.md §3 asks for a scenario or an explicit justification for user-observable output changes - the PR's test plan explicitly discloses the gap for the e2e device scenario (good), but says nothing about this one at all, so it reads as simply uncovered rather than a disclosed, accepted gap.

One more thing worth a look, not blocking: builtin_codex_bridge_engine_inventory (via builtin_engine_inventory's static table) still unconditionally claims lemonade runs "with ROCm llama.cpp backend" - the exact stale claim this PR exists to fix. The new lemonade_engine_note helper's own doc comment calls this out by name as a deliberate scope choice (no live GPU probe in front of the codex-bridge inventory), so this may be intentional - but it's the kind of thing worth confirming explicitly rather than leaving implicit, since it's the same wording this PR is removing from two other call sites.

Comment thread apps/rocm/src/therock.rs Outdated
The GPU is detected but is not supported by ROCm; \
the Vulkan backend can still be used."
the Vulkan backend can still be used. \
Try `rocm serve <model>` — it runs on Lemonade's Vulkan backend instead."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a second user-observable CLI text change in this PR (alongside the device: row), but it has no Gherkin scenario and no justification note in the PR description - unlike the device: row change, whose missing e2e coverage is explicitly disclosed in the test plan as a known, accepted gap. Right now this one just reads as uncovered. Worth either a scenario asserting rocm install sdk's refusal message on an unsupported GPU names rocm serve <model>, or an explicit note in the PR description on why unit-test coverage (which does exist, in the two tests just below this hunk) is sufficient here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went with the PR-description note rather than a scenario, and the reason is stronger than "a scenario costs runtime": a scenario for this branch cannot run on any lane we have. Full write-up is now a dedicated section in the PR description ("Why the rocm install sdk remediation sentence is unit-covered and not a scenario"); the short version:

unsupported_host_gpu_message is reached from resolve_family only after all three earlier doors are closed — no --family, no ROCM_CLI_THEROCK_FAMILY, no managed runtime — and then only when detect_host_gfx_target() itself returns a target normalize_therock_family does not cover and gfx_target_has_no_rocm_build confirms. That needs a real Renoir/Cezanne Vega APU on the runner. Every self-hosted lane reports platform=mi300x os=linux, and detect_host_gfx_target reads real sysfs/WMI with no test override, so the branch cannot be made to fire in e2e. A scenario written for it would be permanently skipped, which reads greener than it is.

What a lane can drive is the sibling branch, and that is now asserted — including the negative you found in your other thread: runtime-19 checks the --family gfx90c refusal points at rocm examine and does not contain rocm serve, precisely because that lane is ROCm-capable. So the one door a lane can open is covered by proving the absence of the prediction rather than enforcing it.

Holding this sentence up: the two unit tests just below (therock.rs:7681, therock.rs:7807), plus the fact that the claim is not a bare string pin — gfx_target_has_no_rocm_build is what routed the user here, and selects_vulkan_when_rocm_unsupported in engines/lemonade/src/backend_alignment.rs is what makes llama.cpp actually pick vulkan once rocm reports unsupported.

One wording change you should know about, since the exact text is what those unit tests sit next to: I merged main into this branch (it was 36 commits behind) and that brought in #683's whole-tree ASCII output contract, which this branch predates. xtask's every_literal_that_can_reach_a_redirectable_sink_is_ascii flagged this literal — and only this one out of everything the branch adds — for the em dash. It is now an ASCII hyphen:

Try `rocm serve <model>` - it runs on Lemonade's Vulkan backend instead.

Both unit tests match on rocm serve <model>, so they are unaffected. Merge commit is daed78d5; full gate re-run there on MI300X: 4178 tests passed / 0 failed, both clippy gates clean, cargo fmt --all --check clean.

Comment thread apps/rocm/src/main.rs
/// places that print this note statically (the dashboard TUI, `rocmd`'s own
/// catalog) have no host probe in front of them and keep the unconditional
/// ROCm wording, since they describe the engine in the abstract.
const fn lemonade_engine_note(level: rocm_core::GpuSupportLevel) -> &'static str {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper's own doc comment discloses that builtin_codex_bridge_engine_inventory/builtin_engine_inventory (main.rs ~2726-2747) keep the unconditional "ROCm llama.cpp backend" wording for lemonade, since that call site has no live GPU probe in front of it. That may be the right scope call, but it's worth confirming explicitly - it's the exact stale claim about ROCm support this PR exists to fix in the other two call sites, just surfacing through a third one (the codex bridge inventory) that's silently left out.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right that this needed to be decided explicitly rather than left in a doc comment, and on looking at it I don't think the scope call it described actually held. Fixed in 49e23e3 instead of justified.

The old comment claimed the bridge inventory could not be host-aware because it is "the static catalogue". That isn't what blocked it — both call sites can see the host:

  • build_codex_bridge_snapshot (main.rs:2706) already gathers the ExamineSummary carrying detected_gfx_target and compatible_therock_family, the exact two inputs gpu_support_level takes. It now hoists that gather and passes the level down, so the engine rows cost no detection the snapshot wasn't already paying for.
  • The bridge's engines tool (main.rs:8123) takes the same cheap detect_host_gpu_summary probe that rocm engines already uses, via a new host_gpu_support_level helper.

So there was no call site here that genuinely could not answer the question, which is what made "left alone deliberately" the wrong reading. As you put it, this is the same stale ROCm claim ROCMAI-550 exists to remove, surfacing through a third user-facing surface — a bridge client renders summary to the operator.

Only the summary string moves. Field set, field names and the installed_binary: true beside it are untouched, so the bridge's JSON shape is unchanged — this isn't the contract change the old comment warned about.

Two places still print the note statically, and that is now stated as deliberate rather than implied: the dashboard TUI's engine-manager pane renders a hard-coded descriptive list with no probe behind it, and rocmd's catalog in apps/rocmd/src/common.rs answers for whichever host a client asks about, not the one it runs on. Either becoming host-aware is its own change.

Gates on the Linux box at 49e23e3: 439 crates compiled, 3852 tests passed / 0 failed, clippy --workspace --all-targets -D warnings clean, and clippy --locked -p e2e-cucumber --test e2e -D warnings clean.

…l is produced

The gfx90c refusals in `therock.rs` gained a `rocm serve <model>` remediation
sentence, a second user-observable output change in this PR with no scenario
behind it. AGENTS.md §3 wants remediation advice asserted where the message is
really printed, not only against the formatter.

runtime-19 already opens the one door of the two a lane can reach -- naming the
target with `--family` -- so the assertion goes there. The detected-host door
needs a Vega APU no runner has and stays unit-covered, as its existing comment
already records.

Also names the codex-bridge engine inventory at both the static table and the
bridge helper as a deliberate scope boundary, instead of only in
`lemonade_engine_note`'s doc comment where a reader of the table would miss it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — both points checked against the code before acting. Pushed 65a24f8a..7162f911.

Blocking: the therock.rs remediation text. Your reading is right. Against the merge base (666cb34a) both messages gained the same trailing sentence, and nothing else in them moved:

+ Try `rocm serve <model>` — it runs on Lemonade's Vulkan backend instead.

One correction to be fair to the test plan: it did list "unit tests for the rocm install sdk next-step suggestion". But §3 says in as many words that a unit test does not discharge the requirement, so your point stands — there was no scenario and no disclosed gap.

I chose coverage over disclosure, because it turned out the gap was smaller than either of us assumed. runtime-19 (@id:runtime-install-sdk-refuses-unsupported-gpu-target) already drives install sdk --family gfx90c --dry-run and already captures stderr — so the new sentence was already flowing through an ungated lane, just unasserted. Added a third Then pinning it. Verified cargo xtask e2e -- -n "runtime-19" green, 4/4 steps, and falsified it: with the sentence removed from unsupported_requested_target_message that step fails and the other three still pass.

§3 also wants remediation advice tied to the behaviour, not just the string, and that part genuinely cannot run here, so the step's doc comment names what does prove it rather than pinning the wording alone: gfx_target_has_no_rocm_build is what routed the user into the refusal, and selects_vulkan_when_rocm_unsupported in engines/lemonade/src/backend_alignment.rs is what makes llama.cpp pick vulkan once rocm comes back unsupported — which is exactly the claim the sentence makes.

That leaves one genuine gap, now disclosed in the test plan rather than inherited silently: runtime-19 only opens the --family/env door, so unsupported_host_gpu_message — the detected-host door a real Ryzen 4000/5000 laptop opens — stays unit-covered only. It needs a Vega APU no runner has, and detect_host_gfx_target reads real sysfs/WMI with no override to fake one. That is the same gap runtime-19's own comment already records for the refusal itself; this change just extends it to the new sentence instead of leaving it unstated.

Non-blocking: the codex-bridge inventory. Confirmed, and deliberate — with one correction to the premise. The lemonade_engine_note doc comment names the dashboard TUI and rocmd's catalog, but it does not name the codex bridge, which is presumably why you had to ask. That is a fair hit, so I fixed the visibility rather than just answering here: the scope boundary is now stated on builtin_engine_inventory (the static table itself) and on builtin_codex_bridge_engine_inventory, and the bridge is named explicitly in lemonade_engine_note's list.

On the substance, I considered making it support-aware — build_codex_bridge_snapshot already calls ExamineSummary::gather(), so detected_gfx_target and compatible_therock_family are right there and threading a GpuSupportLevel through would be about five lines. I left it, for two reasons. First, that list is the host-blind catalogue, in the same shape as the unconditional installed_binary: true sitting beside it: every field but default_for_platform describes the engine, not the machine — and a consumer that wants this host's real support level reads it off the examine block in the same snapshot. Second, engines[].summary is a JSON contract field, so changing it is a third user-observable output change, in a PR you are already (correctly) blocking over an undisclosed second one. Doing it here would reproduce the problem rather than close it. Happy to raise a follow-up if you would rather the bridge carry the live wording.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed from scratch at the current head (this PR has moved substantially since my last review, which GitHub auto-dismissed). This version delivers what ROCMAI-550 asks for: a GpuSupportLevel classification, a GPU-aware Lemonade engine note, an install-sdk remediation sentence, and live device reporting (via the Healthcheck RPC, not guessed) in the serve summary and services list. I grepped the repo for the stale "ROCm llama.cpp backend" claim this ticket is about; the only remaining occurrences are the 4 locations this PR's own new doc comments document as a deliberate, explained exclusion - nothing stale slipped through. CI is fully green across all 24 checks, nothing flaky triggered.

One real issue, left as an inline comment below: the new remediation sentence added to unsupported_requested_target_message makes an unconditional claim about which backend the host will use, but that function fires on a rejected requested override (a --family flag or a stale ROCM_CLI_THEROCK_FAMILY env value) independent of the host's actual GPU. Your own new test in this same diff (cli_family_overrides_a_stale_env_value_naming_an_unsupported_gpu) demonstrates exactly this kind of override/real-host mismatch - which means on a host that actually has a ROCm-capable GPU, this message would tell the user rocm serve runs on Vulkan when it would really run on ROCm/HIP. The identical sentence is correct in the sibling unsupported_host_gpu_message, since that path only fires when host detection itself found no ROCm build, so it's describing a confirmed fact about this host.

Non-blocking, for your awareness:

  • host_gpu.rs's ExamineSummary::rocm_support_line hand-rolls the same classification the new gpu_support_level() function now exists to express, instead of calling it - currently equivalent, but a future change to one could silently desync from the other.
  • README.md/docs/testing.md/docs/manual-testing.md aren't updated for the new device: lines in the serve summary and services list text output.
  • The lemonade e2e device-reporting Then-step is appended after assertions that are xfailed on that lane (EAI-7423), so it doesn't actually run there yet; only the vLLM canary scenario reliably exercises the new device-reporting path in CI today.
  • render_services_json doesn't get the new device field that the text view does, so --json and text output diverge in informational content.

Comment thread apps/rocm/src/therock.rs Outdated
That GPU is not supported by ROCm; the Vulkan backend can still be \
used. Nothing was installed and no other family was substituted."
used. Nothing was installed and no other family was substituted. \
Try `rocm serve <model>` — it runs on Lemonade's Vulkan backend instead."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is added to the requested-target rejection path, which fires independent of the host's real GPU (e.g. a stale ROCM_CLI_THEROCK_FAMILY env value naming an unsupported target on a host that actually has a ROCm-capable GPU - exactly the scenario your own new test cli_family_overrides_a_stale_env_value_naming_an_unsupported_gpu exercises). On such a host, rocm serve <model> would actually run on ROCm/HIP, not Vulkan, making this an incorrect claim rather than a confirmed fact. The identical sentence in unsupported_host_gpu_message above is fine because that path only fires when host detection itself returned the no-build target. Worth either dropping this sentence from the requested-target message, or rephrasing it to not assert a specific backend when the host's actual support level is unknown at this point.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 16ccfb3 — you were right, and the precondition difference is exactly as you described.

Checking it before changing anything: unsupported_requested_target_message is raised from the two guards at the top of resolve_family (the --family branch and the ROCM_CLI_THEROCK_FAMILY branch), both of which sit above detect_managed_therock_family and detect_host_gfx_target. No host probe has run at that point. unsupported_host_gpu_message is only reachable further down, after detect_host_gfx_target() returned a target that normalize_therock_family didn't cover and gfx_target_has_no_rocm_build confirmed — so there the sentence is a fact about this host.

What I chose: reworded rather than dropped, to Run rocm examine to see what this host itself supports.

I considered gating it on the real GpuSupportLevel, but getting one there means calling detect_host_gfx_target() inside a branch deliberately placed ahead of all detection — it would contradict the function's own doc comment ("nothing was detected here"), and the existing test comment on requesting_gfx90c_by_name_fails_instead_of_installing_another_family explicitly treats reaching detection as the failure signal. Dropping it outright was the safe option but loses the next step, so pointing at rocm examine seemed strictly better: true on every host, still actionable, and it's the command this PR just taught to report the real support level.

One thing your review surfaced that I think matters more than the wording: runtime-19 was asserting the bug. It drives --family gfx90c, so it goes through this exact function, and its new Then was checking for rocm serve <model> + Vulkan. The CI runner it passes on reports platform=mi300x os=linux gpu=true effective_engine=vllm — a fully ROCm-capable host. So the false claim was being pinned green on precisely the kind of machine where it's false. That step now asserts the rocm examine pointer and the negative (!output.contains("rocm serve")), so the lane now proves the absence of the prediction instead of enforcing it.

Evidence the test fails without the fix — reverted just the message string on the devbox, left the tests alone:

test therock::tests::requested_target_error_does_not_claim_which_backend_serving_will_use ... FAILED
test therock::tests::requested_target_error_names_where_the_target_came_from ... FAILED

the requested target says nothing about this host's GPU, so the refusal must not
predict what `rocm serve` would run on: ... Try `rocm serve <model>` — it runs on
Lemonade's Vulkan backend instead.

Restored, both pass. The new test also asserts the positive side on the sibling path (unsupported_host_gpu_message keeps rocm serve <model> + Vulkan), so it pins the asymmetry rather than banning the sentence outright — if someone later copies it back across, that test is what catches it.

`unsupported_requested_target_message` is raised from the `--family` and
`ROCM_CLI_THEROCK_FAMILY` branches at the top of `resolve_family`, before
any host probe runs, so the rejected target says nothing about the GPU in
this machine. It had borrowed its sibling's closing sentence promising
that `rocm serve <model>` runs on Lemonade's Vulkan backend -- true in
`unsupported_host_gpu_message`, which is only reached once host detection
itself returns a target with no ROCm build, but false here: a stale env
value naming gfx90c refuses just the same on a host whose own GPU is
fully ROCm-capable, and there serving would run on ROCm/HIP.

Point at `rocm examine` instead. It is true on every host, it is still a
next step, and it is the command this PR just taught to report the real
support level.

runtime-19 was asserting the wrong half of this: it drives `--family
gfx90c`, so it was pinning the Vulkan-serving claim green on exactly the
ROCm-capable runners where it was false. It now asserts the examine
pointer and the negative.

Also route `ExamineSummary::rocm_support_line` through `gpu_support_level`
rather than re-deriving the same classification, and document the serve
summary's `device:` row in README and the manual-testing GPU check.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the re-review — the blocking finding was correct and is fixed in 16ccfb3. Details on the fix and the falsification evidence are in-thread; summary plus the four non-blocking items here.

Blocking — requested-target path claimed a backend it never checked. Verified the precondition difference at the source before touching it: both resolve_family guards that raise unsupported_requested_target_message sit above every detection call, so the rejected target says nothing about this host. Reworded to point at rocm examine instead of predicting Vulkan. Also fixed runtime-19, which was asserting the bug — it drives --family gfx90c and its runner is platform=mi300x effective_engine=vllm, i.e. the Vulkan claim was being pinned green on a ROCm-capable host. It now asserts the examine pointer and the negative.

Non-blocking:

  1. rocm_support_line → gpu_support_level() — taken. Routed through the shared function, with every GpuSupportLevel variant spelled out so adding one is a compile error here rather than a silently blank line. This is exactly the drift the PR is about, and it was cheap.

  2. Docs — taken, where it's load-bearing. README now describes the device: row (and notes rocm services reports the same). docs/manual-testing.md §3 got the more useful edit: its GPU check was "the engine uses ROCm GPU execution / does not fall back to Vulkan", which was previously verified by inferring from the absence of a fallback warning — there's now a direct device: row to read, so the step says to read it and what a failing value looks like. Left docs/testing.md alone; it documents the harness, not serve output.

  3. Lemonade e2e step after xfailed assertions — disclosed, not moved. You're right that it isn't coverage there. I looked at whether moving it up is valid and concluded it isn't: serve-lemonade-inference is xfail'd at scenario level on effective_engine=lemonade, os=linux, so no step of it is a live assertion on that lane regardless of position — and the EAI-7423 failure mode (serve reaches ready, shuts down ~0.08s later) means the service is gone before a healthcheck could report a device anyway, so the step would fail there on its own merits. Reordering would have looked like a fix without being one. Added a comment on the scenario saying so, and naming serve-05 (vLLM) as the canary that actually exercises device reporting per-PR.

  4. render_services_json — left, deliberately, and I think it wants its own PR. It isn't a rendering choice: that function is serde_json::to_string_pretty(&records) over ManagedServiceRecord loaded straight off disk, whereas device is read live from the engine's Healthcheck RPC and never persisted. Adding it means either widening the persisted record schema or giving --json a shape that diverges from the record it currently mirrors, plus a live RPC per service on the machine-readable path (with its own timeout/failure semantics to define). That's a JSON contract change with real design questions, not a one-line addition, so I'd rather it be proposed and reviewed on its own than smuggled in here. The text/JSON divergence is real in the meantime — flagging it rather than leaving it silent.

Verification (devbox, 16ccfb3a, remote HEAD confirmed to match):

No new assertion interpolates a rendered summary, so no new rust/cleartext-logging surface. Not re-requesting review per the owner's standing request.

@r0x0r
r0x0r requested a review from juhovainio October 16, 2026 14:21
r0x0r added 2 commits October 16, 2026 17:44
…tory

The codex-bridge engine inventory kept the unconditional "with ROCm
llama.cpp backend" wording that ROCMAI-550 removed from `rocm examine`
and `rocm engines`, so a bridge client still rendered a ROCm claim to
the operator on a host that cannot use it.

Both call sites can answer the question: the snapshot already gathers
the ExamineSummary carrying the two inputs, and the `engines` tool takes
the same cheap host probe `rocm engines` uses. Only the `summary` string
changes; the field set and the bridge's JSON shape are untouched.

The dashboard TUI pane and rocmd's own catalog keep the static wording
deliberately -- the pane has no probe behind it, and rocmd answers for
whichever host a client asks about, not the one it runs on.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Resolves the one conflict, apps/rocm/src/main.rs, by taking main's
post-#667 file (chat/MCP-bridge extraction removed 185 functions) and
re-applying this branch's 23 hunks onto it. One hunk follows its
function: `run_internal_mcp_call` now lives in apps/rocm/src/chat.rs, so
the `engines` tool's call to `builtin_codex_bridge_engine_inventory`
is updated there and `host_gpu_support_level` is imported from the crate
root.

Verified by construction: the `fn` name sets of main, this branch and the
resolution comm out to resolved == main's set plus exactly the five
functions this branch adds (`fetch_service_device`,
`host_gpu_support_level`, `lemonade_engine_note` and the two new examine
tests), with nothing of main's dropped.

Also converts this branch's one new non-ASCII production literal to ASCII
for the output contract #683 added, which this branch predates: the em
dash in `unsupported_host_gpu_message`'s "Try `rocm serve <model>`"
sentence becomes a hyphen, matching how #683 converted the rest of the
tree. `xtask`'s `every_literal_that_can_reach_a_redirectable_sink_is_ascii`
named that literal and only that one; everything else this branch adds
with non-ASCII is a comment or lives under the exempt `tests/` prefix.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the GPU-support-level messaging work (ROCm/HIP vs Vulkan-only vs Unknown) across host_gpu.rs, the codex-bridge/MCP snapshot, rocm engines/rocm examine, and the install-sdk refusal paths.

The core idea is sound: a 3-tier GpuSupportLevel replaces a static "ROCm llama.cpp backend" claim with one that reflects what the host actually supports, and it's backed by real e2e coverage (serve-05/06/07, runtime-19) plus an honest EAI-7423 xfail scoped correctly to effective_engine=lemonade, os=linux.

One thing worth flagging before this merges: the install-sdk refusal messages are not symmetric between the host-detected path (unsupported_host_gpu_message, adds "Try rocm serve <model>") and the explicit --family/env path (unsupported_requested_target_message, points at rocm examine instead) — but that's deliberate and well-documented/tested here (requested_target_error_does_not_claim_which_backend_serving_will_use), so just calling it out as intentional, not a problem.

Blocking:

  • New GPU-aware behavior on the codex-bridge/MCP surface (host_gpu_support_level, the new branch in builtin_codex_bridge_engine_inventory, and the "engines" MCP tool call site in chat.rs) has zero test coverage. append_examine_engine_inventory (rocm examine) got solid new tests for both VulkanOnly and RocmHip, but neither render_engine_inventory_text_with_paths (rocm engines) nor the codex-bridge path exercise the same branch directly — see inline comment.

Non-blocking:

  • Stale doc reference in host_gpu.rs — see inline comment.
  • The lemonade note-substitution logic (if *engine == "lemonade" { lemonade_engine_note(...) }) is now duplicated three times across render_engine_inventory_text_with_paths, append_examine_engine_inventory, and builtin_codex_bridge_engine_inventory. Not blocking, but a shared helper would avoid the three call sites drifting independently later.

Ticket scope: Confirmed against ROCMAI-550 (epic ROCMAI-517) — the change matches the ticket's intent (stop asserting a static ROCm claim, report real support level instead). One review pass raised that docs/rocm-docs/, docs/ux-guidelines.md, doctor skill files, and --help text might still carry the old unconditional claim; a follow-up repo-wide grep came back clean on all of those. Flagging the discrepancy rather than silently picking one — worth a quick independent grep before merge if you want to be certain, but I did not find the stale string anywhere outside what's already fixed in this diff.

CI: The PR was just updated with a merge of main (resolving an earlier merge conflict cleanly — verified independently, chat.rs's call site already correctly passes host_gpu_support_level(Some(paths))). Most checks are still pending against the new merge commit as of this review; nothing has failed so far.

Comment thread crates/rocm-core/src/host_gpu.rs Outdated
/// `Unknown` deliberately covers both "nothing detected" and "a gfx target we
/// don't recognize" rather than being called `Unsupported` — see the doc
/// comment on [`gfx_target_has_no_rocm_build`] and
/// `GFX_TARGETS_WITHOUT_ROCM_SUPPORT` in `diagnose.rs` for why conflating

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc comment points to GFX_TARGETS_WITHOUT_ROCM_SUPPORT in diagnose.rs, but that constant doesn't exist anywhere in the repo (checked via a full-repo grep). Either the name changed before this landed or it was never added — worth fixing so the doc doesn't send readers chasing a symbol that isn't there.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in bce633d6. GFX_TARGETS_WITHOUT_ROCM_SUPPORT appears exactly once in the repo — in this doc comment. diagnose.rs has no such constant (its consts are the KEYWORDS_* tables, MIN_SCORE_FOR_MATCH, SHM_MIN_BYTES, etc.), and nothing matching GFX_TARGETS* or *WITHOUT_ROCM* exists anywhere else either. It was never a real symbol, not a rename.

The sentence needed the second reference because gfx_target_has_no_rocm_build alone doesn't carry the "confirmed" half — that comes from its conjunction with normalize_therock_family. So rather than point at a different file, the doc now names the conjunct that actually establishes it:

Only that predicate's conjunction with [normalize_therock_family] establishes the "confirmed" half; an unrecognized target satisfies neither conjunct, so it is a gap in what this catalog knows rather than a fact about the card.

Both links are now intra-module rustdoc links, so a future rename breaks the doc build instead of rotting silently.

Comment thread apps/rocm/src/main.rs Outdated
/// The same two inputs `rocm examine` derives it from, read through
/// `detect_host_gpu_summary` rather than the full examine gather — exactly what
/// `render_engine_inventory_text_with_paths` (i.e. `rocm engines`) already does.
fn host_gpu_support_level(paths: Option<&AppPaths>) -> rocm_core::GpuSupportLevel {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function (and the new GPU-aware branch it feeds into builtin_codex_bridge_engine_inventory, plus the "engines" MCP tool call site in chat.rs) has no test coverage. append_examine_engine_inventory got examine_engine_inventory_names_vulkan_for_a_gpu_with_no_rocm_support / ..._keeps_rocm_note_for_a_rocm_capable_gpu for the equivalent rocm examine branch — this surface (and render_engine_inventory_text_with_paths's own rocm engines output) should get the same treatment before merge, since it's the same user-facing correction (no more asserting ROCm support on a Vulkan-only host) just reached through the MCP bridge instead of the CLI.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, and the gap was mine — I pushed 49e23e30 earlier today without tests. Fixed in bce633d6.

Both surfaces were untestable for the same reason: the branch's input is a live sysfs probe, and no CI runner has a GPU that is detected and confirmed to have no ROCm build. So each now takes the level the way append_examine_engine_inventory already did — the caller probes once, the renderer is handed the answer:

  • render_engine_inventory_text_for_host(paths, host_default_engine, level) — split out of render_engine_inventory_text_with_paths, which still does the one detect_host_gpu_summary call and derives both arguments from it, so the * marker and the note cannot disagree about the same host.
  • internal_mcp_engines_tool_result(level) — split out of the "engines" match arm in chat.rs. Same text, same engines key, same entries.

Six new tests mirroring yours, plus three on the probe wiring:

test proves
engines_inventory_names_vulkan_for_a_gpu_with_no_rocm_support rocm engines names the Vulkan backend, drops the ROCm claim, and leaves vLLM's row alone
engines_inventory_keeps_rocm_note_for_a_rocm_capable_gpu a ROCm host keeps the ROCm wording and is not told it's on Vulkan
engines_inventory_does_not_claim_vulkan_for_an_unrecognized_gpu Unknown is a detection gap, not a confirmed absence
codex_bridge_engine_inventory_names_vulkan_… / …_keeps_rocm_note_… / …_does_not_claim_vulkan_… the same three on CodexBridgeEngine.summary, with installed_binary and vLLM's row asserted unchanged so the JSON shape is pinned too
engines_mcp_tool_names_vulkan_… / …_keeps_rocm_note_… asserted on structuredContent.engines[].summary — the exact string an MCP client renders
engines_mcp_tool_answers_for_the_host_it_runs_on the tool reaches the inventory through the live probe, not a level nailed to a constant
gpu_support_level_for_host_gpu_reads_a_vulkan_only_summary_as_vulkan_only (+ two) covers host_gpu_support_level's derivation

Falsified every one — five mutations on the devbox, each run confirmed non-zero test count:

  1. engine_inventory_note's engine == "lemonade" → false: 3 new tests red (engines_inventory_names_vulkan…, codex_bridge_…_names_vulkan…, engines_mcp_tool_names_vulkan…), plus your examine_engine_inventory_names_vulkan….
  2. lemonade_engine_note's VulkanOnly => → RocmHip =>: 6 new tests red — both directions on all three surfaces.
  3. VulkanOnly => → Unknown =>: 4 new tests red, including both …_does_not_claim_vulkan_for_an_unrecognized_gpu, so the Unknown arm has teeth.
  4. gpu_support_level_for_host_gpu arguments transposed: gpu_support_level_for_host_gpu_reads_a_vulkan_only_summary_as_vulkan_only red. The two Option<String> fields are positional and same-typed, so that mutation compiles clean and classifies exactly the Vulkan-only host as RocmHip — it was the one genuinely silent failure mode here.
  5. "engines" arm hard-coded to GpuSupportLevel::VulkanOnly: engines_mcp_tool_answers_for_the_host_it_runs_on red.

On your duplication point: I did extract the helper — engine_inventory_note(engine, catalog_note, level) — and all three renderers now route through it. The tests deliberately drive the renderers and never that helper, per your #703 note; mutation 1 above is the proof that going through it is what they're actually checking.

One honest limitation: host_gpu_support_level(Some(paths)) at the call site is covered only by engines_mcp_tool_answers_for_the_host_it_runs_on, which can catch a hard-coded level that disagrees with the runner but not one that happens to render identically to it (Unknown and RocmHip share wording). The discriminating coverage for a real Vulkan-only host stays with the e2e lane.

Gates on the devbox, all green: cargo test --workspace --all-targets --no-fail-fast (rc 0, 35 suites, all 12 new tests present), cargo clippy --workspace --all-targets -- -D warnings (rc 0), cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings (rc 0), cargo fmt --all clean.

… and the bridge

`append_examine_engine_inventory` got tests for both support levels when
ROCMAI-550 landed; the other two surfaces carrying the same correction did
not. `rocm engines` and the codex-bridge inventory (including its `engines`
MCP tool) could have gone back to asserting ROCm support on a Vulkan-only
host with nothing red.

Both were untestable for the same reason: the branch's input is a live sysfs
probe, and no CI runner has a GPU that is detected *and* confirmed to have no
ROCm build. So each surface now takes the level the way
`append_examine_engine_inventory` already did -- the caller probes once, the
renderer is given the answer. `render_engine_inventory_text_for_host` and
`internal_mcp_engines_tool_result` are those renderers; neither changes what
is printed.

The three copies of the lemonade substitution collapse into
`engine_inventory_note`, so the surfaces cannot drift. The new tests drive
the renderers, not that helper: asserting on it would restate the condition
rather than prove a surface prints it.

`gpu_support_level_for_host_gpu` reads the two `HostGpuSummary` fields once
instead of at each call site. They are same-typed positional arguments, so a
transposed call site would have classified a Vulkan-only host as `RocmHip`
and still compiled; it now has a test.

Also drops the dangling `GFX_TARGETS_WITHOUT_ROCM_SUPPORT` reference from the
`GpuSupportLevel` doc comment -- no such constant exists in `diagnose.rs` or
anywhere else, so the doc pointed readers at nothing.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the blocking item and both non-blocking ones in bce633d6; details in the two inline threads.

On your third point — the independent grep you asked for. I ran it and it agrees with yours: no stale unconditional claim survives outside this diff.

grep -rn "ROCm llama.cpp backend" across the whole tree returns 14 hits, and every one is accounted for:

  • Deliberately host-blind (named as such in lemonade_engine_note's doc comment): crates/rocm-dash-tui/src/ui/engine_manager.rs:39 and apps/rocmd/src/common.rs:186.
  • The static catalogues the renderers substitute into, never printed for lemonade by a host-aware surface: engine_inventory() and builtin_engine_inventory() in apps/rocm/src/main.rs.
  • A different claim entirely: engines/lemonade/src/lib.rs:386 and its e2e assertion — "Aligned Lemonade's ROCm llama.cpp backend to match the installed ROCm SDK", which is about backend-version alignment on a host that already has one.
  • Parser fixtures, not user-facing output: tests/e2e-cucumber/src/capability.rs:976 and xtask/src/e2e_prewarm.rs:1477,1499, both feeding parse_engines_list / default_engine_from_inventory.
  • The rest are doc comments in main.rs explaining the substitution.

Specifically on the four places the earlier pass worried about:

  • docs/rocm-docs/ — one mention of Lemonade in the entire tree (index.rst:15, "adapters for Lemonade and vLLM"), no backend claim.
  • docs/ux-guidelines.md — no mention of Lemonade at all.
  • Doctor skill files — skills/rocm-doctor/reference.md:125 already reads the right way round: "A Vulkan-backed engine (the Lemonade default) or a managed-runtime wheel needs no system SDK". The other llama.cpp hits there are the MSVC-redist and HIP-SDK fix rows, unrelated.
  • --help text — no clap about/long_about carries the string; the global grep above is exhaustive and none of the hits are argument help.

So: nothing to fix, second pair of eyes agrees.

Also leaving unsupported_host_gpu_message / unsupported_requested_target_message exactly as they are, per your note that the asymmetry is intentional.

Re-requesting your review.

@r0x0r
r0x0r requested a review from juhovainio October 16, 2026 15:44

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at the current head (bce633d6) after the prior review (at daed78d5) was superseded by new commits.

Blocking

  • apps/rocm/src/therock.rs (unsupported_host_gpu_message): the new closing sentence always promises Lemonade's Vulkan fallback via rocm serve. But select_serve_engine lets a configured default_engine override (e.g. vllm) win regardless of host GPU support, so a user on an unsupported-ROCm GPU with such a config gets a message promising a Vulkan fallback that rocm serve won't actually take. Please condition the message on the engine that will actually be selected, or soften it to not name a specific backend unconditionally.

Non-blocking

  • apps/rocm/src/main.rs (render_services_text / fetch_service_device): the live per-service Healthcheck RPC on each rocm services list render has no visible timeout bound. The in-code comment explains why this reads live instead of from the persisted ManagedServiceRecord (avoiding staleness) — that part's fine — but worth confirming there's an upstream timeout in engine_request, otherwise a hung/slow third-party engine plugin could stall the whole listing.
  • The gpu_support_level derivation (rocm_core::gpu_support_level(detected_gfx_target, compatible_therock_family)) is duplicated verbatim between build_codex_bridge_snapshot and examine_human_report; an analogous helper to the PR's own gpu_support_level_for_host_gpu would avoid the duplicated positional-argument call.
  • Ticket scope (ROCMAI-550) nuance: the ticket's literal wording asks for a three-tier model ("full ROCm/HIP support, Vulkan only, or unsupported"), but this ships two confirmed tiers plus Unknown for detection gaps rather than a confirmed-unsupported verdict. Seems like a deliberate, defensible choice to avoid false-negative claims, but flagging since it's a wording deviation from the ticket.
  • The PR's description implies the stale "ROCm llama.cpp backend" claim has been fully purged; confirmed true in README/docs/rocm-doctor, but apps/rocmd/src/common.rs:186 (rocmd_engine_inventory) still hard-codes that exact phrase and is untouched by this PR. May be intentionally out of scope (rocmd answers for an arbitrary queried host, not necessarily the one it runs on) but it's a live remaining instance worth a look.

CI / mergeability
CI is green aside from the known Strix Halo Windows self-hosted download-infra flake (repo.amd.com 403) and its consolidated-report rollup failure — not a regression. Separately, the PR currently shows mergeable=CONFLICTING and will need a rebase/merge-conflict resolution regardless of this review.

Comment thread apps/rocm/src/therock.rs Outdated
The GPU is detected but is not supported by ROCm; \
the Vulkan backend can still be used."
the Vulkan backend can still be used. \
Try `rocm serve <model>` - it runs on Lemonade's Vulkan backend instead."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is unconditional, but select_serve_engine can select a configured default_engine override (e.g. vllm) regardless of host GPU support. On an unsupported-ROCm GPU with such a config, rocm serve won't actually take the Vulkan/Lemonade path this message promises. Consider conditioning this on the engine that will actually be selected, or softening the wording.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in f54a8d2f — you and @nowycondro landed on the same defect from two directions, and you were both right. select_serve_engine (apps/rocm/src/serve_cmd.rs) ranks explicit --engine first, then configured_default, and only then the host preference, so a host that reaches unsupported_host_gpu_message can still have rocm serve <model> select vLLM.

I pinned the engine rather than conditioning the wording on config. apps/rocm/src/therock.rs:3607 now reads:

Try rocm serve <model> --engine lemonade - it runs on Lemonade's Vulkan backend instead.

The reason for pinning over conditioning: resolve_family would have to read RocmCliConfig to condition, which adds a config load to an install-path error and still only covers the configured-default source — the recipe-preferred-engine source (which @copilot-pull-request-reviewer raised separately) would remain. An explicit --engine out-ranks every selection source in that function, so the sentence is true whatever the config and whatever the recipe says.

New test at apps/rocm/src/therock.rs:8399, unsupported_gpu_error_pins_the_engine_a_configured_default_could_override. It pins both halves — the engine-pinned command, and the absence of the bare one still carrying the promise. Falsified on the MI300X box by reverting the message to the bare command: that test and requested_target_error_does_not_claim_which_backend_serving_will_use both go red (the latter because it asserts the sibling's positive side).

Comment thread apps/rocm/src/main.rs Outdated
// actually reports right now. Only meaningful once the engine is ready
// to answer, same gate `rocm serve`'s summary uses.
if record.status == "ready"
&& let Some(device) = fetch_service_device(paths, &record.engine, &record.service_id)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this issues a live Healthcheck RPC per listed service with no visible timeout bound here. The comment above explains why it reads live rather than from the persisted record (good reasoning), but worth confirming engine_request enforces a timeout upstream — otherwise a slow/hung third-party engine plugin could stall the whole rocm services list render.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked it, and the answer to "worth confirming engine_request enforces a timeout upstream" is no, it does not — so this needed a fix, not a confirmation. engine_request_with_env_root (apps/rocm/src/engine_ipc.rs) ends in a plain child.wait() with no deadline; there is no *_TIMEOUT anywhere in that module. I had half-written a doc comment claiming a timeout before I went and read it, which is a good argument for your asking.

What is bounded is the engine side, and only for the built-ins: rocm serve's three engines take the in-process fast path, and every endpoint probe they make is timeout-bounded (engines/vllm's HEALTHCHECK_TIMEOUT_MS, engines/lemonade's rocm_core::INFERENCE_PROBE_TIMEOUT). An external plugin is a spawned child and is genuinely unbounded.

Fixed in f54a8d2f by bounding the aggregate at the call site rather than per call — fetch_service_devices (engine_ipc.rs:365) under SERVICE_DEVICE_PROBE_BUDGET (:345, 10s). Once the budget is spent the remaining services simply get no device: row. render_services_text (services_text.rs:441) goes through it.

Two things I deliberately did not do, both stated in the doc comment rather than left implicit:

  • Not parallelised. engine_request routes the built-ins through with_scoped_builtin_engine_env, which takes a process-wide mutex and mutates process-global ROCM_CLI_* env vars for the duration. Concurrent probes would serialise on that lock anyway and racing them would be unsound.
  • The budget is checked between probes, not inside one. A single external plugin that never exits still blocks. Bounding that means a detached thread, and abandoning a worker that holds the env lock mid-probe would leak mutated process env into whatever ran next — worse than the problem. That is the pre-existing behaviour of every engine call site in the CLI and this does not make it worse.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review (medium/high-effort pass). 3 findings posted inline, ranked by severity: (1) a correctness issue where unsupported_host_gpu_message's new "Try rocm serve <model>" sentence can be false when the user has a configured default_engine override; (2) a documentation/behavior mismatch where vLLM's actual device string (rocm_gpu) doesn't match the format the PR documents (rocm gpu 0/rocm gpu <n>); (3) a minor efficiency note that rocm services list now does live per-service network healthchecks where it previously only read local state. No other blocking issues found — the core GpuSupportLevel plumbing, the rocm serve/services list device-row gating, and the refactor of append_examine_engine_inventory/render_engine_inventory_text_for_host all look correct and consistently wired.

Comment thread apps/rocm/src/therock.rs Outdated
The GPU is detected but is not supported by ROCm; \
the Vulkan backend can still be used."
the Vulkan backend can still be used. \
Try `rocm serve <model>` - it runs on Lemonade's Vulkan backend instead."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence unconditionally promises that rocm serve <model> "runs on Lemonade's Vulkan backend", but this branch (unsupported_host_gpu_message) only confirms the host GPU has no ROCm build — it says nothing about which engine rocm serve will actually pick. Per select_serve_engine (apps/rocm/src/serve_cmd.rs:60-78), a configured_default_engine (set via rocm config set-default-engine, which is never validated against the host's GPU) outranks the host's engine preference. So a user who previously ran rocm config set-default-engine vllm (e.g. after moving a config from a ROCm box, or by mistake) would have rocm serve <model> attempt vllm on this Vulkan-only host and fail under the project's strict no-CPU-fallback policy — directly contradicting this message's promise.

Notably, the sibling message unsupported_requested_target_message in this same PR deliberately avoids making this exact claim (see its doc comment a few lines below) because it "never looked at this host" — but unsupported_host_gpu_message did look at the host's GPU, just not at the user's configured engine override, so the same class of false-promise risk applies here too. The new unit tests (therock.rs ~7677, ~7807) only pin the string "rocm serve <model>", not that running it actually succeeds via Lemonade under a configured-engine override.

Failure scenario: host is gfx90c (Vulkan-only, confirmed via detect_host_gfx_target), rocm_cli_config.default_engine == Some("vllm"). rocm install sdk (no --family) hits this branch and tells the user to "Try rocm serve <model>" promising Vulkan/Lemonade; rocm serve <model> instead selects vllm (configured default wins) and fails since there's no ROCm GPU.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in f54a8d2f. Your failure scenario is exact, including that rocm config set-default-engine never validates against the GPU — I checked select_serve_engine (apps/rocm/src/serve_cmd.rs) and configured_default does sit above the host-GPU preference, so rocm serve <model> on a gfx90c host with default_engine = "vllm" attempts vLLM and dies under the no-CPU-fallback policy while this message promised Lemonade.

apps/rocm/src/therock.rs:3607 now pins the engine:

Try rocm serve <model> --engine lemonade - it runs on Lemonade's Vulkan backend instead.

An explicit --engine out-ranks every other selection source in that function, so this is true for every config and for a recipe whose preferred_engines would otherwise win — which closes the second half of the same hole that a config-conditional message would have left open.

Your point that the sibling message deliberately avoids the claim is why I did not just drop the sentence here: the asymmetry is real (this branch did look at the host) and worth keeping, it was only over-claiming the engine, not the backend. The doc comment above the function now says exactly that, and the new test unsupported_gpu_error_pins_the_engine_a_configured_default_could_override (therock.rs:8399) pins it against the configured-default regression specifically.

On your note that the old tests only pinned the string "rocm serve <model>": true, and that is why the new test asserts the negative too (no bare rocm serve <model> followed by the promise), not just the presence of the longer form — the longer form contains the shorter one as a prefix, so a presence-only assertion could not tell them apart.

Comment thread docs/manual-testing.md Outdated
- The engine uses ROCm GPU execution. The deployment summary's `device:` row
reports what the engine actually selected, so check it by reading that row
rather than inferring it from the absence of a fallback warning: expect
`rocm gpu <n>`. `rocm services` shows the same `device` while it runs.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tells the manual tester to expect rocm gpu <n> in the new device: row for a vLLM/ROCm run, matching the format documented in README.md:680 (rocm gpu 0) and the doc comment on DeploymentSummary.device (serve_summary.rs:59). But vLLM's own Healthcheck handler (engines/vllm/src/state.rs:60-66) hardcodes the device string as the literal constant "rocm_gpu" (underscore, no GPU index) whenever state.is_some():

let device = if state.is_some() {
    "rocm_gpu"
} else {
    "unknown"
};

fetch_service_device (apps/rocm/src/main.rs) displays response.device verbatim with no reformatting. So for the vLLM engine — the only engine that reports a rocm* device — the actual printed line will read device: rocm_gpu, not device: rocm gpu 0 as this doc (and the README) describe. A tester following this doc literally would flag a false failure, or a reader would be misled about the real format. This is also the serve-05 vLLM canary the PR added device:-row e2e coverage for, and the PR's own test-plan notes it could not actually exercise that step live due to an unrelated environment gap — so this mismatch was never observed end-to-end.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right about the mismatch and I fixed the code rather than the doc, in f54a8d2f. One correction to the attribution, which does not change the finding: the section at that line is ## 3. Lemonade GPU Verification (rocm serve qwen --engine lemonade ...), so the doc's own command is Lemonade, not vLLM. But your underlying claim is exactly right and the mismatch was real in two places:

  • engines/vllm/src/state.rs hardcoded the literal "rocm_gpu", which fetch_service_device prints verbatim. README.md's rocm gpu 0 and this doc's rocm gpu <n> described a format vLLM never produced.
  • engines/aim/src/state.rs (new on main since this PR opened) had the same literal.
  • Even for Lemonade, reported_device(None, "rocm") returns a bare "rocm gpu", so rocm gpu <n> was not guaranteed there either.

What changed:

  • engines/vllm/src/state.rs:87 gains a reported_device mirroring Lemonade's: "rocm gpu {index}", falling back to "rocm gpu". write_running_state now persists gpu_indices (introspection only, never read back to relaunch) so the ordinal is available. Test: reported_device_matches_the_shape_lemonade_reports (state.rs:759).
  • engines/aim/src/state.rs:43 reports "rocm gpu". No ordinal: AIM's state records no gpu_indices (the container gets the device list), so claiming one would be inventing it — stated in the comment rather than left implicit.
  • docs/manual-testing.md:286-293 now describes the real contract: rocm gpu <n>, or a bare rocm gpu when no ordinal was pinned, and says the first word is the half that decides the check. README.md's example is now accurate for every engine.

Side effect worth flagging: main's every_advised_command_parses guard reads a backticked `rocm gpu 0` as advice to run a command. That gate was failing on this branch after the merge — six findings across README.md, docs/manual-testing.md and the two engines. They are now NOT_INVOCATIONS entries with reasons (apps/rocm/src/advised_commands.rs), which is the escape hatch that guard documents.

I could not verify the printed row on real hardware: the dev box has no amd-smi/rocm-smi and no usable GPU telemetry, so this is unit-level plus the strengthened e2e step, not an observed device: line.

Comment thread apps/rocm/src/main.rs Outdated
// actually reports right now. Only meaningful once the engine is ready
// to answer, same gate `rocm serve`'s summary uses.
if record.status == "ready"
&& let Some(device) = fetch_service_device(paths, &record.engine, &record.service_id)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor efficiency/behavior-change note: before this PR, rocm services list (render_services_text) only read the locally persisted ManagedServiceRecords — pure local I/O. Now, for every record with status == "ready", it calls fetch_service_device, which issues a live engine Healthcheck RPC. For the Lemonade engine this RPC itself performs real HTTP requests to the running server's /v1/health and (on a miss) /v1/models endpoints (engines/lemonade/src/process.rs::query_loaded_model_endpoint), each with its own multi-second timeout, on every invocation (the latch only short-circuits the separate "has inference been verified" check, not this listed query).

So rocm services list --all with several ready services now makes that many live network round-trips to each server before printing anything, and a single slow/stalled/unresponsive server can noticeably delay what used to be an instant, purely-local listing command. This is probably an acceptable tradeoff for the new device: info, but worth confirming it's intentional — e.g. whether services list should cap total wait time or run these checks concurrently rather than sequentially per record.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and bounded in f54a8d2f. Your reading of the cost is right: render_services_text was pure local I/O, and the row made it one live Healthcheck RPC per ready record, sequentially, each with its own multi-second engine-side timeout. @juhovainio raised the same surface from the timeout angle, and the answer to his question turned out to be that engine_request enforces no deadline at all — engine_request_with_env_root ends in a bare child.wait().

What changed: fetch_service_devices (apps/rocm/src/engine_ipc.rs:365) runs the probes under one shared SERVICE_DEVICE_PROBE_BUDGET (:345, 10s). Once spent, the remaining services get no device: row rather than each adding its own wait. So rocm services list --all is now flat in the number of services instead of linear in it, and the row is informational so dropping it is strictly better than a long hang. services_text.rs:441 is the call site.

On your "run these concurrently" suggestion — I looked and it is not available: engine_request routes the three built-in engines through with_scoped_builtin_engine_env, which takes a process-wide mutex and mutates process-global ROCM_CLI_* env vars for the duration of the call. Concurrent probes would serialise on that lock anyway, and racing them would be unsound rather than merely useless. A cap was the option actually on the table, which is why I took it. The doc comment on fetch_service_devices says this so the next person does not re-derive it.

One honest limit, also in the doc comment: the budget is checked between probes, not inside one, so a single external plugin that never exits still blocks. Bounding that needs a detached thread, and abandoning a worker that holds the env lock mid-probe would leak mutated process env into the next operation. That is a worse trade than the one it fixes, and it is pre-existing behaviour of every engine call site in the CLI.

@nowycondro

Copy link
Copy Markdown
Collaborator

🤖 Automated review (Claude Code) · bce633d

This supplements my review posted above (PR review #5655784868) with the required format.

Findings (COMMENT only, non-blocking severity noted per item):

  1. Blocking-candidate (medium confidence) — apps/rocm/src/therock.rs (unsupported_host_gpu_message): the new closing sentence unconditionally promises rocm serve <model> will run on Lemonade's Vulkan backend. But select_serve_engine (apps/rocm/src/serve_cmd.rs) allows a user's configured_default_engine to override the host-GPU-based engine choice, and that override is never validated against host GPU support. On a Vulkan-only host with default_engine configured to something like vllm, this message's promise is false. This matches the open, unresolved concern from juhovainio's CHANGES_REQUESTED review at this same head commit (bce633d) — I independently traced the same code path and confirm the issue still holds as of this commit.

  2. Non-blocking (medium confidence) — docs/manual-testing.md / README.md / apps/rocm/src/serve_summary.rs: docs describe the device line format as rocm gpu 0 / rocm gpu <n>, but vLLM's Healthcheck handler (engines/vllm/src/state.rs) hardcodes the literal string "rocm_gpu" (underscore, no index) which is surfaced verbatim. For vLLM serves the actual rendered text is device: rocm_gpu, not the documented format. The PR's own test plan acknowledges this e2e step couldn't be exercised live, which is consistent with this gap going unnoticed.

  3. Non-blocking / efficiency (low confidence on real-world impact) — apps/rocm/src/main.rs (render_services_text / fetch_service_device): rocm services list now performs a live per-service Healthcheck RPC instead of only reading local persisted state, with no visible timeout bound in the diff itself. This mirrors a concern already raised by juhovainio's review at bce633d — worth confirming there's an upstream timeout in engine_request so a stalled third-party engine can't stall the whole listing.

What I read: the full diff for this PR at bce633d, plus the surrounding non-diff context for select_serve_engine, GpuSupportLevel/gpu_support_level derivation, append_examine_engine_inventory/render_engine_inventory_text_for_host, DeploymentSummary construction sites, and the e2e feature/step files. I also read the prior review thread (copilot, juhovainio, r0x0r) to check whether previously flagged issues were addressed.

What I could not verify: no Rust toolchain was available in this sandbox, so none of this was confirmed by cargo build/cargo test — this is a static reading of the diff and surrounding code, not a dynamic verification. I also could not independently exercise the vLLM/Lemonade e2e scenarios.

No other correctness issues found beyond what's listed above and what's already been raised in the existing review thread.

🤖 Generated with Claude Code

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · bce633d

Review

Full independent review of the whole PR diff (base d4e7c75 → head bce633d6, 14 files, +856/-53), using a two-worker fan-out plus direct source verification of every surviving finding. Outcome: needs work.

Intent: implements most of ROCMAI-550 ("clear messages for AMD GPUs that ROCm doesn't support but Vulkan does") — adds a GpuSupportLevel classification, GPU-aware Lemonade engine wording in rocm examine/rocm engines/the codex-bridge inventory, an install-sdk remediation sentence, and live backend/device reporting in rocm serve's summary and rocm services list. The ticket is still "In Progress" and the PR does not claim to close it; its own scope note says part of the ticket (gfx90c detection/rocm install sdk wording) was already covered by a prior PR (#648), which is consistent with what the ticket itself asks for.

Blocking

  • GpuSupportLevel::Unknown still falls through to the stale "ROCm llama.cpp backend" claim — apps/rocm/src/main.rs:2831-2838 (lemonade_engine_note): the match has an explicit VulkanOnly arm and a catch-all _ arm returning "default embedded Lemonade server with ROCm llama.cpp backend". _ catches both RocmHip (correct) and Unknown (wrong) — and per host_gpu.rs, Unknown covers both an unrecognized gfx target and no AMD GPU detected at all. So on a host with no AMD GPU, rocm examine, rocm engines, and the codex-bridge/MCP "engines" tool all still unconditionally print the exact claim this PR (ROCMAI-550) exists to remove. The one test covering this case, engines_inventory_does_not_claim_vulkan_for_an_unrecognized_gpu, only asserts the negative (!output.contains("no ROCm/HIP support")) and never asserts the positive ROCm claim is absent, so this passes CI undetected.
    Confidence 90 · logic · Fix: add an explicit GpuSupportLevel::Unknown arm with wording that doesn't assert either backend (e.g. "GPU support not yet determined"), and extend the existing test to assert the ROCm string is also absent.

  • unsupported_host_gpu_message's new remediation sentence can be false under a configured engine override — apps/rocm/src/therock.rs:3463: unconditionally appends "Try \rocm serve ` - it runs on Lemonade's Vulkan backend instead."Butselect_serve_engine (apps/rocm/src/serve_cmd.rs:60-109) checks a configured default_engine*before* any host-GPU preference (lines 73-78 return early on a non-empty configured value; the host-GPU branch at line 80 is only reached if that's unset). So a user on an unsupported-ROCm-GPU host with e.g.default_engine = "vllm"configured gets a message promising a Vulkan/Lemonade fallback thatrocm serve will not actually select. The sibling message (unsupported_requested_target_message) had this exact class of bug fixed on this same branch (commit 16ccfb3) with the stated reasoning "only reached once host detection itself confirmed no ROCm build" — that reasoning holds for host detection, but doesn't account for select_serve_engine's config-override precedence, so the asymmetry fix was incomplete. Confidence 85 · logic · Fix: condition the sentence on what select_serve_engine` will actually pick (or soften it to not name a specific backend unconditionally, the way its sibling message was changed to do).

  • Two CI checks are currently failing: E2E tests (Strix Halo, Windows, current) and E2E consolidated report (self-hosted) (re-verified live against head bce633d6 immediately before this review). This review could not fetch the job logs (network egress to the Actions API was blocked in this review environment), so the root cause is not independently confirmed from here — the job names suggest a Windows self-hosted hardware lane and its rollup, which has been attributed elsewhere to download-infrastructure flakiness unrelated to this diff, but that should be reconfirmed by someone who can read the logs rather than assumed clean.
    Confidence 100 (checks are red, verified live) · process · Fix: confirm cause, re-run if infra-flake, or fix if real.

Non-blocking

  • rocm services list --json (and the MCP services tool) never gain the new device field — ManagedServiceRecord (crates/rocm-core/src/service_record.rs:29) is serialized verbatim for JSON (render_services_json, apps/rocm/src/main.rs:6834) and carries no device field; only the text renderer calls the new fetch_service_device. A scripting/MCP consumer gets strictly less information than a human reading text output now does.
    Confidence 85 · architectural · Fix: either add device to the JSON record (live-fetched, not persisted) or document the asymmetry.

  • No test exercises the positive render path of the new device: line, and the one e2e assertion that does touch it is weak — tests/e2e-cucumber/tests/e2e/serving_steps.rs's assert_services_list_reports_device only checks that a line starting with device: exists; it never checks the value. This is currently safe only because fetch_service_device (apps/rocm/src/main.rs:10323) already filters out empty/"unknown" values before the CLI ever writes the label — an invariant the test itself doesn't pin, so if that filter weakened, this step wouldn't catch it. On the unit side, serve_summary.rs's device_row_shown_only_when_known test only covers "shown when known" / "absent by default" — nothing seeds a real ready-state Healthcheck response end-to-end and asserts the exact rendered value.
    Confidence 75 · test-coverage · Fix: add a test that seeds a concrete device string and asserts it appears verbatim.

  • vLLM's actual device string doesn't match the documented format — engines/vllm/src/state.rs:64 hardcodes the literal "rocm_gpu" (no GPU index, underscore not space), while this PR's own doc comment (serve_summary.rs), README.md, and docs/manual-testing.md all document the shape as "rocm gpu 0" / "rocm gpu <n>" (matching Lemonade's {backend} gpu {index} format in engines/lemonade/src/state.rs:60-64). Not a functional bug — the string is still truthful — but this data was never user-facing before this PR, and now it is, inconsistently.
    Confidence 80 · mechanical · Fix: either have vLLM's healthcheck report an index-bearing device string to match, or adjust the documented examples to not imply one exists for vLLM.

  • gpu_support_level(...) is called with raw positional args, duplicated at two sites — build_codex_bridge_snapshot (apps/rocm/src/main.rs:2715) and examine_human_report (apps/rocm/src/main.rs:~5097) both call rocm_core::gpu_support_level(examine.detected_gfx_target.as_deref(), examine.compatible_therock_family.as_deref()) directly instead of through a single-call helper analogous to the PR's own gpu_support_level_for_host_gpu. Both arguments are Option<&str>, so a future transposition at either site would compile and silently misclassify.
    Confidence 75 · mechanical · Fix: add an ExamineSummary::gpu_support_level(&self) (or similar) and route both call sites through it.

  • apps/rocmd/src/common.rs:186 (rocmd_engine_inventory) still hardcodes the unconditional claim — "embedded Lemonade server with ROCm llama.cpp backend", with no GpuSupportLevel parameter and no comment in this file explaining the exemption. The only stated rationale for excluding rocmd lives in a doc comment on a different function in a different file ("the still-host-blind consumers ... keep the static row"), but README.md describes rocmd as "the background telemetry daemon used by the dashboard" that's auto-started locally — which is in tension with treating it as answering for an arbitrary, not-necessarily-local host. May be intentional, but it's unverified from the code as it stands.
    Confidence 70 · logic · Fix: add a comment at the rocmd call site itself stating the rationale, or make it GPU-aware too if it in fact usually runs on the host it describes.

  • rocm services list's text render now makes a live, synchronous, unparallelized RPC per listed service (fetch_service_device → engine_request, apps/rocm/src/main.rs:6730-6804) where it previously only read local disk state. Each call is individually timeout-bounded, so it won't hang indefinitely, but total latency now scales linearly with the number of concurrently-ready services being listed.
    Confidence 70 · architectural · Fix: consider parallelizing the per-service fetch, or note the latency tradeoff is accepted.

Decisions for the author

  • GpuSupportLevel ships three tiers (RocmHip/VulkanOnly/Unknown) rather than the ticket's literal "full ROCm/HIP support, Vulkan only, or unsupported" — tradeoff. Collapsing "unrecognized gfx target" and "no GPU detected" into Unknown instead of a confirmed Unsupported avoids a false-negative claim on a detection gap, and is well-documented in host_gpu.rs's doc comments with a dedicated test guarding the distinction. Worth the author explicitly confirming this reading of the ticket's "unsupported" tier is the intended scope, since it is a visible wording deviation from the ticket text.

Positive signals

  • gpu_support_level/gpu_support_level_for_host_gpu (crates/rocm-core/src/host_gpu.rs) are genuinely a single source of truth with a dedicated test (gpu_support_level_for_host_gpu_reads_a_vulkan_only_summary_as_vulkan_only) specifically guarding against the two positional Option<&str> arguments being silently transposed.
  • lemonade_engine_note (apps/rocm/src/main.rs:2831) is a single shared helper reused by all three GPU-aware call sites (rocm examine, rocm engines, codex-bridge) rather than duplicated inline per site, so the three renderers can't independently drift apart on the parts it does cover.
  • serve_summary.rs's redacted() test helper (masks api key: ... before interpolating a rendered summary into a panic message) is a thoughtful, specific guard against leaking secrets into retained CI logs.
  • The PR description's test-plan section is unusually candid about what wasn't verified end-to-end (both GPU canaries failing locally for disclosed, pre-existing environment reasons) rather than silently claiming full coverage.

Deployment notes

  • The PR currently shows mergeable: CONFLICTING / mergeStateStatus: DIRTY against current main (re-verified live immediately before posting this review) — it needs a rebase/conflict resolution before it can be merged, independent of the findings above.
  • This surfaces a live per-service network RPC inside rocm services list and rocm serve's summary that did not exist before (see the latency non-blocking item above); operators scripting against these commands at scale should be aware the command's cost profile changed.

What this covered

Read the complete diff (base d4e7c75194d1643eacc3b1bda33afe10766def73 → head bce633d6e143d72ee8a147bb189f7dbbd72552f0, all 14 changed files) directly for crates/rocm-core/src/host_gpu.rs, crates/rocm-core/src/lib.rs, apps/rocm/src/serve_cmd.rs, apps/rocm/src/serve_summary.rs, apps/rocm/src/therock.rs, README.md, and docs/manual-testing.md, including the functions each diff touches, their callers, and surrounding tests. apps/rocm/src/main.rs, apps/rocm/src/chat.rs, apps/rocm/src/providers.rs, and the four e2e test files were covered by two independent fan-out workers; every finding either worker reported above was independently re-derived and confirmed against the source by this review before inclusion (exact line reads, not taken on the workers' word). Also independently grepped the whole repository tree at head for the literal phrase "ROCm llama.cpp backend" to check for other remaining stale-claim sites. Checked live CI status and PR mergeability twice (once mid-review, once immediately before posting); could not fetch Actions job logs (network egress to the Actions API is blocked in this review sandbox) so the two failing checks' root cause is reported but not independently confirmed. Did not run cargo test/cargo clippy locally (no Rust build performed in this review session) — all findings are from static reading of the diff and surrounding code, not execution. Retrieved the named ticket (ROCMAI-550) directly from the tracker to check the description's claims against it, rather than from the PR body alone.

This review's own live-state check (required before reviewing, to avoid duplicating prior coverage) necessarily surfaced the existing review thread on this PR as a side effect of querying gh pr view --json reviews; this review did not treat any of that thread's conclusions as given — every finding reported above was re-derived from the source code directly (function bodies, call sites, and tests read and traced independently) before being included. Not reconciled against that thread beyond that.

Resolves the conflicts with main's `main.rs` modularization (Phase 6f,
ROCMAI-208/213/214/215) and addresses the outstanding review findings in
the same pass, because several of the fixes land in code this merge had
to relocate.

Conflict resolution. Every large hunk had an empty `origin/main` side:
main moved the code out of `main.rs`, it did not delete it. The HEAD
block was dropped and this branch's additions re-applied in the module
main moved them to.

- `examine_human_report`, `render_engine_inventory_text_with_paths` /
  `_for_host` and `append_examine_engine_inventory` ->
  `examine_report_text.rs`, with the seven `append_examine_engine_inventory`
  test call sites and the five relocated inventory tests.
- `render_services_text`'s `device:` row -> `services_text.rs`, with the
  `!rendered.contains("device:")` assertion that pins the `"unknown"` case.
- `fetch_service_device` -> `engine_ipc.rs`, beside the `engine_request`
  it calls; `serve_cmd.rs` imports it from there.
- The codex-bridge inventory stayed in `main.rs`, so its three tests
  went back into `main.rs`'s test module.

All 21 test functions this branch adds are present and run; `engine_inventory()`
now carries a third engine (`aim`) from main and the renderers handle it
unchanged. No scenario ids were added by this branch (it only appends
`Then` steps to serve-05, serve-07 and runtime-19), so nothing needed
renumbering.

Review findings.

- `unsupported_host_gpu_message` pins `--engine lemonade` instead of
  suggesting a bare `rocm serve <model>`: a configured `default_engine`
  out-ranks the host preference in `select_serve_engine`, so the bare
  command could attempt vLLM on a GPU with no ROCm build.
- `lemonade_engine_note` now matches `GpuSupportLevel` exhaustively.
  `Unknown` no longer inherits the ROCm claim through a catch-all; it
  asserts no backend and points at `rocm examine`.
- `fetch_service_device` hands the engine the service's 0600 key file via
  `ROCM_SERVE_API_KEY_FILE`, so a public / `--require-api-key` service
  does not 401 its own healthcheck and lose the row.
- `rocm services list` probes under one aggregate budget
  (`fetch_service_devices`), so the row cannot turn a local-only render
  into an unbounded wait.
- `engines/vllm` and `engines/aim` report the device in the same
  `<backend> gpu [<n>]` shape `engines/lemonade` does, instead of the
  literal `"rocm_gpu"` that README.md and `docs/manual-testing.md` never
  described.
- The `services list` e2e step parses the device value, checks it against
  the engine that served, and cross-checks it with the `rocm serve`
  deployment summary.

Verified on the Linux MI300X box: `cargo fmt --all --check`, `cargo
clippy --workspace --all-targets -- -D warnings`, `cargo clippy --locked
-p e2e-cucumber --test e2e -- -D warnings` and `cargo test --workspace
--all-targets --no-fail-fast` all rc 0, 4823 tests passed / 0 failed.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

Merged main, addressed the outstanding findings — f54a8d2f

The branch was 36 commits behind and conflicting. This push resolves that and fixes the open review findings in the same commit, because several of the fixes land in code the merge had to relocate.

Merge resolution

main ran a large main.rs modularization (Phase 6f, ROCMAI-208/213/214/215). Seven of the eight conflict hunks had an empty origin/main side — main moved that code out of main.rs, it did not delete it. For each one the HEAD block was dropped and this branch's genuinely-new content re-applied in the module main moved it to:

this branch's content new home
examine_human_report's gpu_support_level derivation, render_engine_inventory_text_with_paths → _for_host split, append_examine_engine_inventory's new parameter, both engine_inventory_note call sites apps/rocm/src/examine_report_text.rs
render_services_text's device: row apps/rocm/src/services_text.rs
fetch_service_device apps/rocm/src/engine_ipc.rs (beside the engine_request it calls); serve_cmd.rs imports it from there
builtin_codex_bridge_engine_inventory, engine_inventory_note, lemonade_engine_note, host_gpu_support_level stayed in main.rs (main did not move them)

Tests were the thing most at risk here, since three of the dropped blocks were test modules. Relocated and verified present by name in the final tree, all 21 of them:

  • examine_report_text.rs — the seven append_examine_engine_inventory call sites, plus examine_engine_inventory_names_vulkan_… / …_keeps_rocm_note_… and the three engines_inventory_….
  • services_text.rs — the !rendered.contains("device:") assertion inside render_services_text_lists_live_services_by_default_and_all_on_request, which pins that an engine reporting "unknown" prints no row.
  • main.rs — the three codex_bridge_engine_inventory_….
  • chat.rs, serve_summary.rs, crates/rocm-core/src/host_gpu.rs, therock.rs auto-merged; their tests were checked by name too.

Other merge notes: engine_inventory() now carries a third engine (aim) from main, which the renderers handle unchanged. No scenario renumbering was needed — this branch adds no scenarios, only Then steps to the existing serve-05, serve-07 and runtime-19. I checked every feature file for duplicate ids against origin/main: none.

Findings addressed

The engine promise was wrong (@juhovainio, @nowycondro, @copilot-pull-request-reviewer — three threads, one defect). unsupported_host_gpu_message promised rocm serve <model> runs on Lemonade/Vulkan, but select_serve_engine ranks a configured default_engine above the host preference, and rocm config set-default-engine never validates against the GPU. apps/rocm/src/therock.rs:3607 now pins the engine — rocm serve <model> --engine lemonade — which out-ranks every selection source, so the sentence holds for any config and any recipe. New test therock.rs:8399 pins the pinned form and the absence of the bare one (the bare command is a prefix of the pinned one).

Unknown still asserted ROCm support (@copilot-pull-request-reviewer). lemonade_engine_note (main.rs:2748) now matches GpuSupportLevel exhaustively instead of folding Unknown into a _ => ROCm arm. Unknown claims no backend and points at rocm examine; it does not get the Vulkan wording either, because a detection gap is not a confirmed absence. Both Unknown tests now assert all three facts.

The healthcheck was unauthenticated (@copilot-pull-request-reviewer). Verified the mechanism: the engines read ROCM_SERVE_API_KEY_FILE for their own /v1/models probe, so a public / --require-api-key service 401'd its own healthcheck and the row vanished for exactly the protected services. engine_request_with_scoped_env (engine_ipc.rs:152) now threads the service's 0600 key file — inside the same BUILTIN_ENGINE_ENV_LOCK critical section for the built-in path, on the child Command for an external plugin.

services list cost and timeouts (@juhovainio, @nowycondro). The answer to "does engine_request enforce a timeout upstream" is no — it ends in a bare child.wait(). fetch_service_devices (engine_ipc.rs:365) now runs the probes under one SERVICE_DEVICE_PROBE_BUDGET (10s), so the cost is flat in the number of services instead of linear. Not parallelised, and the reason is in the doc comment: with_scoped_builtin_engine_env takes a process-wide mutex and mutates process-global env, so concurrent probes would serialise anyway and racing them would be unsound. The budget is checked between probes, not inside one — a hung external plugin still blocks, which is pre-existing CLI-wide behaviour and is stated rather than implied.

The documented device: format did not exist (@nowycondro). engines/vllm hardcoded the literal "rocm_gpu" and engines/aim (new on main) did the same, while README.md and docs/manual-testing.md described rocm gpu <n>. Fixed on the code side: vLLM now reports rocm gpu <n> (persisting gpu_indices for the ordinal) and AIM reports rocm gpu, both matching Lemonade's shape; the docs now also describe the ordinal-less form, which Lemonade could already produce. One attribution correction in that thread: the section is the Lemonade one, not vLLM — the finding stands regardless.

The e2e assertions had no teeth (@copilot-pull-request-reviewer, two threads). assert_services_list_reports_device now parses the value (<backend> gpu [<n>], nothing trailing), checks the backend against the engine that served (vulkan from vllm/aim fails; unknown fails everywhere), and cross-checks it against the rocm serve deployment summary via a new E2eWorld::serve_summary. The summary check is two-sided — equal if present, and if absent the summary must not have said ready — because a managed launch can legitimately return before the weights load.

Judged already handled, not silently skipped: the two Gherkin-coverage threads. A scenario for the Vulkan-only branch needs a Renoir/Cezanne Vega APU on a runner and detect_host_gfx_target has no test override, so it would be permanently skipped — the reasoning is in the PR description and restated in each thread. runtime-19 asserts the half a lane can reach (the negative).

Failing checks

Both were on the stale 2026-10-17 run and neither is this PR's:

  • E2E tests (Strix Halo, Windows, current) — repo.amd.com returned HTTP 403 for therock-dist-windows-gfx1151-7.14.1.tar.gz after 6 attempts, so the Lemonade backend install failed and rocm engines install exited 1. Transient upstream artifact availability. main's own run on 11a60691 (today) has this lane green.
  • E2E consolidated report (self-hosted) — strictly downstream of the above: lane e2e-gpu-strix-windows ran (needs result: failure) but uploaded no e2e-gpu-strix-windows-report. The lane died before writing a results dir, so the guard that every lane contributes an artifact fired. It appears "only on this PR" because only this PR had the Windows lane fail; there is nothing to fix here, and a fresh run on this head should decide it.

Verification — Linux MI300X box, exit codes captured

Darwin cannot build this workspace (engines/lemonade: st_mode is u16 there), so everything was run remotely against the exact merged tree (git diff origin/main..HEAD applied onto origin/main; the patch sha256 matches the pushed commit byte for byte).

cargo build --workspace --all-targets                          rc 0
cargo fmt --all --check                                        rc 0
cargo clippy --workspace --all-targets -- -D warnings          rc 0
cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings rc 0
cargo test --workspace --all-targets --no-fail-fast            rc 0   4823 passed / 0 failed

All 21 of this branch's test functions were confirmed ... ok by name in that log, including every relocated one.

Falsification, on the box, each run confirming a non-zero test count:

  1. lemonade_engine_note's Unknown arm restored to the ROCm string → engines_inventory_does_not_claim_vulkan_for_an_unrecognized_gpu and codex_bridge_engine_inventory_does_not_claim_vulkan_for_an_unrecognized_gpu red.
  2. unsupported_host_gpu_message reverted to the bare rocm serve <model> → unsupported_gpu_error_pins_the_engine_a_configured_default_could_override and requested_target_error_does_not_claim_which_backend_serving_will_use red.

Both restored, all green again.

Also caught by the merge: main's new every_advised_command_parses guard reads a backticked `rocm gpu 0` as advice to run a command, and was failing on this branch with six findings. They are now NOT_INVOCATIONS entries with reasons in apps/rocm/src/advised_commands.rs — the escape hatch that guard documents.

Push hygiene: pushed with SKIP=cargo-clippy,cargo-test because those two pre-push hooks cannot compile on Darwin. That is legitimate only because both were really run on the Linux box first, with the results above. Nothing else was skipped — verify-commits (signature + DCO), license-headers, cargo fmt, cargo clippy (e2e harness + steps), lychee and the manifest/catalog/TPN hooks all ran and passed locally. No --no-verify, no force-push.

What I could not verify

The dev box has no amd-smi/rocm-smi and no usable GPU telemetry despite KFD nodes, so none of the real-hardware reporting this PR is about was observed there: no live device: row, no Vulkan-only host exercising the VulkanOnly branch, and no protected service proving the API-key fix turns a 401 into a device row. Those remain unit-level plus the strengthened e2e steps, which is why the e2e assertions got the attention they did in this round.

@r0x0r
r0x0r requested a review from nowycondro November 16, 2026 13:34
@r0x0r

r0x0r commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up to the review fixes: main moved again while that round was in flight (#1041 and #1069 landed), so this branch needed a second merge. Pushed as 069ba0e4b; the PR is mergeable again.

One file conflicted — apps/rocm/src/chat/mcp_bridge.rs — in two places, both the same shape, each side having appended to the same list:

  • The use crate::{...} group is resolved as the union: this branch's host_gpu_support_level and main's render_model_registry_text_with_context_and_host are both kept. Main's new test calls the latter, so taking either side alone would have failed to compile rather than failing quietly.
  • The test module's two appended blocks — this branch's engines-tool Vulkan tests and main's model-tool context-size tests — are independent and both kept whole. Neither was spliced into the other.

Re-verified on Linux after the merge: cargo fmt --all --check, clippy --workspace --all-targets, clippy -p e2e-cucumber --test e2e, feature_naming, check-architecture-doc / check-crate-edges / check-module-size (427 files, none over), and cargo test -p rocm --bin rocm (1899 passed).

Re-requesting review, since all four blocking findings from #pullrequestreview-5917831375 are now addressed. The one thing I still cannot claim is the end-to-end Windows result: the fix for finding #1 is pinned by a unit test that publishes the endpoint inside the probe closure and fails without the production call, but only a Strix Halo Windows lane re-run confirms the five scenarios actually go green.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the current diff and have three things I'd want fixed before merge, plus some smaller notes. Details are inline.

Blocking

  1. The "about 18 seconds worst case, however many servers" claim (README and the services list help) is only true for built-in engines. External plugin engines are waited on with no deadline, as engine_ipc.rs itself says, so the doc promises a bound the code doesn't give. Narrow the wording or skip the probe for non-built-in engines.
  2. The PR body no longer matches the diff. It cites serve-55 while the diff adds serve-59, and it omits the rocm engines banner rewrite, the host-aware bridge engines tool, the vLLM/AIM device string change (rocm_gpu becomes rocm gpu N (name)), the new gpu_name/gpu_indices state keys, refresh_published_anthropic_endpoints, the service_log_follow.rs move and the help-text emphasis test. It also names no lane for the serve-05/07 device assertions, and there is no scenario or stated reason for the rocm engines/bridge changes.
  3. The cross-surface device: assertion in serving_steps.rs looks vacuous: it strips a device: prefix the serve summary doesn't print, so the comparison likely never runs.

Non-blocking

  • ~78 lines of new logic in main.rs; a sibling module would be better.
  • Positional gpu_support_level calls bypass the transposition guard added in this PR.
  • Review-history narration in comments will go stale.
  • retry_service_device makes a hard failure wait out the full probe budget.
  • "no AMD GPU was detected" can contradict the same report on Windows.
  • examine-49 picks its expected command from the line under test, and the rocm engines assertion is over-broad.

Also, ROCMAI-550 is marked Done while this is open, and the docs/skill/dash-TUI items listed under "Deferred" in the description no longer have a tracker home. Is there a follow-up for them?

CI is green on the current head. The anchor, doc and test points above are from reading the diff; I haven't run the e2e scenarios, which need a GPU lane.

Comment thread README.md

`list` asks each running server which backend and GPU it is actually on, so it
can take a few seconds rather than returning instantly. The query is capped in
total — about 18 seconds worst case, however many servers are listed — and a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"About 18 seconds worst case, however many servers are listed" only holds for built-in engines: engine_ipc.rs documents that engine_request waits on an external plugin child with no deadline, so a ready plugin-backed service can still hang rocm services list and the chat services command. Narrow this wording (and the ServicesCommand::List help in cli.rs) to built-in engines, or skip the probe for non-built-in engines.

Comment thread apps/rocm/src/cli.rs
///
/// In the default table, each running server is asked which backend and GPU
/// it is actually on, so this takes a few seconds rather than returning
/// instantly. The asking is capped in total (about 18 seconds worst case,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same claim as the README: the aggregate cap does not cover external plugin engines (unbounded child wait). Say "built-in engines", or only probe those.

Comment thread README.md
The device line is omitted when the engine cannot say yet. A server is reported
ready as soon as it answers, which can be before it has finished loading the
model, and until then the engine does not know which device it ended up on;
`rocm serve` waits a few seconds for it rather than holding your terminal, and

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Waits a few seconds" understates it: the code is a 6s start gate plus one in-flight probe (about 14s per the engine_ipc.rs comment) on a launch that may then print no row. Align this with docs/testing.md and the code.

/// The value of the first `device:` row in `text`, trimmed.
fn device_row_value(text: &str) -> Option<&str> {
text.lines()
.find_map(|line| line.trim_start().strip_prefix("device:"))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This strips a device: prefix, but the serve summary renders padded table labels (the serve_summary.rs unit test matches starts_with("device") with no colon). If so, device_row_value(summary) is always None, the assert_eq!(summary_device, device) below never runs, and the step passes vacuously. Parse the row the way the unit test does, and fail the step if a printed row was not compared.

Comment thread apps/rocm/src/main.rs
/// drift into disagreeing about the same host. Tests exercise the renderers
/// rather than this function: asserting on it directly would only restate the
/// condition below instead of proving a surface prints it.
pub(crate) fn engine_inventory_note(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds ~78 lines of host-aware inventory logic to main.rs (engine_inventory_note, lemonade_engine_note, host_gpu_support_level). The architecture guardrails ask for new domain logic in its own file with a docs/architecture.md entry; examine_report_text.rs (which hosts the renderers) or a small sibling module would fit.

Comment thread apps/rocm/src/engine_ipc.rs Outdated
/// means (review on #678); giving the tool the row is a change in its own
/// right, not something this constant may be read as already covering.
///
/// # The decision, recorded (review on #678)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review-history framing ("review on #678") will be stale after merge. Keep the rationale for the budget constant here and move the history to the PR description. Same in therock/family.rs, pinned_gpu_name in host_gpu.rs, the gpu_visibility.rs test doc and serving_steps.rs ("An earlier revision asserted...").

if let Some(device) = probe() {
return Some(device);
}
if elapsed() >= SERVE_DEVICE_PROBE_BUDGET {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

retry_service_device treats every None alike, so a hard failure (engine missing, bad env, plugin error) also burns the whole 6s budget plus an in-flight probe on the default rocm serve path. Separate "unknown yet" from "call failed" so a failure returns immediately.

`device:` row of `rocm services list` for the backend it \
used\n"
),
None => " rocm_support: unknown - no AMD GPU was detected; run \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"no AMD GPU was detected" is derived only from detected_gfx_target == None. On Windows an AMD adapter can be present with a driver and no gfx target, so this line contradicts the rest of the same report and sends the user to rocm diagnose as if nothing were there. Say "no gfx target detected", or carry adapter presence into the classification.

// both, so a lane with an unrecognized GPU would have failed against
// *correct* output; no current lane has one, which is the only reason
// it stayed green.
(Some(_), None, _) if line.contains("is not in this build's catalog") => (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expected command is picked by line.contains("is not in this build's catalog"), i.e. from the line under test, so a wrong-tier line cannot fail here (the PR body concedes this). Key it on something independent of the printed line, or name the unit test that pins the tier.

!output.contains("Lemonade server with ROCm llama.cpp backend"),
"an unrecognized GPU must not be told it has ROCm support either:\n{output}"
);
assert!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

output.contains("rocm examine") is checked over the whole rocm engines text, so unrelated rows can satisfy it. Assert on the lemonade row instead, as the bridge test does with lemonade.summary.

main took `serve-59` for the endpoint-key scenario, so this branch's APU
pre-flight scenario moves to `serve-60`; `therock/family.rs` names it in
prose and follows.

Signed-off-by: Roman Sirokov <roman@flowrl.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…nthropic endpoint

The Anthropic proxy is a thread of the engine's own process, and it
publishes exactly once -- as soon as the server lists a model. The engine
adapter rewrites the same state file *whole* immediately after it spawns
that server, and since ROCMAI-550 it spends a host GPU probe building the
document first (`pinned_gpu_name`: pnputil plus a PowerShell CIM query on
Windows, seconds). The publication lands inside that window and is
replaced, and because nothing publishes twice the `anthropic endpoint:`
row never comes back -- five Windows e2e scenarios saw a ready service
with a working `device:` row and no endpoint. The file lock main added in
ROCMAI-453 orders the two writers but does not stop the later one
replacing what the earlier recorded; that is what a full rewrite is for.

`publish_anthropic_endpoint` now notes the publication for this process
before it touches the file, and `write_json_state_file` carries that note
into the document it is about to write. Scoped to this process and this
state file, so a dead endpoint a previous run left on disk is still
cleared by the new run's first write, which `reset_for_restart` relies on.

Also puts the engine state file into the e2e failure message: the row
being absent because nothing published and the row being absent because
something replaced it look identical in `services list`, and telling them
apart cost a Windows lane run per guess.

Signed-off-by: Roman Sirokov <roman@flowrl.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
Picks up #678's fix for the Anthropic endpoint being dropped by the engine's
post-spawn whole-file state rewrite, which is what left this branch's Strix
Halo Windows lane red: the five failing scenarios are #678's, inherited here
because this branch stacks on it, not defects of the Vulkan probe.

Merged clean -- no conflicts.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

The previous fix did not work; here is the real cause, and the lane is now green

The previous fix did not work

refresh_published_anthropic_endpoints in services_text.rs (the "stale snapshot" theory: the listing snapshots records, then spends up to ~18s on the new device probe, then renders from the stale snapshot) was merged, and I re-ran the Windows lane on that head — run 43885834281, job 133363771260. The result was byte-identical to before it: 263 scenarios (256 passed, 7 failed), Reconciliation: 2 xfail … 5 unexpected failure(s), the same five scenarios, the same messages. The re-read could not help, because the key it re-reads was not in the engine state file at all.

The real cause

The engine's own post-spawn state rewrite was deleting the Anthropic endpoint the proxy had just published — and nothing publishes twice, so the row never came back.

The proxy is not a separate writer. run_builtin_engine_serve_http starts anthropic_proxy::spawn_alongside_engine on a thread and then runs the engine adapter in the same process. Both write the one engine state file. The sequence inside engines/lemonade's serve_router is:

  1. spawn_server() — lemond starts;
  2. write_running_state_for_model(…, "running") — a whole-file rewrite, which since this PR evaluates rocm_core::pinned_gpu_name() while building its document. On Windows that is pnputil plus a PowerShell CIM query — seconds, not microseconds;
  3. meanwhile the proxy's 2s poll sees the server list a model, binds, and publishes.

Step 3 lands inside step 2's window, and step 2 then writes a document that never contained the key. Before this PR, step 2's document was built from cheap syscalls and was on disk long before the server was even listening, so the ordering could not invert — which is why the lane is green on main and red here, and why only Windows fails: on Linux the same probe is a couple of sysfs reads.

This also matches the failing job's own evidence exactly: a ready service with a correct device: rocm (AMD Radeon(TM) 8060S Graphics) row — the probe ran and succeeded, inside the write that clobbered the endpoint — and no anthropic endpoint: row anywhere, through 31 consecutive services list invocations over the step's full 30s budget.

The file lock #1092 (ROCMAI-453) added does not cover this, and its own doc says so: it orders the two writers so neither sees a half-written file, but "a full rewrite through write_json_state_file still replaces every key merged before it". That is what a full rewrite is for — it is how a new run clears what the previous run left, which ManagedServiceRecord::reset_for_restart relies on.

What changed

3d59dbcb8:

  • rocm_core::publish_anthropic_endpoint notes the publication for this process (keyed by state-file path) before it touches the file, then merges as before. The in-memory note is released before the file lock is taken, so the two locks are never held in both orders.
  • rocm_core::write_json_state_file carries that note into the document it is about to write, under the file lock. There is no gap: a rewrite that gets the lock first already sees the note; one that gets it after the merge is carrying the key anyway.
  • Scoped to this process and this file, so a dead endpoint a previous run left on disk is still cleared by the new run's first write.
  • rocmd's proxy publishes through that function instead of a bare merge.
  • The e2e failure message now carries services list --json and the engine state file itself. "Nothing published" and "something replaced it" look identical in the listing, and telling them apart cost a Windows lane run per guess.

Tests: a_whole_file_state_write_keeps_the_endpoint_this_process_published and a_rewrite_still_clears_an_endpoint_this_process_did_not_publish in rocm-core, and a_running_state_write_keeps_the_anthropic_endpoint_this_process_published in engines/lemonade driving the real write_running_state. Mutation-checked: with the carry removed, both the rocm-core and the lemonade test fail, and the lemonade failure prints exactly the Windows shape — a complete document with status: running, gpu_name, and no anthropic_* keys.

The Windows lane now passes

Run 44051420537, job 133930090064, on the fixed head:

266 scenarios (264 passed, 2 failed)
Reconciliation: 2 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 0 unexpected failure

All five previously failing scenarios (serve-50 … serve-54) pass. The 2 remaining xfails are the pre-existing --engine vllm pair, unrelated to this PR.

Also in this push

origin/main merged (8fec79c2d). main took serve-59 for the endpoint-key scenario, so this branch's APU pre-flight scenario is now serve-60; therock/family.rs names it in prose and follows. cargo test -p e2e-cucumber --test feature_naming passes.

Not fixed here

engines/vllm and engines/aim write their state files with their own unlocked write_state rather than through rocm_core::write_json_state_file, so they get neither the lock nor this carry-over. That gap predates this PR and is already noted in merge_json_state_file's own documentation; it is a separate change.

Gates, all on Linux (this Mac cannot build rocm-engine-lemonade): cargo fmt --all --check, cargo clippy --workspace --all-targets -D warnings, cargo clippy -p e2e-cucumber --test e2e -D warnings, cargo test -p e2e-cucumber --test feature_naming, cargo xtask check-architecture-doc, check-crate-edges, check-module-size, cargo test -p rocm --bin rocm (1926 passed), cargo test -p rocm-core, cargo test -p rocm-engine-lemonade. Five rocmd supervise:: process-tree-kill tests fail on that box; they fail identically on a pristine origin/main checkout there, so they are environmental (the container cannot kill a process tree), not from this branch.

@r0x0r
r0x0r requested a review from juhovainio November 17, 2026 09:27
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Heads-up on a scenario-number collision between this PR and a sibling, so whoever merges second is not surprised by a red feature_naming.

After main took serve-59 (the endpoint-key ownership scenario), both of these renumbered independently and now claim the same number:

Each is correct within its own branch, because the rule is that a scenario's number equals its position in its file. The collision only exists across them, so whichever merges first forces the other to renumber. That is cheap and safe: cargo test -p e2e-cucumber --test feature_naming fails loudly on a duplicate or out-of-position number, so this cannot slip through silently — it just needs doing, along with the prose references (docs/testing.md's scenario lists and repro filter, and apps/rocm/src/therock/family.rs for #678 / service_launch/readiness.rs for #854).

The @id: tags are the binding key and are unaffected either way, so no step binding moves.

No action needed now; I will renumber whichever branch lands second.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work here. The core design is sound: a single support-level classifier with exhaustive matches, the examine/serve/install messages checked against the state they describe (examine-49, serve-60), and a pure-move split of the log-follow code. I read the full diff; CI is green. I'm requesting changes for a few things that need to be fixed before merge, plus some smaller items.

Needs fixing

  1. Commit messages no longer match the branch. Each lands in public history. Several describe work the final diff does not contain: 794e4b6 (Vulkan probe, host_gpu/vulkan.rs, examine-49/50/51 as first written), and 30144c5, c3e6bf2 and 4ff8d08 (clearing the endpoint key file for plugin children; the stdio_client change was reset to main). Others cite scenario ids that were renumbered (examine-31, serve-55), and some carry "review round on #678" chatter. 16ccfb3 also has a Co-Authored-By: Claude trailer. These should be folded/dropped/reworded, which means a force-push; check the live review state before doing it.
  2. PR body names a scenario that isn't in the diff. It says serve-55 drives detection and the pre-flight from the same planted /sys; the diff adds serve-60. Re-read the body against the final diff.
  3. The device: row has no coverage on the blocking mock e2e job. It is exercised end to end only by serve-05 (GPU lane) and serve-07 (@requires-gpu, xfail'd on lemonade+linux); the blocking job has only closure-injected unit tests. The PR text should say which lane covers it. Likewise serve-60 shows only that the pre-flight accepts the host (it stops at --gpu 99), not that serve --engine lemonade then launches under gpu_required.
  4. ROCMAI-550 scope. The ticket asks for README, docs/rocm-docs, --help, the doctor skill files and docs/ux-guidelines.md to be searched for the static "ROCm llama.cpp backend" claim. docs/rocm-docs, skills/rocm-doctor and ux-guidelines.md are untouched, and the claim survives in the dash-TUI engine manager, rocmd's host snapshot, and the RocmHip Lemonade note (see inline). The PR body says this is deferred, but the ticket is already Done; please either reopen/split a follow-up ticket or finish the sweep.

Smaller items

  • The PUBLISHED_ENDPOINTS carry in rocm-core is process-wide and never evicted (inline). The race it patches comes from this PR's own GPU-name probe inside the post-spawn state write; resolving the name before the spawn would avoid the need for it.
  • AIM/vLLM device changes from the literal rocm_gpu to rocm [gpu n] [(name)]. Please confirm by grep that nothing else (dash-TUI, rocmd snapshot, scripts, docs) compares against the old string.
  • Review chatter in source comments ("review on #678" in engine_ipc.rs, therock/family.rs, host_gpu.rs, a cli.rs test doc) and a contributor aside in docs/testing.md should be trimmed; a few multi-paragraph comments in host_gpu/render.rs are also out of order.
  • The PR is large (~5.6k lines). The log-follow move is forced by the module-size limit and I verified it is a pure move; the anthropic-endpoint carry and the help-text markdown test are separable if you want to shrink it.
  • Optional: the device is a formatted string that the e2e step and device_worth_reporting parse back; a small struct in the protocol crate would avoid that re-parse (a contract change, so only a suggestion).

Inline comments below cover the specifics.

/// nothing and a global would quietly cross-contaminate if one ever did (and
/// does cross-contaminate between tests sharing a process, which is the same
/// bug in miniature).
static PUBLISHED_ENDPOINTS: OnceLock<Mutex<HashMap<PathBuf, (String, String)>>> = OnceLock::new();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Process-wide note that is never evicted: write_json_state_file re-injects this endpoint on every full rewrite of the path for the rest of the process, including after the proxy has gone or the service restarted in-process. The doc rests on "nothing today runs two engines in one process" without naming where that holds. The race exists because pinned_gpu_name (seconds on Windows) was put inside the post-spawn write_running_state; resolving the name before spawning would keep the rewrite fast and leave write_json_state_file a plain replace. Is the global needed?

Comment thread apps/rocm/src/engine_ipc.rs Outdated
/// How long one call can take, since `rocm services list` makes one per ready
/// service: [`engine_request`] itself imposes no deadline, so the bound comes
/// from the engine. The three built-in engines answer in-process and every
/// endpoint probe they make is timeout-bounded (`engines/vllm`'s

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bound on fetch_service_device (and the ~18s/~14s figures in the README and docs/testing.md) rests on every endpoint probe of the built-in engines being timeout-bounded, but only vLLM and Lemonade are cited. AIM's healthcheck also derives listed from the endpoint; please name where its bound is defined, or soften the claim.

Comment thread apps/rocm/src/main.rs
"default embedded Lemonade server; this GPU has no ROCm/HIP support, so it runs on the Vulkan llama.cpp backend"
}
rocm_core::GpuSupportLevel::RocmHip => {
"default embedded Lemonade server with ROCm llama.cpp backend"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This RocmHip note still asserts the ROCm llama.cpp backend from the gfx table. docs/manual-testing.md in this same PR says Lemonade can select ROCm or Vulkan on a supported GPU, so on covered hosts this is the same unestablished claim the PR removes elsewhere. Consider wording that defers to the device: row for the backend actually used.

),
None => format!(
" rocm_support: GPU detected ({}) with full ROCm/HIP support; \
install the SDK for it with `rocm install sdk`\n",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This advises rocm install sdk whenever no managed runtime is recorded; a ROCm-capable host with a working system ROCm gets an install that clears nothing. Make it conditional ("for a managed SDK") or confirm that case cannot reach here.

Comment thread docs/testing.md Outdated
Vulkan-only here is inferred from the gfx table, not probed: this change has
three levels and this section describes three. Probing the host for a Vulkan
driver, and the further levels that evidence makes knowable, is separate work.
Written this way deliberately — a forward reference to an unmerged change reads

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is a note to reviewers about the PR, not testing guidance; please drop it.

Comment thread apps/rocm/src/therock/family.rs Outdated
///
/// Asked of [`gpu_support_level`] rather than re-tested here. The install path
/// used to call `gfx_target_has_no_rocm_build` directly, which was *exactly*
/// equivalent (review on #678 checked that it was) and still wrong to keep: it

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source comments shouldn't cite "review on #678"; state the reason (one classifier for examine and install) and drop the review reference. Same in engine_ipc.rs (two places), host_gpu.rs and the cli.rs test doc.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (merge-base 9101286a to head 3d59dbcb). Implements the GPU/backend-support classifier and its primary CLI/engine-protocol surfaces described in ROCMAI-550, explicitly deferring (per the PR's own description) the dash-TUI/rocmd static-claim surfaces, Windows verification, and hardware-grounded e2e coverage for the VulkanOnly level.

Blocking

  • AIM's reported_device hardcodes a "rocm" backend claim and unconditionally probes/names a GPU, regardless of device_policy — engines/aim/src/state.rs:68-74, :164. On a single-AMD-GPU host, a DevicePolicy::CpuOnly launch (which correctly attaches zero GPU devices per engines/aim/src/process.rs's resolve_use_gpu/docker.rs) still has write_running_state record a real gpu_name, so reported_device renders e.g. "rocm (AMD Instinct MI300X)" for a container that used no GPU — the exact "invented GPU/backend claim" ROCMAI-550 exists to eliminate. The doc comment ("AIM only ever runs on ROCm") is contradicted by DevicePolicy::CpuOnly being a real, tested, reachable variant inside this crate. No test exercises this path. Verified caveat: DevicePolicy::CpuOnly is currently unreachable through any rocm-cli-driven path (the shared parse_device_policy gate in apps/rocm/src/gpu_policy.rs rejects cpu/cpu_only outright for both rocm serve and the IPC launch path), so this is latent rather than live-exploitable today — but it ships in compiled code and becomes live the moment that gate loosens or AIM is invoked directly.
    Confidence 80 · logic/architectural · Fix: gate the backend string/gpu_name probe on the stored device_policy, add a test, correct the doc comment.

  • AIM never records a GPU ordinal in state, unlike lemonade/vllm — engines/aim/src/state.rs:164 vs engines/aim/src/process.rs:285-297 (plan_gpus resolves one) and docker.rs (uses it to set HIP_VISIBLE_DEVICES). Compare engines/lemonade/src/state.rs:91,101, engines/vllm/src/state.rs:181,189. On a multi-GPU host AIM's device: row is a bare "rocm" where lemonade/vllm would report "rocm gpu <n> (<name>)" for the same hardware — a real inconsistency on exactly the axis this PR is about. The justifying doc comment ("the container gets the device list, not an ordinal") is inaccurate given HIP_VISIBLE_DEVICES narrows it the same way lemonade/vllm's does.
    Confidence 75 · architectural · Fix: thread gpus.indices through to write_running_state, or correct the doc comment.

  • AIM's merge_state_field/write_state bypass the file lock and the Anthropic-endpoint carry-forward fix this PR adds elsewhere — engines/aim/src/state.rs:185-191,220-227, plain fs::read/fs::write, no FileLock, no call to carry_published_anthropic_endpoint (cf. crates/rocm-core/src/anthropic_endpoint.rs + crates/rocm-core/src/endpoint_readiness.rs:144-159, wired for lemonade via write_json_state_file_with). record_served_model_id fires right as the container is confirmed ready — the same event that gates the Anthropic-proxy thread's publish — so AIM's own thread can overwrite the file with a stale copy and silently drop a just-published endpoint, the same failure mode this PR fixes for lemonade, via a different writer.
    Confidence 75 · architectural/logic · Fix: route AIM's writers through rocm_core::merge_json_state_file/write_json_state_file, or document the exemption.

  • docs/architecture.md's gfx_target.rs entry is stale — docs/architecture.md (the gfx_target.rs row), zero diff in this PR despite gfx_target.rs gaining ~155 lines that add the entire GpuSupportLevel classifier (gpu_support_level, gpu_support_level_for_host_gpu, VULKAN_ONLY_SUPPORT_CLAIM) — the central piece of this whole PR. The entry still only describes "pure gfx-target lookups." AGENTS.md states the CI gate (check-architecture-doc) verifies presence only, not accuracy, which is why this is still green — but the project's own guidance puts keeping the description true on the author.
    Confidence 80 · documentation · Fix: update the entry.

  • Untested match arm: gpu_support_level's (_, Some(_)) => RocmHip ignores detected_gfx_target entirely — crates/rocm-core/src/gfx_target.rs:608. Every test call site passes Some/Some, None/None, or Some/None; none passes None/Some(_), so mutating the pattern to (Some(_), Some(_)) survives the whole suite. Low current real-world impact since every caller derives the family from an already-detected target, but the public signature permits the combination.
    Confidence 80 · logic/test-quality · Fix: add the missing test case, or narrow the inputs.

  • Untested branch: gpu_name_for_single_gpu_host's ordinal-mismatch guard can be deleted without failing any test — crates/rocm-core/src/host_gpu.rs:544-546. No test supplies gpu_indices=[1] with present_amd_gpus=Some(1); the existing test uses present_amd_gpus=Some(2), so the trailing count check alone already covers it, independent of this branch.
    Confidence 85 · logic/test-quality · Fix: add the missing test case.

  • bridge_snapshot.rs's gpu_support_level wiring is new behavior this PR introduces, with zero test coverage — apps/rocm/src/bridge_snapshot.rs:40-43, file has zero #[test] functions. The downstream renderer (builtin_codex_bridge_engine_inventory) is well-tested elsewhere, but only via a test that injects a pre-computed GpuSupportLevel, bypassing this exact call site — so a positional transposition of the two same-typed arguments (a bug class the classifier's own doc comment warns about) would pass every existing test.
    Confidence 72 · test-coverage · Fix: add a test exercising build_codex_bridge_snapshot's own wiring.

Non-blocking

None scoring ≥70 survived independent verification beyond the items above; see What this covered for lower-confidence candidates that were dropped.

Decisions for the author

  • The VulkanOnly level has no hardware-grounded e2e coverage — answered-by-intent. Verified directly: the only planted gfx90c fixture in the whole e2e suite (host_with_a_kfd_less_gfx90c) is used only by model_serving.feature's serve-60-style scenario (rocm install sdk --dry-run / rocm serve), never by any rocm examine scenario; examine-49's self-consistency check is genuinely non-tautological (it cross-references two independently-rendered fields of the actual CLI output, not a re-call of the classifier) but is never exercised with (Some("gfx90c"), None) on any current lane, so the VulkanOnly wording is pinned only by Rust unit tests against planted inputs. This matches the PR description's own "Deferred, explicitly" section verbatim: "The VulkanOnly level has no hardware-grounded e2e scenario... it is a known, accepted gap here, not an oversight." Would otherwise be a blocking test-coverage finding; confirm this remains the accepted plan.
  • rocm examine --json does not expose the derived GpuSupportLevel/advice, only the raw facts — non-blocking-improvement. apps/rocm/src/examine_cmd.rs has zero diff in this PR (pre-existing gap, not introduced or worsened here), but it sits in tension with the PR's own "one classifier every surface renders from" framing for JSON consumers.
  • Two independent GPU-detection entry points feed gpu_support_level — tradeoff/question. bridge_snapshot.rs's build_codex_bridge_snapshot derives its facts from ExamineSummary::gather(), while main.rs's host_gpu_support_level uses detect_host_gpu_summary(paths) — two separate functions. Both appeared consistent on direct reading, but nothing pins that they always agree; worth a maintainer call on whether that's enforced by construction or should be.

Positive signals

  • Every GpuSupportLevel consumer found (render.rs's support line, main.rs's lemonade_engine_note, examine_report_text.rs, mcp_bridge.rs) uses a compile-time-exhaustive match with no wildcard arm, so a new variant is a compile error rather than a silently stale sentence — genuinely delivers the PR's stated design goal.
  • VULKAN_ONLY_SUPPORT_CLAIM centralizes identical user-facing wording across surfaces with a dedicated word-for-word pin test guarding against independent drift.
  • rocm services list --json's lack of a device field is honestly and specifically documented (README.md, confirmed directly) rather than silently diverging from the text table.
  • The actual GPU-allocation logic (resolve_use_gpu, parse_device_policy) correctly fails fast rather than silently falling back to CPU for GpuRequired/GpuPreferred — AGENTS.md's "no silent CPU fallback" rule is honored in the allocation path even where the reporting path (see Blocking) is not.

Deployment notes

  • User-facing CLI output changes: rocm examine now prints a rocm_support line on every host (additive), and rocm serve/rocm services list grow a device: row — anything scripting/scraping that output should be aware of the new lines.
  • The blocking findings above affect only the aim engine; lemonade and vllm were directly checked and do not share them.

What this covered

Full diff 9101286a...3d59dbcb (40 files, 5599+/1454-, 56 commits). Multi-pass review split into 7 coupled-file groups: (1) core rocm-core GPU/gfx classifier, (2) engine protocol/IPC + Anthropic-endpoint race fix, (3) per-engine state/device-reporting (aim/lemonade/vllm), (4) examine/serve/services render surfaces, (5) CLI wiring/chat bridge/log-follow, (6) e2e feature files + docs/README, (7) e2e step-definition Rust files.

Groups (1) and (2) were run as genuine independent fan-out workers and their findings are included above after my own verification. Groups (3)-(7) are disclosed as not independent: the originally-spawned workers for those five groups did not return verifiably through this session's own notification channel despite being pinged directly and re-spawned fresh once; separately, this session received repeated, increasingly detailed unsolicited messages purporting to relay those workers' "verbatim" findings through an unverifiable channel, which were deliberately not used as input to avoid anchoring a review on unattributable content. Per this method's own rule for a worker that does not return, I ran groups (3)-(7) directly myself in this driver context instead of dropping them, with the explicit understanding that this is not independent of my own judgement or of groups (1)/(2)'s findings. Within that constraint I verified file:line evidence for every finding above by reading the actual repository myself (crates/rocm-core/src/gfx_target.rs, crates/rocm-core/src/host_gpu.rs, engines/aim/src/{state,process,docker}.rs, engines/lemonade/src/state.rs, engines/vllm/src/state.rs, apps/rocm/src/{bridge_snapshot,gpu_policy,serve_cmd,examine_cmd,main,services_text}.rs, apps/rocm/src/chat/mcp_bridge.rs, apps/rocmd/src/anthropic_proxy.rs, crates/rocm-core/src/anthropic_endpoint.rs, crates/rocm-core/src/endpoint_readiness.rs, docs/architecture.md, README.md, and the e2e feature/step files for the VulkanOnly-coverage question), not from any relayed summary.

Not independently re-verified by me within groups (3)-(7): serve_summary.rs's redacted() masking correctness, service_log_follow.rs's full correctness (cursor/footer edge cases), services_text.rs's split-completeness against the new service_log_follow.rs, cli.rs/main.rs dead-code/wiring beyond lemonade_engine_note's exhaustiveness (confirmed), broader Gherkin scenario quality beyond the VulkanOnly-coverage question, and docs/testing.md/docs/manual-testing.md accuracy beyond the two README sections checked directly. Several lower-confidence candidates (a cosmetic double-parens case in format_reported_device, a silent Option-drop in carry_published_anthropic_endpoint for a currently-unreachable non-object input, a test that doesn't actually exercise the carry-forward wiring it's named for) scored below 70 on direct inspection and were dropped per this method's threshold rather than reported.

CI: 50 of 51 checks green at the reviewed head (3d59dbcb) as of this review, including clippy (which also runs check-module-size/check-architecture-doc), the full workspace test suite, CodeQL, and all security scans; E2E tests (MI300X, current) was still in progress/re-running at review time with no failure reported. No failing check was observed.

Not reconciled against this PR's prior review discussion during the findings pass (reached blind, per the method); the long prior-review history on this PR was not read until after the findings above were locked.


🤖 Automated by an AI agent

Comment thread engines/aim/src/state.rs
None,
value_string(state, "gpu_name").as_deref(),
)
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — reported_device hardcodes the "rocm" backend claim and write_running_state (line 164) calls pinned_gpu_name(&[]) unconditionally, regardless of device_policy.

Traced directly: engines/aim/src/process.rs's resolve_use_gpu correctly returns false for DevicePolicy::CpuOnly, and docker.rs correctly omits --device=/dev/kfd/--device=/dev/dri/HIP_VISIBLE_DEVICES in that case — the actual container launch is correct. But write_running_state is called with the same shape regardless of device_policy (process.rs:177-186, :376-383), so on a single-AMD-GPU host a CpuOnly launch still writes a real gpu_name into state, and reported_device still renders e.g. "rocm (AMD Instinct MI300X)" for a container that used zero GPU devices — exactly the "invented GPU/backend claim" this PR (ROCMAI-550) exists to eliminate.

The doc comment above ("AIM only ever runs on ROCm, so the backend half is fixed") is itself contradicted by the code: DevicePolicy::CpuOnly is a real, tested, reachable variant inside this crate (parse_device_policy_arg, resolve_use_gpu_cpu_only_never_uses_gpu). No test in this file exercises write_running_state/reported_device with CpuOnly — the only reported_device test uses GpuRequired.

Important caveat, verified directly: DevicePolicy::CpuOnly is currently unreachable through any rocm-cli-driven path — apps/rocm/src/gpu_policy.rs's parse_device_policy (the single shared gate used by both rocm serve and the IPC launch path) unconditionally rejects "cpu"/"cpu_only" before any engine is invoked. So this is a latent defect in AIM's own crate, not a live bug through today's CLI — but it ships in compiled, user-facing code and becomes live the moment the gate is loosened or the engine binary is invoked directly, which the engine's own thorough CpuOnly modeling suggests was anticipated.

Confidence 80. Suggested fix: gate the backend string and the gpu_name probe on the stored device_policy (report "cpu" or omit the name when CpuOnly), add a test for that path, and correct the doc comment.

Comment thread engines/aim/src/state.rs Outdated
// container is handed a device list, not an ordinal — and
// `rocm_core::pinned_gpu_name` still declines on any host where
// more than one AMD GPU could be the one that answered.
"gpu_name": rocm_core::pinned_gpu_name(&[]),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — AIM never records a GPU ordinal in state, unlike lemonade/vllm, producing a cross-engine device: inconsistency on equivalent hardware.

write_running_state's JSON literal has no gpu_indices key, and reported_device always passes None as the ordinal to format_reported_device. But engines/aim/src/process.rs's plan_gpus (lines 285-297) resolves a real single GPU ordinal, and docker.rs uses it to set HIP_VISIBLE_DEVICES inside the container (so AIM does pin a specific, known device). Compare engines/lemonade/src/state.rs:91,101 and engines/vllm/src/state.rs:181,189, which both record gpu_indices and thread it into pinned_gpu_name. On a multi-GPU host, AIM's device: row is the bare "rocm" (no name, no ordinal) while lemonade/vllm report "rocm gpu <n> (<name>)" for identical hardware — a real inconsistency on exactly the axis (device: agreement across engines) this PR is about.

The justifying doc comment ("the container gets the device list... so claiming one would be inventing it") is inaccurate: the container's visible GPU set is narrowed via HIP_VISIBLE_DEVICES to the resolved ordinal, the same mechanism lemonade/vllm use.

Confidence 75. Suggested fix: thread gpus.indices through to write_running_state and report it the same way lemonade/vllm do, or correct the doc comment if this is an accepted scope limit.

detected_gfx_target: Option<&str>,
compatible_therock_family: Option<&str>,
) -> GpuSupportLevel {
match (detected_gfx_target, compatible_therock_family) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (test-quality) — untested match arm: this arm ignores detected_gfx_target entirely, and no test exercises gpu_support_level(None, Some(_)).

(_, Some(_)) => GpuSupportLevel::RocmHip discards detected_gfx_target via the wildcard. Verified directly: every test call site in this file (gpu_support_level(Some("gfx90a"), Some("gfx90a")), gpu_support_level(None, None), gpu_support_level(Some("gfx_totally_unrecognized"), None), gpu_support_level(Some("gfx90c"), None)) passes Some/Some, None/None, or Some/None — none passes None/Some(_). Mutating this arm's pattern to (Some(_), Some(_)) would not be caught by any test in the suite.

Real-world impact is currently low since every caller derives compatible_therock_family from an already-Some detected target, but the function's public signature permits the combination.

Confidence 80. Suggested fix: add a test for gpu_support_level(None, Some("gfx90a")), or narrow the inputs so the illegal state is unrepresentable.

// validation on a host now reporting one GPU) also declines to name.
if gpu_indices.iter().any(|&index| index != 0) {
return None;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (test-quality) — this ordinal-mismatch guard can be deleted without failing any existing test.

The doc comment above says this exists specifically so a pinned ordinal other than 0 (e.g. --gpu 1 that got past validation on a host now reporting one GPU) also declines to name. But no test supplies gpu_indices=[1] together with present_amd_gpus=Some(1): the existing a_second_gpu_on_the_host_withholds_the_name test uses present_amd_gpus=Some(2), so the trailing present_amd_gpus != Some(1) check two lines below already returns None regardless of this branch, independent of it.

Confidence 85. Suggested fix: add a test calling gpu_name_for_single_gpu_host(&[1], Some(1), false, ...) and asserting None.

let gpu_support_level = rocm_core::gpu_support_level(
examine.detected_gfx_target.as_deref(),
examine.compatible_therock_family.as_deref(),
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (test coverage) — this gpu_support_level wiring is new behavior this PR introduces, and nothing tests it.

build_codex_bridge_snapshot computes gpu_support_level from ExamineSummary::gather()'s two fields, independently of main.rs's host_gpu_support_level (which uses detect_host_gpu_summary(paths)). Verified: this file has zero #[test] functions. The sibling renderer builtin_codex_bridge_engine_inventory (used downstream) is well-tested via chat/mcp_bridge.rs's internal_mcp_engines_tool_result, but that test injects a pre-computed GpuSupportLevel directly, bypassing this exact call site — so a positional transposition of the two Option<&str> arguments at lines 40-43 (a bug class gpu_support_level's own doc comment explicitly warns about) would compile and pass every existing test.

Confidence 72. Suggested fix: add a test that plants an ExamineSummary (or exercises build_codex_bridge_snapshot end-to-end) and asserts the resulting snapshot's engine notes match the expected GpuSupportLevel.

Comment thread docs/architecture.md

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (documentation completeness) — the gfx_target.rs entry (line 313) is stale: it omits the entire new GpuSupportLevel classifier this PR adds to that file.

The entry reads "Pure gfx-target lookups: extracting a gfx token, mapping a target to its TheRock family, the AMD PCI-id and marketing-name tables, and kernel gfx-version decoding" — no mention of GpuSupportLevel, gpu_support_level(), gpu_support_level_for_host_gpu(), or VULKAN_ONLY_SUPPORT_CLAIM, all of which this PR adds to this exact file (confirmed via git diff: gfx_target.rs gains ~155 lines introducing exactly this classifier, the central piece of this whole PR). This line of docs/architecture.md itself has zero diff in this PR.

AGENTS.md is explicit that cargo xtask check-architecture-doc only verifies an entry's presence, not that its description stays true ("The check reads no prose, so keeping an entry's description true is still on you") — which is why this didn't fail CI, but it is still a violation of the project's own stated authoring requirement, and it's introduced by this PR's own changes to the file in question.

Confidence 80. Suggested fix: update the entry to mention the GpuSupportLevel classifier.

Comment thread engines/aim/src/state.rs

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — merge_state_field (lines 185-191, pre-existing/unchanged by this PR) and write_state (lines 220-227) bypass the file lock and the Anthropic-endpoint carry-forward fix this PR adds elsewhere, leaving AIM exposed to the race the fix targets. (Posted at file level because lines 185-227 are unchanged context outside this PR's diff hunks, so an inline comment there is rejected.)

merge_state_field/write_state do a plain fs::read_to_string/fs::write with no FileLock and no call to carry_published_anthropic_endpoint. Compare crates/rocm-core/src/anthropic_endpoint.rs's fix, wired through crates/rocm-core/src/endpoint_readiness.rs's write_json_state_file_with — a path AIM's own writers never go through. record_served_model_id (line 173, which calls merge_state_field) is invoked right as the container is confirmed ready, the same event that gates the Anthropic-proxy thread's publish in apps/rocmd/src/anthropic_proxy.rs. Two threads of the same rocm __engine-serve-http aim ... process can race: AIM's own thread reads state before the proxy's publish lands, then overwrites the file with its stale copy, silently dropping the just-published endpoint — the same failure mode this PR's fix targets for lemonade, via a different writer.

Confidence 75 (traced directly via the absence of locking/carry-forward in this file and the call-site timing; the actual interleaving was not reproduced by running concurrent threads).

Suggested fix: route AIM's merge_state_field/write_state through rocm_core::merge_json_state_file/write_json_state_file, or document why AIM is intentionally exempt.

Main landed ROCMAI-703 (#1050), which rewrites the same deployment-summary
prose and adds its own serve scenario, so all three conflicts are with it.

tests/e2e-cucumber/features/model_serving.feature: main took `serve-60` for
its smoke-test-failure scenario. Main's keeps 60; this branch's APU scenario
becomes `serve-61` and moves after it, so each number still equals its
position. `apps/rocm/src/therock/family.rs`'s prose reference follows. The two
`-n "serve-60"` repro filters in docs are main's own and are left alone.

docs/testing.md: two independent sections appended at the same place, kept
whole. They are unnumbered `##` headings, so nothing renumbers.

README.md: both sides rewrite the one paragraph describing the deployment
summary, so this is a real merge rather than a choice between them. The
summary sentence keeps this branch's `device:` row and main's endpoint/model/
smoke-test list; main's smoke-test-failure paragraph and this branch's
"device line is omitted when the engine cannot say yet" paragraph both
survive; and the shared closing sentence is kept once. Taking either side
whole would have silently dropped the other's user-facing contract.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Correction to my earlier note: the colliding number has moved.

main has since landed ROCMAI-703 (#1050), which took serve-60 for its smoke-test-failure scenario. Both branches renumbered past it independently, so the collision is now at serve-61, not serve-60:

As before, each is correct within its own branch (a scenario's number equals its position in its own file); the clash only exists across the two, so whichever merges first forces the other to renumber. feature_naming fails loudly on a duplicate, so it cannot slip through silently.

Worth noting this is the third time these numbers have moved — #854's pair has gone 57/58 → 59/60 → 60/61 → 61/62 purely from main inserting scenarios ahead of them. The @id: tags are the binding key and have never changed, so no step binding has moved with them, and nothing about the tests' meaning has changed.

I will renumber whichever branch lands second.

r0x0r added a commit that referenced this pull request Nov 17, 2026
Tracks #678 past its merge of ROCMAI-703 (#1050), which renumbered that
branch's APU scenario to serve-61 and reworked the README's
deployment-summary paragraph. Merged clean -- no conflicts here.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main landed the WSL librocdxg work (#1091), which appends to the same two
examine files this branch does.

tests/e2e-cucumber/features/examine.feature: main took `examine-49` through
`examine-53`. This branch's support-level scenario becomes `examine-54` and
moves after them, so each number still equals its position. Three references
follow it: `host_gpu/render.rs`'s doc comment, and the prose and repro filter
in `docs/testing.md`. The `examine-49`..`examine-53` references further down
that file are main's own librocdxg paragraphs and are left alone.

tests/e2e-cucumber/tests/e2e/examine_steps.rs: two independent step blocks
appended at the same place, both kept. The split left each side's final
function unterminated while sharing a single closing brace, so the brace is
reproduced for each rather than closing only the block that came last.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
#678 merged #1091, which took `examine-49` through `examine-53` and moved
#678's own support-level scenario to `examine-54`. This branch's four Vulkan
scenarios collided with that block, so they become `examine-55` through
`examine-58`. They already sat last in the file, so only the numbers moved.

Nine references follow them: three comments in `examine_steps.rs`, the
`host_gpu/vulkan.rs` row in `docs/architecture.md`, and the four support-level
table rows plus the repro filter in `docs/testing.md`. Two neighbouring
`examine-51` references are main's librocdxg ones -- the `render.rs` test
comment and the librocdxg paragraphs in `docs/testing.md` -- and are left
alone.

crates/rocm-core/src/host_gpu/render.rs: both sides added a field to the same
`format!` template. Resolved as the union -- this branch's `vulkan: {}` and
main's extra placeholder before `managed_runtimes` -- because the argument
list git had already merged carries both `self.vulkan_line()` and
`self.runtime_librocdxg_line()`. Taking either template alone would have left
the placeholders and the arguments out of step.

docs/testing.md: this branch raises the support levels from three to five,
which is its own change, so its wording wins over the base's.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main landed runtime provenance (#1090), which appends its own scenario to
examine.feature and took `examine-54`. This branch's support-level scenario
becomes `examine-55` and moves after it, so each number still equals its
position.

Two references follow it: `host_gpu/render.rs`'s doc comment and the prose
and repro filter in `docs/testing.md`. The other `examine-54` references in
that file belong to main's provenance scenario
(`@id:examine-reports-where-the-active-runtime-came-from`) and are left alone.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
Main's #1090 took `examine-54`, so #678's support-level scenario moved to
`examine-55` and this branch's four Vulkan scenarios shift again, to
`examine-56` through `examine-59`.

Nine references follow them: three comments in `examine_steps.rs`, the
`host_gpu/vulkan.rs` row in `docs/architecture.md`, and the four
support-level table rows plus their repro filter in `docs/testing.md`. The
`examine-55` repro filter immediately above that table is #678's and stays,
as do the `examine-54` references to main's provenance scenario.

docs/testing.md: this branch raises the support levels from three to five,
which is its own change, so its wording wins over the base's.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (base 9e50eac5a03df41a2f23e73dac3a7ec470ea78a7 → head 30f91a09c6f27419e0511914fd8c6208a587b1bd, 40 files, +5605/-1458). Ticket ROCMAI-550 is named throughout the description/title; not retrieved (internal tracker identifier, kept out of a public review to avoid pulling internal-system content into it). The description is otherwise unusually detailed and largely accurate against the diff — its own "Deferred, explicitly" section discloses several gaps this review independently confirmed (e.g. the VulkanOnly tier having no hardware-grounded e2e scenario) — but it omits an entire subsystem of the diff; see Blocking below.

Blocking

  • engine_ipc.rs's new doc comment describes "unknown"-until-ready device gating as uniform across engines; Lemonade's gate differs from vLLM/AIM's — apps/rocm/src/engine_ipc.rs:366-370 (device_worth_reporting doc), against engines/lemonade/src/lib.rs's healthcheck_service_at (gates on full inference-verified ready) vs. engines/vllm/src/state.rs / engines/aim/src/state.rs (gate merely on state.is_some()). The comment cites both vLLM and Lemonade as sharing "no state on disk" semantics; they don't.
    Confidence 85 · logic/doc-accuracy · Fix: reword to describe the actual per-engine difference, or make Lemonade's gate consistent with the other two.

  • The e2e "both surfaces agree" device check can never fire under any current CI lane's harness — tests/e2e-cucumber/tests/e2e/serving_steps.rs:3807 (assert_services_list_reports_device's device_row_value(summary) check). world.serve_summary is captured via run_rocm_bounded's file-redirected stdout, so serve_cmd.rs's summary_mode gate is always false and the CLI takes the device-row-free print_managed_launch_plain path; even were that not so, serve_summary.rs's render_summary writes the row with no colon (" {label:<label_width$} {value}") while device_row_value looks for "device:". Either way the comparison is structurally dead for serve-05 and serve-07, the only two scenarios that use it. AGENTS.md's own standard: "a check whose name or comment claims a branch it cannot fail on is a defect, not missing coverage... rank it worse than no test at all."
    Confidence 85 · logic/test-coverage · Fix: drive rocm serve through a real PTY here (as serve-28/41/60 already do) and align the two renderers' row formats, or state plainly that the cross-check is currently unreachable.

  • vLLM and AIM remain exposed to the exact Anthropic-endpoint publish race this PR hardens only for Lemonade, and the new module doc overclaims general coverage — engines/vllm/src/state.rs:189,235 and engines/aim/src/state.rs:164,220 (both add the new rocm_core::pinned_gpu_name probe, but both write_state remain a raw, unlocked fs::write), vs. engines/lemonade/src/state.rs:160-161 (the only write_state routed through rocm_core::write_json_state_file, the sole caller of carry_published_anthropic_endpoint, crates/rocm-core/src/endpoint_readiness.rs:156); crates/rocm-core/src/anthropic_endpoint.rs:16-20's doc states generally "the engine adapters write it whole... keep the whole-file write from dropping what the sibling thread published" — true only for Lemonade. The PR author's own review-thread comment traces this exact mechanism for Lemonade and notes "Not fixed here: engines/vllm and engines/aim... that gap predates this PR... it is a separate change" — accurate that the unlocked write predates this PR, but by the same comment's own causal chain, the slow probe that makes the race practically reachable is newly added to vLLM/AIM by this very diff, which the "predates this PR" framing undersells. The current PR description doesn't mention this scope limitation at all.
    Confidence 85 · architectural · Fix: route vLLM's/AIM's write_state through rocm_core::write_json_state_file, or state the scope limitation explicitly and track the remainder.

  • docs/manual-testing.md: this diff drops --foreground from the example rocm serve command, but the prose ~34 lines later still calls it "that foreground server" with no stop instruction — command at line 408 (diff: ...--device gpu_required --foreground --port 11435 → ...--device gpu_required --port 11435), contradicted by "With that foreground server stopped, check that..." at line 442 (HEAD), which — unlike the parallel flow later in the same file — gives no rocm services stop <id> --yes-style instruction.
    Confidence 90 · mechanical · Fix: restore --foreground if intended, or reword the follow-on prose for a backgrounded service and add the stop step.

  • No test in any of the three engines drives the device field through the real healthcheck_service/healthcheck_service_at RPC entry point against a populated/ready state — only the private reported_device formatter is unit-tested directly: engines/vllm/src/state.rs:753, engines/aim/src/state.rs:416, engines/lemonade/src/device_policy.rs:259,270. Grepping each engine's lib.rs for .device/response.device returns zero hits. These tests would all still pass if the RPC handler's wiring to the formatter were reverted.
    Confidence 70 · test-coverage · Fix: add, for at least one engine, a test driving the real RPC handler end-to-end and asserting .device.

  • The PR description does not mention the Anthropic-endpoint publish-race subsystem of the diff at all — apps/rocmd/src/anthropic_proxy.rs, crates/rocm-core/src/anthropic_endpoint.rs, crates/rocm-core/src/endpoint_readiness.rs, plus the pinned_gpu_name calls newly added to every engine's state-writing path, are a substantial, independently-discoverable piece of this diff with no mention in the PR body (whose "Also in this diff"/"Known costs"/"Deferred, explicitly" sections are otherwise thorough). This is very plausibly connected to the previous finding going unnoticed through this PR's many review rounds — a reviewer working from the description would not know to check this subsystem's completeness.
    Confidence 80 · description-scope (Step 9) · Fix: cover this subsystem in the PR body per AGENTS.md §5 ("write the PR body from the final diff").

Non-blocking

  • publish_anthropic_endpoint records its note unconditionally before the merge it precedes is known to succeed, with no rollback on non-retryable failure — crates/rocm-core/src/anthropic_endpoint.rs:110-131. A narrow edge case (needs a non-retryable merge failure on first publish) that could carry a dead endpoint into a later rewrite.
    Confidence 70 · logic.
  • The new carry-forward mechanism's tests are all single-threaded/sequential — crates/rocm-core/src/anthropic_endpoint.rs's three new tests never drive the publisher and the rewriter on different threads with the lock actually contended, despite the repo having exactly that infrastructure elsewhere (endpoint_readiness.rs's concurrent_merges_across_threads_and_processes_keep_every_key).
    Confidence 75 · test-coverage.
  • lemonade_engine_note's RocmHip arm hardcodes a string literal byte-identical to, but untied from, engine_inventory()'s own lemonade catalog entry (apps/rocm/src/main.rs:1083-1098 vs. ~1189-1191), which it always overrides. A future edit to one would silently stop matching the other.
    Confidence 70 · architectural.

Decisions for the author

  • tradeoff — bundling the Anthropic-endpoint race hardening into this PR's scope (defensible: the new host-GPU probe is what creates/widens the exposure) vs. AGENTS.md §11's one-logical-change guidance. Given the hardening is incomplete (see Blocking), a split might have gotten the gap more scrutiny.
  • non-blocking-improvement — gpu_support_level's two positional Option<&str> arguments could be tightened into a type that prevents transposition, closing the gap the function's own doc comment already flags (no current call site has them backwards).

Positive signals

  • GpuSupportLevel's 3 variants are exercised as a genuine state matrix in host_gpu/render.rs's tests (RocmHip with matching/mismatched/no install, VulkanOnly, Unknown-detected-unrecognized, Unknown-not-detected), each with its own assertion.
  • pinned_gpu_name's decision logic is a pure, fully unit-tested helper, with a test asserting the expensive probe closure is never invoked when the decision short-circuits — exactly the kind of test that would catch a broken guard.
  • service_log_follow.rs's "pure move, not a rewrite" claim was mechanically verified byte-identical against the pre-change services_text.rs content, not just trusted.
  • The device-row feature has an explicit state matrix in both serve_summary.rs and services_text.rs, including a probe-not-even-called assertion for the reuse path, not just a value-absent one.
  • VULKAN_ONLY_SUPPORT_CLAIM consolidates three independently-worded copies of the same user claim into one pinned, tested constant.
  • docs/testing.md/docs/manual-testing.md's other new technical claims (probe-budget numbers, format_reported_device's exact shape, scenario IDs) were all independently cross-checked against code and found accurate — an unusually well-verified documentation PR apart from the one contradiction above.
  • advised_commands.rs's mechanical scan automatically covers every new remediation command string with no special-casing required.
  • The PR's own "Deferred, explicitly" section's disclosure of the VulkanOnly e2e-coverage gap was independently confirmed accurate by this review.

Deployment notes

  • rocm services list's documented ~18s worst-case block (a process-wide mutex serializing concurrent CLI invocations) is extended, not introduced, by this PR, and is already disclosed by the author as an accepted, reversible cost.
  • Windows hosts running the vLLM or AIM engine are where the unprotected race (see Blocking) is reachable in practice — Linux's GPU probe is cheap enough that the window is negligible, per the author's own traced root-cause analysis for the Lemonade case.
  • No schema migration: new engine-protocol/state fields are additive and optional.

What this covered

Full diff read via 9 coupled file-group workers fanned out in parallel (grouped by shared contract/coupling per this method's own sizing guidance, given the diff's size): core GPU detection/classification (crates/rocm-core/src/{gfx_target,host_gpu,host_gpu/*}.rs); family catalog + examine report + advised-commands + README/architecture docs; engine IPC/protocol + all three engines' state + MCP/bridge rendering; serve/services rendering + the new service_log_follow.rs split; CLI/main/providers wiring; the Anthropic-endpoint race-fix subsystem; docs/manual-testing.md/docs/testing.md; and the e2e feature/step files (examine+runtime, then model-serving), each split further where one worker's group was still too large. Every file in every group was read in full at the PR head via git show/git diff against the fetched refs (no checkout), plus surrounding caller/callee code as needed; two workers explicitly noted sampling rather than exhaustively line-reading very large, mostly-untouched pre-existing regions of docs/testing.md (~5600 lines) and examine_steps.rs/runtime_steps.rs (~3600/~1900 lines) beyond the new content and its directly-adjacent helpers — disclosed per this method's rule rather than silently partial. CI was read directly from the GitHub API: all required checks (clippy, module-size, architecture-doc, full e2e suite, security scans) pass at the reviewed head.

Did not run: no Rust test binary or the e2e-cucumber suite was executed by this review (all "would this still pass reverted" verdicts are from static tracing, disclosed per finding); a dedicated "history"/git blame pass and a separate "prior PR review comments" pass were not run as independent passes — each file-group worker instead incidentally used git log/git diff against the merge-base where it bore on a specific claim (e.g. the service-log-follow "pure move" check), which is narrower than the method's normal separate-pass treatment, disclosed here as a cost-driven deviation.

Findings were reached blind, from the code only; the PR description and ticket were read afterward (Step 9) and this PR's extensive prior review history (dozens of rounds, none carrying this method's marker) was read only after that, per the task's own instructions, to check whether these specific findings had already surfaced — none of the six Blocking findings above appear in the prior discussion under any wording this review could find, though the vLLM/AIM race gap (third Blocking item) was separately, partially disclosed by the author in a review reply (not in the PR description itself) while fixing the Lemonade-specific case; that reply is quoted and reasoned about directly in that finding rather than treated as settling it, consistent with this review not being reconciled against prior discussion.


🤖 Automated by an AI agent

device_worth_reporting(response.device)
}

/// The engine's reported device, or `None` when it said nothing usable.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this doc comment describes "unknown"-until-ready gating as uniform across engines; it isn't.

The comment says "unknown" is "the engines' own device value when no state is on disk for this service id yet (see engines/vllm/src/state.rs and engines/lemonade/src/lib.rs)". That's accurate for vLLM/AIM (Some(state) => reported_device(state), None => "unknown" — gated only on state.is_some()), but not for Lemonade: healthcheck_service_at in engines/lemonade/src/lib.rs gates on ready (let device = if ready { ... } else { "unknown".to_owned() }), which stays "unknown" even once a state file clearly exists, until the server's own inference probe has verified it can serve. So the three engines disagree on when device leaves "unknown", and this new comment asserts a uniformity that doesn't hold — the kind of claim AGENTS.md §3 says must be checked against the code it describes.

Confidence 85 (traced directly against engines/lemonade/src/lib.rs's healthcheck_service_at, engines/vllm/src/state.rs, and engines/aim/src/state.rs). Suggested fix: reword to name the actual per-engine gating difference, or make Lemonade's gate consistent with the other two if the divergence is unintentional.

let Some(summary) = world.serve_summary.as_deref() else {
return;
};
if let Some(summary_device) = device_row_value(summary) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this cross-surface "both surfaces agree" check can never actually fire under any current CI lane.

device_row_value(summary) looks for a line matching strip_prefix("device:") (colon-terminated). But world.serve_summary is the stdout of a rocm serve invocation captured via run_rocm_bounded (tests/e2e-cucumber/tests/e2e.rs), which redirects stdout to a plain file — never a terminal. apps/rocm/src/serve_cmd.rs's summary_mode = background && IsTerminal::is_terminal(&stdout()) is therefore always false here, so the CLI takes the print_managed_launch_plain branch (apps/rocm/src/service_launch/start.rs), whose output has no device line at all. And even if summary_mode were somehow true, serve_summary.rs's render_summary writes the device row as " {label:<label_width$} {value}" — a padded label with no colon (confirmed by its own test asserting starts_with("device"), not "device:") — so strip_prefix("device:") still wouldn't match. Either way, device_row_value(summary) is always None, so the assert_eq! on the two lines above this one never executes, for both scenarios that use this step (serve-05, serve-07).

This is the class AGENTS.md's own test-quality standard calls out explicitly: "a check whose name or comment claims a branch it cannot fail on is a defect, not missing coverage... rank it worse than no test at all." A real regression in cross-surface device agreement would pass silently.

Confidence 85 (fully traced: the summary_mode gate, print_managed_launch_plain's body, render_summary's no-colon format and its own unit test proving that format, and run_rocm_bounded's file-redirected stdout — not executed, per this review's constraints). Suggested fix: either drive rocm serve through a real PTY here (as serve-28/41/60 already do elsewhere) so summary_mode activates, and align device_row_value's parsing with whatever format render_summary actually emits — or state plainly in this step's own comment that the cross-check is currently unreachable.

Comment thread engines/vllm/src/state.rs
// path. `rocm_core::pinned_gpu_name` answers `null` when the host
// probe cannot be attributed to the pinned ordinal, so the row
// falls back to `rocm gpu <n>` rather than naming another card.
"gpu_name": rocm_core::pinned_gpu_name(&request.gpu_indices),

ghost Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — vLLM (and AIM, same pattern) remain exposed to the exact Anthropic-endpoint publish race this PR hardens only for Lemonade.

This PR adds the rocm_core::pinned_gpu_name(...) probe (new here — confirmed via diff against merge-base) to vLLM's and AIM's write_running_state too, but their write_state (engines/vllm/src/state.rs ~line 235, engines/aim/src/state.rs ~line 220) still does a raw, unlocked fs::write, never going through rocm_core::write_json_state_file — the only function carry_published_anthropic_endpoint is wired into (crates/rocm-core/src/endpoint_readiness.rs:156). crates/rocm-core/src/anthropic_endpoint.rs's new module doc says generally "Both write the one engine state file, and the engine adapters write it whole... keep the whole-file write from dropping what the sibling thread published" — true only for Lemonade.

The author's own review-thread comment on this PR traces the exact mechanism for Lemonade (a multi-second Windows GPU probe inside a post-spawn whole-file rewrite racing the Anthropic proxy's publish) and explicitly notes "Not fixed here: engines/vllm and engines/aim write their state files with their own unlocked write_state... that gap predates this PR... it is a separate change." That's only half right: the unlocked write predates this PR, but the slow probe that makes the race practically reachable is new here, added to vLLM/AIM by this same diff — by the author's own causal chain for Lemonade, this PR is what newly widens the window for these two engines as well. The current PR description doesn't mention this scope limitation at all.

Confidence 85 (the pinned_gpu_name/raw-fs::write pattern is confirmed by direct read of both files; the causal argument is the PR's own, applied consistently). Suggested fix: either route vLLM's/AIM's write_state through rocm_core::write_json_state_file, or state the scope limitation in the PR description and track the remainder as a tightly-scoped, explicit follow-up.

Comment thread engines/aim/src/state.rs Outdated
// container is handed a device list, not an ordinal — and
// `rocm_core::pinned_gpu_name` still declines on any host where
// more than one AMD GPU could be the one that answered.
"gpu_name": rocm_core::pinned_gpu_name(&[]),

ghost Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same finding as the one on engines/vllm/src/state.rs:189 (this comment is just the second affected site, so it doesn't scroll out of view): this PR adds the pinned_gpu_name probe here too, but AIM's write_state (~line 220) is also a raw, unlocked fs::write that never goes through rocm_core::write_json_state_file, so it gets neither the file lock nor the new carry-forward protection. See the other comment for the full trace and the author's own partial acknowledgment of this gap.

Comment thread docs/manual-testing.md

```powershell
rocm serve qwen --engine lemonade --device gpu_required --foreground --port 11435
rocm serve qwen --engine lemonade --device gpu_required --port 11435

ghost Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this diff drops --foreground from the example command, but the prose ~34 lines below (line 442 at HEAD) still calls it "that foreground server" and gives no stop instruction.

Diff against merge-base: rocm serve qwen --engine lemonade --device gpu_required --foreground --port 11435 → rocm serve qwen --engine lemonade --device gpu_required --port 11435. Per this same file's own stated convention ("rocm serve returns to the shell once the server is ready. Background is the default... Only --foreground or --verbose attach to the log stream"), the server launched by the edited command now runs in the background — yet "With that foreground server stopped, check that a catalogued recipe reaches Lemonade..." (line 442) still describes it as foreground and, unlike the parallel flow later in the same section (which explicitly says rocm services stop <service-id> --yes), gives no instruction for how to actually stop it before continuing.

Confidence 90 (direct diff + content read). Suggested fix: either restore --foreground if that's genuinely intended, or reword the follow-on prose to describe a backgrounded service and add the stop command.

Comment thread engines/vllm/src/state.rs
/// is the format README.md and `docs/manual-testing.md` never described and
/// which made one row disagree with itself across engines.
#[test]
fn reported_device_matches_the_shape_lemonade_reports() {

ghost Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (test-coverage) — no test in any of the three engines drives the device field through the real healthcheck_service/healthcheck_service_at RPC entry point against a populated/ready state; only the private reported_device formatter is tested directly.

This test (reported_device_matches_the_shape_lemonade_reports) calls reported_device(...) directly, as does AIM's reported_device_for_existing_state_matches_the_shared_engine_shape (engines/aim/src/state.rs:416) and Lemonade's reported_device_reflects_backend_pinned_ordinal_and_gpu_name/reported_device_names_the_gpu_when_no_ordinal_was_pinned (engines/lemonade/src/device_policy.rs:259,270). Grepping each engine's lib.rs (where the real healthcheck_service/healthcheck_service_at RPC handlers live) for .device/response.device returns zero hits in all three. So: would these tests still pass if the RPC handler's Some(state) => reported_device(state) wiring were reverted to its old hardcoded form? Yes — nothing here would notice, since none of them call the RPC handler itself.

Confidence 70 (confirmed by reading all three engines' full test suites; the current wiring is in fact correct, so this is a coverage gap rather than a live defect). Suggested fix: add, for at least one engine, a test that calls the real RPC handler with a populated/ready state and asserts the returned .device.

/// Called more than once for one endpoint when the merge has to be retried
/// (`rocmd`'s proxy retries a state file caught mid-write); the note is the
/// same each time.
pub fn publish_anthropic_endpoint(

ghost Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking — the in-memory "published" note is recorded unconditionally before the merge it precedes is known to succeed, and is never rolled back on a non-retryable failure.

published.insert(...) runs before crate::merge_json_state_file(...) is called, and the function just returns that call's Result with no cleanup on Err. If the merge fails non-retryably (e.g. JsonStateRefusal::NotAnObject, or a FileLock::acquire error — apps/rocmd/src/anthropic_proxy.rs's publish_with_pause doesn't retry these), the stale note persists in PUBLISHED_ENDPOINTS for the rest of the process, and a later full-state rewrite for the same path would still carry a dead endpoint into the document via carry_published_anthropic_endpoint.

Confidence 70 (straight-line control-flow trace, single-threaded). This is a narrow edge case (requires a non-retryable merge failure on first publish) — flagging as non-blocking rather than blocking. Suggested fix: only insert the note after the merge succeeds, or clear it on non-retryable failure.

Comment thread apps/rocm/src/main.rs
/// now answers for every level, and this says what that answer is: the support
/// level, not the backend. The backend is settled by launching one, which that
/// line in turn points at.
const fn lemonade_engine_note(level: rocm_core::GpuSupportLevel) -> &'static str {

ghost Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking — RocmHip's arm hardcodes a string literal byte-identical to, but untied from, engine_inventory()'s own lemonade catalog entry ("default embedded Lemonade server with ROCm llama.cpp backend", apps/rocm/src/main.rs ~line 1189), which engine_inventory_note always overrides for every GpuSupportLevel including this one.

Since catalog_note is already in scope at the one call site (engine_inventory_note), this arm could take it as a parameter and return it verbatim instead of re-stating it — the catalog entry is now dead data on every real call path (bridge_snapshot.rs, examine_report_text.rs, chat/mcp_bridge.rs), and nothing ties the two literals together, so a future edit to one (e.g. a typo fix in the catalog) would silently stop matching the other. The only test covering this (examine_report_text.rs's *_keeps_rocm_note_for_a_rocm_capable_gpu) asserts against its own separately hand-written literal, not against engine_inventory() itself.

Confidence 70 (confirmed both literals are currently byte-identical and unconnected). Suggested fix: have this arm take catalog_note and return it for RocmHip instead of re-stating the string.

Roman Sirokov added 4 commits November 17, 2026 17:03
…tate under the lock

AIM's `reported_device` hardcoded `rocm` and probed the host for a GPU name
whatever the launch's `device_policy` was, so a `cpu_only` container — which
`resolve_use_gpu`/`docker::RunSpec` correctly give no /dev/kfd, no /dev/dri and
no HIP_VISIBLE_DEVICES — still rendered e.g. `rocm (AMD Instinct MI300X)`. That
is the invented device claim this branch exists to remove, shipped in compiled
code even though `parse_device_policy` currently refuses `cpu` at the CLI. The
policy the launch recorded now decides, and the probe is not made at all when
no GPU was attached.

AIM also recorded no `gpu_indices`, so on a multi-GPU host it answered a bare
`rocm` where lemonade and vLLM answer `rocm gpu <n> (<name>)` for the same
hardware — the engine does pin an ordinal, `plan_gpus` resolves it and
`HIP_VISIBLE_DEVICES` narrows the container to it. `run_container` now reports
the ordinals it attached so the state file and the container cannot disagree.

Both engines' `write_state` were a lock-free `fs::write` that bypassed
`carry_published_anthropic_endpoint`. The unlocked write predates this branch,
but the slow host GPU probe that makes the race practically reachable is added
to these two state writes by this branch itself, so the exposure is this
branch's to close: routing them through `rocm_core::write_json_state_file`
orders them against every merge and carries a sibling thread's publication
through the rewrite, as lemonade's already does. The status writers patch
through `merge_json_state_file` instead of rewriting the document whole.

`healthcheck_service_at` is split out of both handlers, the way lemonade's
already was, so a test can drive the real RPC entry point: every device test in
the three engines called the private formatter, so reverting a handler's wiring
to a hardcoded literal kept the suite green.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ld not

`gpu_support_level`'s `(_, Some(_)) => RocmHip` arm ignores the detected
target on purpose — `resolve_family` asks it about a `--family` the user
named, where there may be no detected target — but every call in the suite
passed Some/Some, None/None or Some/None, so narrowing the arm to
`(Some(_), Some(_))` was invisible.

`gpu_name_for_single_gpu_host`'s ordinal guard was in the same position: the
only test with a non-zero ordinal also supplied a host count of 2, which the
check two lines below already declines on, so the guard could be deleted
outright with the suite still green. Both mutations now turn it red.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…et it read both renderers

`assert_services_list_reports_device`'s comparison against `rocm serve`'s
summary was dead twice over. `world.serve_summary` comes from
`run_rocm_bounded`, which redirects stdout to a file, so `serve_cmd.rs`'s
`summary_mode` is false and the CLI prints `print_managed_launch_plain`, which
has no device row in any state; and `device_row_value` looked for `"device:"`
while `render_summary` writes a padded label with no colon, so it could not
have matched a summary even if one carried the row.

The parser now reads both renderers' shapes (and still refuses the
`device_policy:` row the serve plan prints), and the step says in full that on
today's lanes the comparison does not run and the backend assertion above it is
what discriminates — rather than letting the step's name imply coverage it has
not got. Making it live means driving the vLLM and Lemonade per-PR serve
canaries through a `TuiSession`, which is a change to those canaries' own
harness.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…onade serve flow

`device_worth_reporting` claimed the `"unknown"`-until-ready gate was uniform
and cited vLLM and Lemonade as sharing it. They do not: vLLM and AIM answer as
soon as a state file exists, while Lemonade keeps saying `"unknown"` until its
own inference probe returns. That difference is the whole reason
`SERVE_DEVICE_PROBE_BUDGET` exists, so the comment now names it. The probe
bound alongside it also cites AIM's own `HEALTHCHECK_TIMEOUT_MS`, which it
relied on without naming.

`docs/architecture.md`'s `gfx_target.rs` entry still described pure lookups
after that file gained the whole `GpuSupportLevel` classifier; the gate checks
presence, not accuracy, so nothing caught it.

`docs/manual-testing.md` dropped `--foreground` from the Lemonade serve
example but still called the result "that foreground server" 34 lines later,
with no way to stop it. The prose now matches a backgrounded service and gives
the stop command, as the parallel flow below it already does.

Also drops review-thread references from source comments and a note to
reviewers from docs/testing.md; each says the reason instead.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

ghost commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Three CHANGES_REQUESTED reviews (5936137783, 5936357643, 5939041029) went unread until now — apologies. Head is now 8b811d6b9. Going through every blocking finding, fixed or not, with what proves it.

Fixed

1. AIM's reported_device hardcoded rocm and probed a GPU regardless of device_policy (5936357643). Fixed. write_running_state no longer calls pinned_gpu_name when the recorded policy is CpuOnly, and reported_device reads that policy back and answers cpu with no name. The doc comment's "AIM only ever runs on ROCm" is gone — you were right that CpuOnly is reachable in-crate, and a container docker::RunSpec gives no /dev/kfd, no /dev/dri and no HIP_VISIBLE_DEVICES is not running on ROCm whatever the CLI's parse_device_policy currently permits. Test: a_cpu_only_launch_claims_neither_rocm_nor_a_gpu_name. Mutation: removing the CpuOnly arm and forcing uses_gpu = true → red.

2. AIM recorded no GPU ordinal (5936357643). Fixed. run_container now returns a ContainerRun carrying the ordinals it actually attached (empty for CPU-only, and for a container it found already running — that launch pinned them and this call cannot know which), write_running_state records gpu_indices, and reported_device reads it through the shared first_gpu_index_from_state. AIM now answers rocm gpu <n> (<name>) like the other two. Mutation: dropping the gpu_indices key from the state write → red.

3. vLLM's and AIM's unlocked write_state, and the module doc overclaiming coverage (both 5936357643 and 5939041029). Fixed, by routing rather than by narrowing the doc.

On the "predates this PR" framing — you are right and I was wrong. The unlocked fs::write predates this branch; the multi-second host GPU probe that makes the race practically reachable is added to vLLM's and AIM's state-writing paths by this very diff. By my own causal argument for Lemonade, closing that window is this diff's job for all three engines, not a separate change. Both write_state now go through rocm_core::write_json_state_file, and the status writers through merge_json_state_file instead of read-patch-rewrite. anthropic_endpoint.rs's module doc and merge_json_state_file's doc no longer imply general coverage — they name write_json_state_file as the only carrier and say an out-of-tree plugin writing the file itself gets neither it nor the lock. Test, in each engine: a_running_state_write_keeps_the_anthropic_endpoint_this_process_published. Mutation: restoring the raw fs::write → red in both.

4. device_worth_reporting's doc claimed uniform "unknown"-until-ready gating (5939041029). Fixed. It now names the real split: vLLM and AIM answer as soon as a state file exists; Lemonade keeps answering "unknown" until its own inference probe returns. That difference is precisely why SERVE_DEVICE_PROBE_BUDGET exists, so the comment now says so rather than asserting a uniformity that would make the budget look pointless. I did not make Lemonade consistent with the other two — its gate is stricter on purpose (it reports a device only once it has verified it can serve), and loosening it is a behaviour change outside this branch. Also fixed the neighbouring probe-bound comment, which relied on AIM's HEALTHCHECK_TIMEOUT_MS without naming it.

5. docs/architecture.md's gfx_target.rs row was stale (5936357643). Fixed — it now names GpuSupportLevel, gpu_support_level, gpu_support_level_for_host_gpu and VULKAN_ONLY_SUPPORT_CLAIM. Correct that check-architecture-doc reads no prose, which is why CI stayed green.

6. docs/manual-testing.md's dropped --foreground (5939041029). Fixed, and this one was mine: I removed --foreground to fix an earlier finding and left the prose 34 lines down calling it "that foreground server" with no way to stop it. It now describes a backgrounded service and gives rocm services stop <service-id> --yes, matching the parallel flow further down the same file.

7. gpu_support_level's (_, Some(_)) arm was untested (5936357643). Fixed: the test now passes None/Some(_) (and Some("gfx90c")/Some("gfx90a"), where the family must outvote a target with no ROCm build). Mutation: narrowing the arm to (Some(_), Some(_)) → red. I kept the wildcard rather than narrowing the inputs, because resolve_family legitimately asks this about a --family the user named, with no detected target at all.

8. gpu_name_for_single_gpu_host's ordinal guard was deletable (5936357643). Fixed: a_pinned_ordinal_the_host_count_contradicts_withholds_the_name supplies gpu_indices=[1] with present_amd_gpus=Some(1) and asserts the name closure is never even called. Mutation: deleting the branch → red, and only that test goes red, which confirms your reading that the count check alone covered every other case.

9. No test drove the device through the real RPC entry point (5939041029). Fixed. healthcheck_service_at is split out of vLLM's and AIM's handlers (the way Lemonade's already was), and each engine has healthcheck_rpc_answers_with_the_device_the_state_file_describes, which drives the handler against a planted state and asserts .device — plus the no-state "unknown" case. Mutation: reverting the handler's Some(state) => reported_device(state) wiring to a hardcoded "rocm_gpu" → red in both, and it is the only test that goes red, which is exactly the gap you described.

Partly fixed, and said plainly

The e2e cross-surface device check is structurally dead (5939041029). Both halves of your trace are correct, and I confirmed them. device_row_value looked for "device:" while render_summary writes a padded label with no colon; that is fixed — the helper now reads both renderers' shapes (and still refuses the device_policy: row the serve plan prints). But the deeper half stands: world.serve_summary comes from run_rocm_bounded's file-redirected stdout, so summary_mode is false and the CLI prints print_managed_launch_plain, which carries no device row in any state. The comparison therefore still does not run on any lane.

I took your second option and said so in full, in the step's own comment and in the PR body: on today's lanes the comparison does not fire, the backend/shape assertion above it is what discriminates for serve-05 and serve-07, and making it live means driving the vLLM and Lemonade per-PR serve canaries through a TuiSession. I deliberately did not do that here: it re-plumbs the two canaries that gate every PR, including their readiness polling and failure-evidence capture, and I cannot run those GPU lanes from where this was verified. A canary broken by a harness change is a worse outcome than a cross-check that honestly declares itself dormant. Happy to take the opposite call if you'd rather.

Not fixed, with reasons

Commit-message rewrites (5936137783, item 1). Agreed on the substance — several messages describe work the final diff no longer contains, cite renumbered scenario ids, carry review chatter, and 16ccfb3a has a Co-Authored-By: Claude trailer. Not done, because PR #1036 (split/vulkan-probe) is stacked on this branch and folding/rewording means a force-push that breaks it. Raised with the author as a decision: either land #1036 first and rewrite then, or squash-merge this PR so the branch's individual messages never reach main. Not something to do quietly behind a stacked PR.

ROCMAI-550 scope (5936137783, item 4). Still deferred and now recorded as such in the body: docs/rocm-docs, skills/rocm-doctor and docs/ux-guidelines.md are untouched, and the static claim survives in the dash-TUI engine manager and rocmd's host snapshot. The ticket is already Done, so this needs a follow-up ticket rather than a note — flagged to the author, not widened here. I did fix the one in-scope item you flagged inline only partially: main.rs's RocmHip Lemonade note still asserts the ROCm llama.cpp backend; deferring it to the device: row is a wording change with its own pinned tests and I left it with the rest of that sweep.

PUBLISHED_ENDPOINTS is process-wide and never evicted (5936137783). Your suggestion is the better design: resolving the GPU name before the spawn would keep the post-spawn rewrite fast and leave write_json_state_file a plain replace, removing the need for the carry entirely. I did not do it blind — the name is resolved from the ordinals the launch pins, which for the self-managed engines are settled at different points in each adapter's launch path, so it is a restructuring of three launch paths rather than a move. Recorded in the body as the known cost and the known better shape.

publish_anthropic_endpoint notes before the merge succeeds (5939041029, non-blocking). Not changed. Noting first is load-bearing — it is what leaves no window for a rewrite that takes the file lock first — so the fix is to clear the note on a non-retryable failure, not to reorder. Worth doing; it needs the retryable/non-retryable split made explicit and I did not want to add it in a round already this large.

Confirmed by grep, as asked

Nothing compares against AIM/vLLM's old literal rocm_gpu healthcheck device string. The surviving occurrences are a different protocol field — DetectResponse.devices[].kind and DetectCapabilities.rocm_gpu — in each engine's lib.rs, examine_report_text.rs, dash.rs's fixture and scripts/vllm_therock_gpu_test.py (which reads kind == "rocm_gpu" from Detect, not the healthcheck). No dash-TUI, rocmd-snapshot, script or doc reads the healthcheck device string literally.

Also trimmed

The review-thread references in engine_ipc.rs (two), therock/family.rs, host_gpu.rs and the cli.rs test doc, and the note-to-reviewers paragraph in docs/testing.md — each now states the reason instead. The PR body is rewritten from the final diff: serve-55 → serve-61, examine-49 → examine-55, which lane actually covers the device: row, what serve-61 does and does not show, and the Anthropic-endpoint subsystem, which the body had never mentioned at all. That omission is very plausibly why finding 3 went unnoticed for several rounds, as you said.

Verification

Linux devbox: cargo fmt --all --check; cargo clippy --workspace --all-targets -- -D warnings; cargo clippy -p e2e-cucumber --test e2e -- -D warnings; cargo test -p rocm-core (1627); cargo test -p rocm --bin rocm (1947); cargo test -p rocm-engine-aim (95) / -p rocm-engine-vllm (172) / -p rocm-engine-protocol; cargo test -p e2e-cucumber --test feature_naming; cargo xtask check-architecture-doc / check-crate-edges / check-module-size. Every new test was mutation-checked individually — break the production change, confirm red, restore.

One conflict, in `apps/rocm/src/chat/mcp_bridge.rs`, and it is the
code-was-removed shape rather than two additions colliding.

Main's #980 deletes `internal_mcp_arguments` outright and replaces it with
`internal_mcp_tool_call` plus `mcp_call_cli_arguments`, so that a pasted MCP
envelope is unwrapped in one named step instead of silently inside the
dispatcher -- the point of that change being that the approval card and the
executed call cannot disagree. This branch still carried the old function.

Resolved by taking main's design whole and keeping only this branch's own
addition from the other side, the `engines` MCP tool-result helper. Keeping
this branch's `internal_mcp_arguments` would have reinstated the unwrapping
main had just removed, which is the opposite of what #980 is for; grep
confirms no caller of it survives.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main landed the engine stdio protocol versioning (#1073), which appends to the
same two places in `apps/rocm/src/engine_ipc.rs` this branch does: the module's
free functions and its test module.

Both conflicts are the same shape -- each side's last item was left
unterminated by the split while a single closing brace sat below for both -- so
the brace is reproduced for each block rather than closing only whichever came
last. Nothing is interleaved: this branch's device-probe helpers and tests and
main's `answer_builtin_engine_stdio` and its protocol-contract test are kept
whole and side by side.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r force-pushed the ROCMAI-550-vulkan-support-messaging branch from 20c48b4 to f04357f Compare November 17, 2026 18:13
@r0x0r

ghost commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Force-pushed the commit-message rewrite you asked for. 20c48b444 → f04357fba.

The tree is byte-identical. That was the safety property I wanted before touching a 65-commit branch, and it is verifiable rather than asserted — the root tree object is ba47776b602cf43cd17926f2a70708a7467f77dc on both the old and new heads, and git diff between them is empty. Only commit messages changed; no content moved.

What the rewrite did, across all 65 commits:

All 65 commits are re-signed and still signed-off (filter-branch drops GPG signatures, so the rewrite re-signed via commit-tree -S): 0 unsigned, 0 missing a Signed-off-by. Pushed with --force-with-lease pinned to the expected old SHA.

#1036 is repaired and did not need a force-push. Rewriting this branch gave every commit a new SHA, so #1036's merge-base fell back to main and it went CONFLICTING. A plain merge of the new head conflicted for the same reason. Instead it is re-parented onto the reworded base by a merge commit whose tree is byte-identical to its previous tip (b8ddd7885… both sides) — its content is untouched and its own history is intact. Its diff against this branch is the same 47 files as before the rewrite, file-for-file.

Both PRs are MERGEABLE again. The old heads are kept locally as refs/backup/pr678-preforce and refs/backup/pr1036-preforce in case anything needs recovering.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants