diff --git a/README.md b/README.md index 1af112fe7..443a5b761 100644 --- a/README.md +++ b/README.md @@ -682,6 +682,12 @@ family is substituted for it. `rocm examine` reports the same conclusion on a These GPUs still work with the Vulkan backend, which needs no ROCm SDK. +`rocm examine` prints that `rocm_support` line on **every** host, not only this +case, at one of three levels: ROCm/HIP-supported, Vulkan-only (the case above), +or unknown. Unknown covers both no GPU detected and a target this build's +catalog does not recognise — detection gaps rather than a verdict about the +hardware, so neither is reported as "no ROCm build". + #### Compiler toolchain (--devel) Pass `--devel` to also install the compiler and headers needed to build GPU @@ -1390,18 +1396,33 @@ the running service, or stop it first to serve on the new port. By default the server runs in the background under ROCm CLI's supervision and prints a deployment summary: a progress indicator while it starts, then a table -with the status, the full inference endpoint, the API-qualified model name, and a -quick smoke test (time to first token and approximate tokens per second). On an -interactive terminal, if the smoke test cannot complete, the summary is headed -`(smoke test failed)`, shows the status as `started, smoke test failed` instead -of `ready`, and a note gives the reason and points at +with the status, the engine and the backend device the engine reports (for +example `rocm gpu 0 (AMD Instinct MI300X)`, or `vulkan (AMD Radeon Graphics)` +where the engine pinned no GPU ordinal). It is read back from the running +engine, so it is that engine's own account of the backend it was launched on +rather than an independent measurement of the hardware. The summary also carries +the full inference endpoint, the API-qualified model name, and a quick smoke +test (time to first token and approximate tokens per second). `rocm services` +reports the same `device` for each running service. + +On an interactive terminal, if the smoke test cannot complete, the summary is +headed `(smoke test failed)`, shows the status as `started, smoke test failed` +instead of `ready`, and a note gives the reason and points at `rocm logs --service `. ROCm CLI does not stop the server, and the command still exits 0. Which failures are detected, and which still read as `ready`, is listed under [Serve Smoke Test Failure](https://github.com/ROCm/rocm-cli/blob/main/docs/testing.md#serve-smoke-test-failure) -in the testing guide. Control -returns to your shell with the server still running; manage it later with `rocm -services` (below). +in the testing guide. + +The device line is omitted when the engine cannot say yet. A server is reported +ready as soon as it answers, which can be before it has finished loading the +model, and until then the engine does not know which device it ended up on; +`rocm serve` waits a few seconds for it rather than holding your terminal, and +`rocm services` shows it once it is known. The GPU name is omitted on a machine +with more than one AMD GPU, where naming one of them could name the wrong one. + +Control returns to your shell with the server still running; manage it later +with `rocm services` (below). `--verbose` (or `--foreground`) instead attaches to the server in the current terminal and streams every engine log line. Use it to debug a startup problem. @@ -1646,6 +1667,15 @@ It does not suggest running the stop again: when only a process under the recorded one survived, a second stop can report the server stopped while that process still runs. +`list` asks each running server which backend and GPU it is actually on, so it +can take a few seconds rather than returning instantly. The query is capped in +total — about 18 seconds worst case, however many servers are listed — and a +server that does not answer in time simply has no `device` line rather than +holding up the rest of the list. The same applies to asking about servers in +`rocm chat`, which prints the same list. `list --json` is unaffected: it emits +the stored service records, which have no `device` field, asks no server +anything, and returns as fast as it always did. + `restart` starts a server again only once its stop has confirmed that every recorded process is gone; if the stop cannot confirm that, it says so, starts nothing, and leaves the stop request recorded. diff --git a/apps/rocm/src/advised_commands.rs b/apps/rocm/src/advised_commands.rs index 84a5ee852..e6d48b3e5 100644 --- a/apps/rocm/src/advised_commands.rs +++ b/apps/rocm/src/advised_commands.rs @@ -1304,6 +1304,24 @@ const NOT_INVOCATIONS: &[(&str, &str, &str)] = &[ "the doc's expected result is that the parser rejects the unknown option \ with exit 2, the code the catalog publishes for an unknown fix-id", ), + // The `device:` row's value (ROCMAI-550). Every engine reports the backend + // it selected as ` [gpu ] [()]` -- `vulkan (AMD Radeon + // Graphics)`, `rocm gpu 1 (AMD Instinct MI300X)` -- so the ROCm half of + // that value reads as a `rocm` invocation to this scan while naming no + // command at all. The engines no longer need an entry each: they build the + // string through `rocm_engine_protocol::format_reported_device` rather + // than holding a `"rocm gpu ..."` literal. These are the two docs that + // quote the format back. + ( + "README.md", + "rocm gpu 0 (AMD Instinct MI300X)", + "an example `device:` value in the deployment-summary description, not a command", + ), + ( + "docs/manual-testing.md", + "rocm gpu 0 (AMD Instinct MI300X)", + "the `device:` value the tester is told to expect, not a command to run", + ), ]; /// Natural-language examples must reach the planner with a multi-word request: diff --git a/apps/rocm/src/bridge_snapshot.rs b/apps/rocm/src/bridge_snapshot.rs index 62e353f4b..a5c7b648b 100644 --- a/apps/rocm/src/bridge_snapshot.rs +++ b/apps/rocm/src/bridge_snapshot.rs @@ -32,15 +32,24 @@ use crate::managed_service_status::load_managed_services; pub(crate) fn build_codex_bridge_snapshot(paths: &AppPaths) -> Result { let config = RocmCliConfig::load(paths).unwrap_or_default(); + // Hoisted out of the struct literal so the engine rows can be told what this + // host actually supports. The probe is already paid for: `gpu_support_level` + // takes exactly the two fields this summary already carries, so the engine + // list costs no detection the snapshot was not doing anyway. + let examine = ExamineSummary::gather()?; + let gpu_support_level = rocm_core::gpu_support_level( + examine.detected_gfx_target.as_deref(), + examine.compatible_therock_family.as_deref(), + ); Ok(CodexBridgeSnapshot { protocol: "rocmd-codex-bridge-v0".to_owned(), generated_at_unix_ms: rocm_core::unix_time_millis(), - examine: ExamineSummary::gather()?, + examine, gpu: build_codex_bridge_gpu_snapshot(&config), config, automation_runtime: AutomationRuntimeState::load(paths)?, recent_automation_events: load_recent_automation_events(paths, 32)?, - engines: builtin_codex_bridge_engine_inventory(), + engines: builtin_codex_bridge_engine_inventory(gpu_support_level), services: load_managed_services(paths)?, }) } @@ -63,7 +72,22 @@ pub(crate) fn build_codex_bridge_gpu_snapshot(config: &RocmCliConfig) -> CodexBr } } -pub(crate) fn builtin_codex_bridge_engine_inventory() -> Vec { +/// The engine catalogue in the codex-bridge snapshot and the bridge's `engines` +/// tool, built from [`engine_inventory`] with lemonade's note replaced +/// by `lemonade_engine_note` for the given support level. +/// +/// It takes the level rather than reading the static row because the stale +/// "with ROCm llama.cpp backend" claim is exactly what ROCMAI-550 exists to +/// remove, and this is a user-facing surface like the other two: a bridge +/// client renders `summary` to the operator. +/// +/// Only `summary` moves. The field set, its names and `installed_binary: true` +/// beside it are untouched, so the bridge's JSON shape is the same; what +/// changes is that one string stops asserting ROCm support on a host that has +/// none. +pub(crate) fn builtin_codex_bridge_engine_inventory( + gpu_support_level: rocm_core::GpuSupportLevel, +) -> Vec { let default_engine = default_engine_for_platform(); let binary_path = daemon_binary_path() .ok() @@ -72,7 +96,7 @@ pub(crate) fn builtin_codex_bridge_engine_inventory() -> Vec .iter() .map(|(id, summary)| CodexBridgeEngine { id: (*id).to_owned(), - summary: (*summary).to_owned(), + summary: crate::engine_inventory_note(id, summary, gpu_support_level).to_owned(), default_for_platform: *id == default_engine, installed_binary: true, binary_path: binary_path.clone(), diff --git a/apps/rocm/src/chat/mcp_bridge.rs b/apps/rocm/src/chat/mcp_bridge.rs index cf5ea7c6a..fea0554d7 100644 --- a/apps/rocm/src/chat/mcp_bridge.rs +++ b/apps/rocm/src/chat/mcp_bridge.rs @@ -44,9 +44,10 @@ use crate::uninstall_plan::render_uninstall_dry_run; use crate::update_cmd::render_update_text; use crate::{ build_codex_bridge_gpu_snapshot, build_codex_bridge_snapshot, - builtin_codex_bridge_engine_inventory, cli_version_string, record_cli_audit_event, - render_config_text, render_engine_inventory_text_with_paths, render_examine_text, - render_examine_text_with_paths, render_model_registry_text_with_context_and_host, + builtin_codex_bridge_engine_inventory, cli_version_string, host_gpu_support_level, + record_cli_audit_event, render_config_text, render_engine_inventory_text_with_paths, + render_examine_text, render_examine_text_with_paths, + render_model_registry_text_with_context_and_host, render_model_registry_verbose_text_with_context_and_host, render_setup_status_text, therock_install_version_selector, }; @@ -186,13 +187,9 @@ pub(crate) fn run_internal_mcp_call( serde_json::json!(gpu), )) } - "engines" => { - let engines = builtin_codex_bridge_engine_inventory(); - Ok(internal_mcp_tool_success( - format!("Found {} engine entries.", engines.len()), - serde_json::json!({ "engines": engines }), - )) - } + "engines" => Ok(internal_mcp_engines_tool_result(host_gpu_support_level( + Some(paths), + ))), "services" => { let services = load_managed_services(paths)?; Ok(internal_mcp_tool_success( @@ -388,6 +385,24 @@ pub(crate) fn run_internal_mcp_call( } } +/// The `engines` MCP tool result for a host at `gpu_support_level`. +/// +/// Split from the match arm above only so the GPU-aware branch is reachable +/// from a test: the arm's own argument is a live host probe, and no CI runner +/// has a GPU that is detected *and* confirmed to have no ROCm build, so the +/// Vulkan-only wording a bridge client would render could not otherwise be +/// asserted on. Nothing else moved — the summary text, the `engines` key and +/// the entries themselves are what the arm built inline. +fn internal_mcp_engines_tool_result( + gpu_support_level: rocm_core::GpuSupportLevel, +) -> serde_json::Value { + let engines = builtin_codex_bridge_engine_inventory(gpu_support_level); + internal_mcp_tool_success( + format!("Found {} engine entries.", engines.len()), + serde_json::json!({ "engines": engines }), + ) +} + /// The validated tool call `run_internal_mcp_call` executes for `arguments`. /// /// `arguments` is used exactly as given: it is the object the dash approval @@ -1361,6 +1376,73 @@ mod tests { assert!(text.lines().any(|line| line == version_line), "{text}"); } + /// The bridge's `engines` tool carries the same claim `rocm engines` and + /// `rocm examine` carry, so it gets the same pair of tests. These assert on + /// the serialized payload because `structuredContent.engines[].summary` is + /// the exact string an MCP client renders to the operator. + #[test] + fn engines_mcp_tool_names_vulkan_for_a_gpu_with_no_rocm_support() { + let result = internal_mcp_engines_tool_result(rocm_core::GpuSupportLevel::VulkanOnly); + + let summary = engines_tool_lemonade_summary(&result); + assert!( + summary.contains("no ROCm/HIP support") && summary.contains("Vulkan llama.cpp backend"), + "the MCP client must be handed the real backend: {summary}" + ); + assert!( + !summary.contains("with ROCm llama.cpp backend"), + "the stale ROCm claim must not reach an MCP client: {summary}" + ); + assert!(mcp_tool_result_text(&result).contains("engine entries")); + } + + #[test] + fn engines_mcp_tool_keeps_rocm_note_for_a_rocm_capable_gpu() { + let result = internal_mcp_engines_tool_result(rocm_core::GpuSupportLevel::RocmHip); + + assert_eq!( + engines_tool_lemonade_summary(&result), + "default embedded Lemonade server with ROCm llama.cpp backend" + ); + } + + /// Pins the wiring the two tests above cannot see: that the tool reaches + /// the host-aware inventory through the live probe rather than a level + /// nailed to a constant. A hard-coded level that disagrees with this host + /// fails here. + #[test] + fn engines_mcp_tool_answers_for_the_host_it_runs_on() { + let (_root, paths) = test_paths("mcp-engines-host-probe"); + + let result = run_internal_mcp_call(&paths, "engines", serde_json::json!({}), false) + .expect("the engines mcp tool is read-only and should run"); + + assert_eq!( + result.get("isError").and_then(serde_json::Value::as_bool), + Some(false) + ); + assert_eq!( + result.get("structuredContent"), + internal_mcp_engines_tool_result(host_gpu_support_level(Some(&paths))) + .get("structuredContent"), + "the `engines` tool must report this host's support level" + ); + } + + /// The lemonade `summary` as an MCP client reads it out of the tool result. + fn engines_tool_lemonade_summary(result: &serde_json::Value) -> String { + result + .pointer("/structuredContent/engines") + .and_then(serde_json::Value::as_array) + .expect("the engines tool must return an `engines` array") + .iter() + .find(|engine| engine.get("id").and_then(serde_json::Value::as_str) == Some("lemonade")) + .and_then(|engine| engine.get("summary")) + .and_then(serde_json::Value::as_str) + .expect("the engines tool must carry a lemonade summary") + .to_owned() + } + /// What the chat `model` tool hands back has to be what `rocm model` /// prints, not the `--verbose` diagnostics. The verbose dump is every /// recipe with its aliases, engine recipes, flags and policy notes; on a diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 55f8eb157..87f438146 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -63,11 +63,14 @@ pub(crate) struct Cli { pub(crate) enum Command { /// Check this computer's GPU, ROCm install, engines, and setup folders. /// - /// A GPU that is recognized but has no ROCm build at all - `gfx90c`, the - /// Vega iGPU in Renoir, Lucienne, Cezanne and Barcelo APUs - is reported on - /// a `rocm_support` line saying so, so that an empty - /// `compatible_therock_family` does not read as a failed probe. Such a GPU - /// still works through the Vulkan backend. + /// A `rocm_support` line is printed on every host, at one of three + /// levels: ROCm/HIP-supported, Vulkan-only, or unknown. Vulkan-only covers + /// a GPU that is recognized but has no ROCm build at all (`gfx90c`, the + /// Vega iGPU in Renoir, Lucienne, Cezanne and Barcelo APUs), which still + /// works through Lemonade's Vulkan backend. Unknown covers both no GPU + /// detected and a target this build's catalog does not recognize: those + /// are detection gaps, not a verdict on the hardware, which is why an + /// empty `compatible_therock_family` does not read as a failed probe. Examine { /// Emit the machine-readable Examination JSON (for diagnosis tooling). #[arg(long)] @@ -476,13 +479,16 @@ rocm model --verbose" /// Anthropic `/v1/messages` endpoint is published beside the OpenAI /// one (see `rocm services list`). By default the /// server runs as a managed background service and prints a deployment summary - /// (status, endpoint, model, and smoke-test metrics); use --verbose to stream engine - /// logs in this terminal instead. When streaming, press Ctrl-D to detach and leave the - /// server running, or Ctrl-C to stop it. If that stop answers that it could not confirm - /// the server exited, it says so on stderr and exits with an error; if the stop itself - /// fails, it says why on stderr, signals the server process directly and exits 0, - /// noting on stderr if that signal fails too. Inspect or stop servers later with - /// `rocm services`. + /// (status, endpoint, model, smoke-test metrics, and the backend device the engine + /// reports it is running on). The device line is read back from the running server, so + /// it is omitted when the engine cannot say yet: a server that answers is not always + /// done loading its weights. `rocm services list` shows it once it is known. Use + /// --verbose to stream engine logs in this terminal instead. When streaming, press Ctrl-D + /// to detach and leave the server running, or Ctrl-C to stop it. If that stop answers + /// that it could not confirm the server exited, it says so on stderr and exits with an + /// error; if the stop itself fails, it says why on stderr, signals the server process + /// directly and exits 0, noting on stderr if that signal fails too. Inspect or stop + /// servers later with `rocm services`. /// /// Under --device gpu_required (the default) this command refuses on a host with no /// usable AMD GPU. Reusing an already-running managed server is the one exception: it @@ -1206,6 +1212,13 @@ pub(crate) enum ComfyuiCommand { pub(crate) enum ServicesCommand { /// Show currently running local model servers, and count the records that /// are no longer running. + /// + /// In the default table, each running server is asked which backend and GPU + /// it is actually on, so this takes a few seconds rather than returning + /// instantly. The asking is capped in total (about 18 seconds worst case, + /// however many servers are listed), and a server that does not answer in + /// time simply has no device line. `--json` asks nothing and carries no + /// device field: it is the stored records, returned as fast as before. List { /// Include failed, stopped, and old service records. #[arg(short, long)] @@ -2752,4 +2765,68 @@ mod tests { assert_ascii_help(&mut Cli::command(), "rocm"); } + + /// Sibling of `every_help_text_in_the_command_tree_is_ascii`, for the same + /// reason one level down: clap renders a doc comment verbatim, with no + /// markdown processing anywhere in its pipeline, so `**every**` written in + /// a doc comment reaches the user's terminal as literal asterisks. That had + /// already happened on `Examine`, where the ASCII test could not see it + /// because `*` and `_` are ASCII. + /// + /// Only `**` and `__` are rejected, and only those: they have no honest + /// use in plain help text. A single `*` does (`gfx110*`, a glob in an + /// example), and backticks are the repo's deliberate house style for + /// quoting commands in help, so neither is banned here. + /// + /// The hidden `__engine-*` subcommands are a real `__` in rendered help — + /// their own usage lines print their names — so each command's internal + /// names are removed before the search rather than the check being + /// dropped. The list is read off the command tree, not hard-coded, so a + /// new hidden command does not make this fail spuriously. + #[test] + fn no_help_text_in_the_command_tree_contains_markdown_emphasis() { + fn assert_no_markdown_emphasis(cmd: &mut clap::Command, path: &str) { + let mut internal: Vec = cmd + .get_subcommands() + .map(|sub| sub.get_name().to_owned()) + .filter(|name| name.starts_with("__")) + .collect(); + if cmd.get_name().starts_with("__") { + internal.push(cmd.get_name().to_owned()); + } + + // Both renderings, for the same reason as the ASCII test: `-h` + // truncates at the first paragraph, so emphasis in a later one is + // only visible in the long form. + for rendered in [cmd.render_help(), cmd.render_long_help()] { + let mut rendered = rendered.to_string(); + for name in &internal { + rendered = rendered.replace(name.as_str(), ""); + } + for marker in ["**", "__"] { + assert!( + !rendered.contains(marker), + "`{path} --help` must not contain markdown emphasis (`{marker}`); \ + clap renders doc comments verbatim, so the user sees the markup \ + itself: {rendered}" + ); + } + } + // Collect first: `find_subcommand_mut` needs a mutable borrow that + // cannot overlap the iterator borrowing `cmd`. + let names: Vec = cmd + .get_subcommands() + .map(|sub| sub.get_name().to_owned()) + .collect(); + for name in names { + let child_path = format!("{path} {name}"); + let child = cmd + .find_subcommand_mut(&name) + .expect("subcommand name was just enumerated from this command"); + assert_no_markdown_emphasis(child, &child_path); + } + } + + assert_no_markdown_emphasis(&mut Cli::command(), "rocm"); + } } diff --git a/apps/rocm/src/engine_ipc.rs b/apps/rocm/src/engine_ipc.rs index 0735f9b68..57978d80d 100644 --- a/apps/rocm/src/engine_ipc.rs +++ b/apps/rocm/src/engine_ipc.rs @@ -53,13 +53,14 @@ use std::ffi::OsString; use std::io::{self, Read}; use std::path::{Path, PathBuf}; use std::sync::{Mutex, OnceLock}; +use std::time::{Duration, Instant}; use anyhow::{Context, Result, bail}; use rocm_core::{AppPaths, ENGINE_ENVS_ROOT_ENV, engine_binary_path, engine_plugin_dirs}; use rocm_engine_protocol::{ DevicePolicy, EngineMethod, EnginePluginDescriptor, EngineRecipeHint, EngineRequestEnvelope, - EngineResponseEnvelope, EngineStdioOptions, decode_engine_response, engine_request_envelope, - optional_arg, run_engine_stdio, + EngineResponseEnvelope, EngineStdioOptions, HealthcheckRequest, HealthcheckResponse, + decode_engine_response, engine_request_envelope, optional_arg, run_engine_stdio, }; use serde::Serialize; use serde::de::DeserializeOwned; @@ -117,15 +118,59 @@ pub(crate) fn engine_request_with_env_root( request: &T, env_root: Option<&Path>, ) -> Result +where + T: Serialize, + R: DeserializeOwned, +{ + engine_request_with_scoped_env(paths, engine, method, request, env_root, None) +} + +/// [`engine_request_with_env_root`], additionally handing a **built-in** engine +/// the 0600 endpoint key file for one service via `ROCM_SERVE_API_KEY_FILE`. +/// +/// An engine that probes its own server's HTTP endpoint while answering a +/// request reads that variable to authenticate +/// (`rocm_engine_protocol::resolve_endpoint_api_key`, reached from +/// `engines/lemonade`'s and `engines/vllm`'s `/v1/models` probes). Without it, +/// a public or `--require-api-key` service answers 401 and the engine reports +/// the service as not listed - indistinguishable, from the CLI's side, from a +/// server that is simply down. It is threaded per call rather than per process +/// because the key is per service id. +/// +/// **External plugin children are not given it.** The built-in engines run +/// in-process, so the variable is set and restored around one call +/// ([`with_scoped_builtin_engine_env`]); a plugin is a third-party binary this +/// process spawns, and a variable put in its environment is readable from +/// `/proc//environ` and inherited by every grandchild it starts. That is +/// more exposure than an informational `device:` row is worth — the only caller +/// that passes a key file is [`fetch_service_device`]. The cost is that a +/// plugin-backed service with endpoint auth reports no device; it is a row the +/// summary already declares optional. +/// +/// This is a reduction in incidental exposure, not a security boundary: a +/// plugin child is handed `ROCM_CLI_DATA_DIR` by [`apply_app_path_env`] and the +/// key file's location under it is a documented, deterministic convention +/// (`rocm_engine_protocol::endpoint_key_file_path`), so a plugin that wants the +/// key can still derive the path. What changes is that it must go looking. +fn engine_request_with_scoped_env( + paths: Option<&AppPaths>, + engine: &str, + method: EngineMethod, + request: &T, + env_root: Option<&Path>, + endpoint_key_file: Option<&Path>, +) -> Result where T: Serialize, R: DeserializeOwned, { let stream_progress = matches!(&method, EngineMethod::Install); let envelope = engine_request_envelope(method, request)?; - if let Some(envelope) = with_scoped_builtin_engine_env(paths, env_root, || { - builtin_engine_request(engine, &envelope) - }) { + if let Some(envelope) = + with_scoped_builtin_engine_env(paths, env_root, endpoint_key_file, || { + builtin_engine_request(engine, &envelope) + }) + { return decode_engine_response(envelope); } @@ -218,9 +263,10 @@ impl Drop for ScopedEnvVar { fn with_scoped_builtin_engine_env( paths: Option<&AppPaths>, env_root: Option<&Path>, + endpoint_key_file: Option<&Path>, f: impl FnOnce() -> R, ) -> R { - if paths.is_none() && env_root.is_none() { + if paths.is_none() && env_root.is_none() && endpoint_key_file.is_none() { return f(); } let lock = BUILTIN_ENGINE_ENV_LOCK.get_or_init(|| Mutex::new(())); @@ -234,6 +280,12 @@ fn with_scoped_builtin_engine_env( if let Some(env_root) = env_root { vars.push(ScopedEnvVar::set_path(ENGINE_ENVS_ROOT_ENV, env_root)); } + if let Some(endpoint_key_file) = endpoint_key_file { + vars.push(ScopedEnvVar::set_path( + rocm_engine_protocol::ENDPOINT_API_KEY_FILE_ENV, + endpoint_key_file, + )); + } let result = f(); drop(vars); result @@ -274,6 +326,284 @@ pub(crate) fn run_builtin_engine_stdio(engine: &str) -> Result<()> { Ok(()) } +/// Best-effort query of the backend/device a running managed service actually +/// selected (e.g. `"vulkan gpu 0"` / `"rocm gpu 0"`), via the same +/// `Healthcheck` RPC every engine plugin already answers +/// (`engines/lemonade` reads it back from its own `backend_requested` state; +/// `engines/vllm` and `engines/aim` report the ROCm device they pinned). Read +/// from the engine, never guessed from the launch-time GPU selection, so it +/// reflects what the server is actually running on. `None` on any failure - +/// this is informational text, not worth failing `serve`/`services list` over. +/// +/// How long one call can take, since `rocm services list` makes one per ready +/// service: [`engine_request`] itself imposes no deadline, so the bound comes +/// from the engine. The three built-in engines answer in-process and every +/// endpoint probe they make is timeout-bounded (`engines/vllm`'s and +/// `engines/aim`'s own `HEALTHCHECK_TIMEOUT_MS`, `engines/lemonade`'s +/// `rocm_core::INFERENCE_PROBE_TIMEOUT`), so a stalled server costs seconds, +/// not forever. An *external* plugin is a spawned child that +/// `engine_request` waits on unbounded, exactly as every other engine call +/// site in the CLI does. [`fetch_service_devices`] is what keeps a list of +/// services from multiplying that cost by the number of services. +pub(crate) fn fetch_service_device( + paths: &AppPaths, + engine: &str, + service_id: &str, +) -> Option { + // Hand the engine this service's 0600 key file. Both built-in engines + // authenticate their own `/v1/models` probe from + // `ROCM_SERVE_API_KEY_FILE`, so without it a public or `--require-api-key` + // service answers 401, the engine reports it as not listed, and the device + // comes back `"unknown"` - the row would vanish for exactly the services + // that are protected. `None` for a loopback service, which has no key file + // and needs none. + let endpoint_key_file = crate::endpoint_keys::endpoint_key_file_if_present(paths, service_id); + let response: HealthcheckResponse = engine_request_with_scoped_env( + Some(paths), + engine, + EngineMethod::Healthcheck, + &HealthcheckRequest { + service_id: service_id.to_owned(), + }, + None, + endpoint_key_file.as_deref(), + ) + .ok()?; + device_worth_reporting(response.device) +} + +/// The engine's reported device, or `None` when it said nothing usable. +/// +/// `"unknown"` is the engines' own "I cannot say yet", and it is not more +/// informative than showing nothing — a `device: unknown` row claims the CLI +/// asked and learned something, which is the opposite of what happened. +/// +/// *When* an engine stops saying it is not uniform, and that difference is why +/// [`SERVE_DEVICE_PROBE_BUDGET`] exists at all: +/// +/// * `engines/vllm` and `engines/aim` answer as soon as a state file exists +/// for the service id (`Some(state) => reported_device(state)`, +/// `None => "unknown"`), so the row is available from the launch write +/// onwards, readiness or not; +/// * `engines/lemonade` keeps answering `"unknown"` while a state file plainly +/// exists — its `healthcheck_service_at` reports a device only once the +/// service is `ready`, which there means its own inference probe has come +/// back. +/// +/// So this filter is what keeps the slowest of the three from printing a row +/// that says nothing; it is not a uniform no-state-on-disk guard. +/// +/// A named function rather than an inline `then_some` because the only caller +/// is an RPC to a live engine: in default CI that call never succeeds, so the +/// filter is unreachable there and deleting it would be invisible until a +/// freshly-launched Lemonade service printed `device: unknown` on a GPU lane. +fn device_worth_reporting(device: String) -> Option { + (!device.trim().is_empty() && device != "unknown").then_some(device) +} + +/// How long `rocm serve`'s deployment summary may keep asking the engine which +/// device it selected, and how long it waits between asks. +/// +/// A managed launch reports `ready` as soon as its server answers, but +/// `engines/lemonade` reports `device: "unknown"` until its own +/// `inference_verified` probe has come back — so asking once, immediately, +/// loses the row on exactly the Vulkan case ROCMAI-550 is about. One retry +/// window closes the common case: the default smoke test has just put a real +/// completion through the server, so the engine's own probe is a cache hit. +/// +/// **It cannot close every case, and is not meant to.** A server that reported +/// `ready` while weights are still loading can stay unverified for minutes, and +/// waiting that out would hold the terminal hostage for an informational row +/// that `rocm services list` will show as soon as it exists. So the row is +/// *optional by contract* — see [`crate::serve_summary::DeploymentSummary`]'s +/// `device` field, `docs/manual-testing.md` and README.md, which all say it can +/// be absent. +/// +/// The budget is a start gate, like [`SERVICE_DEVICE_PROBE_BUDGET`]: it is +/// tested before each attempt, never inside one, so the worst-case addition to +/// `rocm serve` is this budget plus one in-flight probe — about 14s with +/// Lemonade's 8s `rocm_core::INFERENCE_PROBE_TIMEOUT`, and only on a launch +/// that was going to have no row at all. +const SERVE_DEVICE_PROBE_BUDGET: Duration = Duration::from_secs(6); +/// Pause between the attempts [`SERVE_DEVICE_PROBE_BUDGET`] allows. Short +/// enough that the common case (the engine is a second away from verifying) +/// resolves quickly, long enough not to spin on an engine that answers +/// instantly. +const SERVE_DEVICE_PROBE_INTERVAL: Duration = Duration::from_secs(2); + +/// [`fetch_service_device`], retried briefly, for `rocm serve`'s one-shot +/// deployment summary. +/// +/// `None` is a legitimate answer and must stay cheap to get: see +/// [`SERVE_DEVICE_PROBE_BUDGET`] for why the wait is bounded and why the +/// summary prints no `device:` row rather than blocking on one. +pub(crate) fn fetch_service_device_briefly( + paths: &AppPaths, + engine: &str, + service_id: &str, +) -> Option { + let started = Instant::now(); + retry_service_device( + || started.elapsed(), + || fetch_service_device(paths, engine, service_id), + std::thread::sleep, + ) +} + +/// The retry loop of [`fetch_service_device_briefly`], with the clock, the +/// probe and the wait injected. +/// +/// Split out for the same reason as [`collect_service_devices`]: driving this +/// against the live clock would mean a test that really sleeps out +/// [`SERVE_DEVICE_PROBE_BUDGET`] against a real engine. What has to be +/// guaranteed is not just that a late answer is reported but that the loop +/// *stops* — an unbounded retry here would hang every `rocm serve` whose engine +/// never verifies. +fn retry_service_device(mut elapsed: C, mut probe: P, mut wait: S) -> Option +where + C: FnMut() -> Duration, + P: FnMut() -> Option, + S: FnMut(Duration), +{ + loop { + if let Some(device) = probe() { + return Some(device); + } + if elapsed() >= SERVE_DEVICE_PROBE_BUDGET { + return None; + } + wait(SERVE_DEVICE_PROBE_INTERVAL); + } +} + +/// How long [`fetch_service_devices`] may spend on live `Healthcheck` RPCs in +/// total, across however many services were listed. +/// +/// `rocm services list` was pure local I/O before the `device:` row; the row is +/// worth a short wait but not an unbounded one, and the cost has to stay flat +/// in the number of services rather than growing with it. Sized to let a +/// couple of healthy engines answer (`engines/lemonade`'s endpoint probe is +/// the slowest at `rocm_core::INFERENCE_PROBE_TIMEOUT`, 8s worst case, and +/// only on a miss) while still bounding the whole render. +/// +/// **Worst-case wall time for `rocm services list` is this budget plus one +/// in-flight probe, i.e. ~18s** with the built-in engines, not 10s. The budget +/// is a *start* gate: it is tested before each probe, never inside one, so a +/// probe begun at 9.99s still runs to its own timeout. The arithmetic is 10s +/// of elapsed budget plus Lemonade's 8s `INFERENCE_PROBE_TIMEOUT`. That bound +/// applies equally to `rocm chat` when its natural-language interpreter +/// resolves a request to the `services` command: that path runs the real +/// command in process (`chat::mcp_bridge`'s `run_rocm_read_only_in_process`) +/// and so renders through the same +/// [`crate::services_text::render_services_text`]. +/// +/// It does *not* apply to the structured MCP tool that is also named +/// `services`. That one renders through `chat::mcp_bridge`'s +/// `render_services_tool_result_text`, a different and older function with no +/// `device:` row, so a bridge client calling the tool makes no probe and waits +/// for nothing — unlike the sibling `engines` MCP tool, which ROCMAI-550 did +/// make host-aware. The shared name is the trap, so this says which surface it +/// means; giving the tool the row is a change in its own +/// right, not something this constant may be read as already covering. +/// +/// # The decision, recorded +/// +/// This is accepted rather than merely observed. `rocm services list` and +/// `rocm chat`'s `services` command can both block ~18s on a host whose +/// engines are slow to answer, and for the duration the built-in probe holds +/// the process-wide engine env lock [`with_scoped_builtin_engine_env`] takes, +/// so a concurrent caller waits behind it. +/// +/// Two cheaper options were considered and not taken. Moving the probe off the +/// render path means the row stops reflecting the host at the moment it is +/// printed, which is the whole reason ROCMAI-550 added it — the `device:` row +/// exists to answer "what is this actually running on *now*". Probing once and +/// printing nothing for the rest makes the output depend on service ordering, +/// which reads as a bug to anyone who lists twice and sees different rows. +/// +/// Both remain reasonable calls and this one is reversible: it is a constant and +/// a start gate, not a structural choice. If a maintainer prefers a 10s hard +/// cap, the change is a per-probe deadline, which needs a cancellable probe +/// (see below) rather than a smaller number here. +/// +/// Why not a per-probe deadline that would hold the total at 10s: the probe is +/// `engine_request`, which for a built-in engine is a synchronous in-process +/// call and for an external plugin is a `wait()` on a spawned child. Neither +/// can be cancelled from here without either a thread per probe (ruled out by +/// the process-wide env/mutex scoping `fetch_service_devices` documents) or a +/// kill path for a child mid-RPC. Both are larger changes than the row is +/// worth; the start gate is what keeps the cost flat in the number of +/// services, which is the growth that actually matters. +pub(crate) const SERVICE_DEVICE_PROBE_BUDGET: Duration = Duration::from_secs(10); + +/// Devices for several services at once, keyed by service id, under one shared +/// time budget. +/// +/// The probes run sequentially on purpose. They cannot usefully be run in +/// parallel: [`engine_request`] routes the built-in engines through +/// [`with_scoped_builtin_engine_env`], which takes a process-wide mutex *and* +/// mutates process-wide `ROCM_CLI_*` env vars for the duration, so concurrent +/// callers would serialize on that lock anyway and racing them would be +/// unsound. What this does instead is cap the aggregate: once +/// [`SERVICE_DEVICE_PROBE_BUDGET`] is spent, the remaining services are left +/// without a `device:` row rather than each adding its own wait. The row is +/// informational, so dropping it is strictly better than making +/// `rocm services list --all` hang behind one unresponsive server. +/// +/// The budget is checked between probes, not inside one: a single external +/// plugin that never exits still blocks, because `engine_request` waits on its +/// child unbounded. That is the pre-existing behavior of every engine call +/// site in the CLI and is not made worse here. See +/// [`SERVICE_DEVICE_PROBE_BUDGET`] for what that makes the worst-case wall +/// time of a `rocm services list`. +pub(crate) fn fetch_service_devices<'a, I>( + paths: &AppPaths, + services: I, +) -> std::collections::HashMap +where + I: IntoIterator, +{ + let started = Instant::now(); + collect_service_devices( + services, + || started.elapsed(), + |service_id, engine| fetch_service_device(paths, engine, service_id), + ) +} + +/// The budget loop of [`fetch_service_devices`], with the clock and the probe +/// injected. +/// +/// Split out so the budget-exhausted exit has a test. In the real function the +/// clock is `Instant::elapsed` and the probe is a live engine RPC, so covering +/// that branch would otherwise mean making a test wait out +/// [`SERVICE_DEVICE_PROBE_BUDGET`] against real engines — which is both slow +/// and exactly the timing-dependent shape that flakes. What the branch has to +/// guarantee is not just that later services lack a `device:` row but that +/// they were never *probed*, since the whole point is to stop paying; only an +/// injected probe can observe that. +fn collect_service_devices<'a, I, C, P>( + services: I, + mut elapsed: C, + mut probe: P, +) -> std::collections::HashMap +where + I: IntoIterator, + C: FnMut() -> Duration, + P: FnMut(&str, &str) -> Option, +{ + let mut devices = std::collections::HashMap::new(); + for (service_id, engine) in services { + if elapsed() >= SERVICE_DEVICE_PROBE_BUDGET { + break; + } + if let Some(device) = probe(service_id, engine) { + devices.insert(service_id.to_owned(), device); + } + } + devices +} + /// The answer a built-in engine's `handle` gives to the request `text` read on /// stdin: the version is judged before the method or payload is decoded (see /// [`rocm_engine_protocol::answer_engine_request`]), as a separately built @@ -405,6 +735,27 @@ mod tests { use super::*; use crate::tests::test_paths; + /// The `unknown` filter, which nothing else can reach: its caller is an RPC + /// to a live engine, so in default CI it never runs and deleting it stays + /// green. + #[test] + fn a_device_the_engine_does_not_know_is_not_reported_as_one() { + for nothing in ["unknown", "", " "] { + assert_eq!( + device_worth_reporting(nothing.to_owned()), + None, + "`{nothing}` is the engine saying it has no answer yet, and a \ + `device:` row would claim otherwise" + ); + } + for device in ["rocm gpu 0", "vulkan (AMD Radeon Graphics)"] { + assert_eq!( + device_worth_reporting(device.to_owned()), + Some(device.to_owned()) + ); + } + } + /// `rocm` must actually ASK for the withholding, not merely be able to. /// /// The protocol crate's own test proves the flag causes the clearing; this @@ -429,6 +780,200 @@ mod tests { } } + /// The endpoint key file reaches a built-in engine through the process + /// environment, and is gone again afterwards. + /// + /// Both halves matter and neither was covered. Without the variable a + /// `--require-api-key` service's own `/v1/models` probe answers 401, the + /// engine reports the service as unlisted, and the `device:` row silently + /// vanishes for exactly the protected services. Without the restore, one + /// service's key would leak into the next service's probe in the same + /// `rocm services list` — `with_scoped_builtin_engine_env` mutates + /// process-wide state, which is why the probes are serialized in the first + /// place. + #[test] + fn the_endpoint_key_file_is_scoped_to_one_builtin_engine_call() { + let key = rocm_engine_protocol::ENDPOINT_API_KEY_FILE_ENV; + // SAFETY-adjacent: the scoped setter takes the same process-wide lock + // this test observes through, so no other engine call can interleave. + let before = std::env::var_os(key); + let seen = with_scoped_builtin_engine_env( + None, + None, + Some(Path::new("/tmp/rocm-e2e-endpoint-key")), + || std::env::var_os(key), + ); + assert_eq!( + seen.as_deref(), + Some(std::ffi::OsStr::new("/tmp/rocm-e2e-endpoint-key")), + "the engine must be able to read the key file path it was given" + ); + assert_eq!( + std::env::var_os(key), + before, + "the key must not outlive the call it was scoped to" + ); + + // And nothing is set when no key file was supplied: a loopback service + // has none, and inventing one would point the engine at a file that + // does not exist. + assert_eq!( + with_scoped_builtin_engine_env(None, None, None, || std::env::var_os(key)), + before + ); + } + + /// Every service inside the budget is probed and reported. + #[test] + fn service_device_probes_run_while_the_budget_holds() { + let mut probed = Vec::new(); + let devices = collect_service_devices( + [("svc-a", "lemonade"), ("svc-b", "vllm")], + || Duration::ZERO, + |service_id, engine| { + probed.push(service_id.to_owned()); + Some(format!("{engine}-device")) + }, + ); + + assert_eq!(probed, ["svc-a", "svc-b"]); + assert_eq!( + devices.get("svc-a").map(String::as_str), + Some("lemonade-device") + ); + assert_eq!( + devices.get("svc-b").map(String::as_str), + Some("vllm-device") + ); + } + + /// Once [`SERVICE_DEVICE_PROBE_BUDGET`] is spent the remaining services are + /// *not probed at all*, and what was already learned is still rendered. + /// + /// This is the branch that keeps `rocm services list` — and the chat + /// `services` tool, which renders through the same + /// `render_services_text` — from scaling its wall time with the number of + /// services. Dropping the later rows is only half of it; the point is to + /// stop paying for them, so the probe counter below is the real assertion. + /// Asserting it against the live clock and live engines would mean a test + /// that waits out ten real seconds, which is why the loop takes both + /// injected. + #[test] + fn service_device_probes_stop_once_the_budget_is_exhausted() { + let mut probed = Vec::new(); + // Inside the budget for the first service, over it from then on. + let mut tick = 0; + let devices = collect_service_devices( + [ + ("svc-fast", "lemonade"), + ("svc-hung", "vllm"), + ("svc-never-reached", "aim"), + ], + || { + tick += 1; + if tick <= 1 { + Duration::ZERO + } else { + SERVICE_DEVICE_PROBE_BUDGET + } + }, + |service_id, _engine| { + probed.push(service_id.to_owned()); + Some("rocm gpu 0".to_owned()) + }, + ); + + assert_eq!( + probed, + ["svc-fast"], + "the budget must stop further probing, not merely hide its results" + ); + assert_eq!( + devices.get("svc-fast").map(String::as_str), + Some("rocm gpu 0"), + "what was learned before the budget ran out is still reported" + ); + assert!( + !devices.contains_key("svc-hung") && !devices.contains_key("svc-never-reached"), + "an unprobed service must get no `device:` row: {devices:?}" + ); + } + + /// The serve summary asks again when the first answer is `None`: the + /// engine can be seconds from verifying its own inference probe, which is + /// the `device: "unknown"` window `engines/lemonade` has right after a + /// launch reports ready. + #[test] + fn the_serve_summary_probe_waits_for_a_device_that_is_not_ready_yet() { + let mut attempts = 0; + let mut waited = Duration::ZERO; + let device = retry_service_device( + || Duration::ZERO, + || { + attempts += 1; + (attempts > 2).then(|| "vulkan gpu 0".to_owned()) + }, + |interval| waited += interval, + ); + + assert_eq!(device.as_deref(), Some("vulkan gpu 0")); + assert_eq!(attempts, 3); + assert_eq!( + waited, + SERVE_DEVICE_PROBE_INTERVAL * 2, + "the retries must pace themselves rather than spin" + ); + } + + /// And it gives up. A lemonade service whose weights are still loading can + /// stay `unknown` for minutes; the summary prints no `device:` row rather + /// than holding the terminal, which is why the row is documented as + /// optional. + #[test] + fn the_serve_summary_probe_gives_up_within_its_budget() { + let mut attempts = 0; + // Inside the budget for the first attempt, past it from then on. + let mut tick = 0; + let device = retry_service_device( + || { + tick += 1; + if tick <= 1 { + Duration::ZERO + } else { + SERVE_DEVICE_PROBE_BUDGET + } + }, + || { + attempts += 1; + None + }, + |_| {}, + ); + + assert_eq!(device, None); + assert_eq!( + attempts, 2, + "the budget must end the retries, not merely discard their results" + ); + } + + /// A service whose engine cannot answer costs its own probe and nothing + /// more: it is skipped, and the services after it are still probed. + #[test] + fn a_service_whose_probe_fails_does_not_suppress_the_rest() { + let devices = collect_service_devices( + [("svc-quiet", "lemonade"), ("svc-ok", "vllm")], + || Duration::ZERO, + |service_id, _engine| (service_id == "svc-ok").then(|| "rocm gpu 1".to_owned()), + ); + + assert!(!devices.contains_key("svc-quiet"), "{devices:?}"); + assert_eq!( + devices.get("svc-ok").map(String::as_str), + Some("rocm gpu 1") + ); + } + /// Every engine built into this binary holds the engine protocol contract on /// both of its paths: the decoded request `rocm` hands it in-process, and /// `answer_builtin_engine_stdio`, the function `rocm __engine-stdio ` @@ -685,7 +1230,7 @@ mod tests { let done = Arc::clone(&done); std::thread::spawn(move || { while !done.load(Ordering::Relaxed) { - with_scoped_builtin_engine_env(Some(&paths), None, || { + with_scoped_builtin_engine_env(Some(&paths), None, None, || { std::thread::sleep(std::time::Duration::from_millis(2)); }); // Outside the lock, so the test's own turn is not starved. diff --git a/apps/rocm/src/examine_report_text.rs b/apps/rocm/src/examine_report_text.rs index 816cc854f..5e71bcb26 100644 --- a/apps/rocm/src/examine_report_text.rs +++ b/apps/rocm/src/examine_report_text.rs @@ -74,7 +74,9 @@ use crate::runtimes_cmd::{ runtime_keys_text, runtime_usability_status, setup_runtime_paths, toolchain_state_text, }; use crate::therock; -use crate::{PlatformRuleOut, engine_inventory, engine_ruled_out_by_platform}; +use crate::{ + PlatformRuleOut, engine_inventory, engine_inventory_note, engine_ruled_out_by_platform, +}; pub(crate) fn render_launch_summary(paths: &AppPaths, config: &RocmCliConfig) -> String { // User-facing text, so it must agree with what `serve` would pick — the same @@ -132,7 +134,17 @@ pub(crate) fn examine_human_report( let mut output = render_examine_plain_header(&summary); output.push_str(&summary.render_text()); append_examine_runtime_state(&mut output, paths, config)?; - append_examine_engine_inventory(&mut output, paths, config, &summary.default_engine); + let gpu_support_level = rocm_core::gpu_support_level( + summary.detected_gfx_target.as_deref(), + summary.compatible_therock_family.as_deref(), + ); + append_examine_engine_inventory( + &mut output, + paths, + config, + &summary.default_engine, + gpu_support_level, + ); Ok((output, summary)) } @@ -190,7 +202,27 @@ pub(crate) fn render_engine_inventory_text_with_paths(paths: Option<&AppPaths>) // Mark the engine this GPU actually serves on as primary. Using the platform // constant put the `*` on Lemonade even on Instinct, where `serve` picks vLLM. let host_gpu = rocm_core::detect_host_gpu_summary(paths); - let host_default_engine = rocm_core::default_engine_for_host(&host_gpu); + render_engine_inventory_text_for_host( + paths, + rocm_core::default_engine_for_host(&host_gpu), + rocm_core::gpu_support_level_for_host_gpu(&host_gpu), + ) +} + +/// `rocm engines` for a host whose preferred engine and GPU support level the +/// caller has already probed. +/// +/// Split from [`render_engine_inventory_text_with_paths`] for the same reason +/// [`append_examine_engine_inventory`] takes both as parameters: the host probe +/// reads real sysfs, so a test cannot otherwise reach the Vulkan-only branch - +/// no CI runner has a GPU that is detected *and* confirmed to have no ROCm +/// build. Both values come from one `detect_host_gpu_summary` call above, so +/// the marker and the note still cannot disagree about the same host. +fn render_engine_inventory_text_for_host( + paths: Option<&AppPaths>, + host_default_engine: &str, + gpu_support_level: rocm_core::GpuSupportLevel, +) -> String { // A configured `default_engine` still wins over the host preference, mirroring // `select_serve_engine` (and `append_examine_engine_inventory`). Without this, // the legend could mark an engine `serve` will not actually default to. @@ -218,7 +250,7 @@ pub(crate) fn render_engine_inventory_text_with_paths(paths: Option<&AppPaths>) output, " Built-in engines are included with rocm-cli. External plugins are optional." ); - let _ = writeln!(output, " ROCm GPU execution is required."); + let _ = writeln!(output, "{}", gpu_execution_line(gpu_support_level)); if let Some(paths) = paths { let _ = writeln!(output, " Plugin folders:"); for (index, path) in engine_plugin_dirs(paths).iter().enumerate() { @@ -233,6 +265,7 @@ pub(crate) fn render_engine_inventory_text_with_paths(paths: Option<&AppPaths>) } for (name, note) in engine_inventory() { let marker = (*name == default_engine).then_some(DEFAULT_ENGINE_MARKER); + let note = engine_inventory_note(name, note, gpu_support_level); write_marked_row(&mut output, "", marker, format_args!("{name:10} {note}")); append_engine_detect_summary(&mut output, name, paths); } @@ -244,6 +277,43 @@ pub(crate) fn render_engine_inventory_text_with_paths(paths: Option<&AppPaths>) output } +/// The `rocm engines` banner stating what these engines will and will not run +/// on, for a host at `level`. +/// +/// This line used to be the constant `" ROCm GPU execution is required."`, +/// printed a few rows above the per-engine notes. That was consistent while +/// lemonade's note was the same static ROCm-only sentence, and stopped being +/// consistent the moment [`crate::engine_inventory_note`] made that note +/// host-aware: on a `VulkanOnly` host the banner asserted ROCm was required +/// directly above a row saying this GPU has no ROCm support — reinstating, on +/// the same screen, the stale claim ROCMAI-550 exists to remove. +/// +/// What is actually invariant across every level is the *refusal*, not the +/// backend: rocm-cli serves on a GPU and never silently falls back to CPU, and +/// Vulkan is one of the GPU paths it serves on rather than a fallback from +/// one. So "GPU execution is required" is said at every level and the ROCm +/// half is said only where the classifier establishes it. `Unknown` drops it +/// too, and that is the point of preferring this over a `VulkanOnly`-only +/// special case: "ROCm is required" on a host whose support level is unknown +/// is the same unfounded claim, just on the hosts the CLI knows least about. +/// `Unknown` does not re-name `rocm examine` either — lemonade's own row +/// already does, two lines below. +/// +/// Matched exhaustively, like the notes it has to agree with, so a fourth +/// support level is a compile error here instead of silently inheriting a +/// claim about ROCm. +const fn gpu_execution_line(level: rocm_core::GpuSupportLevel) -> &'static str { + match level { + rocm_core::GpuSupportLevel::RocmHip => " ROCm GPU execution is required.", + rocm_core::GpuSupportLevel::VulkanOnly => { + " GPU execution is required; on this GPU that is the Vulkan backend, not ROCm." + } + rocm_core::GpuSupportLevel::Unknown => { + " GPU execution is required; CPU execution is not supported." + } + } +} + pub(crate) fn append_examine_runtime_state( output: &mut String, paths: &AppPaths, @@ -363,6 +433,7 @@ fn append_examine_engine_inventory( paths: &AppPaths, config: &RocmCliConfig, host_default_engine: &str, + gpu_support_level: rocm_core::GpuSupportLevel, ) { // Mirrors `select_serve_engine`'s guard: a blank configured value must not // out-rank the host preference, or `effective_default` below would become @@ -445,6 +516,7 @@ fn append_examine_engine_inventory( .or(entry.last_installed_env_id.as_deref()) }) .unwrap_or(""); + let note = engine_inventory_note(engine, note, gpu_support_level); write_marked_row( output, " ", @@ -738,7 +810,13 @@ mod tests { assert!(rendered.contains("Local model engines")); assert!(rendered.contains("Built-in engines are included with rocm-cli")); - assert!(rendered.contains("ROCm GPU execution is required")); + // Not `"ROCm GPU execution is required"`: this renderer probes the real + // host, so the ROCm half of that sentence is now whatever this runner's + // `GpuSupportLevel` is (see `gpu_execution_line`). The part that holds + // on every runner — and the part this test is about — is that engines + // need a GPU at all. Which backend is claimed for which level is pinned + // by `engines_banner_and_engine_note_cannot_contradict_each_other`. + assert!(rendered.contains("GPU execution is required")); assert!(rendered.contains("Plugin folders:")); assert!(rendered.contains(&paths.primary_engine_plugin_dir().display().to_string())); } @@ -856,7 +934,13 @@ mod tests { }; let mut output = String::new(); - append_examine_engine_inventory(&mut output, &paths, &config, "vllm"); + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "vllm", + rocm_core::GpuSupportLevel::Unknown, + ); assert!( output.contains("configured_default_engine: "), @@ -1190,7 +1274,13 @@ mod tests { // Host default deliberately disagrees with the configured value: an // engine the user chose must outrank whatever the GPU would prefer. - append_examine_engine_inventory(&mut output, &paths, &config, "lemonade"); + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "lemonade", + rocm_core::GpuSupportLevel::Unknown, + ); assert!(output.contains("engine_inventory:")); assert!(output.contains("configured_default_engine: vllm")); @@ -1207,7 +1297,13 @@ mod tests { let config = RocmCliConfig::default(); let mut output = String::new(); - append_examine_engine_inventory(&mut output, &paths, &config, "vllm"); + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "vllm", + rocm_core::GpuSupportLevel::Unknown, + ); let _ = fs::remove_dir_all(root); let legend = format!("legend: {DEFAULT_ENGINE_MARKER} = default engine"); @@ -1227,7 +1323,13 @@ mod tests { let config = RocmCliConfig::default(); let mut output = String::new(); - append_examine_engine_inventory(&mut output, &paths, &config, "vllm"); + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "vllm", + rocm_core::GpuSupportLevel::RocmHip, + ); let _ = fs::remove_dir_all(root); for (engine, _) in engine_inventory() { @@ -1255,7 +1357,13 @@ mod tests { }; let mut output = String::new(); - append_examine_engine_inventory(&mut output, &paths, &config, "vllm"); + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "vllm", + rocm_core::GpuSupportLevel::Unknown, + ); let _ = fs::remove_dir_all(root); assert!( @@ -1275,7 +1383,13 @@ mod tests { let config = RocmCliConfig::default(); let mut output = String::new(); - append_examine_engine_inventory(&mut output, &paths, &config, "vllm"); + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "vllm", + rocm_core::GpuSupportLevel::Unknown, + ); // The configured line still reads as unset — the host default must not be // mistaken for something the user chose. @@ -1307,7 +1421,13 @@ mod tests { }; let mut output = String::new(); - append_examine_engine_inventory(&mut output, &paths, &config, "vllm"); + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "vllm", + rocm_core::GpuSupportLevel::Unknown, + ); assert!( output.contains("effective_default_engine: lemonade"), @@ -1341,7 +1461,13 @@ mod tests { }; let mut output = String::new(); - append_examine_engine_inventory(&mut output, &paths, &config, "vllm"); + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "vllm", + rocm_core::GpuSupportLevel::Unknown, + ); assert!( !output.contains("configured_default_note"), @@ -1349,4 +1475,187 @@ mod tests { ); let _ = fs::remove_dir_all(root); } + + #[test] + fn examine_engine_inventory_names_vulkan_for_a_gpu_with_no_rocm_support() { + // A gfx90c host (or any confirmed-no-ROCm-build GPU) actually runs + // Lemonade's Vulkan backend, not the ROCm one the static catalog text + // always claimed. + let (root, paths) = test_paths("examine-engine-inventory-vulkan-only"); + let config = RocmCliConfig::default(); + let mut output = String::new(); + + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "lemonade", + rocm_core::GpuSupportLevel::VulkanOnly, + ); + + assert!( + output.contains("no ROCm/HIP support") && output.contains("Vulkan llama.cpp backend"), + "{output}" + ); + assert!( + !output.contains("note=default embedded Lemonade server with ROCm"), + "{output}" + ); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn examine_engine_inventory_keeps_rocm_note_for_a_rocm_capable_gpu() { + let (root, paths) = test_paths("examine-engine-inventory-rocm-capable"); + let config = RocmCliConfig::default(); + let mut output = String::new(); + + append_examine_engine_inventory( + &mut output, + &paths, + &config, + "lemonade", + rocm_core::GpuSupportLevel::RocmHip, + ); + + assert!( + output.contains("note=default embedded Lemonade server with ROCm llama.cpp backend"), + "{output}" + ); + let _ = fs::remove_dir_all(root); + } + + /// `rocm engines` makes the same claim `rocm examine` does, through a + /// different renderer, so it needs the same pair of tests: + /// `examine_engine_inventory_names_vulkan_for_a_gpu_with_no_rocm_support` + /// is this test's counterpart on the examine surface. + #[test] + fn engines_inventory_names_vulkan_for_a_gpu_with_no_rocm_support() { + let (root, paths) = test_paths("engines-inventory-vulkan-only"); + + let output = render_engine_inventory_text_for_host( + Some(&paths), + "lemonade", + rocm_core::GpuSupportLevel::VulkanOnly, + ); + + assert!( + output.contains("no ROCm/HIP support") && output.contains("Vulkan llama.cpp backend"), + "`rocm engines` must name the backend this GPU really runs on:\n{output}" + ); + assert!( + !output.contains("Lemonade server with ROCm llama.cpp backend"), + "`rocm engines` must not keep asserting ROCm support on a host without it:\n{output}" + ); + // The substitution is lemonade's alone; vLLM's row is not host-aware. + assert!( + output.contains("Linux/WSL ROCm GPU serving engine through external vLLM"), + "only lemonade's note is support-level aware:\n{output}" + ); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn engines_inventory_keeps_rocm_note_for_a_rocm_capable_gpu() { + let (root, paths) = test_paths("engines-inventory-rocm-capable"); + + let output = render_engine_inventory_text_for_host( + Some(&paths), + "vllm", + rocm_core::GpuSupportLevel::RocmHip, + ); + + assert!( + output.contains("default embedded Lemonade server with ROCm llama.cpp backend"), + "a ROCm-capable host must keep the ROCm wording:\n{output}" + ); + assert!( + !output.contains("Vulkan llama.cpp backend"), + "a ROCm-capable host must not be told it is on Vulkan:\n{output}" + ); + let _ = fs::remove_dir_all(root); + } + + /// An unrecognized or undetected GPU is a gap in what the catalog knows, + /// not a confirmed absence of ROCm - printing the Vulkan wording for it + /// would turn that gap into a false claim about the hardware, which is the + /// distinction `GpuSupportLevel::Unknown` exists to preserve. It must not + /// get the *ROCm* wording either: that was the original bug, reappearing + /// on precisely the hosts where support is least known. + #[test] + fn engines_inventory_does_not_claim_vulkan_for_an_unrecognized_gpu() { + let (root, paths) = test_paths("engines-inventory-unknown"); + + let output = render_engine_inventory_text_for_host( + Some(&paths), + "lemonade", + rocm_core::GpuSupportLevel::Unknown, + ); + + assert!( + !output.contains("no ROCm/HIP support"), + "an unrecognized GPU must not be reported as confirmed ROCm-less:\n{output}" + ); + assert!( + !output.contains("Lemonade server with ROCm llama.cpp backend"), + "an unrecognized GPU must not be told it has ROCm support either:\n{output}" + ); + assert!( + output.contains("rocm examine"), + "claiming no backend still has to leave a way to find out:\n{output}" + ); + let _ = fs::remove_dir_all(root); + } + + /// `rocm engines` must not assert and deny ROCm support on one screen. + /// + /// The three tests above each check one half of the page — the per-engine + /// note — and would all pass with the banner above it flatly contradicting + /// them, which is exactly the state this test was added for: the banner was + /// the constant `"ROCm GPU execution is required."` printed a few lines + /// over "this GPU has no ROCm/HIP support". + /// + /// Stated as a relation between the two lines rather than as expected text + /// per level, so it keeps holding if either sentence is reworded: the claim + /// under test is that the banner requires ROCm on exactly the hosts whose + /// row says they have it. That covers every level and not just the + /// `VulkanOnly` one — `Unknown` must not be told ROCm is required either, + /// when its own row says the support level is not yet known. + #[test] + fn engines_banner_and_engine_note_cannot_contradict_each_other() { + let (root, paths) = test_paths("engines-banner-vs-note"); + + for level in [ + rocm_core::GpuSupportLevel::RocmHip, + rocm_core::GpuSupportLevel::VulkanOnly, + rocm_core::GpuSupportLevel::Unknown, + ] { + let output = render_engine_inventory_text_for_host(Some(&paths), "lemonade", level); + + // The invariant that survives every level: these engines serve on a + // GPU, never on the CPU. Dropping the banner entirely would make the + // contradiction assertions below vacuously true. + assert!( + output.contains("GPU execution is required"), + "{level:?}: the GPU requirement must still be stated:\n{output}" + ); + + let banner_requires_rocm = output.contains("ROCm GPU execution is required"); + let note_claims_rocm = + output.contains("default embedded Lemonade server with ROCm llama.cpp backend"); + assert_eq!( + banner_requires_rocm, + matches!(level, rocm_core::GpuSupportLevel::RocmHip), + "{level:?}: the banner may require ROCm only where the classifier \ + establishes it:\n{output}" + ); + assert_eq!( + banner_requires_rocm, note_claims_rocm, + "{level:?}: the banner and lemonade's row must make the same ROCm claim \ + about the same host:\n{output}" + ); + } + + let _ = fs::remove_dir_all(root); + } } diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index 5d385cf23..20e2e64ce 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -66,6 +66,7 @@ mod serve_summary; mod service_control; mod service_hints; mod service_launch; +mod service_log_follow; mod service_records; mod services_cmd; mod services_text; @@ -169,13 +170,14 @@ use crate::model_fit::{ use crate::runtimes_cmd::{current_runtime_manifest, runtimes}; use crate::serve_cmd::{ServeArgs, serve}; use crate::service_control::run_internal_sandbox_tool; +use crate::service_log_follow::follow_service_logs; // Reached only as `super::run_approved_service_action` from // `advised_commands.rs`'s test, so a non-test build does not carry it. #[cfg(test)] use crate::services_cmd::run_approved_service_action; use crate::services_cmd::services; use crate::services_text::{ - follow_service_logs, render_internal_status_text, render_logs_browser_text, render_logs_text, + render_internal_status_text, render_logs_browser_text, render_logs_text, render_service_logs_text, }; use crate::setup_cmd::{render_setup_status_text, setup}; @@ -1024,6 +1026,85 @@ fn strip_subcommands(cmd: clap::Command) -> clap::Command { bare } +/// The inventory note to print for `engine`, given the row `catalog_note` the +/// static catalogue carries for it and what this host actually supports. +/// +/// The one place the lemonade substitution lives. All three host-aware +/// inventory renderers — [`render_engine_inventory_text_for_host`] +/// (`rocm engines`), [`append_examine_engine_inventory`] (`rocm examine`) and +/// [`builtin_codex_bridge_engine_inventory`] (the codex bridge and its +/// `engines` MCP tool) — route their note through this, so the three cannot +/// drift into disagreeing about the same host. Tests exercise the renderers +/// rather than this function: asserting on it directly would only restate the +/// condition below instead of proving a surface prints it. +pub(crate) fn engine_inventory_note( + engine: &str, + catalog_note: &'static str, + gpu_support_level: rocm_core::GpuSupportLevel, +) -> &'static str { + if engine == "lemonade" { + lemonade_engine_note(gpu_support_level) + } else { + catalog_note + } +} + +/// The lemonade engine note shown in `rocm examine`, `rocm engines` and the +/// codex-bridge engine inventory, adjusted for whether this host's detected GPU +/// can actually use the ROCm llama.cpp backend. +/// +/// All three of those have a live `GpuSupportLevel` in scope. The two places +/// that still print the note statically — the dashboard TUI's engine-manager +/// pane and `rocmd`'s own `rocmd_engine_inventory` in +/// `apps/rocmd/src/host_snapshot.rs` — keep the +/// unconditional ROCm wording, and that is deliberate rather than an oversight: +/// the TUI pane renders a hard-coded descriptive list with no probe behind it, +/// and `rocmd` is the daemon answering for whichever host a client asks about, +/// not necessarily the one it runs on. Both are their own change if they ever +/// need to be host-aware. +/// +/// The match is exhaustive rather than a catch-all on purpose. `Unknown` is +/// "no GPU detected yet, or a gfx target this catalog does not recognize", and +/// folding it into the ROCm arm reproduced the exact bug ROCMAI-550 exists to +/// fix - asserting ROCm support on a host where nothing establishes it, just +/// on the hosts where the CLI knows least. It does not get the Vulkan wording +/// either: that is a confirmed absence, and a detection gap is not one. So it +/// claims no backend and names the command that can answer. A new +/// `GpuSupportLevel` variant now has to be classified here rather than +/// silently inheriting the ROCm claim. +/// +/// What the `Unknown` arm promises is bounded by what the named command +/// delivers. It used to say `rocm examine` would show "which llama.cpp backend +/// this GPU uses", and examine showed no such thing — it printed a +/// `rocm_support` line only for `VulkanOnly`, so on exactly the hosts that got +/// this note it printed nothing at all. `ExamineSummary::rocm_support_line` +/// now answers for every level, and this says what that answer is: the support +/// level, not the backend. The backend is settled by launching one, which that +/// line in turn points at. +const fn lemonade_engine_note(level: rocm_core::GpuSupportLevel) -> &'static str { + match level { + rocm_core::GpuSupportLevel::VulkanOnly => { + "default embedded Lemonade server; this GPU has no ROCm/HIP support, so it runs on the Vulkan llama.cpp backend" + } + rocm_core::GpuSupportLevel::RocmHip => { + "default embedded Lemonade server with ROCm llama.cpp backend" + } + rocm_core::GpuSupportLevel::Unknown => { + "default embedded Lemonade server; run `rocm examine` for this host's GPU support level, which says how to find the backend" + } + } +} + +/// This host's GPU support level from the cheap host probe, for the call sites +/// that need it without gathering a whole [`ExamineSummary`]. +/// +/// The same two inputs `rocm examine` derives it from, read through +/// `detect_host_gpu_summary` rather than the full examine gather — exactly what +/// `render_engine_inventory_text_with_paths` (i.e. `rocm engines`) already does. +pub(crate) fn host_gpu_support_level(paths: Option<&AppPaths>) -> rocm_core::GpuSupportLevel { + rocm_core::gpu_support_level_for_host_gpu(&rocm_core::detect_host_gpu_summary(paths)) +} + fn refresh_startup_update_check_quietly() { let Ok(paths) = AppPaths::discover() else { return; diff --git a/apps/rocm/src/providers.rs b/apps/rocm/src/providers.rs index f3b9fd7e6..0656df470 100644 --- a/apps/rocm/src/providers.rs +++ b/apps/rocm/src/providers.rs @@ -1932,6 +1932,7 @@ mod tests { anthropic_endpoint: None, service_id: "svc-ready".to_owned(), status: "ready".to_owned(), + device: None, already_running: false, metrics, smoke_failure, diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index f359699db..93d47d515 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -55,7 +55,7 @@ use rocm_engine_protocol::{ use crate::DevicePolicyArg; use crate::endpoint_keys; use crate::engine_install_env::engine_manages_own_runtime; -use crate::engine_ipc::engine_request; +use crate::engine_ipc::{engine_request, fetch_service_device_briefly}; use crate::engine_runtime_select::{ EngineSelection, resolve_engine_selection, validate_engine_selection_runtime, }; @@ -456,6 +456,32 @@ fn reuse_pregate_opens( .flatten() } +/// Whether the deployment summary may spend a device probe, given what the +/// launch report says happened. +/// +/// `!already_running` is the load-bearing half, not a cosmetic echo of the +/// smoke test's gate. A service that was already running is `"ready"` by +/// definition, so keying on the status alone opens this on *every* repeat +/// `rocm serve` of a live model — and the probe is not free: on a miss it costs +/// `engine_ipc::SERVE_DEVICE_PROBE_BUDGET` plus one in-flight probe, about 14s +/// with Lemonade's 8s inference probe, paid sequentially before the summary's +/// remaining waits. That is a long stall bought for a row this command cannot +/// learn anything new for — the engine answered the same thing when the service +/// was launched, and a *running* service's device is what `rocm services list` +/// reads, with its own budget for exactly that. +/// +/// `probe` is a closure rather than a value for that reason: on the reuse path +/// the cost must not be paid at all, which is what the unit test pins. +fn serve_device_probe_opens( + already_running: bool, + status: &str, + probe: impl FnOnce() -> Option, +) -> Option { + (!already_running && status == "ready") + .then(probe) + .flatten() +} + /// The no-usable-GPU bail's host precondition, read from the visibility-resolved /// set of usable AMD GPU ordinals. /// @@ -1252,6 +1278,25 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { }; spinner.clear(); + // Read what the engine actually selected rather than guessing from + // the pre-launch GPU plan: `gpu_indices` above can be empty for a + // Vulkan-only GPU that ROCm/amd-smi never enumerates, even though + // the server is running fine on it. + // + // Only for a launch this command performed — see + // [`serve_device_probe_opens`] for why skipping the reuse path is + // the point rather than tidiness. + // + // Asked more than once, briefly: `engines/lemonade` answers + // `"unknown"` until its own inference probe has come back, which is + // a real window right after a launch reports ready. The wait is + // bounded (`engine_ipc::SERVE_DEVICE_PROBE_BUDGET`) and `None` is a + // supported outcome — the summary then has no `device:` row, and + // `rocm services list` shows it once the engine can say. + let device = serve_device_probe_opens(report.already_running, &report.status, || { + fetch_service_device_briefly(&paths, &selected_engine, &report.service_id) + }); + let mut notes = collect_serve_notes( cpu_only, &gpu_selection, @@ -1316,6 +1361,7 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { anthropic_endpoint, service_id: report.service_id.clone(), status: report.status.clone(), + device, already_running: report.already_running, metrics, smoke_failure, @@ -1481,6 +1527,63 @@ mod tests { ); } + /// Regression coverage for the device probe's gate (ROCMAI-550 review). + /// + /// The gate shipped as `report.status == "ready"` alone, under a comment + /// claiming it was "the same gate as the smoke test" — which it was not: the + /// smoke test and the Anthropic-endpoint wait beside it both also require + /// `!already_running`. A reused service is `"ready"` by definition, so the + /// omission made every repeat `rocm serve` of a live model pay the probe's + /// full retry budget (plus one in-flight probe, ~14s with Lemonade) before + /// printing a summary of a launch that did not happen. + /// + /// The probe is asserted *not consulted*, not merely that the row is absent: + /// the cost is the bug, and a gate that probed and then discarded the answer + /// would satisfy a value-only assertion. + #[test] + fn the_device_probe_is_not_spent_on_a_service_that_was_already_running() { + let mut probed = false; + assert!( + serve_device_probe_opens(true, "ready", || { + probed = true; + Some("gpu") + }) + .is_none(), + "a reused service must not open the device probe: it is `ready` by \ + definition, so the budget would be spent on every repeat `serve`" + ); + assert!( + !probed, + "the probe must not even be called on the reuse path — the wall time \ + is the regression, not the `device:` row" + ); + + // A launch that is not ready has nothing to ask the engine about yet. + let mut probed_starting = false; + assert!( + serve_device_probe_opens(false, "starting", || { + probed_starting = true; + Some("gpu") + }) + .is_none() + ); + assert!(!probed_starting); + + // The row itself still works for the case it was added for: a fresh + // launch that reached ready, where the engine knows what it selected. + assert_eq!( + serve_device_probe_opens(false, "ready", || Some("gpu")), + Some("gpu"), + "a fresh ready launch must still read the engine's device, or the \ + Vulkan-only `device:` row this PR adds is gone" + ); + // `None` from the engine stays a supported answer: no row, no error. + assert_eq!( + serve_device_probe_opens(false, "ready", || Option::<&str>::None), + None + ); + } + #[test] fn serve_engine_selection_uses_shared_recipe_when_no_override_exists() { let recipe = resolve_builtin_model_recipe("qwen32b").expect("qwen32b recipe"); diff --git a/apps/rocm/src/serve_summary.rs b/apps/rocm/src/serve_summary.rs index c30bcc654..92998faf6 100644 --- a/apps/rocm/src/serve_summary.rs +++ b/apps/rocm/src/serve_summary.rs @@ -61,6 +61,21 @@ pub(crate) struct DeploymentSummary { pub service_id: String, /// `"ready"`, `"starting"`, or an existing service's status. pub status: String, + /// The backend/device the engine actually selected (e.g. `"vulkan gpu 0"`, + /// `"rocm gpu 0 (AMD Instinct MI300X)"`), read from the engine's own + /// `Healthcheck` response. + /// + /// **`None` is a supported outcome, and renders no `device:` row at all.** + /// A launch is `ready` once its server answers, but an engine can still be + /// unable to say which device it is on: `engines/lemonade` reports + /// `"unknown"` until its own inference probe returns, which for a model + /// still loading its weights can be minutes away. `rocm serve` retries + /// within `engine_ipc::SERVE_DEVICE_PROBE_BUDGET` and then prints the + /// summary without the row rather than making the user wait for it; + /// `rocm services list` shows it as soon as the engine can answer. So + /// neither a caller nor a test may read a missing row as a failure to + /// select a device. + pub device: Option, /// True when an equivalent server was already running and nothing was spawned. pub already_running: bool, pub metrics: SmokeMetrics, @@ -199,6 +214,11 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { ] }; + let device_row = summary + .device + .as_deref() + .map(|device| ("device", device.to_owned())); + let anthropic_rows: Vec<(&str, String)> = summary .anthropic_endpoint .iter() @@ -216,6 +236,7 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { ), ("engine", summary.engine.clone()), ], + device_row.into_iter().collect(), model_rows, vec![("endpoint", summary.chat_endpoint.clone())], anthropic_rows, @@ -556,6 +577,22 @@ mod tests { use super::*; use std::fs; + /// A rendered summary with the api key masked, for panic messages. + /// + /// `render_summary` prints the key by design, and a failing assertion that + /// interpolated the whole summary would copy it into CI logs, which are + /// retained and widely readable. Assert on the real text; print this. + fn redacted(rendered: &str) -> String { + rendered + .lines() + .map(|line| match line.split_once("api key: ") { + Some((prefix, _)) => format!("{prefix}api key: "), + None => line.to_owned(), + }) + .collect::>() + .join("\n") + } + fn base_summary() -> DeploymentSummary { DeploymentSummary { engine: "vllm".to_owned(), @@ -565,6 +602,7 @@ mod tests { anthropic_endpoint: None, service_id: "vllm-qwen-1720000000".to_owned(), status: "ready".to_owned(), + device: None, already_running: false, metrics: SmokeMetrics { ttft: Some(Duration::from_millis(180)), @@ -694,17 +732,19 @@ mod tests { line.trim_start().starts_with(label) && line.trim_end().ends_with(value) }) }; - assert!(row("requested model", "qwen"), "{rendered}"); + assert!(row("requested model", "qwen"), "{}", redacted(&rendered)); assert!( row("resolved model", "Qwen3-4B-Instruct-2507-GGUF"), - "{rendered}" + "{}", + redacted(&rendered) ); // The bare `model` row is not used when the two differ. assert!( !rendered .lines() .any(|line| line.trim_start().starts_with("model ")), - "{rendered}" + "{}", + redacted(&rendered) ); } @@ -723,7 +763,11 @@ mod tests { let mut summary = base_summary(); summary.status = "starting".to_owned(); let rendered = render_summary(&summary); - assert!(rendered.contains("not ready yet"), "heading: {rendered}"); + assert!( + rendered.contains("not ready yet"), + "heading: {}", + redacted(&rendered) + ); assert!(rendered.contains("did not report ready")); assert!(rendered.contains("rocm logs --service")); } @@ -738,14 +782,20 @@ mod tests { summary.status = "running".to_owned(); summary.metrics = SmokeMetrics::default(); let rendered = render_summary(&summary); - assert!(rendered.contains("not ready yet"), "heading: {rendered}"); + assert!( + rendered.contains("not ready yet"), + "heading: {}", + redacted(&rendered) + ); assert!( rendered.contains("no smoke test was run"), - "the skipped smoke test must be explained: {rendered}" + "the skipped smoke test must be explained: {}", + redacted(&rendered) ); assert!( rendered.contains("still loading"), - "the user needs to know this resolves on its own: {rendered}" + "the user needs to know this resolves on its own: {}", + redacted(&rendered) ); assert!(rendered.contains("rocm services list")); } @@ -821,6 +871,28 @@ mod tests { ); } + #[test] + fn device_row_shown_only_when_known() { + let without_device = render_summary(&base_summary()); + assert!( + !without_device.contains("device"), + "{}", + redacted(&without_device) + ); + + let mut summary = base_summary(); + summary.device = Some("vulkan gpu 0".to_owned()); + let rendered = render_summary(&summary); + assert!( + rendered + .lines() + .any(|line| line.trim_start().starts_with("device") + && line.trim_end().ends_with("vulkan gpu 0")), + "{}", + redacted(&rendered) + ); + } + #[test] fn oom_note_is_withheld_for_a_ready_serve_or_a_clean_log() { // A serve that became ready is healthy even if the log mentions memory. @@ -1008,14 +1080,22 @@ mod tests { fn api_key_client_config_shown_only_when_present() { // Loopback / no key: the summary must not mention an api key at all. let plain = render_summary(&base_summary()); - assert!(!plain.contains("api key"), "{plain}"); + assert!(!plain.contains("api key"), "{}", redacted(&plain)); // Public bind: the key is shown once as secure client config. let mut summary = base_summary(); summary.api_key = Some("endpoint-secret".to_owned()); let rendered = render_summary(&summary); - assert!(rendered.contains("api key: endpoint-secret"), "{rendered}"); - assert!(rendered.contains("shown only now"), "{rendered}"); + assert!( + rendered.contains("api key: endpoint-secret"), + "{}", + redacted(&rendered) + ); + assert!( + rendered.contains("shown only now"), + "{}", + redacted(&rendered) + ); } #[test] diff --git a/apps/rocm/src/service_log_follow.rs b/apps/rocm/src/service_log_follow.rs new file mode 100644 index 000000000..8e981bc04 --- /dev/null +++ b/apps/rocm/src/service_log_follow.rs @@ -0,0 +1,1265 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The `--follow` tail behind `rocm logs --service --follow` and +//! `rocm services logs --follow`. +//! +//! Split out of [`services_text.rs`](../services_text.rs) as a pure move when +//! that module outgrew the file-size limit (see `docs/architecture.md`); this +//! is the same code, not a rewrite. It is new code to begin with, not one of +//! the clusters mechanically relocated from `main.rs`. +//! +//! This module owns `follow_service_logs` and the seams its tests drive +//! (`follow_service_logs_with`, `follow_service_logs_via`, `follow_loop`), the +//! growing-file cursor and the pieces it yields (`FollowCursor`, `LogSegment`, +//! `write_log_segments`), the terminal footer that keeps the "how to stop" +//! hint under the newest output (`FollowFooter`, `follow_hint`, +//! `terminal_cols`), and the two small helpers the loop uses to say something +//! of its own (`newly_ready`, `write_note`). +//! +//! A `FollowCursor` follows a growing file; it is not `rocm_core::LogTail`, +//! the result of a bounded tail read, which is what the log *views* in +//! [`services_text.rs`](../services_text.rs) use. +//! +//! Only `follow_service_logs` is `pub(crate)`: `dispatch()` calls it for `rocm +//! logs --service --follow` and `services_cmd.rs` calls it for `rocm +//! services logs --follow`. Everything else here has no caller outside +//! this module and stays private. +//! +//! The reads it needs stay in [`services_text.rs`](../services_text.rs) and are +//! called from here: `load_managed_service` for the liveness re-read and +//! `render_service_logs_text` for the snapshot printed before the follow +//! starts. Nothing in `services_text.rs` calls back into this module. +//! +//! The regression tests for everything above moved here with it, reaching +//! shared fixtures (`test_paths`) through `crate::tests::` the same way the +//! other split-out modules do. + +use std::fs; +use std::io::{self, Read as _, Seek as _, SeekFrom}; +use std::path::{Path, PathBuf}; +use std::thread; +use std::time::Duration; + +use anyhow::{Result, bail}; +use rocm_core::{AppPaths, ManagedServiceRecord}; + +use crate::managed_service_status::managed_service_is_live; +use crate::services_text::{load_managed_service, render_service_logs_text}; + +/// How often `--follow` looks for new log output. +const LOG_FOLLOW_POLL: Duration = Duration::from_millis(500); +/// How often `--follow` re-reads the service record to see whether it stopped. +const LOG_FOLLOW_LIVENESS_EVERY: Duration = Duration::from_secs(2); + +/// Reads whatever a growing log file gained since the last [`Self::poll`]. +/// Not `rocm_core::LogTail`, which is what a bounded read of the end of a log +/// returns. +/// +/// Download bars (tqdm, huggingface_hub) redraw in place with `\r`, so a +/// finished segment ends at either `\r` or `\n`; a trailing partial segment is +/// held back until its terminator arrives. +struct FollowCursor { + path: PathBuf, + offset: u64, + pending: Vec, +} + +/// One completed piece of log output and whether it ended in a carriage return. +#[derive(Debug, PartialEq, Eq)] +struct LogSegment { + text: String, + redraw: bool, +} + +impl FollowCursor { + /// Start following at the file's current end. + fn at_end(path: &Path) -> Self { + let offset = fs::metadata(path).map_or(0, |meta| meta.len()); + Self { + path: path.to_path_buf(), + offset, + pending: Vec::new(), + } + } + + fn poll(&mut self) -> Vec { + let Ok(mut file) = fs::File::open(&self.path) else { + return Vec::new(); + }; + let len = file.metadata().map_or(0, |meta| meta.len()); + if len < self.offset { + // Truncated or replaced (a restart reuses the path): start over. + self.offset = 0; + self.pending.clear(); + } + if file.seek(SeekFrom::Start(self.offset)).is_err() { + return Vec::new(); + } + let mut fresh = Vec::new(); + if file.read_to_end(&mut fresh).is_err() { + return Vec::new(); + } + self.offset += fresh.len() as u64; + self.pending.extend_from_slice(&fresh); + + let mut segments = Vec::new(); + let mut start = 0; + let mut index = 0; + while index < self.pending.len() { + let byte = self.pending[index]; + // A `\r` at the very end of the data may be the first half of a `\r\n` + // that the next poll completes; hold it back rather than emit an + // empty segment (which would erase a final redraw frame on a tty). + if byte == b'\r' && index + 1 == self.pending.len() { + break; + } + if byte == b'\n' || byte == b'\r' { + let mut redraw = byte == b'\r'; + // The footer's row bookkeeping assumes the log text moves no cursor. + let text = rocm_core::terminal::strip_terminal_control_sequences( + &String::from_utf8_lossy(&self.pending[start..index]), + ); + // `\r\n` is one line break, not a redraw followed by an empty line. + if redraw && self.pending.get(index + 1) == Some(&b'\n') { + redraw = false; + index += 1; + } + segments.push(LogSegment { text, redraw }); + start = index + 1; + } + index += 1; + } + self.pending.drain(..start); + segments + } + + /// The unterminated output still held back, for when following ends: a + /// last write cut short by a crash is otherwise never shown. + fn finish(&mut self) -> Option { + let mut bytes = std::mem::take(&mut self.pending); + if bytes.last() == Some(&b'\r') { + bytes.pop(); + } + // Same stripping as `poll`: a cut-off write can end mid-escape. + (!bytes.is_empty()).then(|| LogSegment { + text: rocm_core::terminal::strip_terminal_control_sequences(&String::from_utf8_lossy( + &bytes, + )), + redraw: false, + }) + } +} + +/// Writes `segments`; returns whether the last one was a redraw, which leaves the +/// cursor on that row (the next segment erases it first) instead of a fresh one. +fn write_log_segments( + out: &mut impl io::Write, + segments: &[LogSegment], + tty: bool, +) -> Result { + for segment in segments { + match (tty, segment.redraw) { + // Redraw the same terminal row, as the engine's own bar does. + (true, true) => write!(out, "\r\x1b[2K{}", segment.text)?, + (true, false) => writeln!(out, "\r\x1b[2K{}", segment.text)?, + (false, _) => writeln!(out, "{}", segment.text)?, + } + } + out.flush()?; + Ok(tty && segments.last().is_some_and(|segment| segment.redraw)) +} + +/// The "how to stop" hint kept on the row just below the newest log output of a +/// terminal follow, so it stays in view however much the log scrolls. +/// +/// While drawn, the cursor sits on the footer row. [`Self::erase`] puts it back +/// where the log output left off: on a fresh row, or, after a redraw segment +/// (a download bar), at the end of that still-open row, one row up. +struct FollowFooter { + text: String, + drawn: bool, + /// The row above the footer is an unfinished redraw row. + open_row: bool, +} + +/// Led by the part that says the server keeps running: service ids run 40+ +/// characters, so on a narrow terminal the end of the hint is what gets cut. +fn follow_hint(service_id: &str) -> String { + format!("Ctrl-C stops following, the server keeps running. Following {service_id}") +} + +impl FollowFooter { + fn new(service_id: &str) -> Self { + Self { + text: follow_hint(service_id), + drawn: false, + open_row: false, + } + } + + fn erase(&mut self, out: &mut impl io::Write) -> Result<()> { + if self.drawn { + write!(out, "\r\x1b[2K")?; + if self.open_row { + write!(out, "\x1b[1A")?; + } + self.drawn = false; + } + Ok(()) + } + + /// Clear the footer and end an unfinished redraw row, leaving the cursor on + /// a fresh row. + fn finish(&mut self, out: &mut impl io::Write) -> Result<()> { + self.erase(out)?; + if self.open_row { + writeln!(out)?; + self.open_row = false; + } + out.flush()?; + Ok(()) + } + + /// One line of the follow's own, written below the log output with the + /// footer redrawn under it. An unfinished download bar row is ended first, + /// not overwritten. + fn write_notice( + &mut self, + out: &mut impl io::Write, + text: &str, + cols: Option, + ) -> Result<()> { + self.finish(out)?; + writeln!(out, "{text}")?; + self.draw(out, false, cols) + } + + /// `cols` truncates the text so it never wraps onto a second row, which the + /// one-row erase could not clear. + fn draw( + &mut self, + out: &mut impl io::Write, + open_row: bool, + cols: Option, + ) -> Result<()> { + if open_row { + writeln!(out)?; + } + let shown: String = match cols { + Some(cols) => { + crate::cli_progress::truncate_to_width(&self.text, cols.saturating_sub(1)) + } + None => self.text.clone(), + }; + write!(out, "\r\x1b[2K{shown}")?; + out.flush()?; + self.drawn = true; + self.open_row = open_row; + Ok(()) + } + + /// Write a batch of new output with the footer kept below it. A batch with + /// nothing in it leaves the screen alone (no flicker on an idle log). + fn write_batch( + &mut self, + out: &mut impl io::Write, + segments: &[LogSegment], + cols: Option, + ) -> Result<()> { + if segments.is_empty() { + return Ok(()); + } + self.erase(out)?; + let open_row = write_log_segments(out, segments, true)?; + self.draw(out, open_row, cols) + } +} + +fn terminal_cols() -> Option { + crossterm::terminal::size() + .ok() + .map(|(cols, _)| usize::from(cols)) +} + +/// `rocm logs --service --follow`: print the recent output, then keep +/// printing new output until the service is no longer starting or running. +/// Ctrl-C ends the follow and leaves the service running. +pub(crate) fn follow_service_logs(paths: &AppPaths, service_id: &str) -> Result<()> { + use std::io::IsTerminal as _; + let mut out = io::stdout(); + let tty = out.is_terminal(); + follow_service_logs_with( + paths, + service_id, + &mut out, + &mut io::stderr(), + tty, + LOG_FOLLOW_POLL, + LOG_FOLLOW_LIVENESS_EVERY, + &terminal_cols, + ) +} + +/// True once, the first time `status` reads `ready`. +fn newly_ready(announced: &mut bool, status: &str) -> bool { + let ready = status == "ready" && !*announced; + *announced |= ready; + ready +} + +/// A line about the follow itself (the hint, "is ready.", "stopped following."). +/// On a terminal it shares the log's stream and row layout; piped, it goes to +/// `err` so `rocm logs -f | grep error` sees log text only. +fn write_note( + out: &mut impl io::Write, + err: &mut impl io::Write, + tty: bool, + line: &str, +) -> Result<()> { + if tty { + writeln!(out, "{line}")?; + } else { + writeln!(err, "{line}")?; + } + Ok(()) +} + +#[allow(clippy::too_many_arguments)] +fn follow_service_logs_with( + paths: &AppPaths, + service_id: &str, + out: &mut impl io::Write, + err: &mut impl io::Write, + tty: bool, + poll: Duration, + liveness_every: Duration, + cols: &dyn Fn() -> Option, +) -> Result<()> { + follow_service_logs_via( + paths, + service_id, + &|| load_managed_service(paths, service_id), + out, + err, + tty, + poll, + liveness_every, + cols, + ) +} + +/// [`follow_service_logs_with`] with the record read injected: a `ready` status +/// only sticks while the endpoint answers, so a test cannot plant one on disk. +#[allow(clippy::too_many_arguments)] +fn follow_service_logs_via( + paths: &AppPaths, + service_id: &str, + load: &dyn Fn() -> Result, + out: &mut impl io::Write, + err: &mut impl io::Write, + tty: bool, + poll: Duration, + liveness_every: Duration, + cols: &dyn Fn() -> Option, +) -> Result<()> { + let record = load()?; + // Open the tail before the snapshot: output written in between is then shown + // twice (in the snapshot and by the first poll) rather than lost, and the + // tail can start mid-line. + let mut tail = FollowCursor::at_end(&record.log_path); + write!(out, "{}", render_service_logs_text(paths, service_id)?)?; + out.flush()?; + if !managed_service_is_live(&record) { + write_note( + out, + err, + tty, + &format!( + "\nService {service_id} is {}; not following.", + record.status + ), + )?; + // Same outcome as a follow that sees the server fail: not a success. + if record.status == "failed" { + bail!("service {service_id} failed; see the log output above"); + } + return Ok(()); + } + // On a terminal the hint is a footer that stays under the newest output; a + // pipe or log file gets it once, since it has no row to keep it on. + let mut footer = tty.then(|| FollowFooter::new(service_id)); + match footer.as_mut() { + Some(footer) => footer.draw(out, false, cols())?, + None => write_note(out, err, tty, &format!("\n{}", follow_hint(service_id)))?, + } + out.flush()?; + + let result = follow_loop( + load, + &record, + &mut tail, + &mut footer, + out, + err, + tty, + poll, + liveness_every, + cols, + ); + if result.is_err() + && let Some(footer) = footer.as_mut() + { + // Leave the terminal on a clean row so the caller's `Error:` is not glued + // to the footer or to an unfinished download bar. + let _ = footer.finish(out); + } + result +} + +#[allow(clippy::too_many_arguments)] +fn follow_loop( + load: &dyn Fn() -> Result, + record: &ManagedServiceRecord, + tail: &mut FollowCursor, + footer: &mut Option, + out: &mut impl io::Write, + err: &mut impl io::Write, + tty: bool, + poll: Duration, + liveness_every: Duration, + cols: &dyn Fn() -> Option, +) -> Result<()> { + let service_id = record.service_id.as_str(); + let mut announced_ready = record.status == "ready"; + let mut since_check = Duration::ZERO; + loop { + thread::sleep(poll); + match footer.as_mut() { + Some(footer) => footer.write_batch(out, &tail.poll(), cols())?, + None => { + write_log_segments(out, &tail.poll(), tty)?; + } + } + since_check += poll; + if since_check < liveness_every { + continue; + } + since_check = Duration::ZERO; + let record = load()?; + if newly_ready(&mut announced_ready, &record.status) { + let line = format!("Service {service_id} is ready."); + match footer.as_mut() { + Some(footer) => footer.write_notice(out, &line, cols())?, + None => write_note(out, err, tty, &line)?, + } + } + if !managed_service_is_live(&record) { + // Output written between the last poll and the exit. + let mut last = tail.poll(); + last.extend(tail.finish()); + let mut open_row = false; + if let Some(footer) = footer.as_mut() { + footer.erase(out)?; + open_row = footer.open_row; + } + open_row = write_log_segments(out, &last, tty)? || (open_row && last.is_empty()); + if open_row { + writeln!(out)?; + } + write_note( + out, + err, + tty, + &format!( + "Service {service_id} is {}; stopped following.", + record.status + ), + )?; + // Same outcome as `serve --wait`: a server that failed is not a success. + if record.status == "failed" { + bail!("service {service_id} failed; see the log output above"); + } + return Ok(()); + } + } +} + +#[cfg(test)] +mod tests { + use std::thread; + use std::time::Duration; + + use super::*; + use crate::tests::test_paths; + + #[test] + fn log_tail_yields_only_new_complete_segments() -> Result<()> { + let (root, paths) = test_paths("log-tail"); + paths.ensure()?; + let log = root.join("tail.log"); + fs::write(&log, "old line\n")?; + let mut tail = FollowCursor::at_end(&log); + assert!(tail.poll().is_empty()); + + // tqdm-style bar redraws with `\r`; the last piece has no terminator yet. + fs::write(&log, "old line\n 28%|##\r 33%|###\rdone\r\nhalf")?; + let segments = tail.poll(); + assert_eq!( + segments, + vec![ + LogSegment { + text: " 28%|##".into(), + redraw: true + }, + LogSegment { + text: " 33%|###".into(), + redraw: true + }, + LogSegment { + text: "done".into(), + redraw: false + }, + ] + ); + // The held-back partial segment completes on the next poll. + let mut file = fs::OpenOptions::new().append(true).open(&log)?; + std::io::Write::write_all(&mut file, b" line\n")?; + assert_eq!( + tail.poll(), + vec![LogSegment { + text: "half line".into(), + redraw: false + }] + ); + + // A truncated log (restart reusing the path) is read from the start. + fs::write(&log, "new\n")?; + assert_eq!( + tail.poll(), + vec![LogSegment { + text: "new".into(), + redraw: false + }] + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn log_tail_holds_back_a_trailing_cr_and_flushes_the_rest_on_finish() -> Result<()> { + let (root, paths) = test_paths("log-tail-edges"); + paths.ensure()?; + let log = root.join("edges.log"); + fs::write(&log, "")?; + let mut tail = FollowCursor::at_end(&log); + + // A `\r` ending the data may be half of a `\r\n`: it is not a redraw yet. + fs::write(&log, " 99%|#\r")?; + assert!(tail.poll().is_empty()); + let mut file = fs::OpenOptions::new().append(true).open(&log)?; + std::io::Write::write_all(&mut file, b"\n")?; + assert_eq!( + tail.poll(), + vec![LogSegment { + text: " 99%|#".into(), + redraw: false + }], + "the held `\\r` joins the next `\\n` instead of yielding an empty segment" + ); + + // Output cut short before any terminator is shown when following ends. + std::io::Write::write_all(&mut file, b"crashed mid-wri")?; + assert!(tail.poll().is_empty()); + assert_eq!( + tail.finish(), + Some(LogSegment { + text: "crashed mid-wri".into(), + redraw: false + }) + ); + assert_eq!(tail.finish(), None); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn write_log_segments_redraws_in_place_only_on_a_tty() -> Result<()> { + let segments = [ + LogSegment { + text: "28%".into(), + redraw: true, + }, + LogSegment { + text: "done".into(), + redraw: false, + }, + ]; + let mut tty = Vec::new(); + write_log_segments(&mut tty, &segments, true)?; + // A redraw segment leaves the cursor on its row; the next segment (or + // line) erases the row first, and a plain line ends with a newline. + assert_eq!(String::from_utf8(tty)?, "\r\x1b[2K28%\r\x1b[2Kdone\n"); + + let mut piped = Vec::new(); + write_log_segments(&mut piped, &segments, false)?; + assert_eq!(String::from_utf8(piped)?, "28%\ndone\n"); + Ok(()) + } + + fn seg(text: &str, redraw: bool) -> LogSegment { + LogSegment { + text: text.into(), + redraw, + } + } + + const ERASE: &str = "\r\x1b[2K"; + + #[test] + fn follow_footer_is_redrawn_under_each_batch_of_new_lines() -> Result<()> { + let mut footer = FollowFooter::new("svc_a"); + let hint = footer.text.clone(); + let mut out = Vec::new(); + footer.draw(&mut out, false, None)?; + assert_eq!(String::from_utf8(out.clone())?, format!("{ERASE}{hint}")); + + out.clear(); + footer.write_batch(&mut out, &[seg("one", false), seg("two", false)], None)?; + // Footer row cleared, lines written over it, footer redrawn on the next row. + assert_eq!( + String::from_utf8(out)?, + format!("{ERASE}{ERASE}one\n{ERASE}two\n{ERASE}{hint}") + ); + Ok(()) + } + + #[test] + fn follow_footer_leaves_an_idle_screen_alone() -> Result<()> { + let mut footer = FollowFooter::new("svc_a"); + let mut out = Vec::new(); + footer.draw(&mut out, false, None)?; + out.clear(); + footer.write_batch(&mut out, &[], None)?; + assert!(out.is_empty(), "{out:?}"); + Ok(()) + } + + #[test] + fn follow_footer_keeps_a_download_bar_on_its_own_row_above_it() -> Result<()> { + let mut footer = FollowFooter::new("svc_a"); + let hint = footer.text.clone(); + let mut out = Vec::new(); + footer.draw(&mut out, false, None)?; + out.clear(); + + footer.write_batch(&mut out, &[seg("28%", true)], None)?; + // The bar row stays open; the footer goes on the row below it. + assert_eq!( + String::from_utf8(out.clone())?, + format!("{ERASE}{ERASE}28%\n{ERASE}{hint}") + ); + + out.clear(); + footer.write_batch(&mut out, &[seg("43%", true)], None)?; + // Erase the footer, step back up onto the bar row, redraw it in place. + assert_eq!( + String::from_utf8(out.clone())?, + format!("{ERASE}\x1b[1A{ERASE}43%\n{ERASE}{hint}") + ); + + out.clear(); + footer.write_batch(&mut out, &[seg("done", false)], None)?; + // A finished line replaces the bar, and the footer is back on a fresh row. + assert_eq!( + String::from_utf8(out)?, + format!("{ERASE}\x1b[1A{ERASE}done\n{ERASE}{hint}") + ); + Ok(()) + } + + #[test] + fn follow_footer_is_cut_to_one_terminal_row() -> Result<()> { + let mut footer = FollowFooter::new("svc_a"); + let mut out = Vec::new(); + footer.draw(&mut out, false, Some(30))?; + // One column short of the width, so it never wraps, and what is kept is + // the head of the hint: the part saying the server keeps running. + let drawn = String::from_utf8(out)?; + let shown = drawn.strip_prefix(ERASE).expect("erase prefix"); + assert_eq!(unicode_width::UnicodeWidthStr::width(shown), 29, "{shown}"); + assert!(shown.starts_with("Ctrl-C stops following, the"), "{shown}"); + Ok(()) + } + + #[test] + fn log_tail_strips_cursor_escapes_from_engine_output() -> Result<()> { + let (root, _paths) = test_paths("tail-escapes"); + fs::create_dir_all(&root)?; + let log = root.join("engine.log"); + fs::write(&log, "")?; + let mut tail = FollowCursor::at_end(&log); + fs::write(&log, "before\x1b[1Aafter\n")?; + let segments = tail.poll(); + assert_eq!(segments.len(), 1); + assert_eq!(segments[0].text, "beforeafter"); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn log_tail_finish_strips_cursor_escapes_from_a_cut_off_write() -> Result<()> { + let (root, _paths) = test_paths("tail-finish-escapes"); + fs::create_dir_all(&root)?; + let log = root.join("engine.log"); + fs::write(&log, "")?; + let mut tail = FollowCursor::at_end(&log); + fs::write(&log, "last\x1b[1Aword")?; + assert!(tail.poll().is_empty()); + let segment = tail.finish().expect("unterminated output is returned"); + assert_eq!(segment.text, "lastword"); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn follow_service_logs_keeps_the_hint_under_new_output_on_a_tty() -> Result<()> { + let (root, paths) = test_paths("follow-tty"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "svc_follow_tty", + "vllm", + "qwen3.5", + "Qwen/Qwen3.5", + "127.0.0.1", + 11438, + "managed", + std::process::id(), + None, + None, + None, + ); + record.status = "starting".to_owned(); + record.write()?; + fs::write(&record.log_path, "before\n")?; + + let log_path = record.log_path; + let manifest_paths = paths.clone(); + let writer = thread::spawn(move || -> Result<()> { + let appended = (|| -> Result<()> { + thread::sleep(Duration::from_millis(150)); + let mut file = fs::OpenOptions::new().append(true).open(&log_path)?; + std::io::Write::write_all(&mut file, b"Downloading 43%\n")?; + thread::sleep(Duration::from_millis(150)); + Ok(()) + })(); + let mut record = load_managed_service(&manifest_paths, "svc_follow_tty")?; + record.status = "stopped".to_owned(); + record.write()?; + appended + }); + let mut out = Vec::new(); + let mut err = Vec::new(); + follow_service_logs_with( + &paths, + "svc_follow_tty", + &mut out, + &mut err, + true, + Duration::from_millis(20), + Duration::from_millis(40), + &|| None, + )?; + writer.join().expect("writer thread")?; + let out = String::from_utf8(out)?; + let hint = FollowFooter::new("svc_follow_tty").text; + // The hint is drawn again right under the line that scrolled it away, + // and only the stop message follows it, after the footer row is cleared. + assert!( + out.contains(&format!("Downloading 43%\n{ERASE}{hint}")), + "{out:?}" + ); + assert!( + out.ends_with(&format!( + "{hint}{ERASE}Service svc_follow_tty is stopped; stopped following.\n" + )), + "{out:?}" + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn follow_service_logs_ends_an_open_download_bar_row_before_the_stop_message() -> Result<()> { + let (root, paths) = test_paths("follow-tty-bar"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "svc_follow_bar", + "vllm", + "qwen3.5", + "Qwen/Qwen3.5", + "127.0.0.1", + 11440, + "managed", + std::process::id(), + None, + None, + None, + ); + record.status = "starting".to_owned(); + record.write()?; + fs::write(&record.log_path, "")?; + + let log_path = record.log_path; + let manifest_paths = paths.clone(); + let writer = thread::spawn(move || -> Result<()> { + let appended = (|| -> Result<()> { + thread::sleep(Duration::from_millis(150)); + let mut file = fs::OpenOptions::new().append(true).open(&log_path)?; + std::io::Write::write_all(&mut file, b"28%\r43%\r\r")?; + thread::sleep(Duration::from_millis(150)); + Ok(()) + })(); + let mut record = load_managed_service(&manifest_paths, "svc_follow_bar")?; + record.status = "stopped".to_owned(); + record.write()?; + appended + }); + let mut out = Vec::new(); + let mut err = Vec::new(); + follow_service_logs_with( + &paths, + "svc_follow_bar", + &mut out, + &mut err, + true, + Duration::from_millis(20), + Duration::from_millis(40), + &|| None, + )?; + writer.join().expect("writer thread")?; + let out = String::from_utf8(out)?; + // The footer is erased and the cursor goes back up onto the bar's row; that row + // is then ended with a newline so the stop message starts on its own row. + assert!( + out.ends_with(&format!( + "{ERASE}\x1b[1A\nService svc_follow_bar is stopped; stopped following.\n" + )), + "{out:?}" + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn follow_service_logs_draws_no_footer_for_a_stopped_service_on_a_tty() -> Result<()> { + let (root, paths) = test_paths("follow-tty-stopped"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "svc_follow_tty_stopped", + "vllm", + "qwen3.5", + "Qwen/Qwen3.5", + "127.0.0.1", + 11439, + "managed", + 0, + None, + None, + None, + ); + record.status = "stopped".to_owned(); + record.write()?; + let mut out = Vec::new(); + let mut err = Vec::new(); + follow_service_logs_with( + &paths, + "svc_follow_tty_stopped", + &mut out, + &mut err, + true, + Duration::from_millis(1), + Duration::from_millis(1), + &|| None, + )?; + let out = String::from_utf8(out)?; + assert!(!out.contains("Following"), "{out:?}"); + assert!(out.contains("not following."), "{out:?}"); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn follow_service_logs_prints_snapshot_and_returns_for_a_stopped_service() -> Result<()> { + let (root, paths) = test_paths("follow-stopped"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "svc_follow_stopped", + "vllm", + "qwen3.5", + "Qwen/Qwen3.5", + "127.0.0.1", + 11436, + "managed", + 0, + None, + None, + None, + ); + record.status = "stopped".to_owned(); + record.write()?; + fs::write(&record.log_path, "Downloading 86%\n")?; + + let mut out = Vec::new(); + let mut err = Vec::new(); + follow_service_logs_with( + &paths, + "svc_follow_stopped", + &mut out, + &mut err, + false, + Duration::from_millis(1), + Duration::from_millis(1), + &|| None, + )?; + let out = String::from_utf8(out)?; + let err = String::from_utf8(err)?; + assert!(out.contains("Downloading 86%"), "{out}"); + // The notice is not log text, so a pipe keeps it off stdout. + assert!(!out.contains("not following"), "{out}"); + assert!( + err.contains("Service svc_follow_stopped is stopped; not following."), + "{err}" + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn newly_ready_fires_once_on_the_first_ready_status() { + let mut announced = false; + assert!(!newly_ready(&mut announced, "starting")); + assert!(newly_ready(&mut announced, "ready")); + assert!(!newly_ready(&mut announced, "ready")); + // A follow that began on an already-ready server has announced nothing new. + let mut announced = true; + assert!(!newly_ready(&mut announced, "ready")); + } + + #[test] + fn follow_service_logs_fails_when_the_service_failed() -> Result<()> { + let (root, paths) = test_paths("follow-ready-failed"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "svc_follow_fail", + "vllm", + "qwen3.5", + "Qwen/Qwen3.5", + "127.0.0.1", + 11438, + "managed", + std::process::id(), + None, + None, + None, + ); + record.status = "starting".to_owned(); + record.write()?; + fs::write(&record.log_path, "before\n")?; + + let manifest_paths = paths.clone(); + let writer = thread::spawn(move || -> Result<()> { + thread::sleep(Duration::from_millis(200)); + let mut record = load_managed_service(&manifest_paths, "svc_follow_fail")?; + record.status = "failed".to_owned(); + record.write() + }); + let mut out = Vec::new(); + let mut err = Vec::new(); + let result = follow_service_logs_with( + &paths, + "svc_follow_fail", + &mut out, + &mut err, + false, + Duration::from_millis(20), + Duration::from_millis(40), + &|| None, + ); + writer.join().expect("writer thread")?; + let err = String::from_utf8(err)?; + assert!(err.contains("is failed; stopped following"), "{err}"); + let error = result.expect_err("a failed service must not exit 0"); + assert!( + error.to_string().contains("svc_follow_fail failed"), + "{error}" + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn follow_service_logs_streams_new_output_until_the_service_stops() -> Result<()> { + let (root, paths) = test_paths("follow-live"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "svc_follow_live", + "vllm", + "qwen3.5", + "Qwen/Qwen3.5", + "127.0.0.1", + 11437, + "managed", + std::process::id(), + None, + None, + None, + ); + record.status = "starting".to_owned(); + record.write()?; + fs::write(&record.log_path, "before\n")?; + + let log_path = record.log_path; + let manifest_paths = paths.clone(); + let writer = thread::spawn(move || -> Result<()> { + let appended = (|| -> Result<()> { + thread::sleep(Duration::from_millis(150)); + let mut file = fs::OpenOptions::new().append(true).open(&log_path)?; + std::io::Write::write_all(&mut file, b"Downloading 43%\n")?; + thread::sleep(Duration::from_millis(150)); + Ok(()) + })(); + // Stop the service even when the append failed, so the follow loop + // under test always ends instead of hanging the run. + let mut record = load_managed_service(&manifest_paths, "svc_follow_live")?; + record.status = "stopped".to_owned(); + record.write()?; + appended + }); + let mut out = Vec::new(); + let mut err = Vec::new(); + follow_service_logs_with( + &paths, + "svc_follow_live", + &mut out, + &mut err, + false, + Duration::from_millis(20), + Duration::from_millis(40), + &|| None, + )?; + writer.join().expect("writer thread")?; + let out = String::from_utf8(out)?; + let err = String::from_utf8(err)?; + // Piped: stdout is the log text alone, with no terminal escapes, and the + // follow's own lines go to stderr, the hint exactly once. + assert!(out.contains("Downloading 43%"), "{out}"); + assert!(!out.contains('\x1b'), "{out:?}"); + assert!(!out.contains("Following"), "{out}"); + assert!(!out.contains("stopped following"), "{out}"); + assert_eq!(err.matches("Following svc_follow_live").count(), 1, "{err}"); + assert!(err.contains("stopped following"), "{err}"); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + fn starting_record(paths: &AppPaths, id: &str, port: u16) -> Result { + let mut record = ManagedServiceRecord::new( + paths, + id, + "vllm", + "qwen3.5", + "Qwen/Qwen3.5", + "127.0.0.1", + port, + "managed", + std::process::id(), + None, + None, + None, + ); + record.status = "starting".to_owned(); + record.write()?; + Ok(record) + } + + /// A record read that walks through `statuses` (the last one repeats). + fn scripted_statuses( + base: ManagedServiceRecord, + statuses: &'static [&'static str], + ) -> impl Fn() -> Result { + let calls = std::cell::Cell::new(0_usize); + move || { + let call = calls.get(); + calls.set(call + 1); + let mut record = base.clone(); + record.status = statuses[call.min(statuses.len() - 1)].to_owned(); + Ok(record) + } + } + + #[test] + fn follow_service_logs_announces_ready_once_when_the_server_comes_up() -> Result<()> { + let (root, paths) = test_paths("follow-ready-line"); + paths.ensure()?; + let record = starting_record(&paths, "svc_follow_ready", 11441)?; + fs::write(&record.log_path, "before\n")?; + // Initial read, then one read per liveness check. + let load = scripted_statuses( + record, + &["starting", "starting", "ready", "ready", "stopped"], + ); + let mut out = Vec::new(); + let mut err = Vec::new(); + follow_service_logs_via( + &paths, + "svc_follow_ready", + &load, + &mut out, + &mut err, + false, + Duration::from_millis(5), + Duration::from_millis(5), + &|| None, + )?; + let err = String::from_utf8(err)?; + assert_eq!( + err.matches("Service svc_follow_ready is ready.").count(), + 1, + "{err}" + ); + assert!(!String::from_utf8(out)?.contains("is ready.")); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn follow_service_logs_ends_a_download_bar_row_before_the_ready_line_on_a_tty() -> Result<()> { + let (root, paths) = test_paths("follow-ready-bar"); + paths.ensure()?; + let record = starting_record(&paths, "svc_follow_rbar", 11442)?; + fs::write(&record.log_path, "")?; + let log_path = record.log_path.clone(); + let appender = thread::spawn(move || -> Result<()> { + thread::sleep(Duration::from_millis(50)); + let mut file = fs::OpenOptions::new().append(true).open(&log_path)?; + std::io::Write::write_all(&mut file, b"28%\r43%\r")?; + Ok(()) + }); + // The bar is on screen when the status turns ready (liveness check 3 on). + let calls = std::cell::Cell::new(0_usize); + let base = record; + let load = move || -> Result { + let call = calls.get(); + calls.set(call + 1); + if call >= 3 { + thread::sleep(Duration::from_millis(30)); + } + let mut record = base.clone(); + record.status = match call { + 0..=7 => "starting", + 8..=9 => "ready", + _ => "stopped", + } + .to_owned(); + Ok(record) + }; + let mut out = Vec::new(); + let mut err = Vec::new(); + follow_service_logs_via( + &paths, + "svc_follow_rbar", + &load, + &mut out, + &mut err, + true, + Duration::from_millis(10), + Duration::from_millis(10), + &|| None, + )?; + appender.join().expect("appender thread")?; + let out = String::from_utf8(out)?; + // The bar's row is ended with a newline, then the ready line gets its own + // row; it never replaces the bar. + assert!( + out.contains(&format!( + "{ERASE}\x1b[1A\nService svc_follow_rbar is ready.\n" + )), + "{out:?}" + ); + assert!(err.is_empty(), "a terminal follow writes nothing to stderr"); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn follow_service_logs_clears_the_footer_when_the_follow_errors() -> Result<()> { + let (root, paths) = test_paths("follow-error"); + paths.ensure()?; + let record = starting_record(&paths, "svc_follow_gone", 11443)?; + fs::write(&record.log_path, "")?; + let manifest = record.manifest_path; + // The service record disappears mid-follow, so the next liveness read fails. + let remover = thread::spawn(move || -> Result<()> { + thread::sleep(Duration::from_millis(100)); + fs::remove_file(manifest)?; + Ok(()) + }); + let mut out = Vec::new(); + let mut err = Vec::new(); + let result = follow_service_logs_with( + &paths, + "svc_follow_gone", + &mut out, + &mut err, + true, + Duration::from_millis(20), + Duration::from_millis(40), + &|| None, + ); + remover.join().expect("remover thread")?; + assert!(result.is_err(), "a vanished record must end the follow"); + let out = String::from_utf8(out)?; + // The footer row is erased, so `Error:` starts on a clean row. + assert!(out.ends_with(ERASE), "{out:?}"); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn follow_service_logs_fails_for_a_service_that_is_already_failed() -> Result<()> { + let (root, paths) = test_paths("follow-already-failed"); + paths.ensure()?; + let mut record = starting_record(&paths, "svc_follow_dead", 11444)?; + record.status = "failed".to_owned(); + record.supervisor_pid = 0; + record.write()?; + let mut out = Vec::new(); + let mut err = Vec::new(); + let result = follow_service_logs_with( + &paths, + "svc_follow_dead", + &mut out, + &mut err, + false, + Duration::from_millis(1), + Duration::from_millis(1), + &|| None, + ); + let error = result.expect_err("a failed service must not exit 0"); + assert!( + error.to_string().contains("svc_follow_dead failed"), + "{error}" + ); + assert!(String::from_utf8(err)?.contains("is failed; not following.")); + let _ = fs::remove_dir_all(root); + Ok(()) + } +} diff --git a/apps/rocm/src/services_cmd.rs b/apps/rocm/src/services_cmd.rs index f2cbc5262..bf15eb3a9 100644 --- a/apps/rocm/src/services_cmd.rs +++ b/apps/rocm/src/services_cmd.rs @@ -37,13 +37,14 @@ use crate::service_hints::{ self, sandbox_tool_arg_from_service_tool, service_action_command, service_action_past_tense, service_action_verb, }; +use crate::service_log_follow::follow_service_logs; use crate::service_records::{ prune_managed_service_records, remove_managed_service_record_with, service_prune_min_age_hours, }; use crate::services_text::{ - follow_service_logs, render_service_action_result, - render_service_action_result_with_helper_note, render_service_logs_text, render_services_json, - render_services_text, service_stop_unconfirmed, validate_service_id, + render_service_action_result, render_service_action_result_with_helper_note, + render_service_logs_text, render_services_json, render_services_text, service_stop_unconfirmed, + validate_service_id, }; pub(crate) fn services(command: Option) -> Result<()> { diff --git a/apps/rocm/src/services_text.rs b/apps/rocm/src/services_text.rs index cd6af59fb..91c174538 100644 --- a/apps/rocm/src/services_text.rs +++ b/apps/rocm/src/services_text.rs @@ -7,12 +7,12 @@ //! stop and restart paths build on. //! //! Mechanically relocated from `main.rs` with no behavior change, and later -//! narrowed: the process-control cluster (stop, restart and the internal -//! sandbox-tool runner) lives in `service_control.rs` (see -//! `docs/architecture.md`). The `--follow` tail (`FollowCursor`, `FollowFooter`, -//! `follow_service_logs`) is new code written here, not relocated. A -//! `FollowCursor` follows a growing file; it is not `rocm_core::LogTail`, the -//! result of a bounded tail read that the log views here use. +//! narrowed twice: the process-control cluster (stop, restart and the internal +//! sandbox-tool runner) lives in `service_control.rs`, and the `--follow` tail +//! (`follow_service_logs`, `FollowCursor`, `FollowFooter`) in +//! `service_log_follow.rs` (see `docs/architecture.md`). The log *views* here +//! read a bounded tail (`rocm_core::LogTail`), which is not what a +//! `FollowCursor` does. //! //! This module owns: the CLI/TUI logs browser (`render_logs_text`, //! `render_logs_browser_text`, `render_logs_browser_page_text`, @@ -23,10 +23,7 @@ //! status surfaces (`render_services_text`, `render_services_json`, //! `render_service_logs_text`, `render_service_logs_text_with_options`, //! `render_service_action_result`, `read_managed_service_manifest`, -//! `load_managed_service`), the `--follow` tail behind `rocm logs --service -//! --follow` and `rocm services logs --follow` -//! (`follow_service_logs`, `follow_service_logs_with`, the `FollowCursor` and -//! `LogSegment` types, `write_log_segments`), the internal `rocmd`-compatible +//! `load_managed_service`), the internal `rocmd`-compatible //! status surface (`render_internal_status_text`), the `stop_server` answer shape //! (`internal_stop_server_value`, `service_stop_unconfirmed`) and its renderer //! (`render_unconfirmed_service_stop`, `process_check_command`), plus @@ -42,9 +39,9 @@ //! `render_internal_status_text`, `render_service_logs_text`, //! `render_logs_browser_text`, and `render_logs_text` — re-imported via //! `use crate::services_text::{...};`. `services_cmd.rs`'s `services()` calls -//! `render_services_text`, `render_services_json`, `render_service_logs_text` -//! and `follow_service_logs` through its own imports, `dispatch()` calls -//! `follow_service_logs` for `rocm logs --service --follow`, and its `run_approved_service_action` and +//! `render_services_text`, `render_services_json` and +//! `render_service_logs_text` through its own imports, and its +//! `run_approved_service_action` and //! `finish_service_action` also read `render_service_action_result`, //! `service_stop_unconfirmed` and `validate_service_id`. `service_records.rs` //! reads `validate_service_id` (in `service_record_artifacts` and @@ -56,7 +53,10 @@ //! `service_control.rs` and from `managed_service_status.rs`'s own test module; //! `read_managed_service_manifest`, `internal_stop_server_value` and //! `process_check_command` are `pub(crate)` for `service_control.rs` (the last -//! one only for its tests); `read_log_tail`, `render_logs_text`, +//! one only for its tests); `load_managed_service` and +//! `render_service_logs_text` are also what `service_log_follow.rs` calls back +//! into this module for (the liveness re-read and the snapshot it prints before +//! the follow starts); `read_log_tail`, `render_logs_text`, //! `render_logs_browser_text`, `render_services_text` and //! `render_service_logs_text` are also `pub(crate)` and read from //! `chat/mcp_bridge.rs`'s read-only command runner; `validate_service_id` is also @@ -69,7 +69,8 @@ //! The regression tests for what stays here live in this module's own //! `mod tests`, reaching shared fixtures (`test_paths`) through `crate::tests::` //! the same way other mechanically relocated modules do. The tests that drive -//! the stop and restart paths moved to `service_control.rs`; `assert_no_retry_line` +//! the stop and restart paths moved to `service_control.rs` and the ones that +//! drive the `--follow` tail to `service_log_follow.rs`; `assert_no_retry_line` //! stays here as a `pub(crate)` helper because it checks this module's //! unconfirmed-stop rendering and `service_launch.rs`'s tests read it too. Two //! tests that call `load_managed_service` only incidentally — their primary @@ -79,17 +80,17 @@ //! classification respectively — stay in `main.rs`'s shared `mod tests` //! instead. +use std::collections::HashMap; use std::fmt::Write as _; use std::fs; -use std::io::{self, Read as _, Seek as _, SeekFrom}; use std::path::{Path, PathBuf}; -use std::thread; use std::time::Duration; use anyhow::{Context, Result, bail}; use rocm_core::{AppPaths, ManagedServiceRecord}; use rocm_engine_protocol::DEFAULT_LOG_TAIL_LINES; +use crate::engine_ipc::fetch_service_devices; use crate::managed_service_status::{ load_managed_services, local_server_sidebar_status, managed_service_is_live, managed_service_sidebar_counts, refresh_managed_service_runtime_liveness, @@ -400,7 +401,31 @@ fn format_anthropic_endpoint_line(url: &str, responding: bool) -> String { } } +/// `rocm services list`, and `rocm chat`'s `services` command, which runs this +/// in process and renders the same text. Not the MCP tool of the same name, +/// which has its own renderer and no `device:` row. +/// +/// **No longer a pure local read.** Since ROCMAI-550 it makes one live +/// `Healthcheck` RPC per *ready* service to fill the `device:` row, bounded in +/// aggregate by `engine_ipc::SERVICE_DEVICE_PROBE_BUDGET` — see that constant +/// for the ~18s worst case and for which `services` surface pays it. The chat +/// command pays it too, rather than show a device-less view of the same list. pub(crate) fn render_services_text(paths: &AppPaths, all: bool) -> Result { + render_services_text_with_devices(paths, all, |ready| fetch_service_devices(paths, ready)) +} + +/// [`render_services_text`] with the live device probe injected. +/// +/// Split out because the probe is the one thing a unit test cannot have: in +/// default CI it answers nothing, so deleting the `device:` row would stay +/// green outside the GPU lanes. The seam takes only the probe — which services +/// are offered to it and where the row lands stay in the shared body, so a +/// test can pin both. +fn render_services_text_with_devices( + paths: &AppPaths, + all: bool, + probe_devices: impl FnOnce(Vec<(&str, &str)>) -> HashMap, +) -> Result { // Counts describe the WHOLE registry, not the rows this view keeps. The // default view hides records that are not live, and counting the filtered // list made the header say "none ready" on a host whose only local servers @@ -414,7 +439,7 @@ pub(crate) fn render_services_text(paths: &AppPaths, all: bool) -> Result>(); @@ -429,11 +454,9 @@ pub(crate) fn render_services_text(paths: &AppPaths, all: bool) -> Result 0 { write_past_attempts_hint( &mut output, @@ -449,18 +472,37 @@ pub(crate) fn render_services_text(paths: &AppPaths, all: bool) -> Result Result, -} - -/// One completed piece of log output and whether it ended in a carriage return. -#[derive(Debug, PartialEq, Eq)] -struct LogSegment { - text: String, - redraw: bool, -} - -impl FollowCursor { - /// Start following at the file's current end. - fn at_end(path: &Path) -> Self { - let offset = fs::metadata(path).map_or(0, |meta| meta.len()); - Self { - path: path.to_path_buf(), - offset, - pending: Vec::new(), - } - } - - fn poll(&mut self) -> Vec { - let Ok(mut file) = fs::File::open(&self.path) else { - return Vec::new(); - }; - let len = file.metadata().map_or(0, |meta| meta.len()); - if len < self.offset { - // Truncated or replaced (a restart reuses the path): start over. - self.offset = 0; - self.pending.clear(); - } - if file.seek(SeekFrom::Start(self.offset)).is_err() { - return Vec::new(); - } - let mut fresh = Vec::new(); - if file.read_to_end(&mut fresh).is_err() { - return Vec::new(); - } - self.offset += fresh.len() as u64; - self.pending.extend_from_slice(&fresh); - - let mut segments = Vec::new(); - let mut start = 0; - let mut index = 0; - while index < self.pending.len() { - let byte = self.pending[index]; - // A `\r` at the very end of the data may be the first half of a `\r\n` - // that the next poll completes; hold it back rather than emit an - // empty segment (which would erase a final redraw frame on a tty). - if byte == b'\r' && index + 1 == self.pending.len() { - break; - } - if byte == b'\n' || byte == b'\r' { - let mut redraw = byte == b'\r'; - // The footer's row bookkeeping assumes the log text moves no cursor. - let text = rocm_core::terminal::strip_terminal_control_sequences( - &String::from_utf8_lossy(&self.pending[start..index]), - ); - // `\r\n` is one line break, not a redraw followed by an empty line. - if redraw && self.pending.get(index + 1) == Some(&b'\n') { - redraw = false; - index += 1; - } - segments.push(LogSegment { text, redraw }); - start = index + 1; - } - index += 1; - } - self.pending.drain(..start); - segments - } - - /// The unterminated output still held back, for when following ends: a - /// last write cut short by a crash is otherwise never shown. - fn finish(&mut self) -> Option { - let mut bytes = std::mem::take(&mut self.pending); - if bytes.last() == Some(&b'\r') { - bytes.pop(); - } - // Same stripping as `poll`: a cut-off write can end mid-escape. - (!bytes.is_empty()).then(|| LogSegment { - text: rocm_core::terminal::strip_terminal_control_sequences(&String::from_utf8_lossy( - &bytes, - )), - redraw: false, - }) - } -} - -/// Writes `segments`; returns whether the last one was a redraw, which leaves the -/// cursor on that row (the next segment erases it first) instead of a fresh one. -fn write_log_segments( - out: &mut impl io::Write, - segments: &[LogSegment], - tty: bool, -) -> Result { - for segment in segments { - match (tty, segment.redraw) { - // Redraw the same terminal row, as the engine's own bar does. - (true, true) => write!(out, "\r\x1b[2K{}", segment.text)?, - (true, false) => writeln!(out, "\r\x1b[2K{}", segment.text)?, - (false, _) => writeln!(out, "{}", segment.text)?, - } - } - out.flush()?; - Ok(tty && segments.last().is_some_and(|segment| segment.redraw)) -} - -/// The "how to stop" hint kept on the row just below the newest log output of a -/// terminal follow, so it stays in view however much the log scrolls. -/// -/// While drawn, the cursor sits on the footer row. [`Self::erase`] puts it back -/// where the log output left off: on a fresh row, or, after a redraw segment -/// (a download bar), at the end of that still-open row, one row up. -struct FollowFooter { - text: String, - drawn: bool, - /// The row above the footer is an unfinished redraw row. - open_row: bool, -} - -/// Led by the part that says the server keeps running: service ids run 40+ -/// characters, so on a narrow terminal the end of the hint is what gets cut. -fn follow_hint(service_id: &str) -> String { - format!("Ctrl-C stops following, the server keeps running. Following {service_id}") -} - -impl FollowFooter { - fn new(service_id: &str) -> Self { - Self { - text: follow_hint(service_id), - drawn: false, - open_row: false, - } - } - - fn erase(&mut self, out: &mut impl io::Write) -> Result<()> { - if self.drawn { - write!(out, "\r\x1b[2K")?; - if self.open_row { - write!(out, "\x1b[1A")?; - } - self.drawn = false; - } - Ok(()) - } - - /// Clear the footer and end an unfinished redraw row, leaving the cursor on - /// a fresh row. - fn finish(&mut self, out: &mut impl io::Write) -> Result<()> { - self.erase(out)?; - if self.open_row { - writeln!(out)?; - self.open_row = false; - } - out.flush()?; - Ok(()) - } - - /// One line of the follow's own, written below the log output with the - /// footer redrawn under it. An unfinished download bar row is ended first, - /// not overwritten. - fn write_notice( - &mut self, - out: &mut impl io::Write, - text: &str, - cols: Option, - ) -> Result<()> { - self.finish(out)?; - writeln!(out, "{text}")?; - self.draw(out, false, cols) - } - - /// `cols` truncates the text so it never wraps onto a second row, which the - /// one-row erase could not clear. - fn draw( - &mut self, - out: &mut impl io::Write, - open_row: bool, - cols: Option, - ) -> Result<()> { - if open_row { - writeln!(out)?; - } - let shown: String = match cols { - Some(cols) => { - crate::cli_progress::truncate_to_width(&self.text, cols.saturating_sub(1)) - } - None => self.text.clone(), - }; - write!(out, "\r\x1b[2K{shown}")?; - out.flush()?; - self.drawn = true; - self.open_row = open_row; - Ok(()) - } - - /// Write a batch of new output with the footer kept below it. A batch with - /// nothing in it leaves the screen alone (no flicker on an idle log). - fn write_batch( - &mut self, - out: &mut impl io::Write, - segments: &[LogSegment], - cols: Option, - ) -> Result<()> { - if segments.is_empty() { - return Ok(()); - } - self.erase(out)?; - let open_row = write_log_segments(out, segments, true)?; - self.draw(out, open_row, cols) - } -} - -fn terminal_cols() -> Option { - crossterm::terminal::size() - .ok() - .map(|(cols, _)| usize::from(cols)) -} - -/// `rocm logs --service --follow`: print the recent output, then keep -/// printing new output until the service is no longer starting or running. -/// Ctrl-C ends the follow and leaves the service running. -pub(crate) fn follow_service_logs(paths: &AppPaths, service_id: &str) -> Result<()> { - use std::io::IsTerminal as _; - let mut out = io::stdout(); - let tty = out.is_terminal(); - follow_service_logs_with( - paths, - service_id, - &mut out, - &mut io::stderr(), - tty, - LOG_FOLLOW_POLL, - LOG_FOLLOW_LIVENESS_EVERY, - &terminal_cols, - ) -} - -/// True once, the first time `status` reads `ready`. -fn newly_ready(announced: &mut bool, status: &str) -> bool { - let ready = status == "ready" && !*announced; - *announced |= ready; - ready -} - -/// A line about the follow itself (the hint, "is ready.", "stopped following."). -/// On a terminal it shares the log's stream and row layout; piped, it goes to -/// `err` so `rocm logs -f | grep error` sees log text only. -fn write_note( - out: &mut impl io::Write, - err: &mut impl io::Write, - tty: bool, - line: &str, -) -> Result<()> { - if tty { - writeln!(out, "{line}")?; - } else { - writeln!(err, "{line}")?; - } - Ok(()) -} - -#[allow(clippy::too_many_arguments)] -fn follow_service_logs_with( - paths: &AppPaths, - service_id: &str, - out: &mut impl io::Write, - err: &mut impl io::Write, - tty: bool, - poll: Duration, - liveness_every: Duration, - cols: &dyn Fn() -> Option, -) -> Result<()> { - follow_service_logs_via( - paths, - service_id, - &|| load_managed_service(paths, service_id), - out, - err, - tty, - poll, - liveness_every, - cols, - ) -} - -/// [`follow_service_logs_with`] with the record read injected: a `ready` status -/// only sticks while the endpoint answers, so a test cannot plant one on disk. -#[allow(clippy::too_many_arguments)] -fn follow_service_logs_via( - paths: &AppPaths, - service_id: &str, - load: &dyn Fn() -> Result, - out: &mut impl io::Write, - err: &mut impl io::Write, - tty: bool, - poll: Duration, - liveness_every: Duration, - cols: &dyn Fn() -> Option, -) -> Result<()> { - let record = load()?; - // Open the tail before the snapshot: output written in between is then shown - // twice (in the snapshot and by the first poll) rather than lost, and the - // tail can start mid-line. - let mut tail = FollowCursor::at_end(&record.log_path); - write!(out, "{}", render_service_logs_text(paths, service_id)?)?; - out.flush()?; - if !managed_service_is_live(&record) { - write_note( - out, - err, - tty, - &format!( - "\nService {service_id} is {}; not following.", - record.status - ), - )?; - // Same outcome as a follow that sees the server fail: not a success. - if record.status == "failed" { - bail!("service {service_id} failed; see the log output above"); - } - return Ok(()); - } - // On a terminal the hint is a footer that stays under the newest output; a - // pipe or log file gets it once, since it has no row to keep it on. - let mut footer = tty.then(|| FollowFooter::new(service_id)); - match footer.as_mut() { - Some(footer) => footer.draw(out, false, cols())?, - None => write_note(out, err, tty, &format!("\n{}", follow_hint(service_id)))?, - } - out.flush()?; - - let result = follow_loop( - load, - &record, - &mut tail, - &mut footer, - out, - err, - tty, - poll, - liveness_every, - cols, - ); - if result.is_err() - && let Some(footer) = footer.as_mut() - { - // Leave the terminal on a clean row so the caller's `Error:` is not glued - // to the footer or to an unfinished download bar. - let _ = footer.finish(out); - } - result -} - -#[allow(clippy::too_many_arguments)] -fn follow_loop( - load: &dyn Fn() -> Result, - record: &ManagedServiceRecord, - tail: &mut FollowCursor, - footer: &mut Option, - out: &mut impl io::Write, - err: &mut impl io::Write, - tty: bool, - poll: Duration, - liveness_every: Duration, - cols: &dyn Fn() -> Option, -) -> Result<()> { - let service_id = record.service_id.as_str(); - let mut announced_ready = record.status == "ready"; - let mut since_check = Duration::ZERO; - loop { - thread::sleep(poll); - match footer.as_mut() { - Some(footer) => footer.write_batch(out, &tail.poll(), cols())?, - None => { - write_log_segments(out, &tail.poll(), tty)?; - } - } - since_check += poll; - if since_check < liveness_every { - continue; - } - since_check = Duration::ZERO; - let record = load()?; - if newly_ready(&mut announced_ready, &record.status) { - let line = format!("Service {service_id} is ready."); - match footer.as_mut() { - Some(footer) => footer.write_notice(out, &line, cols())?, - None => write_note(out, err, tty, &line)?, - } - } - if !managed_service_is_live(&record) { - // Output written between the last poll and the exit. - let mut last = tail.poll(); - last.extend(tail.finish()); - let mut open_row = false; - if let Some(footer) = footer.as_mut() { - footer.erase(out)?; - open_row = footer.open_row; - } - open_row = write_log_segments(out, &last, tty)? || (open_row && last.is_empty()); - if open_row { - writeln!(out)?; - } - write_note( - out, - err, - tty, - &format!( - "Service {service_id} is {}; stopped following.", - record.status - ), - )?; - // Same outcome as `serve --wait`: a server that failed is not a success. - if record.status == "failed" { - bail!("service {service_id} failed; see the log output above"); - } - return Ok(()); - } - } -} - pub(crate) fn render_service_action_result(tool: &str, value: &serde_json::Value) -> String { let output = value.get("output").unwrap_or(value); if tool == "stop_server" && service_stop_unconfirmed(value) { @@ -1824,840 +1471,71 @@ pub(crate) mod tests { } #[test] - fn log_tail_yields_only_new_complete_segments() -> Result<()> { - let (root, paths) = test_paths("log-tail"); - paths.ensure()?; - let log = root.join("tail.log"); - fs::write(&log, "old line\n")?; - let mut tail = FollowCursor::at_end(&log); - assert!(tail.poll().is_empty()); - - // tqdm-style bar redraws with `\r`; the last piece has no terminator yet. - fs::write(&log, "old line\n 28%|##\r 33%|###\rdone\r\nhalf")?; - let segments = tail.poll(); - assert_eq!( - segments, - vec![ - LogSegment { - text: " 28%|##".into(), - redraw: true - }, - LogSegment { - text: " 33%|###".into(), - redraw: true - }, - LogSegment { - text: "done".into(), - redraw: false - }, - ] - ); - // The held-back partial segment completes on the next poll. - let mut file = fs::OpenOptions::new().append(true).open(&log)?; - std::io::Write::write_all(&mut file, b" line\n")?; - assert_eq!( - tail.poll(), - vec![LogSegment { - text: "half line".into(), - redraw: false - }] - ); - - // A truncated log (restart reusing the path) is read from the start. - fs::write(&log, "new\n")?; - assert_eq!( - tail.poll(), - vec![LogSegment { - text: "new".into(), - redraw: false - }] - ); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn log_tail_holds_back_a_trailing_cr_and_flushes_the_rest_on_finish() -> Result<()> { - let (root, paths) = test_paths("log-tail-edges"); + fn render_service_logs_text_tails_manifest_log() -> Result<()> { + let (root, paths) = test_paths("service-logs"); paths.ensure()?; - let log = root.join("edges.log"); - fs::write(&log, "")?; - let mut tail = FollowCursor::at_end(&log); - - // A `\r` ending the data may be half of a `\r\n`: it is not a redraw yet. - fs::write(&log, " 99%|#\r")?; - assert!(tail.poll().is_empty()); - let mut file = fs::OpenOptions::new().append(true).open(&log)?; - std::io::Write::write_all(&mut file, b"\n")?; - assert_eq!( - tail.poll(), - vec![LogSegment { - text: " 99%|#".into(), - redraw: false - }], - "the held `\\r` joins the next `\\n` instead of yielding an empty segment" - ); - - // Output cut short before any terminator is shown when following ends. - std::io::Write::write_all(&mut file, b"crashed mid-wri")?; - assert!(tail.poll().is_empty()); - assert_eq!( - tail.finish(), - Some(LogSegment { - text: "crashed mid-wri".into(), - redraw: false - }) - ); - assert_eq!(tail.finish(), None); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn write_log_segments_redraws_in_place_only_on_a_tty() -> Result<()> { - let segments = [ - LogSegment { - text: "28%".into(), - redraw: true, - }, - LogSegment { - text: "done".into(), - redraw: false, - }, - ]; - let mut tty = Vec::new(); - write_log_segments(&mut tty, &segments, true)?; - // A redraw segment leaves the cursor on its row; the next segment (or - // line) erases the row first, and a plain line ends with a newline. - assert_eq!(String::from_utf8(tty)?, "\r\x1b[2K28%\r\x1b[2Kdone\n"); - - let mut piped = Vec::new(); - write_log_segments(&mut piped, &segments, false)?; - assert_eq!(String::from_utf8(piped)?, "28%\ndone\n"); - Ok(()) - } - - fn seg(text: &str, redraw: bool) -> LogSegment { - LogSegment { - text: text.into(), - redraw, - } - } - - const ERASE: &str = "\r\x1b[2K"; - - #[test] - fn follow_footer_is_redrawn_under_each_batch_of_new_lines() -> Result<()> { - let mut footer = FollowFooter::new("svc_a"); - let hint = footer.text.clone(); - let mut out = Vec::new(); - footer.draw(&mut out, false, None)?; - assert_eq!(String::from_utf8(out.clone())?, format!("{ERASE}{hint}")); - - out.clear(); - footer.write_batch(&mut out, &[seg("one", false), seg("two", false)], None)?; - // Footer row cleared, lines written over it, footer redrawn on the next row. - assert_eq!( - String::from_utf8(out)?, - format!("{ERASE}{ERASE}one\n{ERASE}two\n{ERASE}{hint}") - ); - Ok(()) - } - - #[test] - fn follow_footer_leaves_an_idle_screen_alone() -> Result<()> { - let mut footer = FollowFooter::new("svc_a"); - let mut out = Vec::new(); - footer.draw(&mut out, false, None)?; - out.clear(); - footer.write_batch(&mut out, &[], None)?; - assert!(out.is_empty(), "{out:?}"); - Ok(()) - } - - #[test] - fn follow_footer_keeps_a_download_bar_on_its_own_row_above_it() -> Result<()> { - let mut footer = FollowFooter::new("svc_a"); - let hint = footer.text.clone(); - let mut out = Vec::new(); - footer.draw(&mut out, false, None)?; - out.clear(); - - footer.write_batch(&mut out, &[seg("28%", true)], None)?; - // The bar row stays open; the footer goes on the row below it. - assert_eq!( - String::from_utf8(out.clone())?, - format!("{ERASE}{ERASE}28%\n{ERASE}{hint}") - ); - - out.clear(); - footer.write_batch(&mut out, &[seg("43%", true)], None)?; - // Erase the footer, step back up onto the bar row, redraw it in place. - assert_eq!( - String::from_utf8(out.clone())?, - format!("{ERASE}\x1b[1A{ERASE}43%\n{ERASE}{hint}") - ); - - out.clear(); - footer.write_batch(&mut out, &[seg("done", false)], None)?; - // A finished line replaces the bar, and the footer is back on a fresh row. - assert_eq!( - String::from_utf8(out)?, - format!("{ERASE}\x1b[1A{ERASE}done\n{ERASE}{hint}") - ); - Ok(()) - } - - #[test] - fn follow_footer_is_cut_to_one_terminal_row() -> Result<()> { - let mut footer = FollowFooter::new("svc_a"); - let mut out = Vec::new(); - footer.draw(&mut out, false, Some(30))?; - // One column short of the width, so it never wraps, and what is kept is - // the head of the hint: the part saying the server keeps running. - let drawn = String::from_utf8(out)?; - let shown = drawn.strip_prefix(ERASE).expect("erase prefix"); - assert_eq!(unicode_width::UnicodeWidthStr::width(shown), 29, "{shown}"); - assert!(shown.starts_with("Ctrl-C stops following, the"), "{shown}"); - Ok(()) - } - - #[test] - fn log_tail_strips_cursor_escapes_from_engine_output() -> Result<()> { - let (root, _paths) = test_paths("tail-escapes"); - fs::create_dir_all(&root)?; - let log = root.join("engine.log"); - fs::write(&log, "")?; - let mut tail = FollowCursor::at_end(&log); - fs::write(&log, "before\x1b[1Aafter\n")?; - let segments = tail.poll(); - assert_eq!(segments.len(), 1); - assert_eq!(segments[0].text, "beforeafter"); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn log_tail_finish_strips_cursor_escapes_from_a_cut_off_write() -> Result<()> { - let (root, _paths) = test_paths("tail-finish-escapes"); - fs::create_dir_all(&root)?; - let log = root.join("engine.log"); - fs::write(&log, "")?; - let mut tail = FollowCursor::at_end(&log); - fs::write(&log, "last\x1b[1Aword")?; - assert!(tail.poll().is_empty()); - let segment = tail.finish().expect("unterminated output is returned"); - assert_eq!(segment.text, "lastword"); - let _ = fs::remove_dir_all(root); - Ok(()) - } - #[test] - fn follow_service_logs_keeps_the_hint_under_new_output_on_a_tty() -> Result<()> { - let (root, paths) = test_paths("follow-tty"); - paths.ensure()?; let mut record = ManagedServiceRecord::new( &paths, - "svc_follow_tty", + "svc_qwen35_primary", "vllm", "qwen3.5", "Qwen/Qwen3.5", "127.0.0.1", - 11438, + 11435, "managed", std::process::id(), + Some("therock-release".to_owned()), None, - None, - None, + Some("gpu_preferred".to_owned()), ); - record.status = "starting".to_owned(); + record.status = "ready".to_owned(); record.write()?; - fs::write(&record.log_path, "before\n")?; - - let log_path = record.log_path; - let manifest_paths = paths.clone(); - let writer = thread::spawn(move || -> Result<()> { - let appended = (|| -> Result<()> { - thread::sleep(Duration::from_millis(150)); - let mut file = fs::OpenOptions::new().append(true).open(&log_path)?; - std::io::Write::write_all(&mut file, b"Downloading 43%\n")?; - thread::sleep(Duration::from_millis(150)); - Ok(()) - })(); - let mut record = load_managed_service(&manifest_paths, "svc_follow_tty")?; - record.status = "stopped".to_owned(); - record.write()?; - appended - }); - let mut out = Vec::new(); - let mut err = Vec::new(); - follow_service_logs_with( - &paths, - "svc_follow_tty", - &mut out, - &mut err, - true, - Duration::from_millis(20), - Duration::from_millis(40), - &|| None, - )?; - writer.join().expect("writer thread")?; - let out = String::from_utf8(out)?; - let hint = FollowFooter::new("svc_follow_tty").text; - // The hint is drawn again right under the line that scrolled it away, - // and only the stop message follows it, after the footer row is cleared. - assert!( - out.contains(&format!("Downloading 43%\n{ERASE}{hint}")), - "{out:?}" - ); - assert!( - out.ends_with(&format!( - "{hint}{ERASE}Service svc_follow_tty is stopped; stopped following.\n" - )), - "{out:?}" - ); + + let mut log = String::new(); + for index in 1..=90 { + let _ = writeln!(log, "entry-{index:03}"); + } + fs::write(&record.log_path, log)?; + + let rendered = render_service_logs_text(&paths, "svc_qwen35_primary")?; + assert!(rendered.contains("Service Log")); + assert!(rendered.contains("Service: svc_qwen35_primary")); + assert!(rendered.contains("Engine: vllm")); + assert!(rendered.contains("Status: starting")); + assert!(rendered.contains("File locations: shown")); + assert!(rendered.contains(&format!( + " Details file: {}", + record.manifest_path.display() + ))); + assert!(rendered.contains(&format!(" Log file: {}", record.log_path.display()))); + assert!(!rendered.contains("entry-010")); + assert!(rendered.contains("entry-011")); + assert!(rendered.contains("entry-090")); + let _ = fs::remove_dir_all(root); Ok(()) } - #[test] - fn follow_service_logs_ends_an_open_download_bar_row_before_the_stop_message() -> Result<()> { - let (root, paths) = test_paths("follow-tty-bar"); + /// What `rocm services logs` prints for a service whose log holds `bytes` + /// (or has no file when `None`), with the log's size on disk, so a message + /// is asserted together with the state it describes. + fn render_service_log_with(name: &str, bytes: Option<&[u8]>) -> Result<(String, Option)> { + let (root, paths) = test_paths(name); paths.ensure()?; - let mut record = ManagedServiceRecord::new( + let record = ManagedServiceRecord::new( &paths, - "svc_follow_bar", + "svc_tail_notice", "vllm", "qwen3.5", "Qwen/Qwen3.5", "127.0.0.1", - 11440, + 11435, "managed", std::process::id(), + Some("therock-release".to_owned()), None, - None, - None, - ); - record.status = "starting".to_owned(); - record.write()?; - fs::write(&record.log_path, "")?; - - let log_path = record.log_path; - let manifest_paths = paths.clone(); - let writer = thread::spawn(move || -> Result<()> { - let appended = (|| -> Result<()> { - thread::sleep(Duration::from_millis(150)); - let mut file = fs::OpenOptions::new().append(true).open(&log_path)?; - std::io::Write::write_all(&mut file, b"28%\r43%\r\r")?; - thread::sleep(Duration::from_millis(150)); - Ok(()) - })(); - let mut record = load_managed_service(&manifest_paths, "svc_follow_bar")?; - record.status = "stopped".to_owned(); - record.write()?; - appended - }); - let mut out = Vec::new(); - let mut err = Vec::new(); - follow_service_logs_with( - &paths, - "svc_follow_bar", - &mut out, - &mut err, - true, - Duration::from_millis(20), - Duration::from_millis(40), - &|| None, - )?; - writer.join().expect("writer thread")?; - let out = String::from_utf8(out)?; - // The footer is erased and the cursor goes back up onto the bar's row; that row - // is then ended with a newline so the stop message starts on its own row. - assert!( - out.ends_with(&format!( - "{ERASE}\x1b[1A\nService svc_follow_bar is stopped; stopped following.\n" - )), - "{out:?}" - ); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn follow_service_logs_draws_no_footer_for_a_stopped_service_on_a_tty() -> Result<()> { - let (root, paths) = test_paths("follow-tty-stopped"); - paths.ensure()?; - let mut record = ManagedServiceRecord::new( - &paths, - "svc_follow_tty_stopped", - "vllm", - "qwen3.5", - "Qwen/Qwen3.5", - "127.0.0.1", - 11439, - "managed", - 0, - None, - None, - None, - ); - record.status = "stopped".to_owned(); - record.write()?; - let mut out = Vec::new(); - let mut err = Vec::new(); - follow_service_logs_with( - &paths, - "svc_follow_tty_stopped", - &mut out, - &mut err, - true, - Duration::from_millis(1), - Duration::from_millis(1), - &|| None, - )?; - let out = String::from_utf8(out)?; - assert!(!out.contains("Following"), "{out:?}"); - assert!(out.contains("not following."), "{out:?}"); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn follow_service_logs_prints_snapshot_and_returns_for_a_stopped_service() -> Result<()> { - let (root, paths) = test_paths("follow-stopped"); - paths.ensure()?; - let mut record = ManagedServiceRecord::new( - &paths, - "svc_follow_stopped", - "vllm", - "qwen3.5", - "Qwen/Qwen3.5", - "127.0.0.1", - 11436, - "managed", - 0, - None, - None, - None, - ); - record.status = "stopped".to_owned(); - record.write()?; - fs::write(&record.log_path, "Downloading 86%\n")?; - - let mut out = Vec::new(); - let mut err = Vec::new(); - follow_service_logs_with( - &paths, - "svc_follow_stopped", - &mut out, - &mut err, - false, - Duration::from_millis(1), - Duration::from_millis(1), - &|| None, - )?; - let out = String::from_utf8(out)?; - let err = String::from_utf8(err)?; - assert!(out.contains("Downloading 86%"), "{out}"); - // The notice is not log text, so a pipe keeps it off stdout. - assert!(!out.contains("not following"), "{out}"); - assert!( - err.contains("Service svc_follow_stopped is stopped; not following."), - "{err}" - ); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn newly_ready_fires_once_on_the_first_ready_status() { - let mut announced = false; - assert!(!newly_ready(&mut announced, "starting")); - assert!(newly_ready(&mut announced, "ready")); - assert!(!newly_ready(&mut announced, "ready")); - // A follow that began on an already-ready server has announced nothing new. - let mut announced = true; - assert!(!newly_ready(&mut announced, "ready")); - } - - #[test] - fn follow_service_logs_fails_when_the_service_failed() -> Result<()> { - let (root, paths) = test_paths("follow-ready-failed"); - paths.ensure()?; - let mut record = ManagedServiceRecord::new( - &paths, - "svc_follow_fail", - "vllm", - "qwen3.5", - "Qwen/Qwen3.5", - "127.0.0.1", - 11438, - "managed", - std::process::id(), - None, - None, - None, - ); - record.status = "starting".to_owned(); - record.write()?; - fs::write(&record.log_path, "before\n")?; - - let manifest_paths = paths.clone(); - let writer = thread::spawn(move || -> Result<()> { - thread::sleep(Duration::from_millis(200)); - let mut record = load_managed_service(&manifest_paths, "svc_follow_fail")?; - record.status = "failed".to_owned(); - record.write() - }); - let mut out = Vec::new(); - let mut err = Vec::new(); - let result = follow_service_logs_with( - &paths, - "svc_follow_fail", - &mut out, - &mut err, - false, - Duration::from_millis(20), - Duration::from_millis(40), - &|| None, - ); - writer.join().expect("writer thread")?; - let err = String::from_utf8(err)?; - assert!(err.contains("is failed; stopped following"), "{err}"); - let error = result.expect_err("a failed service must not exit 0"); - assert!( - error.to_string().contains("svc_follow_fail failed"), - "{error}" - ); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn follow_service_logs_streams_new_output_until_the_service_stops() -> Result<()> { - let (root, paths) = test_paths("follow-live"); - paths.ensure()?; - let mut record = ManagedServiceRecord::new( - &paths, - "svc_follow_live", - "vllm", - "qwen3.5", - "Qwen/Qwen3.5", - "127.0.0.1", - 11437, - "managed", - std::process::id(), - None, - None, - None, - ); - record.status = "starting".to_owned(); - record.write()?; - fs::write(&record.log_path, "before\n")?; - - let log_path = record.log_path; - let manifest_paths = paths.clone(); - let writer = thread::spawn(move || -> Result<()> { - let appended = (|| -> Result<()> { - thread::sleep(Duration::from_millis(150)); - let mut file = fs::OpenOptions::new().append(true).open(&log_path)?; - std::io::Write::write_all(&mut file, b"Downloading 43%\n")?; - thread::sleep(Duration::from_millis(150)); - Ok(()) - })(); - // Stop the service even when the append failed, so the follow loop - // under test always ends instead of hanging the run. - let mut record = load_managed_service(&manifest_paths, "svc_follow_live")?; - record.status = "stopped".to_owned(); - record.write()?; - appended - }); - let mut out = Vec::new(); - let mut err = Vec::new(); - follow_service_logs_with( - &paths, - "svc_follow_live", - &mut out, - &mut err, - false, - Duration::from_millis(20), - Duration::from_millis(40), - &|| None, - )?; - writer.join().expect("writer thread")?; - let out = String::from_utf8(out)?; - let err = String::from_utf8(err)?; - // Piped: stdout is the log text alone, with no terminal escapes, and the - // follow's own lines go to stderr, the hint exactly once. - assert!(out.contains("Downloading 43%"), "{out}"); - assert!(!out.contains('\x1b'), "{out:?}"); - assert!(!out.contains("Following"), "{out}"); - assert!(!out.contains("stopped following"), "{out}"); - assert_eq!(err.matches("Following svc_follow_live").count(), 1, "{err}"); - assert!(err.contains("stopped following"), "{err}"); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - fn starting_record(paths: &AppPaths, id: &str, port: u16) -> Result { - let mut record = ManagedServiceRecord::new( - paths, - id, - "vllm", - "qwen3.5", - "Qwen/Qwen3.5", - "127.0.0.1", - port, - "managed", - std::process::id(), - None, - None, - None, - ); - record.status = "starting".to_owned(); - record.write()?; - Ok(record) - } - - /// A record read that walks through `statuses` (the last one repeats). - fn scripted_statuses( - base: ManagedServiceRecord, - statuses: &'static [&'static str], - ) -> impl Fn() -> Result { - let calls = std::cell::Cell::new(0_usize); - move || { - let call = calls.get(); - calls.set(call + 1); - let mut record = base.clone(); - record.status = statuses[call.min(statuses.len() - 1)].to_owned(); - Ok(record) - } - } - - #[test] - fn follow_service_logs_announces_ready_once_when_the_server_comes_up() -> Result<()> { - let (root, paths) = test_paths("follow-ready-line"); - paths.ensure()?; - let record = starting_record(&paths, "svc_follow_ready", 11441)?; - fs::write(&record.log_path, "before\n")?; - // Initial read, then one read per liveness check. - let load = scripted_statuses( - record, - &["starting", "starting", "ready", "ready", "stopped"], - ); - let mut out = Vec::new(); - let mut err = Vec::new(); - follow_service_logs_via( - &paths, - "svc_follow_ready", - &load, - &mut out, - &mut err, - false, - Duration::from_millis(5), - Duration::from_millis(5), - &|| None, - )?; - let err = String::from_utf8(err)?; - assert_eq!( - err.matches("Service svc_follow_ready is ready.").count(), - 1, - "{err}" - ); - assert!(!String::from_utf8(out)?.contains("is ready.")); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn follow_service_logs_ends_a_download_bar_row_before_the_ready_line_on_a_tty() -> Result<()> { - let (root, paths) = test_paths("follow-ready-bar"); - paths.ensure()?; - let record = starting_record(&paths, "svc_follow_rbar", 11442)?; - fs::write(&record.log_path, "")?; - let log_path = record.log_path.clone(); - let appender = thread::spawn(move || -> Result<()> { - thread::sleep(Duration::from_millis(50)); - let mut file = fs::OpenOptions::new().append(true).open(&log_path)?; - std::io::Write::write_all(&mut file, b"28%\r43%\r")?; - Ok(()) - }); - // The bar is on screen when the status turns ready (liveness check 3 on). - let calls = std::cell::Cell::new(0_usize); - let base = record; - let load = move || -> Result { - let call = calls.get(); - calls.set(call + 1); - if call >= 3 { - thread::sleep(Duration::from_millis(30)); - } - let mut record = base.clone(); - record.status = match call { - 0..=7 => "starting", - 8..=9 => "ready", - _ => "stopped", - } - .to_owned(); - Ok(record) - }; - let mut out = Vec::new(); - let mut err = Vec::new(); - follow_service_logs_via( - &paths, - "svc_follow_rbar", - &load, - &mut out, - &mut err, - true, - Duration::from_millis(10), - Duration::from_millis(10), - &|| None, - )?; - appender.join().expect("appender thread")?; - let out = String::from_utf8(out)?; - // The bar's row is ended with a newline, then the ready line gets its own - // row; it never replaces the bar. - assert!( - out.contains(&format!( - "{ERASE}\x1b[1A\nService svc_follow_rbar is ready.\n" - )), - "{out:?}" - ); - assert!(err.is_empty(), "a terminal follow writes nothing to stderr"); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn follow_service_logs_clears_the_footer_when_the_follow_errors() -> Result<()> { - let (root, paths) = test_paths("follow-error"); - paths.ensure()?; - let record = starting_record(&paths, "svc_follow_gone", 11443)?; - fs::write(&record.log_path, "")?; - let manifest = record.manifest_path; - // The service record disappears mid-follow, so the next liveness read fails. - let remover = thread::spawn(move || -> Result<()> { - thread::sleep(Duration::from_millis(100)); - fs::remove_file(manifest)?; - Ok(()) - }); - let mut out = Vec::new(); - let mut err = Vec::new(); - let result = follow_service_logs_with( - &paths, - "svc_follow_gone", - &mut out, - &mut err, - true, - Duration::from_millis(20), - Duration::from_millis(40), - &|| None, - ); - remover.join().expect("remover thread")?; - assert!(result.is_err(), "a vanished record must end the follow"); - let out = String::from_utf8(out)?; - // The footer row is erased, so `Error:` starts on a clean row. - assert!(out.ends_with(ERASE), "{out:?}"); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn follow_service_logs_fails_for_a_service_that_is_already_failed() -> Result<()> { - let (root, paths) = test_paths("follow-already-failed"); - paths.ensure()?; - let mut record = starting_record(&paths, "svc_follow_dead", 11444)?; - record.status = "failed".to_owned(); - record.supervisor_pid = 0; - record.write()?; - let mut out = Vec::new(); - let mut err = Vec::new(); - let result = follow_service_logs_with( - &paths, - "svc_follow_dead", - &mut out, - &mut err, - false, - Duration::from_millis(1), - Duration::from_millis(1), - &|| None, - ); - let error = result.expect_err("a failed service must not exit 0"); - assert!( - error.to_string().contains("svc_follow_dead failed"), - "{error}" - ); - assert!(String::from_utf8(err)?.contains("is failed; not following.")); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn render_service_logs_text_tails_manifest_log() -> Result<()> { - let (root, paths) = test_paths("service-logs"); - paths.ensure()?; - - let mut record = ManagedServiceRecord::new( - &paths, - "svc_qwen35_primary", - "vllm", - "qwen3.5", - "Qwen/Qwen3.5", - "127.0.0.1", - 11435, - "managed", - std::process::id(), - Some("therock-release".to_owned()), - None, - Some("gpu_preferred".to_owned()), - ); - record.status = "ready".to_owned(); - record.write()?; - - let mut log = String::new(); - for index in 1..=90 { - let _ = writeln!(log, "entry-{index:03}"); - } - fs::write(&record.log_path, log)?; - - let rendered = render_service_logs_text(&paths, "svc_qwen35_primary")?; - assert!(rendered.contains("Service Log")); - assert!(rendered.contains("Service: svc_qwen35_primary")); - assert!(rendered.contains("Engine: vllm")); - assert!(rendered.contains("Status: starting")); - assert!(rendered.contains("File locations: shown")); - assert!(rendered.contains(&format!( - " Details file: {}", - record.manifest_path.display() - ))); - assert!(rendered.contains(&format!(" Log file: {}", record.log_path.display()))); - assert!(!rendered.contains("entry-010")); - assert!(rendered.contains("entry-011")); - assert!(rendered.contains("entry-090")); - - let _ = fs::remove_dir_all(root); - Ok(()) - } - - /// What `rocm services logs` prints for a service whose log holds `bytes` - /// (or has no file when `None`), with the log's size on disk, so a message - /// is asserted together with the state it describes. - fn render_service_log_with(name: &str, bytes: Option<&[u8]>) -> Result<(String, Option)> { - let (root, paths) = test_paths(name); - paths.ensure()?; - let record = ManagedServiceRecord::new( - &paths, - "svc_tail_notice", - "vllm", - "qwen3.5", - "Qwen/Qwen3.5", - "127.0.0.1", - 11435, - "managed", - std::process::id(), - Some("therock-release".to_owned()), - None, - Some("gpu_preferred".to_owned()), + Some("gpu_preferred".to_owned()), ); record.write()?; let size = match bytes { @@ -2954,6 +1832,10 @@ pub(crate) mod tests { ); assert!(rendered.contains("- svc-ready")); assert!(rendered.contains(" stop: rocm services stop svc-ready --yes")); + // No real engine state file backs this record, so the engine's own + // Healthcheck reports `device: "unknown"` - not more informative than + // omitting the line, so it must not appear. + assert!(!rendered.contains("device:"), "{rendered}"); assert!(!rendered.contains("- svc-failed")); assert!(all.contains("- svc-failed")); assert!(all.contains(" restart: rocm services restart svc-failed --yes")); @@ -2961,6 +1843,205 @@ pub(crate) mod tests { Ok(()) } + /// The *positive* half of the `device:` row, which nothing in default CI + /// could assert before: it needs a running engine to answer a `Healthcheck`, + /// and this suite has none, so the row rendered nowhere and deleting the + /// `if let` that writes it stayed green off the GPU lanes. + /// + /// Three things are pinned here, all of which a plausible edit could undo + /// while the rest of the suite stayed green: that only `ready` services are + /// probed at all (a `starting` server has nothing to report and probing it + /// would spend the shared budget on a certain miss), that a device that + /// *is* known reaches the output, and that it lands between `engine:` and + /// `model:` rather than merely appearing somewhere in the text. + #[test] + fn render_services_text_shows_the_device_a_ready_engine_reports() -> Result<()> { + use std::io::{Read, Write}; + use std::net::TcpListener; + + let (root, paths) = test_paths("services-list-device"); + paths.ensure()?; + let listener = TcpListener::bind(("127.0.0.1", 0))?; + let ready_port = listener.local_addr()?.port(); + let server = thread::spawn(move || -> Result<()> { + for _ in 0..2 { + let (mut stream, _) = listener.accept()?; + stream.set_read_timeout(Some(Duration::from_secs(2)))?; + let mut request = [0_u8; 512]; + let read = stream.read(&mut request)?; + let request = String::from_utf8_lossy(&request[..read]).into_owned(); + let body = if request.starts_with("POST /v1/chat/completions ") { + r#"{"choices":[{"message":{"content":"ok"}}]}"# + } else { + r#"{"data":[{"id":"Qwen/Qwen3.5"}]}"# + }; + write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body + )?; + } + Ok(()) + }); + let current_pid = std::process::id(); + for (service_id, status, port) in [ + ("svc-ready", "ready", ready_port), + ("svc-starting", "starting", 11438_u16), + ] { + let mut record = ManagedServiceRecord::new( + &paths, + service_id, + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + port, + "managed", + current_pid, + Some("therock-release".to_owned()), + None, + Some("gpu_required".to_owned()), + ); + record.status = status.to_owned(); + record.write()?; + } + + let mut offered: Vec<(String, String)> = Vec::new(); + let rendered = render_services_text_with_devices(&paths, false, |ready| { + offered = ready + .iter() + .map(|(id, engine)| ((*id).to_owned(), (*engine).to_owned())) + .collect(); + HashMap::from([( + "svc-ready".to_owned(), + "rocm gpu 0 (AMD Instinct MI300X)".to_owned(), + )]) + })?; + server + .join() + .expect("fake models server should not panic")?; + let _ = fs::remove_dir_all(root); + + assert_eq!( + offered, + [("svc-ready".to_owned(), "vllm".to_owned())], + "only a service ready to answer may be probed" + ); + assert!( + rendered.contains(" device: rocm gpu 0 (AMD Instinct MI300X)\n"), + "the device the engine reported must reach the list:\n{rendered}" + ); + assert!( + rendered + .contains(" engine: vllm\n device: rocm gpu 0 (AMD Instinct MI300X)\n model: "), + "the device belongs with the engine that selected it, above the \ + model:\n{rendered}" + ); + // The service the probe said nothing about keeps no row of its own. + assert_eq!( + rendered.matches(" device: ").count(), + 1, + "only the service with a reported device gets a row:\n{rendered}" + ); + Ok(()) + } + + /// The `anthropic endpoint:` row must survive the device probe's gap. + /// + /// `render_services_text_with_devices` snapshots every record once, up + /// front. That used to be a local read taken effectively at print time; + /// since the device probe went in between, the snapshot can be seconds old + /// by the time a row is written. The serving process publishes the + /// Anthropic endpoint to the engine state file at about the moment the + /// engine first lists its model — the same moment the probe starts getting + /// an answer — so that gap is precisely the window in which a freshly + /// ready service renders with no `anthropic endpoint:` row at all. + /// + /// The probe closure here *is* that window: it publishes the endpoint + /// while it runs. Rendering from the pre-probe snapshot therefore shows no + /// row and this test fails; only a re-read after the probe passes it. The + /// fixture cannot be "the endpoint is simply on the record", which the + /// stopped-service test above already covers and which passes either way. + #[test] + fn services_listing_shows_an_anthropic_endpoint_published_while_the_probe_ran() -> Result<()> { + use std::io::{Read, Write}; + use std::net::TcpListener; + + let (root, paths) = test_paths("services-anthropic-after-probe"); + paths.ensure()?; + let listener = TcpListener::bind(("127.0.0.1", 0))?; + let ready_port = listener.local_addr()?.port(); + let server = thread::spawn(move || -> Result<()> { + for _ in 0..2 { + let (mut stream, _) = listener.accept()?; + stream.set_read_timeout(Some(Duration::from_secs(2)))?; + let mut request = [0_u8; 512]; + let read = stream.read(&mut request)?; + let request = String::from_utf8_lossy(&request[..read]).into_owned(); + let body = if request.starts_with("POST /v1/chat/completions ") { + r#"{"choices":[{"message":{"content":"ok"}}]}"# + } else { + r#"{"data":[{"id":"Qwen/Qwen3.5"}]}"# + }; + write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body + )?; + } + Ok(()) + }); + let mut record = ManagedServiceRecord::new( + &paths, + "svc-anthropic-late", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + ready_port, + "managed", + std::process::id(), + Some("therock-release".to_owned()), + None, + Some("gpu_required".to_owned()), + ); + record.status = "ready".to_owned(); + record.write()?; + // Nothing is published when the snapshot is taken: no state file exists, + // so `refresh_from_engine_state` has nothing to adopt. + assert!(!record.engine_state_path.exists()); + let state_path = record.engine_state_path; + + let rendered = render_services_text_with_devices(&paths, false, |_ready| { + // What the serving process does while the probe is in flight. + if let Some(parent) = state_path.parent() { + fs::create_dir_all(parent).expect("engine state dir"); + } + fs::write( + &state_path, + r#"{"status": "ready", "anthropic_endpoint_url": "http://127.0.0.1:1", "anthropic_endpoint_mode": "proxy"}"#, + ) + .expect("publish the anthropic endpoint mid-probe"); + HashMap::new() + })?; + server + .join() + .expect("fake models server should not panic")?; + let _ = fs::remove_dir_all(root); + + // Port 1 is closed, so the row says "(not responding)". What is pinned + // is that the row exists and names the published URL — the five Strix + // Halo scenarios that regressed look for exactly this prefix. + assert!( + rendered.contains(" anthropic endpoint: http://127.0.0.1:1"), + "an endpoint published during the device probe must still reach the \ + row, not be lost to the pre-probe snapshot:\n{rendered}" + ); + Ok(()) + } + /// The default view lists only live servers. The counts used to be taken /// from that already-filtered list, so a host whose local servers had all /// failed was told "none ready" and shown an empty list - the records were diff --git a/apps/rocm/src/therock/family.rs b/apps/rocm/src/therock/family.rs index c60753627..3afc4e153 100644 --- a/apps/rocm/src/therock/family.rs +++ b/apps/rocm/src/therock/family.rs @@ -12,9 +12,9 @@ use super::source::{FamilyResolution, TheRockChannel}; use anyhow::{Result, bail}; use rocm_core::{ - AppPaths, detect_host_gfx_target, detect_host_gpu_diagnostics, detect_managed_therock_family, - gfx_target_has_no_rocm_build, known_therock_families, normalize_therock_family, - runtime_is_windows, + AppPaths, GpuSupportLevel, VULKAN_ONLY_SUPPORT_CLAIM, detect_host_gfx_target, + detect_host_gpu_diagnostics, detect_managed_therock_family, gpu_support_level, + known_therock_families, normalize_therock_family, runtime_is_windows, }; use std::fmt::Write as _; @@ -99,7 +99,7 @@ pub(super) fn resolve_family( // left over from an earlier session happens to name gfx90c, and the // reverse for an env value once `--family` isn't the one in play. if let Some(value) = family_override { - if gfx_target_has_no_rocm_build(value) { + if target_is_vulkan_only(value) { bail!(unsupported_requested_target_message( value.trim(), "--family" @@ -115,7 +115,7 @@ pub(super) fn resolve_family( } if let Some(value) = env_override.as_deref() { - if gfx_target_has_no_rocm_build(value) { + if target_is_vulkan_only(value) { bail!(unsupported_requested_target_message( value.trim(), "ROCM_CLI_THEROCK_FAMILY" @@ -156,7 +156,7 @@ pub(super) fn resolve_family( // there is nothing to install for this GPU. Other unmapped targets keep the // generic error below, which points at `--family`. if let Some(raw_arch) = host_arch.as_deref() - && gfx_target_has_no_rocm_build(raw_arch) + && target_is_vulkan_only(raw_arch) { let message = unsupported_host_gpu_message(raw_arch); let diagnostics = detect_host_gpu_diagnostics(); @@ -172,15 +172,66 @@ pub(super) fn resolve_family( ) } +/// Whether `value` is a GPU target the CLI recognizes and knows has no ROCm +/// build — [`GpuSupportLevel::VulkanOnly`] — which is what `install sdk` +/// refuses with the two messages below. +/// +/// Asked of [`gpu_support_level`] rather than re-tested here. The install path +/// used to call `gfx_target_has_no_rocm_build` directly, which was *exactly* +/// equivalent, and still wrong to keep: it +/// made this a second, independent definition of the tier whose claim these +/// messages then go on to make, so a change to the classification would have +/// left `rocm examine` and `rocm install sdk` disagreeing about the same GPU +/// while both looked correct in isolation. One classifier, one answer. +/// +/// The family argument is computed rather than passed as `None` because this +/// is also asked of a target the *user* named, before any host probe: for +/// `--family gfx1100` the covering family is what makes the answer `RocmHip`, +/// and hard-coding `None` here would classify every requested target as if no +/// family covered it. On the host branch the caller has already established +/// that `normalize_therock_family` answers `None`, so the extra call there is +/// a no-op that costs a string compare. +fn target_is_vulkan_only(value: &str) -> bool { + matches!( + gpu_support_level(Some(value), normalize_therock_family(value).as_deref()), + GpuSupportLevel::VulkanOnly + ) +} + /// The install error for a detected GPU target that no TheRock package family /// covers. It deliberately does not list `--family` values: none of them was /// built for this GPU. +/// +/// The remediation pins the engine with `--engine lemonade` rather than +/// suggesting a bare `rocm serve `. This branch knows the *host* has no +/// ROCm build, which is not the same as knowing which engine `serve` would +/// pick: `select_serve_engine` (`serve_cmd.rs`) ranks an explicit `--engine` +/// first, then a configured `default_engine`, and only then the host's own +/// preference — so a user who once ran `rocm config set-default-engine vllm` +/// (never validated against the GPU) would have the bare command attempt vLLM +/// on a host with no ROCm, and fail under the no-CPU-fallback policy. Naming +/// the engine makes the sentence true whatever the config says, because the +/// explicit flag out-ranks every other selection source. +/// +/// **The named command is not refused on the host that is shown it.** `serve`'s +/// no-usable-GPU pre-flight is `host_has_no_usable_gpu` (`serve_cmd.rs`) over +/// `rocm_core::usable_amd_gpu_indices`, and on Linux that composes the KFD +/// compute-node count with the DRM `amdgpu` card count: a *zero* KFD count +/// falls back to DRM rather than refusing. A gfx90c APU whose kernel exposes no +/// compute node can only have been named by the DRM `ip_discovery` read, which +/// walks the same `card*/device` entries that count — so this branch is +/// reachable only on a host with at least one such card, and the pre-flight +/// therefore sees ordinal 0. `rocm_core`'s +/// `a_gfx90c_host_the_kernel_gives_no_compute_node_still_has_a_usable_gpu` +/// drives both halves off one planted `/sys`, and serve-63 runs the real binary +/// against that same host shape on a GPU-less lane. fn unsupported_host_gpu_message(raw_arch: &str) -> String { format!( "no ROCm SDK build exists for the detected GPU target `{raw_arch}`, so \ `rocm install sdk` has nothing to install for it.\n\ - The GPU is detected but is not supported by ROCm; \ - the Vulkan backend can still be used." + The GPU is detected but is {VULKAN_ONLY_SUPPORT_CLAIM}. \ + Try `rocm serve --engine lemonade` - it runs on Lemonade's \ + Vulkan backend instead." ) } @@ -191,12 +242,24 @@ fn unsupported_host_gpu_message(raw_arch: &str) -> String { /// here: the user named the target, so the error names where it came from and /// does not claim a GPU is present. Like that message it offers no `--family` /// values, since none of them was built for this GPU. +/// +/// That same distinction is why this one must not borrow its sibling's closing +/// sentence. `unsupported_host_gpu_message` can promise that `rocm serve` runs +/// on Lemonade's Vulkan backend because it is only reached once host detection +/// has itself come back with a target that has no ROCm build. This message is +/// raised from the `--family` / `ROCM_CLI_THEROCK_FAMILY` branches at the top of +/// [`resolve_family`], before any host probe runs, so the rejected target says +/// nothing about the GPU in this machine: a stale env value naming gfx90c +/// refuses just the same on a host whose own GPU is fully ROCm-capable, and +/// there `rocm serve` would run on ROCm/HIP. So it points at `rocm examine`, +/// which reports this host's real support level, instead of asserting one. fn unsupported_requested_target_message(value: &str, source: &str) -> String { format!( "no ROCm SDK build exists for the GPU target `{value}` requested by \ {source}, so `rocm install sdk` has nothing to install for it.\n\ - That GPU is not supported by ROCm; the Vulkan backend can still be \ - used. Nothing was installed and no other family was substituted." + That GPU is {VULKAN_ONLY_SUPPORT_CLAIM}. Nothing was installed and no \ + other family was substituted. \ + Run `rocm examine` to see what this host itself supports." ) } @@ -304,12 +367,67 @@ mod tests { assert_eq!(raw_arch_agreeing_with_family(None, "gfx103X-dgpu"), None); } + /// The gate on both install refusals is the classifier's `VulkanOnly` + /// tier, not a private re-derivation of it. + /// + /// Every case is one `gpu_support_level` answers, so a change there has to + /// come through here: `gfx90c` is the recognized-with-no-ROCm-build tier + /// and is refused; a covered target is `RocmHip` and is not; an + /// unrecognized one is `Unknown`, a gap in this build's catalog rather + /// than a fact about the card, and refusing it would be the exact false + /// claim `GpuSupportLevel`'s doc comment exists to prevent. + #[test] + fn the_install_refusal_gate_is_the_classifiers_vulkan_only_tier() { + for refused in ["gfx90c", "GFX90C", "gfx90c:xnack-", " gfx90c "] { + assert!( + target_is_vulkan_only(refused), + "`{refused}` is the Vulkan-only tier and install must refuse it" + ); + } + for allowed in ["gfx1100", "gfx1151", "gfx90a", "gfx950-dcgpu"] { + assert!( + !target_is_vulkan_only(allowed), + "`{allowed}` has a ROCm build; install must not refuse it" + ); + } + for unknown in ["gfx803", "not-a-target", ""] { + assert!( + !target_is_vulkan_only(unknown), + "`{unknown}` is unrecognized, which is a gap in this catalog and \ + not a confirmed absence of a ROCm build: refusing it would claim \ + something about hardware nobody probed" + ); + } + } + + /// Both refusals and `rocm examine`'s `rocm_support` line make the same + /// claim about a Vulkan-only GPU, so they render it from the same string. + /// + /// Asserting the constant is *in* each message is what makes it one + /// claim; the wording itself is pinned word-for-word in `rocm-core` + /// (`the_vulkan_only_claim_is_pinned_word_for_word`), so neither test + /// alone can be satisfied by quietly rewriting the other's expectation. + #[test] + fn both_install_refusals_render_the_shared_vulkan_only_claim() { + for message in [ + unsupported_host_gpu_message("gfx90c"), + unsupported_requested_target_message("gfx90c", "--family"), + ] { + assert!( + message.contains(VULKAN_ONLY_SUPPORT_CLAIM), + "the claim must come from the shared constant, or install and \ + examine can drift apart while both look right: {message}" + ); + } + } + #[test] fn unsupported_gpu_error_names_the_gpu_and_does_not_suggest_families() { let message = unsupported_host_gpu_message("gfx90c"); assert!(message.contains("no ROCm SDK build"), "{message}"); assert!(message.contains("`gfx90c`"), "{message}"); assert!(message.contains("Vulkan"), "{message}"); + assert!(message.contains("rocm serve "), "{message}"); assert!(!message.contains("--family"), "{message}"); for family in known_therock_families() { assert!( @@ -319,6 +437,79 @@ mod tests { } } + /// Knowing the host has no ROCm build is not knowing which engine `serve` + /// would pick. + /// + /// `select_serve_engine` ranks a configured `default_engine` above the + /// host's own preference, and `rocm config set-default-engine` never + /// validates that choice against the GPU. So on a gfx90c host whose config + /// says `vllm`, a bare `rocm serve ` attempts vLLM and fails under + /// the no-CPU-fallback policy — while this message promised Lemonade's + /// Vulkan backend. Pinning `--engine lemonade` makes the sentence true for + /// every config, because an explicit `--engine` out-ranks all of it. + #[test] + fn unsupported_gpu_error_pins_the_engine_a_configured_default_could_override() { + let message = unsupported_host_gpu_message("gfx90c"); + + assert!( + message.contains("rocm serve --engine lemonade"), + "the remediation must name the engine it promises, or a configured \ + `default_engine` makes it false: {message}" + ); + // Guards the exact regression: the bare command with the claim still + // attached. `--engine lemonade` keeps `rocm serve ` as a prefix, + // so the check has to look at what directly follows it. + assert!( + !message.contains("rocm serve ` "), + "a bare `rocm serve ` cannot carry this promise: {message}" + ); + } + + /// The requested-target refusal must not promise which backend `rocm serve` + /// will use, because it never looked at this host. + /// + /// Both refusals used to close with the same "`rocm serve ` runs on + /// Lemonade's Vulkan backend" sentence. In `unsupported_host_gpu_message` + /// that is a fact: it is only reached after `detect_host_gfx_target` returns + /// a target `target_is_vulkan_only` confirms. This one fires from the + /// `--family` / `ROCM_CLI_THEROCK_FAMILY` branches before any probe runs — + /// see `cli_family_overrides_a_stale_env_value_naming_an_unsupported_gpu` + /// for an override that deliberately disagrees with its host — so on a + /// ROCm-capable machine it told the user Vulkan when serving would really + /// run on ROCm/HIP. It now points at `rocm examine` instead, which is true + /// on every host and still leaves somewhere to go. + #[test] + fn requested_target_error_does_not_claim_which_backend_serving_will_use() { + for source in ["--family", "ROCM_CLI_THEROCK_FAMILY"] { + let message = unsupported_requested_target_message("gfx90c", source); + assert!( + !message.contains("rocm serve"), + "the requested target says nothing about this host's GPU, so the \ + refusal must not predict what `rocm serve` would run on: {message}" + ); + assert!( + !message.contains("Lemonade"), + "naming the engine makes the same unchecked promise about this \ + host as naming the command: {message}" + ); + assert!( + message.contains("rocm examine"), + "dropping the claim must still leave the user a next step: {message}" + ); + } + + // The sibling path did look at this host, so it keeps the claim: this is + // the asymmetry, not a blanket ban on the sentence. It pins the engine + // for a different reason — see + // `unsupported_gpu_error_pins_the_engine_a_configured_default_could_override`. + let detected = unsupported_host_gpu_message("gfx90c"); + assert!( + detected.contains("rocm serve --engine lemonade"), + "{detected}" + ); + assert!(detected.contains("Vulkan"), "{detected}"); + } + /// Asking for gfx90c by name must fail, not quietly install for some other /// GPU. /// diff --git a/apps/rocmd/src/anthropic_proxy.rs b/apps/rocmd/src/anthropic_proxy.rs index 73df2dceb..51937bb6d 100644 --- a/apps/rocmd/src/anthropic_proxy.rs +++ b/apps/rocmd/src/anthropic_proxy.rs @@ -46,9 +46,8 @@ use axum::response::{IntoResponse, Response}; use axum::routing::{get, post}; use axum::{Json, Router}; use rocm_core::{ - ANTHROPIC_ENDPOINT_MODE_STATE_KEY, ANTHROPIC_ENDPOINT_URL_STATE_KEY, AnthropicEndpointMode, - JsonStateFileRefused, format_http_base_url, http_get_with_auth, http_post_json_with_auth, - merge_json_state_file, openai_models_endpoint_has_model, + AnthropicEndpointMode, JsonStateFileRefused, format_http_base_url, http_get_with_auth, + http_post_json_with_auth, openai_models_endpoint_has_model, publish_anthropic_endpoint, }; use serde_json::{Value, json}; @@ -153,11 +152,17 @@ const PUBLISH_RETRY_PAUSE: Duration = Duration::from_millis(200); /// Record the endpoint in the engine's state file. /// -/// `merge_json_state_file` refuses a state file it cannot merge into rather -/// than replace it. The vLLM and AIM engines still rewrite their state files -/// in place, so a file refused as empty or not valid JSON can be one caught -/// mid-write and whole again a moment later: that refusal is tried again, up -/// to [`PUBLISH_ATTEMPTS`] attempts [`PUBLISH_RETRY_PAUSE`] apart. Any other +/// Through `rocm_core::publish_anthropic_endpoint`, not a bare merge: the +/// engine adapter running in this same process rewrites that file *whole* a +/// moment after it spawns the server, and only a publication that function has +/// noted survives such a rewrite. See its documentation for why a bare merge +/// loses the endpoint for good. +/// +/// It refuses a state file it cannot merge into rather than replace it. The +/// vLLM and AIM engines still rewrite their state files in place, so a file +/// refused as empty or not valid JSON can be one caught mid-write and whole +/// again a moment later: that refusal is tried again, up to +/// [`PUBLISH_ATTEMPTS`] attempts [`PUBLISH_RETRY_PAUSE`] apart. Any other /// failure (a whole document that is not an object, or failing to lock, read /// or write the file) is returned at once. fn publish( @@ -178,13 +183,9 @@ fn publish_with_pause( ) -> anyhow::Result<()> { // The listener URL carries no `/v1`: Anthropic SDKs append `/v1/messages`. let base = url.trim_end_matches('/').trim_end_matches("/v1"); - let patch = json!({ - ANTHROPIC_ENDPOINT_URL_STATE_KEY: base, - ANTHROPIC_ENDPOINT_MODE_STATE_KEY: mode.as_str(), - }); let mut attempt = 1; loop { - let error = match merge_json_state_file(&config.state_path, &patch) { + let error = match publish_anthropic_endpoint(&config.state_path, base, mode) { Ok(()) => return Ok(()), Err(error) => error, }; diff --git a/crates/rocm-core/src/anthropic_endpoint.rs b/crates/rocm-core/src/anthropic_endpoint.rs index 744ae7d02..93e8eb7a3 100644 --- a/crates/rocm-core/src/anthropic_endpoint.rs +++ b/crates/rocm-core/src/anthropic_endpoint.rs @@ -12,10 +12,29 @@ //! engine publishes the proxy's address under the state-file keys below, and [`crate::ManagedServiceRecord::refresh_from_engine_state`] adopts //! it, so every reader (`rocm serve`, `rocm services list`, the readiness probe) //! sees the same endpoint. +//! +//! The publisher is a *thread of the engine's own process*, not a separate +//! writer: `rocm`'s `run_builtin_engine_serve_http` starts the proxy and then +//! runs the engine adapter in the same process. Both write the one engine state +//! file, and the engine adapters write it whole. [`publish_anthropic_endpoint`] +//! and `carry_published_anthropic_endpoint` are the two halves that keep the +//! whole-file write from dropping what the sibling thread published. +//! +//! The carry reaches a whole-file write only when that write goes through +//! [`crate::write_json_state_file`], which is +//! `carry_published_anthropic_endpoint`'s single caller. All three built-in +//! adapters' `write_state` do (`engines/lemonade`, `engines/vllm`, +//! `engines/aim`). An out-of-tree engine plugin that writes the state file +//! itself takes neither this nor the state directory's lock, and would drop +//! the publication. -use std::path::Path; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use std::sync::{Mutex, OnceLock, PoisonError}; use std::time::{Duration, Instant}; +use anyhow::Result; + use crate::openai_models_endpoint_has_model; /// Engine state-file key naming the Anthropic-compatible endpoint's base URL. @@ -56,6 +75,117 @@ pub fn anthropic_endpoint_ready( .unwrap_or(false) } +/// What *this process* has published, keyed by the engine state file it went +/// into, so [`carry_published_anthropic_endpoint`] can put it back. +/// +/// Deliberately process-local rather than read back off disk. A previous run of +/// the same service id left its own endpoint in that file, and that URL is dead +/// — its proxy died with the process that bound the port. The first state write +/// of a new run is what clears it (`ManagedServiceRecord::reset_for_restart` +/// relies on exactly that), so a disk-backed carry-over would resurrect a dead +/// endpoint on every restart. Only the publication this process made is ours to +/// preserve. +/// +/// Keyed by path rather than held as a single value because the key is per +/// service: nothing today runs two engines in one process, but a map costs +/// nothing and a global would quietly cross-contaminate if one ever did (and +/// does cross-contaminate between tests sharing a process, which is the same +/// bug in miniature). +static PUBLISHED_ENDPOINTS: OnceLock>> = OnceLock::new(); + +fn published_endpoints() -> &'static Mutex> { + PUBLISHED_ENDPOINTS.get_or_init(|| Mutex::new(HashMap::new())) +} + +/// Record `base_url` as this service's Anthropic endpoint in `state_path`, and +/// remember it for the rest of this process. +/// +/// `base_url` carries no `/v1`: Anthropic SDKs append `/v1/messages` to it. +/// +/// The in-memory note is taken *before* the file is touched, and the note's +/// mutex is released before the file lock is asked for. Both matter: +/// +/// * noting first leaves no window — a whole-file rewrite that gets the file +/// lock first already sees the note and carries the endpoint, and one that +/// gets it after the merge below is carrying it too; +/// * releasing first keeps the two locks from ever being held in both orders, +/// since `carry_published_anthropic_endpoint` is called with the file lock +/// held. +/// +/// Called more than once for one endpoint when the merge has to be retried +/// (`rocmd`'s proxy retries a state file caught mid-write); the note is the +/// same each time. +pub fn publish_anthropic_endpoint( + state_path: &Path, + base_url: &str, + mode: AnthropicEndpointMode, +) -> Result<()> { + { + let mut published = published_endpoints() + .lock() + .unwrap_or_else(PoisonError::into_inner); + published.insert( + state_path.to_path_buf(), + (base_url.to_owned(), mode.as_str().to_owned()), + ); + } + crate::merge_json_state_file( + state_path, + &serde_json::json!({ + ANTHROPIC_ENDPOINT_URL_STATE_KEY: base_url, + ANTHROPIC_ENDPOINT_MODE_STATE_KEY: mode.as_str(), + }), + ) +} + +/// `state`, with the Anthropic endpoint this process published for `state_path` +/// put back into it — or `None` when there is nothing to put back, so the +/// caller writes the document it already has and pays no clone. +/// +/// [`crate::write_json_state_file`] is the only caller. A full rewrite replaces +/// every key, by design: it serializes a document its caller built rather than +/// patching the file, which is how a new run of a service clears what the +/// previous run left. The one thing it must not replace is a publication made +/// by *its own process*, because the publisher is a thread of that process +/// (`rocm`'s `run_builtin_engine_serve_http` starts the Anthropic proxy and +/// then runs the engine adapter) and nothing publishes a second time. +/// +/// This is not a theoretical race. The engine adapter's post-spawn `running` +/// write happens immediately after the server process is started, and since +/// ROCMAI-550 it spends a host GPU probe (`crate::pinned_gpu_name`: `pnputil` +/// plus a PowerShell CIM query on Windows, seconds) building the document +/// before writing it. That is long enough for the server to come up, for the +/// proxy's poll to see a listed model, and for the publication to land inside +/// the window and be replaced — leaving a ready service that never shows an +/// `anthropic endpoint:` row. +/// +/// The cross-process lock [`crate::merge_json_state_file`] and +/// [`crate::write_json_state_file`] share does not cover this. It orders the +/// two writers so neither sees a half-written file; it does not stop the +/// later one replacing what the earlier one recorded, which is exactly what a +/// full rewrite is for. +pub(crate) fn carry_published_anthropic_endpoint( + state_path: &Path, + state: &serde_json::Value, +) -> Option { + let (url, mode) = published_endpoints() + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(state_path) + .cloned()?; + let mut carried = state.clone(); + let object = carried.as_object_mut()?; + object.insert( + ANTHROPIC_ENDPOINT_URL_STATE_KEY.to_owned(), + serde_json::Value::String(url), + ); + object.insert( + ANTHROPIC_ENDPOINT_MODE_STATE_KEY.to_owned(), + serde_json::Value::String(mode), + ); + Some(carried) +} + /// The Anthropic endpoint the serving process has published in `state_path`, /// waiting up to `wait` for it to appear. /// @@ -120,4 +250,123 @@ mod tests { fn mode_strings_are_the_values_published_in_state() { assert_eq!(AnthropicEndpointMode::Proxy.as_str(), "proxy"); } + + /// The regression ROCMAI-550's Windows lane hit, driven through the real + /// writers: publish the endpoint, then have the engine adapter rewrite the + /// state file whole — which is what `write_running_state` does right after + /// it spawns the server, with a multi-second host GPU probe inside it. + /// + /// The rewritten document is deliberately *not* a copy of what is on disk: + /// a "fix" that merely re-read the file before writing would not be what + /// the adapters do, and would not satisfy this. + #[test] + fn a_whole_file_state_write_keeps_the_endpoint_this_process_published() { + let dir = unique_dir("carry"); + let state_path = dir.join("state.json"); + crate::write_json_state_file(&state_path, &serde_json::json!({ "status": "starting" })) + .unwrap(); + publish_anthropic_endpoint( + &state_path, + "http://127.0.0.1:41000", + AnthropicEndpointMode::Proxy, + ) + .unwrap(); + + crate::write_json_state_file( + &state_path, + &serde_json::json!({ "status": "running", "server_pid": 7 }), + ) + .unwrap(); + + let state: serde_json::Value = + serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); + assert_eq!( + state[ANTHROPIC_ENDPOINT_URL_STATE_KEY], "http://127.0.0.1:41000", + "an engine rewriting the whole state file must not drop the \ + endpoint its own process published: nothing publishes twice, so \ + the `anthropic endpoint:` row never comes back" + ); + assert_eq!(state[ANTHROPIC_ENDPOINT_MODE_STATE_KEY], "proxy"); + // The rewrite still replaces everything else, which is what it is for. + assert_eq!(state["status"], "running"); + assert_eq!(state["server_pid"], 7); + let _ = std::fs::remove_dir_all(dir); + } + + /// The carry is scoped to what *this* process published for *this* file. + /// + /// Both halves matter. A previous run of the same service left a dead + /// endpoint in that file — its proxy died with the process that bound the + /// port — and the new run's first whole-file write is what clears it, which + /// `ManagedServiceRecord::reset_for_restart` relies on; carrying anything + /// read off disk would resurrect it. And a process that published for one + /// service must not hand that endpoint to another. + #[test] + fn a_rewrite_still_clears_an_endpoint_this_process_did_not_publish() { + let dir = unique_dir("unpublished"); + let published_to = dir.join("published.json"); + let other = dir.join("other.json"); + publish_anthropic_endpoint( + &published_to, + "http://127.0.0.1:41001", + AnthropicEndpointMode::Proxy, + ) + .unwrap(); + std::fs::write( + &other, + r#"{"anthropic_endpoint_url": "http://127.0.0.1:1", "anthropic_endpoint_mode": "proxy"}"#, + ) + .unwrap(); + + crate::write_json_state_file(&other, &serde_json::json!({ "status": "starting" })).unwrap(); + + let state: serde_json::Value = + serde_json::from_slice(&std::fs::read(&other).unwrap()).unwrap(); + assert_eq!( + state, + serde_json::json!({ "status": "starting" }), + "only this process's own publication for this exact state file may \ + survive a whole-file write; a dead endpoint from a previous run \ + must not be restored" + ); + let _ = std::fs::remove_dir_all(dir); + } + + /// Publishing still writes the keys through to the file, which is what + /// every out-of-process reader (`rocm services list`, `rocm serve`'s + /// summary) actually looks at — the note is an addition, not a substitute. + #[test] + fn publishing_writes_the_endpoint_into_the_state_file_it_names() { + let dir = unique_dir("publish"); + let state_path = dir.join("state.json"); + std::fs::write(&state_path, r#"{"status": "running"}"#).unwrap(); + publish_anthropic_endpoint( + &state_path, + "http://127.0.0.1:41002", + AnthropicEndpointMode::Proxy, + ) + .unwrap(); + assert_eq!( + wait_for_published_anthropic_endpoint(&state_path, Duration::ZERO).as_deref(), + Some("http://127.0.0.1:41002") + ); + let state: serde_json::Value = + serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); + assert_eq!(state[ANTHROPIC_ENDPOINT_MODE_STATE_KEY], "proxy"); + // The merge keeps what was already recorded for the service. + assert_eq!(state["status"], "running"); + let _ = std::fs::remove_dir_all(dir); + } + + /// A directory no other test in this process shares: the publication note + /// is keyed by path and lives for the whole test binary. + fn unique_dir(tag: &str) -> std::path::PathBuf { + let dir = std::env::temp_dir().join(format!( + "rocm-core-anthropic-{tag}-{}-{}", + std::process::id(), + crate::unix_time_millis() + )); + std::fs::create_dir_all(&dir).unwrap(); + dir + } } diff --git a/crates/rocm-core/src/endpoint_readiness.rs b/crates/rocm-core/src/endpoint_readiness.rs index 52895a8be..63335b490 100644 --- a/crates/rocm-core/src/endpoint_readiness.rs +++ b/crates/rocm-core/src/endpoint_readiness.rs @@ -70,8 +70,9 @@ pub fn json_state_lock_path(path: &Path) -> PathBuf { /// run under one cross-process lock ([`json_state_lock_path`]) and no merge /// can drop a key another merge recorded. The lock orders writers rather than /// combining them: a full rewrite through [`write_json_state_file`] still -/// replaces every key merged before it, and a writer that bypasses both -/// functions (the vLLM and AIM engines' own state writes) takes no lock at all. +/// replaces every key merged before it. Every built-in engine adapter's state +/// writes go through one of the two; a writer that bypasses both (an +/// out-of-tree engine plugin writing the file itself) takes no lock at all. /// /// The result is published atomically /// ([`crate::atomic_write::write_file_atomically`]), so a reader that takes no @@ -124,6 +125,16 @@ fn merge_json_state_file_with( /// Takes the same lock as [`merge_json_state_file`] and publishes the same /// atomic way, so a full rewrite cannot land in the middle of another /// writer's merge. +/// +/// It replaces every key merged before it — that is what a full rewrite is for, +/// and how a new run of a service clears what the previous run recorded — with +/// one exception: an Anthropic endpoint *this process* published +/// (`crate::publish_anthropic_endpoint`) is carried into the document first. +/// The publisher is a thread of the same process as the engine adapter calling +/// this, it publishes exactly once, and since ROCMAI-550 the adapter's +/// post-spawn rewrite spends a multi-second host GPU probe building its +/// document — wide enough for the publication to land inside it and be lost. +/// See `anthropic_endpoint::carry_published_anthropic_endpoint`. pub fn write_json_state_file(path: &Path, value: &serde_json::Value) -> Result<()> { write_json_state_file_with(path, value, || {}) } @@ -137,8 +148,15 @@ fn write_json_state_file_with( before_publish: impl FnOnce(), ) -> Result<()> { let _lock = FileLock::acquire(json_state_lock_path(path))?; + // Read inside the file lock, which leaves no gap: a publisher notes the + // endpoint before it asks for this lock, so one that is waiting here has + // already been noted and is carried, and one that has not noted yet merges + // after this rewrite and lands on its own. `publish_anthropic_endpoint` + // releases its in-memory note before taking the file lock, so the two locks + // are never held in both orders. + let carried = crate::anthropic_endpoint::carry_published_anthropic_endpoint(path, value); before_publish(); - publish_json_state(path, value) + publish_json_state(path, carried.as_ref().unwrap_or(value)) } /// Why [`merge_json_state_file`] refused an existing state file. diff --git a/crates/rocm-core/src/gfx_target.rs b/crates/rocm-core/src/gfx_target.rs index 18c429343..a73e36e97 100644 --- a/crates/rocm-core/src/gfx_target.rs +++ b/crates/rocm-core/src/gfx_target.rs @@ -11,6 +11,8 @@ //! the strings its probes read, and [`crate::managed_runtime`] reads the family //! mapping from here. +use crate::host_gpu::HostGpuSummary; + pub fn extract_first_gfx_token(text: &str) -> Option { text.split(|ch: char| !(ch.is_ascii_alphanumeric() || ch == '-' || ch == '_')) .map(str::trim) @@ -549,6 +551,82 @@ pub(crate) fn gfx_target_from_gc_version(major: u32, minor: u32, revision: u32) Some(format!("gfx{major}{minor:x}{revision:x}")) } +/// How well this host's GPU is supported, for messages that tell the user what +/// they can actually do rather than just reporting raw detection fields. +/// +/// `Unknown` deliberately covers both "nothing detected" and "a gfx target we +/// don't recognize" rather than being called `Unsupported` — see the doc +/// comment on [`gfx_target_has_no_rocm_build`] for why conflating "confirmed +/// no ROCm build" with "unrecognized" would turn a detection gap into a false +/// claim about the hardware. Only that predicate's conjunction with +/// [`normalize_therock_family`] establishes the "confirmed" half; an +/// unrecognized target satisfies neither conjunct, so it is a gap in what this +/// catalog knows rather than a fact about the card. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum GpuSupportLevel { + /// A TheRock package family covers this GPU: full ROCm/HIP support. + RocmHip, + /// Recognized, confirmed to have no ROCm/HIP build, but Vulkan-based + /// engines (the default Lemonade backend) still work on it. + VulkanOnly, + /// No GPU detected yet, or a gfx target this catalog doesn't recognize. + Unknown, +} + +/// The one claim every [`GpuSupportLevel::VulkanOnly`] surface makes, written +/// once. +/// +/// `rocm examine`'s `rocm_support` line and `rocm install sdk`'s two refusals +/// (`unsupported_host_gpu_message` / `unsupported_requested_target_message` in +/// `apps/rocm/src/therock/family.rs`) all tell the user the same two things +/// about a Vulkan-only GPU: ROCm will not run on it, and Vulkan will. They are +/// three different sentences around it — each adds its own subject and its own +/// next step, because what they may promise differs — but the claim itself is +/// shared, so it lives here next to the classifier that decides when it is +/// true rather than being retyped at each site. +/// +/// Deliberately just the claim, with no leading subject and no trailing +/// punctuation: a caller supplies both. Changing the wording here changes all +/// three surfaces together, which is the point. +pub const VULKAN_ONLY_SUPPORT_CLAIM: &str = + "not supported by ROCm; the Vulkan backend can still be used"; + +/// Classify the host's GPU support level from the same two facts +/// `ExamineSummary` and `resolve_family` already compute: the detected gfx +/// target and whichever TheRock family (if any) covers it. +/// +/// Not only for a *host*: `resolve_family` also asks this of a target the user +/// named with `--family` or `ROCM_CLI_THEROCK_FAMILY`, to decide whether to +/// refuse it. The two arguments are facts about a GPU target, not about this +/// machine, so that is the same question. What differs is what the caller may +/// then say — a requested target establishes nothing about the hardware here, +/// which is why the two install refusals end differently. +pub fn gpu_support_level( + detected_gfx_target: Option<&str>, + compatible_therock_family: Option<&str>, +) -> GpuSupportLevel { + match (detected_gfx_target, compatible_therock_family) { + (_, Some(_)) => GpuSupportLevel::RocmHip, + (Some(gfx), None) if gfx_target_has_no_rocm_build(gfx) => GpuSupportLevel::VulkanOnly, + _ => GpuSupportLevel::Unknown, + } +} + +/// [`gpu_support_level`] for a [`HostGpuSummary`] the caller already probed. +/// +/// The two fields are read here, once, rather than at each call site: they are +/// positional arguments of the same type on [`gpu_support_level`], so a call +/// site that transposed them would classify a Vulkan-only host as `RocmHip` +/// and still compile. `rocm engines` and the codex-bridge `engines` tool both +/// derive the level from this struct, so both go through this. +#[must_use] +pub fn gpu_support_level_for_host_gpu(host_gpu: &HostGpuSummary) -> GpuSupportLevel { + gpu_support_level( + host_gpu.gfx_target.as_deref(), + host_gpu.therock_family.as_deref(), + ) +} + #[cfg(test)] mod tests { use super::*; @@ -627,6 +705,24 @@ mod tests { ); } + /// The shared claim is user-visible text on three surfaces, so its wording + /// is pinned here rather than only asserted as "whatever the constant + /// says". + /// + /// `rocm examine`'s `rocm_support` line and both `rocm install sdk` + /// refusals now interpolate [`VULKAN_ONLY_SUPPORT_CLAIM`], which is what + /// stops them drifting — but it also means a careless edit to the constant + /// silently rewords all three at once, and their own tests would still + /// pass because they assert containment of the same constant. This test is + /// the one place that has to be changed deliberately. + #[test] + fn the_vulkan_only_claim_is_pinned_word_for_word() { + assert_eq!( + VULKAN_ONLY_SUPPORT_CLAIM, + "not supported by ROCm; the Vulkan backend can still be used" + ); + } + #[test] fn gfx_target_has_no_rocm_build_names_only_gfx90c() { assert!(gfx_target_has_no_rocm_build("gfx90c")); @@ -935,4 +1031,81 @@ mod tests { assert_eq!(parse_linux_kfd_gfx_target("121600"), None); assert_eq!(parse_linux_kfd_gfx_target("not-a-target"), None); } + + #[test] + fn gpu_support_level_for_host_gpu_reads_a_covered_summary_as_rocm_hip() { + let host_gpu = HostGpuSummary { + name: Some("AMD Instinct MI300X".to_owned()), + gfx_target: Some("gfx942".to_owned()), + therock_family: Some("gfx94X-dcgpu".to_owned()), + }; + assert_eq!( + gpu_support_level_for_host_gpu(&host_gpu), + GpuSupportLevel::RocmHip + ); + } + + #[test] + fn gpu_support_level_for_host_gpu_reads_a_vulkan_only_summary_as_vulkan_only() { + let host_gpu = HostGpuSummary { + name: Some("AMD Radeon Graphics".to_owned()), + gfx_target: Some("gfx90c".to_owned()), + therock_family: None, + }; + assert_eq!( + gpu_support_level_for_host_gpu(&host_gpu), + GpuSupportLevel::VulkanOnly + ); + } + + #[test] + fn gpu_support_level_for_host_gpu_reads_an_undetected_summary_as_unknown() { + assert_eq!( + gpu_support_level_for_host_gpu(&HostGpuSummary::default()), + GpuSupportLevel::Unknown + ); + } + + /// A covering TheRock family is the whole answer, which is why that arm + /// ignores `detected_gfx_target` — and `resolve_family` depends on it: + /// it asks this about a target named with `--family`, where there may be + /// no detected target at all. Until the `None`/`Some(_)` case below, every + /// call in this suite passed `Some`/`Some`, `None`/`None` or `Some`/`None`, + /// so the arm could be narrowed to `(Some(_), Some(_))` with nothing + /// noticing. + #[test] + fn gpu_support_level_is_rocm_hip_when_a_family_covers_it() { + assert_eq!( + gpu_support_level(Some("gfx90a"), Some("gfx90a")), + GpuSupportLevel::RocmHip + ); + assert_eq!( + gpu_support_level(None, Some("gfx90a")), + GpuSupportLevel::RocmHip + ); + // Even for a target this file's own table says has no ROCm build: the + // family the caller resolved is a statement about ROCm support, and a + // detected target cannot outvote it. + assert_eq!( + gpu_support_level(Some("gfx90c"), Some("gfx90a")), + GpuSupportLevel::RocmHip + ); + } + + #[test] + fn gpu_support_level_is_unknown_without_detection_or_for_an_unrecognized_target() { + assert_eq!(gpu_support_level(None, None), GpuSupportLevel::Unknown); + assert_eq!( + gpu_support_level(Some("gfx_totally_unrecognized"), None), + GpuSupportLevel::Unknown + ); + } + + #[test] + fn gpu_support_level_is_vulkan_only_for_gfx90c() { + assert_eq!( + gpu_support_level(Some("gfx90c"), None), + GpuSupportLevel::VulkanOnly + ); + } } diff --git a/crates/rocm-core/src/host_gpu.rs b/crates/rocm-core/src/host_gpu.rs index 9a722f8f5..55c356e57 100644 --- a/crates/rocm-core/src/host_gpu.rs +++ b/crates/rocm-core/src/host_gpu.rs @@ -479,6 +479,139 @@ pub fn detect_host_gpu_summary(paths: Option<&AppPaths>) -> HostGpuSummary { detect_host_gpu_summary_fast(paths) } +/// The GPU name an engine may record at launch for a service pinned to +/// `gpu_indices`, for the `device:` row `rocm serve` and `rocm services list` +/// print (ROCMAI-550). +/// +/// `None` whenever the name cannot be attributed to the *pinned* device, which +/// is most of what this function is for. Every name probe on this host — +/// [`detect_host_gpu_summary`] on Linux, the Windows display inventory — answers +/// for the host's primary/preferred adapter, not for an ordinal. On a multi-GPU +/// host that would print `gpu 1 ()`: a row that looks specific +/// and is wrong, which is worse than the bare `gpu 1` it replaced. So the name +/// is reported only when the host has exactly **one** AMD GPU, where "the +/// primary one" and "the pinned one" cannot differ. +/// +/// The three built-in engines all call this instead of probing themselves, so +/// the rule lives in one place and `engines/aim` (which records no ordinal at +/// all) is held to it as well. +/// +/// Why not map the ordinal to a device properly: the ordinals in `gpu_indices` +/// are HIP ordinals *after* the visibility mask +/// ([`usable_amd_gpu_indices`]), so ordinal 0 under `ROCR_VISIBLE_DEVICES=1` +/// is the host's second card. Attributing a name would mean enumerating +/// devices in HIP order and re-applying that mask, and neither KFD topology +/// nodes nor DRM cards carry a marketing name — so it is a real probe to build +/// and one no lane here can verify (none of the CI hosts has two AMD GPUs). +/// Until that exists, a mask in effect is treated exactly like a second GPU: +/// no name. +#[must_use] +pub fn pinned_gpu_name(gpu_indices: &[u32]) -> Option { + // One Windows display-inventory read per call, shared by the two questions + // that need it. Both the count and the name come from the same `pnputil` / + // CIM query, and asking twice meant two PowerShell/WMI probes on the serve + // launch path. The read is eager rather than lazy only on the platform that + // uses it: off Windows nothing is probed here at all, and on Windows the + // count already needed it before any of the decisions below could run. + // + // The cost, stated rather than implied: this + // deduplicates *within* a call and is **not** cached across calls. Each + // engine's `write_running_state` calls it, and that runs more than once per + // launch (`starting`, then `running`), so a Windows launch pays that many + // PowerShell/CIM probes. It is on the launch path only — no request path + // reaches it — which is why the cost is accepted rather than cached: a + // cache would have to be invalidated on GPU hotplug to stay correct, and no + // caller is hot enough to earn that. Unverified on Windows; nothing + // available to this change can build the MSVC target. + let windows_inventory = runtime_is_windows() + .then(detect_windows_examine_inventory) + .flatten(); + gpu_name_for_single_gpu_host( + gpu_indices, + amd_gpu_count_for_naming(windows_inventory.as_ref()), + gpu_visibility_mask_in_effect(), + || host_primary_gpu_name(windows_inventory.as_ref()), + ) +} + +/// The decision [`pinned_gpu_name`] makes, with the host probes lifted out. +/// +/// Split out because every input here is a host fact a unit test cannot plant +/// (a second GPU, a Windows display inventory), while the rule itself — "a name +/// only when nothing else on this host could be the one that was pinned" — is +/// the whole point and is exactly what a regression would silently undo. +fn gpu_name_for_single_gpu_host( + gpu_indices: &[u32], + present_amd_gpus: Option, + mask_in_effect: bool, + name: F, +) -> Option +where + F: FnOnce() -> Option, +{ + // A pinned ordinal other than 0 cannot be the only GPU on the host, so it + // is already outside what the probes can attribute. Checked separately from + // the count so a disagreement between the two (a `--gpu 1` that got past + // validation on a host now reporting one GPU) also declines to name. + if gpu_indices.iter().any(|&index| index != 0) { + return None; + } + if mask_in_effect || present_amd_gpus != Some(1) { + return None; + } + name() +} + +/// How many AMD GPUs this host presents, for [`pinned_gpu_name`]'s +/// "could this name belong to another card?" test. `None` when it cannot be +/// established, which declines the name rather than guessing. +/// +/// `windows_inventory` is the display listing the caller already read, or +/// `None` off Windows (and when the listing failed); see [`pinned_gpu_name`] +/// for why it is passed in rather than read here. +fn amd_gpu_count_for_naming(windows_inventory: Option<&WindowsExamineInventory>) -> Option { + #[cfg(target_os = "linux")] + { + if !runtime_is_windows() { + // WSL2 exposes no per-device topology; the ROCm-over-dxg path + // reaches a single device, which is what `usable_amd_gpu_indices` + // reports there too. + if is_wsl_host() { + return Some(1); + } + return gpu_visibility::combine_amd_gpu_counts( + linux::linux_kfd_gpu_node_count(), + linux::linux_drm_amdgpu_card_count(), + ); + } + } + windows_inventory.map(WindowsExamineInventory::amd_display_count) +} + +/// The host's primary/preferred AMD GPU name — correct only on a host with one +/// AMD GPU, which is why [`pinned_gpu_name`] is the only caller. +fn host_primary_gpu_name(windows_inventory: Option<&WindowsExamineInventory>) -> Option { + if runtime_is_windows() { + return windows_inventory.and_then(WindowsExamineInventory::amd_display_name); + } + detect_host_gpu_summary(None).name +} + +/// Whether a GPU visibility mask is narrowing what the engine can see, so a +/// HIP ordinal no longer indexes the host's own device order. +#[cfg(target_os = "linux")] +fn gpu_visibility_mask_in_effect() -> bool { + gpu_visibility::visibility_mask_in_effect() +} + +/// Platform stub: the mask readers are Linux-only (see +/// [`visibility_mask_from_env`]), so nothing here can observe a mask. A mask +/// reader for another platform must extend this rather than leave it `false`. +#[cfg(not(target_os = "linux"))] +const fn gpu_visibility_mask_in_effect() -> bool { + false +} + #[cfg(windows)] fn detect_host_gpu_summary_fast(_paths: Option<&AppPaths>) -> HostGpuSummary { let windows_inventory = detect_windows_examine_inventory(); @@ -832,6 +965,82 @@ mod tests { } } + #[test] + fn a_second_gpu_on_the_host_withholds_the_name() { + let mut probed = false; + let name = gpu_name_for_single_gpu_host(&[1], Some(2), false, || { + probed = true; + Some("AMD Instinct MI300X".to_owned()) + }); + assert_eq!(name, None); + assert!( + !probed, + "a name that cannot be attributed must not be probed" + ); + + // Ordinal 0 on that same host is no safer: the probe answers for the + // host's primary adapter, which the mask/enumeration order need not + // agree with. + assert_eq!( + gpu_name_for_single_gpu_host(&[0], Some(2), false, || Some("AMD".to_owned())), + None + ); + } + + /// The ordinal guard on its own, with the host count agreeing there is one + /// GPU — the disagreement its doc comment names (a `--gpu 1` that got past + /// validation on a host now reporting a single card). + /// + /// Every other case here supplies a count that already declines, so + /// deleting the ordinal branch left the whole suite green. This is the + /// only call that reaches it. + #[test] + fn a_pinned_ordinal_the_host_count_contradicts_withholds_the_name() { + let mut probed = false; + let name = gpu_name_for_single_gpu_host(&[1], Some(1), false, || { + probed = true; + Some("AMD Instinct MI300X".to_owned()) + }); + assert_eq!(name, None); + assert!( + !probed, + "an ordinal the host's own count contradicts must not be probed" + ); + } + + #[test] + fn a_single_unmasked_gpu_is_named() { + for indices in [&[][..], &[0][..]] { + assert_eq!( + gpu_name_for_single_gpu_host(indices, Some(1), false, || Some( + "AMD Radeon Graphics".to_owned() + )), + Some("AMD Radeon Graphics".to_owned()) + ); + } + } + + #[test] + fn a_visibility_mask_withholds_the_name() { + assert_eq!( + gpu_name_for_single_gpu_host(&[0], Some(1), true, || Some("AMD".to_owned())), + None + ); + } + + #[test] + fn an_unknown_gpu_count_withholds_the_name() { + assert_eq!( + gpu_name_for_single_gpu_host(&[0], None, false, || Some("AMD".to_owned())), + None + ); + // And a host with no AMD GPU at all has nothing to name. + assert_eq!( + gpu_name_for_single_gpu_host(&[], Some(0), false, || Some("AMD".to_owned())), + None + ); + } + /// The summary takes the examination's kernel release on Linux without a /// second read, and an empty one is a failed read, not a release. Off Linux, /// or with no examination, it reads for itself. The reads are counted, so a diff --git a/crates/rocm-core/src/host_gpu/gpu_visibility.rs b/crates/rocm-core/src/host_gpu/gpu_visibility.rs index 9fb70cef4..8d8fdcacf 100644 --- a/crates/rocm-core/src/host_gpu/gpu_visibility.rs +++ b/crates/rocm-core/src/host_gpu/gpu_visibility.rs @@ -184,7 +184,7 @@ fn windows_gpu_listing_from(adapters: usize, hip_mask: Option) -> Window /// /// `None` only when NEITHER surface could be read (availability truly unknown). #[cfg(any(target_os = "linux", test))] -fn combine_amd_gpu_counts(kfd: Option, drm: Option) -> Option { +pub(super) fn combine_amd_gpu_counts(kfd: Option, drm: Option) -> Option { match kfd { // KFD is compute-authoritative: prefer it whenever it sees a GPU. Some(k) if k > 0 => Some(k), @@ -305,9 +305,110 @@ fn usable_amd_gpu_indices_from( mask_tokens_within(hip_space, hip) } +/// Whether any GPU visibility mask is set in the environment. +/// +/// Linux-only, mirroring [`visibility_mask_from_env`]: no other platform has a +/// mask reader here, so there is nothing for this to answer about. +/// +/// A `bool` wrapper so callers outside this module can ask the question +/// without `GpuVisibilityMask` — the mask's shape is this module's business, +/// and exposing it only to answer "is one set?" would widen the type's +/// visibility for nothing. Read by `pinned_gpu_name`, which refuses to name a +/// GPU when a mask is in effect: the host probes answer for the primary +/// adapter, and a mask can make the pinned ordinal refer to a different one. +#[cfg(target_os = "linux")] +pub(super) fn visibility_mask_in_effect() -> bool { + visibility_mask_from_env().is_some() +} + #[cfg(test)] mod tests { use super::*; + use crate::gfx_target::gfx_target_has_no_rocm_build; + use crate::test_support::temp_app_paths; + use std::fs; + + /// ROCMAI-550, and the one claim the Lemonade remediation rests on: the host + /// that gets told `rocm serve --engine lemonade` is a host + /// `rocm serve` will actually launch on. + /// + /// The shape is a Renoir/Cezanne laptop: the Vega iGPU is on the bus and + /// bound to `amdgpu`, but the kernel's KFD topology lists no compute node + /// for it, so `gfx90c` can only have been named by the DRM `ip_discovery` + /// read. `apps/rocm/src/therock/family.rs` prints + /// `unsupported_host_gpu_message` for exactly that target, and + /// `apps/rocm/src/serve_cmd.rs`'s `host_has_no_usable_gpu` — + /// `matches!(visible, Some([]))` over [`usable_amd_gpu_indices`] — decides + /// whether the command it names is refused. Both halves are driven here off + /// **one** planted `/sys`, so this is the composition and not two + /// independent claims: KFD contributes `Some(0)`, DRM contributes `Some(1)`, + /// and [`combine_amd_gpu_counts`]'s zero-KFD fallback is what turns that + /// into a usable ordinal. Delete that fallback and the detection half still + /// passes while the pre-flight half goes red, which is the failure the + /// reviewer asked to see ruled out. + /// + /// Planted rather than run on hardware because no CI lane has a gfx90c APU; + /// see `linux::linux_drm_amdgpu_card_count_in`. + #[test] + fn a_gfx90c_host_the_kernel_gives_no_compute_node_still_has_a_usable_gpu() { + let (root, _paths) = temp_app_paths("gfx90c-drm-only-host"); + let drm = root.join("sys/class/drm"); + let device = drm.join("card0/device"); + fs::create_dir_all(&device).expect("plant the DRM card"); + fs::write(device.join("vendor"), "0x1002\n").expect("plant the PCI vendor"); + // GC 9.0.12 is the Vega iGPU in Renoir / Lucienne / Cezanne / Barcelo. + let gc = device.join("ip_discovery/die/0/GC/0"); + fs::create_dir_all(&gc).expect("plant the GC IP block"); + for (file, value) in [("major", "9"), ("minor", "0"), ("revision", "12")] { + fs::write(gc.join(file), value).expect("plant the GC IP version"); + } + // A connector sub-node, which is not a card and must not be counted. + fs::create_dir_all(drm.join("card0-eDP-1")).expect("plant the DRM connector"); + // The KFD topology is readable and lists only the CPU node: this host's + // iGPU has no compute node, which is the whole premise. + let nodes = root.join("sys/class/kfd/kfd/topology/nodes"); + fs::create_dir_all(nodes.join("0")).expect("plant the KFD CPU node"); + fs::write( + nodes.join("0").join("properties"), + "cpu_cores_count 16\nsimd_count 0\ngfx_target_version 0\n", + ) + .expect("plant the KFD CPU node properties"); + + let detected = super::super::linux::detect_linux_drm_ip_discovery_gfx_target_in(&drm); + let kfd = super::super::linux::linux_kfd_gpu_node_count_in(&nodes); + let cards = super::super::linux::linux_drm_amdgpu_card_count_in(&drm); + fs::remove_dir_all(&root).ok(); + + // Half one: this is a host `unsupported_host_gpu_message` speaks for. + assert_eq!( + detected.as_deref(), + Some("gfx90c"), + "the DRM ip-discovery read is the only route left on a host with no \ + KFD compute node; without it there is no detected target and the \ + Lemonade advice is never printed" + ); + assert!(gfx_target_has_no_rocm_build("gfx90c")); + + // Half two: and `rocm serve`'s pre-flight does not refuse it. + assert_eq!( + (kfd, cards), + (Some(0), Some(1)), + "the kernel must report no compute node and the bus exactly one card" + ); + let present = combine_amd_gpu_counts(kfd, cards); + let usable = usable_amd_gpu_indices_from(present.expect("GPU presence"), None); + assert_eq!( + usable, + Some(vec![0]), + "serve pins HIP ordinal 0 on this host" + ); + assert!( + !matches!(usable.as_deref(), Some([])), + "`serve_cmd::host_has_no_usable_gpu` is `matches!(visible, Some([]))`; \ + anything else here means the advised command is refused for the very \ + condition it was printed to clear" + ); + } fn wsl_summary(librocdxg_in_cache: bool, on_ld_library_path: bool) -> super::super::WslSummary { super::super::WslSummary { diff --git a/crates/rocm-core/src/host_gpu/linux.rs b/crates/rocm-core/src/host_gpu/linux.rs index 701071d85..47dd97682 100644 --- a/crates/rocm-core/src/host_gpu/linux.rs +++ b/crates/rocm-core/src/host_gpu/linux.rs @@ -204,7 +204,21 @@ fn natural_node_order(name: &str) -> (u64, String) { /// DRM class dir can't be read; `Some(0)` when it is readable with no AMD card. #[cfg(target_os = "linux")] pub(super) fn linux_drm_amdgpu_card_count() -> Option { - let entries = fs::read_dir(host_path("/sys/class/drm")).ok()?; + linux_drm_amdgpu_card_count_in(&host_path("/sys/class/drm")) +} + +/// The DRM card count, against a caller-supplied `/sys/class/drm`. +/// +/// Split out for the same reason as `linux_kfd_gpu_node_count_in`: this half of +/// the serve pre-flight's "is there a GPU here" answer is what carries a host +/// whose KFD reports no compute node — the gfx90c APUs +/// `unsupported_host_gpu_message` sends to Lemonade among them — and no CI lane +/// has that hardware. Planting the directory is the only way the composed +/// answer can be asserted at all; see +/// `a_gfx90c_host_the_kernel_gives_no_compute_node_still_has_a_usable_gpu`. +#[cfg(any(target_os = "linux", test))] +pub(super) fn linux_drm_amdgpu_card_count_in(class_dir: &Path) -> Option { + let entries = fs::read_dir(class_dir).ok()?; let count = entries .flatten() .filter(|entry| { @@ -236,7 +250,7 @@ pub(crate) fn linux_kfd_gpu_node_count() -> Option { /// branch is driven by tests -- the unreadable branches key off the real /// `/dev/kfd`, which a planted directory cannot stand in for. #[cfg(any(target_os = "linux", test))] -fn linux_kfd_gpu_node_count_in(nodes_dir: &Path) -> Option { +pub(super) fn linux_kfd_gpu_node_count_in(nodes_dir: &Path) -> Option { match fs::read_dir(nodes_dir) { Ok(entries) => Some( entries @@ -305,8 +319,22 @@ fn kfd_gfx_target_version_is_gpu(value: &str) -> bool { #[cfg(target_os = "linux")] fn detect_linux_drm_ip_discovery_gfx_target() -> Option { - let drm_dir = host_path("/sys/class/drm"); - let entries = fs::read_dir(drm_dir).ok()?; + detect_linux_drm_ip_discovery_gfx_target_in(&host_path("/sys/class/drm")) +} + +/// The DRM ip-discovery target read, against a caller-supplied `/sys/class/drm`. +/// +/// Split out for the same reason as [`linux_drm_amdgpu_card_count_in`], and +/// against the *same* directory on purpose: on a host whose KFD exposes no +/// compute node this is the only route left that can name a target, so it is +/// what puts a GPU in front of `unsupported_host_gpu_message` — and it walks +/// `card*/device` under `is_amdgpu_device`, exactly the cards that count then +/// hands to the serve pre-flight. Pointing one planted tree at both is what +/// makes "the host that gets the Lemonade advice is a host serve accepts" +/// assertable rather than argued. +#[cfg(any(target_os = "linux", test))] +pub(super) fn detect_linux_drm_ip_discovery_gfx_target_in(class_dir: &Path) -> Option { + let entries = fs::read_dir(class_dir).ok()?; for entry in entries.flatten() { let card_path = entry.path(); let Some(card_name) = card_path.file_name().and_then(|value| value.to_str()) else { diff --git a/crates/rocm-core/src/host_gpu/render.rs b/crates/rocm-core/src/host_gpu/render.rs index 09d6626ea..125c9f384 100644 --- a/crates/rocm-core/src/host_gpu/render.rs +++ b/crates/rocm-core/src/host_gpu/render.rs @@ -6,25 +6,144 @@ use super::ExamineSummary; use super::os_facts::format_gib_value; -use crate::gfx_target::gfx_target_has_no_rocm_build; +use crate::gfx_target::{GpuSupportLevel, VULKAN_ONLY_SUPPORT_CLAIM, gpu_support_level}; impl ExamineSummary { - /// The `rocm_support` line of the examine text, or an empty string. + /// The `rocm_support` line of the examine text. /// - /// Present only when a GPU was detected but no ROCm package family covers - /// it (for example the gfx90c Vega iGPU in Renoir / Cezanne APUs). Without - /// it `compatible_therock_family: ` reads like a probe failure, when - /// the GPU was in fact recognized and simply has no ROCm build. + /// ROCMAI-550 asks `rocm examine` to "report the support level with one + /// line of what the user can do", so every level gets a line — including + /// the ones where the answer is "this build cannot tell". The earlier draft + /// printed only the `VulkanOnly` line, which made this report unable to + /// answer the very question the other surfaces send people here to ask: + /// `lemonade_engine_note` in `apps/rocm/src/main.rs` tells an `Unknown` + /// host to run `rocm examine`, and silence there is not an answer. + /// + /// Without the `VulkanOnly` line specifically, + /// `compatible_therock_family: ` reads like a probe failure when + /// the GPU was in fact recognized and simply has no ROCm build (the gfx90c + /// Vega iGPU in Renoir / Cezanne APUs). The `Unknown` line exists for the + /// mirror-image misreading: the same `` value, where the CLI + /// genuinely does not know, must not be read as either claim. + /// + /// Classified by [`gpu_support_level`] rather than re-deriving the same + /// `(detected, covering family)` test here: this line and that function are + /// two renderings of one judgement, and the whole point of this change is + /// that support claims come from a single place. Every variant is spelled + /// out so that adding one is a compile error here rather than a silently + /// blank line. fn rocm_support_line(&self) -> String { - match ( - self.detected_gfx_target.as_deref(), - self.compatible_therock_family.as_deref(), - ) { - (Some(gfx), None) if gfx_target_has_no_rocm_build(gfx) => format!( - " rocm_support: GPU detected ({gfx}) but not supported by ROCm; \ - the Vulkan backend can still be used\n" + let detected = self.detected_gfx_target.as_deref(); + match gpu_support_level(detected, self.compatible_therock_family.as_deref()) { + // `VulkanOnly` is only ever returned for a detected target, so the + // fallback below is unreachable; it keeps the arm total without a + // panic path. + GpuSupportLevel::VulkanOnly => format!( + " rocm_support: GPU detected ({}) but {VULKAN_ONLY_SUPPORT_CLAIM} - \ + serve on it with `rocm serve --engine lemonade`\n", + detected.unwrap_or_default() ), - _ => String::new(), + // The covering family is the fact; it is already printed on its own + // line above, so this one says what it means rather than repeating + // it. + // + // The advice is conditional because AGENTS.md wants a named command + // to be able to clear the condition it is printed for. On a host + // that already has a managed SDK, `rocm install sdk` clears + // nothing — it is the install that is already done — so the line + // says so and points at what is left to do instead. + // `detected_therock_family` is the installed managed runtime (see + // `detect_managed_therock_family`), as opposed to + // `compatible_therock_family`, which is what this GPU *could* run. + // + // The two are computed from unrelated facts and really can differ. + // `compatible_therock_family` is `normalize_therock_family` applied + // to the gfx target detected right now; `detect_managed_therock_family` + // is the family of the most recently installed managed runtime on + // disk and never looks at the GPU at all. Swap the card, plug in an + // eGPU, or keep two cards and boot the other one, and the install + // on disk is for a family this GPU was never built for. Saying + // "already installed - serve on it" there would promise a serve + // this host cannot do, which is exactly the kind of unestablished + // claim ROCMAI-550 exists to remove, so the match is a condition of + // that sentence rather than an assumption behind it. + GpuSupportLevel::RocmHip => { + let compatible = self.compatible_therock_family.as_deref(); + match self.detected_therock_family.as_deref() { + Some(installed) if Some(installed) == compatible => format!( + " rocm_support: GPU detected ({}) with full ROCm/HIP support, and the \ + managed ROCm SDK for {installed} is already installed - serve on it with \ + `rocm serve `\n", + detected.unwrap_or("") + ), + // An install is present but for another family, so + // `rocm install sdk` still has something to do and is still + // advice that can clear the condition it is printed for. + Some(installed) => format!( + " rocm_support: GPU detected ({}) with full ROCm/HIP support, but the \ + managed ROCm SDK installed here is for {installed}, not {} - install the \ + one this GPU needs with `rocm install sdk`\n", + detected.unwrap_or(""), + compatible.unwrap_or("") + ), + None => format!( + " rocm_support: GPU detected ({}) with full ROCm/HIP support; \ + install the SDK for it with `rocm install sdk`\n", + detected.unwrap_or("") + ), + } + } + // The command names `rocm services list` rather than the serve + // summary on purpose. The summary reads the device once, right + // after `ready`, and lemonade answers `unknown` until its own + // inference probe returns — so the row it prints is legitimately + // absent for a short window. Pointing at the surface that shows it + // *once known* keeps the advice able to clear the condition it is + // printed for, which naming the summary would not. + // + // Not folded in with `VulkanOnly`: a detection gap is not a + // confirmed absence of ROCm support, and saying "Vulkan" here would + // be the same false claim in the other direction. What the user can + // do is let an engine decide and then read back what it chose - + // which is the `device:` row this same ticket added, and the only + // thing on this host that can actually settle the question. + // Two different hosts reach `Unknown`, and they need different + // advice because only one of them can serve anything. + // + // A detected-but-unrecognised target means a GPU *is* present, so + // `rocm serve` runs and the engine's own `device:` row settles what + // this catalog could not. With nothing detected, that same command + // is refused outright by the no-usable-GPU pre-flight under the + // default `gpu_required` policy — naming it would be advice that + // cannot clear the condition it is printed for (AGENTS.md section 3). + // `rocm diagnose` is the command that can: its `fix-3-rocm-kernel`, + // `fix-4-render-group` and `fix-5-amdgpu-load` entries are exactly + // the reasons a present GPU goes undetected. + // + // Kept to one sentence each: the level word already says the CLI + // cannot answer, so spelling out "can confirm neither ROCm support + // nor its absence" in both arms was length without information. + // The distinction it was carrying lives in this comment and in + // `GpuSupportLevel`'s own docs, which is where it is load-bearing. + // The command is deliberately NOT `--engine lemonade`. An + // unrecognised target is a gap in this build's catalog, and a newer + // ROCm-capable GPU falls into it just as readily as a Vulkan-only + // one — naming the Vulkan backend would make exactly the kind of + // unestablished claim this level exists to avoid, only in the other + // direction. Plain `rocm serve` lets the engine choose, and the + // `device:` row then reports what it actually used, which is the + // answer this host needs and nothing here can predict. + GpuSupportLevel::Unknown => match detected { + Some(gfx) => format!( + " rocm_support: unknown - {gfx} is not in this build's \ + catalog; serve with `rocm serve `, then read the \ + `device:` row of `rocm services list` for the backend it \ + used\n" + ), + None => " rocm_support: unknown - no AMD GPU was detected; run \ + `rocm diagnose` to find out why none is visible\n" + .to_owned(), + }, } } @@ -222,6 +341,29 @@ mod tests { assert!(rendered.contains("model_cache_entries: 3")); } + /// The exact placeholders the report prints for the three GPU fields when + /// it has no value, pinned because they are read back as data. + /// + /// examine-55 decides which support tier a host is in by filtering these + /// out of `detected_gfx_target`, `compatible_therock_family` and + /// `detected_therock_family`. Reword one and that scenario stops seeing an + /// absent value as absent: a lane would be classified into the wrong tier + /// and its tier-specific command assertion would silently stop describing + /// the host it ran on. The two strings differ on purpose — "not detected" + /// is about a managed install, "unknown" about a probe — so neither may + /// stand in for the other. + #[test] + fn examine_render_spells_its_absent_value_placeholders() { + let rendered = examine_summary_with_gpu(None, None).render_text(); + for field in [ + "detected_gfx_target: ", + "compatible_therock_family: ", + "detected_therock_family: ", + ] { + assert!(rendered.contains(field), "missing `{field}`:\n{rendered}"); + } + } + fn examine_summary_with_gpu( detected_gfx_target: Option<&str>, compatible_therock_family: Option<&str>, @@ -279,24 +421,74 @@ mod tests { rendered.contains("the Vulkan backend can still be used"), "{rendered}" ); + // AGENTS.md §3: advice naming a command is asserted together with the + // behaviour. Saying the Vulkan backend "can still be used" without + // naming how to use it leaves the user where they started, so the + // command is part of the claim, not decoration — without this + // assertion the recommendation could be deleted or garbled and this + // test would still pass. + // + // That the named command *parses* (the subcommand exists and accepts + // `--engine lemonade`) is proved mechanically rather than here: + // `advised_commands::every_advised_command_parses` in `apps/rocm` + // scans production string literals under `crates/`, including this + // one, and routes each through the real clap tree. rocm-core cannot + // see the CLI's argument parser, so this half pins that the sentence + // is printed at all for a Vulkan-only host. + assert!( + rendered.contains("serve on it with `rocm serve --engine lemonade`"), + "a Vulkan-only host must be told the command that serves on it:\n{rendered}" + ); } + /// An unmapped target must not be *claimed* unsupported, even though every + /// host now gets a `rocm_support` line. + /// + /// This replaces `examine_render_omits_the_support_note_for_supported_or_missing_gpus`, + /// which asserted the line was absent for supported, missing and unmapped + /// GPUs. ROCMAI-550 deliberately reverses the absence: silence could not + /// answer the question `lemonade_engine_note` sends users here to ask. The + /// half of that test worth keeping is the discrimination it protected — + /// "no ROCm build" is a fact about gfx90c, and a gap in this catalog is not + /// the same claim about gfx803. #[test] - fn examine_render_omits_the_support_note_for_supported_or_missing_gpus() { + fn examine_render_never_claims_an_unmapped_target_is_unsupported() { + for (gfx, family) in [ + ("gfx1103", normalize_therock_family("gfx1103")), + ("gfx803", None), + ] { + let rendered = examine_summary_with_gpu(Some(gfx), family.as_deref()).render_text(); + assert!( + rendered.contains("rocm_support"), + "every host gets a support line now: {rendered}" + ); + } + + // Covered by a family: the line must not say it has no ROCm build. let family = normalize_therock_family("gfx1103"); - let supported = examine_summary_with_gpu(Some("gfx1103"), family.as_deref()); - let rendered = supported.render_text(); - assert!(!rendered.contains("rocm_support"), "{rendered}"); + let supported = examine_summary_with_gpu(Some("gfx1103"), family.as_deref()).render_text(); + assert!( + !supported.contains("not supported by ROCm"), + "a covered GPU must not be reported as unsupported: {supported}" + ); - let missing = examine_summary_with_gpu(None, None); - let rendered = missing.render_text(); - assert!(!rendered.contains("rocm_support"), "{rendered}"); + // Unmapped: a gap in the catalog, NOT a confirmed absence of support. + let unknown = examine_summary_with_gpu(Some("gfx803"), None).render_text(); + assert!( + !unknown.contains("not supported by ROCm"), + "an unmapped target must not be claimed unsupported: {unknown}" + ); - // An unmapped target the CLI knows nothing about is not claimed to be - // unsupported; only targets known to have no ROCm build are. - let unknown = examine_summary_with_gpu(Some("gfx803"), None); - let rendered = unknown.render_text(); - assert!(!rendered.contains("rocm_support"), "{rendered}"); + // Nothing detected: likewise a gap, not a verdict about hardware. + let missing = examine_summary_with_gpu(None, None).render_text(); + assert!( + missing.contains("rocm_support"), + "an undetected GPU still gets a line: {missing}" + ); + assert!( + !missing.contains("not supported by ROCm"), + "an undetected GPU must not be claimed unsupported: {missing}" + ); } #[test] @@ -486,4 +678,216 @@ mod tests { )); assert!(rendered.contains("rocm install sdk --channel release --format wheel")); } + + #[test] + fn examine_answers_the_support_question_the_unknown_engine_note_sends_users_here_for() { + // Both `Unknown` sub-cases must answer, and both must leave somewhere + // to go — but not the same somewhere. A detected-but-unrecognised + // target has a GPU, so `rocm serve` runs and its `device:` row settles + // the backend. With nothing detected that command is refused by the + // no-usable-GPU pre-flight, so the only honest next step is finding out + // why the GPU is invisible. See + // `no_gpu_advice_names_a_command_that_host_can_run`, which pins that + // distinction against the behaviour rather than the wording. + for (detected, expected_command) in [ + (None, "rocm diagnose"), + // Plain `rocm serve`, not `--engine lemonade`: see the arm's own + // comment. The reasoning this test already gave — "`rocm serve` + // runs and its `device:` row settles the backend" — is only true of + // the unpinned command. + (Some("gfx803"), "rocm serve "), + ] { + let rendered = examine_summary_with_gpu(detected, None).render_text(); + let line = rendered + .lines() + .find(|line| line.trim_start().starts_with("rocm_support:")) + .unwrap_or_else(|| { + panic!( + "`rocm examine` is where the Unknown-tier engine note sends \ + the user; it must print a support level:\n{rendered}" + ) + }); + // Says which level it is, and leaves a command that can settle it. + assert!(line.contains("unknown"), "{line}"); + assert!( + line.contains(expected_command), + "an `unknown` answer still has to leave somewhere to go, and it \ + has to be somewhere this host can actually go: {line}" + ); + } + + // The detected sub-case also names where the backend shows up; the + // undetected one has no server to read it from. + let unrecognised = examine_summary_with_gpu(Some("gfx803"), None).render_text(); + assert!( + unrecognised.contains("device:"), + "a host that can serve must be told where the real backend shows up: \ + {unrecognised}" + ); + } + + /// The advice a host is given must name a command that host can actually + /// run — asserted against the serve pre-flight's own predicate, not against + /// the wording. + /// + /// `rocm serve` refuses on a host with no usable AMD GPU under the default + /// `gpu_required` policy, so telling an undetected host to serve a model is + /// advice that cannot clear the condition it is printed for. Both `Unknown` + /// sub-cases are driven here because they differ precisely on this: a + /// detected-but-unrecognised target has a GPU and may serve, nothing + /// detected may not. + /// + /// Written against `gpu_support_level` + the rendered line rather than a + /// string match on the sentence: a reworded message that still sent a + /// GPU-less host to `rocm serve` would pass a wording assertion and fail + /// this one. + #[test] + fn no_gpu_advice_names_a_command_that_host_can_run() { + let undetected = examine_summary_with_gpu(None, None).render_text(); + let line = undetected + .lines() + .find(|l| l.trim_start().starts_with("rocm_support:")) + .expect("every host gets a support line"); + assert_eq!( + gpu_support_level(None, None), + GpuSupportLevel::Unknown, + "precondition: nothing detected is the Unknown tier" + ); + assert!( + !line.contains("rocm serve"), + "`rocm serve` is refused on a host with no usable AMD GPU, so it \ + cannot be the next step offered to one: {line}" + ); + assert!( + line.contains("rocm diagnose"), + "dropping the unusable command must still leave a usable one: {line}" + ); + + // The sibling sub-case keeps the serve advice, because that host has a + // GPU and the engine's own device row is what settles the question. + let unrecognised = examine_summary_with_gpu(Some("gfx-not-in-catalog"), None).render_text(); + let line = unrecognised + .lines() + .find(|l| l.trim_start().starts_with("rocm_support:")) + .expect("every host gets a support line"); + assert!( + line.contains("rocm serve"), + "a detected GPU can serve, so the advice that settles its backend \ + must not be withheld: {line}" + ); + } + + #[test] + fn examine_render_reports_a_support_level_for_every_host() { + let family = normalize_therock_family("gfx1103"); + let supported = examine_summary_with_gpu(Some("gfx1103"), family.as_deref()); + let rendered = supported.render_text(); + assert!( + rendered.contains("rocm_support: GPU detected (gfx1103) with full ROCm/HIP support"), + "{rendered}" + ); + assert!(rendered.contains("rocm install sdk"), "{rendered}"); + + // Same GPU, but the SDK it was being told to install is already there: + // the advice has to be something that is still left to do (AGENTS.md + // wants a named command to be able to clear the condition). The + // installed family must be the GPU's own - see + // `an_sdk_for_another_family_is_not_reported_as_ready_for_this_gpu`. + let mut installed = supported; + installed.detected_therock_family = family; + let rendered = installed.render_text(); + assert!( + rendered.contains("rocm_support: GPU detected (gfx1103) with full ROCm/HIP support"), + "{rendered}" + ); + assert!( + rendered.contains("managed ROCm SDK for gfx110X-all is already installed"), + "{rendered}" + ); + assert!( + !rendered.contains("rocm install sdk"), + "an install that is already done must not be the advice: {rendered}" + ); + + // No GPU found: the level is reported as unknown, and nothing about the + // hardware is asserted in either direction. + let missing = examine_summary_with_gpu(None, None); + let rendered = missing.render_text(); + assert!( + rendered.contains("rocm_support: unknown - no AMD GPU was detected"), + "{rendered}" + ); + assert!( + !rendered.contains("Vulkan backend can still be used"), + "nothing was detected, so no backend may be promised: {rendered}" + ); + + // An unmapped target the CLI knows nothing about is not claimed to be + // unsupported; only targets known to have no ROCm build are. + let unknown = examine_summary_with_gpu(Some("gfx803"), None); + let rendered = unknown.render_text(); + assert!( + rendered.contains("rocm_support: unknown - gfx803 is not in this build's catalog"), + "{rendered}" + ); + assert!( + !rendered.contains("not supported by ROCm"), + "an unrecognized target must not be reported as confirmed \ + ROCm-less: {rendered}" + ); + assert!( + !rendered.contains("full ROCm/HIP support"), + "nor as supported: {rendered}" + ); + } + + /// A managed SDK on disk is not, by itself, an SDK for *this* GPU. + /// + /// `detected_therock_family` is the most recently installed managed runtime + /// (`detect_managed_therock_family` reads the registry and never looks at + /// the GPU); `compatible_therock_family` is normalised from the gfx target + /// detected right now. A GPU swap, an eGPU or a second card leaves them + /// pointing at different families, and the `already installed - serve on + /// it` sentence then promises a serve this host cannot do. + /// + /// The matching half is asserted by + /// `examine_render_reports_a_support_level_for_every_host`; only this one + /// fails if the two families stop being compared, because the wording of + /// the matching case is unchanged. + #[test] + fn an_sdk_for_another_family_is_not_reported_as_ready_for_this_gpu() { + let compatible = normalize_therock_family("gfx1103").expect("gfx1103 has a family"); + let mut host = examine_summary_with_gpu(Some("gfx1103"), Some(&compatible)); + // A real family, and not this GPU's: the gfx110X split between the + // discrete and the all-targets build is exactly the pair a user is + // likeliest to have both of. + host.detected_therock_family = Some("gfx110X-dgpu".to_owned()); + assert_ne!( + host.detected_therock_family.as_deref(), + Some(compatible.as_str()), + "precondition: the installed family is not the compatible one" + ); + + let rendered = host.render_text(); + let line = rendered + .lines() + .find(|line| line.trim_start().starts_with("rocm_support:")) + .expect("every host gets a support line") + .to_owned(); + + assert!( + !line.contains("already installed"), + "an SDK built for another family must not be reported as this GPU's: {line}" + ); + assert!( + line.contains("gfx110X-dgpu") && line.contains(compatible.as_str()), + "the line has to name both families, or the user cannot see which \ + one is missing: {line}" + ); + assert!( + line.contains("rocm install sdk"), + "there is still an install left to do, so the advice must name the \ + command that does it: {line}" + ); + } } diff --git a/crates/rocm-core/src/host_gpu/windows_inventory.rs b/crates/rocm-core/src/host_gpu/windows_inventory.rs index ec6db3e7b..d09728ae5 100644 --- a/crates/rocm-core/src/host_gpu/windows_inventory.rs +++ b/crates/rocm-core/src/host_gpu/windows_inventory.rs @@ -146,8 +146,7 @@ impl WindowsExamineInventory { /// vendor, so the rows also hold devices like "AMD Bluetooth Capture Audio /// Device" and "Intel(R) UHD Graphics". A row counts only with an AMD PCI /// device id (`PCI\VEN_1002`) or a marketing name `rocm examine` recognizes. - #[cfg(any(windows, test))] - fn amd_display_count(&self) -> usize { + pub(crate) fn amd_display_count(&self) -> usize { self.displays .iter() .filter(|display| { @@ -1009,4 +1008,31 @@ Class Name: Display Some("2026-05-02") ); } + + #[test] + fn the_windows_display_count_counts_amd_adapters_only() { + let inventory = WindowsExamineInventory { + cpu_model: None, + system_ram_gib: None, + displays: vec![ + WindowsDisplayAdapter { + name: "Intel(R) UHD Graphics".to_owned(), + driver_version: None, + driver_date: None, + pnp_device_id: Some("PCI\\VEN_8086&DEV_9A49".to_owned()), + }, + WindowsDisplayAdapter { + name: "AMD Radeon(TM) Graphics".to_owned(), + driver_version: None, + driver_date: None, + pnp_device_id: Some("PCI\\VEN_1002&DEV_15BF".to_owned()), + }, + ], + }; + assert_eq!(inventory.amd_display_count(), 1); + assert_eq!( + inventory.amd_display_name(), + Some("AMD Radeon(TM) Graphics".to_owned()) + ); + } } diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index 90385d208..f990b62c6 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -74,7 +74,7 @@ pub mod windows_driver; pub use amd_smi::{amd_smi_json, device_readable_bounded, resolve_amd_smi_binary}; pub use anthropic_endpoint::{ ANTHROPIC_ENDPOINT_MODE_STATE_KEY, ANTHROPIC_ENDPOINT_URL_STATE_KEY, AnthropicEndpointMode, - anthropic_endpoint_ready, wait_for_published_anthropic_endpoint, + anthropic_endpoint_ready, publish_anthropic_endpoint, wait_for_published_anthropic_endpoint, }; pub use automation::{ AuditEventRecord, AutomationEventRecord, AutomationProposalRecord, AutomationRuntimeState, @@ -136,6 +136,9 @@ pub use fix::{ apply as apply_fix, catalog_manifest, catalog_manifest_json, exit as fix_exit, list_recipes as list_fix_recipes, }; +pub use gfx_target::{ + GpuSupportLevel, VULKAN_ONLY_SUPPORT_CLAIM, gpu_support_level, gpu_support_level_for_host_gpu, +}; pub use gfx_target::{ extract_first_gfx_token, gfx_target_has_no_rocm_build, known_therock_families, normalize_therock_family, @@ -147,8 +150,8 @@ pub use host_gpu::{ WslHostDriverProbe, WslSummary, default_engine_for_host, default_engine_for_platform, detect_gpu_driver_version, detect_host_gfx_target, detect_host_gpu_diagnostics, detect_host_gpu_summary, detect_system_ram_gib, has_usable_amd_gpu, is_amdgpu_device, - is_wsl_host, preferred_serve_engine_for_host_gpu_summary, usable_amd_gpu_indices, - windows_amd_gpu_listing, + is_wsl_host, pinned_gpu_name, preferred_serve_engine_for_host_gpu_summary, + usable_amd_gpu_indices, windows_amd_gpu_listing, }; #[cfg(any(target_os = "linux", test))] use host_gpu::{KfdGpuNode, kfd_gpu_nodes_in}; diff --git a/crates/rocm-engine-protocol/src/lib.rs b/crates/rocm-engine-protocol/src/lib.rs index 9fcab5380..18533ad1f 100644 --- a/crates/rocm-engine-protocol/src/lib.rs +++ b/crates/rocm-engine-protocol/src/lib.rs @@ -696,6 +696,80 @@ pub struct LaunchResponse { pub state_path: String, } +/// The [`HealthcheckResponse::device`] string, in the one shape every engine +/// reports and the `device:` row of `rocm serve` / `rocm services list` prints +/// verbatim. +/// +/// ROCMAI-550 asks those two surfaces for "the llama.cpp backend actually +/// chosen (`vulkan`/`rocm`) **and the device name**", so both halves are named +/// arguments here rather than text each engine formats for itself: +/// +/// | backend | ordinal | name | rendered | +/// | --- | --- | --- | --- | +/// | `vulkan` | `Some(0)` | `Some("AMD Radeon Graphics")` | `vulkan gpu 0 (AMD Radeon Graphics)` | +/// | `vulkan` | `Some(0)` | `None` | `vulkan gpu 0` | +/// | `vulkan` | `None` | `Some("AMD Radeon Graphics")` | `vulkan (AMD Radeon Graphics)` | +/// | `vulkan` | `None` | `None` | `vulkan` | +/// +/// The last row deliberately drops the `gpu` noun rather than printing a +/// dangling `vulkan gpu`. That bare form is not hypothetical: it is what the +/// ticket's own host produces. `gpu_indices` comes from +/// `engines/lemonade`'s `resolve_serve_gpu_indices`, whose probe +/// (`rocm_core::usable_amd_gpu_indices`) returns `None` off Linux, so a default +/// `rocm serve` on Windows pins no ordinal and records an empty list — and +/// Windows on a gfx90c Vega iGPU is exactly the machine ROCMAI-550 was filed +/// from. `device: vulkan gpu` would answer "which GPU?" with the word "gpu". +/// +/// The name is whatever the engine recorded for the host at launch; it is not +/// per-ordinal. On a multi-GPU host the ordinal is the precise half and the +/// name is context, which is why the ordinal is never dropped in favour of it. +#[must_use] +pub fn format_reported_device( + backend: &str, + gpu_index: Option, + gpu_name: Option<&str>, +) -> String { + use std::fmt::Write as _; + + let mut device = backend.to_owned(); + if let Some(index) = gpu_index { + let _ = write!(device, " gpu {index}"); + } + if let Some(name) = gpu_name.map(str::trim).filter(|name| !name.is_empty()) { + let _ = write!(device, " ({name})"); + } + device +} + +/// The first pinned GPU ordinal recorded in a service state file, if any. +/// +/// The ordinal half of what [`format_reported_device`] renders, read from the +/// state file every engine writes at launch. It lives beside that function for +/// the same reason that one does: `rocm serve`'s `device:` row and +/// `rocm services list` print whichever engine answered verbatim, so the two +/// engines that pin ordinals (`engines/lemonade`, `engines/vllm`) have to agree +/// on *which* ordinal they report, not only on how it is spelled. They had +/// byte-identical private copies of this, which is one keystroke away from +/// disagreeing silently — the same duplication `format_reported_device` was +/// centralized to end. +/// +/// `None` covers every shape that is not a usable ordinal: no `gpu_indices` +/// key, a non-array value, an empty list (a launch that pinned nothing, which +/// is the default on Windows — see `format_reported_device`), a non-integer +/// entry, and a value too large for `u32`. Those are "no ordinal to report", +/// not errors: state files are written by engine processes of possibly +/// different versions, and a row that says less is better than one that +/// invents an ordinal or refuses to render. +#[must_use] +pub fn first_gpu_index_from_state(state: &Value) -> Option { + state + .get("gpu_indices") + .and_then(Value::as_array) + .and_then(|indices| indices.first()) + .and_then(Value::as_u64) + .and_then(|value| u32::try_from(value).ok()) +} + #[derive(Debug, Clone, Serialize, Deserialize)] pub struct HealthcheckResponse { /// Engine-reported service state. `ready` is reserved for a service that has @@ -776,6 +850,94 @@ pub struct LogsResponse { #[cfg(test)] mod tests { + /// The `device:` row has to carry a device, not the word "gpu". + /// + /// The ordinal-less case is the one ROCMAI-550 was filed about: on Windows + /// `rocm_core::usable_amd_gpu_indices` cannot probe, so a default + /// `rocm serve` records an empty `gpu_indices` and the old formatting + /// rendered a bare `vulkan gpu`. The name is what makes that row answer the + /// question; with neither, the noun goes too rather than dangling. + #[test] + fn reported_device_never_renders_a_dangling_gpu_noun() { + assert_eq!( + super::format_reported_device("vulkan", Some(0), Some("AMD Radeon Graphics")), + "vulkan gpu 0 (AMD Radeon Graphics)" + ); + assert_eq!( + super::format_reported_device("vulkan", None, Some("AMD Radeon Graphics")), + "vulkan (AMD Radeon Graphics)" + ); + assert_eq!( + super::format_reported_device("rocm", Some(1), None), + "rocm gpu 1" + ); + // The regression this exists to stop: nothing known beyond the backend + // must not be padded out with a noun that names no device. + assert_eq!( + super::format_reported_device("vulkan", None, None), + "vulkan" + ); + } + + /// A name the host probe could not fill in arrives as `Some("")` or + /// `Some(" ")` just as easily as `None` (the Windows display-adapter probe + /// and the Linux DRM read both build it from scraped text), and + /// `vulkan ()` is worse than no parenthetical at all. + #[test] + fn reported_device_treats_a_blank_gpu_name_as_absent() { + assert_eq!( + super::format_reported_device("vulkan", Some(0), Some("")), + "vulkan gpu 0" + ); + assert_eq!( + super::format_reported_device("vulkan", None, Some(" ")), + "vulkan" + ); + assert_eq!( + super::format_reported_device("rocm", Some(2), Some(" MI300X ")), + "rocm gpu 2 (MI300X)" + ); + } + + /// Every shape a state file can carry in `gpu_indices` resolves to "an + /// ordinal" or "no ordinal", never to a panic or an invented value. + /// + /// This used to be two byte-identical private helpers in `engines/lemonade` + /// and `engines/vllm`, neither of which tested the shapes below; only the + /// happy path was reachable from either engine's own tests. Centralizing it + /// is what makes the edge cases testable in one place for both callers. + #[test] + fn the_first_pinned_ordinal_is_read_and_every_other_shape_is_absent() { + use serde_json::json; + + assert_eq!( + super::first_gpu_index_from_state(&json!({"gpu_indices": [1, 2]})), + Some(1), + "the first pinned ordinal is the one reported" + ); + // A launch that pinned nothing — the default on Windows, which is the + // host ROCMAI-550 was filed from. + assert_eq!( + super::first_gpu_index_from_state(&json!({"gpu_indices": []})), + None + ); + assert_eq!(super::first_gpu_index_from_state(&json!({})), None); + // A state file written by some other version of an engine must degrade + // to "no ordinal", not to a wrong one. + assert_eq!( + super::first_gpu_index_from_state(&json!({"gpu_indices": "0"})), + None + ); + assert_eq!( + super::first_gpu_index_from_state(&json!({"gpu_indices": [-1]})), + None + ); + assert_eq!( + super::first_gpu_index_from_state(&json!({"gpu_indices": [u64::from(u32::MAX) + 1]})), + None + ); + } + fn resolve_response(served: Option<&str>) -> ResolveModelResponse { ResolveModelResponse { canonical_model_id: "image:tag".to_owned(), diff --git a/docs/architecture.md b/docs/architecture.md index ae418380a..e7311e833 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -124,9 +124,10 @@ The `rocm` binary: the clap command tree, a handler module for most commands, an | [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) | Spawning the detached managed engine child both serve paths share: process and stdio plumbing and the serialized spawn transaction (`spawn_managed_engine_child`). | relocation | | [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) | The background launch path (`start_managed_service`) and the plain line-by-line rendering of its launch report. | relocation | | [`service_launch/test_support.rs`](../apps/rocm/src/service_launch/test_support.rs) | Test fixtures shared by more than one `service_launch/` child. | flat | +| [`service_log_follow.rs`](../apps/rocm/src/service_log_follow.rs) | The `--follow` tail behind `rocm logs --service --follow` and `rocm services logs --follow`: the growing-file cursor (`FollowCursor`), the terminal footer that keeps the "how to stop" hint under the newest output (`FollowFooter`), and the follow loop that ends when the service does. Split out of [`services_text.rs`](../apps/rocm/src/services_text.rs) as a pure move; it calls back into it for the record read and the opening snapshot, and nothing there calls into here. | domain | | [`service_records.rs`](../apps/rocm/src/service_records.rs) | Removing a managed-service record and every file it owns — including any copy of its endpoint key an interrupted save left behind, found with one listing of the services folder per command (`staged_endpoint_keys_by_service`) — and the `rocm services prune` plan, render and apply steps with their age gate. `remove_managed_service_record_with` takes `--force` and the read of each recorded PID: `--force` removes a record only when none of its recorded processes is confirmed running, one could not be queried and a stop was requested, and reports which. | relocation | | [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) | `rocm services` (`services`) for `list`, `logs`, `stop`, `restart`, `remove` and `prune`, and the approval-gated stop and restart path. | relocation | -| [`services_text.rs`](../apps/rocm/src/services_text.rs) | Log rendering and the `rocm services` and `rocm logs` status, browser and `--follow` surfaces, managed-service record reads, and the `stop_server` answer shape. | relocation | +| [`services_text.rs`](../apps/rocm/src/services_text.rs) | Log rendering and the `rocm services` and `rocm logs` status and browser surfaces, managed-service record reads, and the `stop_server` answer shape. The `--follow` tail lives in [`service_log_follow.rs`](../apps/rocm/src/service_log_follow.rs). | relocation | | [`setup_cmd.rs`](../apps/rocm/src/setup_cmd.rs) | `rocm setup` (`setup`): the `setup status` text and `setup reset`, which clears the saved first-time prompt state. | relocation | | [`shell_quote.rs`](../apps/rocm/src/shell_quote.rs) | The CLI's one POSIX shell quoter (`shell_quote`, over `rocm-core`'s) and `has_terminal_control_character`, the predicate the chat tool-call guards refuse on. | flat | | [`storage.rs`](../apps/rocm/src/storage.rs) | `rocm storage`: a read-only report of what runtimes, downloads and engine installs occupy on disk, and guarded removal of old installs and downloads. | domain | @@ -309,7 +310,7 @@ The library every first-party app and engine crate depends on. It holds host exa | [`fix/render.rs`](../crates/rocm-core/src/fix/render.rs) | Rendering the catalog for people: the listing, a recipe's plan and verify lines, the flag summary `rocm diagnose` shares, and the refusal of a diagnosis position as a fix-id. | flat | | [`fix/runners.rs`](../crates/rocm-core/src/fix/runners.rs) | The host-mutating runners, one per auto-applicable fix, each printing the exact change, honouring `--dry-run` and asking consent before any write, except the report-only arms (fix-2 on Linux, fix-9 without `--device-index`). On Windows, fix-2, fix-6 and fix-9 (with `--device-index`) first read the env-scope value their plan depends on, and if that read fails they stop with exit 4 before the runner's own preview or any prompt, under `--dry-run` too. | flat | | [`foreign_temp_probe.rs`](../crates/rocm-core/src/foreign_temp_probe.rs) | Test probe that a writer publishing through `atomic_write` never touches another writer's temp file; built only for tests and the `test-support` feature. | flat | -| [`gfx_target.rs`](../crates/rocm-core/src/gfx_target.rs) | Pure gfx-target lookups: extracting a gfx token, mapping a target to its TheRock family, the AMD PCI-id and marketing-name tables, and kernel gfx-version decoding. | domain | +| [`gfx_target.rs`](../crates/rocm-core/src/gfx_target.rs) | Pure gfx-target lookups: extracting a gfx token, mapping a target to its TheRock family, the AMD PCI-id and marketing-name tables, and kernel gfx-version decoding. Also the `GpuSupportLevel` classifier (`gpu_support_level`, `gpu_support_level_for_host_gpu`, `VULKAN_ONLY_SUPPORT_CLAIM`) that turns those two facts into the one ROCm/Vulkan support claim `examine`, `install` and the engine notes all render, so no surface invents its own. | domain | | [`gguf.rs`](../crates/rocm-core/src/gguf.rs) | Reads a GGUF model's shape (layers, heads, head width, parameter count, data offset) from its header, never the weights. | domain | | [`gguf_fit.rs`](../crates/rocm-core/src/gguf_fit.rs) | Pure arithmetic on whether a GGUF model fits in GPU memory at a context size: weights, KV cache, compute buffer and reserve against VRAM. | domain | | [`gguf_source.rs`](../crates/rocm-core/src/gguf_source.rs) | Where a GGUF header comes from: a local file, or a Hugging Face file read by HTTP range requests. | domain | diff --git a/docs/manual-testing.md b/docs/manual-testing.md index ca491f3a2..26db54be8 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -345,7 +345,10 @@ checks, which cannot be made anywhere else: `compatible_therock_family`, plus a `rocm_support` line saying the GPU is detected but not supported by ROCm and that the Vulkan backend can still be used. A `` target, or a missing `rocm_support` line, means detection - regressed. + regressed. That line is printed on **every** host, at one of three levels — + ROCm/HIP-supported, Vulkan-only (this case), or unknown — so on a supported + card or a machine with no GPU it must still be there, saying the level that + host's hardware implies rather than being absent. - A plain `rocm install sdk`, with no `--family`, gives the same refusal with the wording for a *detected* GPU rather than a requested one. - `rocm diagnose` reports `info-20-gpu-no-rocm-support`, naming `gfx90c` and @@ -422,12 +425,28 @@ Lemonade is the default local assistant/server engine. Serve a small assistant model with the managed runtime: ```powershell -rocm serve qwen --engine lemonade --device gpu_required --foreground --port 11435 +rocm serve qwen --engine lemonade --device gpu_required --port 11435 ``` Expected result: -- The engine uses its selected supported GPU backend, ROCm or Vulkan. +- The engine uses its selected supported GPU backend, ROCm or Vulkan. The + deployment summary's `device:` row is the engine's own account of the backend + it was launched on (Lemonade reads back the `backend_requested` it recorded), + so check it by reading that row rather than inferring it from the absence of a + fallback warning. Every engine reports it in the same + ` [gpu ] [()]` shape — so expect something like + `rocm gpu 0 (AMD Instinct MI300X)`, with the ordinal omitted when the launch + pinned none (no `--gpu`, nothing recorded in the engine's `gpu_indices`) and + the name omitted when the host probe could not read one or the machine has + more than one AMD GPU (where naming one could name the wrong one). `rocm + services` shows the same `device` while it runs. The first word is the half + that decides this check. +- The whole `device:` row may be missing from the deployment summary, and that + is not a failure of this check: the engine reports no device until it has + verified it can serve, which a model still loading its weights has not. Run + `rocm services list` and read the row there instead — the check is on what the + row says, not on which surface said it first. - The run does not fall back to CPU, and does not select a stale ROCm binary when the installed runtime selects Vulkan. - If the GPU cannot be used, the command fails with a clear error. @@ -440,9 +459,16 @@ in `MSVCP140.dll`), the Visual C++ runtime on the machine is probably older than fix-21-msvc-redist-outdated` prints the update plan and changes nothing. After the update, restart Lemonade and repeat the serve command above. -With that foreground server stopped, check that a catalogued recipe reaches -Lemonade with a build it can load, from a short name alone — no `--engine`, no -quantization variant typed by hand: +That server is in the background — `rocm serve` returns to the shell once it is +ready — so stop it before continuing, or it keeps the GPU and port 11435: + +```powershell +rocm services stop --yes +``` + +With it stopped, check that a catalogued recipe reaches Lemonade with a build +it can load, from a short name alone — no `--engine`, no quantization variant +typed by hand: ```powershell rocm serve qwen-small --managed diff --git a/docs/testing.md b/docs/testing.md index d5d2027da..a27abd359 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -4871,6 +4871,57 @@ others assert on the outcome the smoke test returns. cargo xtask e2e -- -n "serve-60" ``` +## The Reported Backend Device And Support Level + +ROCMAI-550 made two surfaces report what this host can actually do instead of a +static ROCm claim. Both are worth knowing about before reading a failure in +them. + +**`rocm serve` and `rocm services list` print a `device:` row** — the backend +and GPU the engine says it selected, read back over the engine's own +`Healthcheck` RPC rather than inferred from the launch plan. Every engine +renders it through `rocm_engine_protocol::format_reported_device`, so the shape +is one and the same everywhere: a backend word, then an optional ` gpu `, +then an optional ` ()`. + +**The row is optional, and a missing one is not a defect.** An engine answers +`unknown` until it can really say — `engines/lemonade` until its own inference +probe returns, which for a model still loading its weights is minutes rather +than seconds — and the CLI drops `unknown` rather than printing it. `rocm serve` +retries within `engine_ipc::SERVE_DEVICE_PROBE_BUDGET` (~6s, plus one in-flight +probe) and then prints the summary without the row; `rocm services list` shows +it once the engine can answer. So: + +- a test may assert that a row, where present, is well-formed and that the two + surfaces agree — which is what `the services list reports the engine's actual + backend device` (serve-05, serve-07) does; +- a test may **not** assert that a `ready` serve printed one. + +The GPU name is optional in the same way, and deliberately so: +`rocm_core::pinned_gpu_name` reports a name only on a host with exactly one AMD +GPU and no visibility mask, because every name probe here answers for the +host's primary adapter rather than for a HIP ordinal. On a multi-GPU host the +row is the backend and ordinal with no name — correct and unspecific, rather +than specific and wrong. No CI lane has two AMD GPUs, so that branch is covered +by unit tests in [host_gpu.rs](../crates/rocm-core/src/host_gpu.rs) only. + +**`rocm examine` prints a `rocm_support` line on every host**, at one of three +levels (`gpu_support_level`): ROCm/HIP-supported, Vulkan-only, or unknown. +Unknown covers both "nothing detected" and "a target this build's catalog does +not recognise" — conflating the latter with "confirmed to have no ROCm build" +would turn a detection gap into a claim about the hardware. `examine-55` holds +the line against the `detected_gfx_target` and `compatible_therock_family` the +same report prints, so every lane checks the level its own hardware implies: + +```bash +cargo xtask e2e -- -n '(examine-55)' +``` + +Vulkan-only here is inferred from the gfx table, not probed: there are three +support levels and this section describes three. Probing the host for a Vulkan +driver, and the further levels that evidence would make knowable, is separate +work. + ## Windows Tool Notes The TheRock SDK wheel install path should not require users to install global diff --git a/engines/aim/src/process.rs b/engines/aim/src/process.rs index b6460269f..10681d5e6 100644 --- a/engines/aim/src/process.rs +++ b/engines/aim/src/process.rs @@ -164,7 +164,7 @@ pub(crate) fn launch_service(request: LaunchRequest) -> Result { let state_path = crate::state::state_path(&request.service_id); - let served_model_id = run_container( + let run = run_container( &request.service_id, &request.model_ref, &request.host, @@ -178,10 +178,11 @@ pub(crate) fn launch_service(request: LaunchRequest) -> Result { &state_path, &request.service_id, &request.model_ref, - served_model_id.as_deref(), + run.canonical_name.as_deref(), &request.host, request.port, device_policy, + &run.gpu_indices, engine_recipe.as_ref(), )?; @@ -246,8 +247,22 @@ fn plan_gpus(requested: &[u32], usable: Option>, windows: bool) -> Resu }) } +/// What [`run_container`] established about the container it left running. +#[derive(Debug, Default, PartialEq, Eq)] +struct ContainerRun { + /// The model name the image says it serves under, if known. + canonical_name: Option, + /// The GPU ordinals the container was narrowed to through + /// `HIP_VISIBLE_DEVICES`, for [`crate::state::write_running_state`] to + /// record so the `device:` row can name one (ROCMAI-550). Empty when no + /// GPU was attached, and when the container was already running — that + /// launch chose the ordinals and this call cannot know which. + gpu_indices: Vec, +} + /// Starts the named service's container unless it is already running, and -/// returns the model name the image says it serves under, if known. +/// reports the model name the image says it serves under together with the +/// GPUs the container was given. /// /// Container names are unique in Docker, so `docker run --name X` against a /// name still held by an exited container (a crash, a reboot, a failed @@ -261,15 +276,18 @@ fn run_container( device_policy: &DevicePolicy, requested_gpus: &[u32], extra_env: Option<&BTreeMap>, -) -> Result> { +) -> Result { let name = crate::docker::container_name(service_id); // Normalize once so the pull, the metadata and the run all use the same image. let image_ref = crate::registry::normalize_image_ref(model_ref); match crate::docker::container_state(&name)? { ContainerState::Running => { - return Ok(crate::registry::local_image_metadata(&image_ref) - .and_then(|metadata| metadata.canonical_name)); + return Ok(ContainerRun { + canonical_name: crate::registry::local_image_metadata(&image_ref) + .and_then(|metadata| metadata.canonical_name), + gpu_indices: Vec::new(), + }); } ContainerState::Exited(_) => crate::docker::remove(&name)?, ContainerState::Missing => {} @@ -313,6 +331,9 @@ fn run_container( if let Some(api_key) = rocm_engine_protocol::resolve_endpoint_api_key() { env.insert("VLLM_API_KEY".to_owned(), api_key); } + // The one place the attached ordinals are decided, so the container and the + // state file cannot disagree about them. + let gpu_indices = if use_gpu { gpus.indices } else { Vec::new() }; crate::docker::run(&crate::docker::RunSpec { name: &name, image: &image_ref, @@ -320,11 +341,14 @@ fn run_container( port, container_port: crate::DEFAULT_AIM_PORT, use_gpu, - gpu_indices: if use_gpu { &gpus.indices } else { &[] }, + gpu_indices: &gpu_indices, render_gid, env: &env, })?; - Ok(metadata.canonical_name) + Ok(ContainerRun { + canonical_name: metadata.canonical_name, + gpu_indices, + }) } /// Enforces `device_policy` instead of treating GPU use as a soft heuristic: @@ -357,7 +381,7 @@ fn resolve_use_gpu( pub(crate) fn serve_http(request: ServeHttpRequest) -> Result<()> { let name = crate::docker::container_name(&request.service_id); - let label_name = run_container( + let run = run_container( &request.service_id, &request.model_ref, &request.host, @@ -377,10 +401,11 @@ pub(crate) fn serve_http(request: ServeHttpRequest) -> Result<()> { &request.state_path, &request.service_id, &request.model_ref, - label_name.as_deref(), + run.canonical_name.as_deref(), &request.host, request.port, request.device_policy, + &run.gpu_indices, request.engine_recipe.as_ref(), )?; @@ -391,7 +416,7 @@ pub(crate) fn serve_http(request: ServeHttpRequest) -> Result<()> { &name, &request.host, request.port, - label_name.as_deref(), + run.canonical_name.as_deref(), aim_ready_timeout(), ) { Ok(served_model_id) => { diff --git a/engines/aim/src/state.rs b/engines/aim/src/state.rs index c81710afb..b7e1ac607 100644 --- a/engines/aim/src/state.rs +++ b/engines/aim/src/state.rs @@ -17,8 +17,19 @@ const HEALTHCHECK_TIMEOUT_MS: u64 = 700; pub(crate) fn healthcheck_service(request: HealthcheckRequest) -> Result { require_nonempty(&request.service_id, "service_id")?; - let state_path = state_path(&request.service_id); - let state = read_service_state(&state_path).ok(); + healthcheck_service_at(&state_path(&request.service_id)) +} + +/// [`healthcheck_service`] against an explicit state path. +/// +/// Split out for the same reason `engines/lemonade`'s `healthcheck_service_at` +/// is: [`state_path`] resolves through the caller's real `AppPaths`, so the +/// RPC handler itself could not be driven from a test, and the `device` field +/// it answers with was only ever covered by calling [`reported_device`] +/// directly — a test that stays green if this handler stops consulting it at +/// all (see `healthcheck_rpc_answers_with_the_device_the_state_file_describes`). +pub(crate) fn healthcheck_service_at(state_path: &Path) -> Result { + let state = read_service_state(state_path).ok(); let endpoint_url = state .as_ref() @@ -33,27 +44,62 @@ pub(crate) fn healthcheck_service(request: HealthcheckRequest) -> Result reported_device(state), + None => "unknown".to_owned(), }; let listed = !endpoint_url.is_empty() && query_loaded_model_endpoint(&endpoint_url, model_ref.as_deref()).unwrap_or(false); let ready = listed && model_ref.as_deref().is_some_and(|model_ref| { - inference_verified(&state_path, state.as_ref(), &endpoint_url, model_ref) + inference_verified(state_path, state.as_ref(), &endpoint_url, model_ref) }); Ok(HealthcheckResponse::for_readiness( listed, ready, &state_status, - device, + &device, )) } +/// The device string reported once state exists for the service. +/// +/// The shape is whatever [`rocm_engine_protocol::format_reported_device`] +/// renders — the same one `engines/lemonade` and `engines/vllm` use, because +/// `rocm serve`'s `device:` row and `rocm services list` print whichever engine +/// answered verbatim and must not change format per engine. The ordinal comes +/// from [`rocm_engine_protocol::first_gpu_index_from_state`], the same read +/// those two make of the same `gpu_indices` key. +/// +/// The backend is **not** fixed. AIM's images run on ROCm, but +/// [`rocm_engine_protocol::DevicePolicy::CpuOnly`] is a real variant of this +/// crate's own launch path (`process::resolve_use_gpu`) and +/// `docker::RunSpec` then hands the container no `/dev/kfd`, no `/dev/dri` and +/// no `HIP_VISIBLE_DEVICES` at all. Claiming `rocm`, let alone naming a card, +/// for a container that was given no GPU is exactly the invented device row +/// ROCMAI-550 exists to remove, so the policy the launch recorded decides: +/// `cpu_only` reports `cpu` and names nothing. A state file from before this +/// key was written has no policy, and ROCm is the right assumption there — +/// `cpu_only` could not be reached through `rocm serve` then either. +fn reported_device(state: &Value) -> String { + if recorded_device_policy(state) == Some(DevicePolicy::CpuOnly) { + return rocm_engine_protocol::format_reported_device("cpu", None, None); + } + rocm_engine_protocol::format_reported_device( + "rocm", + rocm_engine_protocol::first_gpu_index_from_state(state), + value_string(state, "gpu_name").as_deref(), + ) +} + +/// The `device_policy` the launch recorded, or `None` when the state file +/// predates the key or holds something that is not one of the three policies. +fn recorded_device_policy(state: &Value) -> Option { + serde_json::from_value(state.get("device_policy")?.clone()).ok() +} + pub(crate) fn endpoint_response(request: EndpointRequest) -> Result { require_nonempty(&request.service_id, "service_id")?; let state = read_service_state(&state_path(&request.service_id)) @@ -112,9 +158,13 @@ pub(crate) fn write_running_state( host: &str, port: u16, device_policy: DevicePolicy, + gpu_indices: &[u32], engine_recipe: Option<&EngineRecipeHint>, ) -> Result<()> { let endpoint = endpoint_url(host, port); + // No GPU was attached, so there is no device to probe for or name. See + // [`reported_device`] for why this is gated rather than probed anyway. + let uses_gpu = device_policy != DevicePolicy::CpuOnly; // `extra_env` can carry secrets (e.g. an HF token passed via `--engine-env`); // this state file is status/introspection only (never read back to relaunch // the container, see `crate::process::launch_service`), so redact values @@ -133,6 +183,23 @@ pub(crate) fn write_running_state( "port": port, "endpoint_url": endpoint, "device_policy": device_policy, + // The ordinals `process::plan_gpus` resolved and `docker::RunSpec` + // narrowed the container to with `HIP_VISIBLE_DEVICES` — the same + // key, read the same way, as `engines/lemonade` and + // `engines/vllm`, so one multi-GPU host does not report + // `rocm gpu 1 (…)` for those two and a bare `rocm` for this one. + // Empty for a CPU-only launch, and for a container this call found + // already running (that launch pinned it, and this one cannot know + // what it chose). + "gpu_indices": gpu_indices, + // The host's GPU name, so the `device:` row names a device and not + // only a backend (ROCMAI-550). Captured once here rather than + // probed per healthcheck: `rocm services list` healthchecks every + // ready service on its render path. `rocm_core::pinned_gpu_name` + // declines on any host where more than one AMD GPU could be the + // one that answered, and is not asked at all when no GPU was + // attached. + "gpu_name": uses_gpu.then(|| rocm_core::pinned_gpu_name(gpu_indices)).flatten(), "engine_recipe": engine_recipe, "started_at_unix_ms": current_unix_millis(), }), @@ -153,24 +220,26 @@ pub(crate) fn read_status(state_path: &Path) -> Option { value_string(&read_service_state(state_path).ok()?, "status") } +/// Patch one key into the service's state file under the state directory's +/// lock ([`rocm_core::merge_json_state_file`]) — the same lock the +/// inference-verification latch and `rocmd`'s Anthropic-endpoint publish take, +/// so no writer drops a key another writer recorded. +/// +/// It used to read, patch and `fs::write` with no lock at all, which could +/// drop either of those. A state file that is empty, torn or not a JSON object +/// is now an error and left as it was, rather than replaced wholesale. fn merge_state_field(state_path: &Path, key: &str, value: Value) -> Result<()> { - let mut state = read_service_state(state_path).unwrap_or_else(|_| json!({})); - if let Some(object) = state.as_object_mut() { - object.insert(key.to_owned(), value); - } - write_state(state_path, &state) + rocm_core::merge_json_state_file(state_path, &json!({ key: value })) } pub(crate) fn write_terminal_state(state_path: &Path, status: &str) -> Result<()> { - let mut state = read_service_state(state_path).unwrap_or_else(|_| json!({})); - if let Some(object) = state.as_object_mut() { - object.insert("status".to_owned(), json!(status)); - object.insert( - "stopped_at_unix_ms".to_owned(), - json!(current_unix_millis()), - ); - } - write_state(state_path, &state) + rocm_core::merge_json_state_file( + state_path, + &json!({ + "status": status, + "stopped_at_unix_ms": current_unix_millis(), + }), + ) } pub(crate) fn read_service_state(path: &Path) -> Result { @@ -179,22 +248,20 @@ pub(crate) fn read_service_state(path: &Path) -> Result { serde_json::from_str(&text).with_context(|| format!("failed to parse {}", path.display())) } -/// Serializes `value` to `path` as pretty JSON, creating the parent directory -/// if needed. The low-level primitive [`write_running_state`], -/// [`merge_state_field`] and [`write_terminal_state`] all go through — it is -/// not the only way this module's state files are written: -/// inference-verification latches go through -/// `rocm_core::merge_json_state_file` instead (see [`inference_verified`]), -/// which patches the file under the state directory's lock rather than -/// routing through here. This function takes no such lock and writes in -/// place, so the lock does not order it against those latches. +/// Replace the service's state file whole, under the state directory's lock +/// and published atomically ([`rocm_core::write_json_state_file`]). +/// +/// [`write_running_state`] is the only caller; the status writers patch +/// ([`merge_state_field`]) instead. It used to be a lock-free `fs::write`, +/// which is a problem this PR's own change made reachable: the document being +/// written now includes a host GPU probe (`rocm_core::pinned_gpu_name`, up to +/// seconds on Windows), and `rocmd`'s Anthropic-proxy thread — a thread of +/// *this* process — publishes its endpoint into the same file in that window. +/// Going through `write_json_state_file` both orders this against every merge +/// and carries that publication through the rewrite, the way +/// `engines/lemonade`'s `write_state` does. fn write_state(path: &Path, value: &Value) -> Result<()> { - if let Some(parent) = path.parent() { - fs::create_dir_all(parent) - .with_context(|| format!("failed to create {}", parent.display()))?; - } - fs::write(path, serde_json::to_vec_pretty(value)?) - .with_context(|| format!("failed to write {}", path.display())) + rocm_core::write_json_state_file(path, value) } pub(crate) fn endpoint_url(host: &str, port: u16) -> String { @@ -360,6 +427,7 @@ mod tests { "127.0.0.1", 8000, DevicePolicy::GpuPreferred, + &[0], Some(&hint), ) .unwrap(); @@ -368,6 +436,206 @@ mod tests { assert!(on_disk.contains("HF_TOKEN")); } + /// ROCMAI-550 changed the device this engine reports for an existing + /// service from the literal `"rocm_gpu"` to the shared + /// ` [gpu ] [()]` shape, and nothing pinned it. + /// + /// This drives the real persist-then-report path: + /// [`write_running_state`] writes the state file, [`read_service_state`] + /// reads it back and [`reported_device`] renders it — exactly what + /// [`healthcheck_service`] does for a tracked service, minus the + /// `state_path` lookup, which resolves through the caller's real + /// `AppPaths` and so cannot be driven from a unit test (the no-state half + /// is covered by + /// `healthcheck_reports_unknown_for_a_service_with_no_state_file`). + /// + /// Reverting the string to `"rocm_gpu"`, dropping the `gpu_name` field or + /// dropping the `gpu_indices` field from the state write fails this. + #[test] + fn reported_device_for_existing_state_matches_the_shared_engine_shape() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("svc-device.json"); + write_running_state( + &path, + "svc-device", + "some/model", + Some("Org/Served"), + "127.0.0.1", + 8000, + DevicePolicy::GpuRequired, + &[1], + None, + ) + .unwrap(); + + // The probed name is whatever the test host has, so assert the shape + // rather than a value: `rocm gpu 1`, optionally followed by + // `()`, and never the old `rocm_gpu`. + let state = read_service_state(&path).unwrap(); + let device = reported_device(&state); + assert!( + device == "rocm gpu 1" || (device.starts_with("rocm gpu 1 (") && device.ends_with(')')), + "AIM must report the shared ` gpu [()]` shape, with the \ + ordinal the container was narrowed to: {device}" + ); + assert!( + !device.contains("rocm_gpu"), + "the pre-ROCMAI-550 literal must not come back: {device}" + ); + // The state file must actually carry both keys the row is built from, + // even when the host probe found no name. + assert!( + state.get("gpu_name").is_some(), + "write_running_state must record gpu_name: {state}" + ); + assert_eq!( + state.get("gpu_indices"), + Some(&json!([1])), + "write_running_state must record the ordinals the container was \ + given, the way lemonade and vLLM do: {state}" + ); + + // And the name, when the host has one, reaches the row. + assert_eq!( + reported_device(&json!({ "gpu_indices": [0], "gpu_name": "AMD Instinct MI300X" })), + "rocm gpu 0 (AMD Instinct MI300X)" + ); + } + + /// A `cpu_only` launch attaches no GPU at all (`process::resolve_use_gpu` + /// returns `false`, and `docker::RunSpec` then passes no `/dev/kfd`, no + /// `/dev/dri` and no `HIP_VISIBLE_DEVICES`), so neither the `rocm` backend + /// nor a GPU name is a fact about it. Before ROCMAI-550's fix this path + /// still probed the host and rendered e.g. + /// `rocm (AMD Instinct MI300X)` for a container that used no GPU — the + /// invented claim the whole ticket exists to remove. + /// + /// Drops to red if `write_running_state` probes for a name regardless of + /// the policy, or if `reported_device` goes back to hardcoding `rocm`. + #[test] + fn a_cpu_only_launch_claims_neither_rocm_nor_a_gpu_name() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("svc-cpu.json"); + write_running_state( + &path, + "svc-cpu", + "some/model", + Some("Org/Served"), + "127.0.0.1", + 8000, + DevicePolicy::CpuOnly, + &[], + None, + ) + .unwrap(); + + let state = read_service_state(&path).unwrap(); + assert_eq!( + state.get("gpu_name"), + Some(&Value::Null), + "a cpu-only launch must not have the host GPU probed for it: {state}" + ); + assert_eq!(reported_device(&state), "cpu"); + + // And the rendering is driven by the recorded policy, not by the + // absence of a name: a state file that somehow carries both still + // reports `cpu`. + assert_eq!( + reported_device(&json!({ + "device_policy": "cpu_only", + "gpu_indices": [0], + "gpu_name": "AMD Instinct MI300X", + })), + "cpu" + ); + // A state file written before the key existed keeps the old reading. + assert_eq!( + reported_device(&json!({ "gpu_name": "AMD Instinct MI300X" })), + "rocm (AMD Instinct MI300X)" + ); + } + + /// The `device` the Healthcheck RPC *itself* answers with, not the private + /// formatter it is supposed to call. + /// + /// Every other device test in the three engines calls `reported_device` + /// directly, so reverting the handler's `Some(state) => reported_device` + /// wiring to a hardcoded literal kept the whole suite green. This drives + /// the handler. + /// + /// The planted state carries no endpoint, so no HTTP probe is made and + /// `listed`/`ready` are false — which is the point: this engine reports + /// the device as soon as it has state, independently of readiness (see + /// `apps/rocm/src/engine_ipc.rs`'s `device_worth_reporting`). + #[test] + fn healthcheck_rpc_answers_with_the_device_the_state_file_describes() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("svc-rpc.json"); + write_state( + &path, + &json!({ + "service_id": "svc-rpc", + "status": "running", + "device_policy": "gpu_required", + "gpu_indices": [2], + "gpu_name": "AMD Instinct MI300X", + }), + ) + .unwrap(); + + let response = healthcheck_service_at(&path).unwrap(); + assert_eq!(response.device, "rocm gpu 2 (AMD Instinct MI300X)"); + assert_eq!(response.status, "running"); + assert!(!response.model_loaded); + + // And a service with no state on disk is the `"unknown"` the CLI + // filters out rather than prints. + let response = healthcheck_service_at(&dir.path().join("absent.json")).unwrap(); + assert_eq!(response.device, "unknown"); + } + + /// The post-spawn whole-file rewrite must not drop the Anthropic endpoint + /// a sibling thread of this same process published moments earlier. + /// + /// `rocm`'s `run_builtin_engine_serve_http` runs `rocmd`'s Anthropic proxy + /// as a thread of the process running this adapter, and ROCMAI-550 put a + /// host GPU probe (seconds, on Windows) inside the rewrite — wide enough + /// for the publish to land inside it. Routing [`write_state`] through + /// `rocm_core::write_json_state_file` is what carries it through; putting + /// the raw `fs::write` back drops the key and fails this. + #[test] + fn a_running_state_write_keeps_the_anthropic_endpoint_this_process_published() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("svc-anthropic.json"); + rocm_core::publish_anthropic_endpoint( + &path, + "http://127.0.0.1:9931", + rocm_core::AnthropicEndpointMode::Proxy, + ) + .unwrap(); + + write_running_state( + &path, + "svc-anthropic", + "some/model", + None, + "127.0.0.1", + 8000, + DevicePolicy::GpuRequired, + &[0], + None, + ) + .unwrap(); + + let state = read_service_state(&path).unwrap(); + assert_eq!( + value_string(&state, rocm_core::ANTHROPIC_ENDPOINT_URL_STATE_KEY).as_deref(), + Some("http://127.0.0.1:9931"), + "the whole-file rewrite dropped the endpoint the proxy thread \ + published: {state}" + ); + } + #[test] fn status_helpers_round_trip_and_keep_other_fields() { let dir = tempfile::tempdir().unwrap(); diff --git a/engines/lemonade/src/device_policy.rs b/engines/lemonade/src/device_policy.rs index 00ca729d6..76d161062 100644 --- a/engines/lemonade/src/device_policy.rs +++ b/engines/lemonade/src/device_policy.rs @@ -134,14 +134,40 @@ pub(crate) const fn device_policy_name(policy: &DevicePolicy) -> &'static str { } } -/// The device string reported once a model is loaded. Reflects the backend that -/// actually ran (`rocm`, `vulkan`, …) and the pinned GPU ordinal when known, -/// rather than a hardcoded value, so the report matches observed execution. +/// The device string reported once a model is loaded: the backend this service +/// was launched on (`rocm`, `vulkan`, …), the pinned GPU ordinal when known, and +/// the GPU name recorded at launch. +/// +/// Read back from the service's own recorded `backend_requested` rather than +/// from a hardcoded value — so it follows the backend selection this launch +/// actually made, including a ROCm-to-Vulkan fallback. It is this engine's +/// account of what it asked llama.cpp for, not an independent measurement of +/// which device executed, which is why the CLI's docs say the row *reports* a +/// device rather than naming the one observed. +/// +/// Formatted by [`rocm_engine_protocol::format_reported_device`] so all three +/// engines render one shape; see that function for why the ordinal-less row +/// drops the `gpu` noun rather than printing a bare `vulkan gpu`. The ordinal +/// is read by [`rocm_engine_protocol::first_gpu_index_from_state`] for the same +/// reason — `engines/vllm` reads the same `gpu_indices` key out of a state file +/// of the same shape, and kept a byte-identical copy of that read until it +/// moved next to the formatter. pub(crate) fn reported_device(state: Option<&Value>, backend: &str) -> String { - match state.and_then(first_gpu_index_from_state) { - Some(index) => format!("{backend} gpu {index}"), - None => format!("{backend} gpu"), - } + rocm_engine_protocol::format_reported_device( + backend, + state.and_then(rocm_engine_protocol::first_gpu_index_from_state), + state.and_then(gpu_name_from_state).as_deref(), + ) +} + +/// The GPU name recorded in service state at launch, if any. +/// +/// Read back rather than re-probed: `healthcheck_service` runs on the +/// `rocm services list` render path (see `fetch_service_devices`), where an +/// extra host probe per service is latency the row is not worth, and the launch +/// host is the one whose GPU actually ran the server. +fn gpu_name_from_state(state: &Value) -> Option { + crate::state::value_string(state, "gpu_name") } /// A one-line notice naming the backend actually serving when it is a GPU @@ -176,16 +202,6 @@ pub(crate) fn announce_non_rocm_backend(backend: &str, log_path: Option<&Path>) Some(notice) } -/// The first pinned GPU ordinal recorded in service state, if any. -fn first_gpu_index_from_state(state: &Value) -> Option { - state - .get("gpu_indices") - .and_then(Value::as_array) - .and_then(|indices| indices.first()) - .and_then(Value::as_u64) - .and_then(|value| u32::try_from(value).ok()) -} - #[cfg(test)] mod tests { use super::*; @@ -240,11 +256,28 @@ mod tests { } #[test] - fn reported_device_reflects_backend_and_pinned_ordinal() { - let state = json!({ "gpu_indices": [1] }); - assert_eq!(reported_device(Some(&state), "vulkan"), "vulkan gpu 1"); + fn reported_device_reflects_backend_pinned_ordinal_and_gpu_name() { + let state = json!({ "gpu_indices": [1], "gpu_name": "AMD Instinct MI300X" }); + assert_eq!( + reported_device(Some(&state), "vulkan"), + "vulkan gpu 1 (AMD Instinct MI300X)" + ); // Without a recorded ordinal the backend is still reported truthfully. - assert_eq!(reported_device(None, "rocm"), "rocm gpu"); + assert_eq!(reported_device(None, "rocm"), "rocm"); + } + + #[test] + fn reported_device_names_the_gpu_when_no_ordinal_was_pinned() { + let state = json!({ "gpu_indices": [], "gpu_name": "AMD Radeon(TM) Graphics" }); + assert_eq!( + reported_device(Some(&state), "vulkan"), + "vulkan (AMD Radeon(TM) Graphics)" + ); + + // An older state file, written before `gpu_name` was recorded, still + // reports its backend rather than a dangling noun. + let legacy = json!({ "gpu_indices": [] }); + assert_eq!(reported_device(Some(&legacy), "vulkan"), "vulkan"); } #[test] diff --git a/engines/lemonade/src/state.rs b/engines/lemonade/src/state.rs index 82c7df0ed..81dca5318 100644 --- a/engines/lemonade/src/state.rs +++ b/engines/lemonade/src/state.rs @@ -90,6 +90,16 @@ pub(crate) fn write_running_state_for_model( "endpoint_url": endpoint_url(&request.host, request.port), "device_policy": device_policy_name(&request.device_policy), "gpu_indices": request.gpu_indices, + // The name of the GPU this service was pinned to, so + // `healthcheck_service` can name a device and not just a backend + // (ROCMAI-550). Captured here, once, rather than probed per + // healthcheck: `rocm services list` healthchecks every ready + // service on its render path. `rocm_core::pinned_gpu_name` answers + // `null` whenever the host probe cannot be attributed to the pinned + // ordinal (a second AMD GPU on the host, a visibility mask); the + // key is recorded either way, so its absence means an older state + // file rather than a probe that declined. + "gpu_name": rocm_core::pinned_gpu_name(&request.gpu_indices), "runtime_id": request.runtime_id.as_deref().unwrap_or(runtime.manifest.env_id.as_str()), "env_id": request.env_id.as_deref().unwrap_or(runtime.manifest.env_id.as_str()), "runtime_kind": "lemonade_embeddable", @@ -141,6 +151,13 @@ pub(crate) fn read_service_state(path: &Path) -> Result { /// with every merge (`rocm_core::write_json_state_file`), so a full rewrite /// never lands inside another writer's merge, and publishes atomically, so a /// lock-free reader such as `rocm services list` never sees half a document. +/// +/// It replaces every key merged before it, with one exception that +/// `write_json_state_file` owns: an Anthropic endpoint published by this very +/// process is carried through. The proxy that publishes it is a thread of this +/// process and publishes exactly once, so the rewrite that happens moments +/// after the server is spawned would otherwise lose it for good — see +/// `a_running_state_write_keeps_the_anthropic_endpoint_this_process_published`. fn write_state(path: &Path, value: &Value) -> Result<()> { rocm_core::write_json_state_file(path, value) } @@ -351,6 +368,82 @@ mod tests { use std::io::{Read, Write}; use std::time::Duration; + /// ROCMAI-550 regression, on the real writer: the `running` state write + /// happens immediately after `serve_router` spawns the server, and since + /// ROCMAI-550 it spends a host GPU probe (`rocm_core::pinned_gpu_name`, + /// `pnputil` plus a PowerShell CIM query on Windows) building its document + /// before writing it. The Anthropic proxy `rocm` starts on a thread of this + /// same process publishes its endpoint as soon as the server lists a model + /// — inside that window — and nothing publishes twice, so a rewrite that + /// drops the endpoint drops it for the life of the service. That is what + /// left five Windows e2e scenarios with a ready service and no + /// `anthropic endpoint:` row. + /// + /// This drives the production call, not the primitive underneath it, so it + /// also holds the routing: a `write_state` that went back to a plain + /// `fs::write` would fail here while `rocm-core`'s own tests stayed green. + #[test] + fn a_running_state_write_keeps_the_anthropic_endpoint_this_process_published() { + let dir = scratch_dir("anthropic-endpoint-survives-rewrite"); + let request = ServeHttpRequest { + service_id: "svc".to_owned(), + model_ref: "Qwen3-0.6B-GGUF".to_owned(), + host: "127.0.0.1".to_owned(), + port: 11435, + device_policy: rocm_engine_protocol::DevicePolicy::GpuRequired, + gpu_indices: Vec::new(), + runtime_id: None, + env_id: None, + state_path: dir.join("svc.json"), + log_path: None, + engine_recipe: None, + }; + let runtime = LemonadeRuntime { + manifest: LemonadeInstallManifest { + env_id: "test".to_owned(), + version: "test".to_owned(), + runtime_dir: dir.join("runtime"), + lemond: dir.join("lemond"), + lemonade: dir.join("lemonade"), + backend_recipe: crate::LLAMACPP_RECIPE.to_owned(), + backend_name: "rocm".to_owned(), + installed_at_unix_ms: 0, + }, + }; + + // The serve's own first write, then the proxy thread's publication. + write_running_state(&request, &runtime, std::process::id(), None, "starting").unwrap(); + rocm_core::publish_anthropic_endpoint( + &request.state_path, + "http://127.0.0.1:41000", + rocm_core::AnthropicEndpointMode::Proxy, + ) + .unwrap(); + + // The post-spawn rewrite, which replaces the whole document. + write_running_state( + &request, + &runtime, + std::process::id(), + Some(4242), + "running", + ) + .unwrap(); + + let state = read_service_state(&request.state_path).unwrap(); + assert_eq!( + state["anthropic_endpoint_url"], "http://127.0.0.1:41000", + "the rewrite dropped the endpoint its own process published, so \ + `rocm services list` will never show an `anthropic endpoint:` row \ + for this service: {state}" + ); + assert_eq!(state["anthropic_endpoint_mode"], "proxy"); + // The rewrite still does its own job. + assert_eq!(state["status"], "running"); + assert_eq!(state["server_pid"], 4242); + fs::remove_dir_all(dir).ok(); + } + #[test] fn identity_from_state_prefers_server_pid_and_its_start_ticks() { let state = json!({ diff --git a/engines/vllm/src/state.rs b/engines/vllm/src/state.rs index 7e87aa893..7acaf5ae0 100644 --- a/engines/vllm/src/state.rs +++ b/engines/vllm/src/state.rs @@ -34,7 +34,19 @@ pub(crate) struct ServiceFiles { pub(crate) fn healthcheck_service(request: HealthcheckRequest) -> Result { require_nonempty(&request.service_id, "service_id")?; let files = service_files(&request.service_id)?; - let state = read_service_state(&files.state_path).ok(); + healthcheck_service_at(&files.state_path) +} + +/// [`healthcheck_service`] against an explicit state path. +/// +/// Split out the way `engines/lemonade`'s `healthcheck_service_at` already is: +/// [`service_files`] resolves through the caller's real `AppPaths`, so the RPC +/// handler itself could not be driven from a test, and the `device` it answers +/// with was only covered by calling [`reported_device`] directly — a test that +/// stays green if this handler stops consulting it at all (see +/// `healthcheck_rpc_answers_with_the_device_the_state_file_describes`). +pub(crate) fn healthcheck_service_at(state_path: &Path) -> Result { + let state = read_service_state(state_path).ok(); let endpoint_url = state.as_ref().and_then(endpoint_url_from_state); let model_ref = state .as_ref() @@ -51,7 +63,7 @@ pub(crate) fn healthcheck_service(request: HealthcheckRequest) -> Result Result reported_device(state), + None => "unknown".to_owned(), }; Ok(HealthcheckResponse::for_readiness( listed, ready, &state_status, - device, + &device, )) } +/// The device string reported once state exists for the service. +/// +/// vLLM only ever runs on ROCm here (`device_policy` is pinned to +/// `gpu_required` at launch), so the backend half is fixed — but the shape has +/// to match what `engines/lemonade` reports, because +/// `rocm serve`'s `device:` row and `rocm services list` print whichever engine +/// answered verbatim. It used to be the literal `"rocm_gpu"`, so the same row +/// read `rocm_gpu` for vLLM and `vulkan gpu 0` for Lemonade, and neither +/// README.md nor `docs/manual-testing.md` described the former. Both halves are +/// now formatted by [`rocm_engine_protocol::format_reported_device`], shared +/// with Lemonade, which is also what keeps an unpinned launch from rendering a +/// dangling `rocm gpu`. +/// +/// The ordinal comes from +/// [`rocm_engine_protocol::first_gpu_index_from_state`] for the same reason: +/// this engine and Lemonade read the same `gpu_indices` key out of state files +/// of the same shape, and had byte-identical private copies of that read. +fn reported_device(state: &Value) -> String { + rocm_engine_protocol::format_reported_device( + "rocm", + rocm_engine_protocol::first_gpu_index_from_state(state), + value_string(state, "gpu_name").as_deref(), + ) +} + pub(crate) fn endpoint_response(request: EndpointRequest) -> Result { require_nonempty(&request.service_id, "service_id")?; let files = service_files(&request.service_id)?; @@ -150,6 +186,19 @@ pub(crate) fn write_running_state( "port": request.port, "endpoint_url": endpoint_url(&request.host, request.port), "device_policy": "gpu_required", + // Recorded so `healthcheck_service` can report the GPU this service + // was actually pinned to, in the same shape `engines/lemonade` + // reports. Introspection only, like the rest of this file — never + // read back to relaunch. + "gpu_indices": request.gpu_indices, + // The name of the GPU this service was pinned to, so the `device:` + // row names a device and not only a backend (ROCMAI-550). Captured + // once here rather than probed per healthcheck, because `rocm + // services list` healthchecks every ready service on its render + // path. `rocm_core::pinned_gpu_name` answers `null` when the host + // probe cannot be attributed to the pinned ordinal, so the row + // falls back to `rocm gpu ` rather than naming another card. + "gpu_name": rocm_core::pinned_gpu_name(&request.gpu_indices), "runtime_id": runtime.runtime_id, "requested_runtime_id": request.runtime_id, "env_id": request.env_id.as_deref().unwrap_or(runtime.env_id.as_str()), @@ -168,16 +217,18 @@ pub(crate) fn write_running_state( ) } +/// Stamps a terminal status, patching the file under the state directory's +/// lock ([`rocm_core::merge_json_state_file`]) rather than reading it and +/// writing it back unlocked — so it cannot drop a key the inference latch or +/// `rocmd`'s Anthropic-endpoint publish recorded in between. pub(crate) fn write_terminal_state(state_path: &Path, status: &str) -> Result<()> { - let mut state = read_service_state(state_path).unwrap_or_else(|_| json!({})); - if let Some(object) = state.as_object_mut() { - object.insert("status".to_owned(), Value::String(status.to_owned())); - object.insert( - "stopped_at_unix_ms".to_owned(), - Value::from(current_unix_millis() as u64), - ); - } - write_state(state_path, &state) + rocm_core::merge_json_state_file( + state_path, + &json!({ + "status": status, + "stopped_at_unix_ms": current_unix_millis() as u64, + }), + ) } pub(crate) fn read_service_state(path: &Path) -> Result { @@ -186,25 +237,20 @@ pub(crate) fn read_service_state(path: &Path) -> Result { serde_json::from_str(&text).with_context(|| format!("failed to parse {}", path.display())) } -/// Serializes `value` to `path` as pretty JSON, creating the parent directory -/// if needed. The low-level primitive [`write_running_state`] and -/// [`write_terminal_state`] both go through — it is not the only way this -/// module's state files are written: inference-verification latches go -/// through `rocm_core::merge_json_state_file` instead (see -/// [`inference_verified`]), which patches the file under the state -/// directory's lock rather than routing through here. This function takes no -/// such lock and writes in place, so the lock does not order it against those -/// latches. +/// Replace the service's state file whole, under the state directory's lock +/// and published atomically ([`rocm_core::write_json_state_file`]). +/// +/// [`write_running_state`] is the only caller; the terminal status is a patch +/// ([`write_terminal_state`]). It used to be a lock-free `fs::write`, which is +/// a problem this PR's own change made reachable: the document being written +/// now includes a host GPU probe (`rocm_core::pinned_gpu_name`, up to seconds +/// on Windows), and `rocmd`'s Anthropic-proxy thread — a thread of *this* +/// process — publishes its endpoint into the same file in that window. Going +/// through `write_json_state_file` both orders this against every merge and +/// carries that publication through the rewrite, as `engines/lemonade`'s +/// `write_state` does. fn write_state(path: &Path, value: &Value) -> Result<()> { - if let Some(parent) = path.parent() { - fs::create_dir_all(parent) - .with_context(|| format!("failed to create {}", parent.display()))?; - } - fs::write( - path, - serde_json::to_vec_pretty(value).context("failed to serialize vLLM state")?, - ) - .with_context(|| format!("failed to write {}", path.display())) + rocm_core::write_json_state_file(path, value) } pub(crate) fn endpoint_url(host: &str, port: u16) -> String { @@ -706,4 +752,99 @@ mod tests { fn identity_from_state_without_pid_is_none() { assert!(identity_from_state(&json!({ "status": "running" })).is_none()); } + + /// The `device:` row `rocm serve` and `rocm services list` print comes + /// straight from this string, so vLLM has to speak the same shape + /// `engines/lemonade` does. It used to be the literal `"rocm_gpu"`, which + /// is the format README.md and `docs/manual-testing.md` never described and + /// which made one row disagree with itself across engines. + #[test] + fn reported_device_matches_the_shape_lemonade_reports() { + assert_eq!( + reported_device(&json!({ "gpu_indices": [1], "gpu_name": "AMD Instinct MI300X" })), + "rocm gpu 1 (AMD Instinct MI300X)" + ); + assert_eq!( + reported_device(&json!({ "gpu_indices": [1] })), + "rocm gpu 1" + ); + // No pinned ordinal recorded: the backend is still reported truthfully, + // exactly as `engines/lemonade`'s `reported_device(None, "rocm")` does, + // and without the dangling `gpu` noun that names no device. + assert_eq!(reported_device(&json!({ "status": "running" })), "rocm"); + assert_eq!(reported_device(&json!({ "gpu_indices": [] })), "rocm"); + assert_eq!( + reported_device(&json!({ "gpu_name": "AMD Instinct MI300X" })), + "rocm (AMD Instinct MI300X)" + ); + } + + /// The `device` the Healthcheck RPC *itself* answers with, not the private + /// formatter it is supposed to call. + /// + /// Every other device test in the three engines calls `reported_device` + /// directly, so reverting this handler's `Some(state) => reported_device` + /// wiring to the old hardcoded literal kept the whole suite green. This + /// drives the handler. + /// + /// The planted state carries no endpoint, so no HTTP probe is made and + /// `listed`/`ready` are false — which is the point: this engine reports + /// the device as soon as it has state, independently of readiness (unlike + /// `engines/lemonade`, see `apps/rocm/src/engine_ipc.rs`'s + /// `device_worth_reporting`). + #[test] + fn healthcheck_rpc_answers_with_the_device_the_state_file_describes() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("svc-rpc.json"); + write_state( + &path, + &json!({ + "status": "running", + "gpu_indices": [3], + "gpu_name": "AMD Instinct MI300X", + }), + ) + .unwrap(); + + let response = healthcheck_service_at(&path).unwrap(); + assert_eq!(response.device, "rocm gpu 3 (AMD Instinct MI300X)"); + assert_eq!(response.status, "running"); + assert!(!response.model_loaded); + + // And a service with no state on disk is the `"unknown"` the CLI + // filters out rather than prints. + let response = healthcheck_service_at(&dir.path().join("absent.json")).unwrap(); + assert_eq!(response.device, "unknown"); + } + + /// The post-spawn whole-file rewrite must not drop the Anthropic endpoint + /// a sibling thread of this same process published moments earlier. + /// + /// `rocm`'s `run_builtin_engine_serve_http` runs `rocmd`'s Anthropic proxy + /// as a thread of the process running this adapter, and ROCMAI-550 put a + /// host GPU probe (seconds, on Windows) inside the rewrite — wide enough + /// for the publish to land inside it. Routing [`write_state`] through + /// `rocm_core::write_json_state_file` is what carries it through; putting + /// the raw `fs::write` back drops the key and fails this. + #[test] + fn a_running_state_write_keeps_the_anthropic_endpoint_this_process_published() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("svc-anthropic.json"); + rocm_core::publish_anthropic_endpoint( + &path, + "http://127.0.0.1:9932", + rocm_core::AnthropicEndpointMode::Proxy, + ) + .unwrap(); + + write_state(&path, &json!({ "status": "running", "gpu_indices": [0] })).unwrap(); + + let state = read_service_state(&path).unwrap(); + assert_eq!( + value_string(&state, rocm_core::ANTHROPIC_ENDPOINT_URL_STATE_KEY).as_deref(), + Some("http://127.0.0.1:9932"), + "the whole-file rewrite dropped the endpoint the proxy thread \ + published: {state}" + ); + } } diff --git a/tests/e2e-cucumber/features/examine.feature b/tests/e2e-cucumber/features/examine.feature index a5a7d1c52..27b03d497 100644 --- a/tests/e2e-cucumber/features/examine.feature +++ b/tests/e2e-cucumber/features/examine.feature @@ -793,3 +793,24 @@ Feature: GPU detection and system inspection Given the active runtime is a nightly runtime installed for one exact GPU When the user inspects the system Then the runtime state names the channel, package source, source layout and exact GPU target that runtime was installed from + + # ROCMAI-550. `rocm examine` answers "can this machine run ROCm?" with a + # `rocm_support` line on EVERY host -- the ROCm-capable runners and the no-GPU + # mock lane included, not only the Vulkan-only hosts the ticket started from. + # Other surfaces send people here for that answer (`lemonade_engine_note` in + # apps/rocm/src/main.rs tells an unclassifiable host to run `rocm examine`), so + # silence there is a dead end rather than a smaller report. + # + # Deliberately untagged: the line's content is keyed to what this host turns + # out to be, so the lanes disagreeing about their GPUs is the coverage rather + # than an obstacle to it. The assertion is a cross-check and not a tautology -- + # the claim is held against `detected_gfx_target` and + # `compatible_therock_family` from the same report, so a line reading "full + # ROCm/HIP support" on the mock lane, or "not supported by ROCm" on an MI300X, + # fails here. + @id:examine-reports-this-hosts-rocm-support-level + Scenario: examine-55 - System inspection says whether this machine's GPU is supported by ROCm + When the user inspects the system + Then the inspection states this host's ROCm support level + And the support level agrees with the GPU the inspection detected + And the support level leaves a command that fits this host diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index d6db3d214..a5fed92b1 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -52,6 +52,7 @@ Feature: Model serving When the user sends a chat completion request Then the response contains a model reply And the response identifies the correct model + And the services list reports the engine's actual backend device # Large-model coverage (dogfooding W9): serve a representative large model for # each GPU platform end-to-end at least once. The model follows the lane's serve @@ -71,6 +72,14 @@ Feature: Model serving # Lemonade serve + inference (GGUF model). Engine coverage: Lemonade. The # lemonade per-PR canary: one real lemonade serve runs on every PR (the # counterpart to the vLLM canary above). + # + # ROCMAI-550's device-reporting Then is listed here for the lanes that run it, + # but it is NOT coverage on lemonade+linux: the whole scenario is xfail'd there + # (EAI-7423 -- the managed serve reaches ready then shuts down ~0.08s later), + # so no step of it is a live assertion on that lane. Moving the Then earlier + # would not change that: the service is gone before a healthcheck could report + # a device. The vLLM canary (serve-05) is what actually exercises device + # reporting on every PR today. @id:serve-lemonade-inference @requires-gpu @requires-engine:lemonade Scenario: serve-07 - A model served on lemonade responds to inference requests Given a managed runtime is active @@ -78,6 +87,7 @@ Feature: Model serving When the user sends a chat completion request Then the response contains a model reply And the response identifies the correct model + And the services list reports the engine's actual backend device # Explicit HF-checkpoint canary: `owner/repo:variant` runs the selected GPU # llama-server directly and fails if that backend binary is missing. Named @@ -903,6 +913,7 @@ Feature: Model serving And the user is told the model needs an explicit GGUF variant And no quantization is suggested for a repo the CLI has not inspected And no local server was recorded + # The pinned Lemonade catalog's smallest vision GGUF needs its mmproj file. # Serving its registered name must retain the router, not flatten the entry # into a single-checkpoint direct serve. Use a scenario-owned port and service; @@ -1019,3 +1030,27 @@ Feature: Model serving When the user lists the engines of that install Then vLLM is reported as installed but not usable And the report names the extension that failed to import + + # ROCMAI-550, and the evidence that the Vulkan remediation is not a dead end. + # `rocm install sdk` on a gfx90c APU refuses and tells the user to run `rocm + # serve --engine lemonade`; AGENTS.md section 3 only allows that if + # serve's no-usable-GPU pre-flight accepts the very host the refusal was + # printed on. That host is a Renoir/Cezanne laptop: the iGPU is on the bus and + # bound to amdgpu, but the kernel's KFD topology lists no compute node for it, + # so the target can only have come from the DRM ip-discovery read. + # + # The assertion is positive rather than "the GPU refusal is absent": an + # out-of-range `--gpu` is validated AFTER the pre-flight and prints the usable + # ordinals it found, so the refusal text here can only be produced by a + # pre-flight that resolved exactly one usable GPU on this host. Take away + # `combine_amd_gpu_counts`'s zero-KFD fallback to DRM and this turns into "no + # usable AMD GPU detected", which is what the step rejects by name. + # + # Linux only, and needs no AMD hardware: the host is planted under + # ROCM_CLI_TEST_HOST_ROOT (an `e2e-test-hooks` build), so the GPU-less mock + # lane is where this runs. + @id:serve-accepts-an-apu-the-kernel-gives-no-compute-node @requires-os:linux + Scenario: serve-63 - Serving is not refused on a Vulkan-only APU the kernel gives no compute node + Given a host whose only GPU is a gfx90c Vega iGPU the kernel reports no compute node for + When the user is told to serve on Lemonade because this GPU has no ROCm build + Then serving got past the no-GPU pre-flight and reported this host's usable GPU diff --git a/tests/e2e-cucumber/features/runtime_setup.feature b/tests/e2e-cucumber/features/runtime_setup.feature index 2114a7b35..96b9eac92 100644 --- a/tests/e2e-cucumber/features/runtime_setup.feature +++ b/tests/e2e-cucumber/features/runtime_setup.feature @@ -441,11 +441,21 @@ Feature: Runtime configuration # build" is absent however the run then ends. The second guards the weaker # failure where a refusal is printed but still dangles a family name in front # of the user. + # + # ROCMAI-550 added a third Then. Having taken every package family away, the + # refusal now has to leave the user somewhere to go, and it points at `rocm + # examine`. That is a remediation line naming a command, so AGENTS.md §3 wants + # it asserted where the refusal is actually produced rather than only against + # the formatter -- this is the one door of the two that a lane can open, so it + # is asserted here. It also pins the negative: this branch refuses a requested + # target before probing the host at all, so it must not predict which backend + # serving would use on a runner whose own GPU is ROCm-capable. @id:runtime-install-sdk-refuses-unsupported-gpu-target Scenario: runtime-19 - Asking for a GPU target ROCm does not build is refused outright When the user dry-runs an SDK install for the gfx90c Vega iGPU Then the install is refused as having no ROCm build for that GPU And the refusal offers no package family to install instead + And the refusal points at examining what this host supports # When the machine already has its own (unmanaged) ROCm and that ROCm is newer # than the SDK the install picks, the preview says why the older SDK is diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 6174cab2a..8964f0c95 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -83,6 +83,14 @@ pub struct E2eWorld { pub artifact_marker_path: Option, pub endpoint: Option, pub model_name: Option, + /// Stdout of the `rocm serve` that this scenario's Given step launched, + /// kept so a Then can assert on the deployment summary it printed. + /// + /// Separate from `cli_output`, which belongs to the scenario's When step: + /// these serves happen in a *precondition*, and overwriting `cli_output` + /// from a Given would change what every existing Then on those scenarios + /// reads. `None` when the scenario served nothing. + pub serve_summary: Option, pub chat_response: Option, /// The reply to a request sent to the served model's Anthropic-compatible /// `/v1/messages` endpoint. @@ -389,6 +397,7 @@ impl Default for E2eWorld { artifact_marker_path: None, endpoint: None, model_name: None, + serve_summary: None, chat_response: None, anthropic_response: None, anthropic_stream: None, diff --git a/tests/e2e-cucumber/tests/e2e/examine_steps.rs b/tests/e2e-cucumber/tests/e2e/examine_steps.rs index a3b34fef3..294fbe85f 100644 --- a/tests/e2e-cucumber/tests/e2e/examine_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/examine_steps.rs @@ -3436,3 +3436,183 @@ async fn user_follows_the_runtime_librocdxg_advice(world: &mut E2eWorld) { world.cli_stderr = Some(stderr); world.cli_rc = Some(rc); } + +/// The `rocm_support` line's value, or `None` when the report has no such line. +fn rocm_support_line(output: &str) -> Option<&str> { + field_value(output, "rocm_support") +} + +/// What `rocm examine` reported as the detected GPU target and the ROCm package +/// family covering it, with the report's `` placeholder read back as +/// "absent" so the two can be reasoned about as the CLI's own +/// `gpu_support_level` inputs. +fn detected_gpu_facts(output: &str) -> (Option<&str>, Option<&str>) { + let field = |name| field_value(output, name).filter(|value| *value != ""); + ( + field("detected_gfx_target"), + field("compatible_therock_family"), + ) +} + +/// ROCMAI-550: the report answers the support question on every host, including +/// the ones where the honest answer is "this build cannot tell". +#[then("the inspection states this host's ROCm support level")] +async fn assert_examine_states_a_support_level(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no command was run"); + let line = rocm_support_line(output).unwrap_or_else(|| { + panic!( + "`rocm examine` printed no `rocm_support` line. Other surfaces send \ + the user here to find out whether this GPU is supported, so there \ + is no host on which the answer may be silence:\n{output}" + ) + }); + assert!( + !line.is_empty(), + "the `rocm_support` line states nothing:\n{output}" + ); +} + +/// The cross-check. The claim is judged against the two facts the same report +/// prints, not against a hardcoded expectation, so this says something on every +/// lane: the mock lane detects no GPU and must not be told it has ROCm support; +/// the GPU runners have a covering package family and must not be told their +/// card has no ROCm build. +#[then("the support level agrees with the GPU the inspection detected")] +async fn assert_support_level_agrees_with_detection(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no command was run"); + let line = rocm_support_line(output).expect("no rocm_support line"); + let (detected, family) = detected_gpu_facts(output); + + match (detected, family) { + // A package family covers this GPU: full ROCm/HIP support, and the + // report must not hedge it into either of the weaker answers. + (_, Some(_)) => { + assert!( + line.contains("full ROCm/HIP support"), + "a GPU with a covering ROCm package family is supported, and the \ + support line must say so: {line}\n{output}" + ); + assert!( + !line.contains("not supported by ROCm") && !line.starts_with("unknown"), + "{line}\n{output}" + ); + } + // Nothing detected at all. The honest answer is "unknown"; claiming + // either support or a working Vulkan backend would be inventing + // hardware. + (None, None) => { + assert!( + line.starts_with("unknown"), + "no GPU was detected, so neither support nor its absence can be \ + claimed: {line}\n{output}" + ); + assert!( + !line.contains("Vulkan backend can still be used"), + "no backend may be promised for a GPU that was not found: \ + {line}\n{output}" + ); + } + // Detected, with no family covering it. Both answers are legitimate + // here and which one is correct depends on the catalog: `gfx90c` is + // *known* to have no ROCm build (Vulkan-only), anything unrecognised is + // a gap in what this build knows (unknown). What is never legitimate is + // claiming ROCm support for a GPU no package family covers. + (Some(gfx), None) => { + assert!( + line.contains(gfx), + "the support line must name the target it is judging: {line}\n{output}" + ); + assert!( + line.contains("not supported by ROCm") || line.contains("unknown"), + "no ROCm package family covers {gfx}, so the line must report \ + either a Vulkan-only GPU or an unknown one: {line}\n{output}" + ); + assert!( + !line.contains("full ROCm/HIP support"), + "a GPU with no covering package family must not be reported as \ + ROCm-supported: {line}\n{output}" + ); + } + } +} + +/// AGENTS.md wants a line that tells the user what they can do, and a named +/// command that can actually clear the condition it was printed for. +/// +/// So the assertion is the *specific* command each tier owes, not "some `rocm` +/// word appears": `contains("rocm ")` is satisfied by `rocm examine` echoing +/// its own name, by a stray `rocm diagnose` on a ROCm-capable host, and by +/// every wrong-tier command there is. Which tier applies is read from the same +/// three fields the renderer classifies on, so this says something different — +/// and falsifiable — on each lane rather than collapsing to one weak check. +/// +/// The placeholders those fields fall back to are pinned by +/// `rocm_core`'s `examine_render_spells_its_absent_value_placeholders`, so this +/// cannot silently start reading every host as "nothing detected" if the +/// renderer's wording changes. +#[then("the support level leaves a command that fits this host")] +async fn assert_support_level_advice_fits(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no command was run"); + let line = rocm_support_line(output).expect("no rocm_support line"); + let (detected, family) = detected_gpu_facts(output); + // `detected_therock_family` is the managed SDK that *is* installed, as + // opposed to `compatible_therock_family`, which is what this GPU could run. + let installed_sdk = field_value(output, "detected_therock_family") + .filter(|value| *value != "" && *value != ""); + + let (expected, forbidden): (&str, &[&str]) = match (detected, family, installed_sdk) { + // ROCm-capable with the managed SDK for THIS GPU's family already in + // place: installing one clears nothing, so the line must point at + // serving instead. + (_, Some(family), Some(installed)) if installed == family => { + ("rocm serve ", &["rocm install sdk"]) + } + // ROCm-capable with a managed SDK installed, but for another family — + // a GPU swap, an eGPU, or a second card. The install on disk was never + // built for this GPU, so the one it needs is still the next step and + // the line must not offer to serve on what is there. + (_, Some(_), Some(_)) => ("rocm install sdk", &["serve on it"]), + // ROCm-capable with no managed SDK yet: that install is the next step. + (_, Some(_), None) => ("rocm install sdk", &[]), + // A GPU no package family covers. Two different tiers land here and + // they do NOT owe the same command, which this arm used to miss: + // + // * merely unrecognized — the catalog has no opinion, and it may be a + // newer ROCm-capable card, so pinning the Vulkan backend would claim + // something nothing established. Plain `rocm serve `, and + // `--engine lemonade` is forbidden. + // * confirmed to have no ROCm build (gfx90c) — Vulkan is the backend + // that can serve it, so the advice names it. + // + // Told apart by the line's own level wording, which is what the user + // reads. Before this split the arm required `--engine lemonade` for + // both, so a lane with an unrecognized GPU would have failed against + // *correct* output; no current lane has one, which is the only reason + // it stayed green. + (Some(_), None, _) if line.contains("is not in this build's catalog") => ( + "rocm serve ", + &["rocm install sdk", "--engine lemonade"], + ), + (Some(_), None, _) => ( + "rocm serve --engine lemonade", + &["rocm install sdk"], + ), + // Nothing detected: `rocm serve` is refused outright by the + // no-usable-GPU pre-flight, so naming it would be advice that cannot + // clear the condition it is printed for. + (None, None, _) => ("rocm diagnose", &["rocm serve", "rocm install sdk"]), + }; + + assert!( + line.contains(expected), + "this host's support tier owes `{expected}`, and the line has to name a \ + command that can actually clear the condition it was printed for: \ + {line}\n{output}" + ); + for command in forbidden { + assert!( + !line.contains(command), + "`{command}` cannot clear anything on this host: {line}\n{output}" + ); + } +} diff --git a/tests/e2e-cucumber/tests/e2e/runtime_steps.rs b/tests/e2e-cucumber/tests/e2e/runtime_steps.rs index 1912039f1..269f71507 100644 --- a/tests/e2e-cucumber/tests/e2e/runtime_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/runtime_steps.rs @@ -2056,3 +2056,29 @@ async fn assert_no_runtime_registered(world: &mut E2eWorld) { "the refused adoption left a runtime registered:\n{stdout}" ); } + +/// Having removed every `--family` the user could try, the refusal has to leave +/// somewhere to go — and that somewhere must be true on whatever host runs it. +/// +/// This scenario asks for `--family gfx90c`, which is refused by +/// `unsupported_requested_target_message` before any host probe runs, so the +/// runner's own GPU has no bearing on the refusal. That makes this lane the +/// proof that the refusal does not predict a backend: these runners are +/// ROCm-capable, so an earlier draft's "`rocm serve ` runs on Lemonade's +/// Vulkan backend" was being asserted green here on exactly the kind of host +/// where it is false. `rocm examine` holds either way, and is the command that +/// reports the real support level this PR added. +#[then("the refusal points at examining what this host supports")] +async fn assert_unsupported_family_points_at_examine(world: &mut E2eWorld) { + let output = unsupported_family_attempt(world); + assert!( + output.contains("rocm examine"), + "the refusal took every package family away, so it must leave the user a \ + next step:\n{output}" + ); + assert!( + !output.contains("rocm serve"), + "this refusal never looked at the host's GPU, so it must not predict \ + which backend serving would use here:\n{output}" + ); +} diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index c649d4de6..307ac836f 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -246,6 +246,7 @@ async fn serve_and_wait( if rc == 0 && model_is_ready(&MODELS_URL, Some(ready_substr), timeout_secs).await { world.endpoint = Some(SERVE_BASE_URL.to_string()); world.model_name = Some(model.to_string()); + world.serve_summary = Some(stdout.clone()); return stdout; } let invocation = format!("`rocm {}`", args.join(" ")); @@ -857,6 +858,7 @@ async fn serve_gpu_model_with_key(world: &mut E2eWorld, api_key: Option<&str>) { { world.endpoint = Some(SERVE_BASE_URL.to_string()); world.model_name = Some(model.to_string()); + world.serve_summary = Some(stdout); return; } // `serve_failure_evidence` reads and copies out the log BEFORE it stops @@ -1853,12 +1855,7 @@ fn docker_call_log(world: &E2eWorld) -> std::path::PathBuf { /// by `e2e-test-hooks` builds) so the family the CLI detects does not depend on the /// runner's hardware. `gfx_target_version` is the kernel's decimal encoding. fn plant_single_gpu(world: &mut E2eWorld, gfx_target_version: u32) { - let host_root = world - .isolated_root - .as_ref() - .expect("scenario has no isolated root") - .path() - .join("host-root"); + let host_root = planted_host_root(world); let node = host_root.join("sys/class/kfd/kfd/topology/nodes/1"); std::fs::create_dir_all(&node).expect("failed to create the planted KFD node"); // `cpu_cores_count 0` with a nonzero `simd_count` makes the node a GPU. @@ -1867,9 +1864,7 @@ fn plant_single_gpu(world: &mut E2eWorld, gfx_target_version: u32) { format!("cpu_cores_count 0\nsimd_count 56\ngfx_target_version {gfx_target_version}\n"), ) .expect("failed to write the planted KFD node properties"); - world - .command_env - .push(("ROCM_CLI_TEST_HOST_ROOT", host_root.into_os_string())); + use_planted_host_root(world); } /// gfx942, TheRock family `gfx94X-dcgpu`. @@ -1884,6 +1879,119 @@ async fn host_with_a_w7900(world: &mut E2eWorld) { plant_single_gpu(world, 110_000); } +/// A Renoir/Cezanne laptop, planted under the simulated host root: the Vega +/// iGPU is on the PCI bus and bound to `amdgpu`, and the kernel's KFD topology +/// lists **only a CPU node** for it. +/// +/// Not `plant_single_gpu`: that writes a KFD GPU node, which is precisely the +/// thing this host does not have. With no compute node the target can only be +/// named by the DRM `ip_discovery` read (GC 9.0.12 -> `gfx90c`), and the usable +/// GPU count can only come from `combine_amd_gpu_counts`' zero-KFD fallback to +/// the DRM card count. Both are driven off this one directory, which is what +/// makes the scenario a statement about one host rather than two. +fn plant_gfx90c_apu_without_a_kfd_node(world: &mut E2eWorld) { + let host_root = planted_host_root(world); + let device = host_root.join("sys/class/drm/card0/device"); + std::fs::create_dir_all(&device).expect("failed to create the planted DRM card"); + std::fs::write(device.join("vendor"), "0x1002\n").expect("failed to write the planted vendor"); + let gc = device.join("ip_discovery/die/0/GC/0"); + std::fs::create_dir_all(&gc).expect("failed to create the planted GC IP block"); + for (file, value) in [("major", "9"), ("minor", "0"), ("revision", "12")] { + std::fs::write(gc.join(file), value).expect("failed to write the planted GC IP version"); + } + let cpu_node = host_root.join("sys/class/kfd/kfd/topology/nodes/0"); + std::fs::create_dir_all(&cpu_node).expect("failed to create the planted KFD node"); + std::fs::write( + cpu_node.join("properties"), + "cpu_cores_count 16\nsimd_count 0\ngfx_target_version 0\n", + ) + .expect("failed to write the planted KFD node properties"); + use_planted_host_root(world); +} + +/// Where a scenario's simulated `/` lives, by the same convention +/// [`plant_single_gpu`] uses. +fn planted_host_root(world: &E2eWorld) -> std::path::PathBuf { + world + .isolated_root + .as_ref() + .expect("scenario has no isolated root") + .path() + .join("host-root") +} + +/// Point the *next* `rocm` invocation at the planted host. +/// +/// Needed once per invocation: `run_rocm_with_scenario_env` takes +/// `world.command_env` by `mem::take`, so a step that runs two commands would +/// silently run the second one against the real machine — which for this +/// scenario would mean asserting the serve pre-flight's answer for the runner's +/// own hardware instead of for the host the refusal was printed on. +fn use_planted_host_root(world: &mut E2eWorld) { + let host_root = planted_host_root(world); + world + .command_env + .push(("ROCM_CLI_TEST_HOST_ROOT", host_root.into_os_string())); +} + +#[given("a host whose only GPU is a gfx90c Vega iGPU the kernel reports no compute node for")] +async fn host_with_a_kfd_less_gfx90c(world: &mut E2eWorld) { + plant_gfx90c_apu_without_a_kfd_node(world); +} + +/// Take the route the CLI itself sends this host down: let the SDK install +/// refuse, then run the command that refusal names. +/// +/// Both halves run against the same planted host, in order, so the scenario +/// cannot pass by printing the advice on one machine and testing it on another. +/// `--gpu 99` is what makes the result readable: it is validated *after* the +/// no-usable-GPU pre-flight and reports the ordinals that pre-flight resolved, +/// turning "the host was accepted" into something the output states rather than +/// something inferred from the absence of a refusal. The model ref is arbitrary +/// and nothing is launched: both bails are reached before any runtime or engine +/// work. +#[when("the user is told to serve on Lemonade because this GPU has no ROCm build")] +async fn user_follows_the_lemonade_advice(world: &mut E2eWorld) { + let (install_stdout, install_stderr, install_rc) = + crate::run_rocm_with_scenario_env(world, &["install", "sdk", "--dry-run"]); + let refusal = format!("{install_stdout}\n{install_stderr}"); + assert_ne!( + install_rc, 0, + "a gfx90c host has no ROCm SDK build to install:\n{refusal}" + ); + assert!( + refusal.contains("gfx90c") && refusal.contains("rocm serve --engine lemonade"), + "the planted host must be detected as gfx90c and advised to serve on \ + Lemonade; without that this scenario is testing nothing:\n{refusal}" + ); + + use_planted_host_root(world); + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env( + world, + &["serve", "some-model", "--engine", "lemonade", "--gpu", "99"], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("serving got past the no-GPU pre-flight and reported this host's usable GPU")] +async fn assert_serve_accepted_the_drm_only_gpu(world: &mut E2eWorld) { + let output = serve_output(world); + assert!( + !output.contains("no usable AMD GPU"), + "`rocm install sdk` told this host to run `rocm serve --engine lemonade`, \ + and serve's pre-flight refused it for having no GPU - advice that cannot \ + clear the condition it was printed for:\n{output}" + ); + assert!( + output.contains("--gpu index 99 is not present on this host") + && output.contains("usable GPU indices: [0]"), + "the pre-flight must have resolved exactly the iGPU's ordinal, and said \ + so where `--gpu` is validated:\n{output}" + ); +} + fn run_serve_aim(world: &mut E2eWorld, args: &[&str]) { let mut full = vec!["serve"]; full.extend_from_slice(args); @@ -2714,12 +2822,42 @@ async fn listed_anthropic_endpoint(world: &mut E2eWorld) -> Option { } } +/// What a missing `anthropic endpoint:` row has to be explained by: the +/// listing, and the engine state file the row is read out of. +/// +/// The row is absent for two very different reasons — the serving process never +/// published, or something wrote the file whole and replaced the key — and the +/// listing alone cannot tell them apart. That cost a round trip to the Windows +/// GPU lane per guess while ROCMAI-550's regression was being chased, so the +/// file goes in the failure message. `--json` is the records verbatim, so it +/// carries `engine_state_path`; it also asks no device probe, so reading it +/// costs nothing a failing step cannot afford. +fn missing_anthropic_endpoint_evidence(world: &mut E2eWorld) -> String { + let listing = world.cli_output.clone().unwrap_or_default(); + let (records, _, _) = crate::run_rocm(world, &["services", "list", "--json"]); + let states = serde_json::from_str::(&records) + .ok() + .and_then(|records| records.as_array().cloned()) + .unwrap_or_default() + .iter() + .filter_map(|record| record.get("engine_state_path")?.as_str().map(str::to_owned)) + .map(|path| { + let body = std::fs::read_to_string(&path) + .unwrap_or_else(|error| format!("")); + format!("{path}:\n{body}") + }) + .collect::>() + .join("\n"); + format!("{listing}\nservices list --json:\n{records}\nengine state:\n{states}") +} + #[then("the service details include an Anthropic-compatible messages endpoint")] async fn assert_service_lists_anthropic_endpoint(world: &mut E2eWorld) { let listed = listed_anthropic_endpoint(world).await; - let output = world.cli_output.clone().unwrap_or_default(); - let endpoint = listed - .unwrap_or_else(|| panic!("no `anthropic endpoint:` row in `services list`:\n{output}")); + let Some(endpoint) = listed else { + let evidence = missing_anthropic_endpoint_evidence(world); + panic!("no `anthropic endpoint:` row in `services list`:\n{evidence}"); + }; assert!( endpoint.starts_with("http://") && !endpoint.contains("/v1"), "the Anthropic endpoint is a base URL clients append `/v1/messages` to, got `{endpoint}`" @@ -2740,9 +2878,11 @@ async fn assert_service_still_lists_openai_endpoint(world: &mut E2eWorld) { /// POST `body` to the served model's Anthropic endpoint and return the JSON /// reply, whatever its status. async fn post_anthropic_messages(world: &mut E2eWorld, body: serde_json::Value) { - let base = listed_anthropic_endpoint(world) - .await - .expect("no Anthropic endpoint is listed for the served model"); + let listed = listed_anthropic_endpoint(world).await; + let Some(base) = listed else { + let evidence = missing_anthropic_endpoint_evidence(world); + panic!("no Anthropic endpoint is listed for the served model:\n{evidence}"); + }; let budget = Duration::from_secs(crate::inference_timeout_for(world)); let reply = tokio::spawn(async move { reqwest::Client::builder() @@ -2889,9 +3029,11 @@ async fn assert_anthropic_response_model_correct(world: &mut E2eWorld) { #[when("the user sends a streaming request to the Anthropic messages endpoint")] async fn user_sends_streaming_anthropic_messages_request(world: &mut E2eWorld) { - let base = listed_anthropic_endpoint(world) - .await - .expect("no Anthropic endpoint is listed for the served model"); + let listed = listed_anthropic_endpoint(world).await; + let Some(base) = listed else { + let evidence = missing_anthropic_endpoint_evidence(world); + panic!("no Anthropic endpoint is listed for the served model:\n{evidence}"); + }; let budget = Duration::from_secs(crate::inference_timeout_for(world)); // Same budget and model name as the non-streaming request above. let body = serde_json::json!({ @@ -3027,11 +3169,11 @@ async fn assert_both_endpoints_healthy(world: &mut E2eWorld) { // `services list` probes each endpoint it shows and marks one that does not // answer with `(not responding)`; a healthy service carries neither mark. let listed = listed_anthropic_endpoint(world).await; + if listed.is_none() { + let evidence = missing_anthropic_endpoint_evidence(world); + panic!("no `anthropic endpoint:` row in `services list`:\n{evidence}"); + } let output = world.cli_output.clone().unwrap_or_default(); - assert!( - listed.is_some(), - "no `anthropic endpoint:` row in `services list`:\n{output}" - ); assert!( output.contains("status: ready"), "the OpenAI-compatible endpoint is not reported ready:\n{output}" @@ -3674,6 +3816,214 @@ async fn assert_service_in_list(world: &mut E2eWorld) { ); } +/// The `device:` value as `rocm serve` / `rocm services list` print it, split +/// into the backend it names and the GPU ordinal it pinned, if any. +/// +/// Every engine renders it through +/// `rocm_engine_protocol::format_reported_device`, so the grammar is +/// ``, then an optional ` gpu `, then an optional ` ()`. +/// `None` for anything else, which is what makes a hardcoded or malformed value +/// fail rather than pass. +fn parse_device_row(value: &str) -> Option<(&str, Option)> { + // The name is the only part that may contain spaces, and it is always last + // and parenthesised. Split on the FIRST " (" so a name with its own + // parentheses ("AMD Radeon (TM) Graphics") still leaves the head intact. + let head = match value.split_once(" (") { + Some((head, name)) if name.ends_with(')') => head, + Some(_) => return None, + None => value, + }; + let mut parts = head.split_whitespace(); + let backend = parts.next()?; + let ordinal = match parts.next() { + Some("gpu") => Some(parts.next()?.parse().ok()?), + // A bare `gpu` with no ordinal is the shape ROCMAI-550 removed; so is + // any other trailing word. + Some(_) => return None, + None => None, + }; + parts.next().is_none().then_some((backend, ordinal)) +} + +/// How long the `device:` row may take to show up after the serve returned. +/// +/// It is not available the instant a service is recorded `ready`: +/// `engines/lemonade` reports `"unknown"` until its own `inference_verified` +/// probe has come back, and `fetch_service_device` drops `"unknown"` rather +/// than printing it. So there is a real window in which a correct +/// implementation has no row yet, and failing on the first look would make +/// this step flake on timing rather than on behavior. +const DEVICE_ROW_DEADLINE: Duration = Duration::from_secs(90); + +/// The value of the first row labelled `device` in `text`, trimmed. +/// +/// Two renderers, two shapes, and this has to read both: `services_text.rs` +/// writes `device: `, while `serve_summary.rs`'s `render_summary` +/// writes a column of padded labels and no colon at all +/// (` device ` — pinned by its own +/// `device_row_shown_only_when_known` test, which asserts +/// `starts_with("device")`). An earlier revision looked only for `"device:"`, +/// so it could never have matched a deployment summary even when one carried +/// the row. +/// +/// The label has to end where it ends: `rocm serve`'s non-interactive plan +/// prints a `device_policy:` row, and matching that here would compare a +/// policy against a device. +fn device_row_value(text: &str) -> Option<&str> { + text.lines().find_map(|line| { + let rest = line.trim_start().strip_prefix("device")?; + let rest = rest.strip_prefix(':').unwrap_or(rest); + if !rest.starts_with(char::is_whitespace) { + return None; + } + Some(rest.trim()).filter(|value| !value.is_empty()) + }) +} + +/// ROCMAI-550: `rocm services list` must show the backend/device the engine +/// actually selected (read live from its `Healthcheck` RPC), not just that a +/// service exists. Reuses `assert_service_in_list`'s model-name match so this +/// only checks the row for the service this scenario just started. +/// +/// It asserts the VALUE, not merely the presence of the row. A row whose value +/// were empty, malformed, or nailed to a constant would satisfy "starts with +/// `device:`" while reporting nothing real, so this parses it and holds it to +/// three independent facts: +/// +/// 1. it has the ` [gpu ] [()]` shape every engine reports; +/// 2. the backend is one this engine can actually have selected - only +/// Lemonade has a Vulkan path, so `vulkan` from `vllm` or `aim` is wrong, +/// and `unknown` is wrong from any of them; +/// 3. it equals what the `rocm serve` deployment summary printed for the same +/// service — *when* that summary printed a device row at all, which is a +/// documented maybe rather than a promise (see the comment at that +/// comparison, and `serve_summary::DeploymentSummary::device`). Facts 1 and +/// 2 are what discriminate on today's lanes; fact 3 does not run, because +/// the summary these scenarios capture is pipe-redirected and therefore the +/// row-less plain form. The comment at that comparison says so in full +/// rather than letting the name of this step imply coverage it has not got. +/// The row is polled rather than read once: see [`DEVICE_ROW_DEADLINE`] for +/// the window in which a correct implementation legitimately has no row yet. +/// Running out of that window is an assertion naming what was last seen, not a +/// panic on the first look. +#[then("the services list reports the engine's actual backend device")] +async fn assert_services_list_reports_device(world: &mut E2eWorld) { + let model = world.model_name.clone().unwrap_or_default(); + let deadline = Instant::now() + DEVICE_ROW_DEADLINE; + // The last thing looked at, so a timeout can show it rather than just + // saying it timed out: the service's own block when there was one, else the + // whole listing (which is the evidence for the service having gone missing). + let mut last_seen; + let device = loop { + let (stdout, _, _) = crate::run_rocm(world, &["services", "list"]); + match stdout + .split("\n- ") + .find(|block| block.to_lowercase().contains(&model.to_lowercase())) + { + Some(block) => { + last_seen = block.to_owned(); + if let Some(device) = device_row_value(&last_seen) { + break device.to_owned(); + } + } + None => last_seen = stdout, + } + assert!( + Instant::now() < deadline, + "after {}s `rocm services list` still reports no `device:` line for \ + {model} - the engine's Healthcheck RPC should report the backend it \ + actually selected once the service is serving:\n{last_seen}", + DEVICE_ROW_DEADLINE.as_secs() + ); + tokio::time::sleep(Duration::from_secs(3)).await; + }; + let block = last_seen.as_str(); + let (backend, _ordinal) = parse_device_row(&device).unwrap_or_else(|| { + panic!( + "`device: {device}` is not the ` [gpu ] [()]` shape \ + every engine reports, so it cannot be the engine's own answer:\n{block}" + ) + }); + + let engine = block + .lines() + .find_map(|line| line.trim_start().strip_prefix("engine:")) + .map_or_else( + || panic!("services-list entry for {model} has no engine:\n{block}"), + str::trim, + ); + // Only Lemonade has a Vulkan llama.cpp backend; vLLM and AIM are ROCm-only, + // so `vulkan` from either would mean the row is not coming from the engine + // that served. `unknown` is the engines' own "I have no state for this", + // which `fetch_service_device` is supposed to drop rather than print. + let allowed: &[&str] = match engine { + "lemonade" => &["rocm", "vulkan"], + _ => &["rocm"], + }; + assert!( + allowed.contains(&backend), + "`device: {device}` names backend `{backend}`, which the `{engine}` \ + engine cannot have selected (expected one of {allowed:?}):\n{block}" + ); + + // The same fact, read from the other surface. `rocm serve` printed this row + // at launch; nothing else in these scenarios inspects that summary, so + // without this the summary half of ROCMAI-550 has no live assertion at all. + // + // Only when the summary HAS the row. Its absence is not a defect and must + // not be asserted against, in either direction: + // + // * `serve` asks the engine once the launch reports `ready` and retries + // only within `engine_ipc::SERVE_DEVICE_PROBE_BUDGET` (~6s), because the + // row is informational and the alternative is holding a terminal; + // * `engines/lemonade` answers `"unknown"` until its own + // `inference_verified` probe returns, which for a model still loading can + // be far longer than that - which is exactly why the Given step then + // polls `/v1/models` for up to minutes; + // * so a correct `rocm serve` can print `status ready` with no `device:` + // row, while `rocm services list` - asked later, after this step has + // waited - has one. + // + // An earlier revision asserted that a `ready` summary must carry the row. + // That is the exact false failure this note describes, and it was invisible + // only because serve-07 is xfail'd on lemonade+linux (see the comment on + // that scenario in model_serving.feature). What remains asserted is the + // half that is a real invariant: the two surfaces read one engine over one + // RPC, so where both speak they must say the same thing. + // + // **State plainly what this cannot do today.** On every current lane the + // comparison below does not run at all, and not for the reason above: + // `world.serve_summary` is `rocm serve`'s stdout captured by + // `run_rocm_bounded`, which redirects it to a file, so + // `serve_cmd.rs`'s `summary_mode` (`background && stdout().is_terminal()`) + // is false and the CLI takes `print_managed_launch_plain`, which renders no + // device row in any state. The device row is an interactive-summary + // feature. So this is a cross-check that is *armed* — `device_row_value` + // reads both renderers' shapes, and it fires for any Given that captures a + // summary through a PTY — but today nothing drives these two scenarios + // that way, and the backend assertion above is what actually discriminates + // for serve-05 and serve-07. Making it live means driving the vLLM and + // Lemonade per-PR serve canaries through a `TuiSession` (as serve-28/41/60 + // do for their mock launches), which is a change to the canaries' own + // harness and is tracked separately rather than smuggled in here. + // And only when a summary was recorded at all. The backend assertion above + // is this step's real invariant and has already run; the cross-surface + // comparison is an extra that needs a Given which captured `rocm serve`'s + // own output. serve-05 and serve-07 both have one today, but panicking here + // would fail any future scenario reusing this Then for the backend check + // alone — a failure about the fixture rather than about the product. + let Some(summary) = world.serve_summary.as_deref() else { + return; + }; + if let Some(summary_device) = device_row_value(summary) { + assert_eq!( + summary_device, device, + "`rocm serve` and `rocm services list` must report the same device \ + for one service; they read it from the same Healthcheck RPC" + ); + } +} + #[then("the list reports the attempt and how to look at it")] async fn assert_past_attempts_reported(world: &mut E2eWorld) { let stdout = world