Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 47 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,33 @@ serving-instance registry needs the Unix domain socket telemetry, so that view
is Linux/WSL2 only. Use `rocm dash --demo` for a synthetic session that runs
anywhere without a GPU or daemon.

On Linux and WSL, the embedded daemon also discovers loaded models from a
separately running local Lemonade server at `127.0.0.1:13305`. Configure
`dashboard.daemon.lemonade_host` and `dashboard.daemon.lemonade_port` in
`config.json` for another endpoint, or set `dashboard.daemon.enable_lemonade`
to `false` to disable external discovery. The file is `~/.rocm/config.json`, or
`config.json` in the directory named by `ROCM_CLI_CONFIG_DIR`. Editing it is the
only way to set these fields. Native Windows has no live dashboard telemetry, so it
has no external discovery. A healthy server with no loaded models
does not count as a running model. Discovery does not register an external server
as a managed service or give ROCm CLI ownership of its process. External models
are visible in Observe, but are excluded from the Services stop/restart controls.
The tok/s figure for a Lemonade model is the rate of the endpoint's most recent
completed request, not live throughput: it stays the same while the server is
idle. The dashboard explains it with a legend (`Lemonade: last req tok/s, not live`)
under the Instances table and the Services overlay rows, and a `last req` prefix in
the instance detail and the quick preview. Because it is not live, the node-wide
totals (the tok/W and tok/s heroes on Home and Observe) and the per-instance tok/W
leave it out, and the agent snapshot marks it with `gen_tps_last_request`. A managed Lemonade server that requires an API key stays listed with its
registry status and throughput unavailable; the dashboard never reads the key.

The engine pane distinguishes a ROCm CLI-managed Lemonade installation from an
externally running server. When installation detection is unavailable, it shows
an unchecked status ("unknown, run `rocm engines list`") rather than claiming
Lemonade is not installed. Installation status does not depend on GPU telemetry.
It is read once, when the dashboard starts. After you install Lemonade from the
Engines overlay, restart the dashboard to refresh it.

## Configure ROCm and serve a model

Before serving a model, check whether a managed ROCm runtime is already
Expand Down Expand Up @@ -1236,7 +1263,26 @@ server instead (an already-serving one is reported at once). Watch the download
Which model form to pass depends on the engine your GPU selects. The Lemonade
engine (Ryzen AI or Radeon) serves llama.cpp **GGUF** models. Pass a GGUF repo
with an explicit quantization variant, for example,
`rocm serve unsloth/Qwen3-0.6B-GGUF:Q4_0`.
`rocm serve unsloth/Qwen3-0.6B-GGUF:Q4_0`. Lemonade also accepts
registered GGUF names such as `Qwen3-0.6B-GGUF`, or the `qwen-smoke` alias.
On Linux, direct serving resolves simple single-GGUF entries from the installed
Lemonade catalog. Entries needing additional files or model-specific options,
well-formed entries of an unrecognized shape, and registered names on Windows use
Lemonade's router instead. That includes a registered checkpoint that is not a
plain `owner/repo:variant`, such as one whose variant is `*`. Explicit `owner/repo:variant` references require the selected GPU
llama-server backend. Direct serving uses that backend, including Vulkan when
supported; it does not select a stale ROCm binary merely because it is on disk.
When Lemonade selects a GPU backend other than ROCm, such as Vulkan, serving names
that backend instead of presenting it as ROCm. Neither route silently substitutes
the default model.

On Linux, if Lemonade's router fails to load or verify a registered model, `rocm
serve` stops the router and serves that model directly through the llama-server of
the backend Lemonade selected, when the catalog entry resolves to a simple
checkpoint. The serve log names the failure and the backend. This never uses a CPU
backend: with no GPU llama-server for the selected backend, or on Windows, the
router's error is returned and the service is marked failed. A server that did not
become ready, a failed backend selection and a later server exit are never retried.

The vLLM engine (Instinct) serves
**safetensors** repos, such as `rocm serve Qwen/Qwen2.5-0.5B-Instruct`.
Expand Down
29 changes: 29 additions & 0 deletions apps/rocm/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -692,6 +692,11 @@ rocm logs --search error timeout")]
status: bool,
},
/// Launch the unified telemetry dashboard (TUI) with an embedded daemon.
///
/// On Linux/WSL, discovers loaded models from local Lemonade at 127.0.0.1:13305
/// by default. Configure dashboard.daemon.enable_lemonade, lemonade_host, and
/// lemonade_port in config.json. External models have no Services stop/restart
/// controls; discovery is read-only and quitting leaves external servers running.
Dash {
/// Replay a recorded session NDJSON instead of connecting to a live daemon.
#[arg(long, value_name = "FILE")]
Expand Down Expand Up @@ -1493,6 +1498,30 @@ mod tests {
);
}

#[test]
fn dash_help_scopes_lemonade_discovery_to_linux_and_wsl() {
// The embedded daemon that does the discovery is Unix-only, so the help
// must not promise it on every platform. README.md says the same.
let help = Cli::command()
.find_subcommand_mut("dash")
.expect("dash subcommand")
.render_long_help()
.to_string();
let flat = help.split_whitespace().collect::<Vec<_>>().join(" ");
assert!(
flat.contains("On Linux/WSL, discovers loaded models from local Lemonade"),
"{help}"
);
// This pins the wording only. Scenario
// `@id:dash-lemonade-external-protocol-discovery` proves the claim: the
// external server receives no request beyond reads after the dashboard
// quits.
assert!(
flat.contains("quitting leaves external servers running"),
"{help}"
);
}

#[test]
fn the_framework_choice_is_offered_on_the_command_line() {
// Guards the actual regression: the variants existed in the library and
Expand Down
94 changes: 90 additions & 4 deletions apps/rocm/src/dash.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,11 @@ pub fn runner_options(
instance_tick: Duration::from_secs_f64(d.instance_tick_secs),
disable_vllm_metrics: false,
vllm_metrics_host: "127.0.0.1".into(),
// Lemonade discovery stays opt-in (mirrors a no-flag embedded daemon).
enable_lemonade: false,
lemonade_host: "127.0.0.1".into(),
lemonade_port: 13305,
// Read-only discovery of local external Lemonade; managed registry
// discovery remains independent of this switch.
enable_lemonade: d.enable_lemonade,
lemonade_host: d.lemonade_host.clone(),
lemonade_port: d.lemonade_port,
persist_dir: Some(paths.telemetry_state_dir()),
// D7 seam consumer: managed services from `rocm serve --managed`.
services_dir: Some(paths.services_dir()),
Expand Down Expand Up @@ -258,6 +259,29 @@ fn automation_summaries(config: &RocmCliConfig) -> Vec<AutomationSummary> {
.collect()
}

/// Read-only managed-engine inventory, independent of GPU telemetry and
/// endpoint liveness. An unavailable detect response means unknown, not absent.
fn lemonade_managed_installed(paths: &AppPaths) -> Option<bool> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Detect request this helper triggers runs synchronously during dash startup. Please confirm the stdio client has a deadline, or move it off the startup path.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked. This Detect request does not spawn a child: engine_request hands a lemonade Detect to the built-in in-process handler (builtin_engine_request), which reads the install manifest and runs the local GPU visibility probe that rocm serve and rocm examine already run. There is no stdio client and no network, so there is no stdio deadline to apply, and I did not add a separate one around it. The PR body says so under the deployment notes.

let detect = crate::engine_ipc::engine_request::<_, rocm_engine_protocol::DetectResponse>(
Some(paths),
"lemonade",
rocm_engine_protocol::EngineMethod::Detect,
&rocm_engine_protocol::DetectRequest::default(),
)
.ok();
managed_installed_from_detect(detect.as_ref())
}

/// Pure verdict over a Detect result: `None` when Detect failed (unknown, not
/// absent); otherwise installed only counts when the runtime is managed. A
/// legacy response without `managed_env` is not proof of a managed install.
fn managed_installed_from_detect(
detect: Option<&rocm_engine_protocol::DetectResponse>,
) -> Option<bool> {
let detect = detect?;
Some(detect.installed && detect.managed_env == Some(true))
}

/// Resolve the TUI args from the unified config + environment.
///
/// MUST be called on a synchronous thread *before* any tokio runtime is entered:
Expand Down Expand Up @@ -328,6 +352,7 @@ pub fn resolved_args(
// `resolved_args` itself has no opinion, so this always starts `false`
// here.
launcher_hosted: false,
lemonade_managed_installed: lemonade_managed_installed(paths),
}
}

Expand Down Expand Up @@ -715,6 +740,7 @@ fn launcher_serving_instances(paths: &AppPaths) -> Vec<rocm_dash_core::metrics::
model_name: svc.model_name,
status: svc.status,
port: svc.port,
managed: svc.managed,
..Default::default()
})
.collect()
Expand Down Expand Up @@ -1080,6 +1106,63 @@ async fn maybe_spawn_embedded_daemon(
mod tests {
use super::*;

fn detect(installed: bool, managed_env: Option<bool>) -> rocm_engine_protocol::DetectResponse {
let mut v = serde_json::json!({
"installed": installed, "env_id": null, "python_version": null,
"torch_version": null, "transformers_version": null,
"available_devices": [], "notes": [],
"capabilities": {"cpu": false, "rocm_gpu": false, "openai_compatible": true,
"tool_calling": false, "quantized_models": "", "reasoning_parser": false}
});
if let Some(m) = managed_env {
v["managed_env"] = m.into();
}
serde_json::from_value(v).unwrap()
}

#[test]
fn managed_installed_verdict_follows_detect_result() {
assert_eq!(
managed_installed_from_detect(Some(&detect(true, Some(true)))),
Some(true)
);
assert_eq!(
managed_installed_from_detect(Some(&detect(true, Some(false)))),
Some(false)
);
// Legacy response without the ownership field proves nothing managed.
assert_eq!(
managed_installed_from_detect(Some(&detect(true, None))),
Some(false)
);
assert_eq!(
managed_installed_from_detect(Some(&detect(false, Some(true)))),
Some(false)
);
// A failed Detect is unknown, never "not installed".
assert_eq!(managed_installed_from_detect(None), None);
}

#[test]
fn lemonade_discovery_defaults_and_overrides_reach_daemon() {
let default = runner_options(&RocmCliConfig::default(), &paths(), false);
assert!(default.enable_lemonade);
assert_eq!(default.lemonade_host, "127.0.0.1");
assert_eq!(default.lemonade_port, 13305);
let config: RocmCliConfig = serde_json::from_str(
r#"{
"dashboard": {"daemon": {"enable_lemonade": false,
"lemonade_host": "localhost", "lemonade_port": 14567}}
}"#,
)
.unwrap();
let overridden = runner_options(&config, &paths(), false);
assert!(!overridden.enable_lemonade);
assert_eq!(overridden.lemonade_host, "localhost");
assert_eq!(overridden.lemonade_port, 14567);
assert_eq!(overridden.services_dir, Some(paths().services_dir()));
}

fn cfg() -> RocmCliConfig {
RocmCliConfig::default()
}
Expand Down Expand Up @@ -1915,6 +1998,9 @@ mod tests {
assert_eq!(inst.model_name, "meta-llama/Llama-3.1-8B");
assert_eq!(inst.port, Some(11435));
assert_eq!(inst.status, rocm_dash_core::metrics::InstanceStatus::Ready);
// Registry records are owned by this CLI; the flag must survive the map
// so lifecycle surfaces treat the launcher row as managed.
assert!(inst.managed, "registry-owned record must stay managed");

let _ = std::fs::remove_dir_all(&root);
}
Expand Down
69 changes: 59 additions & 10 deletions apps/rocm/src/engine_runtime_select.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
//! `service_http_readiness_paths`, `http_get_local_service`,
//! `http_post_local_service_json`, `service_http_readiness_response_ready`,
//! `lemonade_health_ready_for_model`, `model_list_ready_for_model`,
//! `service_model_gpu_ready`, `service_model_reports_rocm_backend`,
//! `service_model_gpu_ready`, `service_model_reports_gpu_backend`,
//! `service_model_names_match`).
//!
//! Mechanically relocated from `main.rs` with no behavior change (the
Expand All @@ -27,7 +27,9 @@
//! immediately after it in the original file, `treat_as_natural_language`, is
//! dispatch-adjacent natural-language-invocation routing that stays in
//! `main.rs` on purpose — it is not part of this module and is not touched by
//! this change.
//! this change. Behavior here has changed since the move: a Lemonade router
//! entry served on Vulkan now counts as ready (`service_model_gpu_ready`), not
//! only one served on ROCm.
//!
//! One item physically interleaved in this span in `main.rs` stayed there
//! instead of moving: `const fn status_for_readiness` sits between
Expand Down Expand Up @@ -76,7 +78,7 @@
//! `service_http_readiness_paths`, `http_get_local_service`,
//! `service_http_readiness_response_ready`, `lemonade_health_ready_for_model`,
//! `model_list_ready_for_model`, `service_model_gpu_ready`, and
//! `service_model_reports_rocm_backend` have no callers outside this module
//! `service_model_reports_gpu_backend` have no callers outside this module
//! and stay private.
//!
//! The cluster's regression tests live in this module's own `mod tests`,
Expand Down Expand Up @@ -495,15 +497,15 @@ fn lemonade_health_ready_for_model(body: &str, canonical_model_id: &str) -> bool
.into_iter()
.filter_map(|field| model.get(field).and_then(serde_json::Value::as_str))
.any(|loaded| service_model_names_match(loaded, canonical_model_id));
name_matches && service_model_reports_rocm_backend(model)
name_matches && service_model_reports_gpu_backend(model)
})
})
}

fn model_list_ready_for_model(
body: &str,
canonical_model_id: &str,
require_rocm_backend: bool,
require_gpu_backend: bool,
) -> bool {
let Ok(value) = serde_json::from_str::<serde_json::Value>(body.trim()) else {
return false;
Expand All @@ -517,30 +519,34 @@ fn model_list_ready_for_model(
.into_iter()
.filter_map(|field| model.get(field).and_then(serde_json::Value::as_str))
.any(|loaded| service_model_names_match(loaded, canonical_model_id));
name_matches && (!require_rocm_backend || service_model_gpu_ready(model))
name_matches && (!require_gpu_backend || service_model_gpu_ready(model))
})
})
}

/// Whether a lemonade `/v1/models` entry is served on GPU. A stock `llama-server`
/// (direct-serve) entry has no `recipe_options` and is accepted (that path only runs
/// GPU backends); a Lemonade-router entry carries `recipe_options`, so it must report a
/// ROCm backend — which keeps a registered-but-unloaded model (empty `recipe_options`)
/// GPU backend (ROCm or Vulkan) — which keeps a registered-but-unloaded model (empty `recipe_options`)
/// from reading as ready.
fn service_model_gpu_ready(model: &serde_json::Value) -> bool {
match model.get("recipe_options") {
None => true,
Some(_) => service_model_reports_rocm_backend(model),
Some(_) => service_model_reports_gpu_backend(model),
}
}

fn service_model_reports_rocm_backend(model: &serde_json::Value) -> bool {
fn service_model_reports_gpu_backend(model: &serde_json::Value) -> bool {
model
.get("recipe_options")
.and_then(|options| options.get("llamacpp_backend"))
.or_else(|| model.get("llamacpp_backend"))
.and_then(serde_json::Value::as_str)
.is_some_and(|backend| backend.trim().to_ascii_lowercase().starts_with("rocm"))
.is_some_and(|backend| {
// The GPU backends the engine can select (ROCm, else Vulkan); never `cpu`.
let backend = backend.trim().to_ascii_lowercase();
backend.starts_with("rocm") || backend.starts_with("vulkan")
})
}

/// Whether two model names refer to the same model, leniently.
Expand Down Expand Up @@ -725,6 +731,49 @@ mod tests {
));
}

#[test]
fn lemonade_router_readiness_accepts_any_gpu_backend_the_engine_selects() {
// The engine selects ROCm first and Vulkan otherwise, and records either as
// `backend_requested`; a router-served model on a Vulkan host must read as
// ready on both probe paths. CPU (or no backend) must never read as ready.
for (backend, ready) in [
("rocm", true),
("rocm-stable", true),
("vulkan", true),
("Vulkan", true),
("cpu", false),
("", false),
] {
let health = json!({
"all_models_loaded": [{
"model_name": "Qwen3-0.6B-GGUF",
"recipe_options": { "llamacpp_backend": backend }
}]
})
.to_string();
let models = json!({
"data": [{
"id": "Qwen3-0.6B-GGUF",
"recipe_options": { "llamacpp_backend": backend }
}]
})
.to_string();
for (path, body) in [("/v1/health", &health), ("/v1/models", &models)] {
assert_eq!(
service_http_readiness_response_ready(
"lemonade",
path,
200,
body,
"Qwen3-0.6B-GGUF"
),
ready,
"{path} with backend `{backend}`"
);
}
}
}

#[test]
fn serve_readiness_wait_withholds_ready_while_the_model_only_lists() -> Result<()> {
use std::io::{Read, Write};
Expand Down
Loading
Loading