From 91964318eeb7e2c9c02532077b492b84d48a1bc8 Mon Sep 17 00:00:00 2001 From: Roman Sirokov Date: Fri, 6 Nov 2026 13:54:52 +0200 Subject: [PATCH 1/5] fix(dash-tui): list running managed services on Windows On Windows there is no telemetry daemon, so nothing ever filled the dashboard's instance list: Home said "Running 0" and Serving "Running now - 0" while rocm services list showed ready services. The live managed-registry probe now also returns the live services as instances (same merge_instance mapping the daemon uses). A background poller reads it once a second under spawn_blocking and posts ClientMsg::ManagedInstances, which AppState::set_managed_instances applies in place so the Home tile's model does not change order each poll. The first poll is immediate. The poller runs only on Windows, never for --demo/--replay or a focused host. has_received_snapshot is untouched, so quit-confirm behaviour is unchanged. Linux is not affected. Also corrects the Observe empty state, the Windows quit-prompt manual-test steps, and README/install docs that said the Windows dashboard shows only GPU/CPU/memory. Known divergence: the probe reads the record's on-disk status, so a crashed service can read as running until a rocm services command reconciles it. Signed-off-by: Roman Sirokov --- README.md | 23 +-- apps/rocm/src/dash.rs | 93 +++++++++-- crates/rocm-dash-tui/src/app/event_loop.rs | 154 ++++++++++++++++++ crates/rocm-dash-tui/src/app/mod.rs | 105 +++++++++++- crates/rocm-dash-tui/src/app/quit_confirm.rs | 4 +- crates/rocm-dash-tui/src/app/test_support.rs | 1 + crates/rocm-dash-tui/src/app/types.rs | 3 +- crates/rocm-dash-tui/src/client.rs | 16 +- crates/rocm-dash-tui/src/registry_probe.rs | 33 ++-- crates/rocm-dash-tui/src/ui/launcher.rs | 6 + crates/rocm-dash-tui/src/ui/quit_confirm.rs | 18 +- crates/rocm-dash-tui/src/ui/tabs/instances.rs | 12 +- docs/manual-testing.md | 30 ++-- docs/rocm-docs/install/installation.md | 5 +- 14 files changed, 426 insertions(+), 77 deletions(-) diff --git a/README.md b/README.md index 313a7f4ee1..ee698e1975 100644 --- a/README.md +++ b/README.md @@ -31,16 +31,17 @@ for Lemonade and vLLM. | Platform | Prebuilt binary | Notes | |---|---|---| | Linux (x86_64) | Yes | Ubuntu 24.04 or newer; full support, including the live dashboard and both inference engines | -| Windows (x86_64) | Yes | CLI and Lemonade serving; live dashboard shows GPU/CPU/memory but not the serving-instance registry; no vLLM | +| Windows (x86_64) | Yes | CLI and Lemonade serving; live dashboard shows GPU/CPU/memory and running managed servers, without per-server throughput; no vLLM | | WSL2 (x86_64) | Yes (Linux binary) | Ubuntu 24.04 or newer; full support, including the live dashboard | | macOS | No | No official installer, release, CI, or QA coverage | For WSL2 setup and diagnosis, see [docs/wsl.md](docs/wsl.md). -Live dashboard telemetry works on Windows too, but shows only GPU/CPU/memory — -the serving-instance registry view needs the `rocmdashd` unix socket, so it's -Linux/WSL2 only (see [Interactive interfaces](#interactive-interfaces)). vLLM +Live dashboard telemetry works on Windows too: GPU/CPU/memory and the managed +servers `rocm services list` reports. Per-server throughput and Docker-discovered +instances need the `rocmdashd` unix socket, so they're Linux/WSL2 only (see +[Interactive interfaces](#interactive-interfaces)). vLLM serving is Linux or WSL2 only (see [docs/vllm.md](docs/vllm.md)). The minimum supported Linux release, native or under WSL2, is Ubuntu 24.04. On @@ -227,9 +228,10 @@ five tabs (switch with `Tab`/`Shift+Tab` or number keys `1`–`5`): | **Observe** | Live GPU utilization, instances, and benchmark telemetry | | **Chat** | Assistant chat backed by a local server or configured provider | -Live mode works on Windows too, but shows only GPU/CPU/memory — the -serving-instance registry needs the Unix domain socket telemetry, so that view -is Linux/WSL2 only. Use `rocm dash --demo` for a synthetic session that runs +Live mode works on Windows too, showing GPU/CPU/memory and the managed servers +`rocm services list` reports. Per-server throughput and Docker-discovered +instances need the Unix domain socket telemetry, so they are Linux/WSL2 only. +Use `rocm dash --demo` for a synthetic session that runs anywhere without a GPU or daemon. ## Configure ROCm and serve a model @@ -1396,9 +1398,10 @@ For `rocm dash`: - `--demo` runs a deterministic synthetic session with no GPU or daemon needed. It works on all platforms. - `--replay ` replays a recorded NDJSON session. -- Live mode works on Windows too, but shows only GPU/CPU/memory — there's no - `rocmdashd` unix socket to connect to there, so the serving-instance - registry view stays empty (Linux and WSL2 get the full live view). +- Live mode works on Windows too, showing GPU/CPU/memory and the managed + servers `rocm services list` reports. There's no `rocmdashd` unix socket + there, so per-server throughput and Docker-discovered instances are missing + (Linux and WSL2 get the full live view). ### Bench diff --git a/apps/rocm/src/dash.rs b/apps/rocm/src/dash.rs index 28346a4c05..d818791d69 100644 --- a/apps/rocm/src/dash.rs +++ b/apps/rocm/src/dash.rs @@ -345,8 +345,9 @@ pub fn resolved_args( /// `rocm_dash_tui::client`'s `#[cfg(windows)]` arm does connect and stream /// live GPU/CPU/memory every tick but — by design, there being no /// `rocmdashd` unix socket to source one from (`maybe_spawn_embedded_daemon` -/// is unix-only) — never sends an instance snapshot, so -/// `AppState::instances` never gets a first one there. +/// is unix-only) — never sends an instance snapshot, so this probe is also +/// where `AppState::instances` comes from there: the dashboard polls it +/// (`ManagedRegistrySnapshot::instances`) once a second off the UI thread. /// /// **Known divergence, same shape as [`services_past_attempts`]'s:** this /// reads the record's raw on-disk `status`, not process liveness. A service @@ -366,11 +367,31 @@ impl ManagedRegistryProbe for LiveManagedRegistryProbe { let discovery = discover_managed_services(&records); ManagedRegistrySnapshot { has_live_service: discovery.svcs.iter().any(|svc| svc.status.is_serving()), + instances: managed_instances(&discovery), managed_ids: discovery.seen, } } } +/// The live managed services as dashboard instances with no telemetry yet — +/// the same mapping the daemon applies to a newly discovered service. +fn managed_instances( + discovery: &rocm_dash_daemon::registry::ManagedDiscovery, +) -> Vec { + discovery + .svcs + .iter() + .map(|svc| { + rocm_dash_core::traits::merge_instance( + svc, + &rocm_dash_core::traits::InstanceSample::default(), + 0, + 0, + ) + }) + .collect() +} + /// Managed-service records that are no longer running, read from the same /// registry `rocm services` reads. A status-only read over already-loaded /// records: no readiness probes, no daemon, no disk I/O of its own. @@ -706,18 +727,7 @@ pub fn run_launcher(chat_mock: bool) -> Result<()> { fn launcher_serving_instances(paths: &AppPaths) -> Vec { use rocm_dash_daemon::registry::{discover_managed_services, load_service_records}; let records = load_service_records(&paths.services_dir()); - discover_managed_services(&records) - .svcs - .into_iter() - .map(|svc| rocm_dash_core::metrics::Instance { - container_id: svc.container_id, - container_name: svc.container_name, - model_name: svc.model_name, - status: svc.status, - port: svc.port, - ..Default::default() - }) - .collect() + managed_instances(&discover_managed_services(&records)) } /// Entry point for a focused launcher flow (Set up / Serve / Diagnose). @@ -1299,6 +1309,61 @@ mod tests { let _ = std::fs::remove_dir_all(&root); } + /// Windows has no daemon, so the dashboard's instance list comes from the + /// probe's `instances` (see `AppState::push_host_gpu_snapshot`). A ready + /// Lemonade record must come through with its model and port; a stopped + /// record and one with no bound port must not. + #[test] + fn live_registry_probe_carries_the_live_managed_instances() { + let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("..") + .join("..") + .join(".rocm-work") + .join("tests") + .join("dash") + .join(format!( + "probe-instances-{}-{}", + std::process::id(), + rocm_core::unix_time_millis() + )); + let p = AppPaths { + config_dir: root.join("config"), + data_dir: root.join("data"), + cache_dir: root.join("cache"), + }; + std::fs::create_dir_all(p.services_dir()).unwrap(); + std::fs::write( + p.services_dir().join("ready.json"), + br#"{"service_id":"lemonade-qwen","engine":"lemonade","model_ref":"Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M","port":11436,"status":"ready","created_at_unix_ms":3}"#, + ) + .unwrap(); + std::fs::write( + p.services_dir().join("stopped.json"), + br#"{"service_id":"lemonade-old","engine":"lemonade","model_ref":"Qwen/Qwen2.5-1.5B-Instruct","port":11435,"status":"stopped","created_at_unix_ms":2}"#, + ) + .unwrap(); + std::fs::write( + p.services_dir().join("unbound.json"), + br#"{"service_id":"lemonade-unbound","engine":"lemonade","port":0,"status":"ready","created_at_unix_ms":1}"#, + ) + .unwrap(); + + let args = resolved_args(&cfg(), &p, ActiveTab::Home); + let instances = probe_snapshot(&args).instances; + assert_eq!( + instances.len(), + 1, + "only the live, bound record: {instances:?}" + ); + let inst = &instances[0]; + assert_eq!(inst.container_id, "lemonade-qwen"); + assert_eq!(inst.model_name, "Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M"); + assert_eq!(inst.port, Some(11436)); + assert_eq!(inst.status, rocm_dash_core::metrics::InstanceStatus::Ready); + + let _ = std::fs::remove_dir_all(&root); + } + /// Paired with `resolved_args_reports_no_live_managed_service_when_none_are_ready_or_running`: /// `managed_ids` and `has_live_service` (`ManagedRegistrySnapshot`) are two /// different questions over the same live read (liveness vs. reachability), so a record diff --git a/crates/rocm-dash-tui/src/app/event_loop.rs b/crates/rocm-dash-tui/src/app/event_loop.rs index 8709b4a53e..ebbb514fe2 100644 --- a/crates/rocm-dash-tui/src/app/event_loop.rs +++ b/crates/rocm-dash-tui/src/app/event_loop.rs @@ -67,6 +67,47 @@ fn seed_registry_probe_and_hosting( state.launcher_hosted = launcher_hosted; } +/// Whether the dashboard polls the managed-service registry for its instance +/// list. Only Windows has no daemon to send instances; a focused host draws no +/// instance list, and `--demo`/`--replay` (`replay`) must never show this +/// host's real services. `windows` is a parameter rather than a `cfg!` so the +/// gate itself is unit-tested on every platform. +const fn polls_managed_registry(focus: Option, replay: bool, windows: bool) -> bool { + windows && !replay && !should_skip_daemon(focus) +} + +/// Read the managed-service registry every `period` and post the live +/// services as `ClientMsg::ManagedInstances`. The probe does a blocking disk +/// read, so it runs under `spawn_blocking` rather than on the event loop — a +/// slow or antivirus-scanned services directory then delays only the next +/// poll, never a frame. The first poll is immediate, so a running service is +/// listed as soon as the dashboard opens. Ends when the receiver is dropped. +fn spawn_managed_registry_poller( + probe: crate::registry_probe::SharedManagedRegistryProbe, + tx: mpsc::UnboundedSender, + period: Duration, +) -> tokio::task::JoinHandle<()> { + tokio::spawn(async move { + let mut ticker = interval(period); + // A slow read must not queue a burst of catch-up polls behind it. + ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + loop { + ticker.tick().await; + let probe = probe.clone(); + let Ok(snapshot) = tokio::task::spawn_blocking(move || probe.snapshot()).await else { + // The probe panicked; keep polling rather than freezing the list. + continue; + }; + if tx + .send(ClientMsg::ManagedInstances(snapshot.instances)) + .is_err() + { + break; + } + } + }) +} + pub async fn run(args: ResolvedArgs) -> color_eyre::Result<()> { // Install the termination-signal watcher BEFORE switching the terminal into // raw/alternate-screen mode. A signal that arrives during startup must find @@ -243,6 +284,11 @@ async fn event_loop(terminal: &mut Tui, args: &ResolvedArgs) -> color_eyre::Resu // Both `--demo` (a generated session replayed) and `--replay ` present // non-live data, so mark the session simulated for the honesty chrome. state.simulated = state.replay.is_some(); + if polls_managed_registry(args.focus, state.simulated, cfg!(windows)) + && let Some(probe) = args.managed_registry_probe.clone() + { + spawn_managed_registry_poller(probe, chat_tx.clone(), Duration::from_secs(1)); + } let mut chat = ChatRuntime::start(&mut state, args, &chat_tx).await; @@ -273,6 +319,7 @@ async fn event_loop(terminal: &mut Tui, args: &ResolvedArgs) -> color_eyre::Resu } Some(ClientMsg::Event(ev)) => state.apply_event(*ev), Some(ClientMsg::HostGpuSnapshot(snap)) => state.push_host_gpu_snapshot(*snap), + Some(ClientMsg::ManagedInstances(live)) => state.set_managed_instances(live), Some(ClientMsg::ReplaySeek) => state.reset_for_seek(), Some(ClientMsg::ReplayPosition { elapsed_s, total_s }) => { if let Some(r) = state.replay.as_mut() { @@ -978,6 +1025,7 @@ mod tests { std::sync::Arc::new(StaticManagedRegistrySnapshot(ManagedRegistrySnapshot { has_live_service: true, managed_ids: std::iter::once("svc-a".to_string()).collect(), + ..Default::default() })); seed_registry_probe_and_hosting(&mut s, Some(probe.clone()), true); @@ -989,4 +1037,110 @@ mod tests { Some(true) ); } + + #[test] + fn polls_managed_registry_state_matrix() { + // The only caller is `event_loop`, which only the Linux e2e lane + // drives, and the Windows arm is the one that matters. `windows` is + // injected so every row runs on every platform's CI job. + assert!( + polls_managed_registry(None, false, true), + "Windows dashboard polls" + ); + assert!( + !polls_managed_registry(None, false, false), + "Linux has a daemon" + ); + assert!( + !polls_managed_registry(None, true, true), + "--demo/--replay must not show this host's real services" + ); + for focus in [Focus::Setup, Focus::Serve, Focus::Examine] { + assert!( + !polls_managed_registry(Some(focus), false, true), + "a focused host draws no instance list: {focus:?}" + ); + } + } + + fn probe_with( + ids: &[&str], + ) -> ( + crate::registry_probe::SharedManagedRegistryProbe, + Vec, + ) { + use crate::registry_probe::{ManagedRegistrySnapshot, StaticManagedRegistrySnapshot}; + let instances: Vec<_> = ids + .iter() + .map(|id| rocm_dash_core::metrics::Instance { + container_id: (*id).to_string(), + status: rocm_dash_core::metrics::InstanceStatus::Ready, + port: Some(11_400), + ..Default::default() + }) + .collect(); + let probe = std::sync::Arc::new(StaticManagedRegistrySnapshot(ManagedRegistrySnapshot { + instances: instances.clone(), + ..Default::default() + })); + (probe, instances) + } + + #[tokio::test] + async fn managed_registry_poller_posts_the_registry_list_immediately_and_repeatedly() { + // The first poll must not wait a full period: a service already + // running when the dashboard opens is listed at once, not after + // "Nothing running" has been on screen for a second. + let (probe, expected) = probe_with(&["svc-a"]); + let (tx, mut rx) = mpsc::unbounded_channel(); + let handle = spawn_managed_registry_poller(probe, tx, Duration::from_millis(20)); + + for _ in 0..2 { + let msg = tokio::time::timeout(Duration::from_secs(5), rx.recv()) + .await + .expect("a poll arrived") + .expect("channel open"); + let ClientMsg::ManagedInstances(got) = msg else { + panic!("unexpected message: {msg:?}"); + }; + assert_eq!(got.len(), expected.len()); + assert_eq!(got[0].container_id, "svc-a"); + } + + drop(rx); + tokio::time::timeout(Duration::from_secs(5), handle) + .await + .expect("the poller stops once the receiver is gone") + .expect("poller did not panic"); + } + + #[tokio::test] + async fn managed_registry_poller_survives_a_panicking_probe() { + // A probe panic (under `spawn_blocking`) must not end the poll loop and + // freeze the instance list for the rest of the session. + #[derive(Debug, Default)] + struct PanicsOnce(std::sync::atomic::AtomicBool); + impl crate::registry_probe::ManagedRegistryProbe for PanicsOnce { + fn snapshot(&self) -> crate::registry_probe::ManagedRegistrySnapshot { + assert!( + self.0.swap(true, std::sync::atomic::Ordering::SeqCst), + "first read fails" + ); + crate::registry_probe::ManagedRegistrySnapshot::default() + } + } + let (tx, mut rx) = mpsc::unbounded_channel(); + let handle = spawn_managed_registry_poller( + std::sync::Arc::new(PanicsOnce::default()), + tx, + Duration::from_millis(20), + ); + + let msg = tokio::time::timeout(Duration::from_secs(5), rx.recv()) + .await + .expect("a later poll arrived") + .expect("channel open"); + assert!(matches!(msg, ClientMsg::ManagedInstances(_)), "{msg:?}"); + handle.abort(); + } } diff --git a/crates/rocm-dash-tui/src/app/mod.rs b/crates/rocm-dash-tui/src/app/mod.rs index 9bbc7ef147..cdd7a1ce1f 100644 --- a/crates/rocm-dash-tui/src/app/mod.rs +++ b/crates/rocm-dash-tui/src/app/mod.rs @@ -599,14 +599,34 @@ impl AppState { /// charts/history exactly like a real snapshot, but deliberately leaves /// `instances` and `has_received_snapshot` untouched. The Windows /// in-process ticker (`client.rs`) always sends an empty `instances` - /// stand-in (no instance-registry daemon exists there yet), so treating - /// it as authoritative the way `push_snapshot` does would permanently - /// blind the issue #145 quit-confirm prompt to models actually being - /// served — see `has_live_instance_with`'s Windows note. + /// stand-in (it has no daemon to discover them), so treating it as + /// authoritative the way `push_snapshot` does would blind the issue #145 + /// quit-confirm prompt to models actually being served — see + /// `has_live_instance_with`'s Windows note. Instances arrive separately, + /// from the managed-service registry, through + /// [`set_managed_instances`](Self::set_managed_instances). pub(crate) fn push_host_gpu_snapshot(&mut self, snap: Snapshot) { self.push_history_and_latest(snap); } + /// Replace `instances` with the managed services the registry poller + /// (`event_loop::spawn_managed_registry_poller`, Windows only) last read, + /// delivered as `ClientMsg::ManagedInstances`. `has_received_snapshot` + /// stays untouched, so the issue #145 quit-confirm gate keeps reading the + /// registry live too. + /// + /// The map is refilled in place rather than rebuilt: a fresh `HashMap` + /// gets a fresh random hasher, so the iteration order — which picks the + /// Home tile's model — would change on every poll. + pub(crate) fn set_managed_instances(&mut self, live: Vec) { + self.instances.clear(); + self.instances.extend( + live.into_iter() + .map(|inst| (inst.container_id.clone(), inst)), + ); + self.clamp_selectors(); + } + /// Shared tail of [`push_snapshot`](Self::push_snapshot) and /// [`push_host_gpu_snapshot`](Self::push_host_gpu_snapshot): the /// history/latest bookkeeping both share, independent of how (or @@ -723,6 +743,83 @@ mod tests { assert_eq!(s.history.len(), 1); } + /// Windows has no daemon to discover instances, so the registry poller's + /// list is what fills `instances`: a served model appears when the poll + /// reports it and drops off once a later poll no longer does. + #[test] + fn set_managed_instances_follows_the_polled_registry_list() { + use crate::registry_probe::{ManagedRegistrySnapshot, StaticManagedRegistrySnapshot}; + use rocm_dash_core::metrics::{Instance, InstanceStatus}; + + let served = Instance { + container_id: "lemonade-qwen".to_string(), + model_name: "Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M".to_string(), + status: InstanceStatus::Ready, + port: Some(11436), + ..Default::default() + }; + let mut s = st(); + s.managed_registry_probe = Some(std::sync::Arc::new(StaticManagedRegistrySnapshot( + ManagedRegistrySnapshot { + has_live_service: true, + managed_ids: std::iter::once(served.container_id.clone()).collect(), + instances: vec![served.clone()], + }, + ))); + + s.set_managed_instances(vec![served.clone()]); + + assert_eq!(s.instances.len(), 1); + let inst = &s.instances["lemonade-qwen"]; + assert_eq!(inst.model_name, served.model_name); + assert_eq!(inst.port, Some(11436)); + assert!( + !s.has_received_snapshot, + "the registry poll must not flip has_received_snapshot" + ); + let snap = s.live_registry_snapshot(); + assert!(s.has_live_instance_with(&snap)); + assert!( + s.has_stoppable_managed_instance_with(&snap), + "a registry-sourced instance must be offered for stopping on quit" + ); + + s.set_managed_instances(Vec::new()); + + assert!( + s.instances.is_empty(), + "a service the registry no longer reports must drop off" + ); + } + + /// The Home tile shows the first serving instance in `instances`' + /// iteration order, so applying the same registry contents poll after + /// poll must not reorder them — otherwise the tile alternates models. + #[test] + fn set_managed_instances_keeps_instance_order_stable_across_polls() { + use rocm_dash_core::metrics::{Instance, InstanceStatus}; + + let polled = || -> Vec { + (0..8) + .map(|n| Instance { + container_id: format!("svc-{n}"), + status: InstanceStatus::Ready, + port: Some(11_400 + n), + ..Default::default() + }) + .collect() + }; + let mut s = st(); + + s.set_managed_instances(polled()); + let first: Vec = s.instances.keys().cloned().collect(); + for _ in 0..5 { + s.set_managed_instances(polled()); + let again: Vec = s.instances.keys().cloned().collect(); + assert_eq!(again, first, "instance order changed between polls"); + } + } + #[test] fn actions_does_not_import_from_scrollbar() { // scrollbar.rs depends on actions.rs (KeyAction/handle_mouse/tab_bar_hit, diff --git a/crates/rocm-dash-tui/src/app/quit_confirm.rs b/crates/rocm-dash-tui/src/app/quit_confirm.rs index c0c6f7096e..21544af3ec 100644 --- a/crates/rocm-dash-tui/src/app/quit_confirm.rs +++ b/crates/rocm-dash-tui/src/app/quit_confirm.rs @@ -38,7 +38,9 @@ impl AppState { /// there either — the live read stays authoritative for the entire /// session rather than only a brief startup window — the registry read is /// cheap and re-run every time this is called, so there is no staleness - /// to outlive. Once a snapshot lands + /// to outlive. (`instances` itself is refilled from the same registry by + /// a background poller once a second — `set_managed_instances` — so it + /// can trail this live read by about a second.) Once a snapshot lands /// on a host where the daemon does connect, `instances` is authoritative /// and the read's *result* is no longer consulted for THIS question (a /// model that legitimately stops later in the session is still reflected diff --git a/crates/rocm-dash-tui/src/app/test_support.rs b/crates/rocm-dash-tui/src/app/test_support.rs index 0920e4a796..2ae8140c4f 100644 --- a/crates/rocm-dash-tui/src/app/test_support.rs +++ b/crates/rocm-dash-tui/src/app/test_support.rs @@ -37,6 +37,7 @@ pub(crate) fn with_registry_probe( ManagedRegistrySnapshot { has_live_service, managed_ids: managed_ids.iter().map(|s| (*s).to_string()).collect(), + ..Default::default() }, ))); } diff --git a/crates/rocm-dash-tui/src/app/types.rs b/crates/rocm-dash-tui/src/app/types.rs index 27fb3962ae..0b0c7fde88 100644 --- a/crates/rocm-dash-tui/src/app/types.rs +++ b/crates/rocm-dash-tui/src/app/types.rs @@ -184,7 +184,8 @@ impl ConnState { /// True for the Windows in-process collector (see /// `client::connect_and_run`'s `#[cfg(windows)]` arm), which tags its /// `daemon_version` this way since there's no real `rocmdashd` behind - /// it — and so no instance registry it could ever populate. + /// it — its instances come only from the managed-service registry + /// (`AppState::set_managed_instances`), never from discovery. pub fn is_in_process(&self) -> bool { matches!(self, Self::Connected { version, .. } if version.starts_with(Self::IN_PROCESS_PREFIX)) } diff --git a/crates/rocm-dash-tui/src/client.rs b/crates/rocm-dash-tui/src/client.rs index 069e2e86b3..1c63b76050 100644 --- a/crates/rocm-dash-tui/src/client.rs +++ b/crates/rocm-dash-tui/src/client.rs @@ -41,10 +41,14 @@ pub enum ClientMsg { /// Deliberately NOT `Event(Snapshot)` — that path treats `instances` as /// authoritative truth and latches `has_received_snapshot` forever, but /// this snapshot's `instances` is always an empty stand-in (Windows has - /// no instance registry daemon yet), so routing it through there would - /// permanently blind the issue #145 quit-confirm prompt to models - /// actually being served. See `AppState::push_host_gpu_snapshot`. + /// no instance registry daemon yet). `AppState::push_host_gpu_snapshot` + /// leaves `instances` alone; they arrive as `ManagedInstances`. HostGpuSnapshot(Box), + /// Windows only: the managed services the registry poller + /// (`event_loop::spawn_managed_registry_poller`) last read, off the UI + /// thread. Replaces `AppState::instances` — see + /// `AppState::set_managed_instances`. + ManagedInstances(Vec), /// Replay-only: the scrubber jumped backward. The TUI must wipe any /// state derived from past events before the burst-emitted events for /// the new playhead position arrive. @@ -230,9 +234,9 @@ fn build_host_gpu_snapshot( /// instead of connecting anywhere: `HostCollector` (already cross-platform) /// for CPU/memory, `WinPerfCollector` (OS perf counters, no AMD tool needed) /// for GPU. Sends `Connected` like the unix arm, but then a stream of -/// `ClientMsg::HostGpuSnapshot` rather than `Event::Snapshot` — deliberately -/// not routed through `Event::Snapshot`, to avoid disturbing the "quit while -/// serving" confirmation prompt (see `AppState::push_host_gpu_snapshot`). +/// `ClientMsg::HostGpuSnapshot` rather than `Event::Snapshot`; instances come +/// from the managed-service registry poller, not from here (see +/// `ClientMsg::ManagedInstances`). #[cfg(windows)] async fn connect_and_run(connect: &str, tx: UnboundedSender) -> anyhow::Result<()> { use rocm_dash_collectors::host::HostCollector; diff --git a/crates/rocm-dash-tui/src/registry_probe.rs b/crates/rocm-dash-tui/src/registry_probe.rs index 87614553db..17a1967cec 100644 --- a/crates/rocm-dash-tui/src/registry_probe.rs +++ b/crates/rocm-dash-tui/src/registry_probe.rs @@ -18,16 +18,20 @@ //! whose daemon client never sends an instance snapshot (Windows — it //! connects and streams live GPU/CPU/memory every tick, but has no //! `rocmdashd` socket to source an instance list from, so -//! `has_received_snapshot` never flips and `AppState::instances` stays empty -//! for the life of the session). A live probe has neither problem — there is -//! no window for it to go stale in. +//! `has_received_snapshot` never flips). A live probe has neither problem — +//! there is no window for it to go stale in. On Windows it is also the source +//! of `AppState::instances` itself: a background poller +//! (`event_loop::spawn_managed_registry_poller`) reads +//! [`ManagedRegistrySnapshot::instances`] off the UI thread once a second. use std::collections::HashSet; use std::sync::Arc; +use rocm_dash_core::metrics::Instance; + /// Live classification of the managed-service registry, as of the moment /// [`ManagedRegistryProbe::snapshot`] was called. -#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[derive(Debug, Clone, Default)] pub struct ManagedRegistrySnapshot { /// Whether any managed-service record is currently a live scrape target — /// the same statuses `Instance::status.is_serving()` treats as "actively @@ -45,6 +49,11 @@ pub struct ManagedRegistrySnapshot { /// Docker- or Lemonade-discovered instance from an external daemon, which /// that command cannot stop. pub managed_ids: HashSet, + /// The live scrape targets themselves (the records `managed_ids` names), + /// as dashboard instances with no telemetry. The Windows in-process ticker + /// has no daemon to send instances, so the registry poller takes them from + /// here. + pub instances: Vec, } /// rocm-core/rocm-dash-daemon-free live registry-read boundary. @@ -63,12 +72,15 @@ pub struct ManagedRegistrySnapshot { /// daemon round-trip or network I/O. In the dashboard, [`snapshot`](Self::snapshot) /// is called every time something needs `live_registry_snapshot`'s answer: /// `request_quit` on the quit keystroke, `quit_confirm_request` on every -/// render while the quit-confirm prompt is open, and -/// `quit_confirm_open_services_key` on the `s` keystroke while it's open. -/// The launcher's own `has_live_instance` check calls it on every render -/// too, but through `StaticManagedRegistrySnapshot` — a read cached once per -/// menu pass — so that per-frame call never reaches an implementation's -/// actual registry I/O; only the dashboard's calls above do. +/// render while the quit-confirm prompt is open, +/// `quit_confirm_open_services_key` on the `s` keystroke while it's open, +/// and — on Windows only — the registry poller +/// (`event_loop::spawn_managed_registry_poller`), once a second under +/// `spawn_blocking`. The launcher's own +/// `has_live_instance` check calls it on every render too, but through +/// `StaticManagedRegistrySnapshot` — a read cached once per menu pass — so +/// that per-frame call never reaches an implementation's actual registry I/O; +/// only the dashboard's calls above do. pub trait ManagedRegistryProbe: std::fmt::Debug + Send + Sync { /// Read the registry now and classify it from the record's own `status` /// field, the same statuses `Instance::status.is_serving()` treats as @@ -122,6 +134,7 @@ mod tests { let probe = StaticManagedRegistrySnapshot(ManagedRegistrySnapshot { has_live_service: true, managed_ids: ids.clone(), + ..Default::default() }); let snap = probe.snapshot(); assert!(snap.has_live_service); diff --git a/crates/rocm-dash-tui/src/ui/launcher.rs b/crates/rocm-dash-tui/src/ui/launcher.rs index 346c4c746b..131fd70756 100644 --- a/crates/rocm-dash-tui/src/ui/launcher.rs +++ b/crates/rocm-dash-tui/src/ui/launcher.rs @@ -321,10 +321,15 @@ fn launcher_state(theme_name: &str, serving: Vec) -> AppState { .iter() .map(|inst| inst.container_id.clone()) .collect(); + // No `instances` here: that field feeds the Windows registry poller + // (`app::event_loop::spawn_managed_registry_poller`), which reads the + // bin-injected live probe rather than this static one, and the launcher + // fills `state.instances` directly on the next line anyway. state.managed_registry_probe = Some(std::sync::Arc::new(StaticManagedRegistrySnapshot( ManagedRegistrySnapshot { has_live_service, managed_ids, + ..Default::default() }, ))); state.instances = serving @@ -892,6 +897,7 @@ mod tests { ManagedRegistrySnapshot { has_live_service: true, managed_ids: std::collections::HashSet::from(["vllm-1".to_string()]), + ..Default::default() }, ))); s.open_quit_confirm(); diff --git a/crates/rocm-dash-tui/src/ui/quit_confirm.rs b/crates/rocm-dash-tui/src/ui/quit_confirm.rs index 3e55a03361..17ff7379ec 100644 --- a/crates/rocm-dash-tui/src/ui/quit_confirm.rs +++ b/crates/rocm-dash-tui/src/ui/quit_confirm.rs @@ -140,11 +140,10 @@ pub(super) fn quit_confirm_body_with( let mut body = Vec::new(); if total == 0 { // `total == 0` here only because `nothing_is_currently_being_served_with` - // returned `false` above: either the startup-race fallback (issue - // #145) is active — the daemon's first snapshot hasn't landed yet — - // or the connection never will succeed at all (Windows: the - // dashboard-embedded daemon client's connect path is unix-only, so - // `instances` stays empty for the whole session). `instances` can't + // returned `false` above: the startup-race fallback (issue #145) is + // active — the daemon's first snapshot hasn't landed yet, or (Windows, + // which has no daemon and refills `instances` from the registry each + // tick) a service was registered since the last tick. `instances` can't // name anything in either case, but the live registry read // (`managed_ids`) needs no daemon snapshot — it is this host's own // `rocm serve` registry, read straight off disk — so it can still @@ -248,11 +247,10 @@ mod tests { #[test] fn quit_confirm_body_prints_the_stop_command_during_the_startup_fallback_when_the_registry_names_an_id() { - // On a host whose daemon client never sends an instance snapshot - // (Windows — it connects and streams live GPU/CPU/memory every tick, - // but has no `rocmdashd` socket to source an instance list from, so - // `has_received_snapshot` never flips for the whole session), - // `instances` never fills, so without this the quit prompt + // Before `instances` has been filled (the daemon's first snapshot + // has not landed, or on Windows the registry gained a service since + // the last registry poll), `has_received_snapshot` is false and + // `instances` is empty, so without this the quit prompt // could only ever print the generic "a model was serving" sentence — // no model name (expected: `instances` is the only source for that), // but also no stop command and no id, even though the live registry diff --git a/crates/rocm-dash-tui/src/ui/tabs/instances.rs b/crates/rocm-dash-tui/src/ui/tabs/instances.rs index 62ebdaf811..f1c7e4fcf8 100644 --- a/crates/rocm-dash-tui/src/ui/tabs/instances.rs +++ b/crates/rocm-dash-tui/src/ui/tabs/instances.rs @@ -206,7 +206,7 @@ pub(in crate::ui) const fn status_meta( fn draw_empty(f: &mut Frame, area: Rect, state: &AppState, theme: &Theme) { let body = if state.conn.is_in_process() { - "no instance registry on this platform · vLLM containers aren't available here" + "no managed services running · serve one from 3 Serving · vLLM containers aren't available here" } else { match &state.conn { ConnState::Connected { .. } => { @@ -273,8 +273,10 @@ pub(in crate::ui) mod tests { use ratatui::backend::TestBackend; // The Windows in-process collector (see `ConnState::is_in_process`) - // has no instance registry at all, ever — distinct from "a real - // daemon is connected but nothing is running" or "still connecting". + // has no daemon: its instances come only from the managed-service + // registry, so an empty list means nothing is served — distinct from + // "a real daemon is connected but nothing is running" or "still + // connecting". let mut state = mk_state(HashMap::new(), 0); state.conn = ConnState::Connected { host: "localhost".into(), @@ -285,8 +287,8 @@ pub(in crate::ui) mod tests { .unwrap(); let grid = buffer_text(&term); assert!( - grid.contains("no instance registry on this platform"), - "should name the platform limitation, not claim a daemon could be started; got:\n{grid}" + grid.contains("no managed services running"), + "should say nothing is served, not claim a daemon could be started; got:\n{grid}" ); assert!( !grid.contains("start daemon"), diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 0a4aaacc9b..9c0ebc54e2 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -799,21 +799,17 @@ Expected result: - A prompt appears naming the serving model and warning that it may still be running in the background after you quit, with the real `rocm services stop --yes` command (the actual id, not a - placeholder) to stop it first. From the dashboard (not the launcher) on - Windows, where the dashboard streams live GPU/CPU/memory but never gets a - serving-instance snapshot (no `rocmdashd` unix socket there), the prompt - still shows that same stop command and the service id, but not the - model's name — the dashboard's own live view is where the name comes - from, and it never fills there. The launcher's own copy of the prompt - always names the model, Windows included — it reads the managed-service - registry directly rather than through the dashboard's daemon connection. + placeholder) to stop it first. On Windows the dashboard has no + `rocmdashd` unix socket, so it reads the same managed-service registry the + launcher does: the prompt names the model there too. Within the first + second after `rocm serve` registers a service, before the dashboard's next + registry read, it may show the stop command and id without the model's name. - From the dashboard (not the launcher), the prompt also offers `s`: pressing it closes the prompt and opens the Services overlay, pre-focused on the serving model, so you can stop it there through the overlay's own - approval+job flow. The launcher's copy of the prompt does not show this - hint or respond to `s` — it has no live Services overlay to jump to — and - neither does the dashboard on Windows, for the same reason the model's name - is missing there. + approval+job flow, on Windows too. The launcher's copy of the prompt does + not show this hint or respond to `s` — it has no live Services overlay to + jump to. - Declining (`n`/Esc) returns to where you were — the dashboard, or the launcher's own front door, whichever you pressed `q` from. Any plain modal that was open when you pressed `q` (the Esc menu, Options, Detail, Theme @@ -908,8 +904,14 @@ Expected result: present. With more than one adapter (for example a hybrid laptop) VRAM total stays unmeasured and shows `—` in the Home hero and in the GPU wall sidebar, never `0.0%`. -- The hero's `T/S` row reads `—`, never `0` or `-0`: Windows has no - serving-instance view, so no throughput is ever measured there. +- The hero's `T/S` row reads `—`, never `0` or `-0`: a managed server is + listed with no throughput telemetry behind it, so no rate is ever measured + there. +- With a managed server running (`rocm serve`, then `rocm services list` + showing it `ready`), the Home tab's `Running` count, the Serving tab's + `Running now` list and the Observe tab list that server within a few seconds + of `rocm dash` opening, not "Nothing running". After `rocm services stop` + it drops off within a few seconds. Its throughput stays unmeasured. - Utilization rises under GPU load (run a Lemonade or ComfyUI job) and falls back when it ends. - Hiding `amdadlx64.dll` (for example on a non-AMD GPU host) leaves diff --git a/docs/rocm-docs/install/installation.md b/docs/rocm-docs/install/installation.md index 3fb9ed7b53..15d61feee3 100644 --- a/docs/rocm-docs/install/installation.md +++ b/docs/rocm-docs/install/installation.md @@ -15,8 +15,9 @@ ROCm CLI ships as a single prebuilt binary. Platform support: For WSL2 setup and diagnosis, see [ROCm CLI on WSL2](wsl.md). -Live dashboard GPU/CPU/memory telemetry works on Windows too; the -serving-instance registry view is Linux/WSL2 only, since there's no +Live dashboard GPU/CPU/memory telemetry and the managed servers +`rocm services list` reports work on Windows too; per-server throughput and +Docker-discovered instances are Linux/WSL2 only, since there's no `rocmdashd` unix socket on Windows (see [Interactive interfaces](../getting-started.md#interactive-interfaces)). vLLM serving is Linux or WSL2 only (see From 5ab336d2a082cfc0fd78f57dd6b054c5658b1e76 Mon Sep 17 00:00:00 2001 From: Roman Sirokov Date: Fri, 13 Nov 2026 15:11:08 +0200 Subject: [PATCH 2/5] test(dash): make the three managed-instance tests fail when the behavior breaks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each of these named a behavior it could not actually observe, so the bug it was written against would have slipped through it. The poller test timed out at 5s against a 20ms period: a first read delayed a full period cleared that bound just as easily as an immediate one, so the "immediate" in its name was unasserted. Period and deadline now straddle each other (1s vs 250ms) so only an immediate first read passes. The selector test left `instance_sel` at its default 0 before shrinking the list, which makes `clamp_selectors` unobservable — drop the call and nothing changes. It now selects a second row first, then empties the list under it. The empty-state test asserted only the diagnosis half of the message and rendered at 80 columns, truncating the remedy it never checked. It now renders wide enough for the whole line and asserts it, tab digit included, so a tab reorder that leaves this string behind is caught. Signed-off-by: Roman Sirokov --- crates/rocm-dash-tui/src/app/event_loop.rs | 26 ++++++++++++++++--- crates/rocm-dash-tui/src/app/mod.rs | 19 ++++++++++++++ crates/rocm-dash-tui/src/ui/tabs/instances.rs | 13 +++++++--- 3 files changed, 51 insertions(+), 7 deletions(-) diff --git a/crates/rocm-dash-tui/src/app/event_loop.rs b/crates/rocm-dash-tui/src/app/event_loop.rs index ebbb514fe2..5194e5cd9e 100644 --- a/crates/rocm-dash-tui/src/app/event_loop.rs +++ b/crates/rocm-dash-tui/src/app/event_loop.rs @@ -1091,14 +1091,32 @@ mod tests { // The first poll must not wait a full period: a service already // running when the dashboard opens is listed at once, not after // "Nothing running" has been on screen for a second. + // + // The period is deliberately much longer than the first read's deadline + // (1s vs 250ms), because the deadline only discriminates if one period + // does not fit inside it: with a period as short as the bound, a poller + // that slept a full period before its first read would clear it anyway + // and the test would pass on the very bug it exists to catch. + const PERIOD: Duration = Duration::from_secs(1); + const FIRST_READ_DEADLINE: Duration = Duration::from_millis(250); + let (probe, expected) = probe_with(&["svc-a"]); let (tx, mut rx) = mpsc::unbounded_channel(); - let handle = spawn_managed_registry_poller(probe, tx, Duration::from_millis(20)); + let handle = spawn_managed_registry_poller(probe, tx, PERIOD); - for _ in 0..2 { - let msg = tokio::time::timeout(Duration::from_secs(5), rx.recv()) + // Poll 0 is the immediate one and gets the tight bound; poll 1 only has + // to prove the loop repeats, so it is allowed a full period plus slack. + for poll in 0..2 { + let deadline = if poll == 0 { + FIRST_READ_DEADLINE + } else { + PERIOD + FIRST_READ_DEADLINE + }; + let msg = tokio::time::timeout(deadline, rx.recv()) .await - .expect("a poll arrived") + .unwrap_or_else(|_| { + panic!("poll {poll} did not arrive within {deadline:?} (period {PERIOD:?})") + }) .expect("channel open"); let ClientMsg::ManagedInstances(got) = msg else { panic!("unexpected message: {msg:?}"); diff --git a/crates/rocm-dash-tui/src/app/mod.rs b/crates/rocm-dash-tui/src/app/mod.rs index cdebc333e6..a086960f7b 100644 --- a/crates/rocm-dash-tui/src/app/mod.rs +++ b/crates/rocm-dash-tui/src/app/mod.rs @@ -793,12 +793,31 @@ mod tests { "a registry-sourced instance must be offered for stopping on quit" ); + // Shrink the list out from under a selection that points past the new + // end. `instance_sel` has to be nonzero first, or the clamp inside + // `set_managed_instances` is unobservable — with the selector already + // at its default 0 the call could be dropped entirely and every + // assertion below would still hold. + let second = Instance { + container_id: "lemonade-llama".to_string(), + status: InstanceStatus::Ready, + port: Some(11437), + ..Default::default() + }; + s.set_managed_instances(vec![served, second]); + s.instance_sel = 1; + s.set_managed_instances(Vec::new()); assert!( s.instances.is_empty(), "a service the registry no longer reports must drop off" ); + assert_eq!( + s.instance_sel, 0, + "the row selection must be clamped to the shrunken list, not left \ + pointing past its end" + ); } /// The Home tile shows the first serving instance in `instances`' diff --git a/crates/rocm-dash-tui/src/ui/tabs/instances.rs b/crates/rocm-dash-tui/src/ui/tabs/instances.rs index f1c7e4fcf8..13c214cd84 100644 --- a/crates/rocm-dash-tui/src/ui/tabs/instances.rs +++ b/crates/rocm-dash-tui/src/ui/tabs/instances.rs @@ -282,13 +282,20 @@ pub(in crate::ui) mod tests { host: "localhost".into(), version: ConnState::in_process_version("0.0.0"), }; - let mut term = Terminal::new(TestBackend::new(80, 20)).unwrap(); + // Wide enough to hold the whole line: the remedy half is what this test + // is really about, and 80 columns would truncate it away unasserted. + let mut term = Terminal::new(TestBackend::new(160, 20)).unwrap(); term.draw(|f| draw_table(f, f.area(), &state, &state.theme)) .unwrap(); let grid = buffer_text(&term); + // Asserted whole, tab digit included. `ActiveTab::from_digit` maps '3' + // to Serving today; a tab reorder that left this line behind would send + // the user to the wrong tab, and nothing else here would catch it. assert!( - grid.contains("no managed services running"), - "should say nothing is served, not claim a daemon could be started; got:\n{grid}" + grid.contains( + "no managed services running · serve one from 3 Serving · vLLM containers aren't available here" + ), + "should say nothing is served AND where to serve one from instead, not claim a daemon could be started; got:\n{grid}" ); assert!( !grid.contains("start daemon"), From de4ed14b1a997c23d4e77ee0e08b2bb5cc85fdfe Mon Sep 17 00:00:00 2001 From: Roman Sirokov Date: Fri, 13 Nov 2026 17:43:03 +0200 Subject: [PATCH 3/5] fix(dash): wrap the instances empty state, and log a panicking registry probe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The in-process empty-state message is 94 characters rendered through an unwrapped `Paragraph`, which ratatui clips rather than wraps. An earlier round on this branch widened that test's backend from 80 to 160 columns so the assertion would pass; that documented the clipping instead of fixing it. Wrap the paragraph the way every other prose paragraph in this UI already does, and assert at 80 columns — reading the panel back through the wrap, so a clipped render still cannot pass. The registry poller's `spawn_blocking` join-failure arm swallowed a probe panic with only a comment. A probe that panics every tick would leave the instance list empty for a whole session with nothing in the log; warn instead, matching `client.rs`'s per-tick Windows failure, and assert the warning fires. Signed-off-by: Roman Sirokov --- crates/rocm-dash-tui/src/app/event_loop.rs | 159 +++++++++++++++++- crates/rocm-dash-tui/src/ui/tabs/instances.rs | 59 ++++++- 2 files changed, 209 insertions(+), 9 deletions(-) diff --git a/crates/rocm-dash-tui/src/app/event_loop.rs b/crates/rocm-dash-tui/src/app/event_loop.rs index 5194e5cd9e..2936fa95ce 100644 --- a/crates/rocm-dash-tui/src/app/event_loop.rs +++ b/crates/rocm-dash-tui/src/app/event_loop.rs @@ -76,6 +76,12 @@ const fn polls_managed_registry(focus: Option, replay: bool, windows: boo windows && !replay && !should_skip_daemon(focus) } +/// Logged when a registry read panics under `spawn_blocking`. Named so the +/// test asserting the warning fires cannot pass against a message that was +/// silently reworded or downgraded below WARN. +const MANAGED_REGISTRY_PROBE_PANIC: &str = + "managed-service registry probe panicked; instance list not updated this poll"; + /// Read the managed-service registry every `period` and post the live /// services as `ClientMsg::ManagedInstances`. The probe does a blocking disk /// read, so it runs under `spawn_blocking` rather than on the event loop — a @@ -94,9 +100,18 @@ fn spawn_managed_registry_poller( loop { ticker.tick().await; let probe = probe.clone(); - let Ok(snapshot) = tokio::task::spawn_blocking(move || probe.snapshot()).await else { - // The probe panicked; keep polling rather than freezing the list. - continue; + let snapshot = match tokio::task::spawn_blocking(move || probe.snapshot()).await { + Ok(snapshot) => snapshot, + Err(e) => { + // The probe panicked; keep polling rather than freezing the + // list — but say so. A probe that panics every tick would + // otherwise leave the instance list silently empty for the + // whole session with nothing in the log to explain it. + // Same shape as `client.rs`'s per-tick Windows + // perf-counter failure. + tracing::warn!(error = %e, "{MANAGED_REGISTRY_PROBE_PANIC}"); + continue; + } }; if tx .send(ClientMsg::ManagedInstances(snapshot.instances)) @@ -1147,6 +1162,15 @@ mod tests { crate::registry_probe::ManagedRegistrySnapshot::default() } } + + // This test reaches the poller's `warn!` callsite too, so it must warm + // the interest cache for the same reason + // `managed_registry_poller_warns_when_the_probe_panics` does — see + // `warm_tracing_interest_cache`. Without the call here, whichever of + // the two tests registers that callsite first decides whether it is + // ever enabled, and this one has no subscriber of its own. + warm_tracing_interest_cache(); + let (tx, mut rx) = mpsc::unbounded_channel(); let handle = spawn_managed_registry_poller( std::sync::Arc::new(PanicsOnce::default()), @@ -1161,4 +1185,133 @@ mod tests { assert!(matches!(msg, ClientMsg::ManagedInstances(_)), "{msg:?}"); handle.abort(); } + + /// Minimal `tracing::Subscriber` recording each event's level and message + /// text, so a test can assert that a SPECIFIC diagnostic fired — not + /// merely that something was logged. Counting events alone would stay + /// green if the `warn!` were downgraded to `debug!` or its text removed. + /// `agent/clients.rs`'s `CapturedWarnEvents` does the same for that + /// module's warnings; it is private to that test module, so this is the + /// local equivalent rather than a shared helper. + #[derive(Default)] + struct CapturedWarnings(std::sync::Mutex>); + + impl CapturedWarnings { + fn warned(&self, message: &str) -> bool { + self.0 + .lock() + .unwrap() + .iter() + .any(|(level, text)| *level == tracing::Level::WARN && text == message) + } + } + + impl tracing::Subscriber for CapturedWarnings { + fn enabled(&self, _metadata: &tracing::Metadata<'_>) -> bool { + true + } + + fn new_span(&self, _span: &tracing::span::Attributes<'_>) -> tracing::span::Id { + tracing::span::Id::from_u64(1) + } + + fn record(&self, _span: &tracing::span::Id, _values: &tracing::span::Record<'_>) {} + + fn record_follows_from(&self, _span: &tracing::span::Id, _follows: &tracing::span::Id) {} + + fn event(&self, event: &tracing::Event<'_>) { + struct MessageVisitor(Option); + impl tracing::field::Visit for MessageVisitor { + fn record_debug( + &mut self, + field: &tracing::field::Field, + value: &dyn std::fmt::Debug, + ) { + // tracing wraps the message in an internal type whose + // `Debug` forwards to `Display`, so `{value:?}` is the + // message text itself, unquoted. + if field.name() == "message" { + self.0 = Some(format!("{value:?}")); + } + } + } + let mut visitor = MessageVisitor(None); + event.record(&mut visitor); + if let Some(message) = visitor.0 { + self.0 + .lock() + .unwrap() + .push((*event.metadata().level(), message)); + } + } + + fn enter(&self, _span: &tracing::span::Id) {} + + fn exit(&self, _span: &tracing::span::Id) {} + } + + /// Register one always-interested process-wide default subscriber before + /// any test can reach the poller's `warn!` callsite. + /// + /// tracing-core caches a callsite's `Interest` the first time that + /// callsite executes, by asking the dispatcher in scope on whichever + /// thread happens to get there first. Under threaded `cargo test` that is + /// a race between this module's two panicking-probe tests, and the other + /// one carries no subscriber — so without this, "never interested" can be + /// cached before the capturing test ever installs its own, and the + /// assertion below fails for a reason that has nothing to do with the + /// production code. A permanent global default whose `enabled()` is + /// always `true` makes "nobody is interested yet" impossible from the + /// first registration onward. Its own buffer is never read; only the + /// thread-local override installed in the test is. Same remedy, and same + /// reasoning, as `agent/clients.rs`'s `warm_tracing_interest_cache`. + fn warm_tracing_interest_cache() { + static ONCE: std::sync::Once = std::sync::Once::new(); + ONCE.call_once(|| { + let _ = tracing::subscriber::set_global_default(CapturedWarnings::default()); + }); + } + + #[tokio::test] + async fn managed_registry_poller_warns_when_the_probe_panics() { + // Surviving the panic (above) is not enough: a probe that panics on + // every tick degrades this feature for a whole session, and without a + // log line the operator sees only a permanently empty instance list + // with nothing anywhere to explain it. + #[derive(Debug)] + struct AlwaysPanics; + impl crate::registry_probe::ManagedRegistryProbe for AlwaysPanics { + fn snapshot(&self) -> crate::registry_probe::ManagedRegistrySnapshot { + panic!("registry read blew up"); + } + } + + // Thread-local, not global: `#[tokio::test]` runs a current-thread + // runtime, so the poller task is polled on this very thread and the + // `warn!` dispatches through this subscriber. The global default the + // warming installs is only there to keep the callsite's cached + // `Interest` from being decided by a subscriber-less thread. + warm_tracing_interest_cache(); + let captured = std::sync::Arc::new(CapturedWarnings::default()); + let guard = tracing::subscriber::set_default(captured.clone()); + + let (tx, _rx) = mpsc::unbounded_channel(); + let handle = spawn_managed_registry_poller( + std::sync::Arc::new(AlwaysPanics), + tx, + Duration::from_millis(20), + ); + + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + while !captured.warned(MANAGED_REGISTRY_PROBE_PANIC) { + assert!( + tokio::time::Instant::now() < deadline, + "a panicking probe must log {MANAGED_REGISTRY_PROBE_PANIC:?} at WARN, not fail silently" + ); + tokio::time::sleep(Duration::from_millis(10)).await; + } + + handle.abort(); + drop(guard); + } } diff --git a/crates/rocm-dash-tui/src/ui/tabs/instances.rs b/crates/rocm-dash-tui/src/ui/tabs/instances.rs index 13c214cd84..7b6080ea90 100644 --- a/crates/rocm-dash-tui/src/ui/tabs/instances.rs +++ b/crates/rocm-dash-tui/src/ui/tabs/instances.rs @@ -10,7 +10,7 @@ use ratatui::Frame; use ratatui::layout::{Constraint, Direction, Layout, Rect}; use ratatui::style::{Modifier, Style}; use ratatui::text::{Line, Span}; -use ratatui::widgets::{Cell, Paragraph, Row, Table}; +use ratatui::widgets::{Cell, Paragraph, Row, Table, Wrap}; use rocm_dash_core::metrics::{ Instance, InstanceStatus, ObservationFreshness, ObservationMetadata, @@ -216,10 +216,23 @@ fn draw_empty(f: &mut Frame, area: Rect, state: &AppState, theme: &Theme) { } }; let inner = panel::bento(f, area, Some("Instances"), BoxRole::Neutral, false, theme); + // Wrapped, not one clipped line. An unwrapped `Paragraph` truncates at the + // panel's right edge rather than wrapping, and the longer branches here + // outrun the columns an 80-column terminal leaves usable inside this + // border: the in-process one (94 chars) stopped dead at + // "· vLLM containers ar", and the `Connected` one (79) is at the edge of + // the same cliff. The narrower the pane, the more of the remedy goes too. + // `Wrap { trim: false }` is what every other prose paragraph in this UI + // uses (`ui/tabs/pane.rs`, `ui/tabs/hardware.rs`, `ui/help_modal.rs`, + // `ui/warnings_modal.rs`, `ui/approval.rs`), so the empty state now + // behaves like the rest of them; shortening the string instead would have + // had to drop either the remedy or the vLLM explanation, and would only + // move the width at which the clipping returns. let p = Paragraph::new(Line::from(Span::styled( body, Style::default().fg(theme.muted), - ))); + ))) + .wrap(Wrap { trim: false }); f.render_widget(p, inner); } @@ -282,17 +295,26 @@ pub(in crate::ui) mod tests { host: "localhost".into(), version: ConnState::in_process_version("0.0.0"), }; - // Wide enough to hold the whole line: the remedy half is what this test - // is really about, and 80 columns would truncate it away unasserted. - let mut term = Terminal::new(TestBackend::new(160, 20)).unwrap(); + // 80 columns — a realistic terminal, the same width + // `ui/tabs/mod.rs`'s panel test uses — NOT a width picked to make the + // assertion pass. The message is 94 chars and the panel's border and + // padding narrow that further, so at 80 columns it survives only + // because `draw_empty` wraps: unwrapped, the render stops dead at + // "· vLLM containers ar" and the user is never told why the vLLM + // containers they are looking for cannot appear here. + let mut term = Terminal::new(TestBackend::new(80, 20)).unwrap(); term.draw(|f| draw_table(f, f.area(), &state, &state.theme)) .unwrap(); let grid = buffer_text(&term); // Asserted whole, tab digit included. `ActiveTab::from_digit` maps '3' // to Serving today; a tab reorder that left this line behind would send // the user to the wrong tab, and nothing else here would catch it. + // Read back through `flatten_wrapped_prose` because the message now + // occupies more than one row: asserting only the first row's worth + // would be the same mistake as widening the backend. + let flat = flatten_wrapped_prose(&grid); assert!( - grid.contains( + flat.contains( "no managed services running · serve one from 3 Serving · vLLM containers aren't available here" ), "should say nothing is served AND where to serve one from instead, not claim a daemon could be started; got:\n{grid}" @@ -303,6 +325,31 @@ pub(in crate::ui) mod tests { ); } + /// Read a rendered panel back as the one sentence the user reads. + /// + /// A wrapped `Paragraph` lays its prose across several buffer rows, each + /// padded to the terminal's width and fenced by the panel's box-drawing + /// border. Treating every border glyph and every run of blanks as a single + /// space rejoins the rows at exactly the single spaces the source string + /// had, so a wrapped message can still be asserted verbatim — and a + /// *clipped* one still cannot, because the clipped characters are simply + /// not in the buffer to rejoin. Only U+2500–U+257F (box drawing) is + /// blanked, so the message's own `·` (U+00B7) survives. + fn flatten_wrapped_prose(grid: &str) -> String { + grid.chars() + .map(|c| { + if matches!(u32::from(c), 0x2500..=0x257F) { + ' ' + } else { + c + } + }) + .collect::() + .split_whitespace() + .collect::>() + .join(" ") + } + #[test] fn status_meta_maps_each_variant() { let theme = Theme::default_dark(); From 10fd5c9e1fd3148ce523c7f2d0f05d13ec1ef535 Mon Sep 17 00:00:00 2001 From: Roman Sirokov Date: Mon, 16 Nov 2026 12:53:56 +0200 Subject: [PATCH 4/5] fix(dash): build the polled-registry fixture the way the poller does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merge of main auto-merged app/mod.rs and app/quit_confirm.rs with no conflict, but main had tightened `has_stoppable_managed_instance_with` to screen on `is_listed_service` (i.e. `Instance::managed`), because the Services overlay that `s` jumps to lists only managed rows. `set_managed_instances_follows_the_polled_registry_list` hand-builds its instance with `..Default::default()`, which leaves `managed` false — a shape the poller cannot actually produce, since it maps through `merge_instance`, which copies `managed` from the record, and `discover_managed_services` only ever emits `managed: true`. So the test's own assertion that a registry-sourced instance is offered for stopping on quit began failing on a merge that compiled clean. Set `managed: true` on the fixture so it matches what the poller produces. The production path was already correct; only the fixture was unrealistic. Signed-off-by: Roman Sirokov --- crates/rocm-dash-tui/src/app/mod.rs | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/crates/rocm-dash-tui/src/app/mod.rs b/crates/rocm-dash-tui/src/app/mod.rs index 16a73b39cc..599d6bfe8c 100644 --- a/crates/rocm-dash-tui/src/app/mod.rs +++ b/crates/rocm-dash-tui/src/app/mod.rs @@ -763,11 +763,20 @@ mod tests { use crate::registry_probe::{ManagedRegistrySnapshot, StaticManagedRegistrySnapshot}; use rocm_dash_core::metrics::{Instance, InstanceStatus}; + // `managed: true` is not decoration. The poller builds these through + // `merge_instance`, which copies `managed` from the registry record, + // and `discover_managed_services` only ever emits `managed: true`, so + // a polled instance is always managed. Hand-building it with + // `..Default::default()` alone would leave the field `false` — a shape + // the poller cannot produce — and `has_stoppable_managed_instance_with` + // screens on exactly that field (`is_listed_service`), because the + // Services overlay `s` jumps to lists only managed rows. let served = Instance { container_id: "lemonade-qwen".to_string(), model_name: "Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M".to_string(), status: InstanceStatus::Ready, port: Some(11436), + managed: true, ..Default::default() }; let mut s = st(); From 75d72eeb590fdec5901deddedacda14f20ea1173 Mon Sep 17 00:00:00 2001 From: Roman Sirokov Date: Mon, 16 Nov 2026 15:13:35 +0200 Subject: [PATCH 5/5] docs(dash): say where the Windows instance list actually comes from MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding: architecture.md's Windows boundary note was narrowly true of client.rs's own HostGpuSnapshot, but as the only place the doc describes Windows instance handling it had come to read as "Windows has no instance list" — which this PR's registry poller makes false. It now names spawn_managed_registry_poller and ClientMsg::ManagedInstances as the path the list arrives by, so the boundary the note draws is about the metrics snapshot rather than about Windows. Signed-off-by: Roman Sirokov --- docs/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture.md b/docs/architecture.md index b97561e7a3..dd4c96ce76 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -583,7 +583,7 @@ Boundaries: - `rocm-core` data reaches it through the [`tool_exec.rs`](../crates/rocm-dash-tui/src/tool_exec.rs) and [`registry_probe.rs`](../crates/rocm-dash-tui/src/registry_probe.rs) seams, which the `rocm` binary implements, or as values the binary plumbs in on `ResolvedArgs`. - A few values that mirror `rocm-core` (the default `rocm serve` port in [`llm.rs`](../crates/rocm-dash-tui/src/llm.rs), the runtime Python layout in [`ui/exec.rs`](../crates/rocm-dash-tui/src/ui/exec.rs)) are literal copies kept because of that rule; a change to the original needs the copy changed too. - `ManagedRegistryProbe` is read live, at each quit attempt and each frame the quit-confirm prompt is open, so an implementation must stay cheap (a local file read). -- **Windows.** There is no daemon socket on Windows: the `cfg(windows)` arm of [`client.rs`](../crates/rocm-dash-tui/src/client.rs) collects host and GPU metrics in-process through `rocm-dash-collectors`, the only use of that dependency, and the snapshot it posts does not replace the instance list. +- **Windows.** There is no daemon socket on Windows: the `cfg(windows)` arm of [`client.rs`](../crates/rocm-dash-tui/src/client.rs) collects host and GPU metrics in-process through `rocm-dash-collectors`, the only use of that dependency, and the snapshot it posts does not replace the instance list. The instance list arrives separately: [`event_loop.rs`](../crates/rocm-dash-tui/src/app/event_loop.rs)'s `spawn_managed_registry_poller` reads the on-disk managed-service registry on a timer and sends it as `ClientMsg::ManagedInstances`. So Windows does have an instance list — it is simply not carried by the metrics snapshot. - **Overlays and approvals.** - Each operational overlay is an `Option` state struct on `AppState` (`None` is closed); every opener calls `close_overlays` first, so at most one is open, and `draw_active_manager` draws it or its job console. - A mutating overlay action stages an `ApprovalRequest` in the overlay's own state; the job starts only on Approve and the `rocm` CLI does the mutation. Read-only actions start their job directly, and so does the bench-run form.