From 9684e3be1f5b0fd88046b579a0eb8d02c31c50c5 Mon Sep 17 00:00:00 2001 From: Eugene Volen Date: Tue, 10 Nov 2026 17:44:31 +0200 Subject: [PATCH] feat(dash): attribute VRAM and power to managed rocm serve services by process ancestry (ROCMAI-941) VRAM is attributed per instance by joining amd-smi's GPU process PIDs to the docker/containerd id in /proc//cgroup. A managed `rocm serve` service is not a container, so it never joined: every managed service showed the whole device's VRAM, two services on one GPU each showed all of it, and the VRAM-weighted power split never applied to them. A service record carries the PIDs of its engine and launcher and their kernel start times. The new ancestry module walks a GPU process's /proc//stat parent links up to a recorded root and returns that service's id. A root whose live start time differs from the recorded one is a recycled PID and claims nothing; an unreadable or looping chain ends with no owner. The runner tries the cgroup join first and ancestry second, so container ids and service ids key the same per-container map the instances already read. attribute_power_w also stops letting a service that is still starting, with no VRAM reading while a neighbour has one, take an equal share: it has not allocated GPU memory, so an equal split showed the serving neighbour about half its power and roughly double its tokens per watt until the newcomer loaded. With no readings at all, or a serving occupant without one, the split stays equal. Linux only: elsewhere ancestry yields no owner and attribution is unchanged. Signed-off-by: Eugene Volen --- apps/rocm/src/dash.rs | 6 +- crates/rocm-dash-collectors/src/ancestry.rs | 230 +++++++++++++++++ crates/rocm-dash-collectors/src/lib.rs | 1 + crates/rocm-dash-core/src/efficiency.rs | 66 +++++ crates/rocm-dash-daemon/src/registry.rs | 150 ++++++++++- crates/rocm-dash-daemon/src/runner.rs | 238 ++++++++++++++++-- .../tests/telemetry_contract.rs | 147 ++++++++++- docs/architecture.md | 7 +- 8 files changed, 809 insertions(+), 36 deletions(-) create mode 100644 crates/rocm-dash-collectors/src/ancestry.rs diff --git a/apps/rocm/src/dash.rs b/apps/rocm/src/dash.rs index 28346a4c05..454efc7b38 100644 --- a/apps/rocm/src/dash.rs +++ b/apps/rocm/src/dash.rs @@ -1157,8 +1157,10 @@ mod tests { /// `running`, and there is no process or listener behind it. That status /// string is the whole of what decides the count — this path never consults /// a pid and never probes the port, and the on-disk view it deserializes - /// into (`rocm_dash_daemon::registry::ServiceRecord`) carries no pid fields - /// at all. Asserting 0 is asserting that no refresh happens on this path — + /// into (`rocm_dash_daemon::registry::ServiceRecord`) does carry pid fields + /// now, for the daemon's process attribution. Discovery on this path derives + /// ancestry roots from them too, but the count asserted here comes from the + /// record's status string alone and never reads a pid. Asserting 0 is asserting that no refresh happens on this path — /// if someone later adds one, this test fails and they are pointed at the /// doc comment explaining the trade-off rather than silently flipping a /// launch-time read into network I/O. diff --git a/crates/rocm-dash-collectors/src/ancestry.rs b/crates/rocm-dash-collectors/src/ancestry.rs new file mode 100644 index 0000000000..5d2f30613a --- /dev/null +++ b/crates/rocm-dash-collectors/src/ancestry.rs @@ -0,0 +1,230 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! GPU process → managed service, by process ancestry. +//! +//! `amd-smi process` reports the host PIDs of GPU compute processes. For a +//! managed `rocm serve` service those are the engine's workers, which descend +//! from the PID rocm-cli recorded for the engine (or for its supervisor), so a +//! walk up `/proc//stat` parent links reaches a recorded root. This is the +//! managed-service counterpart of [`crate::cgroup`], which only recognises +//! docker/containerd ids and can never name a managed service. +//! +//! The matcher is pure: the `/proc//stat` reader is injected, so the +//! whole thing is testable on any host. Only [`owner_of_pid`] reads `/proc`, +//! and it is `cfg`-gated with a non-Linux stub. + +/// How many parent links to follow before giving up. Real engines sit a few +/// levels deep (launcher, server, worker); the bound only stops a corrupt or +/// adversarial parent chain from looping. +const MAX_DEPTH: usize = 32; + +/// The lowest PID that can be a root. +/// +/// PID 0 means "not recorded" and PID 1 is init, an ancestor of every process on +/// the host: a root there would own all of them, including GPU processes that +/// belong to no service. +pub const MIN_ROOT_PID: u32 = 2; + +/// A recorded process that identifies one managed service. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Root { + /// The recorded PID (engine or supervisor). + pub pid: u32, + /// The kernel start time recorded with it, when there is one. A PID can be + /// recycled, so a root whose live start time differs is a different process + /// that merely has the same number, and must not claim anything. + pub start_ticks: Option, + /// The managed service this root belongs to. + pub owner: String, +} + +/// `(parent pid, start time in clock ticks)` from the contents of +/// `/proc//stat`, or `None` when it does not parse. +/// +/// The process name (field 2) is parenthesised and may itself contain spaces +/// and parentheses, so the fields after it are counted from the *last* `)`. +/// Field numbers below are the `proc(5)` ones: state is 3, ppid is 4 and the +/// start time is 22. +#[must_use] +pub fn parse_stat(stat: &str) -> Option<(u32, u64)> { + let rest = &stat[stat.rfind(')')? + 1..]; + let fields: Vec<&str> = rest.split_whitespace().collect(); + let ppid = fields.get(1)?.parse().ok()?; + let start = fields.get(19)?.parse().ok()?; + Some((ppid, start)) +} + +/// The managed service that owns `pid`, if `pid` is, or descends from, one of +/// `roots`. `read_stat` returns the contents of `/proc//stat` for a PID. +/// +/// A root counts only when its recorded start time (if it has one) matches the +/// live process, so a recycled PID never claims a stranger's GPU memory. An +/// unreadable link ends the walk with no owner rather than a guess. +pub fn owner_of(pid: u32, roots: &[Root], read_stat: F) -> Option +where + F: Fn(u32) -> Option, +{ + let mut current = pid; + for _ in 0..MAX_DEPTH { + let (parent, start) = parse_stat(&read_stat(current)?)?; + if let Some(root) = roots + .iter() + .find(|r| r.pid == current && r.pid >= MIN_ROOT_PID) + && root.start_ticks.is_none_or(|recorded| recorded == start) + { + return Some(root.owner.clone()); + } + if parent == 0 || parent == current { + return None; + } + current = parent; + } + None +} + +/// [`owner_of`] against the live `/proc`. +#[cfg(target_os = "linux")] +#[must_use] +pub fn owner_of_pid(pid: u32, roots: &[Root]) -> Option { + owner_of(pid, roots, |p| { + std::fs::read_to_string(format!("/proc/{p}/stat")).ok() + }) +} + +/// Non-Linux stub: there is no `/proc`, so ancestry is never available. +#[cfg(not(target_os = "linux"))] +#[must_use] +pub const fn owner_of_pid(_pid: u32, _roots: &[Root]) -> Option { + None +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashMap; + + /// A `/proc//stat` line: `comm` is field 2, then state, ppid, and 17 + /// filler fields so the start time lands at field 22. + fn stat(pid: u32, comm: &str, ppid: u32, start: u64) -> String { + let filler = vec!["0"; 17].join(" "); + format!("{pid} ({comm}) S {ppid} {filler} {start} 0 0") + } + + fn table(entries: &[(u32, u32, u64)]) -> HashMap { + entries + .iter() + .map(|&(pid, ppid, start)| (pid, stat(pid, "proc", ppid, start))) + .collect() + } + + fn root(pid: u32, start: Option, owner: &str) -> Root { + Root { + pid, + start_ticks: start, + owner: owner.into(), + } + } + + #[test] + fn parse_stat_reads_ppid_and_start_time_even_when_the_name_is_awkward() { + assert_eq!(parse_stat(&stat(10, "vllm", 4, 777)), Some((4, 777))); + // Spaces and parentheses inside the process name. + assert_eq!( + parse_stat(&stat(10, "my (odd) name", 4, 777)), + Some((4, 777)) + ); + assert_eq!(parse_stat("garbage"), None); + assert_eq!(parse_stat("10 (short) S 4"), None); + } + + #[test] + fn a_worker_resolves_to_the_service_whose_engine_is_its_ancestor() { + // 100 (supervisor) -> 200 (engine) -> 300 (TP worker) -> 301 (helper) + let t = table(&[ + (100, 1, 5), + (200, 100, 6), + (300, 200, 7), + (301, 300, 8), + (1, 0, 1), + ]); + let roots = [root(200, Some(6), "svc-a")]; + let read = |p: u32| t.get(&p).cloned(); + assert_eq!(owner_of(301, &roots, read), Some("svc-a".into())); + assert_eq!(owner_of(300, &roots, read), Some("svc-a".into())); + // The root itself counts, and a process outside the tree does not. + assert_eq!(owner_of(200, &roots, read), Some("svc-a".into())); + assert_eq!(owner_of(100, &roots, read), None); + } + + #[test] + fn two_services_are_told_apart() { + let t = table(&[(10, 1, 1), (11, 10, 2), (20, 1, 3), (21, 20, 4), (1, 0, 1)]); + let roots = [root(10, None, "a"), root(20, None, "b")]; + let read = |p: u32| t.get(&p).cloned(); + assert_eq!(owner_of(11, &roots, read), Some("a".into())); + assert_eq!(owner_of(21, &roots, read), Some("b".into())); + } + + #[test] + fn a_recycled_pid_does_not_claim_a_stranger_s_children() { + // The recorded engine had start time 6, but pid 200 is now a different + // process (start 99) that happens to own a GPU worker. + let t = table(&[(200, 1, 99), (300, 200, 7), (1, 0, 1)]); + let roots = [root(200, Some(6), "svc-a")]; + assert_eq!(owner_of(300, &roots, |p| t.get(&p).cloned()), None); + // With no recorded start time the root is taken at its word. + let loose = [root(200, None, "svc-a")]; + assert_eq!( + owner_of(300, &loose, |p| t.get(&p).cloned()), + Some("svc-a".into()) + ); + } + + #[test] + fn a_root_at_init_or_zero_never_owns_anything() { + // Every process descends from init, so a root recorded as PID 1 would + // claim every GPU process on the host, whoever it belongs to. + let t = table(&[(300, 200, 7), (200, 1, 6), (1, 0, 1)]); + let read = |p: u32| t.get(&p).cloned(); + // PID 0 documents a state the walk never reaches (a parent of 0 ends it + // before `current` is assigned); PID 1 is the one the guard decides. + for pid in [0, 1] { + let roots = [root(pid, None, "svc")]; + assert_eq!(owner_of(300, &roots, read), None, "root pid {pid}"); + } + // A real root beside a bogus one still works. + let roots = [root(1, None, "bogus"), root(200, None, "real")]; + assert_eq!(owner_of(300, &roots, read), Some("real".into())); + // A process outside every service tree has no owner even with a bogus root. + let stray = table(&[(900, 1, 50), (1, 0, 1)]); + assert_eq!( + owner_of(900, &[root(1, None, "bogus")], |p| stray.get(&p).cloned()), + None + ); + } + + #[test] + fn owner_of_pid_is_total_on_every_platform() { + // A PID that cannot exist: unreadable on Linux, a stub elsewhere. Never + // a panic, never an owner. + assert_eq!(owner_of_pid(u32::MAX, &[]), None); + assert_eq!(owner_of_pid(u32::MAX, &[root(2, None, "svc")]), None); + } + + #[test] + fn unreadable_links_and_cycles_end_without_an_owner() { + let roots = [root(1000, None, "svc")]; + // The worker's parent has vanished. + let t = table(&[(300, 200, 7)]); + assert_eq!(owner_of(300, &roots, |p| t.get(&p).cloned()), None); + // A parent chain that loops must terminate. + let looped = table(&[(5, 6, 1), (6, 5, 2)]); + assert_eq!(owner_of(5, &roots, |p| looped.get(&p).cloned()), None); + // A process that is its own parent, and init. + let selfref = table(&[(7, 7, 1)]); + assert_eq!(owner_of(7, &roots, |p| selfref.get(&p).cloned()), None); + assert_eq!(owner_of(5, &[], |_| None), None); + } +} diff --git a/crates/rocm-dash-collectors/src/lib.rs b/crates/rocm-dash-collectors/src/lib.rs index b4c0047e02..f196f3d7f5 100644 --- a/crates/rocm-dash-collectors/src/lib.rs +++ b/crates/rocm-dash-collectors/src/lib.rs @@ -7,6 +7,7 @@ //! Every collector implements one of the traits in `rocm_dash_core::traits`. pub mod amd_smi; +pub mod ancestry; pub mod bench_load; pub mod bench_tail; pub mod cgroup; diff --git a/crates/rocm-dash-core/src/efficiency.rs b/crates/rocm-dash-core/src/efficiency.rs index f56e45b69e..ded34043c6 100644 --- a/crates/rocm-dash-core/src/efficiency.rs +++ b/crates/rocm-dash-core/src/efficiency.rs @@ -49,6 +49,13 @@ pub fn gpu_ids_overlap(gpu_ids: &[String], gpus: &[GpuMetrics]) -> bool { /// a GPU has no per-container reading the split on that GPU is equal, rather /// than guessing a skew from partial data. A sole occupant gets the whole GPU. /// +/// An occupant that is still starting and has no reading while another occupant +/// does is left out of the split. (A status of `Unknown` means no readiness +/// information, not "not serving", so it stays in.) It has not +/// allocated GPU memory, so the evidence says it is drawing next to nothing, and +/// an equal share would show the serving neighbour about half its real power and +/// roughly double its real tokens per watt until the newcomer loads. +/// /// Power on a GPU no instance occupies (idle, or a process outside the managed /// set) is attributed to nobody; it still counts toward node power. An entry is /// `None` when none of its GPUs matched a GPU reporting power (id join found @@ -76,6 +83,19 @@ pub fn attribute_power_w( if occupants.is_empty() { continue; } + let reading = |k: usize| { + per_container_vram_mb + .get(&instances[k].container_id) + .is_some_and(|&mb| mb > 0) + }; + let occupants: Vec = if occupants.iter().any(|&k| reading(k)) { + occupants + .into_iter() + .filter(|&k| reading(k) || !instances[k].status.is_starting()) + .collect() + } else { + occupants + }; let weights: Option> = occupants .iter() .map(|&k| { @@ -252,6 +272,7 @@ pub fn tokens_per_watt(gen_tps: Option, power_w: Option) -> Option GpuMetrics { GpuMetrics { @@ -568,6 +589,51 @@ mod tests { } } + #[test] + fn a_starting_occupant_with_no_reading_does_not_dilute_a_serving_neighbour() { + let gpus = [gpu("gpu-0", 400.0)]; + let mut serving = inst("a", &["0"]); + serving.status = InstanceStatus::Ready; + let mut loading = inst("b", &["0"]); + loading.status = InstanceStatus::Starting { phase: None }; + let insts = [serving, loading]; + // `a` has a VRAM reading; `b` is still loading and has none. + let p = attribute_power_w(&insts, &gpus, &vram(&[("a", 30_000)])); + assert_eq!(p, vec![Some(400.0), None], "the loader gets no share yet"); + // Once it has allocated memory it takes part in the split. + let p = attribute_power_w(&insts, &gpus, &vram(&[("a", 30_000), ("b", 10_000)])); + assert_eq!(p, vec![Some(300.0), Some(100.0)]); + // With no readings at all nothing is known, so the split stays equal. + let p = attribute_power_w(&insts, &gpus, &vram(&[])); + assert_eq!(p, vec![Some(200.0), Some(200.0)]); + } + + #[test] + fn a_starting_occupant_is_left_out_but_an_unmeasured_serving_one_still_forces_equal() { + let gpus = [gpu("gpu-0", 400.0)]; + let mut measured = inst("a", &["0"]); + measured.status = InstanceStatus::Ready; + let mut unmeasured = inst("b", &["0"]); + unmeasured.status = InstanceStatus::Ready; + let mut loading = inst("c", &["0"]); + loading.status = InstanceStatus::Starting { phase: None }; + let insts = [measured, unmeasured, loading]; + // Only `a` has a reading. `c` is starting with none, so it is dropped; + // `b` is serving but unmeasured (e.g. a missed /proc join), so leaving + // it out would credit `a` with the whole board: the rest stay equal. + let p = attribute_power_w(&insts, &gpus, &vram(&[("a", 30_000)])); + assert_eq!(p, vec![Some(200.0), Some(200.0), None]); + // Without the starting occupant in the picture the result is the same + // for the other two, so the starting one is what changed the split. + let p2 = attribute_power_w(&insts[..2], &gpus, &vram(&[("a", 30_000)])); + assert_eq!(p2, vec![Some(200.0), Some(200.0)]); + // An `Unknown` status is not "starting": it stays in the split. + let mut unknown = inst("d", &["0"]); + unknown.status = InstanceStatus::Unknown; + let p3 = attribute_power_w(&[insts[0].clone(), unknown], &gpus, &vram(&[("a", 30_000)])); + assert_eq!(p3, vec![Some(200.0), Some(200.0)]); + } + #[test] fn multi_gpu_instance_vram_is_spread_across_its_gpus() { // `a` spans gpu 0+1 (40 GB total -> 20 GB each); `b` is alone on gpu 1 diff --git a/crates/rocm-dash-daemon/src/registry.rs b/crates/rocm-dash-daemon/src/registry.rs index 1bb31a4512..cf39320c33 100644 --- a/crates/rocm-dash-daemon/src/registry.rs +++ b/crates/rocm-dash-daemon/src/registry.rs @@ -57,9 +57,10 @@ use serde::Deserialize; /// Minimal read-only view of a rocm-cli `ManagedServiceRecord` on disk. /// -/// Mirrors the stable subset the scrape seam needs; unknown fields (manifest paths, -/// pids, recipe json, …) are ignored. Every field defaults so partial/older -/// records still parse. +/// Mirrors the stable subset the scrape seam needs; the engine and supervisor +/// pids (with their start times) are kept to match GPU processes by ancestry; +/// unknown fields (manifest paths, recipe json, …) are ignored. Every field +/// defaults so partial/older records still parse. #[derive(Debug, Clone, Deserialize)] pub struct ServiceRecord { #[serde(default)] @@ -86,10 +87,46 @@ pub struct ServiceRecord { /// joins it to GPU power and VRAM. #[serde(default)] pub gpu_indices: Vec, + /// The launcher process rocm-cli recorded for the service. + #[serde(default)] + pub supervisor_pid: u32, + /// Kernel start time of the supervisor, which tells it from a recycled PID. + #[serde(default)] + pub supervisor_start_ticks: Option, + /// The engine server process, once the supervisor has seen it. + #[serde(default)] + pub engine_pid: Option, + /// Kernel start time of the engine server process. + #[serde(default)] + pub engine_start_ticks: Option, #[serde(default)] pub created_at_unix_ms: u128, } +impl ServiceRecord { + /// The recorded processes whose descendants are this service's GPU + /// processes: the engine server and its launcher. Either can be absent on + /// an old or half-written record, and a PID that is zero (not recorded) or + /// 1 (init, an ancestor of everything) is never a root, since it would claim + /// processes that belong to no service. + pub fn ancestry_roots(&self) -> Vec { + let root = |pid: u32, start_ticks| rocm_dash_collectors::ancestry::Root { + pid, + start_ticks, + owner: self.service_id.clone(), + }; + let usable = |pid: u32| pid >= rocm_dash_collectors::ancestry::MIN_ROOT_PID; + let mut roots = Vec::new(); + if let Some(pid) = self.engine_pid.filter(|p| usable(*p)) { + roots.push(root(pid, self.engine_start_ticks)); + } + if usable(self.supervisor_pid) { + roots.push(root(self.supervisor_pid, self.supervisor_start_ticks)); + } + roots + } +} + /// Service statuses worth scraping. /// /// Matches the live set the rocm-cli supervisor overlays onto a record @@ -349,6 +386,13 @@ pub struct ManagedDiscovery { /// A live record with no bound port is NOT counted: it is skipped for a /// different reason and is not a past attempt. pub past_attempts: usize, + /// Ancestry roots of every live service surfaced, so a GPU process can be + /// attributed to the managed service it belongs to. + pub roots: Vec, + /// Ids of records that were read in this pass and found no longer running. + /// A partial read keeps what it did not see; it must still drop the roots + /// of a service it saw stop, or a dead service's PID would stay matchable. + pub retired: HashSet, } /// Convert the loaded registry records into a [`ManagedDiscovery`] — the pure core of the daemon's managed-service discovery tick. @@ -368,10 +412,12 @@ pub fn discover_managed_services(records: &[ServiceRecord]) -> ManagedDiscovery // live record with no bound port. Only the first is a past attempt. if !is_scrapeable_status(&record.status) { out.past_attempts += 1; + out.retired.insert(record.service_id.clone()); } continue; }; out.seen.insert(svc.container_id.clone()); + out.roots.extend(record.ancestry_roots()); if engine_kind_for(record) != Some(EngineKind::Vllm) { out.non_vllm.insert(svc.container_id.clone()); } @@ -784,6 +830,104 @@ mod tests { } } + #[test] + fn ancestry_roots_come_from_the_recorded_engine_and_supervisor() { + let with = record_json("svc", "vllm", 8000, "ready", 1).replace( + r#""supervisor_pid": 4242,"#, + r#""supervisor_pid": 4242, "supervisor_start_ticks": 11, "engine_pid": 4300, "engine_start_ticks": 22,"#, + ); + let rec: ServiceRecord = serde_json::from_str(&with).unwrap(); + let roots = rec.ancestry_roots(); + let got: Vec<_> = roots.iter().map(|r| (r.pid, r.start_ticks)).collect(); + assert_eq!(got, vec![(4300, Some(22)), (4242, Some(11))]); + assert!(roots.iter().all(|r| r.owner == "svc")); + // An old record: no engine yet, no start times. The supervisor alone. + let old: ServiceRecord = + serde_json::from_str(&record_json("svc", "vllm", 8000, "ready", 1)).unwrap(); + let got: Vec<_> = old + .ancestry_roots() + .iter() + .map(|r| (r.pid, r.start_ticks)) + .collect(); + assert_eq!(got, vec![(4242, None)]); + // Zero is "not recorded" and 1 is init: neither is ever a root, or it + // would own every process on the host. + for bad in [0, 1] { + let json = format!( + r#"{{"service_id":"z","engine":"vllm","port":1,"status":"ready","supervisor_pid":{bad},"engine_pid":{bad}}}"# + ); + let rec: ServiceRecord = serde_json::from_str(&json).unwrap(); + assert!( + rec.ancestry_roots().is_empty(), + "pid {bad} must not be a root" + ); + } + } + + #[test] + fn recorded_pids_survive_the_producer_to_daemon_serde_handoff() { + // Same contract as the startup-phase test above, for the pid fields the + // ancestry join reads: serialize the producer's real type and read it + // back as ours, so a rename or retype on either side fails here. + use rocm_core::{AppPaths, ManagedServiceRecord}; + + let root = PathBuf::from("/tmp/rocm-xcrate-pids"); + let paths = AppPaths { + config_dir: root.join("config"), + data_dir: root.join("data"), + cache_dir: root.join("cache"), + }; + let mut produced = ManagedServiceRecord::new( + &paths, + "svc-pids", + "vllm", + "M/pids", + "M/pids", + "127.0.0.1", + 8000, + "serve", + 4242, + None, + None, + None, + ); + produced.status = "ready".to_owned(); + produced.supervisor_start_ticks = Some(111); + produced.engine_pid = Some(4300); + produced.engine_start_ticks = Some(222); + + let json = serde_json::to_string(&produced).expect("producer record serializes"); + let consumed: ServiceRecord = + serde_json::from_str(&json).expect("daemon record deserializes the producer's JSON"); + let got: Vec<_> = consumed + .ancestry_roots() + .iter() + .map(|r| (r.pid, r.start_ticks, r.owner.clone())) + .collect(); + assert_eq!( + got, + vec![ + (4300, Some(222), "svc-pids".to_string()), + (4242, Some(111), "svc-pids".to_string()), + ], + "engine and supervisor pids and start times must cross the serde handoff by name" + ); + } + + #[test] + fn discover_managed_services_collects_roots_only_for_live_services() { + let live = record_json("live", "vllm", 8000, "ready", 1); + let dead = record_json("dead", "vllm", 8001, "stopped", 1).replace("4242", "5252"); + let recs: Vec = [live, dead] + .iter() + .map(|j| serde_json::from_str(j).unwrap()) + .collect(); + let disc = discover_managed_services(&recs); + let owners: Vec<_> = disc.roots.iter().map(|r| r.owner.as_str()).collect(); + assert_eq!(owners, vec!["live"], "a stopped service owns nothing"); + assert!(disc.retired.contains("dead") && !disc.retired.contains("live")); + } + #[test] fn discovered_from_record_carries_gpu_indices_as_gpu_ids() { // The recorded launch GPUs are what join an instance to GPU power and diff --git a/crates/rocm-dash-daemon/src/runner.rs b/crates/rocm-dash-daemon/src/runner.rs index 4af9201368..855c109742 100644 --- a/crates/rocm-dash-daemon/src/runner.rs +++ b/crates/rocm-dash-daemon/src/runner.rs @@ -610,6 +610,7 @@ pub async fn run_loop( Ok(read) => { let disc = crate::registry::discover_managed_services(&read.records); discovery.managed_non_vllm = disc.non_vllm; + discovery.managed_roots = disc.roots; reconcile.reconcile_set( DiscoverySource::Managed, &disc.svcs, @@ -805,10 +806,18 @@ pub async fn run_loop( match g.processes().await { Ok(procs) => { procs_nonempty = !procs.is_empty(); - per_container_used = rocm_dash_core::vram::aggregate_process_vram( - &procs, - rocm_dash_collectors::cgroup::container_id_for_pid, - ); + // A container id when the process is in a docker/containerd + // cgroup, otherwise the managed service whose engine it + // descends from. Both key the same map the instances read. + per_container_used = + rocm_dash_core::vram::aggregate_process_vram(&procs, |pid| { + rocm_dash_collectors::cgroup::container_id_for_pid(pid).or_else(|| { + rocm_dash_collectors::ancestry::owner_of_pid( + pid, + &discovery.managed_roots, + ) + }) + }); } Err(e) => { // Keep the last-known map; device fallback covers the gap. @@ -882,11 +891,17 @@ pub async fn run_loop( // Warn only when a real process scrape happened but an instance with // live GPUs still fell back to device-summed (cgroup join missed). if procs_nonempty { - let vram_fallbacks = count_vram_join_misses(&instances, &gpus, &per_container_used); + let vram_fallbacks = count_vram_join_misses( + &instances, + &gpus, + &per_container_used, + &discovery.managed_roots, + &discovery.managed_non_vllm, + ); if vram_fallbacks > 0 { scrape_warns.push(format!( "vram attribution: {vram_fallbacks} instance(s) fell back to device-summed \ - VRAM (no per-process cgroup match; check /proc access)" + VRAM (no per-process cgroup or ancestry match; check /proc access)" )); } } @@ -934,20 +949,32 @@ pub async fn run_loop( /// Instances that *could* have joined the per-process VRAM map but did not. /// -/// The map is keyed by docker/containerd ids read from `/proc//cgroup`, -/// so only an instance whose id has that shape can ever hit it. A managed -/// `rocm serve` service (named by its service id) falls back to device-summed -/// VRAM by design, and counting it would raise a false "check /proc access" -/// warning on every healthy managed instance. +/// Two kinds of instance can hit the map: one whose id is a docker/containerd +/// id read from `/proc//cgroup`, and a managed `rocm serve` service that +/// has recorded `roots` (its engine/supervisor PIDs) to match by ancestry. A +/// managed service with no roots cannot join and falls back to device-summed +/// VRAM by design, so it is not counted. One that is not yet `Ready`/`Running` +/// may still be loading its model with no GPU process to attribute, so it is +/// not counted either; a serving one always holds VRAM, so a missing entry +/// means its ancestry walk failed (unreadable `/proc`, a recycled root PID). +/// A non-vLLM managed service (e.g. Lemonade on a backend `amd-smi` does not +/// list) is not counted: it may hold GPU memory without ever appearing in the +/// process list, which is not an access problem. fn count_vram_join_misses( instances: &[Instance], gpus: &[GpuMetrics], per_container_used: &HashMap, + roots: &[rocm_dash_collectors::ancestry::Root], + non_vllm: &HashSet, ) -> usize { instances .iter() .filter(|i| { - rocm_dash_collectors::cgroup::is_container_id(&i.container_id) + let can_join = rocm_dash_collectors::cgroup::is_container_id(&i.container_id) + || (matches!(i.status, InstanceStatus::Ready | InstanceStatus::Running) + && !non_vllm.contains(&i.container_id) + && roots.iter().any(|r| r.owner == i.container_id)); + can_join && !i.gpu_ids.is_empty() && rocm_dash_core::efficiency::gpu_ids_overlap(&i.gpu_ids, gpus) && !per_container_used.contains_key(&i.container_id) @@ -1048,6 +1075,10 @@ struct DiscoveryState { /// The subset of the registry ids whose engine is NOT vLLM, excluded from /// the vLLM Prometheus scrape so they aren't mis-parsed. managed_non_vllm: HashSet, + /// Ancestry roots of the managed services the registry surfaced, used to + /// attribute GPU processes to a service when the cgroup join cannot (a + /// managed service is not a container). + managed_roots: Vec, /// The id of the currently-live Lemonade instance, if any. lemonade_id: Option, /// How the registry reads have been going, so a condition that lasts many @@ -1064,6 +1095,13 @@ impl DiscoveryState { fn keep_through_partial_registry(&mut self, disc: crate::registry::ManagedDiscovery) { self.managed_non_vllm.retain(|id| !disc.seen.contains(id)); self.managed_non_vllm.extend(disc.non_vllm); + // Roots of the services this pass saw are replaced; the unseen ones + // keep theirs, as their services stay known. + // A service this pass read and found stopped loses its roots too, or its + // PID would stay matchable for as long as an unrelated record is broken. + self.managed_roots + .retain(|r| !disc.seen.contains(&r.owner) && !disc.retired.contains(&r.owner)); + self.managed_roots.extend(disc.roots); self.known_services.extend(disc.seen); } } @@ -1641,20 +1679,95 @@ mod tests { } #[test] - fn managed_instances_never_count_as_vram_join_misses() { + fn only_instances_that_can_join_count_as_vram_join_misses() { + use rocm_dash_collectors::ancestry::Root; let docker_id = "ab".repeat(32); + let rooted = "vllm-qwen-qwen2-5-0-5b-instru-1793361705247"; + let unrooted = "vllm-other-1793361705248"; let gpus = [gpu("gpu-0", 1000, 8000)]; - let insts = [ - inst_on("vllm-qwen-qwen2-5-0-5b-instru-1793361705247", "0"), - inst_on(&docker_id, "0"), - ]; - // A docker instance with no /proc match is a real miss; the managed - // service, which could never match, is not. + let serving = |id: &str| Instance { + status: InstanceStatus::Ready, + ..inst_on(id, "0") + }; + let insts = [serving(rooted), serving(unrooted), inst_on(&docker_id, "0")]; + let roots = [Root { + pid: 10, + start_ticks: None, + owner: rooted.into(), + }]; + // The docker instance and the rooted managed service could both have + // joined and did not; the managed service with no roots never could. let empty: HashMap = HashMap::new(); - assert_eq!(count_vram_join_misses(&insts, &gpus, &empty), 1); - // Once the docker instance joins, nothing is missed. - let joined: HashMap = [(docker_id, 512)].into(); - assert_eq!(count_vram_join_misses(&insts, &gpus, &joined), 0); + assert_eq!( + count_vram_join_misses(&insts, &gpus, &empty, &roots, &HashSet::new()), + 2 + ); + // With no roots recorded, only the docker instance is a miss. + assert_eq!( + count_vram_join_misses(&insts, &gpus, &empty, &[], &HashSet::new()), + 1 + ); + // Once both joined, nothing is missed. + let joined: HashMap = [(docker_id, 512), (rooted.to_string(), 256)].into(); + assert_eq!( + count_vram_join_misses(&insts, &gpus, &joined, &roots, &HashSet::new()), + 0 + ); + } + + #[test] + fn a_rooted_non_vllm_service_is_not_a_vram_join_miss() { + use rocm_dash_collectors::ancestry::Root; + let id = "lemonade-gguf-1793361705247"; + let gpus = [gpu("gpu-0", 1000, 8000)]; + let serving = Instance { + status: InstanceStatus::Ready, + ..inst_on(id, "0") + }; + let roots = [Root { + pid: 10, + start_ticks: None, + owner: id.into(), + }]; + let empty: HashMap = HashMap::new(); + // As a vLLM service it would be a miss; marked non-vLLM it is not, since + // its backend may hold GPU memory without `amd-smi` listing the process. + assert_eq!( + count_vram_join_misses( + std::slice::from_ref(&serving), + &gpus, + &empty, + &roots, + &HashSet::new() + ), + 1 + ); + let non_vllm: HashSet = [id.to_string()].into(); + assert_eq!( + count_vram_join_misses(&[serving], &gpus, &empty, &roots, &non_vllm), + 0 + ); + } + + #[test] + fn a_rooted_service_still_loading_is_not_a_vram_join_miss() { + use rocm_dash_collectors::ancestry::Root; + let id = "vllm-qwen-1793361705247"; + let gpus = [gpu("gpu-0", 1000, 8000)]; + let loading = Instance { + status: InstanceStatus::Starting { phase: None }, + ..inst_on(id, "0") + }; + let roots = [Root { + pid: 10, + start_ticks: None, + owner: id.into(), + }]; + let empty: HashMap = HashMap::new(); + assert_eq!( + count_vram_join_misses(&[loading], &gpus, &empty, &roots, &HashSet::new()), + 0 + ); } #[test] @@ -2487,6 +2600,85 @@ mod tests { ); } + #[test] + fn partial_registry_read_refreshes_the_roots_it_saw_and_keeps_the_unseen() { + use rocm_dash_collectors::ancestry::Root; + let root = |pid, owner: &str| Root { + pid, + start_ticks: None, + owner: owner.into(), + }; + let mut state = DiscoveryState { + known_services: id_set(&["seen", "unseen"]), + managed_roots: vec![root(10, "seen"), root(20, "unseen")], + ..DiscoveryState::default() + }; + + // `seen` was relaunched under a new engine pid; `unseen` was not read. + state.keep_through_partial_registry(crate::registry::ManagedDiscovery { + seen: id_set(&["seen"]), + roots: vec![root(11, "seen")], + ..Default::default() + }); + + let mut got: Vec<_> = state + .managed_roots + .iter() + .map(|r| (r.pid, r.owner.as_str())) + .collect(); + got.sort_unstable(); + assert_eq!( + got, + vec![(11, "seen"), (20, "unseen")], + "a seen service's roots are replaced, an unseen one's survive" + ); + } + + #[test] + fn partial_registry_read_drops_the_roots_of_a_service_it_saw_stop() { + use rocm_dash_collectors::ancestry::Root; + let root = |pid, owner: &str| Root { + pid, + start_ticks: None, + owner: owner.into(), + }; + let mut state = DiscoveryState { + managed_roots: vec![root(10, "stopped"), root(20, "unseen")], + ..DiscoveryState::default() + }; + // `stopped` was read this pass and is no longer running; an unrelated + // record failed to read, which is what makes this a partial pass. + state.keep_through_partial_registry(crate::registry::ManagedDiscovery { + retired: id_set(&["stopped"]), + ..Default::default() + }); + let owners: Vec<_> = state + .managed_roots + .iter() + .map(|r| r.owner.as_str()) + .collect(); + assert_eq!( + owners, + vec!["unseen"], + "a stopped service's PID must not stay matchable" + ); + } + + #[test] + fn partial_registry_read_that_saw_nothing_keeps_every_root() { + use rocm_dash_collectors::ancestry::Root; + let mut state = DiscoveryState { + managed_roots: vec![Root { + pid: 10, + start_ticks: Some(5), + owner: "a".into(), + }], + ..DiscoveryState::default() + }; + state.keep_through_partial_registry(crate::registry::ManagedDiscovery::default()); + assert_eq!(state.managed_roots.len(), 1); + } + #[test] fn partial_registry_read_that_saw_nothing_changes_nothing() { let mut state = DiscoveryState { diff --git a/crates/rocm-dash-daemon/tests/telemetry_contract.rs b/crates/rocm-dash-daemon/tests/telemetry_contract.rs index c15589325d..aae7b5a41c 100644 --- a/crates/rocm-dash-daemon/tests/telemetry_contract.rs +++ b/crates/rocm-dash-daemon/tests/telemetry_contract.rs @@ -796,21 +796,158 @@ esac .unwrap_or_else(|e| panic!("limit and attributed power never appeared: {e}")); assert!((limit - 750.0).abs() < f32::EPSILON, "limit was {limit}"); - // Neither service joins the per-container VRAM map (managed services can - // not), so the board is split equally and the two shares add back up to + // Neither service joins the per-container VRAM map (these records carry no + // pids, so there is no ancestry root to match), so the board is split equally and the two shares add back up to // what the GPU actually draws. assert!((a - 200.0).abs() < 1e-6, "shared-a got {a} W"); assert!((b - 200.0).abs() < 1e-6, "shared-b got {b} W"); // amd-smi lists a GPU process, so the VRAM scrape is "real", but that - // process is in no container the cgroup join can name. Managed services - // can never join that map by design, so this must not read as a /proc - // problem. + // process is in no container the cgroup join can name, and these services + // have no ancestry roots, so they could not join by design and this must + // not read as a /proc problem. assert!( !warnings.iter().any(|w| w.starts_with("vram attribution")), "false vram-attribution warning for healthy managed services: {warnings:?}" ); } +// ── Managed-service VRAM attribution by process ancestry (ROCMAI-941) ─────── + +/// A real child of `parent`, found through `/proc`, once the shell has forked it. +#[cfg(target_os = "linux")] +fn first_child_of(parent: u32) -> u32 { + for _ in 0..100 { + if let Ok(kids) = std::fs::read_to_string(format!("/proc/{parent}/task/{parent}/children")) + && let Some(pid) = kids.split_whitespace().next().and_then(|p| p.parse().ok()) + { + return pid; + } + std::thread::sleep(Duration::from_millis(50)); + } + panic!("process {parent} never forked a child"); +} + +/// Two fake "engine" processes, killed on drop, so a panic while waiting for a +/// fork (or in any assert) does not leave the shells and their sleeping +/// children behind. +#[cfg(target_os = "linux")] +struct Engines([std::process::Child; 2]); + +#[cfg(target_os = "linux")] +impl Drop for Engines { + fn drop(&mut self) { + for e in &mut self.0 { + let _ = e.kill(); + let _ = e.wait(); + } + } +} + +/// Contract: GPU processes that `amd-smi` lists are attributed to the managed +/// service they descend from, by walking parent links to the PID rocm-cli +/// recorded for it. Two services share one 400 W GPU; each gets its own VRAM +/// (not the whole device) and a power share weighted by it, and the dashboard +/// raises no "vram attribution" warning for them. +/// +/// Uses real processes: each "engine" is a shell with a real child standing in +/// for a GPU worker, so the `/proc//stat` walk is the production one. +#[cfg(target_os = "linux")] +#[tokio::test] +async fn managed_services_get_their_own_vram_and_a_weighted_power_split() { + use rocm_core::fake_executable::write_fake_executable; + use std::process::{Command, Stdio}; + + let tmp = tempfile::TempDir::new().unwrap(); + let services_dir = tmp.path().join("services"); + std::fs::create_dir_all(&services_dir).unwrap(); + + // `; true` stops the shell exec-ing straight into `sleep`, so it really forks. + let spawn = |script: &str| { + Command::new("sh") + .args(["-c", script]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .expect("spawn fake process") + }; + // own-a: the recorded engine forks the worker directly. own-b: only the + // supervisor is recorded (as before the supervisor has seen its engine) and + // the worker sits two hops down, behind an unregistered intermediate shell. + let engines = Engines([ + spawn("sleep 120; true"), + spawn("sh -c 'sleep 120; true'; true"), + ]); + let workers: Vec = vec![ + first_child_of(engines.0[0].id()), + first_child_of(first_child_of(engines.0[1].id())), + ]; + + for (id, pids) in [ + ("own-a", format!(r#""engine_pid":{}"#, engines.0[0].id())), + ( + "own-b", + format!(r#""supervisor_pid":{}"#, engines.0[1].id()), + ), + ] { + let json = format!( + r#"{{"service_id":"{id}","engine":"vllm","model_ref":"M/{id}","canonical_model_id":"M/{id}","host":"127.0.0.1","port":9,"status":"ready","gpu_indices":[0],{pids},"created_at_unix_ms":0}}"# + ); + std::fs::write(services_dir.join(format!("{id}.json")), json).unwrap(); + } + + let fake = tmp.path().join("amd-smi-fake"); + write_fake_executable( + &fake, + format!( + r#"#!/bin/sh +case "$1" in + metric) echo '{{"gpu_data":[{{"gpu":0,"mem_usage":{{"used_vram":{{"value":4000}},"total_vram":{{"value":8000}}}},"usage":{{"gfx_activity":{{"value":50}}}},"temperature":{{"hotspot":{{"value":60}}}},"power":{{"socket_power":{{"value":400.0,"unit":"W"}}}}}}]}}' ;; + process) echo '{{"gpu_data":[{{"gpu":0,"process_list":[{{"process_info":{{"pid":{a},"memory_usage":{{"vram_mem":{{"value":3000,"unit":"MB"}}}}}}}},{{"process_info":{{"pid":{b},"memory_usage":{{"vram_mem":{{"value":1000,"unit":"MB"}}}}}}}}]}}]}}' ;; + *) echo '{{}}' ;; +esac +"#, + a = workers[0], + b = workers[1] + ), + ) + .unwrap(); + + let (tx, mut rx) = broadcast::channel::(64); + let opts = RunnerOptions { + amd_smi_binary: Some(fake.into_os_string()), + amd_smi_skip_device_preflight: true, + ..fast_opts(services_dir) + }; + let _runner = spawn_runner(tx, opts).await; + + let seen = wait_for_snapshot(&mut rx, |snap| { + let get = |id: &str| snap.instances.iter().find(|i| i.container_id == id); + let (a, b) = (get("own-a")?, get("own-b")?); + Some(( + a.vram_used_mb, + b.vram_used_mb, + a.power_w?, + b.power_w?, + snap.warnings.clone(), + )) + }) + .await; + + drop(engines); + let (a_vram, b_vram, a_w, b_w, warnings) = + seen.unwrap_or_else(|e| panic!("per-service VRAM and power never appeared: {e}")); + + // Each service shows what its own GPU process holds, not the device total. + assert_eq!((a_vram, b_vram), (3000, 1000), "VRAM must be per service"); + // Power follows that VRAM split of the 400 W board. + assert!((a_w - 300.0).abs() < 1e-6, "own-a got {a_w} W"); + assert!((b_w - 100.0).abs() < 1e-6, "own-b got {b_w} W"); + assert!( + !warnings.iter().any(|w| w.starts_with("vram attribution")), + "unexpected vram-attribution warning: {warnings:?}" + ); +} + // ── Scenario 5: Omitted counter → None (not zero, not panic) ─────────────── /// Contract: when the `/metrics` endpoint returns HTTP 200 but the body omits diff --git a/docs/architecture.md b/docs/architecture.md index 4244c357c1..d64e70cebe 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -395,7 +395,7 @@ The pure layer under the dashboard: telemetry, benchmark and protocol types, the | [`bench_schema.rs`](../crates/rocm-dash-core/src/bench_schema.rs) | `BenchmarkRow`, the one row type for a benchmark result, and `PassFail`, vendored from the instinct-agent-bench schema. | flat | | [`chat_template.rs`](../crates/rocm-dash-core/src/chat_template.rs) | The chat-template defaults a locally served model needs on an OpenAI-compatible chat request (`local_model_should_disable_thinking`). | flat | | [`config.rs`](../crates/rocm-dash-core/src/config.rs) | The standalone dashboard configuration in `rocm-dash/config.toml` under the config directory (daemon, TUI and per-engine sections) and the default daemon socket path. | flat | -| [`efficiency.rs`](../crates/rocm-dash-core/src/efficiency.rs) | The pure join of throughput and power: GPU-id normalisation, board power attributed to the instances on a GPU by VRAM share, and tokens per watt. `node_throughput` sums the node's generation throughput once and works out its freshness (fresh, held, or unknown), ignoring instances at zero tok/s when judging freshness; the Observe and Home throughput heroes call it directly. `node_power_w` sums board power over the GPUs that report a usable reading, skipping one that is not finite or not positive as `attribute_power_w` does, and is what the Observe, Hardware and Home power figures show and divide by. `node_efficiency` combines the two into the whole-node `NodeEfficiency` (Σ `gen_tps` over that power, including GPUs no instance occupies) and is the one implementation behind the Observe efficiency hero and its trend, the Hardware header, and the assistant's `tokens_per_watt` and `list_instances` tools, which reach `node_throughput` through it (the Home tab's tokens-per-watt figure is a separate sum of per-instance values). | flat | +| [`efficiency.rs`](../crates/rocm-dash-core/src/efficiency.rs) | The pure join of throughput and power: GPU-id normalisation, board power attributed to the instances on a GPU by VRAM share (an occupant that is still `Starting` with no VRAM reading, while another occupant has one, is left out of the split so it does not halve a serving neighbour's power), and tokens per watt. `node_throughput` sums the node's generation throughput once and works out its freshness (fresh, held, or unknown), ignoring instances at zero tok/s when judging freshness; the Observe and Home throughput heroes call it directly. `node_power_w` sums board power over the GPUs that report a usable reading, skipping one that is not finite or not positive as `attribute_power_w` does, and is what the Observe, Hardware and Home power figures show and divide by. `node_efficiency` combines the two into the whole-node `NodeEfficiency` (Σ `gen_tps` over that power, including GPUs no instance occupies) and is the one implementation behind the Observe efficiency hero and its trend, the Hardware header, and the assistant's `tokens_per_watt` and `list_instances` tools, which reach `node_throughput` through it (the Home tab's tokens-per-watt figure is a separate sum of per-instance values). | flat | | [`metrics.rs`](../crates/rocm-dash-core/src/metrics.rs) | The live metric types the dashboard crates share: `Snapshot`, `GpuMetrics`, `Instance` with its `InstanceStatus` predicates and labels, and the observation-freshness metadata. | flat | | [`observation.rs`](../crates/rocm-dash-core/src/observation.rs) | Turns a generation-token counter into an honest rate: a bounded sample window, fresh-or-held reporting with expiry, and directly reported rates, with all time passed in. | flat | | [`partition.rs`](../crates/rocm-dash-core/src/partition.rs) | The GPU compute-partition and memory-partition mode enums. | flat | @@ -426,9 +426,10 @@ The I/O-heavy collector implementations behind `rocm-dash-core`'s traits (amd-sm | --- | --- | --- | | [`lib.rs`](../crates/rocm-dash-collectors/src/lib.rs) | Crate root: declares the public collector modules. | flat | | [`amd_smi.rs`](../crates/rocm-dash-collectors/src/amd_smi.rs) | The amd-smi collector: runs `amd-smi ... --json` under a timeout after the GPU-device pre-flight and parses metrics, power limits, processes and system info tolerantly. | flat | +| [`ancestry.rs`](../crates/rocm-dash-collectors/src/ancestry.rs) | Attributes GPU processes to managed `rocm serve` services. A service record carries the PIDs of its engine and launcher and their kernel start times; `owner_of` walks a GPU process's `/proc//stat` parent links (bounded depth, reader injected so it is testable anywhere) to a recorded root and returns that service's id. A root whose live start time differs from the recorded one is a recycled PID and claims nothing, PID 0 and PID 1 are never roots (init is an ancestor of every process), and an unreadable link ends the walk with no owner. `owner_of_pid` reads the live `/proc` (Linux only; a stub returns `None` elsewhere). The runner tries the cgroup join first and this second, so container ids and service ids key the one per-container VRAM map. | flat | | [`bench_load.rs`](../crates/rocm-dash-collectors/src/bench_load.rs) | The concurrency-sweep load generator behind `rocm bench load`: one aggregate `BenchmarkRow` per concurrency cell, appended to a CSV, with an optional auto-ramp. | flat | | [`bench_tail.rs`](../crates/rocm-dash-collectors/src/bench_tail.rs) | `CsvBenchTailer`: re-reads a benchmark CSV on each drain and yields the rows it has not returned yet. | flat | -| [`cgroup.rs`](../crates/rocm-dash-collectors/src/cgroup.rs) | Attributes GPU processes to Docker containers by reading the 64-hex container id from `/proc//cgroup`. | flat | +| [`cgroup.rs`](../crates/rocm-dash-collectors/src/cgroup.rs) | Attributes GPU processes to Docker containers by reading the 64-hex container id from `/proc//cgroup`; `is_container_id` tests that shape, which a managed `rocm serve` service id never has, so those are attributed by `ancestry.rs`. | flat | | [`docker.rs`](../crates/rocm-dash-collectors/src/docker.rs) | `DockerDiscovery`: finds vLLM containers through `bollard` and builds a `DiscoveredService` from each container's environment and command line. | flat | | [`engine_registry.rs`](../crates/rocm-dash-collectors/src/engine_registry.rs) | `EngineKind`: maps a serving engine (vLLM, Lemonade) to its label, conventional port and sample parser. | flat | | [`host.rs`](../crates/rocm-dash-collectors/src/host.rs) | `HostCollector`: CPU, memory and swap from `sysinfo`, sampled once per tick into `SystemMetrics`. Disk and network rates are not collected yet and stay 0. | flat | @@ -455,7 +456,7 @@ The daemon library: [`runner.rs`](../crates/rocm-dash-daemon/src/runner.rs) driv | [`bench_ring.rs`](../crates/rocm-dash-daemon/src/bench_ring.rs) | `BenchRing`, the bounded benchmark-row history a late-joining client is sent when it subscribes. | flat | | [`demo.rs`](../crates/rocm-dash-daemon/src/demo.rs) | The synthetic-session generator behind `rocm dash --demo`: a deterministic recording of an 8-GPU node serving three vLLM containers, written as `PersistedEntry` lines. | flat | | [`persist.rs`](../crates/rocm-dash-daemon/src/persist.rs) | `SessionWriter`: appends every broadcast `Event` as a `PersistedEntry` line to one `session-*.ndjson` file per daemon session. | flat | -| [`registry.rs`](../crates/rocm-dash-daemon/src/registry.rs) | Reads the managed-service records `rocm serve` writes and turns the live ones into `DiscoveredService` scrape targets, telling a registry read in full, an empty or absent one included, from one that could not be read in full. | flat | +| [`registry.rs`](../crates/rocm-dash-daemon/src/registry.rs) | Reads the managed-service records `rocm serve` writes and turns the live ones into `DiscoveredService` scrape targets, telling a registry read in full, an empty or absent one included, from one that could not be read in full. It also keeps each live service's recorded engine and supervisor PIDs (with their start times) and derives the ancestry roots `ancestry.rs` matches GPU processes against, dropping the roots of a service it read and found stopped even on a partial pass. | flat | | [`runner.rs`](../crates/rocm-dash-daemon/src/runner.rs) | The owner task: drives the collectors on their tick cadences, reconciles discovered services, derives rates and efficiency, and broadcasts `Snapshot`s. | flat | | [`server.rs`](../crates/rocm-dash-daemon/src/server.rs) | The listener, the per-client task loop and the broadcast from the runner to every client, including the history replay on subscribe. | flat | | [`snapshot_ring.rs`](../crates/rocm-dash-daemon/src/snapshot_ring.rs) | `SnapshotRing`, the bounded snapshot history a late-joining client is sent when it subscribes. | flat |