Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 2 additions & 51 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,10 @@ syn = { version = "2", default-features = false, features = ["full", "parsing",
# acceptable cost for a preflight check. Unlisted mounts are detected by the
# device-ID cross-check in `rocm-core::disk_space` and reported as unknown,
# which never blocks an install.
sysinfo = { version = "0.39", features = ["linux-tmpfs"] }
# Default features are off because 0.40's `gpu` feature calls `statx` directly,
# which glibc 2.17 (the manylinux2014 release build) does not provide, so the
# link fails. Only CPU, memory and disk figures are used.
sysinfo = { version = "0.40", default-features = false, features = ["system", "disk", "linux-tmpfs"] }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default-features = false also drops sysinfo's multithread (rayon) default, so CPU refresh becomes single-threaded. Fine for this use; just confirming it is intended.

tokio = { version = "1.48", features = ["macros", "net", "rt-multi-thread", "signal", "sync", "time"] }

[workspace.lints.rust]
Expand Down
7 changes: 1 addition & 6 deletions MANIFEST.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,6 @@ repository.
| directories | 6.0.0 | MIT OR Apache-2.0 |
| dirs | 7.0.0 | MIT OR Apache-2.0 |
| dirs-sys | 0.5.0 | MIT OR Apache-2.0 |
| dispatch2 | 0.3.1 | Zlib OR Apache-2.0 OR MIT |
| displaydoc | 0.2.6 | MIT OR Apache-2.0 |
| document-features | 0.2.12 | MIT OR Apache-2.0 |
| downcast-rs | 1.2.1 | MIT/Apache-2.0 |
Expand Down Expand Up @@ -307,12 +306,8 @@ repository.
| num-traits | 0.2.19 | MIT OR Apache-2.0 |
| num_cpus | 1.17.0 | MIT OR Apache-2.0 |
| num_threads | 0.1.7 | MIT OR Apache-2.0 |
| objc2 | 0.6.4 | MIT |
| objc2-core-foundation | 0.3.2 | Zlib OR Apache-2.0 OR MIT |
| objc2-encode | 4.1.0 | MIT |
| objc2-foundation | 0.3.2 | MIT |
| objc2-io-kit | 0.3.2 | Zlib OR Apache-2.0 OR MIT |
| objc2-open-directory | 0.3.2 | Zlib OR Apache-2.0 OR MIT |
| object | 0.37.3 | Apache-2.0 OR MIT |
| once_cell | 1.21.4 | MIT OR Apache-2.0 |
| once_cell_polyfill | 1.70.2 | MIT OR Apache-2.0 |
Expand Down Expand Up @@ -470,7 +465,7 @@ repository.
| synthez | 0.4.0 | BlueOak-1.0.0 |
| synthez-codegen | 0.4.0 | BlueOak-1.0.0 |
| synthez-core | 0.4.0 | BlueOak-1.0.0 |
| sysinfo | 0.39.6 | MIT |
| sysinfo | 0.40.1 | MIT |
| system-configuration | 0.7.0 | MIT OR Apache-2.0 |
| system-configuration-sys | 0.6.0 | MIT OR Apache-2.0 |
| tar | 0.4.46 | MIT OR Apache-2.0 |
Expand Down
6 changes: 1 addition & 5 deletions THIRD_PARTY_NOTICES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8379,7 +8379,6 @@ The following component(s) are licensed under Apache License 2.0:
- num-conv 0.2.2 <https://github.com/jhpratt/num-conv>
- objc2-core-foundation 0.3.2 <https://github.com/madsmtm/objc2>
- objc2-io-kit 0.3.2 <https://github.com/madsmtm/objc2>
- objc2-open-directory 0.3.2 <https://github.com/madsmtm/objc2>
- pin-project-internal 1.1.13 <https://github.com/taiki-e/pin-project>
- pin-project-lite 0.2.17 <https://github.com/taiki-e/pin-project-lite>
- pin-project 1.1.13 <https://github.com/taiki-e/pin-project>
Expand Down Expand Up @@ -10794,9 +10793,6 @@ The following component(s) are licensed under MIT License:
- async-stream-impl 0.3.6 <https://github.com/tokio-rs/async-stream>
- async-stream 0.3.6 <https://github.com/tokio-rs/async-stream>
- libm 0.2.16 <https://github.com/rust-lang/compiler-builtins>
- objc2-encode 4.1.0 <https://github.com/madsmtm/objc2>
- objc2-foundation 0.3.2 <https://github.com/madsmtm/objc2>
- objc2 0.6.4 <https://github.com/madsmtm/objc2>
- peg-macros 0.6.3 <https://github.com/kevinmehall/rust-peg>
- peg-runtime 0.6.3 <https://github.com/kevinmehall/rust-peg>

Expand Down Expand Up @@ -11167,7 +11163,7 @@ MIT License
================================================================================

The following component(s) are licensed under MIT License:
- sysinfo 0.39.6 <https://github.com/GuillaumeGomez/sysinfo>
- sysinfo 0.40.1 <https://github.com/GuillaumeGomez/sysinfo>

The MIT License (MIT)

Expand Down
30 changes: 27 additions & 3 deletions crates/rocm-core/src/disk_space.rs
Original file line number Diff line number Diff line change
Expand Up @@ -230,11 +230,20 @@ pub fn mount_for_path(path: &Path) -> Option<(PathBuf, u64)> {
/// `/proc/diskstats` and `/sys/block/*/queue/rotational`, neither of which this
/// module uses.
fn listed_mounts() -> Vec<(PathBuf, u64)> {
let disks = sysinfo::Disks::new_with_refreshed_list_specifics(
mounts_of(sysinfo::Disks::new_with_refreshed_list_specifics(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: the Err is dropped silently here. A tracing::debug! of the error would make a permanently "unknown" disk check diagnosable. Not blocking, since unknown never blocks an install.

sysinfo::DiskRefreshKind::nothing().with_storage(),
);
))
}

/// Flatten a disk listing into mount points, or an empty list when the
/// platform could not enumerate disks at all. An empty list leaves
/// `mount_for_path` with nothing to match, so callers get "unknown" and never
/// block the operation.
fn mounts_of<E>(disks: Result<sysinfo::Disks, E>) -> Vec<(PathBuf, u64)> {
let Ok(disks) = disks else {
return Vec::new();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (confidence 70, architectural/consistency): this Err arm drops a total disk-enumeration failure with no diagnostic at all, which is asymmetric with the paired fix in this same PR: HostCollector::from_system (host.rs) logs a tracing::warn! for the analogous System::new_with_specifics failure. rocm-core doesn't depend on tracing (checked crates/rocm-core/Cargo.toml), but it already has a precedent for exactly this situation — a non-fatal, "degrade and keep going" failure reported via a warn-once eprintln! (rotation_failure_warning in crates/rocm-core/src/jsonl_store.rs:310-317). Functionally this is harmless — it correctly degrades to SpaceCheck::Unknown, which the module already documents as "never block" — but a host where disk enumeration totally fails now does so invisibly, with no way to distinguish it from the expected "this one mount isn't listed" case.

Suggested fix: add a one-time eprintln!-based warning in this Err arm, mirroring rotation_failure_warning's pattern, so the degradation is at least discoverable.

};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, non-blocking: a total disk-enumeration failure here is dropped with no diagnostic at all, which is asymmetric with HostCollector::from_system in host.rs in this same PR, which logs a tracing::warn! for the equivalent System::new_with_specifics failure. The behavior is still correct — this degrades to SpaceCheck::Unknown, which the module documents as "never block" — but a host where disk enumeration fails entirely will do so invisibly, indistinguishable from the normal "one mount not listed" case. rocm-core doesn't depend on tracing, so pulling that crate in just for this would be a bigger change than the fix warrants; a one-time eprintln!/log here (mirroring the existing warn-once pattern used elsewhere in this crate, e.g. jsonl_store.rs's rotation_failure_warning) would close the observability gap without adding noise, since this function isn't called in a hot loop anyway through listed_mounts(). Worth confirming this asymmetry was a deliberate choice.

disks
.list()
.iter()
.map(|disk| (disk.mount_point().to_path_buf(), disk.available_space()))
.collect()
Expand Down Expand Up @@ -892,6 +901,21 @@ mod tests {
);
}

#[test]
fn a_failed_disk_enumeration_reports_unknown_space_and_never_blocks() {
// sysinfo 0.40 made listing disks fallible. The failure must surface as
// "no mounts", which `select_mount` cannot match and `classify_space`
// reports as unknown, rather than as a panic or an invented figure.
let mounts = mounts_of::<&str>(Err("the platform could not list disks"));
assert!(mounts.is_empty(), "{mounts:?}");
let selected = select_mount(Path::new("/var/cache/rocm"), &mounts);
assert_eq!(selected, None);
assert!(matches!(
classify_space(u64::MAX, selected.map(|(_, available)| available)),
SpaceCheck::Unknown
));
}

#[test]
fn insufficient_space_refuses_and_sufficient_space_allows() {
// The refusal path itself, driven by a synthetic figure rather than
Expand Down
60 changes: 41 additions & 19 deletions crates/rocm-dash-collectors/src/host.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ use rocm_dash_core::metrics::SystemMetrics;
use sysinfo::{MemoryRefreshKind, RefreshKind, System};

pub struct HostCollector {
sys: System,
// `None` when sysinfo could not initialise on this platform; `tick` then
// reports empty metrics instead of failing the whole dashboard.
sys: Option<System>,
}

impl Default for HostCollector {
Expand All @@ -22,30 +24,39 @@ impl HostCollector {
let refresh = RefreshKind::nothing()
.with_cpu(sysinfo::CpuRefreshKind::everything())
.with_memory(MemoryRefreshKind::everything());
let mut sys = System::new_with_specifics(refresh);
// First refresh primes CPU deltas; the very first read is meaningless.
sys.refresh_cpu_all();
sys.refresh_memory();
Self { sys }
Self::from_system(System::new_with_specifics(refresh))
}

fn from_system<E: std::fmt::Display>(sys: Result<System, E>) -> Self {
match sys {
Ok(mut sys) => {
// First refresh primes CPU deltas; the very first read is meaningless.
sys.refresh_cpu_all();
sys.refresh_memory();
Self { sys: Some(sys) }
}
Err(error) => {
tracing::warn!(%error, "host metrics unavailable: sysinfo failed to initialise");
Self { sys: None }
}
}
}

pub fn tick(&mut self) -> SystemMetrics {
self.sys.refresh_cpu_all();
self.sys.refresh_memory();
let Some(sys) = self.sys.as_mut() else {
return SystemMetrics::default();
};
sys.refresh_cpu_all();
sys.refresh_memory();

let cpu_overall_pct = self.sys.global_cpu_usage();
let cpu_per_core_pct: Vec<f32> = self
.sys
.cpus()
.iter()
.map(sysinfo::Cpu::cpu_usage)
.collect();
let cpu_overall_pct = sys.global_cpu_usage();
let cpu_per_core_pct: Vec<f32> = sys.cpus().iter().map(sysinfo::Cpu::usage).collect();

// sysinfo reports memory in bytes.
let memory_used_mb = self.sys.used_memory() / 1024 / 1024;
let memory_total_mb = self.sys.total_memory() / 1024 / 1024;
let swap_used_mb = self.sys.used_swap() / 1024 / 1024;
let swap_total_mb = self.sys.total_swap() / 1024 / 1024;
let memory_used_mb = sys.used_memory() / 1024 / 1024;
let memory_total_mb = sys.total_memory() / 1024 / 1024;
let swap_used_mb = sys.used_swap() / 1024 / 1024;
let swap_total_mb = sys.total_swap() / 1024 / 1024;

SystemMetrics {
cpu_overall_pct,
Expand Down Expand Up @@ -74,4 +85,15 @@ mod tests {
assert!(!m.cpu_per_core_pct.is_empty(), "expected at least one CPU");
assert!(m.memory_total_mb > 0, "expected nonzero total memory");
}

#[test]
fn a_collector_whose_sysinfo_failed_to_initialise_reports_empty_metrics() {
// sysinfo 0.40 made `System` construction fallible. The dashboard must
// keep ticking, with nothing to show for the host, rather than panic.
let mut c = HostCollector::from_system::<&str>(Err("unsupported platform"));
let m = c.tick();
assert!(m.cpu_per_core_pct.is_empty(), "{m:?}");
assert_eq!(m.memory_total_mb, 0, "{m:?}");
assert_eq!(m.swap_total_mb, 0, "{m:?}");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision for the author (non-blocking-improvement): this test only spot-checks 3 of SystemMetrics's 10 fields (cpu_per_core_pct, memory_total_mb, swap_total_mb). Today's production code returns the whole SystemMetrics::default() atomically on the None branch, so there's no live defect, but a future edit that builds a partial struct instead of ..Default::default() could leave cpu_overall_pct/memory_used_mb/swap_used_mb/the disk/net fields wrong and this test would still pass. SystemMetrics doesn't currently derive PartialEq, so a single assert_eq!(m, SystemMetrics::default()) isn't available as-is — your call whether a stronger whole-struct assertion (adding PartialEq) is worth it here, or whether the three-field spot-check is sufficient.

}
}
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,7 @@ The I/O-heavy collector implementations behind `rocm-dash-core`'s traits (amd-sm
| [`cgroup.rs`](../crates/rocm-dash-collectors/src/cgroup.rs) | Attributes GPU processes to Docker containers by reading the 64-hex container id from `/proc/<pid>/cgroup`. | flat |
| [`docker.rs`](../crates/rocm-dash-collectors/src/docker.rs) | `DockerDiscovery`: finds vLLM containers through `bollard` and builds a `DiscoveredService` from each container's environment and command line. | flat |
| [`engine_registry.rs`](../crates/rocm-dash-collectors/src/engine_registry.rs) | `EngineKind`: maps a serving engine (vLLM, Lemonade) to its label, conventional port and sample parser. | flat |
| [`host.rs`](../crates/rocm-dash-collectors/src/host.rs) | `HostCollector`: CPU, memory and swap from `sysinfo`, sampled once per tick into `SystemMetrics`. Disk and network rates are not collected yet and stay 0. | flat |
| [`host.rs`](../crates/rocm-dash-collectors/src/host.rs) | `HostCollector`: CPU, memory and swap from `sysinfo`, sampled once per tick into `SystemMetrics`. Disk and network rates are not collected yet and stay 0. If `sysinfo` cannot initialise, it logs one warning and every tick reports empty metrics. | flat |
| [`lemonade.rs`](../crates/rocm-dash-collectors/src/lemonade.rs) | The Lemonade Server collector: pure parsers over its stats and health endpoints, and discovery of a running server. | flat |
| [`parallel.rs`](../crates/rocm-dash-collectors/src/parallel.rs) | `parallel_scrape`: runs an async scrape against many targets concurrently on a `JoinSet`. | flat |
| [`vllm_prom.rs`](../crates/rocm-dash-collectors/src/vllm_prom.rs) | The vLLM Prometheus `/metrics` scraper: request gauges, KV-cache usage, the generation-token counter and the latency histogram sums. | flat |
Expand Down
Loading