Repository navigation
ROCMAI-938 follow-up: one throughput aggregate for both Observe heroes, skip faulty power readings - #1030
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
juhovainio
left a comment
There was a problem hiding this comment.
Nice cleanup: both Observe heroes now go through one node_throughput, and the held/idle-vs-no-data distinction is tested with exact values. I found nothing blocking in the logic, but a few things should be tightened before merge (details inline):
- The throughput hero still sums
power_wunfiltered, whilenode_efficiencynow skips non-finite/non-positive readings. The faulty-sensor class fixed next door is still reachable there. - The
efficiency.rsentry indocs/architecture.mdshould mention the new power guard, and the claim that the Hardware header andlist_instancesusenode_throughputshould be double-checked. - The new Observe test asserts the unmarked figure with a literal
"300.0 tok/s*"while the efficiency check usesformat::HELD_MARKER. If the marker ever isn't*, the negative assertion becomes vacuous; using the constant in both places would avoid that.
CI is green on the required checks (GPU lanes skipped by the runner outage; the consolidated e2e report was still pending when I looked).
| let tps = throughput.map_or(0.0, |t| t.tokens_per_sec); | ||
| let has_tps = throughput.is_some(); | ||
| let any_held = throughput.is_some_and(|t| t.held()); | ||
| let power: f32 = snap.gpus.iter().map(|g| g.power_w).sum(); |
There was a problem hiding this comment.
This hero still sums power_w unfiltered, while node_efficiency now skips non-finite/non-positive readings. A single faulty sensor would still skew whatever is derived from power here. Consider the same guard, or a shared helper so the power sum isn't hand-rolled in several places.
There was a problem hiding this comment.
Fixed in ff74b73a. One shared node_power_w (in rocm-dash-core) sums board power over the GPUs with a usable reading, skipping one that is not finite or not positive as attribute_power_w does. It is now used by node_efficiency, this hero, the Hardware header kW figure and Home's power sparkline, so none of them hand-rolls the sum. One visible effect: when no GPU reports usable power, this hero now shows — instead of 0.0 W, which fits a platform with no power source better.
| - [`bench_schema.rs`](../crates/rocm-dash-core/src/bench_schema.rs) — `BenchmarkRow`, the one row type for a benchmark result (identity, platform, backend configuration, request statistics, throughput, latency, quality, rollup and provenance columns), and `PassFail`. Almost every field is an `Option` and the struct is `#[serde(default)]`, so a CSV or a recording that omits columns still deserializes. `CsvBenchTailer` deserializes CSV rows into it, `bench_load.rs` builds it, and the daemon forwards it in `Event::BenchmarkRowsAppended`. | ||
| - [`config.rs`](../crates/rocm-dash-core/src/config.rs) — the standalone dashboard configuration read from `<config dir>/rocm-dash/config.toml` (`default_config_path`; a missing file means defaults and `Config::load` fails only on a read or parse error): `Config` with `DaemonConfig` (listen address, token, tick intervals, benchmark CSV path), `TuiConfig` (connect address, theme, chat endpoint, model and auth-header name) and per-engine `EngineConfig`. It is separate from the unified `rocm` configuration (the `dashboard` section of `rocm-core`'s `RocmCliConfig`, which `rocm dash` reads); in the tree only the TUI's chat-endpoint save (`persist_chat_endpoint` in [`app/chat.rs`](../crates/rocm-dash-tui/src/app/chat.rs)) loads and writes this file. Its default socket path (`$XDG_RUNTIME_DIR`, then `~/.rocm/data/telemetry`, then a user-named directory under the temp dir) is a copy of `rocm-core`'s on purpose, because this crate may have no first-party dependencies (`cargo xtask check-crate-edges` enforces that). | ||
| - [`efficiency.rs`](../crates/rocm-dash-core/src/efficiency.rs) — the pure join of throughput and power. `normalize_gpu_id` strips the `gpu-` prefix so the bare indices Docker discovery records in `Instance.gpu_ids` match amd-smi's `gpu-N` device ids; `attribute_power_w` splits each GPU's board power among the instances that occupy it, weighted by per-container VRAM, equally when any occupant has no reading, and yields `None` for an instance none of whose GPUs reported a positive power reading; `tokens_per_watt` divides generation throughput by that attributed power. `node_efficiency` returns the whole-node figure as a `NodeEfficiency` (Σ `gen_tps` over Σ board power, including GPUs no instance occupies) with a `held` flag that is set when any throughput behind it is held; it is the one implementation behind the Observe efficiency hero and its trend, the Hardware header, and the assistant's `tokens_per_watt` tool (the Home tab's tokens-per-watt figure is a separate sum of per-instance values). The daemon runner calls `attribute_power_w`, `tokens_per_watt` and `gpu_ids_overlap`, which it uses to warn when the id join finds nothing. | ||
| - [`efficiency.rs`](../crates/rocm-dash-core/src/efficiency.rs) — the pure join of throughput and power. `normalize_gpu_id` strips the `gpu-` prefix so the bare indices Docker discovery records in `Instance.gpu_ids` match amd-smi's `gpu-N` device ids; `attribute_power_w` splits each GPU's board power among the instances that occupy it, weighted by per-container VRAM, equally when any occupant has no reading, and yields `None` for an instance none of whose GPUs reported a positive power reading; `tokens_per_watt` divides generation throughput by that attributed power. `node_efficiency` returns the whole-node figure as a `NodeEfficiency` (Σ `gen_tps` over Σ board power, including GPUs no instance occupies) with a three-state freshness (fresh, held, or unknown) worked out by `node_throughput`, which sums the node's throughput once and ignores instances at zero tok/s when judging freshness; both are the one implementation behind the Observe efficiency and throughput heroes and the efficiency trend, the Hardware header, and the assistant's `tokens_per_watt` and `list_instances` tools (the Home tab's tokens-per-watt figure is a separate sum of per-instance values). The daemon runner calls `attribute_power_w`, `tokens_per_watt` and `gpu_ids_overlap`, which it uses to warn when the id join finds nothing. |
There was a problem hiding this comment.
The efficiency.rs entry should also say that node_efficiency skips non-finite/non-positive board-power readings (the new behavior). Please also confirm the Hardware header and list_instances really go through node_throughput/node_efficiency, since this diff doesn't touch those consumers.
There was a problem hiding this comment.
Done in ff74b73a. The entry now says node_power_w skips non-finite and non-positive readings, and spells out who calls what. I checked the consumers against the code: the Hardware header (gpu_section_header) and both assistant tools (node_json, used by list_instances and tokens_per_watt) go through node_efficiency, which calls node_throughput, so they reach it indirectly; the Observe and Home throughput heroes call node_throughput directly. The diff does not touch those consumers because they already used node_efficiency.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 7233622e
Review — no blocking findings
Full review of the whole change (3 files: crates/rocm-dash-core/src/efficiency.rs, crates/rocm-dash-tui/src/ui/tabs/observe.rs, docs/architecture.md; merge-base eb2a8194→head 7233622e).
Follow-up to #1017 (ROCMAI-938 named in the description, not retrieved from any tracker here). The description's claims were checked against #1017's own post-merge automated review (public record on that already-merged PR): its one blocking finding (the two Observe heroes disagreeing on held-ness for a zero-throughput instance) and its non-blocking findings (missing power-finiteness guard, docs/architecture.md omitting list_instances) are each addressed here, with a new test pinning the specific scenario the blocking finding named.
Blocking
None
Non-blocking
-
The Node-throughput hero's own power readout isn't guarded against the sensor fault this PR just fixed for the efficiency quotient beside it —
crates/rocm-dash-tui/src/ui/tabs/observe.rs:184(let power: f32 = snap.gpus.iter().map(|g| g.power_w).sum();, unfiltered),crates/rocm-dash-tui/src/ui/format.rs:204-206(watts():format!("{value:.1} W"), no finite guard).node_efficiency's power sum now skips a non-finite or non-positivepower_w(crates/rocm-dash-core/src/efficiency.rs:205-213), but the throughput hero a few lines below still sums every GPU's raw power unfiltered for its own "power" line. One GPU reportingInfinity/NaN/a negative value rendersinf W/NaN Wdirectly in that hero while the efficiency hero beside it (now protected) shows a sane figure — a new two-heroes-disagree case, in the same panel row, of the exact class this PR's own description says it is closing.hardware.rs'sgpu_section_headersums the same way for its kW figure, so the gap isn't confined to this one line. Confidence 80 · architectural · Fix: filter this sum (andhardware.rs's) the same way, or share one filtered-power helper. -
Home tab's own "Node throughput" hero still has the exact held/zero-throughput bug this PR fixes for Observe —
crates/rocm-dash-tui/src/ui/tabs/home.rs:507-514(any_tps_heldcomputed with nov <= 0.0exclusion, unlike the newnode_throughput). A held instance reporting0.0tok/s next to a fresh busy one marks Home's own, identically-titled "Node throughput" hero as held — the same bug class this PR's title and tests fix for Observe's two heroes, left standing on a third surface sharing the name. Untouched by this diff and outside the PR's stated scope (Observe only), so not blocking, but directly pattern-relevant. Confidence 82 · logic · Fix: migrate Home's aggregate onto the sharednode_throughput, or note the gap as a follow-up. -
One sub-assertion in the new power-sensor test doesn't discriminate the guard it's named for —
crates/rocm-dash-core/src/efficiency.rs:404-409(node_efficiency_skips_a_faulty_power_sensor_like_attribute_power_does'sonly_badcase:gpu("gpu-0", f32::NAN), gpu("gpu-1", -5.0)→ expectsNone).f64::from(NaN) + f64::from(-5.0)isNaN, andNaN > 0.0is alreadyfalse, so this assertion returnsNonewhether or not the new finite/positive filter exists — deleting the filter does not turn this line red. The test's other two cases (inf,neg) do flip correctly under that mutation, so the test as a whole still catches a reverted fix, but this line documents the "no usable telemetry" case rather than proving the guard causes it. Confidence 72 · mechanical · Fix: none required for correctness; optional to swap in a case the filter alone discriminates (e.g. a loneInfinityreading).
Decisions for the author
node_efficiency'stokens_per_sec > 0.0 && power > 0.0gate still collapses a genuinely idle, fully-telemetered node (realΣ gen_tps == 0) into the sameNone/—as true data absence — non-blocking-improvement. The newnode_throughputthis PR introduces already tracks "did anything contribute" separately from the sum's value (its own doc: "A real zero counts: an idle node is zero throughput, not no data", backed bynode_throughput_distinguishes_idle_from_no_data), and the throughput hero now correctly shows a real0.0 tok/sfor this case — butnode_efficiencydiscards that distinction immediately after calling it, still requiringtokens_per_sec > 0.0to return a figure at all. This exact gap was raised twice on the preceding PR #1017 (non-blocking there too) and left as documented, apparently deliberate behavior; this PR rewrites the surrounding function without changing it. Worth a deliberate decision on whether to finally close it, since the tracking it would need already exists for free.
Positive signals
node_throughput's extraction preservesnode_efficiency's prior behavior for every reachable case (traced the "no finite readings" and "all-exactly-zero" paths against the old inline loop by hand) while fixing the real cross-panel disagreement for a zero-tok/s held instance.- The new tests target real, distinct edge cases — idle vs. no-data, infinite vs. negative power sensor, cross-function freshness agreement — rather than restating the happy path, and three of the four are traceably mutation-sensitive.
docs/architecture.md's updated entry names every real call site ofnode_throughput/node_efficiencyaccurately — each one (Observe's two heroes, the efficiency trend, the Hardware header, both assistant tools) was traced against the actual code; no stale or missing caller.
Deployment notes
None — no migration, no breaking API change; node_throughput is a new, additive public function.
What this covered
Every line of crates/rocm-dash-core/src/efficiency.rs (591 lines) and crates/rocm-dash-tui/src/ui/tabs/observe.rs (584 lines) read completely at the PR head, including all test modules, plus docs/architecture.md's diff and the call sites its new wording names (hardware.rs, agent/snapshot.rs, agent/tools.rs, home.rs). Split across three independent fan-out workers: (1) the coupled core+UI code pass, (2) the docs/cross-file-consistency pass, (3) a history and prior-changes pass against this code's own git log and PR #1017's already-merged, public review history. AGENTS.md read in full and checked (module-size, architecture-doc sync, markdown-link, and test-verdict rules); no violation found beyond the items above. eb2a8194…7233622e.
CI re-checked live on this head: every required check SUCCESS (self-hosted GPU E2E lanes SKIPPED, consistent with the description's noted runner outage; the consolidated E2E report itself is SUCCESS).
Did not run: no Rust toolchain available in this sandbox, so cargo test/cargo clippy were not executed — verification above is static, by tracing code paths, arithmetic and existing tests by hand; confidence is capped accordingly. No Gherkin/e2e scenario exists for this change and the description explains why; not independently disputed.
Not reconciled against this PR's existing review thread.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
7233622 to
ff74b73
Compare
|
Thanks both. All of it is in
Not changed
|
Addressed in ff74b73: the throughput hero, the Hardware header and Home now use one shared node_power_w, the architecture doc says what each consumer calls and covers the power guard, and the held-marker assertion uses HELD_MARKER. Every point has a reply: #1030 (comment)
juhovainio
left a comment
There was a problem hiding this comment.
The shared throughput aggregate removes the duplication between the two heroes, and the new tests cover idle vs. no-data, held vs. idle, and the Inf-only power case. Two things need fixing before merge.
Blocking
- The Home power label still sums raw readings (
home.rs, thelet power: f64 = ... .sum()feedingpower_label), while the sparkline right below it now usesnode_power_w. One Inf or negative sensor shows in the label but is skipped in the trace. The newdocs/architecture.mdtext saysnode_power_wis used for the Observe, Hardware and Home power figures, which is false for the Home label. This is the same class of bug the PR fixes. Switch the label tonode_power_w(thepower == 0.0=> "-" rule still works) and add a test with an Inf GPU, or narrow the doc. - The PR body omits user-visible changes in the diff: the Home throughput hero's held rule, the Home sparkline and Hardware kW header moving to the filtered sum, and the Observe hero's
has_powerchange (GPUs reporting 0 W now show "-" instead of "0 W"). Please update the body to match the final diff, and mention thehas_powerchange in the no-scenario justification.
Non-blocking
- The doc comment on
node_efficiencyinefficiency.rsis badly reflowed into one very long line. - The Observe tuple is now a 5-tuple mixing f64/f32; a small struct would read better.
- Please confirm intent: a node held only at zero tok/s reports a fresh 0 tok/s, while
node_efficiencyreturnsNonefor it.
CI has no failing checks; several are still pending. I did not run the tests or the break-the-target checks myself.
| // Power can exceed 100; scale to a nominal 1kW ceiling for the trace. | ||
| &history(state, |s| { | ||
| s.gpus.iter().map(|g| f64::from(g.power_w)).sum::<f64>() / 10.0 | ||
| rocm_dash_core::efficiency::node_power_w(&s.gpus) / 10.0 |
There was a problem hiding this comment.
The sparkline now uses node_power_w, but the power label a few lines above (let power: f64 = latest.map_or(0.0, |s| s.gpus.iter().map(..).sum())) still sums raw readings, so one Inf/negative sensor shows in the label and is skipped in the trace. architecture.md says Home uses node_power_w too. Please switch the label (and add a test with an Inf GPU) or narrow the doc.
There was a problem hiding this comment.
Fixed in 89bdf121. Home's power label now uses node_power_w, like the sparkline, so a faulty sensor no longer shows in one and not the other (the power == 0.0 => — rule still holds). New test home_power_label_skips_a_faulty_sensor_like_the_trace_beside_it puts an infinite GPU next to a usable 420 W one and checks the label shows 420.0 W, never inf W; putting the raw sum back makes it fail. The architecture doc is accurate now, and was re-applied onto main's new per-crate layout.
| let has_tps = throughput.is_some(); | ||
| let any_held = throughput.is_some_and(|t| t.held()); | ||
| let power = rocm_dash_core::efficiency::node_power_w(&snap.gpus); | ||
| (tps, has_tps, power, power > 0.0, any_held) |
There was a problem hiding this comment.
Behavior change: has_power was !snap.gpus.is_empty() and is now power > 0.0, so GPUs reporting 0 W (no telemetry) now render "-" instead of "0 W". Probably right, but it is user-visible and not in the PR body; consider a test pinning it.
There was a problem hiding this comment.
Pinned in 89bdf121: throughput_hero_power_is_a_dash_when_gpus_report_no_usable_power checks that GPUs reading 0 W show power — (not 0.0 W) and that a usable reading still prints; reverting has_power to the old non-empty check fails it. The change is also now in the PR body, including in the no-scenario justification.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head ff74b73a160a
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change. Implements a scoped, accurate follow-up to #1017 (ROCMAI-938): unifies the Observe tab's two held-ness checks behind a new node_throughput helper, and adds node_efficiency's missing finite/positive power-sensor guard. The PR description's own claims match the code exactly; it does not itself claim the Home tab's headline power figure is fixed — only docs/architecture.md does, and that claim is not accurate (see Blocking).
Blocking
- Home tab's headline POWER figure still uses an unfiltered sum, disagreeing with its own trend line beneath it, and with this PR's own doc update —
crates/rocm-dash-tui/src/ui/tabs/home.rs:446-447(and its use at:467forpower_label), vs.crates/rocm-dash-tui/src/ui/tabs/home.rs:501-503.
The Home "POWER" card's headline number is computed ass.gpus.iter().map(|g| f64::from(g.power_w)).sum::<f64>()— no finite/positive guard. Four lines later, the same card's trend/sparkline history for the identical quantity was migrated by this PR torocm_dash_core::efficiency::node_power_w(&s.gpus), which does apply that guard. One GPU reporting a faulty reading (infinity, NaN, or a negative value — exactly the fault class this PR targets) makes the headline wrong (format::watts's{value:.1} Wwould render e.g."inf W"or a negative wattage) while the trend line directly beneath it, drawn from the same snapshot, stays correct — the two halves of one card can visibly disagree for the same tick. This also falsifies the PR's owndocs/architecture.mdupdate to theefficiency.rsbullet, which statesnode_power_w"is used wherever the node's power is shown or divided by (the Observe, Hardware and Home power figures)" — the Home headline is the one place that isn't true.
Confidence 85 · logic · Fix: route the headlinepowercomputation throughrocm_dash_core::efficiency::node_power_w(&s.gpus)as well, so both halves of the card — and the architecture.md claim — match reality.
Non-blocking
None
Decisions for the author
- Splitting the aggregate into two free functions invites a silent partial migration — non-blocking-improvement
node_throughputandnode_power_wmust both be called at every site that shows the node's power/throughput; forgetting one (as happened above) still compiles, since the old raw.sum()is still valid Rust. Worth a cheap guard against recurrence — e.g. a repo-wide check (in the spirit of the existingcheck-crate-edges/check-module-sizextask checks) that flags a rawpower_wsum orgen_tpsfold outsideefficiency.rs. - Two test gaps in
efficiency.rs'snode_throughput, both below this review's confidence bar to report as defects — non-blocking-improvement
No test exercises a negativegen_tpsreading (asymmetric againstnode_power_w's explicit negative-value guard —node_throughput's doc comment only promises robustness for zero tok/s, andgen_tpsrealistically shouldn't go negative, so this reads as an out-of-scope input rather than a bug). Also no test combines a contributing Held instance with a contributing Unknown-freshness instance (only Held+Fresh and Fresh+Unknown are covered) to confirm Held still wins over Unknown absent any Fresh instance.
Positive signals
node_throughput's freshness precedence (Held > Unknown > Fresh, with zero/non-finite throughput excluded from influencing it at all) is subtle, documented precisely, and exercised by tests hitting every combination.node_power_w's filter is kept in deliberate, doc-cross-referenced parity with the pre-existingattribute_power_wfilter (verified bit-for-bit equivalent, including-0.0and subnormal boundary cases) — good discipline against the two aggregates silently drifting apart.node_throughputtakesimpl IntoIterator<Item = &Instance>rather than a concrete collection, letting one implementation serve bothSnapshot'sVec<Instance>andAppState'sHashMap::values()call sites without duplicating the aggregation logic.
Deployment notes
None — internal dashboard TUI logic only; no migration, no external API or config change.
What this covered
Read completely: all 5 changed files (crates/rocm-dash-core/src/efficiency.rs, crates/rocm-dash-tui/src/ui/tabs/hardware.rs, crates/rocm-dash-tui/src/ui/tabs/home.rs — in full, ~1,878 lines — crates/rocm-dash-tui/src/ui/tabs/observe.rs, docs/architecture.md), plus surrounding context (metrics.rs's Instance/GpuMetrics/ObservationFreshness, format.rs's watts/gen_tps_aggregate/tokens_per_watt, agent/snapshot.rs's assistant-tool consumers), and a repo-wide search for every other site touching gen_tps/power_w aggregation to check for further missed migrations (none found beyond the one reported). Base 852ce7b0…head ff74b73a, fetched read-only (no checkout).
efficiency.rs was additionally reviewed by an independent fanned-out background worker (fresh context, no access to this transcript or the above); its report is folded into the findings and decisions above — no blocking findings from that pass.
The TUI-consumer pass (home.rs/observe.rs/hardware.rs) could not be run as an independent fan-out worker: two spawn attempts both hit a shared concurrency limit, and only one retry is taken before falling back per method. That pass was run by me (the driver) instead, so it is not independent — the Blocking finding above comes from that non-independent pass.
Did not run: the history pass (git log/git blame) — the read-only clone used shallow fetches of only the needed commits, so no prior history was available to mine. The prior-changes pass — no earlier review comments on these files were consulted (by design, blind to this PR's own discussion). CI was read live for commit ff74b73a160a3e7639f2f77b6ebb1d66a3e8e96a: all 46 check runs completed, none failing (several skipped, expected for hardware-gated e2e lanes) — no CI-blocking finding. Ticket ROCMAI-938 is named but was deliberately not retrieved: this is a public repository and the identifier reads as an internal tracker reference, so it was left alone even though a tracker tool was technically available. The named GitHub PR #1017 (same repo, public) was read instead, confirming #1030 is a scoped bug-fix follow-up to post-merge review feedback on #1017, consistent with this PR's description.
Not reconciled against any discussion or prior review thread on this PR.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
ff74b73 to
89bdf12
Compare
|
Thanks. Both blocking points and the non-blocking ones are in
|
Addressed in 89bdf12: Home's power label now uses node_power_w (with a faulty-sensor test), the PR body matches the final diff including the has_power change, and the doc comment and tuple are tidied. Every point has a reply: #1030 (comment)
juhovainio
left a comment
There was a problem hiding this comment.
Re-reviewed at 89bdf12. Nothing here looks functionally wrong; the shared node_throughput / node_power_w helpers taking impl IntoIterator is a good consolidation, and the fixtures use values the real source can produce (inf, NaN, negative, 0).
Changes requested, both about tests that don't pin what this PR changes:
- The Hardware kW header now uses the filtered
node_power_w, with no test that fails if it is reverted. - The Home test asserts the power label only, while its comment claims agreement with the sparkline's filtered sum. Reverting the sparkline closure would stay green.
Smaller notes:
power as f32in the throughput hero narrows an f64 sum only to feedformat::watts(inline).- The
docs/architecture.mdstatement that the assistant'slist_instances/tokens_per_watttools go throughnode_throughput: please double-check it against the code, I could not confirm it from the diff.
Please confirm these are intended:
- A node whose only throughput is a held zero shows a fresh, unmarked 0 tok/s on the throughput hero (matches the ROCMAI-938 criteria as I read them).
- Hero power now means "usable power > 0", so GPUs without telemetry show
—instead of0.0 W.
CI: the red Linux jobs (build-and-test, Test (affected crates), E2E, Coverage) all fail at "Install native build deps" with apt DNS resolution errors, so no Linux test step has run on this head. They need a re-run before this can be judged green.
| } | ||
| } | ||
| let total_power: f64 = snap.gpus.iter().map(|g| f64::from(g.power_w)).sum(); | ||
| let total_power = rocm_dash_core::efficiency::node_power_w(&snap.gpus); |
There was a problem hiding this comment.
The kW header now uses the filtered node_power_w, but nothing pins it: reverting this line leaves every test green. Please add a render test beside the Home one with a faulty/negative sensor next to a usable one, and check it goes red when the line is reverted.
There was a problem hiding this comment.
Pinned in 2dd9e687: gpu_section_header_kw_skips_a_faulty_power_sensor puts a usable 500 W next to an infinite and a negative reading and checks the header says 0.5 kW, never inf or a total deflated by the negative one. With the line reverted to the raw sum it fails.
| let tps_str = format::gen_tps_aggregate(has_tps.then_some(tps), any_held); | ||
| let power_str = if has_power { | ||
| format::watts(power) | ||
| format::watts(power as f32) |
There was a problem hiding this comment.
power as f32 narrows the f64 sum back just for format::watts. If watts took f64 the cast could go. Minor / optional.
There was a problem hiding this comment.
Done in 2dd9e687. format::watts takes f64 now, so both as f32 casts (here and in Home) are gone.
| ); | ||
| } | ||
|
|
||
| #[test] |
There was a problem hiding this comment.
The comment in this test says the label agrees with the sparkline's filtered sum, but only the label is asserted. Reverting the history(...) closure to a raw sum would not fail it. Please assert the trace too, or drop the claim from the comment.
There was a problem hiding this comment.
Fixed in 2dd9e687, by asserting the trace rather than dropping the claim. The trace is now its own power_trace function, and the test pushes a snapshot with a 420 W and an infinite GPU into the history and checks the trace is [42] (a raw sum would clamp to 100) as well as the label. Putting the raw sum back into power_trace fails it with "the trace must skip it as well".
… faulty power readings in node efficiency (ROCMAI-938 follow-up) Follow-up to the node-efficiency change. Two things a review found after merge: The efficiency hero took its held-ness from node_efficiency, which ignores instances at zero tok/s, while the throughput hero beside it kept an inline check that did not. A held instance at 0 tok/s next to a fresh one made the throughput hero show the held marker and legend and the efficiency hero show none, for the same snapshot. Move the throughput sum and its freshness into node_throughput, used by node_efficiency and by the throughput hero, so the two cannot disagree. A real zero is still a figure (an idle node), and nothing finite is still no data. node_efficiency also summed board power without the finite/positive guard that attribute_power_w applies, so one sensor reporting infinity made the quotient a plausible-looking 0 and a negative reading deflated the total. It now skips those readings the same way, and has no figure when none is usable. The architecture doc names both assistant tools and the new helper. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
89bdf12 to
2dd9e68
Compare
|
Thanks. Everything in this round is in
|
Addressed in 2dd9e68: the Hardware kW and Home trace now have tests that fail when reverted, the f32 cast is gone, and the consumers and intent questions are answered. Every point has a reply: #1030 (comment)
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 2dd9e6873b68
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change. Follow-up to #1017 (ROCMAI-938, named in the title/description but not retrieved from any tracker — not accessible from here). The description's claims match the code: it does not itself claim the Hardware header gained a "no telemetry" dash state, or that Home's tok/W hero was migrated — only docs/architecture.md implies the former's sibling behavior; see Blocking.
Blocking
-
Hardware header's kW figure never shows "—" for "no usable power telemetry", unlike the Observe and Home power figures this same PR gave that treatment —
crates/rocm-dash-tui/src/ui/tabs/hardware.rs:405-408.total_powercomes fromnode_power_w, whose own doc comment states "zero means 'no telemetry'", and bothobserve.rs's throughput hero (has_power: power > 0.0) andhome.rs's power label (if unavailable || power == 0.0 { "—" } else { ... }) were given that exact guard in this diff. Hardware'sgpu_section_headerinstead formatstotal_powerunconditionally asformat!(" · {:.1} kW", total_power / 1000.0), so an all-zero-power snapshot (a platform with no power source, or three sensors all filtered out as faulty) prints"0.0 kW"— a real, measured-looking reading — while the two sibling screens correctly show a dash for the identical condition. The repo's own existing testdraw_handles_degraded_and_empty_states(hardware.rs, case 3) already constructs exactly this all-power_w == 0.0snapshot and asserts"eff -", but makes no assertion on the kW figure, so the gap is untested. Confidence 85 · logic · Fix: guard the kW span the same way (if total_power > 0.0 { format!(...) } else { "—".to_string() }). -
Home's two hero aggregates (tok/s and tok/W) now disagree with each other on "held" semantics, a fresh inconsistency this PR introduces within Home's own hero band —
crates/rocm-dash-tui/src/ui/tabs/home.rs:381-386(any_tpw_held, untouched by this diff) vs.home.rs:515-517(any_tps_held, migrated by this diff).any_tps_heldnow comes fromnode_throughput(..).held(), which — per this PR's whole point — excludes an instance contributing a real zero from marking the figure held.any_tpw_helda few dozen lines above still uses the old pattern:i.tokens_per_watt.is_some_and(f64::is_finite) && ...Held, with no exclusion for a zero-contributing instance (andtokens_per_watt()'s own doc atefficiency.rs:244-245confirms a realSome(0.0)is a valid value, exactly analogous to a real zero tok/s). Before this PR, Home's tok/s hero used the identical unguarded patternany_tpw_heldstill uses today, so the two Home heroes used to agree (both wrong in the same way); after this PR, a held-but-idle instance next to a fresh busy one makes Home's tok/s hero show an unmarked figure while its tok/W hero, one card over, shows the held marker — the exact cross-hero disagreement this PR's title and tests exist to eliminate, now reproduced inside Home instead of between Observe's two heroes. Confidence 85 · logic · Fix: giveany_tpw_heldthe same "zero tok/W instances don't mark held" exclusionnode_throughputapplies, or route it through a shared helper.
Non-blocking
-
node_throughputhas no guard against a negative (but finite)gen_tpsreading, the same bug class this PR just guards against for power —crates/rocm-dash-core/src/efficiency.rs:160-168. Onlyis_finite()filters the sample; a negative reading is summed intotokens_per_secunfiltered, silently deflating (or flipping the sign of) the total — the identical failure modenode_power_w's neww > 0.0filter was built to close for power. Present unchanged in the pre-PR file too, so not a regression, but this PR touches this exact function to close the matching gap on the power side and leaves it open on the throughput side. Confidence 70 · logic · Fix: decide whether a negativegen_tpsis a realistic input; if so, filter it the same waynode_power_wdoes. -
docs/architecture.md's updatedefficiency.rsentry overclaims for Home —docs/architecture.md:396, in the sentence "node_power_w... is what the Observe, Hardware and Home power figures show and divide by." Home only showsnode_power_w(the power label and sparkline); Home's own tokens-per-watt figure divides per-instanceattribute_power_wvalues, notnode_power_w— which the same sentence concedes two clauses later ("the Home tab's tokens-per-watt figure is a separate sum of per-instance values"). Confidence 70 · mechanical · Fix: narrow the "divide by" clause to Observe and Hardware only. -
Stale doc comment on
draw_hero_band—crates/rocm-dash-tui/src/ui/tabs/observe.rs:69-71says the band "Reusesrocm_dash_core::efficiency::node_efficiency", butdraw_throughput_hero(the other half of the band, touched by this diff) now centrally depends onnode_throughputandnode_power_wtoo. Confidence 70 · mechanical · Fix: mention all three.
Decisions for the author
node_power_wreturns a baref64with an implicit "0.0 = no telemetry" sentinel, while its siblingnode_throughputreturnsOption<NodeThroughput>— tradeoff. The asymmetry is exactly what let the Hardware gap above through uncaught: a caller can "forget" the sentinel and the code still compiles. A uniformOption<f64>would force every caller to handle "no data" explicitly, at the cost of a slightly less convenient API for a plain sum.node_power_wandattribute_power_weach inline the same "finite && positive" filter predicate separately rather than sharing one named helper — non-blocking-improvement. Both are currently correct and verified consistent, but nothing ties them together if one is edited later.
Positive signals
node_throughput's extraction is a faithful, verifiable lift ofnode_efficiency's old inline loop;node_efficiencynow composesnode_throughput+node_power_wexactly as its doc says, and the old duplicate inline loop inobserve.rsis fully removed with no dead code left behind.- State-matrix test coverage (held/fresh/unknown-freshness × real-zero/no-data, for both throughput and power, including NaN/Infinity/negative-sensor cases) is thorough and asserts exact values rather than weak substring checks; several are genuine fail-before/pass-after regression tests, including two added in a late push to this PR (
gpu_section_header_kw_skips_a_faulty_power_sensor, and thepower_traceassertion added alongsidehome_power_label_skips_a_faulty_sensor_like_the_trace_beside_it). - No index-based pairing of independently-ordered lists: both shared aggregates are commutative (sum / OR-of-flags), so
Home'sHashMap::values()iteration order can't introduce a correlation bug. docs/architecture.md's updated entry correctly names every other real call site this PR introduces (Observe's two heroes, the Hardware header, both assistant tools), verified against the actual code.
Deployment notes
None — internal dashboard TUI logic only; no migration, no external API or config change; node_throughput/node_power_w are new, additive public functions.
What this covered
Base f536619a08a3b50dd26aac5c21a3bb9f6bd266d2…head 2dd9e6873b68842f66dd74b24bf5f555ae4c597d, fetched read-only (no checkout). The PR was force-pushed to this head partway through this review (from 89bdf1211d...); the full diff and all touched files were re-read and re-verified against the new head, including the amendment itself (the format::watts signature widened from f32 to f64, a new Hardware kW regression test, and a new power_trace helper + assertion in home.rs).
All 6 changed files read completely: crates/rocm-dash-core/src/efficiency.rs (627L), crates/rocm-dash-tui/src/ui/tabs/observe.rs (627L), crates/rocm-dash-tui/src/ui/tabs/hardware.rs (1332L), crates/rocm-dash-tui/src/ui/tabs/home.rs (1935L), crates/rocm-dash-tui/src/ui/format.rs, and the docs/architecture.md diff — plus surrounding unchanged consumer code (agent/snapshot.rs, agent/tools.rs) and a repo-wide grep for any other raw power_w/gen_tps aggregation site that should have migrated (none found).
Two independent fan-out workers ran the coupled code passes (group 1: efficiency.rs + observe.rs + hardware.rs; group 2: efficiency.rs + home.rs + docs/architecture.md), each reading its files completely, blind to this PR's description and discussion; their candidates were verified against source and consolidated above. A history pass compared this change against predecessor PR #1017's own (already-merged, public) review thread, confirming this PR accurately continues from feedback left open there. AGENTS.md was read in full for agent-instruction adherence (module-size limits — all files well under 1,500 production lines; architecture-doc sync; the "don't pair independently-ordered lists by index" rule, not applicable here; state-matrix/test-verdict conventions) — no violation found beyond the items above. Code comments on every touched function were read and checked against behavior (finding above).
CI was read live for this head: clippy and all required static/security checks reported SUCCESS; several build/test/coverage jobs (build-and-test, Test (affected crates), Coverage, Windows jobs) were still IN_PROGRESS at the time of this review, with none failing.
Not reconciled against this PR's existing review thread.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed the diff; I found nothing blocking. Routing both Observe heroes through one throughput aggregate and skipping faulty power readings looks right, and the scope matches the ticket's "one implementation everywhere" goal.
Non-blocking notes:
- The Hardware header still prints "0.0 kW" when no GPU reports usable power, while Observe and Home now show a dash (inline comment below).
- Please double-check the
docs/architecture.mdclaim thatlist_instancesreachesnode_throughputthroughnode_efficiency. I could only confirm thatagent/snapshot.rsreferencesnode_efficiency. - Optional: add a test for a lone held 0 tok/s instance; it is only covered indirectly today.
- There is no Gherkin scenario for the user-visible
0.0 Wto dash change. The PR body's unit/render-level justification is reasonable to me, but a maintainer may want one.
CI: the red jobs (Coverage rocm-dash, E2E, Test affected crates, build-and-test) all failed at "Install native build deps" with an apt index download error, before any build ran. They need a rerun, and the rocm-dash coverage ratchet has not actually been evaluated yet. This approval reflects a diff read only, not CI results or runtime behavior.
| } | ||
| } | ||
| let total_power: f64 = snap.gpus.iter().map(|g| f64::from(g.power_w)).sum(); | ||
| let total_power = rocm_dash_core::efficiency::node_power_w(&snap.gpus); |
There was a problem hiding this comment.
The header still prints "0.0 kW" when no GPU reports usable power, while Observe and Home now show a dash for the same state. Consider a dash here for consistency, or note that it is intentional.
Follow-up to #1017 (ROCMAI-938), from an automated review that landed after that PR merged.
One throughput aggregate for both Observe heroes. The efficiency hero took its held-ness from
node_efficiency, which ignores instances at zero tok/s when judging freshness, while the throughput hero beside it kept an inline check that did not. For the same snapshot, a held instance at 0 tok/s next to a fresh one made the throughput hero show the held marker and legend and the efficiency hero show none. The throughput sum and its freshness now live innode_throughput(rocm-dash-core), used bynode_efficiencyand by the throughput hero, so the two cannot disagree. A real zero is still a figure (an idle node is zero throughput, not no data), and nothing finite is still no data.Faulty power readings in node efficiency.
node_efficiencysummed board power without the finite/positive guardattribute_power_wapplies, so one sensor reporting infinity made the quotient a plausible-looking 0 and a negative reading deflated the total. It now skips those readings the same way, and has no figure when none is usable.One filtered power sum everywhere. The finite/positive guard now lives in a shared
node_power_w, used bynode_efficiency, the Observe throughput hero, the Hardware header's kW figure and Home's power label and sparkline, so none of them hand-rolls the sum.Home's Node throughput hero moves onto
node_throughputtoo (it takes any iterator of instances, since Home holds a map), so a held instance at 0 tok/s no longer marks it.User-visible changes beyond the held-marker agreement: GPUs that report 0 W (no power telemetry) now show
—in the Observe throughput hero instead of0.0 W, and the Hardware header's kW and Home's power label and trace skip a faulty sensor reading instead of showinginf/negative totals. Home's tokens-per-watt figure is a separate sum of per-instance values and is unchanged.docs/architecture.mdnames who callsnode_throughput,node_power_wandnode_efficiency.Tests
node_throughput_and_node_efficiency_agree_on_freshness,node_throughput_distinguishes_idle_from_no_data,node_efficiency_skips_a_faulty_power_sensor_like_attribute_power_does(core) andthe_two_heroes_agree_for_a_held_instance_at_zero_throughput(Observe, which checks the two heroes' own text, since the Instances table below correctly marks the held row),throughput_hero_power_is_a_dash_when_gpus_report_no_usable_power(Observe),home_power_label_skips_a_faulty_sensor_like_the_trace_beside_it(asserts the label and the trace) andhome_throughput_is_not_marked_held_by_a_held_instance_at_zero_tok_s(Home), andgpu_section_header_kw_skips_a_faulty_power_sensor(Hardware). Each fails when its target is broken: removing the zero-throughput guard fails the first and the Observe test; removing the power filter fails the sensor test; putting the raw sum back into Home's label or trace, the Hardware kW figure, or the oldhas_powercheck fails the corresponding power test.No Gherkin scenario: the changes a user can see are corner cases (two heroes agreeing for a held zero-throughput instance, a dash instead of
0.0 Wwhen no GPU reports power, and a faulty sensor no longer showing asinf), which the dash scenarios do not reach; each is pinned at the unit and render level.Verification
cargo clippy --workspace --all-targets -- -D warningsclean andcargo test --workspace --all-targetspassing in 46 binaries, on the final commit.cargo clippy --workspace --all-targets -- -D warningsclean, 1557 tests in the dash crates pass, and the dashboard e2e (-n dash-) passes 50 of 51 scenarios. The one failure,dash-24, needsamd-smi, which does not run on WSL, and fails the same way on plainmain.