Skip to content

ROCMAI-938 follow-up: one throughput aggregate for both Observe heroes, skip faulty power readings - #1030

Merged
volen-silo merged 1 commit into
mainfrom
rocmai-938-followup
Nov 12, 2026
Merged

volen-silo merged 1 commit into
mainfrom
rocmai-938-followup

Conversation

@volen-silo

@volen-silo volen-silo commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #1017 (ROCMAI-938), from an automated review that landed after that PR merged.

One throughput aggregate for both Observe heroes. The efficiency hero took its held-ness from node_efficiency, which ignores instances at zero tok/s when judging freshness, while the throughput hero beside it kept an inline check that did not. For the same snapshot, a held instance at 0 tok/s next to a fresh one made the throughput hero show the held marker and legend and the efficiency hero show none. The throughput sum and its freshness now live in node_throughput (rocm-dash-core), used by node_efficiency and by the throughput hero, so the two cannot disagree. A real zero is still a figure (an idle node is zero throughput, not no data), and nothing finite is still no data.

Faulty power readings in node efficiency. node_efficiency summed board power without the finite/positive guard attribute_power_w applies, so one sensor reporting infinity made the quotient a plausible-looking 0 and a negative reading deflated the total. It now skips those readings the same way, and has no figure when none is usable.

One filtered power sum everywhere. The finite/positive guard now lives in a shared node_power_w, used by node_efficiency, the Observe throughput hero, the Hardware header's kW figure and Home's power label and sparkline, so none of them hand-rolls the sum.

Home's Node throughput hero moves onto node_throughput too (it takes any iterator of instances, since Home holds a map), so a held instance at 0 tok/s no longer marks it.

User-visible changes beyond the held-marker agreement: GPUs that report 0 W (no power telemetry) now show — in the Observe throughput hero instead of 0.0 W, and the Hardware header's kW and Home's power label and trace skip a faulty sensor reading instead of showing inf/negative totals. Home's tokens-per-watt figure is a separate sum of per-instance values and is unchanged.

docs/architecture.md names who calls node_throughput, node_power_w and node_efficiency.

Tests

node_throughput_and_node_efficiency_agree_on_freshness, node_throughput_distinguishes_idle_from_no_data, node_efficiency_skips_a_faulty_power_sensor_like_attribute_power_does (core) and the_two_heroes_agree_for_a_held_instance_at_zero_throughput (Observe, which checks the two heroes' own text, since the Instances table below correctly marks the held row), throughput_hero_power_is_a_dash_when_gpus_report_no_usable_power (Observe), home_power_label_skips_a_faulty_sensor_like_the_trace_beside_it (asserts the label and the trace) and home_throughput_is_not_marked_held_by_a_held_instance_at_zero_tok_s (Home), and gpu_section_header_kw_skips_a_faulty_power_sensor (Hardware). Each fails when its target is broken: removing the zero-throughput guard fails the first and the Observe test; removing the power filter fails the sensor test; putting the raw sum back into Home's label or trace, the Hardware kW figure, or the old has_power check fails the corresponding power test.

No Gherkin scenario: the changes a user can see are corner cases (two heroes agreeing for a held zero-throughput instance, a dash instead of 0.0 W when no GPU reports power, and a faulty sensor no longer showing as inf), which the dash scenarios do not reach; each is pinned at the unit and render level.

Verification

  • Windows: cargo clippy --workspace --all-targets -- -D warnings clean and cargo test --workspace --all-targets passing in 46 binaries, on the final commit.
  • WSL (Linux): cargo clippy --workspace --all-targets -- -D warnings clean, 1557 tests in the dash crates pass, and the dashboard e2e (-n dash-) passes 50 of 51 scenarios. The one failure, dash-24, needs amd-smi, which does not run on WSL, and fails the same way on plain main.

@volen-silo
volen-silo requested a review from a team as a code owner November 11, 2026 09:45
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice cleanup: both Observe heroes now go through one node_throughput, and the held/idle-vs-no-data distinction is tested with exact values. I found nothing blocking in the logic, but a few things should be tightened before merge (details inline):

  • The throughput hero still sums power_w unfiltered, while node_efficiency now skips non-finite/non-positive readings. The faulty-sensor class fixed next door is still reachable there.
  • The efficiency.rs entry in docs/architecture.md should mention the new power guard, and the claim that the Hardware header and list_instances use node_throughput should be double-checked.
  • The new Observe test asserts the unmarked figure with a literal "300.0 tok/s*" while the efficiency check uses format::HELD_MARKER. If the marker ever isn't *, the negative assertion becomes vacuous; using the constant in both places would avoid that.

CI is green on the required checks (GPU lanes skipped by the runner outage; the consolidated e2e report was still pending when I looked).

let tps = throughput.map_or(0.0, |t| t.tokens_per_sec);
let has_tps = throughput.is_some();
let any_held = throughput.is_some_and(|t| t.held());
let power: f32 = snap.gpus.iter().map(|g| g.power_w).sum();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hero still sums power_w unfiltered, while node_efficiency now skips non-finite/non-positive readings. A single faulty sensor would still skew whatever is derived from power here. Consider the same guard, or a shared helper so the power sum isn't hand-rolled in several places.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ff74b73a. One shared node_power_w (in rocm-dash-core) sums board power over the GPUs with a usable reading, skipping one that is not finite or not positive as attribute_power_w does. It is now used by node_efficiency, this hero, the Hardware header kW figure and Home's power sparkline, so none of them hand-rolls the sum. One visible effect: when no GPU reports usable power, this hero now shows — instead of 0.0 W, which fits a platform with no power source better.

Comment thread docs/architecture.md Outdated
- [`bench_schema.rs`](../crates/rocm-dash-core/src/bench_schema.rs) — `BenchmarkRow`, the one row type for a benchmark result (identity, platform, backend configuration, request statistics, throughput, latency, quality, rollup and provenance columns), and `PassFail`. Almost every field is an `Option` and the struct is `#[serde(default)]`, so a CSV or a recording that omits columns still deserializes. `CsvBenchTailer` deserializes CSV rows into it, `bench_load.rs` builds it, and the daemon forwards it in `Event::BenchmarkRowsAppended`.
- [`config.rs`](../crates/rocm-dash-core/src/config.rs) — the standalone dashboard configuration read from `<config dir>/rocm-dash/config.toml` (`default_config_path`; a missing file means defaults and `Config::load` fails only on a read or parse error): `Config` with `DaemonConfig` (listen address, token, tick intervals, benchmark CSV path), `TuiConfig` (connect address, theme, chat endpoint, model and auth-header name) and per-engine `EngineConfig`. It is separate from the unified `rocm` configuration (the `dashboard` section of `rocm-core`'s `RocmCliConfig`, which `rocm dash` reads); in the tree only the TUI's chat-endpoint save (`persist_chat_endpoint` in [`app/chat.rs`](../crates/rocm-dash-tui/src/app/chat.rs)) loads and writes this file. Its default socket path (`$XDG_RUNTIME_DIR`, then `~/.rocm/data/telemetry`, then a user-named directory under the temp dir) is a copy of `rocm-core`'s on purpose, because this crate may have no first-party dependencies (`cargo xtask check-crate-edges` enforces that).
- [`efficiency.rs`](../crates/rocm-dash-core/src/efficiency.rs) — the pure join of throughput and power. `normalize_gpu_id` strips the `gpu-` prefix so the bare indices Docker discovery records in `Instance.gpu_ids` match amd-smi's `gpu-N` device ids; `attribute_power_w` splits each GPU's board power among the instances that occupy it, weighted by per-container VRAM, equally when any occupant has no reading, and yields `None` for an instance none of whose GPUs reported a positive power reading; `tokens_per_watt` divides generation throughput by that attributed power. `node_efficiency` returns the whole-node figure as a `NodeEfficiency` (Σ `gen_tps` over Σ board power, including GPUs no instance occupies) with a `held` flag that is set when any throughput behind it is held; it is the one implementation behind the Observe efficiency hero and its trend, the Hardware header, and the assistant's `tokens_per_watt` tool (the Home tab's tokens-per-watt figure is a separate sum of per-instance values). The daemon runner calls `attribute_power_w`, `tokens_per_watt` and `gpu_ids_overlap`, which it uses to warn when the id join finds nothing.
- [`efficiency.rs`](../crates/rocm-dash-core/src/efficiency.rs) — the pure join of throughput and power. `normalize_gpu_id` strips the `gpu-` prefix so the bare indices Docker discovery records in `Instance.gpu_ids` match amd-smi's `gpu-N` device ids; `attribute_power_w` splits each GPU's board power among the instances that occupy it, weighted by per-container VRAM, equally when any occupant has no reading, and yields `None` for an instance none of whose GPUs reported a positive power reading; `tokens_per_watt` divides generation throughput by that attributed power. `node_efficiency` returns the whole-node figure as a `NodeEfficiency` (Σ `gen_tps` over Σ board power, including GPUs no instance occupies) with a three-state freshness (fresh, held, or unknown) worked out by `node_throughput`, which sums the node's throughput once and ignores instances at zero tok/s when judging freshness; both are the one implementation behind the Observe efficiency and throughput heroes and the efficiency trend, the Hardware header, and the assistant's `tokens_per_watt` and `list_instances` tools (the Home tab's tokens-per-watt figure is a separate sum of per-instance values). The daemon runner calls `attribute_power_w`, `tokens_per_watt` and `gpu_ids_overlap`, which it uses to warn when the id join finds nothing.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The efficiency.rs entry should also say that node_efficiency skips non-finite/non-positive board-power readings (the new behavior). Please also confirm the Hardware header and list_instances really go through node_throughput/node_efficiency, since this diff doesn't touch those consumers.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ff74b73a. The entry now says node_power_w skips non-finite and non-positive readings, and spells out who calls what. I checked the consumers against the code: the Hardware header (gpu_section_header) and both assistant tools (node_json, used by list_instances and tokens_per_watt) go through node_efficiency, which calls node_throughput, so they reach it indirectly; the Observe and Home throughput heroes call node_throughput directly. The diff does not touch those consumers because they already used node_efficiency.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 7233622e

Review — no blocking findings

Full review of the whole change (3 files: crates/rocm-dash-core/src/efficiency.rs, crates/rocm-dash-tui/src/ui/tabs/observe.rs, docs/architecture.md; merge-base eb2a8194→head 7233622e).
Follow-up to #1017 (ROCMAI-938 named in the description, not retrieved from any tracker here). The description's claims were checked against #1017's own post-merge automated review (public record on that already-merged PR): its one blocking finding (the two Observe heroes disagreeing on held-ness for a zero-throughput instance) and its non-blocking findings (missing power-finiteness guard, docs/architecture.md omitting list_instances) are each addressed here, with a new test pinning the specific scenario the blocking finding named.

Blocking

None

Non-blocking

  • The Node-throughput hero's own power readout isn't guarded against the sensor fault this PR just fixed for the efficiency quotient beside it — crates/rocm-dash-tui/src/ui/tabs/observe.rs:184 (let power: f32 = snap.gpus.iter().map(|g| g.power_w).sum();, unfiltered), crates/rocm-dash-tui/src/ui/format.rs:204-206 (watts(): format!("{value:.1} W"), no finite guard). node_efficiency's power sum now skips a non-finite or non-positive power_w (crates/rocm-dash-core/src/efficiency.rs:205-213), but the throughput hero a few lines below still sums every GPU's raw power unfiltered for its own "power" line. One GPU reporting Infinity/NaN/a negative value renders inf W/NaN W directly in that hero while the efficiency hero beside it (now protected) shows a sane figure — a new two-heroes-disagree case, in the same panel row, of the exact class this PR's own description says it is closing. hardware.rs's gpu_section_header sums the same way for its kW figure, so the gap isn't confined to this one line. Confidence 80 · architectural · Fix: filter this sum (and hardware.rs's) the same way, or share one filtered-power helper.

  • Home tab's own "Node throughput" hero still has the exact held/zero-throughput bug this PR fixes for Observe — crates/rocm-dash-tui/src/ui/tabs/home.rs:507-514 (any_tps_held computed with no v <= 0.0 exclusion, unlike the new node_throughput). A held instance reporting 0.0 tok/s next to a fresh busy one marks Home's own, identically-titled "Node throughput" hero as held — the same bug class this PR's title and tests fix for Observe's two heroes, left standing on a third surface sharing the name. Untouched by this diff and outside the PR's stated scope (Observe only), so not blocking, but directly pattern-relevant. Confidence 82 · logic · Fix: migrate Home's aggregate onto the shared node_throughput, or note the gap as a follow-up.

  • One sub-assertion in the new power-sensor test doesn't discriminate the guard it's named for — crates/rocm-dash-core/src/efficiency.rs:404-409 (node_efficiency_skips_a_faulty_power_sensor_like_attribute_power_does's only_bad case: gpu("gpu-0", f32::NAN), gpu("gpu-1", -5.0) → expects None). f64::from(NaN) + f64::from(-5.0) is NaN, and NaN > 0.0 is already false, so this assertion returns None whether or not the new finite/positive filter exists — deleting the filter does not turn this line red. The test's other two cases (inf, neg) do flip correctly under that mutation, so the test as a whole still catches a reverted fix, but this line documents the "no usable telemetry" case rather than proving the guard causes it. Confidence 72 · mechanical · Fix: none required for correctness; optional to swap in a case the filter alone discriminates (e.g. a lone Infinity reading).

Decisions for the author

  • node_efficiency's tokens_per_sec > 0.0 && power > 0.0 gate still collapses a genuinely idle, fully-telemetered node (real Σ gen_tps == 0) into the same None/— as true data absence — non-blocking-improvement. The new node_throughput this PR introduces already tracks "did anything contribute" separately from the sum's value (its own doc: "A real zero counts: an idle node is zero throughput, not no data", backed by node_throughput_distinguishes_idle_from_no_data), and the throughput hero now correctly shows a real 0.0 tok/s for this case — but node_efficiency discards that distinction immediately after calling it, still requiring tokens_per_sec > 0.0 to return a figure at all. This exact gap was raised twice on the preceding PR #1017 (non-blocking there too) and left as documented, apparently deliberate behavior; this PR rewrites the surrounding function without changing it. Worth a deliberate decision on whether to finally close it, since the tracking it would need already exists for free.

Positive signals

  • node_throughput's extraction preserves node_efficiency's prior behavior for every reachable case (traced the "no finite readings" and "all-exactly-zero" paths against the old inline loop by hand) while fixing the real cross-panel disagreement for a zero-tok/s held instance.
  • The new tests target real, distinct edge cases — idle vs. no-data, infinite vs. negative power sensor, cross-function freshness agreement — rather than restating the happy path, and three of the four are traceably mutation-sensitive.
  • docs/architecture.md's updated entry names every real call site of node_throughput/node_efficiency accurately — each one (Observe's two heroes, the efficiency trend, the Hardware header, both assistant tools) was traced against the actual code; no stale or missing caller.

Deployment notes

None — no migration, no breaking API change; node_throughput is a new, additive public function.

What this covered

Every line of crates/rocm-dash-core/src/efficiency.rs (591 lines) and crates/rocm-dash-tui/src/ui/tabs/observe.rs (584 lines) read completely at the PR head, including all test modules, plus docs/architecture.md's diff and the call sites its new wording names (hardware.rs, agent/snapshot.rs, agent/tools.rs, home.rs). Split across three independent fan-out workers: (1) the coupled core+UI code pass, (2) the docs/cross-file-consistency pass, (3) a history and prior-changes pass against this code's own git log and PR #1017's already-merged, public review history. AGENTS.md read in full and checked (module-size, architecture-doc sync, markdown-link, and test-verdict rules); no violation found beyond the items above. eb2a8194…7233622e.
CI re-checked live on this head: every required check SUCCESS (self-hosted GPU E2E lanes SKIPPED, consistent with the description's noted runner outage; the consolidated E2E report itself is SUCCESS).
Did not run: no Rust toolchain available in this sandbox, so cargo test/cargo clippy were not executed — verification above is static, by tracing code paths, arithmetic and existing tests by hand; confidence is capped accordingly. No Gherkin/e2e scenario exists for this change and the description explains why; not independently disputed.
Not reconciled against this PR's existing review thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks both. All of it is in ff74b73a; the inline replies are on the threads. The rest:

  • Held-marker assertion: the Observe test now uses format::HELD_MARKER in the negative assertion for the throughput figure too, so it cannot go vacuous if the marker changes.
  • Home's Node throughput hero (the same bug on a third surface): migrated onto node_throughput (which now takes any iterator of instances, since Home holds a map), so a held instance at 0 tok/s no longer marks it. New test home_throughput_is_not_marked_held_by_a_held_instance_at_zero_tok_s also checks that a held instance that does contribute still marks it; putting the old inline check back makes it fail.
  • The sensor test: the NaN-and-negative case returned None with or without the filter, as pointed out. Added a lone-infinity case that only the filter decides (unfiltered, tps / inf is a finite 0), and a unit test for node_power_w itself.

Not changed

  • An idle node with real zero throughput still gives no efficiency figure (— on screen, null for the assistant). node_throughput tracks the distinction, but — is the right display when no tokens are produced, and this was raised on ROCMAI-938: one node-efficiency implementation in rocm-dash-core, with a held flag on every surface #1017 too. Left as a deliberate gate rather than a quiet change here.
  • Home's tokens-per-watt aggregate keeps its own held logic. It is a sum of per-instance values, a different quantity, and it has the same zero-instance quirk; that would be its own change.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 11, 2026 11:34

Addressed in ff74b73: the throughput hero, the Hardware header and Home now use one shared node_power_w, the architecture doc says what each consumer calls and covers the power guard, and the held-marker assertion uses HELD_MARKER. Every point has a reply: #1030 (comment)

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared throughput aggregate removes the duplication between the two heroes, and the new tests cover idle vs. no-data, held vs. idle, and the Inf-only power case. Two things need fixing before merge.

Blocking

  1. The Home power label still sums raw readings (home.rs, the let power: f64 = ... .sum() feeding power_label), while the sparkline right below it now uses node_power_w. One Inf or negative sensor shows in the label but is skipped in the trace. The new docs/architecture.md text says node_power_w is used for the Observe, Hardware and Home power figures, which is false for the Home label. This is the same class of bug the PR fixes. Switch the label to node_power_w (the power == 0.0 => "-" rule still works) and add a test with an Inf GPU, or narrow the doc.
  2. The PR body omits user-visible changes in the diff: the Home throughput hero's held rule, the Home sparkline and Hardware kW header moving to the filtered sum, and the Observe hero's has_power change (GPUs reporting 0 W now show "-" instead of "0 W"). Please update the body to match the final diff, and mention the has_power change in the no-scenario justification.

Non-blocking

  • The doc comment on node_efficiency in efficiency.rs is badly reflowed into one very long line.
  • The Observe tuple is now a 5-tuple mixing f64/f32; a small struct would read better.
  • Please confirm intent: a node held only at zero tok/s reports a fresh 0 tok/s, while node_efficiency returns None for it.

CI has no failing checks; several are still pending. I did not run the tests or the break-the-target checks myself.

// Power can exceed 100; scale to a nominal 1kW ceiling for the trace.
&history(state, |s| {
s.gpus.iter().map(|g| f64::from(g.power_w)).sum::<f64>() / 10.0
rocm_dash_core::efficiency::node_power_w(&s.gpus) / 10.0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sparkline now uses node_power_w, but the power label a few lines above (let power: f64 = latest.map_or(0.0, |s| s.gpus.iter().map(..).sum())) still sums raw readings, so one Inf/negative sensor shows in the label and is skipped in the trace. architecture.md says Home uses node_power_w too. Please switch the label (and add a test with an Inf GPU) or narrow the doc.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 89bdf121. Home's power label now uses node_power_w, like the sparkline, so a faulty sensor no longer shows in one and not the other (the power == 0.0 => — rule still holds). New test home_power_label_skips_a_faulty_sensor_like_the_trace_beside_it puts an infinite GPU next to a usable 420 W one and checks the label shows 420.0 W, never inf W; putting the raw sum back makes it fail. The architecture doc is accurate now, and was re-applied onto main's new per-crate layout.

let has_tps = throughput.is_some();
let any_held = throughput.is_some_and(|t| t.held());
let power = rocm_dash_core::efficiency::node_power_w(&snap.gpus);
(tps, has_tps, power, power > 0.0, any_held)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Behavior change: has_power was !snap.gpus.is_empty() and is now power > 0.0, so GPUs reporting 0 W (no telemetry) now render "-" instead of "0 W". Probably right, but it is user-visible and not in the PR body; consider a test pinning it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinned in 89bdf121: throughput_hero_power_is_a_dash_when_gpus_report_no_usable_power checks that GPUs reading 0 W show power — (not 0.0 W) and that a usable reading still prints; reverting has_power to the old non-empty check fails it. The change is also now in the PR body, including in the no-scenario justification.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head ff74b73a160a

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change. Implements a scoped, accurate follow-up to #1017 (ROCMAI-938): unifies the Observe tab's two held-ness checks behind a new node_throughput helper, and adds node_efficiency's missing finite/positive power-sensor guard. The PR description's own claims match the code exactly; it does not itself claim the Home tab's headline power figure is fixed — only docs/architecture.md does, and that claim is not accurate (see Blocking).

Blocking

  • Home tab's headline POWER figure still uses an unfiltered sum, disagreeing with its own trend line beneath it, and with this PR's own doc update — crates/rocm-dash-tui/src/ui/tabs/home.rs:446-447 (and its use at :467 for power_label), vs. crates/rocm-dash-tui/src/ui/tabs/home.rs:501-503.
    The Home "POWER" card's headline number is computed as s.gpus.iter().map(|g| f64::from(g.power_w)).sum::<f64>() — no finite/positive guard. Four lines later, the same card's trend/sparkline history for the identical quantity was migrated by this PR to rocm_dash_core::efficiency::node_power_w(&s.gpus), which does apply that guard. One GPU reporting a faulty reading (infinity, NaN, or a negative value — exactly the fault class this PR targets) makes the headline wrong (format::watts's {value:.1} W would render e.g. "inf W" or a negative wattage) while the trend line directly beneath it, drawn from the same snapshot, stays correct — the two halves of one card can visibly disagree for the same tick. This also falsifies the PR's own docs/architecture.md update to the efficiency.rs bullet, which states node_power_w "is used wherever the node's power is shown or divided by (the Observe, Hardware and Home power figures)" — the Home headline is the one place that isn't true.
    Confidence 85 · logic · Fix: route the headline power computation through rocm_dash_core::efficiency::node_power_w(&s.gpus) as well, so both halves of the card — and the architecture.md claim — match reality.

Non-blocking

None

Decisions for the author

  • Splitting the aggregate into two free functions invites a silent partial migration — non-blocking-improvement
    node_throughput and node_power_w must both be called at every site that shows the node's power/throughput; forgetting one (as happened above) still compiles, since the old raw .sum() is still valid Rust. Worth a cheap guard against recurrence — e.g. a repo-wide check (in the spirit of the existing check-crate-edges/check-module-size xtask checks) that flags a raw power_w sum or gen_tps fold outside efficiency.rs.
  • Two test gaps in efficiency.rs's node_throughput, both below this review's confidence bar to report as defects — non-blocking-improvement
    No test exercises a negative gen_tps reading (asymmetric against node_power_w's explicit negative-value guard — node_throughput's doc comment only promises robustness for zero tok/s, and gen_tps realistically shouldn't go negative, so this reads as an out-of-scope input rather than a bug). Also no test combines a contributing Held instance with a contributing Unknown-freshness instance (only Held+Fresh and Fresh+Unknown are covered) to confirm Held still wins over Unknown absent any Fresh instance.

Positive signals

  • node_throughput's freshness precedence (Held > Unknown > Fresh, with zero/non-finite throughput excluded from influencing it at all) is subtle, documented precisely, and exercised by tests hitting every combination.
  • node_power_w's filter is kept in deliberate, doc-cross-referenced parity with the pre-existing attribute_power_w filter (verified bit-for-bit equivalent, including -0.0 and subnormal boundary cases) — good discipline against the two aggregates silently drifting apart.
  • node_throughput takes impl IntoIterator<Item = &Instance> rather than a concrete collection, letting one implementation serve both Snapshot's Vec<Instance> and AppState's HashMap::values() call sites without duplicating the aggregation logic.

Deployment notes

None — internal dashboard TUI logic only; no migration, no external API or config change.

What this covered

Read completely: all 5 changed files (crates/rocm-dash-core/src/efficiency.rs, crates/rocm-dash-tui/src/ui/tabs/hardware.rs, crates/rocm-dash-tui/src/ui/tabs/home.rs — in full, ~1,878 lines — crates/rocm-dash-tui/src/ui/tabs/observe.rs, docs/architecture.md), plus surrounding context (metrics.rs's Instance/GpuMetrics/ObservationFreshness, format.rs's watts/gen_tps_aggregate/tokens_per_watt, agent/snapshot.rs's assistant-tool consumers), and a repo-wide search for every other site touching gen_tps/power_w aggregation to check for further missed migrations (none found beyond the one reported). Base 852ce7b0…head ff74b73a, fetched read-only (no checkout).

efficiency.rs was additionally reviewed by an independent fanned-out background worker (fresh context, no access to this transcript or the above); its report is folded into the findings and decisions above — no blocking findings from that pass.

The TUI-consumer pass (home.rs/observe.rs/hardware.rs) could not be run as an independent fan-out worker: two spawn attempts both hit a shared concurrency limit, and only one retry is taken before falling back per method. That pass was run by me (the driver) instead, so it is not independent — the Blocking finding above comes from that non-independent pass.

Did not run: the history pass (git log/git blame) — the read-only clone used shallow fetches of only the needed commits, so no prior history was available to mine. The prior-changes pass — no earlier review comments on these files were consulted (by design, blind to this PR's own discussion). CI was read live for commit ff74b73a160a3e7639f2f77b6ebb1d66a3e8e96a: all 46 check runs completed, none failing (several skipped, expected for hardware-gated e2e lanes) — no CI-blocking finding. Ticket ROCMAI-938 is named but was deliberately not retrieved: this is a public repository and the identifier reads as an internal tracker reference, so it was left alone even though a tracker tool was technically available. The named GitHub PR #1017 (same repo, public) was read instead, confirming #1030 is a scoped bug-fix follow-up to post-merge review feedback on #1017, consistent with this PR's description.

Not reconciled against any discussion or prior review thread on this PR.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks. Both blocking points and the non-blocking ones are in 89bdf121 (also rebased onto the restructured docs/architecture.md); inline replies are on the threads.

  • Home power label: switched to node_power_w, with a test using an infinite GPU (see the thread). The doc sentence about Home is now true.
  • PR body: updated to the final diff: the Home throughput hero's held rule, the Home label and trace and the Hardware kW header moving to the filtered sum, and the Observe has_power change, which is also in the no-scenario justification.
  • Doc comment: the node_efficiency comment is reflowed into paragraphs.
  • 5-tuple: replaced by a small ThroughputHero struct.
  • Intent, as asked: a node held only at zero tok/s reports a fresh 0 tok/s from node_throughput (nothing contributes, so nothing is held) and no efficiency figure from node_efficiency (nothing to divide). That is deliberate, and the node_efficiency doc comment now says so.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 11, 2026 14:05

Addressed in 89bdf12: Home's power label now uses node_power_w (with a faulty-sensor test), the PR body matches the final diff including the has_power change, and the doc comment and tuple are tidied. Every point has a reply: #1030 (comment)

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 89bdf12. Nothing here looks functionally wrong; the shared node_throughput / node_power_w helpers taking impl IntoIterator is a good consolidation, and the fixtures use values the real source can produce (inf, NaN, negative, 0).

Changes requested, both about tests that don't pin what this PR changes:

  1. The Hardware kW header now uses the filtered node_power_w, with no test that fails if it is reverted.
  2. The Home test asserts the power label only, while its comment claims agreement with the sparkline's filtered sum. Reverting the sparkline closure would stay green.

Smaller notes:

  • power as f32 in the throughput hero narrows an f64 sum only to feed format::watts (inline).
  • The docs/architecture.md statement that the assistant's list_instances / tokens_per_watt tools go through node_throughput: please double-check it against the code, I could not confirm it from the diff.

Please confirm these are intended:

  • A node whose only throughput is a held zero shows a fresh, unmarked 0 tok/s on the throughput hero (matches the ROCMAI-938 criteria as I read them).
  • Hero power now means "usable power > 0", so GPUs without telemetry show — instead of 0.0 W.

CI: the red Linux jobs (build-and-test, Test (affected crates), E2E, Coverage) all fail at "Install native build deps" with apt DNS resolution errors, so no Linux test step has run on this head. They need a re-run before this can be judged green.

}
}
let total_power: f64 = snap.gpus.iter().map(|g| f64::from(g.power_w)).sum();
let total_power = rocm_dash_core::efficiency::node_power_w(&snap.gpus);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kW header now uses the filtered node_power_w, but nothing pins it: reverting this line leaves every test green. Please add a render test beside the Home one with a faulty/negative sensor next to a usable one, and check it goes red when the line is reverted.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinned in 2dd9e687: gpu_section_header_kw_skips_a_faulty_power_sensor puts a usable 500 W next to an infinite and a negative reading and checks the header says 0.5 kW, never inf or a total deflated by the negative one. With the line reverted to the raw sum it fails.

let tps_str = format::gen_tps_aggregate(has_tps.then_some(tps), any_held);
let power_str = if has_power {
format::watts(power)
format::watts(power as f32)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

power as f32 narrows the f64 sum back just for format::watts. If watts took f64 the cast could go. Minor / optional.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2dd9e687. format::watts takes f64 now, so both as f32 casts (here and in Home) are gone.

);
}

#[test]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment in this test says the label agrees with the sparkline's filtered sum, but only the label is asserted. Reverting the history(...) closure to a raw sum would not fail it. Please assert the trace too, or drop the claim from the comment.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2dd9e687, by asserting the trace rather than dropping the claim. The trace is now its own power_trace function, and the test pushes a snapshot with a 420 W and an infinite GPU into the history and checks the trace is [42] (a raw sum would clamp to 100) as well as the label. Putting the raw sum back into power_trace fails it with "the trace must skip it as well".

… faulty power readings in node efficiency (ROCMAI-938 follow-up)

Follow-up to the node-efficiency change. Two things a review found after merge:

The efficiency hero took its held-ness from node_efficiency, which ignores
instances at zero tok/s, while the throughput hero beside it kept an inline
check that did not. A held instance at 0 tok/s next to a fresh one made the
throughput hero show the held marker and legend and the efficiency hero show
none, for the same snapshot.

Move the throughput sum and its freshness into node_throughput, used by
node_efficiency and by the throughput hero, so the two cannot disagree. A real
zero is still a figure (an idle node), and nothing finite is still no data.

node_efficiency also summed board power without the finite/positive guard that
attribute_power_w applies, so one sensor reporting infinity made the quotient a
plausible-looking 0 and a negative reading deflated the total. It now skips
those readings the same way, and has no figure when none is usable. The
architecture doc names both assistant tools and the new helper.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks. Everything in this round is in 2dd9e687; the inline replies are on the threads. The rest:

  • Hardware kW and Home trace tests: both added/extended as asked, and I checked each goes red when its line is reverted to the raw sum.
  • power as f32: gone; format::watts takes f64.
  • list_instances / tokens_per_watt and node_throughput: the architecture entry is accurate as written. Both tools build their node object in snapshot.rs's node_json, which calls rocm_dash_core::efficiency::node_efficiency, and that calls node_throughput, so they reach it through node_efficiency; the entry says exactly that and names only the two throughput heroes as direct callers. The diff does not touch those consumers because they already went through node_efficiency.
  • Intent, both confirmed: a node whose only throughput is a held zero shows a fresh, unmarked 0 tok/s on the throughput hero (nothing contributes, so nothing is held), and hero power means "usable power above 0", so GPUs without telemetry show — instead of 0.0 W.
  • Red Linux CI jobs: the earlier head failed at "Install native build deps" with apt DNS errors, so no test step ran there. This push starts fresh runs; if a job fails the same way I will re-run it and say so here.
  • Local runs on this commit: on Windows the dash crates and workspace clippy pass; on WSL (Linux) workspace clippy is clean, 1557 tests in the dash crates pass, and the dashboard e2e passes 50 of 51 (dash-24 needs amd-smi, which does not run on WSL, and fails the same way on main).

@volen-silo
volen-silo dismissed juhovainio’s stale review November 11, 2026 16:07

Addressed in 2dd9e68: the Hardware kW and Home trace now have tests that fail when reverted, the f32 cast is gone, and the consumers and intent questions are answered. Every point has a reply: #1030 (comment)

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 2dd9e6873b68

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change. Follow-up to #1017 (ROCMAI-938, named in the title/description but not retrieved from any tracker — not accessible from here). The description's claims match the code: it does not itself claim the Hardware header gained a "no telemetry" dash state, or that Home's tok/W hero was migrated — only docs/architecture.md implies the former's sibling behavior; see Blocking.

Blocking

  • Hardware header's kW figure never shows "—" for "no usable power telemetry", unlike the Observe and Home power figures this same PR gave that treatment — crates/rocm-dash-tui/src/ui/tabs/hardware.rs:405-408. total_power comes from node_power_w, whose own doc comment states "zero means 'no telemetry'", and both observe.rs's throughput hero (has_power: power > 0.0) and home.rs's power label (if unavailable || power == 0.0 { "—" } else { ... }) were given that exact guard in this diff. Hardware's gpu_section_header instead formats total_power unconditionally as format!(" · {:.1} kW", total_power / 1000.0), so an all-zero-power snapshot (a platform with no power source, or three sensors all filtered out as faulty) prints "0.0 kW" — a real, measured-looking reading — while the two sibling screens correctly show a dash for the identical condition. The repo's own existing test draw_handles_degraded_and_empty_states (hardware.rs, case 3) already constructs exactly this all-power_w == 0.0 snapshot and asserts "eff -", but makes no assertion on the kW figure, so the gap is untested. Confidence 85 · logic · Fix: guard the kW span the same way (if total_power > 0.0 { format!(...) } else { "—".to_string() }).

  • Home's two hero aggregates (tok/s and tok/W) now disagree with each other on "held" semantics, a fresh inconsistency this PR introduces within Home's own hero band — crates/rocm-dash-tui/src/ui/tabs/home.rs:381-386 (any_tpw_held, untouched by this diff) vs. home.rs:515-517 (any_tps_held, migrated by this diff). any_tps_held now comes from node_throughput(..).held(), which — per this PR's whole point — excludes an instance contributing a real zero from marking the figure held. any_tpw_held a few dozen lines above still uses the old pattern: i.tokens_per_watt.is_some_and(f64::is_finite) && ...Held, with no exclusion for a zero-contributing instance (and tokens_per_watt()'s own doc at efficiency.rs:244-245 confirms a real Some(0.0) is a valid value, exactly analogous to a real zero tok/s). Before this PR, Home's tok/s hero used the identical unguarded pattern any_tpw_held still uses today, so the two Home heroes used to agree (both wrong in the same way); after this PR, a held-but-idle instance next to a fresh busy one makes Home's tok/s hero show an unmarked figure while its tok/W hero, one card over, shows the held marker — the exact cross-hero disagreement this PR's title and tests exist to eliminate, now reproduced inside Home instead of between Observe's two heroes. Confidence 85 · logic · Fix: give any_tpw_held the same "zero tok/W instances don't mark held" exclusion node_throughput applies, or route it through a shared helper.

Non-blocking

  • node_throughput has no guard against a negative (but finite) gen_tps reading, the same bug class this PR just guards against for power — crates/rocm-dash-core/src/efficiency.rs:160-168. Only is_finite() filters the sample; a negative reading is summed into tokens_per_sec unfiltered, silently deflating (or flipping the sign of) the total — the identical failure mode node_power_w's new w > 0.0 filter was built to close for power. Present unchanged in the pre-PR file too, so not a regression, but this PR touches this exact function to close the matching gap on the power side and leaves it open on the throughput side. Confidence 70 · logic · Fix: decide whether a negative gen_tps is a realistic input; if so, filter it the same way node_power_w does.

  • docs/architecture.md's updated efficiency.rs entry overclaims for Home — docs/architecture.md:396, in the sentence "node_power_w ... is what the Observe, Hardware and Home power figures show and divide by." Home only shows node_power_w (the power label and sparkline); Home's own tokens-per-watt figure divides per-instance attribute_power_w values, not node_power_w — which the same sentence concedes two clauses later ("the Home tab's tokens-per-watt figure is a separate sum of per-instance values"). Confidence 70 · mechanical · Fix: narrow the "divide by" clause to Observe and Hardware only.

  • Stale doc comment on draw_hero_band — crates/rocm-dash-tui/src/ui/tabs/observe.rs:69-71 says the band "Reuses rocm_dash_core::efficiency::node_efficiency", but draw_throughput_hero (the other half of the band, touched by this diff) now centrally depends on node_throughput and node_power_w too. Confidence 70 · mechanical · Fix: mention all three.

Decisions for the author

  • node_power_w returns a bare f64 with an implicit "0.0 = no telemetry" sentinel, while its sibling node_throughput returns Option<NodeThroughput> — tradeoff. The asymmetry is exactly what let the Hardware gap above through uncaught: a caller can "forget" the sentinel and the code still compiles. A uniform Option<f64> would force every caller to handle "no data" explicitly, at the cost of a slightly less convenient API for a plain sum.
  • node_power_w and attribute_power_w each inline the same "finite && positive" filter predicate separately rather than sharing one named helper — non-blocking-improvement. Both are currently correct and verified consistent, but nothing ties them together if one is edited later.

Positive signals

  • node_throughput's extraction is a faithful, verifiable lift of node_efficiency's old inline loop; node_efficiency now composes node_throughput + node_power_w exactly as its doc says, and the old duplicate inline loop in observe.rs is fully removed with no dead code left behind.
  • State-matrix test coverage (held/fresh/unknown-freshness × real-zero/no-data, for both throughput and power, including NaN/Infinity/negative-sensor cases) is thorough and asserts exact values rather than weak substring checks; several are genuine fail-before/pass-after regression tests, including two added in a late push to this PR (gpu_section_header_kw_skips_a_faulty_power_sensor, and the power_trace assertion added alongside home_power_label_skips_a_faulty_sensor_like_the_trace_beside_it).
  • No index-based pairing of independently-ordered lists: both shared aggregates are commutative (sum / OR-of-flags), so Home's HashMap::values() iteration order can't introduce a correlation bug.
  • docs/architecture.md's updated entry correctly names every other real call site this PR introduces (Observe's two heroes, the Hardware header, both assistant tools), verified against the actual code.

Deployment notes

None — internal dashboard TUI logic only; no migration, no external API or config change; node_throughput/node_power_w are new, additive public functions.

What this covered

Base f536619a08a3b50dd26aac5c21a3bb9f6bd266d2…head 2dd9e6873b68842f66dd74b24bf5f555ae4c597d, fetched read-only (no checkout). The PR was force-pushed to this head partway through this review (from 89bdf1211d...); the full diff and all touched files were re-read and re-verified against the new head, including the amendment itself (the format::watts signature widened from f32 to f64, a new Hardware kW regression test, and a new power_trace helper + assertion in home.rs).

All 6 changed files read completely: crates/rocm-dash-core/src/efficiency.rs (627L), crates/rocm-dash-tui/src/ui/tabs/observe.rs (627L), crates/rocm-dash-tui/src/ui/tabs/hardware.rs (1332L), crates/rocm-dash-tui/src/ui/tabs/home.rs (1935L), crates/rocm-dash-tui/src/ui/format.rs, and the docs/architecture.md diff — plus surrounding unchanged consumer code (agent/snapshot.rs, agent/tools.rs) and a repo-wide grep for any other raw power_w/gen_tps aggregation site that should have migrated (none found).

Two independent fan-out workers ran the coupled code passes (group 1: efficiency.rs + observe.rs + hardware.rs; group 2: efficiency.rs + home.rs + docs/architecture.md), each reading its files completely, blind to this PR's description and discussion; their candidates were verified against source and consolidated above. A history pass compared this change against predecessor PR #1017's own (already-merged, public) review thread, confirming this PR accurately continues from feedback left open there. AGENTS.md was read in full for agent-instruction adherence (module-size limits — all files well under 1,500 production lines; architecture-doc sync; the "don't pair independently-ordered lists by index" rule, not applicable here; state-matrix/test-verdict conventions) — no violation found beyond the items above. Code comments on every touched function were read and checked against behavior (finding above).

CI was read live for this head: clippy and all required static/security checks reported SUCCESS; several build/test/coverage jobs (build-and-test, Test (affected crates), Coverage, Windows jobs) were still IN_PROGRESS at the time of this review, with none failing.

Not reconciled against this PR's existing review thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff; I found nothing blocking. Routing both Observe heroes through one throughput aggregate and skipping faulty power readings looks right, and the scope matches the ticket's "one implementation everywhere" goal.

Non-blocking notes:

  • The Hardware header still prints "0.0 kW" when no GPU reports usable power, while Observe and Home now show a dash (inline comment below).
  • Please double-check the docs/architecture.md claim that list_instances reaches node_throughput through node_efficiency. I could only confirm that agent/snapshot.rs references node_efficiency.
  • Optional: add a test for a lone held 0 tok/s instance; it is only covered indirectly today.
  • There is no Gherkin scenario for the user-visible 0.0 W to dash change. The PR body's unit/render-level justification is reasonable to me, but a maintainer may want one.

CI: the red jobs (Coverage rocm-dash, E2E, Test affected crates, build-and-test) all failed at "Install native build deps" with an apt index download error, before any build ran. They need a rerun, and the rocm-dash coverage ratchet has not actually been evaluated yet. This approval reflects a diff read only, not CI results or runtime behavior.

}
}
let total_power: f64 = snap.gpus.iter().map(|g| f64::from(g.power_w)).sum();
let total_power = rocm_dash_core::efficiency::node_power_w(&snap.gpus);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header still prints "0.0 kW" when no GPU reports usable power, while Observe and Home now show a dash for the same state. Consider a dash here for consistency, or note that it is intentional.

@volen-silo
volen-silo added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 8551cde Nov 12, 2026
65 of 72 checks passed
@volen-silo
volen-silo deleted the rocmai-938-followup branch November 12, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants