Skip to content

ROCMAI-941: attribute VRAM and power to managed rocm serve services by process ancestry - #1026

Merged
volen-silo merged 1 commit into
mainfrom
rocmai-941-managed-vram
Nov 12, 2026
Merged

volen-silo merged 1 commit into
mainfrom
rocmai-941-managed-vram

Conversation

@volen-silo

Copy link
Copy Markdown
Collaborator

Implements ROCMAI-941, a follow-up to ROCMAI-62 (#812).

VRAM is attributed per instance by joining amd-smi's GPU process PIDs to the docker/containerd id in /proc/<pid>/cgroup. A managed rocm serve service is not a container, so it never joined: every managed service showed the whole device's VRAM, two services on one GPU each showed all of it, and the VRAM-weighted power split never applied to them (#812 only stopped this raising a false warning).

What changes

  • ancestry.rs (collectors): a service record carries the PIDs of its engine and launcher and their kernel start times. owner_of walks a GPU process's /proc/<pid>/stat parent links (bounded depth, reader injected so it is testable on any host) until it reaches a recorded root, and returns that service's id. A root whose live start time differs from the recorded one is a recycled PID and claims nothing; an unreadable or looping chain ends with no owner.
  • Registry and runner: ServiceRecord reads the recorded pids, discover_managed_services collects the roots of live services, and the runner tries the cgroup join first and ancestry second, so container ids and service ids key the one per-container VRAM map the instances already read.
  • Power split for starting instances: an occupant that is still Starting with no VRAM reading, while a neighbour has one, is left out of the split. Before, it took an equal share, so a serving neighbour showed about half its real power and roughly double its tokens per watt until the newcomer loaded. With no readings at all, or a serving occupant that has no reading, the split stays equal. Unknown status is not treated as starting.

Linux only: elsewhere ancestry yields no owner and attribution is unchanged.

Tests

ancestry unit tests (worker to owning service, two services told apart, recycled PID, unreadable links, cycles, self-parent, awkward process names), registry tests for the recorded pids and roots, the attribute_power_w cases above, and managed_services_get_their_own_vram_and_a_weighted_power_split: a Linux integration test that runs the real daemon loop with two real "engine" shells, each with a real child process standing in for a GPU worker, against a fake amd-smi. It asserts each service shows its own VRAM (3000 and 1000 MB, not the device total) and a 300/100 W split with no false vram-attribution warning. With the ancestry roots disabled it fails with both services at (4000, 4000), the symptom this fixes.

No Gherkin scenario: the dash scenarios need a PTY and a managed vLLM with real GPU process attribution, which no lane has. The daemon-level test above covers the behaviour end to end below the TUI.

Verification

  • Windows: cargo clippy --workspace --all-targets -- -D warnings clean; cargo test --workspace --all-targets passes in 46 binaries; dash crates 1488 tests pass.
  • WSL (Linux): 1518 tests in the dash crates pass, including the real-process integration test; the dashboard e2e (-n dash-) passes 48 of 49 scenarios. The one failure, dash-24, needs amd-smi, which does not run on WSL, and fails the same way on plain main.
  • Not run: real rocm serve on a GPU host with amd-smi (WSL has none), so the attribution is exercised against a fake amd-smi and real process trees, not real hardware.

@volen-silo
volen-silo requested a review from a team as a code owner November 10, 2026 16:45

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (12e87e84), read against main at 3ab57202.

Blocking

  • The branch conflicts with main, and CI has never run on this commit — crates/rocm-dash-core/src/efficiency.rs, docs/architecture.md (PR-level)
    GitHub reports the PR as CONFLICTING. git merge-tree against current main conflicts in exactly those two files: #1017 added node_efficiency and a new use line next to the attribute_power_w edits here, and rewrote the same efficiency.rs bullet in architecture.md. Because of the conflict pull_request workflows were never created: the Actions API returns 0 runs for this head SHA and the rollup holds only the docs-build status, while other PRs opened today have CI, CodeQL, security-scan and self-hosted runs. So no required check has been seen for this change, and the Windows/WSL results in the description are from the pre-conflict base, not the merged result.
    Confidence 100 · mechanical · Fix: merge or rebase onto main, resolve the two files (the conflicts are adjacent additions, not semantic), then let CI run to green on the result.

Non-blocking

One finding, attached inline on crates/rocm-dash-daemon/src/registry.rs:119.

Decisions for the author

None

Positive signals

  • The injected /proc reader in owner_of and the start-time check for a recycled PID keep the matcher pure and testable on any host.
  • The real-process integration test fails with the roots disabled (both services at the device total), so it does prove the symptom it targets.

Deployment notes

None. The new ServiceRecord fields are #[serde(default)], so older registry records still load.

What this covered

Every file in the diff, completely, plus the rocm crate's writers of supervisor_pid/engine_pid/*_start_ticks and rocm_core::process_start_ticks, to check the units and meaning match. Ran cargo test -p rocm-dash-collectors -p rocm-dash-core -p rocm-dash-daemon on the PR head in a private clone (Linux): all pass, including the new ancestry, registry, attribute_power_w and managed_services_get_their_own_vram_and_a_weighted_power_split tests. I did not run the mutation check on the new Starting filter, and the merged result was not built because of the conflict above.
The PR's existing discussion was not read before the findings were reached (there were no review threads or comments on it).

Comment thread crates/rocm-dash-daemon/src/registry.rs Outdated
if let Some(pid) = self.engine_pid.filter(|p| *p > 0) {
roots.push(root(pid, self.engine_start_ticks));
}
if self.supervisor_pid > 0 {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (confidence 85): a recorded supervisor_pid of 1 becomes a root that owns every process on the host.

ancestry_roots only rejects 0 (> 0). owner_of (ancestry.rs:65) matches a root by PID and, with start_ticks: None, accepts it unconditionally. Every host process descends from PID 1, so with supervisor_pid: 1 and no start time, any GPU process that is not under an earlier root (a user's own training job, another tool) resolves to this service and its VRAM is added to the service's reading. The new integration test uses exactly that record shape ("supervisor_pid":1 at telemetry_contract.rs:862) and only passes because each worker meets its engine root before it reaches PID 1, so it cannot catch this. The comment on the zero case ("never a root that could match init") shows the intent; PID 1 is the init case.

Suggest > 1 for the supervisor and the engine, and a test that a worker outside every engine tree gets no owner when a service record has supervisor_pid: 1. Worth also using a non-1 supervisor PID in the integration fixture.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c915e0c5, and it was a real bug. A root is now only a PID of 2 or more, enforced in two places: ServiceRecord::ancestry_roots (an engine or supervisor recorded as 0 or 1 yields no root) and owner_of itself (a root at 0 or 1 never matches, whatever the caller passes). New tests: a worker outside every service tree gets no owner when a record has supervisor_pid: 1, and a real root beside a bogus one still resolves. The integration fixture no longer uses PID 1 as filler; it records no supervisor at all.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 12e87e84

Review — needs work

Full review of the whole change (12e87e84), diffed against its merge-base with main (6862dd3f). Three independent fanned-out passes covered: (1) the ancestry/VRAM-attribution feature (ancestry.rs, lib.rs, registry.rs, runner.rs, telemetry_contract.rs), (2) the starting-occupant power-split change (efficiency.rs), and (3) agent-instruction adherence, history/divergence, prior-change patterns and comment accuracy across all 7 files — plus my own direct verification of the merge state and cross-crate field contracts.

Implements ROCMAI-941 (ticket named, not retrieved — no tracker access here, and this account does not follow referenced links by convention); the description's account of the diff (ancestry-based VRAM/power attribution for managed services, plus excluding a still-Starting occupant from the power split) matches what's actually in the diff.

Blocking

  • The branch conflicts with current main, and CI has never run on this commit — crates/rocm-dash-core/src/efficiency.rs, docs/architecture.md (PR-level)
    Independently reproduced: git merge-tree $(git merge-base origin/main pr/1026) origin/main pr/1026 shows a real <<<<<<< conflict in efficiency.rs's use line (this PR imports InstanceStatus; main's commit e64565d8 — "one node-efficiency implementation... (#1017)" — independently changed the same line to import ObservationFreshness, Snapshot) and a second conflict in docs/architecture.md, where #1017 rewrote the exact same efficiency.rs bullet (for node_efficiency) that this PR also rewrites (for the Starting-occupant behavior) — a naive merge drops one of the two descriptions. mcp__pull_request_read(get_check_runs) returns {"total_count":0} for commit 12e87e845fc1d949988f692682820493cb46a42b: zero CI runs exist for this head, so none of this repo's own required local-equivalent gates (cargo test --workspace --all-targets, cargo clippy -D warnings) have been verified against the merged result.
    Confidence 100 · mechanical · Fix: rebase/merge onto current main, reconcile both conflicting hunks (the import line, and the two independent doc-bullet rewrites so neither behavioral description is lost), then drive CI to green on the result.

  • The new cross-crate pid/start_ticks fields are pinned only by a hand-written JSON literal, not by this file's own established round-trip pattern — crates/rocm-dash-daemon/src/registry.rs (tests ancestry_roots_come_from_the_recorded_engine_and_supervisor, discover_managed_services_collects_roots_only_for_live_services, vs. the pre-existing managed_service_record_startup_phase_survives_cross_crate_serde in the same file)
    The pre-existing test's own doc comment states "nothing but a hand-written JSON literal pins" a producer/consumer field-name contract, so it builds a real rocm_core::ManagedServiceRecord, serializes it, and deserializes it as the daemon's ServiceRecord — exactly so a future rename/retype on the producer side fails loudly. The two new tests instead record_json(...).replace(...) a JSON string. Fields match today (confirmed by reading crates/rocm-core/src/service_record.rs), so there's no live bug — but nothing added here protects the new fields the way this file's own precedent says this exact risk requires.
    Confidence 85 · logic · Fix: extend (or add a sibling to) the cross-crate serde test to set supervisor_pid/engine_pid/*_start_ticks on a real ManagedServiceRecord, round-trip it through ServiceRecord, and assert ancestry_roots() on the result.

  • keep_through_partial_registry's new roots retain/extend logic has no test coverage anywhere — crates/rocm-dash-daemon/src/runner.rs:1082-1083
    managed_roots appears only in production code in the whole tree (runner.rs:613, 817, 1063, 1082-1083) — no test references it. The two existing unit tests for this exact function (partial_registry_read_adds_what_it_saw_and_forgets_nothing, partial_registry_read_that_saw_nothing_changes_nothing) were not updated to assert anything about roots, and the new integration test always succeeds so it never exercises the partial-read path either. The "unseen roots survive a partial read" behavior this PR adds is asserted nowhere.
    Confidence 85 · logic · Fix: add managed_roots assertions to the two existing partial-read unit tests (an unseen service's roots survive; a seen service's roots refresh).

  • The second new efficiency.rs test doesn't exercise the logic it's named for — crates/rocm-dash-core/src/efficiency.rs:240-251 (a_serving_occupant_with_no_reading_still_forces_the_equal_split)
    Both instances in this test are Ready (never Starting), so the new Starting-exclusion filter is inert for this scenario; tracing the arithmetic shows it produces the identical result via the pre-existing equal-split fallback, and it is shaped identically to the already-existing shared_gpu_splits_equally_without_complete_vram test. It stays green with the entire new feature fully reverted — the single most common test-quality defect this method looks for.
    Confidence 85 · logic · Fix: rework the test so a genuinely Starting-with-no-reading occupant sits alongside the unmeasured-but-Ready occupant in one call, so it actually exercises the interaction its name claims.

Non-blocking

  • A pre-existing comment elsewhere is now false because of this change — apps/rocm/src/dash.rs:1160 (test resolved_args_does_not_refresh_liveness_so_a_stale_running_record_is_missed)
    The comment claims the daemon's ServiceRecord "carries no pid fields at all." This PR adds four (supervisor_pid, supervisor_start_ticks, engine_pid, engine_start_ticks), confirmed absent from that struct before this PR. The test's actual behavioral assertion is still correct (that path still never reads a pid) — only the comment's claim about the struct's shape is now stale.
    Confidence 90 · mechanical · Fix: update the comment to say the fields now exist but this read path still ignores them.

  • ancestry.rs's live-/proc entry point has no direct unit test, unlike its sibling module — crates/rocm-dash-collectors/src/ancestry.rs:86-98 (owner_of_pid), vs. cgroup.rs's container_id_for_pid_unreadable_is_none_no_panic
    ancestry.rs's own module doc calls cgroup.rs its managed-service counterpart, and cgroup.rs has a dedicated, unconditional test exercising its real platform-gated entry point on every platform. ancestry.rs's equivalent (owner_of_pid) is only exercised indirectly through the new #[cfg(target_os = "linux")]-gated integration test (so never on the Windows CI job), and the non-Linux stub has zero coverage anywhere.
    Confidence 78 · logic · Fix: add one unconditional unit test for owner_of_pid mirroring cgroup.rs's pattern (e.g. assert owner_of_pid(u32::MAX, &[]) is None on every platform).

  • The new ancestry fallback amplifies an existing blocking-I/O-on-async pattern — crates/rocm-dash-daemon/src/runner.rs:812-818 calling ancestry::owner_of_pid (bounded by MAX_DEPTH = 32)
    run_loop is async fn; no spawn_blocking appears anywhere in runner.rs. The pre-existing cgroup join already did one synchronous /proc read per process inline on the async task; this change adds up to 32 more synchronous reads per process that misses the cgroup join. This repo's own AGENTS.md already acknowledges the daemon's tick is serial and doesn't yet isolate slow/blocking sources, so this enlarges a documented, already-accepted gap rather than introducing a new one — flagged for awareness rather than as a hard blocker given /proc reads are typically sub-millisecond.
    Confidence 72 · architectural · Fix: wrap the per-tick VRAM-attribution pass (or at least the ancestry fallback closure) in tokio::task::spawn_blocking.

Decisions for the author

  • A mixed Starting+Unknown pair sharing a GPU still dilutes a serving neighbour's share — tradeoff
    The new filter excludes a Starting-with-no-reading occupant, but an Unknown-with-no-reading occupant on the same GPU still forces the equal-split fallback (because it, too, has no reading), undoing some of the benefit. This reads as consistent with the codebase's stated policy ("equal split when unsure, never guess a skew from partial data") rather than an oversight — confirm that's the intended scope, or extend the exclusion to Unknown as well.
  • The new integration test's fixture records both use supervisor_pid: 1 (PID 1/init) as filler — non-blocking-improvement
    Harmless today (the engine-root match always short-circuits before reaching it), but could mislead a future reader into thinking the supervisor-fallback path is being exercised when it isn't. A distinct placeholder value would make that clearer.
  • first_child_of in the new integration test leaks its spawned "engine" processes if it panics before polling succeeds — non-blocking-improvement
    Both fake-engine child processes are spawned before the loop that waits for them to fork; if that loop times out and panics, the cleanup/kill loop later in the test never runs. Minor test hygiene, not a correctness defect.

Positive signals

  • ancestry.rs's parse_stat (field 22 of /proc/<pid>/stat, offset from the last )) was independently verified to match rocm-core's own, separately-implemented /proc-start-time parsing exactly — the two crates agree on this despite neither depending on the other.
  • The injected-reader design (owner_of(pid, roots, read_stat: F)) keeps the matcher pure and testable on any host, mirroring the pattern this codebase already uses in cgroup.rs.
  • The first new efficiency.rs test (a_starting_occupant_with_no_reading_does_not_dilute_a_serving_neighbour) is a well-targeted regression test: it was traced to correctly go red under three independent mutations (full revert, flipping the filter's ||/&&, and removing the outer "any occupant has a reading" guard).

Deployment notes

The four new ServiceRecord fields (supervisor_pid, supervisor_start_ticks, engine_pid, engine_start_ticks) all carry #[serde(default)], so older on-disk registry records still deserialize without a migration.

What this covered

Every file in the diff (7 files, 458 insertions / 7 deletions) read completely — by three independent fanned-out workers plus direct verification of the merge state and cross-crate contracts — against merge-base 6862dd3f1dd709d01f3246d2b45c265e566e3865…12e87e845fc1d949988f692682820493cb46a42b. Also read: rocm-core's ManagedServiceRecord/process_start_ticks, the production writers of the new pid fields in apps/rocm/apps/rocmd, every production InstanceStatus producer, attribute_power_w's other call site (demo.rs), cgroup.rs as ancestry.rs's sibling module, and dash.rs's reader of ManagedDiscovery.

Did not run: cargo test/clippy/xtask checks — no Rust toolchain was reachable in this sandbox (network-blocked rustup install); all test-quality and mutation conclusions above are from static tracing, not execution, and are scored accordingly. GitHub Actions has produced zero check runs for this commit — that is an objective fact about the PR itself (counted under Blocking), not a check this review skipped. All four review-code source-reading passes (agent-instruction adherence, history, prior-changes, code-comment accuracy) ran as part of the fan-out.

Not reconciled against any discussion on this PR — any existing review thread was deliberately not read, per this method's independence rule.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@volen-silo
volen-silo force-pushed the rocmai-941-managed-vram branch from 12e87e8 to c915e0c Compare November 11, 2026 10:08
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks both. Everything below is in c915e0c5, rebased onto current main.

Conflict / CI never ran: rebased onto main and resolved the two files. In docs/architecture.md the efficiency.rs bullet now carries both the node_efficiency description from #1017 and the Starting-occupant sentence, so neither is lost. The PR is mergeable again, so CI will run on the result.

Test gaps

  • Cross-crate serde: recorded_pids_survive_the_producer_to_daemon_serde_handoff serializes a real rocm_core::ManagedServiceRecord with the pid and start-tick fields set and reads it back as the daemon's ServiceRecord, like the startup-phase test beside it.
  • keep_through_partial_registry roots: two new tests, one where a seen service's roots are replaced and an unseen one's survive, one where a read that saw nothing keeps every root.
  • The power test: a_serving_occupant_with_no_reading_still_forces_the_equal_split was rewritten as a_starting_occupant_is_left_out_but_an_unmeasured_serving_one_still_forces_equal, so a genuinely Starting, unmeasured occupant sits beside an unmeasured Ready one in one call; it also checks that an Unknown status stays in the split.
  • owner_of_pid_is_total_on_every_platform covers the entry point and the non-Linux stub.

Smaller: the stale "carries no pid fields" comment in apps/rocm/src/dash.rs is corrected; the Starting filter uses the existing is_starting(); the integration test's fake engines are killed on drop, so a panic no longer leaks them; the PID-1 placeholder is gone.

Not changed

  • spawn_blocking for the ancestry fallback: the reads are sub-millisecond, bounded at 32 per process, and the same synchronous pattern the cgroup join already has, and AGENTS.md already notes the daemon's serial tick. A blocking-isolation change belongs to that wider gap, not this PR.
  • Extending the Starting exclusion to Unknown: Unknown means no readiness information, not "not serving", and the policy here is equal split when unsure. Left as is, and noted in the code.

@volen-silo
volen-silo dismissed tomastola’s stale review November 11, 2026 10:08

Addressed in c915e0c: rebased onto main with both conflicts resolved (so CI can run), and the inline PID-1 root bug is fixed with tests. Every point has a reply: #1026 (comment)

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@volen-silo
volen-silo force-pushed the rocmai-941-managed-vram branch from c915e0c to c173064 Compare November 11, 2026 10:46

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers from me. Mergeable cleanly against main, CI on this head is green, and the earlier review points (conflict, cross-crate serde test, partial-registry roots tests, the power test, PID 1 as a root) are fixed in the code. This was a static review: nothing was built or run (no Rust toolchain here).

Non-blocking:

  • Stale descriptions of the old behaviour: the count_vram_join_misses doc in crates/rocm-dash-daemon/src/runner.rs:~944-950 still says a managed service falls back to device-summed VRAM "by design"; the ServiceRecord doc in crates/rocm-dash-daemon/src/registry.rs:~60 still says pids are ignored; the vram.rs entry in docs/architecture.md:389 still says the runner passes cgroup::container_id_for_pid (it now tries cgroup, then ancestry); two comments in crates/rocm-dash-daemon/tests/telemetry_contract.rs (~799-806) say managed services can never join the map. Worth rewording while you are here.
  • A managed service that has roots but no GPU process yet (still loading) shows the device-summed VRAM, which includes its neighbour's memory, next to a neighbour that now shows its own figure (resolve_instance_vram fallback, shown in the detail panel). Showing -/0 for that case, or saying so in the description, would avoid two different bases side by side.
  • count_vram_join_misses still only counts container-shaped ids, so a managed service whose ancestry fails (hidepid, a PID namespace, a missing root) falls back silently, and the integration test's "no vram-attribution warning" assertion would also pass with ancestry broken. Counting managed instances that have roots but no map entry would make the warning meaningful again (its text still says "cgroup match").
  • runner.rs:~814: the cgroup join runs first, so if the daemon and a service live in a docker/containerd cgroup, container_id_for_pid returns an id that matches no instance and ancestry never runs. Trying ancestry first when roots exist would be exact.
  • crates/rocm-dash-collectors/src/ancestry.rs:46-51: parse_stat re-implements the /proc/<pid>/stat start-time parse that rocm-core already has (process_start_ticks); the crate already depends on rocm-core, so exposing one shared parser would avoid the copy.

🤖 by agenthub on AMD AgentHub

@volen-silo
volen-silo added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 11, 2026

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head c173064e3b16

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change.
Implements ROCMAI-941 (a follow-up to ROCMAI-62 / #812); no Jira/ROCMAI tracker tool is available in this environment, so the ticket's own content could not be retrieved — the description's account of it is taken at face value and matches the diff.

Blocking

  • PR head conflicts with current main — docs/architecture.md
    Confirmed read-only with git merge-tree --write-tree origin/main <pr-head> (no checkout): the merge produces a single CONFLICT (content): Merge conflict in docs/architecture.md, from concurrent edits to the rocm-dash-collectors crate-doc bullet list (main has since restructured the surrounding TUI section headers). Every other changed file merges cleanly. This matches GitHub's own mergeable_state: "dirty" on the PR. Confidence 100 (directly verified by running the merge). Fix: rebase onto current main and re-resolve the doc-list conflict.

  • The managed-service VRAM/power fallback warning no longer covers the exact failure mode this PR introduces — crates/rocm-dash-daemon/src/runner.rs:951 (count_vram_join_misses), doc comment at :944-950, pinned by the unchanged test managed_instances_never_count_as_vram_join_misses (:1659-1674)
    count_vram_join_misses only counts a "miss" for an instance whose container_id has the docker/containerd cgroup shape (rocm_dash_collectors::cgroup::is_container_id), on the documented rationale that a managed rocm serve service "could never match" the per-process VRAM map and so counting it "would raise a false ... warning on every healthy managed instance." That was true before this PR, when managed services never joined the map at all. This PR makes them join it via ancestry::owner_of_pid, but leaves count_vram_join_misses and its test unchanged, so a managed service is still unconditionally excluded from miss-counting. Concretely: if a live, GPU-active managed service's ancestry join fails for a real reason this PR's own doc comments anticipate (/proc/<pid>/stat unreadable, no root recorded yet, a PID-recycling mismatch), the dashboard silently falls back to device-summed VRAM for it — exactly the pre-PR bug this PR exists to fix — with zero warning, because the is_container_id gate still filters it out of the miss count regardless of why the join didn't happen. This is new, introduced behavior (a managed service's ancestry join can now fail in a way nothing detects) with no test exercising it. Confidence 85 (traced via the unchanged filter predicate, its own now-stale doc comment, and the pinning test's assertion that "the managed service... could never match" is still asserted as true). Fix: extend the miss-detection to also flag a live managed instance with GPU ids and (now) an eligible ancestry root that still isn't in per_container_used, rather than excluding every managed-service id unconditionally.

Non-blocking

None

Decisions for the author

  • Ancestry attribution degrades silently (to the pre-PR device-summed fallback) when a GPU worker is reparented away from its recorded root — tradeoff
    If an engine process dies and is respawned without the registry record's engine_pid/supervisor_pid being refreshed yet, a tensor-parallel worker reparented to init will walk up to a PID that is never a registered root (MIN_ROOT_PID excludes PID 1) and return None, same as an unreadable link. This degrades gracefully rather than misattributing, and self-heals once the registry record refreshes, so it is not a defect — but it's worth the author confirming this window (and its interaction with the above warning gap) is acceptable as-is.

Positive signals

  • ancestry.rs's PID-recycling defense (Root.start_ticks compared against the live process's own start time) and its bounded, injectable-reader walk are exercised by deliberately adversarial unit tests (cycles, self-parent, unreadable links, "awkward" process names with spaces/parens) — a notably thorough test set for a parser working off raw /proc text.
  • managed_services_get_their_own_vram_and_a_weighted_power_split in telemetry_contract.rs runs the real daemon loop against real forked processes standing in for GPU workers, rather than only unit-testing the pure ancestry/efficiency functions — it would catch a regression the pure-function tests alone could miss.
  • The new docs/architecture.md entry for ancestry.rs and the updated cgroup.rs/efficiency.rs entries were added in the same change, as the project's own AGENTS.md requires.

Deployment notes

Linux-only feature: owner_of_pid is a None-returning stub on other platforms, so Windows/WSL-without-/proc behavior is unchanged. All new ServiceRecord fields are #[serde(default)], so older on-disk records continue to parse. No migration or configuration change.

What this covered

All 8 changed files read in full against f536619a…c173064e (apps/rocm/src/dash.rs, new crates/rocm-dash-collectors/src/ancestry.rs, crates/rocm-dash-collectors/src/lib.rs, crates/rocm-dash-core/src/efficiency.rs, crates/rocm-dash-daemon/src/registry.rs, crates/rocm-dash-daemon/src/runner.rs, crates/rocm-dash-daemon/tests/telemetry_contract.rs, docs/architecture.md), plus cross-crate context needed to judge them in place: rocm-core's ManagedServiceRecord (the producer of the on-disk JSON ServiceRecord mirrors), metrics.rs's InstanceStatus, vram.rs's aggregate_process_vram, and the repo's AGENTS.md.

One independent fan-out worker ran a full blind pass over ancestry.rs/registry.rs (including the cross-crate serde contract) and reported no findings at or above confidence 70; its one aside — ManagedServiceRecord::engine_pid lacking #[serde(default)] unlike its sibling fields — is pre-existing, untouched-by-this-PR code and out of scope, so it is not carried into this report. A second worker, intended to independently cover efficiency.rs/runner.rs/telemetry_contract.rs, failed to spawn twice against a shared concurrency limit (many reviews are running concurrently right now); per the method, that pass was run single-context by me instead rather than dropped, so that portion of the review was not cross-checked by a second independent reader.

CI: all 39 check runs on this commit are success or skipped (no failure/error); none are blocking. Did not run: the history pass (git log/git blame against the modified regions) and the prior-review-comments pass, for time; nothing in the diff suggested either would surface something the other passes missed. Not reconciled against any prior review or discussion thread on this PR — this is a fresh blind pass.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@volen-silo
volen-silo force-pushed the rocmai-941-managed-vram branch from c173064 to 2d0794c Compare November 11, 2026 15:50

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 2d0794c36b31

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change.
Implements ROCMAI-941, a follow-up to ROCMAI-62 (#812); ticket content was not retrieved (no tracker access here). The description's account of the diff (ancestry-based VRAM/power attribution for managed rocm serve services, plus excluding a still-Starting occupant with no VRAM reading from the per-GPU power split) matches what's actually in the diff, and its stated reason for omitting a Gherkin scenario (no CI lane has a PTY plus a managed vLLM with real GPU-process attribution; the new daemon-level integration test is offered as the below-TUI substitute) is a reasoned justification I was able to confirm is non-vacuous, not an unexplained gap.

Blocking

  • CI is red on this exact head — PR-level (gh pr view 1026 --json statusCheckRollup against commit 2d0794c3)
    Seven required checks fail: Coverage (rocm-dash crates, ratcheted), build-and-test (both OS legs), Test (affected crates) (both OS legs), E2E tests (both OS legs). Their logs all fail at the identical step — sudo apt-get update && sudo apt-get install -y pkg-config libcap-dev — with Temporary failure resolving 'azure.archive.ubuntu.com'. This reads as a runner-side network/DNS flake rather than anything this diff caused (nothing in the available log output reaches an actual build or test step). Per this review method, a failing required check is blocking regardless of suspected cause.
    Confidence 100 · mechanical · Fix: re-run the failed jobs; if they fail again the same way, escalate as CI infrastructure rather than code.

  • The VRAM/power-fallback monitoring doesn't cover the new failure mode this PR introduces — crates/rocm-dash-daemon/src/runner.rs:894 (call site), :944-963 (count_vram_join_misses and its doc comment), :1658-1674 (the unchanged pinning test managed_instances_never_count_as_vram_join_misses)
    count_vram_join_misses only counts a "miss" for an instance whose container_id has the docker/containerd cgroup shape (cgroup::is_container_id), on the documented rationale that a managed service "could never match" the per-process VRAM map, so counting it "would raise a false ... warning on every healthy managed instance." That was true before this PR, when a managed service never joined the map at all. This PR makes it join via ancestry::owner_of_pid, but leaves this filter, its doc comment, and its pinning test unchanged — so a managed service is still unconditionally excluded from miss-counting. Concretely: when a live, GPU-active managed service's ancestry join fails for a real reason this module's own docs anticipate (/proc/<pid>/stat unreadable, no root recorded yet, a PID-recycling mismatch, MAX_DEPTH exceeded), the dashboard silently falls back to device-summed VRAM for it — the exact pre-PR symptom this PR exists to fix — with zero warning, because is_container_id still filters it out of the miss count regardless of why the join didn't happen. The new integration test's assertion of no "vram attribution" warning would equally pass with ancestry silently broken.
    Confidence 85 · architectural · Fix: extend the miss-predicate to also flag a live managed instance with GPU ids and an eligible ancestry root that still isn't in the per-container map, and reword the doc comment/test name so they no longer assert a managed service "could never match."

  • docs/architecture.md's row for registry.rs wasn't updated, unlike its sibling rows in the same diff — docs/architecture.md:457
    The diff updates the cgroup.rs and efficiency.rs table rows to describe their new ancestry-related behavior, and adds a brand-new row for ancestry.rs itself — but the registry.rs row is untouched, even though this same diff gives that file a new public method (ServiceRecord::ancestry_roots()), four new fields, and a new ManagedDiscovery.roots responsibility. The row still only describes the pre-existing scrape-target conversion. AGENTS.md §6 requires this doc's entries to stay true ("the check reads no prose, so keeping an entry's description true is still on you"), and the PR's own pattern of updating the other two rows shows this one was simply missed, not out of scope.
    Confidence 85 · mechanical · Fix: extend the registry.rs row to mention it now also derives per-service ancestry roots for GPU-process attribution.

Non-blocking

  • A pre-existing module doc comment is now false — crates/rocm-dash-daemon/src/registry.rs:60-61
    ServiceRecord's doc comment still reads "unknown fields (manifest paths, pids, recipe json, …) are ignored" — directly contradicted a few lines below by this same diff's four new, deliberately-read pid/start-tick fields.
    Confidence 85 · mechanical · Fix: drop "pids" from that example list.

  • An updated comment overclaims that pid fields are never consulted on this path — apps/rocm/src/dash.rs:1158-1162 (test resolved_args_does_not_refresh_liveness_so_a_stale_running_record_is_missed)
    The rewritten comment says the on-disk view "does carry pid fields now, but only for the daemon's process attribution; this read path ignores them." Traced the actual call graph: this exact path (resolved_args → discover_managed_services) does call ServiceRecord::ancestry_roots(), which reads engine_pid/supervisor_pid, for any live record — it only happens to yield nothing here because the test fixture never sets those fields. The test's real assertion (the past_attempts counter itself never reads a pid) is still correct; only the broader "this read path ignores them" claim overstates it.
    Confidence 75 · mechanical · Fix: narrow the claim to the specific counter that ignores pid fields, rather than the whole read path.

  • The supervisor-only attribution path has no end-to-end /proc-walk test — crates/rocm-dash-daemon/tests/telemetry_contract.rs:830-933 vs. crates/rocm-dash-daemon/src/registry.rs:827-920
    The new integration test's fixture only ever sets engine_pid (never supervisor_pid), so every GPU process in it is attributed through the engine root, one hop above the worker. The supervisor-only fallback (used before a supervisor has seen its engine) is proven correct only as synthetic Vec<Root> shape in registry.rs's unit tests, never through a real forked process tree the way the engine-root path now is.
    Confidence 75 · logic · Fix: add an integration-test variant (or a focused test) that records only supervisor_pid and verifies a worker descending through an intermediate, unregistered process is still attributed correctly.

  • One sub-case of a new test doesn't exercise the guard it's paired with — crates/rocm-dash-collectors/src/ancestry.rs:186-204, specifically for pid in [0, 1] (line 191)
    The walk's parent == 0 check returns None before current is ever assigned 0, so current can never equal 0 inside owner_of's loop. That means the pid == 0 iteration of this test passes regardless of whether the r.pid >= MIN_ROOT_PID guard exists at all — only the pid == 1 iteration (where current genuinely reaches 1 mid-walk) exercises that guard. The test's overall conclusion is still true and still catches a broken guard (via the pid == 1 case), so this is a clarity/precision issue, not a coverage hole.
    Confidence 75 · mechanical · Fix: no functional change needed; optionally note in a comment that the pid == 0 iteration documents an unreachable state rather than exercising the filter.

  • A dead service's ancestry root can be retained indefinitely during a sustained partial-registry-read failure, with no PID-reuse guard if it lacks a recorded start time — crates/rocm-dash-daemon/src/runner.rs:1063 (managed_roots field), :1077 (keep_through_partial_registry)
    This mirrors the project's own documented, intentional "a failed read never retires anything" trade-off (registry.rs's module doc) already applied to known_services/managed_non_vllm — so the retention itself isn't new. What's new is that managed_roots carries live-matchable PIDs: if the registry directory has one permanently-unreadable file (triggering the sustained-partial-read mode) while a different, cleanly-read service has genuinely stopped, that service's old root is kept every tick for as long as the unrelated file stays broken. If that root was recorded with start_ticks: None (no engine/supervisor start time ever captured) and its old PID is later reused by an unrelated process, ancestry::owner_of's recycled-PID guard can't catch it (it only triggers when a start time was recorded), and that unrelated process's GPU memory would be misattributed to the long-dead service.
    Confidence 70 · architectural · Fix: either purge a root as soon as its owner's record is read and found non-scrapeable (even during an otherwise-partial pass), or add a test making the current trade-off explicit for managed_roots specifically, since its unit tests (partial_registry_read_refreshes_the_roots_it_saw_and_keeps_the_unseen, partial_registry_read_that_saw_nothing_keeps_every_root) don't cover "read successfully this pass but confirmed stopped."

Decisions for the author

  • Ancestry roots are only collected for services with a bound port, not merely a live/scrapeable status — non-blocking-improvement
    discover_managed_services only calls record.ancestry_roots() once discovered_from_record succeeds, which also requires port != 0. A service whose engine process already exists (and may already hold GPU memory) but hasn't bound its port yet contributes no root that tick. Likely harmless (little VRAM is typically allocated before the port binds) and the doc comment ("every live service surfaced") is technically accurate either way — worth confirming the window doesn't matter for your actual engines.

Positive signals

  • ancestry.rs's parse_stat uses rfind(')'), not find, to handle a process name that itself contains spaces/parentheses — the classic /proc/<pid>/stat parsing trap — and has a dedicated test for exactly that.
  • The PID-recycling guard (Root.start_ticks compared against the live process's own start time) is tested both ways: a mismatched start time correctly refuses to claim, and an absent recorded start time is correctly taken at its word.
  • MIN_ROOT_PID is applied through one shared usable closure for both the engine and supervisor branches in registry.rs's ancestry_roots(), avoiding the classic "filtered one of two near-identical branches" bug.
  • The cross-crate serde test (recorded_pids_survive_the_producer_to_daemon_serde_handoff) builds a real rocm_core::ManagedServiceRecord and round-trips it through the daemon's mirror type, rather than hand-rolling JSON — it would genuinely catch a field rename/retype on either side.
  • Managed-service ids are generated as {engine}-{model-slug}-{millis} (always containing -), so they can never collide with the 64-lowercase-hex shape cgroup::is_container_id checks for — verified by reading the generator, not just assumed, so the "cgroup id and service id key the one map safely" design claim actually holds.
  • The new integration test (managed_services_get_their_own_vram_and_a_weighted_power_split) runs a real daemon loop against real forked processes rather than only unit-testing the pure functions, and was hand-traced to genuinely fail (both services showing the device total) with the ancestry fallback reverted — not vacuous.
  • Both new efficiency.rs tests were hand-traced to flip under either a full revert or an inverted is_starting(), and correctly distinguish Unknown (stays in the split) from Starting (left out when unmeasured).

Deployment notes

Linux-only feature: owner_of_pid is a None-returning stub on other platforms, so Windows/WSL-without-/proc behavior is unchanged. All four new ServiceRecord fields are #[serde(default)], so older on-disk registry records continue to parse without a migration.

What this covered

All 8 changed files read completely against f536619a…2d0794c3 (apps/rocm/src/dash.rs, new crates/rocm-dash-collectors/src/ancestry.rs, crates/rocm-dash-collectors/src/lib.rs, crates/rocm-dash-core/src/efficiency.rs, crates/rocm-dash-daemon/src/registry.rs, crates/rocm-dash-daemon/src/runner.rs, crates/rocm-dash-daemon/tests/telemetry_contract.rs, docs/architecture.md), fetched read-only (PR head never checked out) and diffed via git show/git diff against the merge base. Five independent fanned-out workers covered: (1) ancestry.rs+lib.rs, (2) registry.rs+telemetry_contract.rs, (3) runner.rs+efficiency.rs, (4) dash.rs+docs/architecture.md, (5) agent-instruction adherence (AGENTS.md) plus git-history and prior-fix-pattern passes across all 8 files. Also read for cross-file context: cgroup.rs, metrics.rs's InstanceStatus, rocm_core::ManagedServiceRecord, and rocm_core's service-id generator. Two of the five fan-out workers needed multiple re-spawns (a shared, environment-wide concurrency limit, not a content issue) before launching successfully; all five ultimately ran as genuine independent background passes — none were absorbed into my own context. Independently re-verified (not assumed): the merge state against current main (git merge-tree, clean — no conflict), and the live CI status for this exact head via the GitHub API plus the actual failing-job logs.

Did not run: cargo test/clippy/xtask — no Rust toolchain was reachable here; every "would this test still pass reverted/mutated" verdict above comes from static hand-tracing of the exact control flow and arithmetic, not execution, and is scored accordingly (none at 100). Did not retrieve ROCMAI-941's own ticket content (no tracker tool available in this environment).

Not reconciled against this PR's own prior review threads — this was a fresh blind pass per the method's independence rule. Immediately before publishing, I confirmed via the GitHub API that the head commit had not moved since I started, and that no review from this account exists yet at this exact head commit.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@volen-silo
volen-silo force-pushed the rocmai-941-managed-vram branch from 2d0794c to ffc40c7 Compare November 12, 2026 07:53
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks. This round is in ffc40c76 (head moved from 2d0794c3, also rebased onto current main).

  • CI red: it was the apt DNS failure at "Install native build deps" on all seven jobs, before any test ran, on this PR and on ROCMAI-938 follow-up: one throughput aggregate for both Observe heroes, skip faulty power readings #1030 alike. I re-ran the failed jobs; they pass now (43 passing, only the self-hosted GPU lanes were still running at last look).
  • Monitoring gap: fixed. count_vram_join_misses now also counts a serving (Ready/Running) managed vLLM service that has recorded ancestry roots and still is not in the per-container map, so a failed ancestry join (unreadable /proc, a recycled root PID) warns instead of silently falling back to device-summed VRAM. A service that is still loading is not counted (no GPU process yet), and neither is a non-vLLM managed service, whose backend can hold GPU memory without amd-smi ever listing the process. The doc comment and the pinning test were rewritten to say that, with a test for each case.
  • registry.rs row in docs/architecture.md: now describes the recorded engine/supervisor PIDs and the ancestry roots, including the retire rule below.
  • Stale comments: ServiceRecord's "pids are ignored" is corrected, and the dash.rs comment is narrowed to what is true: discovery on that path derives roots from the pids, but the asserted count comes from the status string alone.
  • Supervisor-only path: the integration test now has one service that records only a supervisor, with the worker two hops down behind an unregistered intermediate shell, next to the engine-rooted one. Disabling the supervisor roots makes it fail with "VRAM must be per service".
  • Dead service's root kept on a partial read: fixed. ManagedDiscovery now carries the ids of records it read and found stopped, and a partial pass drops their roots, so a stopped service's PID cannot stay matchable while an unrelated record is unreadable. New test partial_registry_read_drops_the_roots_of_a_service_it_saw_stop.
  • PID 0 iteration: left in, with a comment saying it documents a state the walk never reaches (a parent of 0 ends it first) and that PID 1 is the case the guard decides.

Not changed: ancestry roots are still collected only once a service has a bound port, as discovered_from_record requires. A service whose engine exists but has not bound its port yet contributes no root that tick; it holds little VRAM before it binds, so the window should not matter for these engines, but I have not measured it on a real vLLM start, and say so rather than claim it.

Local runs on this commit: on Windows 2090 tests across the dash and rocm crates pass and clippy is clean; on WSL (Linux) workspace clippy is clean, 1566 tests in the dash crates pass, and the dashboard e2e passes 50 of 51 (dash-24 needs amd-smi, which does not run on WSL, and fails the same way on main).

@volen-silo
volen-silo added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 12, 2026
…y process ancestry (ROCMAI-941)

VRAM is attributed per instance by joining amd-smi's GPU process PIDs to the
docker/containerd id in /proc/<pid>/cgroup. A managed `rocm serve` service is
not a container, so it never joined: every managed service showed the whole
device's VRAM, two services on one GPU each showed all of it, and the
VRAM-weighted power split never applied to them.

A service record carries the PIDs of its engine and launcher and their kernel
start times. The new ancestry module walks a GPU process's /proc/<pid>/stat
parent links up to a recorded root and returns that service's id. A root whose
live start time differs from the recorded one is a recycled PID and claims
nothing; an unreadable or looping chain ends with no owner. The runner tries the
cgroup join first and ancestry second, so container ids and service ids key the
same per-container map the instances already read.

attribute_power_w also stops letting a service that is still starting, with no
VRAM reading while a neighbour has one, take an equal share: it has not
allocated GPU memory, so an equal split showed the serving neighbour about half
its power and roughly double its tokens per watt until the newcomer loaded. With
no readings at all, or a serving occupant without one, the split stays equal.

Linux only: elsewhere ancestry yields no owner and attribution is unchanged.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
volen-silo force-pushed the rocmai-941-managed-vram branch from ffc40c7 to 9684e3b Compare November 12, 2026 09:25
@volen-silo
volen-silo enabled auto-merge November 12, 2026 09:31
@volen-silo
volen-silo added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 99ef606 Nov 12, 2026
40 checks passed
@volen-silo
volen-silo deleted the rocmai-941-managed-vram branch November 12, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants