Repository navigation
ROCMAI-941: attribute VRAM and power to managed rocm serve services by process ancestry - #1026
Conversation
tomastola
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (12e87e84), read against main at 3ab57202.
Blocking
- The branch conflicts with
main, and CI has never run on this commit —crates/rocm-dash-core/src/efficiency.rs,docs/architecture.md(PR-level)
GitHub reports the PR asCONFLICTING.git merge-treeagainst currentmainconflicts in exactly those two files: #1017 addednode_efficiencyand a newuseline next to theattribute_power_wedits here, and rewrote the sameefficiency.rsbullet inarchitecture.md. Because of the conflictpull_requestworkflows were never created: the Actions API returns 0 runs for this head SHA and the rollup holds only the docs-build status, while other PRs opened today haveCI,CodeQL, security-scan and self-hosted runs. So no required check has been seen for this change, and the Windows/WSL results in the description are from the pre-conflict base, not the merged result.
Confidence 100 · mechanical · Fix: merge or rebase ontomain, resolve the two files (the conflicts are adjacent additions, not semantic), then let CI run to green on the result.
Non-blocking
One finding, attached inline on crates/rocm-dash-daemon/src/registry.rs:119.
Decisions for the author
None
Positive signals
- The injected
/procreader inowner_ofand the start-time check for a recycled PID keep the matcher pure and testable on any host. - The real-process integration test fails with the roots disabled (both services at the device total), so it does prove the symptom it targets.
Deployment notes
None. The new ServiceRecord fields are #[serde(default)], so older registry records still load.
What this covered
Every file in the diff, completely, plus the rocm crate's writers of supervisor_pid/engine_pid/*_start_ticks and rocm_core::process_start_ticks, to check the units and meaning match. Ran cargo test -p rocm-dash-collectors -p rocm-dash-core -p rocm-dash-daemon on the PR head in a private clone (Linux): all pass, including the new ancestry, registry, attribute_power_w and managed_services_get_their_own_vram_and_a_weighted_power_split tests. I did not run the mutation check on the new Starting filter, and the merged result was not built because of the conflict above.
The PR's existing discussion was not read before the findings were reached (there were no review threads or comments on it).
| if let Some(pid) = self.engine_pid.filter(|p| *p > 0) { | ||
| roots.push(root(pid, self.engine_start_ticks)); | ||
| } | ||
| if self.supervisor_pid > 0 { |
There was a problem hiding this comment.
Non-blocking (confidence 85): a recorded supervisor_pid of 1 becomes a root that owns every process on the host.
ancestry_roots only rejects 0 (> 0). owner_of (ancestry.rs:65) matches a root by PID and, with start_ticks: None, accepts it unconditionally. Every host process descends from PID 1, so with supervisor_pid: 1 and no start time, any GPU process that is not under an earlier root (a user's own training job, another tool) resolves to this service and its VRAM is added to the service's reading. The new integration test uses exactly that record shape ("supervisor_pid":1 at telemetry_contract.rs:862) and only passes because each worker meets its engine root before it reaches PID 1, so it cannot catch this. The comment on the zero case ("never a root that could match init") shows the intent; PID 1 is the init case.
Suggest > 1 for the supervisor and the engine, and a test that a worker outside every engine tree gets no owner when a service record has supervisor_pid: 1. Worth also using a non-1 supervisor PID in the integration fixture.
There was a problem hiding this comment.
Fixed in c915e0c5, and it was a real bug. A root is now only a PID of 2 or more, enforced in two places: ServiceRecord::ancestry_roots (an engine or supervisor recorded as 0 or 1 yields no root) and owner_of itself (a root at 0 or 1 never matches, whatever the caller passes). New tests: a worker outside every service tree gets no owner when a record has supervisor_pid: 1, and a real root beside a bogus one still resolves. The integration fixture no longer uses PID 1 as filler; it records no supervisor at all.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 12e87e84
Review — needs work
Full review of the whole change (12e87e84), diffed against its merge-base with main (6862dd3f). Three independent fanned-out passes covered: (1) the ancestry/VRAM-attribution feature (ancestry.rs, lib.rs, registry.rs, runner.rs, telemetry_contract.rs), (2) the starting-occupant power-split change (efficiency.rs), and (3) agent-instruction adherence, history/divergence, prior-change patterns and comment accuracy across all 7 files — plus my own direct verification of the merge state and cross-crate field contracts.
Implements ROCMAI-941 (ticket named, not retrieved — no tracker access here, and this account does not follow referenced links by convention); the description's account of the diff (ancestry-based VRAM/power attribution for managed services, plus excluding a still-Starting occupant from the power split) matches what's actually in the diff.
Blocking
-
The branch conflicts with current
main, and CI has never run on this commit —crates/rocm-dash-core/src/efficiency.rs,docs/architecture.md(PR-level)
Independently reproduced:git merge-tree $(git merge-base origin/main pr/1026) origin/main pr/1026shows a real<<<<<<<conflict inefficiency.rs'suseline (this PR importsInstanceStatus;main's commite64565d8— "one node-efficiency implementation... (#1017)" — independently changed the same line to importObservationFreshness, Snapshot) and a second conflict indocs/architecture.md, where#1017rewrote the exact sameefficiency.rsbullet (fornode_efficiency) that this PR also rewrites (for the Starting-occupant behavior) — a naive merge drops one of the two descriptions.mcp__pull_request_read(get_check_runs)returns{"total_count":0}for commit12e87e845fc1d949988f692682820493cb46a42b: zero CI runs exist for this head, so none of this repo's own required local-equivalent gates (cargo test --workspace --all-targets,cargo clippy -D warnings) have been verified against the merged result.
Confidence 100 · mechanical · Fix: rebase/merge onto currentmain, reconcile both conflicting hunks (the import line, and the two independent doc-bullet rewrites so neither behavioral description is lost), then drive CI to green on the result. -
The new cross-crate pid/start_ticks fields are pinned only by a hand-written JSON literal, not by this file's own established round-trip pattern —
crates/rocm-dash-daemon/src/registry.rs(testsancestry_roots_come_from_the_recorded_engine_and_supervisor,discover_managed_services_collects_roots_only_for_live_services, vs. the pre-existingmanaged_service_record_startup_phase_survives_cross_crate_serdein the same file)
The pre-existing test's own doc comment states "nothing but a hand-written JSON literal pins" a producer/consumer field-name contract, so it builds a realrocm_core::ManagedServiceRecord, serializes it, and deserializes it as the daemon'sServiceRecord— exactly so a future rename/retype on the producer side fails loudly. The two new tests insteadrecord_json(...).replace(...)a JSON string. Fields match today (confirmed by readingcrates/rocm-core/src/service_record.rs), so there's no live bug — but nothing added here protects the new fields the way this file's own precedent says this exact risk requires.
Confidence 85 · logic · Fix: extend (or add a sibling to) the cross-crate serde test to setsupervisor_pid/engine_pid/*_start_tickson a realManagedServiceRecord, round-trip it throughServiceRecord, and assertancestry_roots()on the result. -
keep_through_partial_registry's new roots retain/extend logic has no test coverage anywhere —crates/rocm-dash-daemon/src/runner.rs:1082-1083
managed_rootsappears only in production code in the whole tree (runner.rs:613, 817, 1063, 1082-1083) — no test references it. The two existing unit tests for this exact function (partial_registry_read_adds_what_it_saw_and_forgets_nothing,partial_registry_read_that_saw_nothing_changes_nothing) were not updated to assert anything about roots, and the new integration test always succeeds so it never exercises the partial-read path either. The "unseen roots survive a partial read" behavior this PR adds is asserted nowhere.
Confidence 85 · logic · Fix: addmanaged_rootsassertions to the two existing partial-read unit tests (an unseen service's roots survive; a seen service's roots refresh). -
The second new
efficiency.rstest doesn't exercise the logic it's named for —crates/rocm-dash-core/src/efficiency.rs:240-251(a_serving_occupant_with_no_reading_still_forces_the_equal_split)
Both instances in this test areReady(neverStarting), so the new Starting-exclusion filter is inert for this scenario; tracing the arithmetic shows it produces the identical result via the pre-existing equal-split fallback, and it is shaped identically to the already-existingshared_gpu_splits_equally_without_complete_vramtest. It stays green with the entire new feature fully reverted — the single most common test-quality defect this method looks for.
Confidence 85 · logic · Fix: rework the test so a genuinelyStarting-with-no-reading occupant sits alongside the unmeasured-but-Readyoccupant in one call, so it actually exercises the interaction its name claims.
Non-blocking
-
A pre-existing comment elsewhere is now false because of this change —
apps/rocm/src/dash.rs:1160(testresolved_args_does_not_refresh_liveness_so_a_stale_running_record_is_missed)
The comment claims the daemon'sServiceRecord"carries no pid fields at all." This PR adds four (supervisor_pid,supervisor_start_ticks,engine_pid,engine_start_ticks), confirmed absent from that struct before this PR. The test's actual behavioral assertion is still correct (that path still never reads a pid) — only the comment's claim about the struct's shape is now stale.
Confidence 90 · mechanical · Fix: update the comment to say the fields now exist but this read path still ignores them. -
ancestry.rs's live-/procentry point has no direct unit test, unlike its sibling module —crates/rocm-dash-collectors/src/ancestry.rs:86-98(owner_of_pid), vs.cgroup.rs'scontainer_id_for_pid_unreadable_is_none_no_panic
ancestry.rs's own module doc callscgroup.rsits managed-service counterpart, andcgroup.rshas a dedicated, unconditional test exercising its real platform-gated entry point on every platform.ancestry.rs's equivalent (owner_of_pid) is only exercised indirectly through the new#[cfg(target_os = "linux")]-gated integration test (so never on the Windows CI job), and the non-Linux stub has zero coverage anywhere.
Confidence 78 · logic · Fix: add one unconditional unit test forowner_of_pidmirroringcgroup.rs's pattern (e.g. assertowner_of_pid(u32::MAX, &[])isNoneon every platform). -
The new ancestry fallback amplifies an existing blocking-I/O-on-async pattern —
crates/rocm-dash-daemon/src/runner.rs:812-818callingancestry::owner_of_pid(bounded byMAX_DEPTH = 32)
run_loopisasync fn; nospawn_blockingappears anywhere inrunner.rs. The pre-existing cgroup join already did one synchronous/procread per process inline on the async task; this change adds up to 32 more synchronous reads per process that misses the cgroup join. This repo's ownAGENTS.mdalready acknowledges the daemon's tick is serial and doesn't yet isolate slow/blocking sources, so this enlarges a documented, already-accepted gap rather than introducing a new one — flagged for awareness rather than as a hard blocker given/procreads are typically sub-millisecond.
Confidence 72 · architectural · Fix: wrap the per-tick VRAM-attribution pass (or at least the ancestry fallback closure) intokio::task::spawn_blocking.
Decisions for the author
- A mixed
Starting+Unknownpair sharing a GPU still dilutes a serving neighbour's share — tradeoff
The new filter excludes aStarting-with-no-reading occupant, but anUnknown-with-no-reading occupant on the same GPU still forces the equal-split fallback (because it, too, has no reading), undoing some of the benefit. This reads as consistent with the codebase's stated policy ("equal split when unsure, never guess a skew from partial data") rather than an oversight — confirm that's the intended scope, or extend the exclusion toUnknownas well. - The new integration test's fixture records both use
supervisor_pid: 1(PID 1/init) as filler — non-blocking-improvement
Harmless today (the engine-root match always short-circuits before reaching it), but could mislead a future reader into thinking the supervisor-fallback path is being exercised when it isn't. A distinct placeholder value would make that clearer. first_child_ofin the new integration test leaks its spawned "engine" processes if it panics before polling succeeds — non-blocking-improvement
Both fake-engine child processes are spawned before the loop that waits for them to fork; if that loop times out and panics, the cleanup/kill loop later in the test never runs. Minor test hygiene, not a correctness defect.
Positive signals
ancestry.rs'sparse_stat(field 22 of/proc/<pid>/stat, offset from the last)) was independently verified to matchrocm-core's own, separately-implemented/proc-start-time parsing exactly — the two crates agree on this despite neither depending on the other.- The injected-reader design (
owner_of(pid, roots, read_stat: F)) keeps the matcher pure and testable on any host, mirroring the pattern this codebase already uses incgroup.rs. - The first new
efficiency.rstest (a_starting_occupant_with_no_reading_does_not_dilute_a_serving_neighbour) is a well-targeted regression test: it was traced to correctly go red under three independent mutations (full revert, flipping the filter's||/&&, and removing the outer "any occupant has a reading" guard).
Deployment notes
The four new ServiceRecord fields (supervisor_pid, supervisor_start_ticks, engine_pid, engine_start_ticks) all carry #[serde(default)], so older on-disk registry records still deserialize without a migration.
What this covered
Every file in the diff (7 files, 458 insertions / 7 deletions) read completely — by three independent fanned-out workers plus direct verification of the merge state and cross-crate contracts — against merge-base 6862dd3f1dd709d01f3246d2b45c265e566e3865…12e87e845fc1d949988f692682820493cb46a42b. Also read: rocm-core's ManagedServiceRecord/process_start_ticks, the production writers of the new pid fields in apps/rocm/apps/rocmd, every production InstanceStatus producer, attribute_power_w's other call site (demo.rs), cgroup.rs as ancestry.rs's sibling module, and dash.rs's reader of ManagedDiscovery.
Did not run: cargo test/clippy/xtask checks — no Rust toolchain was reachable in this sandbox (network-blocked rustup install); all test-quality and mutation conclusions above are from static tracing, not execution, and are scored accordingly. GitHub Actions has produced zero check runs for this commit — that is an objective fact about the PR itself (counted under Blocking), not a check this review skipped. All four review-code source-reading passes (agent-instruction adherence, history, prior-changes, code-comment accuracy) ran as part of the fan-out.
Not reconciled against any discussion on this PR — any existing review thread was deliberately not read, per this method's independence rule.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
12e87e8 to
c915e0c
Compare
|
Thanks both. Everything below is in Conflict / CI never ran: rebased onto Test gaps
Smaller: the stale "carries no pid fields" comment in Not changed
|
Addressed in c915e0c: rebased onto main with both conflicts resolved (so CI can run), and the inline PID-1 root bug is fixed with tests. Every point has a reply: #1026 (comment)
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
c915e0c to
c173064
Compare
jussielo-amd
left a comment
There was a problem hiding this comment.
No blockers from me. Mergeable cleanly against main, CI on this head is green, and the earlier review points (conflict, cross-crate serde test, partial-registry roots tests, the power test, PID 1 as a root) are fixed in the code. This was a static review: nothing was built or run (no Rust toolchain here).
Non-blocking:
- Stale descriptions of the old behaviour: the
count_vram_join_missesdoc incrates/rocm-dash-daemon/src/runner.rs:~944-950still says a managed service falls back to device-summed VRAM "by design"; theServiceRecorddoc incrates/rocm-dash-daemon/src/registry.rs:~60still says pids are ignored; thevram.rsentry indocs/architecture.md:389still says the runner passescgroup::container_id_for_pid(it now tries cgroup, then ancestry); two comments incrates/rocm-dash-daemon/tests/telemetry_contract.rs(~799-806) say managed services can never join the map. Worth rewording while you are here. - A managed service that has roots but no GPU process yet (still loading) shows the device-summed VRAM, which includes its neighbour's memory, next to a neighbour that now shows its own figure (
resolve_instance_vramfallback, shown in the detail panel). Showing-/0 for that case, or saying so in the description, would avoid two different bases side by side. count_vram_join_missesstill only counts container-shaped ids, so a managed service whose ancestry fails (hidepid, a PID namespace, a missing root) falls back silently, and the integration test's "no vram-attribution warning" assertion would also pass with ancestry broken. Counting managed instances that have roots but no map entry would make the warning meaningful again (its text still says "cgroup match").runner.rs:~814: the cgroup join runs first, so if the daemon and a service live in a docker/containerd cgroup,container_id_for_pidreturns an id that matches no instance and ancestry never runs. Trying ancestry first when roots exist would be exact.crates/rocm-dash-collectors/src/ancestry.rs:46-51:parse_statre-implements the/proc/<pid>/statstart-time parse thatrocm-corealready has (process_start_ticks); the crate already depends onrocm-core, so exposing one shared parser would avoid the copy.
🤖 by agenthub on AMD AgentHub
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head c173064e3b16
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
Implements ROCMAI-941 (a follow-up to ROCMAI-62 / #812); no Jira/ROCMAI tracker tool is available in this environment, so the ticket's own content could not be retrieved — the description's account of it is taken at face value and matches the diff.
Blocking
-
PR head conflicts with current
main—docs/architecture.md
Confirmed read-only withgit merge-tree --write-tree origin/main <pr-head>(no checkout): the merge produces a singleCONFLICT (content): Merge conflict in docs/architecture.md, from concurrent edits to therocm-dash-collectorscrate-doc bullet list (mainhas since restructured the surrounding TUI section headers). Every other changed file merges cleanly. This matches GitHub's ownmergeable_state: "dirty"on the PR. Confidence 100 (directly verified by running the merge). Fix: rebase onto currentmainand re-resolve the doc-list conflict. -
The managed-service VRAM/power fallback warning no longer covers the exact failure mode this PR introduces —
crates/rocm-dash-daemon/src/runner.rs:951(count_vram_join_misses), doc comment at:944-950, pinned by the unchanged testmanaged_instances_never_count_as_vram_join_misses(:1659-1674)
count_vram_join_missesonly counts a "miss" for an instance whosecontainer_idhas the docker/containerd cgroup shape (rocm_dash_collectors::cgroup::is_container_id), on the documented rationale that a managedrocm serveservice "could never match" the per-process VRAM map and so counting it "would raise a false ... warning on every healthy managed instance." That was true before this PR, when managed services never joined the map at all. This PR makes them join it viaancestry::owner_of_pid, but leavescount_vram_join_missesand its test unchanged, so a managed service is still unconditionally excluded from miss-counting. Concretely: if a live, GPU-active managed service's ancestry join fails for a real reason this PR's own doc comments anticipate (/proc/<pid>/statunreadable, no root recorded yet, a PID-recycling mismatch), the dashboard silently falls back to device-summed VRAM for it — exactly the pre-PR bug this PR exists to fix — with zero warning, because theis_container_idgate still filters it out of the miss count regardless of why the join didn't happen. This is new, introduced behavior (a managed service's ancestry join can now fail in a way nothing detects) with no test exercising it. Confidence 85 (traced via the unchanged filter predicate, its own now-stale doc comment, and the pinning test's assertion that "the managed service... could never match" is still asserted as true). Fix: extend the miss-detection to also flag a live managed instance with GPU ids and (now) an eligible ancestry root that still isn't inper_container_used, rather than excluding every managed-service id unconditionally.
Non-blocking
None
Decisions for the author
- Ancestry attribution degrades silently (to the pre-PR device-summed fallback) when a GPU worker is reparented away from its recorded root — tradeoff
If an engine process dies and is respawned without the registry record'sengine_pid/supervisor_pidbeing refreshed yet, a tensor-parallel worker reparented to init will walk up to a PID that is never a registered root (MIN_ROOT_PIDexcludes PID 1) and returnNone, same as an unreadable link. This degrades gracefully rather than misattributing, and self-heals once the registry record refreshes, so it is not a defect — but it's worth the author confirming this window (and its interaction with the above warning gap) is acceptable as-is.
Positive signals
ancestry.rs's PID-recycling defense (Root.start_tickscompared against the live process's own start time) and its bounded, injectable-reader walk are exercised by deliberately adversarial unit tests (cycles, self-parent, unreadable links, "awkward" process names with spaces/parens) — a notably thorough test set for a parser working off raw/proctext.managed_services_get_their_own_vram_and_a_weighted_power_splitintelemetry_contract.rsruns the real daemon loop against real forked processes standing in for GPU workers, rather than only unit-testing the pure ancestry/efficiency functions — it would catch a regression the pure-function tests alone could miss.- The new
docs/architecture.mdentry forancestry.rsand the updatedcgroup.rs/efficiency.rsentries were added in the same change, as the project's ownAGENTS.mdrequires.
Deployment notes
Linux-only feature: owner_of_pid is a None-returning stub on other platforms, so Windows/WSL-without-/proc behavior is unchanged. All new ServiceRecord fields are #[serde(default)], so older on-disk records continue to parse. No migration or configuration change.
What this covered
All 8 changed files read in full against f536619a…c173064e (apps/rocm/src/dash.rs, new crates/rocm-dash-collectors/src/ancestry.rs, crates/rocm-dash-collectors/src/lib.rs, crates/rocm-dash-core/src/efficiency.rs, crates/rocm-dash-daemon/src/registry.rs, crates/rocm-dash-daemon/src/runner.rs, crates/rocm-dash-daemon/tests/telemetry_contract.rs, docs/architecture.md), plus cross-crate context needed to judge them in place: rocm-core's ManagedServiceRecord (the producer of the on-disk JSON ServiceRecord mirrors), metrics.rs's InstanceStatus, vram.rs's aggregate_process_vram, and the repo's AGENTS.md.
One independent fan-out worker ran a full blind pass over ancestry.rs/registry.rs (including the cross-crate serde contract) and reported no findings at or above confidence 70; its one aside — ManagedServiceRecord::engine_pid lacking #[serde(default)] unlike its sibling fields — is pre-existing, untouched-by-this-PR code and out of scope, so it is not carried into this report. A second worker, intended to independently cover efficiency.rs/runner.rs/telemetry_contract.rs, failed to spawn twice against a shared concurrency limit (many reviews are running concurrently right now); per the method, that pass was run single-context by me instead rather than dropped, so that portion of the review was not cross-checked by a second independent reader.
CI: all 39 check runs on this commit are success or skipped (no failure/error); none are blocking. Did not run: the history pass (git log/git blame against the modified regions) and the prior-review-comments pass, for time; nothing in the diff suggested either would surface something the other passes missed. Not reconciled against any prior review or discussion thread on this PR — this is a fresh blind pass.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
c173064 to
2d0794c
Compare
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 2d0794c36b31
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
Implements ROCMAI-941, a follow-up to ROCMAI-62 (#812); ticket content was not retrieved (no tracker access here). The description's account of the diff (ancestry-based VRAM/power attribution for managed rocm serve services, plus excluding a still-Starting occupant with no VRAM reading from the per-GPU power split) matches what's actually in the diff, and its stated reason for omitting a Gherkin scenario (no CI lane has a PTY plus a managed vLLM with real GPU-process attribution; the new daemon-level integration test is offered as the below-TUI substitute) is a reasoned justification I was able to confirm is non-vacuous, not an unexplained gap.
Blocking
-
CI is red on this exact head — PR-level (
gh pr view 1026 --json statusCheckRollupagainst commit2d0794c3)
Seven required checks fail:Coverage (rocm-dash crates, ratcheted),build-and-test(both OS legs),Test (affected crates)(both OS legs),E2E tests(both OS legs). Their logs all fail at the identical step —sudo apt-get update && sudo apt-get install -y pkg-config libcap-dev— withTemporary failure resolving 'azure.archive.ubuntu.com'. This reads as a runner-side network/DNS flake rather than anything this diff caused (nothing in the available log output reaches an actual build or test step). Per this review method, a failing required check is blocking regardless of suspected cause.
Confidence 100 · mechanical · Fix: re-run the failed jobs; if they fail again the same way, escalate as CI infrastructure rather than code. -
The VRAM/power-fallback monitoring doesn't cover the new failure mode this PR introduces —
crates/rocm-dash-daemon/src/runner.rs:894(call site),:944-963(count_vram_join_missesand its doc comment),:1658-1674(the unchanged pinning testmanaged_instances_never_count_as_vram_join_misses)
count_vram_join_missesonly counts a "miss" for an instance whosecontainer_idhas the docker/containerd cgroup shape (cgroup::is_container_id), on the documented rationale that a managed service "could never match" the per-process VRAM map, so counting it "would raise a false ... warning on every healthy managed instance." That was true before this PR, when a managed service never joined the map at all. This PR makes it join viaancestry::owner_of_pid, but leaves this filter, its doc comment, and its pinning test unchanged — so a managed service is still unconditionally excluded from miss-counting. Concretely: when a live, GPU-active managed service's ancestry join fails for a real reason this module's own docs anticipate (/proc/<pid>/statunreadable, no root recorded yet, a PID-recycling mismatch,MAX_DEPTHexceeded), the dashboard silently falls back to device-summed VRAM for it — the exact pre-PR symptom this PR exists to fix — with zero warning, becauseis_container_idstill filters it out of the miss count regardless of why the join didn't happen. The new integration test's assertion of no "vram attribution" warning would equally pass with ancestry silently broken.
Confidence 85 · architectural · Fix: extend the miss-predicate to also flag a live managed instance with GPU ids and an eligible ancestry root that still isn't in the per-container map, and reword the doc comment/test name so they no longer assert a managed service "could never match." -
docs/architecture.md's row forregistry.rswasn't updated, unlike its sibling rows in the same diff —docs/architecture.md:457
The diff updates thecgroup.rsandefficiency.rstable rows to describe their new ancestry-related behavior, and adds a brand-new row forancestry.rsitself — but theregistry.rsrow is untouched, even though this same diff gives that file a new public method (ServiceRecord::ancestry_roots()), four new fields, and a newManagedDiscovery.rootsresponsibility. The row still only describes the pre-existing scrape-target conversion. AGENTS.md §6 requires this doc's entries to stay true ("the check reads no prose, so keeping an entry's description true is still on you"), and the PR's own pattern of updating the other two rows shows this one was simply missed, not out of scope.
Confidence 85 · mechanical · Fix: extend theregistry.rsrow to mention it now also derives per-service ancestry roots for GPU-process attribution.
Non-blocking
-
A pre-existing module doc comment is now false —
crates/rocm-dash-daemon/src/registry.rs:60-61
ServiceRecord's doc comment still reads "unknown fields (manifest paths, pids, recipe json, …) are ignored" — directly contradicted a few lines below by this same diff's four new, deliberately-read pid/start-tick fields.
Confidence 85 · mechanical · Fix: drop "pids" from that example list. -
An updated comment overclaims that pid fields are never consulted on this path —
apps/rocm/src/dash.rs:1158-1162(testresolved_args_does_not_refresh_liveness_so_a_stale_running_record_is_missed)
The rewritten comment says the on-disk view "does carry pid fields now, but only for the daemon's process attribution; this read path ignores them." Traced the actual call graph: this exact path (resolved_args→discover_managed_services) does callServiceRecord::ancestry_roots(), which readsengine_pid/supervisor_pid, for any live record — it only happens to yield nothing here because the test fixture never sets those fields. The test's real assertion (thepast_attemptscounter itself never reads a pid) is still correct; only the broader "this read path ignores them" claim overstates it.
Confidence 75 · mechanical · Fix: narrow the claim to the specific counter that ignores pid fields, rather than the whole read path. -
The supervisor-only attribution path has no end-to-end
/proc-walk test —crates/rocm-dash-daemon/tests/telemetry_contract.rs:830-933vs.crates/rocm-dash-daemon/src/registry.rs:827-920
The new integration test's fixture only ever setsengine_pid(neversupervisor_pid), so every GPU process in it is attributed through the engine root, one hop above the worker. The supervisor-only fallback (used before a supervisor has seen its engine) is proven correct only as syntheticVec<Root>shape inregistry.rs's unit tests, never through a real forked process tree the way the engine-root path now is.
Confidence 75 · logic · Fix: add an integration-test variant (or a focused test) that records onlysupervisor_pidand verifies a worker descending through an intermediate, unregistered process is still attributed correctly. -
One sub-case of a new test doesn't exercise the guard it's paired with —
crates/rocm-dash-collectors/src/ancestry.rs:186-204, specificallyfor pid in [0, 1](line 191)
The walk'sparent == 0check returnsNonebeforecurrentis ever assigned0, socurrentcan never equal 0 insideowner_of's loop. That means thepid == 0iteration of this test passes regardless of whether ther.pid >= MIN_ROOT_PIDguard exists at all — only thepid == 1iteration (wherecurrentgenuinely reaches1mid-walk) exercises that guard. The test's overall conclusion is still true and still catches a broken guard (via thepid == 1case), so this is a clarity/precision issue, not a coverage hole.
Confidence 75 · mechanical · Fix: no functional change needed; optionally note in a comment that thepid == 0iteration documents an unreachable state rather than exercising the filter. -
A dead service's ancestry root can be retained indefinitely during a sustained partial-registry-read failure, with no PID-reuse guard if it lacks a recorded start time —
crates/rocm-dash-daemon/src/runner.rs:1063(managed_rootsfield),:1077(keep_through_partial_registry)
This mirrors the project's own documented, intentional "a failed read never retires anything" trade-off (registry.rs's module doc) already applied toknown_services/managed_non_vllm— so the retention itself isn't new. What's new is thatmanaged_rootscarries live-matchable PIDs: if the registry directory has one permanently-unreadable file (triggering the sustained-partial-read mode) while a different, cleanly-read service has genuinely stopped, that service's old root is kept every tick for as long as the unrelated file stays broken. If that root was recorded withstart_ticks: None(no engine/supervisor start time ever captured) and its old PID is later reused by an unrelated process,ancestry::owner_of's recycled-PID guard can't catch it (it only triggers when a start time was recorded), and that unrelated process's GPU memory would be misattributed to the long-dead service.
Confidence 70 · architectural · Fix: either purge a root as soon as its owner's record is read and found non-scrapeable (even during an otherwise-partial pass), or add a test making the current trade-off explicit formanaged_rootsspecifically, since its unit tests (partial_registry_read_refreshes_the_roots_it_saw_and_keeps_the_unseen,partial_registry_read_that_saw_nothing_keeps_every_root) don't cover "read successfully this pass but confirmed stopped."
Decisions for the author
- Ancestry roots are only collected for services with a bound port, not merely a live/scrapeable status — non-blocking-improvement
discover_managed_servicesonly callsrecord.ancestry_roots()oncediscovered_from_recordsucceeds, which also requiresport != 0. A service whose engine process already exists (and may already hold GPU memory) but hasn't bound its port yet contributes no root that tick. Likely harmless (little VRAM is typically allocated before the port binds) and the doc comment ("every live service surfaced") is technically accurate either way — worth confirming the window doesn't matter for your actual engines.
Positive signals
ancestry.rs'sparse_statusesrfind(')'), notfind, to handle a process name that itself contains spaces/parentheses — the classic/proc/<pid>/statparsing trap — and has a dedicated test for exactly that.- The PID-recycling guard (
Root.start_tickscompared against the live process's own start time) is tested both ways: a mismatched start time correctly refuses to claim, and an absent recorded start time is correctly taken at its word. MIN_ROOT_PIDis applied through one sharedusableclosure for both the engine and supervisor branches inregistry.rs'sancestry_roots(), avoiding the classic "filtered one of two near-identical branches" bug.- The cross-crate serde test (
recorded_pids_survive_the_producer_to_daemon_serde_handoff) builds a realrocm_core::ManagedServiceRecordand round-trips it through the daemon's mirror type, rather than hand-rolling JSON — it would genuinely catch a field rename/retype on either side. - Managed-service ids are generated as
{engine}-{model-slug}-{millis}(always containing-), so they can never collide with the 64-lowercase-hex shapecgroup::is_container_idchecks for — verified by reading the generator, not just assumed, so the "cgroup id and service id key the one map safely" design claim actually holds. - The new integration test (
managed_services_get_their_own_vram_and_a_weighted_power_split) runs a real daemon loop against real forked processes rather than only unit-testing the pure functions, and was hand-traced to genuinely fail (both services showing the device total) with the ancestry fallback reverted — not vacuous. - Both new
efficiency.rstests were hand-traced to flip under either a full revert or an invertedis_starting(), and correctly distinguishUnknown(stays in the split) fromStarting(left out when unmeasured).
Deployment notes
Linux-only feature: owner_of_pid is a None-returning stub on other platforms, so Windows/WSL-without-/proc behavior is unchanged. All four new ServiceRecord fields are #[serde(default)], so older on-disk registry records continue to parse without a migration.
What this covered
All 8 changed files read completely against f536619a…2d0794c3 (apps/rocm/src/dash.rs, new crates/rocm-dash-collectors/src/ancestry.rs, crates/rocm-dash-collectors/src/lib.rs, crates/rocm-dash-core/src/efficiency.rs, crates/rocm-dash-daemon/src/registry.rs, crates/rocm-dash-daemon/src/runner.rs, crates/rocm-dash-daemon/tests/telemetry_contract.rs, docs/architecture.md), fetched read-only (PR head never checked out) and diffed via git show/git diff against the merge base. Five independent fanned-out workers covered: (1) ancestry.rs+lib.rs, (2) registry.rs+telemetry_contract.rs, (3) runner.rs+efficiency.rs, (4) dash.rs+docs/architecture.md, (5) agent-instruction adherence (AGENTS.md) plus git-history and prior-fix-pattern passes across all 8 files. Also read for cross-file context: cgroup.rs, metrics.rs's InstanceStatus, rocm_core::ManagedServiceRecord, and rocm_core's service-id generator. Two of the five fan-out workers needed multiple re-spawns (a shared, environment-wide concurrency limit, not a content issue) before launching successfully; all five ultimately ran as genuine independent background passes — none were absorbed into my own context. Independently re-verified (not assumed): the merge state against current main (git merge-tree, clean — no conflict), and the live CI status for this exact head via the GitHub API plus the actual failing-job logs.
Did not run: cargo test/clippy/xtask — no Rust toolchain was reachable here; every "would this test still pass reverted/mutated" verdict above comes from static hand-tracing of the exact control flow and arithmetic, not execution, and is scored accordingly (none at 100). Did not retrieve ROCMAI-941's own ticket content (no tracker tool available in this environment).
Not reconciled against this PR's own prior review threads — this was a fresh blind pass per the method's independence rule. Immediately before publishing, I confirmed via the GitHub API that the head commit had not moved since I started, and that no review from this account exists yet at this exact head commit.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
2d0794c to
ffc40c7
Compare
|
Thanks. This round is in
Not changed: ancestry roots are still collected only once a service has a bound port, as Local runs on this commit: on Windows 2090 tests across the dash and |
…y process ancestry (ROCMAI-941) VRAM is attributed per instance by joining amd-smi's GPU process PIDs to the docker/containerd id in /proc/<pid>/cgroup. A managed `rocm serve` service is not a container, so it never joined: every managed service showed the whole device's VRAM, two services on one GPU each showed all of it, and the VRAM-weighted power split never applied to them. A service record carries the PIDs of its engine and launcher and their kernel start times. The new ancestry module walks a GPU process's /proc/<pid>/stat parent links up to a recorded root and returns that service's id. A root whose live start time differs from the recorded one is a recycled PID and claims nothing; an unreadable or looping chain ends with no owner. The runner tries the cgroup join first and ancestry second, so container ids and service ids key the same per-container map the instances already read. attribute_power_w also stops letting a service that is still starting, with no VRAM reading while a neighbour has one, take an equal share: it has not allocated GPU memory, so an equal split showed the serving neighbour about half its power and roughly double its tokens per watt until the newcomer loaded. With no readings at all, or a serving occupant without one, the split stays equal. Linux only: elsewhere ancestry yields no owner and attribution is unchanged. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
ffc40c7 to
9684e3b
Compare
Implements ROCMAI-941, a follow-up to ROCMAI-62 (#812).
VRAM is attributed per instance by joining
amd-smi's GPU process PIDs to the docker/containerd id in/proc/<pid>/cgroup. A managedrocm serveservice is not a container, so it never joined: every managed service showed the whole device's VRAM, two services on one GPU each showed all of it, and the VRAM-weighted power split never applied to them (#812 only stopped this raising a false warning).What changes
ancestry.rs(collectors): a service record carries the PIDs of its engine and launcher and their kernel start times.owner_ofwalks a GPU process's/proc/<pid>/statparent links (bounded depth, reader injected so it is testable on any host) until it reaches a recorded root, and returns that service's id. A root whose live start time differs from the recorded one is a recycled PID and claims nothing; an unreadable or looping chain ends with no owner.ServiceRecordreads the recorded pids,discover_managed_servicescollects the roots of live services, and the runner tries the cgroup join first and ancestry second, so container ids and service ids key the one per-container VRAM map the instances already read.startinginstances: an occupant that is stillStartingwith no VRAM reading, while a neighbour has one, is left out of the split. Before, it took an equal share, so a serving neighbour showed about half its real power and roughly double its tokens per watt until the newcomer loaded. With no readings at all, or a serving occupant that has no reading, the split stays equal.Unknownstatus is not treated as starting.Linux only: elsewhere ancestry yields no owner and attribution is unchanged.
Tests
ancestryunit tests (worker to owning service, two services told apart, recycled PID, unreadable links, cycles, self-parent, awkward process names), registry tests for the recorded pids and roots, theattribute_power_wcases above, andmanaged_services_get_their_own_vram_and_a_weighted_power_split: a Linux integration test that runs the real daemon loop with two real "engine" shells, each with a real child process standing in for a GPU worker, against a fakeamd-smi. It asserts each service shows its own VRAM (3000 and 1000 MB, not the device total) and a 300/100 W split with no false vram-attribution warning. With the ancestry roots disabled it fails with both services at(4000, 4000), the symptom this fixes.No Gherkin scenario: the dash scenarios need a PTY and a managed vLLM with real GPU process attribution, which no lane has. The daemon-level test above covers the behaviour end to end below the TUI.
Verification
cargo clippy --workspace --all-targets -- -D warningsclean;cargo test --workspace --all-targetspasses in 46 binaries; dash crates 1488 tests pass.-n dash-) passes 48 of 49 scenarios. The one failure,dash-24, needsamd-smi, which does not run on WSL, and fails the same way on plainmain.rocm serveon a GPU host withamd-smi(WSL has none), so the attribution is exercised against a fakeamd-smiand real process trees, not real hardware.