Skip to content

fix(examine): keep the rocminfo agent name past its ISA entries - #394

Open
sbmthakur wants to merge 1 commit into
ROCm:mainfrom
sbmthakur:fix/examine-rocminfo-agent-name
Open

sbmthakur wants to merge 1 commit into
ROCm:mainfrom
sbmthakur:fix/examine-rocminfo-agent-name

Conversation

@sbmthakur

@sbmthakur sbmthakur commented Sep 13, 2026 •

Copy link
Copy Markdown

Fixes #393

Motivation

rocminfo's parser in examine drops every GPU agent: a GPU agent lists its ISAs as further Name: lines after its own, and the walker let each one overwrite the agent name, so by the next Agent header nothing started with gfx. When #393 was filed that left every gpus[].gfx_target empty. Since #434 the kernel membership pass fills the target from KFD by PCI address, so the symptom in the report is gone — but rocminfo still contributes nothing: no marketing name for a card the PCI scan could not name (the ROCm-container case #434 was written for), no is_apu from the target's family, and the mapping it would do is positional.

Filling the iGPU's target also exposed a diagnose regression that is live on main today: on an APU+dGPU desktop with upstream PyTorch wheels (gfx900–gfx1102), fix-1-arch scores +55 against the iGPU (gfx1036) and tells the user to reinstall torch for a device fix-9 tells them to hide, while fix-2-unset-override goes quiet. Reported by @rominf in review; this PR fixes it.

Technical Details

The fix takes the first Name: (and Marketing Name:, BDFID:) after an Agent header and ignores the rest. The loop becomes parse_rocminfo_gpu_agents; apply_rocminfo_gpu_agents keeps its signature (main's test seam calls it with a string) and joins agents to entries by PCI address, rendering the BDFID in the dddd:bb:dd.f form pci_ids_match already compares, instead of by position. Only an agent with no usable BDFID: falls back to position, a fallback never displaces an exact match, and an agent whose BDFID: matched nothing becomes its own GPU, carrying its address.

rocminfo never overwrites a gfx_target already present: the membership pass has stamped the kernel's native target by address before it runs, and rocminfo's reading is the same value normally and the masqueraded one under HSA_OVERRIDE_GFX_VERSION. The probe also scrubs that variable from rocminfo's environment (run_scrubbing) — confirmed on the reference host, where HSA_OVERRIDE_GFX_VERSION=10.3.0 rocminfo reports gfx1030 for a gfx1100 card. What rocminfo contributes is the name the kernel cannot supply and an is_apu verdict from the target's family.

is_apu follows that family verdict wherever the table has one (gfx_apu_family_verdict: known APU / known discrete / not covered) and keeps the earlier name-based guess otherwise, on the hipInfo path too. A Raphael iGPU (gfx1036, not in the table) keeps lspci's APU verdict; a GPU appended with no entry to fill is left None when its family is unknown, so it cannot set has_discrete_amd. gfx_is_apu_family keeps its bool contract, so serve is unchanged.

In diagnose, serving_gfx_targets judges the discrete targets only when a host has both kinds of GPU, and check_1 names the integrated target it left out; a mixed host whose discrete target is not yet known falls back to every target rather than going quiet. Hosts with one kind of GPU are judged exactly as before.

Blast radius: the rocminfo probe and family classifier in examine.rs; the two arch checks in diagnose.rs. summary.detected_gfx_target, compatible_therock_family, the membership pass, serve's GPU selection and the sysfs fallback are unchanged.

Test Plan

  • examine.rs units, on rocminfo text: the APU+dGPU shape with ISA lines (fails on main), a lone agent at end of input (fails on main), a CPU-only host, the older shape without an ISA block; the address join with lspci enumerating in the opposite order, the no-BDFID positional fallback, an exact match never displaced, an unmatched-BDFID agent appended with its address, BDFID rendering; rocminfo not overwriting a kernel-filled target (the masqueraded-gfx case); the family-verdict rule and an appended unknown family staying None; and run_scrubbing dropping the override from the child. main's own a_rocminfo_marketing_name_outranks_the_kernel_topology_placeholder still passes against the new mapping.
  • diagnose.rs units: the gfx1100 + gfx1036 host against the upstream wheel list (check_1 does not fire, still fires when the discrete target is the missing one, check_2 still credits the covered card), an APU-only host judged unchanged, a mixed host with no discrete target falling back to all.
  • Gherkin: none added. examine-17 (@id:examine-lists-every-gpu-with-its-address-and-target, @requires-gpu) already asserts every GPU's PCI address and gfx target against the kernel topology on the GPU lanes, which is the observable end of the examine change. The diagnose change only diverges on an iGPU+dGPU host whose wheel misses the iGPU; no CI lane is that shape and the harness has no capability to gate one, so a scenario would skip everywhere — it is covered by the unit tests above instead.

Test Result

Host: Arch (Omarchy 4.0.2, kernel 7.1.9), RX 7900 XT (gfx1100) + Raphael iGPU (gfx1036), rocminfo from ROCm 7.2.4; rebased onto main at c1c2aeb.

cargo test -p rocm-core -- rocminfo_ bdfid family_verdict run_scrubbing appended_unknown a_rocminfo   # all pass
cargo test -p rocm-core -- arch_check override_check serving_targets   # all pass
cargo test -p rocm-core                              # 614 passed, 0 failed, 1 ignored
cargo test -p e2e-cucumber --test feature_naming     # 4 passed
cargo clippy -p rocm-core -p e2e-cucumber --all-targets -- -D warnings   # clean on the pinned 1.96.0
cargo xtask e2e -- -n examine-17                     # 1 scenario (1 passed), 0 unexpected failure(s) — main's per-GPU address/target check, green on the rebased code
rocm examine --json | jq '.gpus[] | {pci_id, gfx_target, is_apu}'   # 0000:03:00.0 gfx1100/false, 0000:11:00.0 gfx1036/true (= KFD topology)
HSA_OVERRIDE_GFX_VERSION=10.3.0 rocminfo | grep 'Name:.*gfx'       # gfx1030 — the override reaches a plain rocminfo; the probe scrubs it

Not run here: the Strix Halo / MI300X / MI350P / WSL2 lanes; examine-17 is gated to them.

Open questions

  1. Precedence of is_apu: this PR keeps a name-based guess where the family table has no verdict. Extending the table with the gfx103x (Van Gogh/Rembrandt/Raphael/Mendocino) and gfx90c/gfx902 integrated parts would let the target decide everywhere, but vram_capacity_is_meaningful changes with it on those hosts; happy to do that as a follow-up if that is the preferred direction.
  2. The fix-9-igpu-dgpu Linux plan hardcodes export HIP_VISIBLE_DEVICES=1 regardless of which ordinal the discrete GPU has (diagnose: fix-9-igpu-dgpu hardcodes HIP_VISIBLE_DEVICES=1, which is the iGPU on a desktop where the dGPU enumerates first #395).

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core parser fix here is right - guarding the Name:/Marketing Name: assignment with is_empty() so rocminfo's ISA sub-entries can't clobber the agent name is correct and well-tested. Two things worth fixing before merge, though, both related to the same underlying issue:

Comment thread crates/rocm-core/src/examine.rs Outdated
}
gpu.is_apu = Some(gfx_is_apu_family(&gfx));
if gpu.is_apu.is_none() {
gpu.is_apu = Some(gfx_is_apu_family(&gfx));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apply_rocminfo_gpu_agents only fills is_apu from rocminfo's authoritative gfx-target-based family check when it's None, but every AMD GPU from probe_gpus_lspci is always pushed with is_apu: Some(...), so this correction branch never actually fires in production - confirmed by tracing the probe order (lspci runs before rocminfo, and rocminfo only sees already-Some entries).

A future/rebranded AMD APU whose marketing name isn't yet in APU_KEYWORDS would get is_apu: Some(false) from lspci's keyword guess, and rocminfo's more accurate verdict would never be consulted to fix it, so has_apu/has_discrete_amd stay permanently wrong for that host. The parallel Windows path (hipInfo.exe) still does the unconditional overwrite, so this also creates a Linux/Windows inconsistency in the same class of logic.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, lspci always sets is_apu first, so that guard was dead. 53723ca replaces it with gfx_apu_family_verdict (known APU / known discrete / unknown): a known verdict overrides the name guess, on the hipInfo path as well; an unknown one (gfx1036) leaves lspci's verdict alone rather than flipping it to discrete. Extending the table is a follow-up.

})
.unwrap_or_default();
assert!(
targets.iter().any(|t| t == &expected),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new regression test only checks that some GPU in gpus[] carries the expected gfx_target, not that it's attached to the correct GPU record, so it can't catch a positional mismatch between rocminfo's agent order and lspci's enumeration order.

On a host with both an APU and a discrete GPU (the exact scenario #393 was about) where rocminfo's agent order differs from lspci's PCI order, apply_rocminfo_gpu_agents's positional zip could swap gfx_targets between the two GPU entries; since the assertion is targets.iter().any(|t| t == &expected), the swapped-but-still-present target still satisfies it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at the mapping: agents now join lspci entries on BDFID: (= lspci's bus/device/function), with positional fallback only for agents that have no BDFID, and a fallback never displaces an exact match. Unit tests cover the reversed-order case. The step now also checks each gpus[] record against KFD's target for that PCI address where /sys/class/kfd is readable; presence remains the floor elsewhere.

@sbmthakur
sbmthakur force-pushed the fix/examine-rocminfo-agent-name branch from 90d83ae to 53723ca Compare September 20, 2026 04:40

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parser fix is right, and the BDFID join answers the earlier ordering concern properly. First-Name:-wins, the end-of-input commit, and pass-1-before-pass-2 all hold on the shapes I tried, and the unit tests cover them well.

One thing I think needs handling before this merges: filling gfx_target for every AMD GPU changes rocm diagnose in a way the description doesn't cover, and it goes the wrong way on exactly the APU+dGPU host this fixes.

check_1_arch_not_in_wheel reads amd_gfx_targets(), which is every AMD GPU with a non-empty target, integrated ones included. Until now that list was empty on this host class: probe_gpus_rocminfo dropped every agent (the bug you're fixing), and probe_gpus_lspci only guesses a target for the names in its APU early-return list, which "Raphael" is not one of. So diagnose.rs:425 was effectively dead there. With the fix it returns both targets, and an iGPU target such as gfx1036 is in no PyTorch ROCm wheel's arch list, so the "not in framework arch list" branch scores +55.

Ran at 53723ca against the same Examination twice — once with both gfx_targets empty (pre-fix), once filled (post-fix), with a typical PyTorch ROCm framework_arch_list:

BEFORE: fix-1-arch not diagnosed
AFTER:  score 55 — GPU gfx target(s) ["gfx1036"] not in framework arch list
        ["gfx900", ..., "gfx1100", "gfx1101", "gfx1102"]

So on your reference box rocm diagnose now tells the user to uninstall and reinstall torch from a different wheel index, while the card they are actually serving on (gfx1100) is fully covered. A discrete-only host is unaffected — gfx1100 is in the list, the branch scores -30 and stays filtered out — so this is specific to the APU+dGPU pairing.

Suggestion: scope the coverage test to the GPUs the user would serve on rather than every AMD target. When has_apu && has_discrete_amd, compare framework_arch_list against the is_apu == Some(false) targets (the ones fix-9 already tells them to pin to); or at minimum skip the +55 branch when every missing target is an APU and some discrete target is covered. Worth a diagnose regression test on the gfx1100 + gfx1036 shape.

If you'd rather keep this PR to the parser and take that as a follow-up, that works — but I don't think it should land silently, since it is user-visible output on the very host in #393.

Two smaller notes, neither blocking:

  • gfx_apu_family_verdict is pub, but it is only used inside examine.rs and its tests; private would do.
  • The doc on apply_rocminfo_gpu_agents says only an agent with no usable BDFID: falls back to position, but any agent pass 1 failed to place falls through to pass 2 — including one whose BDFID matched nothing. The parenthetical half-covers it; worth tightening.

Verified locally at 53723ca on Linux: the 12 new unit tests pass; cargo test --workspace --all-targets is green apart from three pre-existing rocm-dash-tui agent::tests failures in a crate this PR does not touch; both CI clippy invocations are clean; feature_naming passes, so examine-16's id/index and the expectations key line up; and examine-16 fails deterministically on WSL, since probe_with_interpreter skips all GPU probes there, so the non-flaky xfail row is right. I did not run the GPU lanes.

@rominf

rominf commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Approved the pending workflow runs — all three were sitting at action_required because this is a fork PR, so the branch had no CI at all until now. Results, replacing the "I did not run the GPU lanes" caveat in my review:

CI ✅  ·  CodeQL ✅

E2E self-hosted — examine-16 passed on every lane that finished:

Lane Lane examine-16
MI300X ✅ passed
MI350P ✅ passed
rad3 R9700 ✅ passed
Strix Halo, Windows ✅ passed
Strix Halo, WSL2 ✅ not applicable (below)
Strix Halo, Ubuntu ⚠️ cancelled ran; not the hang

The Strix Halo Ubuntu lane hit its 45-minute timeout-minutes cap, which GitHub reports as cancelled. It is not this PR: that run's commands.jsonl shows the suite got through examine-16 and on through logs-01..03, then stopped at serve-01 - Short model names are expanded to their full name on the vllm engine and sat there until the cap. Nothing here touches the serve path. That lane is also independently unhealthy right now — it reported failure on main's own run earlier today (35699859214).

One thing the run did not confirm: the new WSL xfail row. @requires-gpu is not satisfied on that WSL runner, so examine-04, -15 and -16 were all filtered out as not-applicable; examine-08 ran and failed exactly as its row expects, but examine-json-names-target-per-gpu never resolved to anything. The row isn't wrong — it's the right guard for a WSL host reporting driver_status: wsl_rocdxg_ready — it is just unexercised by the lane as configured, so it's worth knowing it is unproven rather than verified.

One note on how much the four green lanes actually prove, since they read stronger than they are: the per-record KFD cross-check only fires for gpus[] entries whose pci_id matches a KFD node, and on a host that falls through to probe_gpus_sysfs_fallback the appended record has an empty pci_id, so the cross-check silently skips. The lane artifacts don't capture gpus[], so I can't tell from CI which lanes exercised it. Not a change request — just that the assertion may be thinner on some lanes than a green tick suggests.

Green CI doesn't bear on the fix-1-arch regression in my review — there is no diagnose coverage for that shape — so that one still stands.

@sbmthakur
sbmthakur force-pushed the fix/examine-rocminfo-agent-name branch from 53723ca to 1800405 Compare September 26, 2026 23:07
@sbmthakur

Copy link
Copy Markdown
Author

The parser fix is right, and the BDFID join answers the earlier ordering concern properly. First-Name:-wins, the end-of-input commit, and pass-1-before-pass-2 all hold on the shapes I tried, and the unit tests cover them well.

One thing I think needs handling before this merges: filling gfx_target for every AMD GPU changes rocm diagnose in a way the description doesn't cover, and it goes the wrong way on exactly the APU+dGPU host this fixes.

check_1_arch_not_in_wheel reads amd_gfx_targets(), which is every AMD GPU with a non-empty target, integrated ones included. Until now that list was empty on this host class: probe_gpus_rocminfo dropped every agent (the bug you're fixing), and probe_gpus_lspci only guesses a target for the names in its APU early-return list, which "Raphael" is not one of. So diagnose.rs:425 was effectively dead there. With the fix it returns both targets, and an iGPU target such as gfx1036 is in no PyTorch ROCm wheel's arch list, so the "not in framework arch list" branch scores +55.

Ran at 53723ca against the same Examination twice — once with both gfx_targets empty (pre-fix), once filled (post-fix), with a typical PyTorch ROCm framework_arch_list:

BEFORE: fix-1-arch not diagnosed
AFTER:  score 55 — GPU gfx target(s) ["gfx1036"] not in framework arch list
        ["gfx900", ..., "gfx1100", "gfx1101", "gfx1102"]

So on your reference box rocm diagnose now tells the user to uninstall and reinstall torch from a different wheel index, while the card they are actually serving on (gfx1100) is fully covered. A discrete-only host is unaffected — gfx1100 is in the list, the branch scores -30 and stays filtered out — so this is specific to the APU+dGPU pairing.

Suggestion: scope the coverage test to the GPUs the user would serve on rather than every AMD target. When has_apu && has_discrete_amd, compare framework_arch_list against the is_apu == Some(false) targets (the ones fix-9 already tells them to pin to); or at minimum skip the +55 branch when every missing target is an APU and some discrete target is covered. Worth a diagnose regression test on the gfx1100 + gfx1036 shape.

If you'd rather keep this PR to the parser and take that as a follow-up, that works — but I don't think it should land silently, since it is user-visible output on the very host in #393.

Two smaller notes, neither blocking:

  • gfx_apu_family_verdict is pub, but it is only used inside examine.rs and its tests; private would do.
  • The doc on apply_rocminfo_gpu_agents says only an agent with no usable BDFID: falls back to position, but any agent pass 1 failed to place falls through to pass 2 — including one whose BDFID matched nothing. The parenthetical half-covers it; worth tightening.

Verified locally at 53723ca on Linux: the 12 new unit tests pass; cargo test --workspace --all-targets is green apart from three pre-existing rocm-dash-tui agent::tests failures in a crate this PR does not touch; both CI clippy invocations are clean; feature_naming passes, so examine-16's id/index and the expectations key line up; and examine-16 fails deterministically on WSL, since probe_with_interpreter skips all GPU probes there, so the non-flaky xfail row is right. I did not run the GPU lanes.

Thanks for catching it @rominf. It's my own host class too, not a different one. I never saw it live because examine reads the managed TheRock torch here, and that wheel's arch list includes gfx1036. On the pytorch.org wheels it behaves exactly as you describe. Pushed 1800405 with the fix in this PR rather than punting it.

I added serving_gfx_targets in diagnose.rs. When a host has both an integrated and a discrete AMD GPU, the arch checks now judge the discrete targets only, the same ones fix-9 tells you to pin to. Hosts with one kind of GPU are judged on everything, same as before. check_1 uses it and adds an evidence line naming the iGPU target it left out. While I was in there I found check_2 (HSA_OVERRIDE unneeded) had the mirror problem, since its all() went false once the iGPU showed up, so it uses the same helper.

Tests on the gfx1100 + gfx1036 shape with the upstream wheel list: check_1 scores 0 and doesn't pick up the +55 even with the ISA keyword, still fires if the discrete target itself is missing, and check_2 still credits the covered card. A Strix Halo-only fixture comes out unchanged.

Also took the two smaller ones. gfx_apu_family_verdict is private now, and the apply_rocminfo_gpu_agents doc says what pass 2 really takes, which is any agent pass 1 didn't place, whether it had no BDFID or an address lspci didn't list.

Ran the diagnose and examine unit tests, clippy with -D warnings on rocm-core and e2e-cucumber, and examine-16 on my box.

@sbmthakur
sbmthakur requested a review from rominf September 26, 2026 23:12

@juhovainio juhovainio left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The branch got squashed to a single commit since the last round, so I re-read the whole diff rather than diffing since my last review - and re-checked it against all three specific concerns raised (my two, plus rominf's blocking one).

  1. My "dead correction branch" finding: fixed. apply_family_verdict now unconditionally overrides is_apu whenever the family table has a verdict, and I traced both of its production call sites (apply_rocminfo_gpu_agents and the Windows probe_hip_sdk_windows path) to confirm neither gates it behind is_apu.is_none() anymore - it genuinely fires now, not just in the isolated unit test.
  2. My "weak regression test" finding: fixed, and more thoroughly than I expected. apply_rocminfo_gpu_agents now joins rocminfo agents to lspci GPUs by PCI address (BDFID) instead of position, with two new unit tests (rocminfo_agents_join_lspci_gpus_by_pci_address_not_position, rocminfo_bdfid_match_is_never_displaced_by_a_positional_agent) that construct exactly the swapped-order APU+dGPU scenario and assert the target lands on the correct GPU record, not just "present somewhere in the list." The new e2e scenario (examine-16) goes further, checking each GPU's target against what KFD's own topology reports for that PCI address.
  3. rominf's blocking finding (diagnose false-positive on APU+dGPU hosts): fixed. The new serving_gfx_targets helper in diagnose.rs judges only the discrete targets against the framework arch list when a host has both an APU and a discrete GPU, and reports which integrated target it excluded. Four new tests pin this (both checks, both the "iGPU present" and "no discrete GPU" cases).

Also confirmed rominf's non-blocking pub note was addressed - gfx_apu_family_verdict is private now.

Verified independently rather than taking the commit message's word for it: built and ran the test suite myself in a disposable worktree at this head (cargo test -p rocm-core --lib examine:: -> 45/45, cargo test -p rocm-core --lib diagnose:: -> 61/61).

One caveat worth flagging plainly: this is a fork PR and all three CI workflows (CI, CodeQL, E2E self-hosted) were sitting at action_required - they haven't actually run at this head, so I have no CI signal to point to, only my own local run above.

Enabled the workflows now to get final checks from CI and approving based on the code and my own verification.

@rominf

rominf commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

🔴 Automated review · review-and-fix · 1800405

Summary

Fixes the examine rocminfo walker so a GPU agent's ISA Name: lines no longer overwrite the agent name, joins agents to lspci entries by BDFID/PCI address instead of position, makes is_apu follow the gfx-family table where it has a verdict, scopes diagnose fix-1/fix-2 arch checks to discrete targets on mixed iGPU+dGPU hosts, and adds e2e scenario examine-16. Outcome: Needs work. Reviewed: the whole change (5 files) against merge base 3edfb693, plus its interaction with current main (#434, #444, #453). Verified: static review only — nothing from the PR tree was built or run (fork PR); findings come from reading the code and git merge-tree against main. Blocking: 3 · Non-blocking: 6.

🚫 Blocking

  1. crates/rocm-core/src/examine.rs:1241,1307 — an HSA_OVERRIDE_GFX_VERSION-overridden target can be reported as native. rocminfo runs with the inherited environment; if the runtime reports the override as the agent Name: (the commonly reported ROCr behaviour — not confirmed on hardware in this review), line 1307 writes it into gpus[].gfx_target as authoritative. check_2_hsa_override_unneeded (diagnose.rs:521) would then compare the overridden target against the wheel list — e.g. a gfx1036 user with HSA_OVERRIDE_GFX_VERSION=10.3.0 sees gfx1030 and gets "+25: the override is hiding the native gfx", i.e. advice to remove the override that makes their GPU work. Before this PR the targets were empty, so that branch never fired. Fix: run rocminfo with HSA_OVERRIDE_GFX_VERSION removed from its environment, or keep a native (KFD/sysfs) target and never overwrite it; add a test for the override case.
  2. Conflicts with main, which already covers most of examine: gfx_target is empty for every GPU even though rocminfo lists them → ISA Name: lines overwrite the agent name #393. examine.rs, diagnose.rs and examine.feature conflict. fix(examine): enumerate Instinct GPUs from lspci's accelerator class #434 added a KFD membership pass that already fills each GPU's native gfx_target by PCI address (and rocminfo runs after it); examine-16 is now taken (@id:examine-json-names-the-setup-runtime-folder); examine-17 already asserts per-GPU address/target against KFD. After a rebase, the PR's "rocminfo is authoritative" overwrite would replace KFD's native target — turning finding 1 into a regression on main, and the description's "fails on main" claim for examine-16 likely no longer holds. Fix: rebase; let rocminfo fill only a still-empty target (as the KFD pass does) and supply names / is_apu; renumber the scenario (examine-18) or fold its per-address assertion into examine-17; update the description.
  3. No Gherkin scenario for the rocm diagnose behaviour change (AGENTS.md §3). fix-1/fix-2 now give different verdicts on mixed iGPU+dGPU hosts and fix-1 prints a new "integrated GPU target(s) … not judged" evidence line — user-observable CLI output covered only by unit tests, with no justification in the PR text. Fix: add a scenario in features/diagnose.feature, or name the existing @id: that covers it and the lane it runs on.

Non-blocking

  • diagnose.rs:386 — serving_gfx_targets returns empty when the discrete GPU has no target (e.g. Windows, where hipInfo fills only the first GPU), so fix-1/fix-2 silently stop judging; fall back to all targets when the discrete list is empty.
  • examine.rs:1315 — the append path still treats an unknown family as discrete, contradicting the rule apply_family_verdict documents (a Raphael iGPU in a container without lspci counts as discrete → has_discrete_amd on → fix-9 stays quiet); use gfx_apu_family_verdict (None when unknown).
  • examine.rs:1291 — pass 2 also places agents whose BDFID matched no lspci entry, not just agents without one: on partitioned Instinct several agents share a BDFID and extras can land on an unrelated card, and pci_bdf_id (:1224) drops the PCI domain so 0000:03:00.0/0001:03:00.0 collide; limit the positional fallback to bdf == None.
  • examine.rs:2133 — nothing tests the hipInfo call site; reverting it to gfx_is_apu_family keeps the suite green.
  • Commit message and PR description still say "only an agent with no usable BDFID: falls back", but the code also falls back for an unmatched BDFID; the doc comment was corrected, these two weren't.
  • examine_steps.rs:557 — kfd_gpu_targets_by_pci_id re-implements main's kfd_node_pci_id decode in lib.rs without its location_id == 0 guard; reuse that helper after the rebase.

The rocminfo walker kept a running agent name that every `Name:` line
overwrote, and a GPU agent lists its ISAs as further `Name:` lines
(`amdgcn-amd-amdhsa--gfx1100`, `...--gfx11-generic`) after `Device Type:`.
By the next `Agent` header the name no longer started with `gfx`, so every
GPU agent was dropped and rocminfo contributed nothing: no marketing name
for a card the PCI scan could not name, no `is_apu` from the target's
family. The per-GPU `gfx_target` that was empty when ROCm#393 was filed is now
filled by the kernel membership pass (ROCm#434); this fixes the parser so the
rest of rocminfo's reading reaches the report.

Take the first `Name:` after an `Agent` header as the agent's name and
ignore the rest, same for `Marketing Name:` and `BDFID:`. The loop is
lifted into `parse_rocminfo_gpu_agents`, and `apply_rocminfo_gpu_agents`
joins agents to entries by PCI address (the `BDFID` rendered in the
`dddd:bb:dd.f` form `pci_ids_match` already compares) instead of by
position, so rocminfo's and lspci's enumeration orders need not agree.
Only an agent with no usable `BDFID:` falls back to position, a fallback
never displaces an exact match, and an agent whose `BDFID:` matched nothing
becomes its own GPU rather than land on an unrelated card.

rocminfo never overwrites a `gfx_target` already present. The membership
pass stamps each entry with the kernel's native target by address before
rocminfo runs; rocminfo's reading is the same value normally and the
masqueraded one under `HSA_OVERRIDE_GFX_VERSION`, which the probe also
scrubs from rocminfo's environment so an override cannot be read back as
the hardware's gfx.

`is_apu` follows the target's family wherever the family table has a
verdict (`gfx_apu_family_verdict`: known APU, known discrete, or not
covered) and keeps the earlier name-based guess otherwise; the hipInfo path
on Windows uses the same rule. A Raphael iGPU (gfx1036, not in the table)
keeps lspci's APU verdict instead of flipping to discrete, and a GPU
appended with no entry to fill is left `None` when its family is unknown.

`rocm diagnose`'s two framework-arch checks read every AMD target. Now
that ROCm#434 fills the iGPU's target, an APU+dGPU desktop on upstream PyTorch
wheels is told to reinstall torch for a device fix-9 tells it to hide, and
the mirror check goes quiet. `serving_gfx_targets` judges the discrete
targets only when a host has both kinds of GPU, naming the integrated
target it left out, and falls back to every target when no discrete one is
known. Hosts with one kind are judged as before.

No new scenario: examine-17 already asserts every GPU's PCI address and
gfx target against the kernel topology, and the diagnose change only
diverges on an iGPU+dGPU host whose wheel misses the iGPU, which no CI lane
provides; that shape is covered by unit tests.

Fixes ROCm#393

Signed-off-by: Shubham Thakur <sbmthakur@gmail.com>
@sbmthakur
sbmthakur force-pushed the fix/examine-rocminfo-agent-name branch from 1800405 to c8c72d3 Compare October 12, 2026 06:15
@sbmthakur

Copy link
Copy Markdown
Author

Thanks @rominf, and sorry for the lag — main moved a long way while this sat. Rebased onto c1c2aeb and reworked the PR around what #434 changed. Pushed c8c72d3. Per finding:

Blocking

  1. Override leak — fixed, and it reproduces on my box. HSA_OVERRIDE_GFX_VERSION=10.3.0 rocminfo here reports the agent Name: as gfx1030, not the real gfx1100, so this wasn't just the commonly-reported case. Two layers now: the target probe runs rocminfo through run_scrubbing, which drops that var from the child env (unit-tested), and rocminfo no longer overwrites a gfx_target at all — see 2.

  2. Conflict with main / fix(examine): enumerate Instinct GPUs from lspci's accelerator class #434 — rebased, and the design is now the one you described. The membership pass stamps each GPU's native target from KFD by address before rocminfo runs, so rocminfo now fills a target only when it is still empty and otherwise supplies just the marketing name and is_apu. The join reuses pci_ids_match by rendering the BDFID into the same dddd:bb:dd.f form the kernel topology uses; pci_bdf_id is gone. apply_rocminfo_gpu_agents keeps its (e, out) signature so your test seam and a_rocminfo_marketing_name_outranks_the_kernel_topology_placeholder still work (that test passes). Dropped my examine-16 and its WSL xfail row entirely: examine-17 already asserts every GPU's address and target against KFD, which is the observable end of this change. Added a unit test for the deference itself — a kernel-filled gfx1100 entry stays gfx1100 when rocminfo reports a masqueraded gfx1030 at the same address.

    One thing worth saying plainly: the parser is still broken on main (every Name: line still clobbers the agent name), so main's own gpu.gfx_target = gfx.clone() overwrite is dead today only because no agent survives. Fixing the parser without the deference rule would have resurrected that overwrite — exactly the regression you flagged.

  3. No diagnose Gherkin — justified rather than added, and I'd value your read. The fix-1 regression you found is live on main now (since fix(examine): enumerate Instinct GPUs from lspci's accelerator class #434 fills the iGPU's target), so this PR carries the fix for it. But the behaviour only diverges on an iGPU+dGPU host whose wheel misses the iGPU; the lanes are MI300X, MI350P, rad3 and single-kind Strix Halo, and the harness has no capability to gate "has both." A scenario would resolve to skip everywhere, which is the unproven-row problem you called out on examine-16's WSL guard. It's covered by five diagnose unit tests and stated in the PR text. If you want a gated scenario, it needs a new capability tag first; happy to add both if you agree.

Non-blocking

  • serving_gfx_targets empty-discrete — fixed, falls back to all known targets rather than judging nothing. Test added.
  • append path unknown family as discrete — fixed, appended GPU takes gfx_apu_family_verdict (None when unknown). Test added.
  • pass 2 placing unmatched-BDFID agents — fixed. Positional fallback is limited to agents with no BDFID; an unmatched one becomes its own GPU, carrying its address. I didn't add the PCI domain to the key: rocminfo's BDFID is domain-less here (768 = bus<<8, same as KFD's location_id), so the join renders domain 0000 the way kfd_node_pci_id does, and a domain in the key would stop every single-domain host matching. Test added.
  • hipInfo call site untested — it now calls the same apply_family_verdict the rocminfo path does, and that helper is unit-tested; the Windows-only wiring can't run on a Linux lane.
  • commit/description wording — fixed, both match the code.
  • reuse kfd_node_pci_id — done in spirit: the rocminfo join uses pci_ids_match and the same address rendering; kfd_gpu_targets_by_pci_id is gone with the dropped scenario.

Re-ran on the rebased tree: the 22 touched units, the full rocm-core suite (614 passed, 0 failed — the /opt/rocm test is fixed on main now), clippy -D warnings on both crates, feature_naming, and examine-17 (main's per-GPU address/target check) — 1 scenario, 1 passed on my box.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · c8c72d3

Review — needs work

Full review of the whole change.
Closes #393: the parser fix answers the root cause and expected behaviour the issue describes. Where the family table has no verdict, is_apu keeps the name guess, as the PR's own open question 1 says.

Blocking

  • The Windows build calls a function that only exists on Linux and in tests — crates/rocm-core/src/examine.rs:1482, crates/rocm-core/src/examine.rs:1842-1843
    pci_ids_match is still #[cfg(any(target_os = "linux", test))]. The new pass 1 in apply_rocminfo_gpu_agents calls it, and that function has no cfg gate. Neither do its callers probe_gpus_rocminfo and probe_gpus_after_lspci. The comment at examine.rs:1678-1681 says the latter "still ha[s] to compile on Windows", because it is reached through a runtime os_family check. A non-test Windows build (the release build, and rocm-core built as a dependency of apps/rocm) therefore cannot resolve the symbol. host_gpu.rs:1003-1007 documents the same trap ("test alone would not compile on Windows CI, which is how this was found"). The only status reported at this head is the docs build, so no CI caught it.
    Confidence 85 · mechanical · Fix: drop the cfg gate from pci_ids_match (it is a pure const fn), and run the Windows clippy job.
  • On a host with a non-zero PCI domain, every GPU is listed twice — crates/rocm-core/src/examine.rs:1433-1441, crates/rocm-core/src/examine.rs:1485-1497, crates/rocm-core/src/examine.rs:1514-1530
    bdfid_to_pci_id always renders domain 0000. lspci -D and the KFD reader (kfd_node_pci_id, which has a 0005: test case in host_gpu.rs) both carry the real domain. On such a host pass 1 matches nothing. Pass 2 skips the agent because pci_id.is_some(), so the agent is appended as a new Gpu beside its own lspci/KFD entry. The AMD GPU count doubles: a single-GPU host enters check_11's multi-GPU path (diagnose.rs:1327), and gpus[] lists phantom cards. Before this change the code had no append-on-mismatch path. The doc comment's premise ("what every single-domain host has") is the gap, and no test covers it.
    Confidence 70 · logic · Fix: when the agent has no domain, compare bus:dev.fn and ignore the entry's domain. Otherwise let a domain mismatch fall through to the positional pass instead of appending. Add a test with a 0001: entry.
  • rocminfo_agent_with_an_unmatched_bdfid_is_appended_not_misplaced cannot fail on the branch it claims to cover — crates/rocm-core/src/examine.rs:5695-5716, guard at crates/rocm-core/src/examine.rs:1490
    Its comment says an unmatched agent is "never a positional guess onto an unrelated card". The fixture's only entry, 0000:03:00.0, is already claimed by agent 1's exact match, so pass 2 has nothing free to assign. Delete the || agents[agent_idx].pci_id.is_some() guard and agent 2 is still appended, so every assertion still passes.
    Confidence 85 · logic · Fix: add a second, unclaimed AMD entry at another address, and assert it keeps an empty gfx_target.

Test verdicts for the other new tests:

  • These fail if the code they cover is reverted, and they exercise their branch:
    • rocminfo_gpu_agents_survive_isa_name_lines
    • rocminfo_last_agent_is_committed_at_end_of_input
    • rocminfo_agents_join_entries_by_pci_address_not_position
    • rocminfo_bdfid_match_is_never_displaced_by_a_positional_agent
    • rocminfo_never_overwrites_a_target_the_kernel_already_filled
    • appended_unknown_family_gpu_is_not_called_discrete
    • family_verdict_overrides_a_name_guess_only_where_the_table_knows
    • run_scrubbing_removes_the_named_var_from_the_child
    • the four diagnose.rs arch/override/serving-target tests
  • rocminfo_agents_without_bdfid_fall_back_to_position passes with pass 2 broken in one way only: if both agents went unmatched they would be appended. That still fails on the len() == 2 assertion, so it holds.

Non-blocking

  • The "never overwrite" rule rests on a premise that is false for targets guessed from the lspci name — crates/rocm-core/src/examine.rs:1459-1463, crates/rocm-core/src/examine.rs:1507-1509, crates/rocm-core/src/examine.rs:1784
    The doc (and the description) say the membership pass "has by now stamped each entry with the kernel's native target". It does not overwrite either. It keeps any target classify_amd_marketing_name already guessed (if gpu.gfx_target.is_empty()), and that rule is pinned by a_target_the_pci_scan_already_resolved_is_never_overwritten. A concrete case in this repo: "krackan" maps to gfx1150 (examine.rs:1145-1150), while host_gpu.rs:643 maps Krackan (1114) to gfx1152. After this change neither KFD nor rocminfo can correct that guess, so check_1 judges gfx1150. On main the broken parser never corrected it either, so this is a fix that does not restore the correction rather than a new regression. The override masquerade the rule guards against is already handled by run_scrubbing.
    Confidence 75 · logic · Fix: let a KFD- or rocminfo-reported target replace a name-derived guess, or correct the doc and the description so they do not claim the kernel's target always wins.
  • A stale comment says rocminfo still matches by position — crates/rocm-core/src/examine.rs:1633-1636
    "rocminfo maps its agents onto the surviving entries by position, so it has to see the reconciled list". After this change the join is by PCI address, and position is only the fallback.
    Confidence 85 · mechanical · Fix: reword it to give the actual reason for the ordering.
  • examine-17 passes on main and cannot tell this change apart — tests/e2e-cucumber/tests/e2e/examine_steps.rs:958-1025
    The description cites @id:examine-lists-every-gpu-with-its-address-and-target as "the observable end of the examine change". It asserts that pci_id and gfx_target are non-empty, and the membership pass already supplies both without rocminfo. On a mixed-target host, which is the APU+dGPU shape this PR fixes, it returns before checking targets at all. Nothing at scenario level observes rocminfo's contributions: the marketing name, is_apu, or the address join.
    Confidence 85 · architectural · Fix: either say in the description that no scenario discriminates the examine change, or add one that asserts the rocminfo-supplied name/is_apu on a lane that can.
  • Another rocminfo read still sees the override — crates/rocm-core/src/host_gpu.rs:72-85 (pre-existing)
    detect_host_gfx_target_with_context runs rocminfo and rocm_agent_enumerator unscrubbed, so under HSA_OVERRIDE_GFX_VERSION the masqueraded target is read back. That is exactly the case run_scrubbing exists for. Only the late fallback is affected.
    Confidence 70 · logic · Fix: route both through the scrubbing runner in a follow-up.

Decisions for the author

  • serving_gfx_targets puts is_apu: None GPUs in the excluded "integrated" bucket — tradeoff
    .partition(|g| g.is_apu == Some(false)) (diagnose.rs:566-570) excludes unknown-family GPUs on a mixed host, and check_1 labels them "integrated GPU target(s) … not judged" (diagnose.rs:627-630).
    • For this rule: an appended Raphael iGPU deliberately stays None (appended_unknown_family_gpu_is_not_called_discrete), and judging it would bring back the +55 false positive this PR removes.
    • Against it: an unclassified discrete card (e.g. a KFD placeholder or appended agent of an RDNA4 family not in the table) is silently left out of the arch check and mislabelled "integrated". fix-9 treats None as neither kind.
    • Either way, a test with three GPUs including a None one would pin the choice. "not judged (integrated or unclassified)" would make the evidence line accurate whichever way you go.

Positive signals

  • Exact-address matching runs as its own pass before any positional fallback. That makes "a fallback never displaces an exact match" a property of the structure, not of input order, and a dedicated test pins it.
  • run_scrubbing_removes_the_named_var_from_the_child proves both halves: a plain run inherits the override and the scrubbed run does not. That is a real check of env_remove on an inherited variable, not a string pin.
  • When no discrete target is known, serving_gfx_targets falls back to judging every target rather than going quiet. That avoids turning a detection gap into a disabled check.

Deployment notes

  • rocm examine --json output changes for AMD GPUs:
    • name can now come from rocminfo's marketing name.
    • is_apu follows the family table where it has a verdict.
    • An agent with an unmatched BDFID adds a gpus[] entry carrying its pci_id.
  • check_2's evidence text changes from "every detected GPU target" to "every GPU target serving uses", and check_1 gains an "integrated GPU target(s) … not judged" line. Nothing else in the repository matches the old string.

What this covered

Every changed line in crates/rocm-core/src/examine.rs and crates/rocm-core/src/diagnose.rs, at c1c2aeb…c8c72d33c57d (one commit). This was split across two independent workers: a correctness pass over both files with their callers, and an impact/history/convention pass. Beyond the changed lines it read the probe ordering, the membership pass, the host_gpu.rs KFD/rocminfo readers, the examine-17 scenario and its steps, expectations.toml (no stale #393 row), fix.rs recipes and the rocm-doctor reference (no update needed), and AGENTS.md/CONTRIBUTING.md. Issue #393 was read by its identifier.

Did not run: no Rust toolchain was available here, so nothing was compiled or executed. Every confidence comes from reading and tracing, and the Windows break in particular is unconfirmed by a build. CI: the only status reported at c8c72d3 is the docs build. The prior-changes pass (earlier review comments on these files) did not run. Not reconciled against the discussion on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

examine: gfx_target is empty for every GPU even though rocminfo lists them → ISA Name: lines overwrite the agent name

4 participants