Skip to content

fix(examine): attribute rocminfo GPU readings by PCI address, not list position (ROCMAI-40) - #1037

Open
rominf wants to merge 5 commits into
mainfrom
ROCMAI-40-fix-rocminfo-gpu-match-by-pci-address
Open

rominf wants to merge 5 commits into
mainfrom
ROCMAI-40-fix-rocminfo-gpu-match-by-pci-address

Conversation

@rominf

@rominf rominf commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Why

rocm examine could report one GPU's model name, gfx target and integrated/discrete verdict on a different GPU. It folded the GPU agents rocminfo reports into its GPU list by list position: the first agent went onto the first listed AMD GPU, and so on, with no address or count check. rocminfo lists only what the ROCm runtime exposes, so under a visibility mask such as ROCR_VISIBLE_DEVICES (for example to hide an iGPU) it sees a subset of the cards, possibly in another order, and each reading landed on the wrong card. On a simulated host with GPUs at 03:00.0 (gfx1100) and 04:00.0 (gfx1101) and a runtime that sees only the second, gpus[0] was reported as the second card's "AMD Radeon RX 7800 XT" / gfx1101, and the visible card got nothing.

A second defect hid the first on real hosts: the parser took the Name: lines inside each agent's ISA Info: block (amdgcn-amd-amdhsa--gfx1151, then amdgcn-amd-amdhsa--gfx11-generic) for the agent's name, so the last one replaced gfx1151 and every GPU agent of a real rocminfo listing was dropped. rocminfo data never reached rocm examine at all; the earlier stand-ins left that block out, which is why no test noticed.

These fields feed rocm examine --json (gpus[], has_apu, has_discrete_amd) and, from there, rocm diagnose: the iGPU/dGPU collision check (fix-9-igpu-dgpu, gated on has_apu && has_discrete_amd), the HSA_OVERRIDE_GFX_VERSION check (check_2_hsa_override_unneeded, which holds the detected targets against the framework's arch list), the unsupported-GPU check (check_20_gpu_unsupported_by_rocm), and the report gate (prepare_report, which refuses when an AMD GPU's target is empty or unreleased).

Scope and blast radius

The ISA Info: parser fix means the whole rocminfo pass (model names, gfx targets, is_apu) takes effect on real Linux hosts for the first time. The other behaviour changes here decide what that newly live pass may write: which readings are attributed to which GPU, which source's gfx target wins, and how is_apu is decided. Shipping the parser fix alone would turn the old position-based overwrite on for real hosts, which is the defect this change fixes, so they stay together. The decision and its blast radius are recorded on ROCMAI-40.

What changes in rocm examine --json

Matching rocminfo agents to listed GPUs (the cases are lettered (a)-(g) in the doc of apply_rocminfo_gpu_agents, and each is cited by a test):

  • The parser skips each agent's ISA Info: block, so a real listing yields its GPU agents.
  • Each agent's BDFID: is matched to the listed AMD GPU with the same bus, device and function. BDFID uses the kernel's pci_dev_id() encoding, the same as the KFD topology's location_id, and both now decode through one shared helper (host_gpu::pci_bus_device_function), which refuses zero (the host bridge, and what rocminfo prints for CPU agents) and values wider than 16 bits.
  • rocminfo prints no PCI domain, so an address shared by GPUs in two domains is refused. A partitioned GC 9.4.3 card is expected to arrive with one address per partition (11:00.0, 11:00.1, ...) on both sides, per upstream kernel and ROCm sources, so its partitions match one to one; entries that do share one address are paired with that address's agents in order, and a reading beyond them is noted.
  • An agent whose address no listed GPU has becomes a GPU entry of its own when every listed GPU has an address; when some listed GPU has none, the agent could be that one and nothing is written. When no AMD GPU is listed at all, every agent becomes its own entry.
  • An agent that prints no BDFID is never paired by position, whatever the counts or targets: nothing says which card it is, so its reading goes into the note below, as AGENTS.md §6 asks for two independently ordered lists with no shared id. That no supported rocminfo omits BDFID is unconfirmed: the captured ROCm 7.14 listing prints it for every agent, and the release that first printed it was not found. So this case may never arise on a supported host; where it does, the reading is reported rather than guessed at.
  • A GPU no agent matches keeps its own target and no model name, instead of another card's.
  • Whenever readings cannot be attributed, entries are left untouched and one new note says so: N GPU reading(s) from rocminfo could not be matched to a listed GPU, so rocminfo's model names, gfx targets and integrated/discrete verdicts for them are not applied to any GPU: <target at bus address ... (reason)>; ....

Which source decides the gfx target:

  • A rocminfo agent fills a gfx target only where the entry has none. rocminfo inherits HSA_OVERRIDE_GFX_VERSION and then reports the override's ISA, so it can no longer replace a card's native target, and check_2 holds the card's own target against the arch list.
  • Regression this accepts: on a host with lspci but no readable KFD topology (some containers), a wrong target guessed from the name table is no longer corrected by rocminfo; rocminfo_never_replaces_a_name_table_guess_even_without_a_kernel_node pins it. Because of the parser defect above, main never actually made that correction on a real host either.
  • A KFD node at the same PCI address now replaces a target the lspci scan guessed from the marketing-name table (previously it only filled an empty one); the kernel's gfx_target_version is not affected by the override. Where the name table and the device-id map disagree (Krackan: gfx1150 vs gfx1152), the kernel's value wins.

Which source decides is_apu: the rules, by source, are one table in the doc of apu_verdict_from_target in linux_gpus.rs. Differences from main a user can see:

  • A rocminfo agent's Memory Properties: line now decides where it is printed (main used only the agent's target): APU means integrated, and a line without it means discrete unless the target is in an APU family. That such a line means "discrete" rests on one AMD documentation sample (an MI300X); no discrete GPU's output was captured for this repository.
  • A target outside the APU families is not taken as proof of a discrete GPU, so an agent with no Memory Properties: line and such a target leaves the verdict as it was (null when nothing else classified it) instead of setting false.
  • A KFD-only or sysfs-only entry with an APU-family target reads is_apu: true instead of null.
  • When a KFD node replaces a name-table guess, the verdict that came with the guess is re-derived from the kernel's target, false included.

On a real Strix Halo (gfx1151, whose lspci row is a nameless "Device"), rocm examine --json before this change reported the GPU as discrete (has_discrete_amd: true, has_apu: false); after it, gfx1151 with is_apu: true, has_apu: true, has_discrete_amd: false, the same with and without HSA_OVERRIDE_GFX_VERSION=11.0.0.

Tests

Scenarios in tests/e2e-cucumber/features/examine.feature, all @requires-os:linux on simulated hosts with no GPU, so they run on the default Linux E2E lane:

  • examine-55, @id:examine-each-gpu-keeps-its-own-reading: a GPU hidden from the runtime does not borrow the visible GPU's name, target or verdict. Its rocminfo stand-in now prints the ISA Info: block as the real tool does.
  • examine-56, @id:examine-overridden-runtime-target-keeps-native: a runtime reporting gfx1100 under HSA_OVERRIDE_GFX_VERSION=11.0.0 does not replace the card's gfx1103; the stand-in leaves a marker that proves it ran with the override.
  • examine-57, @id:examine-kernel-architecture-outranks-the-name-guess: the kernel's target replaces a name-table guess and the verdict follows it.
  • examine-58, @id:examine-kernel-architecture-fills-an-unnamed-gpu: the kernel's APU-family target fills an unnamed GPU and makes it integrated.
  • examine-59, @id:examine-unaddressed-runtime-readings-are-reported-not-guessed: a runtime that lists both GPUs of a two-GPU host without a BDFID (same order, same architectures) leaves both entries as they were, and the note names both readings; the scenario asserts the note and the unchanged entries together.

Unit tests in crates/rocm-core/src/examine/linux_gpus.rs cover what the scenarios cannot isolate: every matching case (a)-(g), partitions at their own addresses and the shared-address fallback, cross-domain addresses, two GPUs on one bus that differ only in device or function, NVIDIA entries beside AMD ones, and the Memory Properties verdict (a_matched_agents_verdict_is_its_memory_properties_reading_or_else_the_target_rule, with_no_amd_gpu_listed_each_agent_is_listed_with_its_own_verdict), which examine-56 cannot separate from the target rule because gfx1103 is an APU family either way. a_real_rocminfo_listing_yields_its_gpu_agent parses a captured real rocminfo listing from a Strix Halo host (crates/rocm-core/tests/fixtures/rocminfo_gfx1151_apu.txt). host_gpu::linux pins the shared address decode to literal addresses.

Docs and test support

  • docs/testing.md describes examine-55 to examine-59 and the commands to run them; docs/host-probing.md states which source decides the target on Linux, links to the is_apu table in the code, and states the two residuals below; docs/manual-testing.md adds a Linux check with ROCR_VISIBLE_DEVICES and HSA_OVERRIDE_GFX_VERSION on a multi-GPU host.
  • KERNEL_TOPOLOGY_GPU_NAME is now public so the e2e step asserts against the placeholder itself. The two planted lspci lines examine-57 and examine-58 use are rocm_core::examine::TEST_FIXTURE_LSPCI_LINE_* constants, compiled only under cfg(test) or rocm-core's test-support feature, which the e2e crate enables from its [dev-dependencies] only; a unit test pins how the name table reads them.

Verification

  • Locally: cargo fmt, cargo clippy --workspace --all-targets -- -D warnings, the rocm-core tests, the feature_naming and skill_reference tests, cargo xtask check-module-size and cargo xtask check-architecture-doc pass, and scenarios examine-48 to examine-59 pass together, so this change's scenarios run beside the other examine scenarios in the same feature file.
  • On a Strix Halo host: rocm examine --json with and without HSA_OVERRIDE_GFX_VERSION=11.0.0 gives the result above.
  • cargo xtask smoke could not run to completion on the test host: its sandbox step needs bwrap to create network namespaces, which the host refused; the steps before it passed.
  • Not verified locally: the full cargo test --workspace on the final change.
  • Not verified: that GC 9.4.3 partitions report distinct location_id/BDFID values; this comes from upstream kernel and ROCm sources, and the unit test plants it.
  • Not verified: multi-GPU hardware. The ROCR_VISIBLE_DEVICES path is covered only by simulated hosts and unit tests; docs/manual-testing.md has the manual check.
  • Assumed: that a Memory Properties: line without APU means a discrete GPU rests on one sample in AMD's rocminfo documentation (an MI300X); an APU-family target outranks it.
  • Assumed: that no supported rocminfo omits BDFID (see above).
  • Residuals (documented in docs/host-probing.md): with no KFD node to fill it first, a rocminfo agent can fill an empty target with an override's ISA; and on a host with lspci but no readable KFD topology, a wrong name-table guess is not corrected, because a correct runtime reading cannot be told apart from an overridden one.

Scope

Not changed here:

  • gpu_policy.rs treats the amd-smi gpu index as a HIP ordinal.
  • On Windows, fill_unidentified_gpus_from_hipinfo pairs by position as a last resort without a count guard.
  • The dash parse_metrics defaults a missing amd-smi gpu field to 0 (unwrap_or(0)).
  • A rocminfo reading with no BDFID is never attributed. Chip IDs are not compared; on a host whose cards are different parts they could identify such a reading, at the cost of a second, weaker identity.
  • gfx_is_apu_family does not list gfx902, gfx909 or gfx942; fix(examine): identify AMD GPUs by device id and one name table, and never downgrade an APU verdict (ROCMAI-534) #521 owns that list.
  • rocminfo could be run with HSA_OVERRIDE_GFX_VERSION removed from its environment, so that its target is the card's own: that would remove the first residual above and let it correct a name-table guess again. Recorded as a follow-up on ROCMAI-40.

Related open pull requests touch the same code:

Suggested squash commit message

The branch's individual commit messages describe intermediate states: an earlier commit still had a positional fallback that a later one removed. They cannot be reworded without a force-push over a reviewed PR, so please use this message when merging:

fix(examine): match rocminfo GPU readings by PCI address (ROCMAI-40)

`rocm examine` on Linux paired the GPU agents `rocminfo` reports with the
listed AMD GPUs by list position. The ROCm runtime can expose only some
of a machine's GPUs (for example under ROCR_VISIBLE_DEVICES), and nothing
guarantees the two lists share an order, so one card's model name, gfx
target and is_apu verdict could land on another card.

The rocminfo parser is fixed as well. Each agent ends with an `ISA Info:`
block whose entries carry `Name:` lines of their own, and the parser read
the last of them as the agent's name. That name is not a gfx target, so
every GPU agent of a real rocminfo listing was dropped (shown with a
captured ROCm 7.14 listing, now a test fixture). Before this change the
rocminfo pass therefore had no effect on a real host. From this change
on, its model names, gfx targets and integrated/discrete verdicts reach
`rocm examine` output, and everything that reads it, for the first time.
That is the widest effect of this change.

Behaviour changes on Linux:

- Each agent is matched to a listed GPU by its `BDFID` against the
  entry's PCI bus, device and function. One helper decodes both BDFID
  and the KFD topology's `location_id`, and refuses zero and values
  wider than 16 bits. A match that two entries in different PCI domains
  share is refused and noted.
- An agent that prints no BDFID is never paired with an entry, by
  position or otherwise. Where any AMD GPU is listed, its reading is
  reported in a note and no entry takes it; where none is, every agent
  is listed as a GPU of its own. Whether any supported rocminfo omits
  BDFID is unconfirmed; the captured ROCm 7.14 listing prints it for
  every agent.
- An agent that matches no entry is listed as a GPU of its own only when
  every AMD entry has an address; otherwise it is noted. Readings in
  excess of the entries at one address are noted rather than applied.
- A rocminfo reading fills a gfx target only where the entry has none,
  because the runtime reports the ISA that HSA_OVERRIDE_GFX_VERSION
  names, and `rocm diagnose`'s override check needs the card's own
  target. As a result, on a host with lspci but no readable KFD topology,
  a wrong target guessed from the marketing-name table is never
  corrected. The previous code would have let rocminfo overwrite it, but
  because of the parser bug it never did so on a real host.
- A KFD node at the same address replaces a target the PCI scan guessed
  from the name table, and the verdict that came with the guess; before,
  it only filled an empty target.
- is_apu: a rocminfo agent's `Memory Properties:` line decides it when
  printed (`APU` means integrated, a line without it discrete), except
  that a target in an APU family stays integrated, because the
  "discrete" reading rests on one documentation sample. Without the
  line, an APU-family target makes a GPU integrated and any other target
  leaves the verdict as it was, so a kernel-only or sysfs-fallback entry
  with an APU-family target is now integrated and one with another
  target stays null rather than being called discrete.

Covered by unit tests over the match's cases, the address decode, the
verdict rules and the captured listing, and by scenarios examine-55 (a
GPU hidden from the runtime keeps its own identity), examine-56 (an
overridden runtime target does not replace the native one), examine-57
(the kernel's target replaces a name-table guess), examine-58 (the
kernel's target fills an unnamed GPU and makes it integrated) and
examine-59 (readings with no address are reported, not applied).

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No ROCMAI-40 rows.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation). (Not applicable: no new subcommand.)
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (No source files added, renamed or removed.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen. (The unattributed-readings note is asserted together with the entries keeping their own targets and names.)
  • Comments and doc lines this PR adds or changes that state how the code behaves were checked against that code, and each that is the reason for a decision names the file and the function, constant or test that makes it true (AGENTS.md §3).
  • Any existing scenario this PR moves off the blocking mock e2e job (for example by a tag) is justified as a deletion (AGENTS.md §8). (None moved.)
  • Commits answering review feedback were diffed whole against the reviewed head and widen no documented user-facing option's meaning or authority (AGENTS.md §11). (c9d495246 against the reviewed head 7882970e: doc comments and docs only, no behaviour change.)
  • Every commit message on the branch describes what its own commit does in words still true at the branch head (AGENTS.md §5). Not met: 6c65b265's message describes the end state, though that commit still had a positional fallback, and 8c919d9c's credits it with full-BDF matching that 6c65b265 already did. Use the squash message below.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@rominf
rominf force-pushed the ROCMAI-40-fix-rocminfo-gpu-match-by-pci-address branch 3 times, most recently from 10aeef8 to 2272f68 Compare November 17, 2026 10:22
…t position (ROCMAI-40)

`rocm examine` paired the agents `rocminfo` reports with the listed AMD
GPUs by position. The ROCm runtime can show only some of a machine's
GPUs (for example under ROCR_VISIBLE_DEVICES), and the listed GPUs can
come in a different order, so one card's marketing name, gfx target and
is_apu verdict landed on another card.

Each agent is now matched to a listed GPU by its `BDFID` against the
entry's PCI address, decoded by one helper shared with the KFD
`location_id` path; the helper refuses zero and values wider than 16
bits. A reading that cannot be attributed by address is not applied to
any entry: the entries are left untouched and a note says the readings
could not be attributed, instead of guessing on a multi-GPU host.

The parser now skips each agent's `ISA Info:` block. Its ISA entries
carry `Name:` lines of their own, which the parser read as the agent's
name, so every GPU agent a real `rocminfo` prints was dropped.

A `rocminfo` reading fills a gfx target only where the entry has none.
The runtime's reading carries any HSA_OVERRIDE_GFX_VERSION, so it must
not replace the card's native target, which `rocm diagnose`'s override
check holds against the framework's arch list. A KFD node at the same
address, which the override cannot touch, replaces a target the PCI
scan guessed from the name table.

`is_apu` comes from rocminfo's `Memory Properties:` line when the agent
prints one (`APU` means integrated, a line without it discrete), with an
APU-family target as a backstop that keeps a GPU integrated, and
otherwise from the APU-family rule on the target the entry ends with. A
target outside the APU families is not taken as proof of a discrete
card, so an entry nothing classified stays null.

Covered by unit tests over the match's state matrix, the address
decode, the verdict rules and a captured real `rocminfo` listing, and
by scenarios examine-55 (a GPU hidden from the runtime keeps its own
identity), examine-56 (an overridden runtime target does not replace
the native one), examine-57 (the kernel's target replaces a name-table
guess) and examine-58 (the kernel's target fills an unnamed GPU and
makes it integrated).

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rocminfo's `Memory Properties:` line says `APU` for an integrated GPU, and the
reading that an empty line means a discrete GPU rests on a single sample in
AMD's rocminfo documentation (an MI300X). A flag without `APU` therefore no
longer makes a GPU discrete when its target is in an APU family: the family
list is the backstop, and the flag still decides for every other target and
whenever it says `APU`. Where the entry already held its target, that target is
the card's native one, so the override safety of keeping held targets is
unchanged.

New test rows pin what the fixtures left unexercised:
- two GPUs on one bus that differ only in device or function each receive
  their own agent, so a match on the bus alone fails;
- an agent with no `Memory Properties:` line takes the target rule, both when
  matched and when listed as a GPU of its own;
- a flag without `APU` on a gfx1151 still reads as integrated;
- an NVIDIA entry beside an AMD one is never given a rocminfo reading;
- an agent that prints no `Device Type:` after a GPU agent is not a GPU.

docs/manual-testing.md gains a Linux check for per-GPU identity with
`ROCR_VISIBLE_DEVICES` and `HSA_OVERRIDE_GFX_VERSION`.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A rocminfo agent that prints no BDFID says nothing about which card it is, so
it is never paired with a listed GPU by position, whatever the counts or the
targets. Its reading goes into the note about readings that could not be
matched, and every entry keeps its own target and name, which is the
project's rule for two independently ordered lists with no shared id. The
note now says that rocminfo's integrated/discrete verdict is withheld as well
as its names and targets, and a surplus reading at an address whose entries
are taken gets a reason of its own. Scenario examine-59 covers it end to end:
a runtime that lists both GPUs of a two-GPU host without addresses leaves both
entries as they were and reports both readings.

Every case in the matching doc is lettered and cited by a test. The verdict
for agents listed as GPUs of their own, when no AMD GPU is listed, has its own
test, which also asserts that such GPUs are AMD GPUs and set has_amd_gpu. The
partition test plants partitions at addresses of their own (11:00.0 to
11:00.2), as upstream kernel and ROCm sources describe for GC 9.4.3, which
has not been verified on hardware here; pairing entries that share one
address in order stays as a fallback. The rocminfo test fixture takes its
Memory Properties flag from a written-out table rather than from the
APU-family function under test.

The KFD address doc says a value wider than 16 bits is refused, the Linux
manual check explains how to read `gfx_target_version` and requires an
override that differs from the card's own target, and docs/testing.md
describes examine-55 to examine-59 with their commands after the last of
them.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-40-fix-rocminfo-gpu-match-by-pci-address branch from 2272f68 to 7882970 Compare November 17, 2026 12:58
@rominf
rominf marked this pull request as ready for review November 17, 2026 13:18
@rominf
rominf requested a review from a team as a code owner November 17, 2026 13:18
@rominf
rominf requested a review from tomastola November 17, 2026 13:18

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this PR against the ROCMAI-40 plan and the repo's contribution rules. I did not find a defect I can prove from the diff, and CI is green, so these are changes I'd like before merge rather than bugs.

Scope. The ticket asks for matching rocminfo readings by PCI address. This PR also:

  • stops pairing by position altogether (the ticket kept an equal-count fallback);
  • stops rocminfo from replacing a gfx target that is already held;
  • lets the KFD node replace an lspci name-table guess;
  • reworks the is_apu rules;
  • fixes the ISA Info parser.

That last fix matters most: before it, every GPU agent of a real listing was dropped, so the whole rocminfo pass (names, targets, is_apu) takes effect on real hosts for the first time. That is a much larger blast radius than the title suggests. Please either split this (ISA parse fix + BDFID match, then the target-precedence and is_apu changes) or record the decision and the blast radius on ROCMAI-40.

Behavior tradeoffs to confirm.

  • An agent with no BDFID is never paired, even for one agent and one AMD entry. The ticket's open assumption that no supported rocminfo omits BDFID is still unconfirmed; please say so in the PR body.
  • Where the KFD topology is unreadable but rocminfo works, a wrong name-table target is now never corrected (main overwrote it). It is documented and pinned by a test, but it is a regression for that shape.
  • "Empty Memory Properties means discrete" rests on a single documentation sample.

Docs. The is_apu rule set is restated in the code doc, docs/host-probing.md, docs/testing.md and the PR body, and will drift. Link to the code table instead. docs/testing.md also narrates PR history.

Commit messages. Three of the commits are fix-ups of the first. Please check that each message still reads true at the branch head, since they land in public history.

Please also confirm the test-support feature enablement for e2e sits under [dev-dependencies].

/// target against the framework's arch list. The APU verdict is decided by
/// [`apu_verdict_with_runtime_flag`] on the target the entry ends with, never
/// on a spoofed agent target laid over a held one.
fn apply_agent_to_gpu(gpu: &mut Gpu, agent: RocminfoGpuAgent) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now never replaces a held target, which reverses main (rocminfo was authoritative). On a host with lspci but no readable KFD topology, a wrong name-table target is never corrected any more. It is documented and tested, but it is a behavior change beyond the matching fix. Consider splitting it out or recording the decision on ROCMAI-40.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is deliberate. The runtime reports the ISA that HSA_OVERRIDE_GFX_VERSION names, and rocm diagnose's override check needs the card's own target, so a rocminfo reading only fills an empty target.

The residual is now stated plainly in the apply_agent_to_gpu doc and in docs/host-probing.md (c9d4952): on a host with lspci but no readable KFD topology, a wrong name-table target is never corrected, pinned by rocminfo_never_replaces_a_name_table_guess_even_without_a_kernel_node. On main this correction never happened on a real host anyway, because the parser dropped every GPU agent.

The decision and blast radius are recorded on ROCMAI-40, with a follow-up: run rocminfo with the override removed from its environment, so it can correct a guess again. The PR body has a new "Scope and blast radius" section and states the regression.

/// - (c) When some AMD entry has no address, such an agent could be that one,
/// so nothing is written and a note says so.
/// - (d) An agent that states no address is never paired with an entry: not
/// by position, whatever the counts or targets, and whether or not other

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agents with no BDFID are never paired, even for a single agent and a single AMD entry, where the ticket plan kept an equal-count positional fallback. The stricter rule fits the no-guessing policy, but please confirm that no supported rocminfo omits BDFID (the ticket's open assumption) and state that in the PR body.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed deliberate: per AGENTS.md §6, two independently ordered lists with no shared id degrade to unknown rather than a guess, so a reading with no BDFID goes into the note even for one agent and one AMD entry.

Whether any supported rocminfo omits BDFID is unconfirmed: the captured ROCm 7.14 listing (REAL_ROCMINFO_GFX1151) prints it for every agent, and I could not find the release that first printed it. That is now stated in the case (d) doc of apply_rocminfo_gpu_agents (c9d4952) and in the PR body, under the matching rules and as an assumption under Verification.

Comment thread docs/testing.md Outdated
RUSTC_WRAPPER= cargo xtask e2e -- -n 'examine-(49|50|51|52|53) '
```

`examine-55` plants a native Linux machine whose kernel topology has two GPUs, at `0000:03:00.0` (gfx1100) and `0000:04:00.0` (gfx1101), with an `lspci` stand-in that lists an empty bus and a `rocminfo` stand-in that lists only the second, as "AMD Radeon RX 7800 XT" with its `BDFID`: the ROCm runtime with the first GPU hidden from it. The stand-in prints each agent the way the real tool does, ending with an `ISA Info:` block whose entries have `Name:` lines of their own. Earlier stand-ins left that block out, which is why no test noticed that the parser took those lines for the agent's name and so dropped every GPU agent of a real `rocminfo` listing. `rocm examine --json` must report the second GPU with that model name, its own gfx1101 and `is_apu: false` (the stand-in prints the empty `Memory Properties:` line a discrete card gets), and the first with gfx1100, no model name (the kernel-topology placeholder, explained by one "marketing name is unknown" note) and `is_apu: null`: nothing that can see it says which package it is (gfx1100 is outside the APU families, which is not taken as proof of a discrete card, no PCI scan named it, and the runtime that could say does not list it). Before the fix the runtime's GPUs were paired with the listed ones by position, so the first GPU took the second's name, target and verdict. It runs on the Linux lanes and needs no GPU. The other rows of the match (agents with no `BDFID`, which are never paired by position whatever the counts or targets and are always noted; an address no listed GPU has; an address shared across PCI domains; more readings at an address than listed GPUs; partitions at addresses of their own; agents listed with no AMD GPU to match) are unit tests in [linux_gpus.rs](../crates/rocm-core/src/examine/linux_gpus.rs), next to `apply_rocminfo_gpu_agents`, and `a_real_rocminfo_listing_yields_its_gpu_agent` parses a captured real listing, [rocminfo_gfx1151_apu.txt](../crates/rocm-core/tests/fixtures/rocminfo_gfx1151_apu.txt). The commands for examine-55 to examine-59 follow examine-59.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph narrates PR history ("Earlier stand-ins left that block out...", "Before the fix...") in a long-lived testing doc, and it restates the is_apu rules that already live in apu_verdict_from_target and docs/host-probing.md. Prefer describing what the scenario asserts and linking to the code doc for the rules.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c9d4952. The examine-55 to examine-59 paragraphs now describe what each scenario asserts; the history narration ("Earlier stand-ins...", "Before the fix...", "Before that rule...") is gone. For the is_apu rules, docs/testing.md and docs/host-probing.md now link to the table in the doc of apu_verdict_from_target instead of restating it, and the PR body does the same. The evidence for reading an empty Memory Properties: line as discrete is stated once, on parse_rocminfo_memory_properties. That commit also corrects the examine-56 paragraph's pointer to its command (after examine-59, not examine-58).

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran an independent pass over this PR (full diff against main at 9e50eac, head 7882970e), focused on the PCI-address matching/attribution logic, the target/is_apu precedence rules, the ISA Info parser fix, the e2e scenario and test-support feature wiring, and the docs/fixtures. Posting this alongside @juhovainio's existing review rather than instead of it — most of what I'd flag at the approach level, they already raised.

Correctness: no blocking defects found. I traced the BDFID/location_id bit decode, the cross-domain and shared-address (partition) fallback, the full apply_rocminfo_gpu_agents matching table (cases a–g), and the is_apu precedence rules against the code, the unit tests, and the captured real rocminfo fixture, and did not find a logic bug, panic path, or a test that would pass against a broken implementation. That part of the change looks solid.

New finding — two commit messages misdescribe their own commits, and will land in public history as-is.

  • 6c65b265 ("attribute rocminfo GPU readings by PCI address, not list position") says "A reading that cannot be attributed by address is not applied to any entry... instead of guessing on a multi-GPU host". That commit's own diff still contains a positional-pairing fallback for the no-BDFID case (equal counts + agreeing targets), documented in its own doc comment as case (d) ("are they paired by position: the last resort...") and covered by a test named agents_without_an_address_pair_by_position_only_when_the_counts_agree. That fallback isn't removed until 7882970e. So 6c65b265's message describes the PR's end state, not what that commit itself does.
  • 8c919d9c ("full-BDF GPU match, APU target beats flag") claims this commit introduces full bus+device+function matching. It doesn't: 6c65b265 already decodes and compares the full bus:device.function string (pci_bus_device_function / bdf.eq_ignore_ascii_case); the new test in 8c919d9c (gpus_on_one_bus_match_by_device_and_function_too) exercises already-existing behavior rather than a change this commit makes. The only production-code change in 8c919d9c is the is_apu precedence rework.

This is exactly what the PR's own checklist asserts was done ("Every commit message on the branch describes what its own commit does in words still true at the branch head (AGENTS.md §5)"), so it's worth fixing before these messages land in the squash/rebase history — reword or fold 6c65b265/8c919d9c so each accurately scopes what it changed.

Corroborating two of @juhovainio's open points, verified independently rather than taken on faith:

  • The is_apu rule table is genuinely restated (not just linked) in both the apu_verdict_from_target doc comment and the new docs/host-probing.md prose — host-probing.md does point back to the code afterward, but only after re-deriving the same rules, so a future change to the table isn't forced to update both.
  • docs/testing.md's new examine-55..59 section does narrate implementation history ("Earlier stand-ins left that block out...", "Before the fix...") rather than only describing current, durable behavior.
  • One of @juhovainio's open questions I can now answer from the code: the test-support feature enablement for rocm-core in tests/e2e-cucumber/Cargo.toml is correctly under [dev-dependencies] (not [dependencies]), and the gated constants are additionally #[cfg(any(test, feature = "test-support"))] at their definition site, so resolver-v2 keeps them out of the production build unit.

On scope. I agree with the standing request: the ISA Info parser fix alone changes the blast radius from "matching precedence on an already-working path" to "the whole rocminfo pass now fires for the first time on real hosts," and that's bundled with four other independent behavior changes (no positional fallback at all, KFD-over-lspci-name-table precedence, rocminfo-never-overwrites-a-held-target, and the is_apu rework) under a ticket titled around PCI-address matching. Nothing I found changes that calculus, and it's still unaddressed at this head (no split, no recorded decision on ROCMAI-40).

Given the unresolved scope question plus the commit-message inaccuracies above (both fixable, neither requiring a rewrite of the logic itself), I'm requesting changes rather than approving, alongside the existing review.

🤖 Automated by an AI agent

The is_apu rules were written out in full in the doc comment of
apu_verdict_from_target, again in docs/host-probing.md and again in
docs/testing.md, so a change to them had three docs to drift in. Both
docs now describe what a user sees in gpus[].is_apu and link to the
table in the code for the rules. The reason the kernel path gets only
the target rule, which only host-probing.md gave, now sits in the doc
above that table.

The evidence that an empty Memory Properties line means a discrete GPU
is one documentation sample. That is now said once, on
parse_rocminfo_memory_properties, and the table row and
apu_verdict_with_runtime_flag point there instead of repeating it.

The examine-55 to examine-59 paragraphs in docs/testing.md no longer
narrate how the code used to behave; they say what each scenario
asserts and how to run it. The paragraph on examine-56 also pointed at
the wrong place for its command, which follows examine-59.

The doc of the no-BDFID case in apply_rocminfo_gpu_agents now says
that whether any supported rocminfo omits BDFID is unconfirmed: the
ROCm 7.14 listing captured for the tests prints it for every agent,
and the release that first printed it was not found. The doc of
apply_agent_to_gpu and docs/host-probing.md state the residual of
never replacing a held target plainly: on a host with lspci but no
readable KFD topology, a wrong name-table target is never corrected,
and the test that pins it is named. No behaviour changes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…o-gpu-match-by-pci-address

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

@juhovainio, answering both review bodies (13:29Z and 14:32Z on 7882970e):

Scope. I recorded the decision and blast radius on ROCMAI-40 rather than splitting. The reason is that the ISA Info: parser fix alone would turn the old position-based overwrite on for real hosts, since on main that pass never ran on one: every GPU agent was dropped. The other changes decide what the newly live pass may write. The PR body now has a "Scope and blast radius" section saying this.

Behaviour tradeoffs. Answered in the inline threads: the no-BDFID assumption is stated as unconfirmed in the code doc and the body, and the held-target regression is stated plainly in both, with the test that pins it. The single-sample basis for "empty Memory Properties means discrete" is stated once, on parse_rocminfo_memory_properties, and in the body.

Docs. c9d4952: the is_apu rules are linked to the apu_verdict_from_target table from docs/testing.md, docs/host-probing.md and the PR body instead of restated, and the history narration in docs/testing.md is gone.

Commit messages. Both findings are correct. 6c65b265's message describes the end state rather than its own commit, which still had the positional fallback. 8c919d9c's message credits it with full-BDF matching that 6c65b265 already did. Rewording them needs a force-push over a reviewed PR, which this repo's workflow (AGENTS.md §1) reserves for explicit approval, so the PR body now carries a corrected "Suggested squash commit message" for the merger, and the checklist item on commit messages is unchecked with the reason. If you prefer the branch history itself rewritten, say so and I will ask for the force-push.

test-support. Confirmed. In tests/e2e-cucumber/Cargo.toml the entry sits in [dev-dependencies]:

rocm-core = { path = "../../crates/rocm-core", features = ["test-support"] }

and both constants in crates/rocm-core/src/examine/linux_gpus.rs are gated with #[cfg(any(test, feature = "test-support"))].

Since 7882970e. Only c9d4952 (doc comments and docs, no behaviour change) and a merge of main (b874d91, no conflicts). Nothing else changed.

@rominf
rominf requested a review from juhovainio November 17, 2026 16:02

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh pass at the current head (b874d918), re-checking this round's open items against the two CHANGES_REQUESTED reviews at 7882970e (diff:38b7c3f6418c). Full diff against main, independent fan-out over the matching/precedence logic (linux_gpus.rs, examine.rs, host_gpu.rs/host_gpu/linux.rs), the e2e harness (examine.feature, examine_steps.rs, the test-support wiring), the docs (host-probing.md, testing.md, manual-testing.md), and git history on these files.

Since the last round, only c9d495246 (doc comments and docs only — verified no production-code behavior change) and a clean merge of main (b874d918, verified no conflicting hunks) landed.

This round's open items:

  • Scope/blast-radius. Recorded on ROCMAI-40 (verified directly on the ticket) and in the PR body's new "Scope and blast radius" section, rather than split. Reasonable given the dependency between the ISA-parser fix and everything that decides what the newly-live rocminfo pass may write.
  • Commit messages (6c65b265, 8c919d9c). Not reworded — the PR explains this needs a force-push the repo's own workflow reserves for explicit approval, leaves the PR checklist item honestly unchecked, and supplies a corrected squash message for the merger. That's a defensible way to handle it without an unrequested force-push.
  • Docs restatement / history narration. Largely fixed in c9d495246: the is_apu table is now linked rather than restated in the code doc/host-probing.md/testing.md, and the "before this fix.../earlier stand-ins..." narration is gone from testing.md. Two smaller instances of the same restate-instead-of-link pattern remain (below).

Correctness. No logic defect found. Matching by BDFID → PCI address, the cross-domain/shared-address/partition fallback, and the is_apu precedence rules all trace correctly against the code, the unit tests, and the captured real rocminfo fixture. Every new/changed test in examine_steps.rs and linux_gpus.rs's test module was checked against "would it still pass if the change were reverted / if one branch were broken alone" — all pass that bar. A history pass found no re-introduction of the pre-existing positional-pairing bug this PR is fixing more thoroughly.

Non-blocking findings:

  • docs/host-probing.md's "lspci but no readable KFD topology" residual (new sentence) restates the apply_agent_to_gpu doc comment on the same residual almost verbatim instead of linking to it, which is the convention the sentence right before it in the same code comment already follows (pointing at docs/host-probing.md instead of repeating itself). Worth converging on one of them per the no-repetition convention this change otherwise cleaned up elsewhere this round.
  • docs/testing.md's new examine-57 entry restates the Krackan Point name-table/device-id-map disagreement that's already fully stated in docs/host-probing.md, and it isn't actually part of what examine-57 plants (examine-57 uses a "Phoenix1"/gfx1103-vs-gfx1100 fixture). A one-line pointer would do.
  • Two doc comments in linux_gpus.rs make opposite-sounding claims about the same unconfirmed hardware scenario (a partitioned GC 9.4.3/Instinct card): the new comment on apply_rocminfo_gpu_agents (case a) says such partitions get "addresses of their own... on both sides," while the pre-existing comment in apply_kernel_gpu_membership says "several KFD nodes legitimately share one physical card's address." Both are explicitly caveated as unverified on hardware, and the code already handles both cases (a distinct-address path and an explicit shared-address fallback), so there's no behavioral bug here — but worth reconciling which is the expected case so a future reader doesn't have to guess.

None of the above block correctness or merge; they're follow-up polish on an already-converged set of design decisions.

CI: all substantive jobs (clippy, build-and-test incl. the ubuntu-26.04 variant, Windows build/test/release, Test (affected crates), E2E tests incl. ubuntu-26.04, Build Linux PR binaries, CodeQL, the security scanners, lychee, license headers, commit-signatures) are green at this head; only summary/aggregator jobs remain queued.


🤖 Automated by an AI agent

@rominf
rominf added this pull request to the merge queue Nov 17, 2026
@rominf
rominf removed this pull request from the merge queue due to a manual request Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants