Skip to content

fix(examine): report amdkfd_loaded and amdgpu_loaded when built into the kernel - #694

Merged
tomastola merged 28 commits into
mainfrom
fix-examine-kfd-builtin
Nov 10, 2026
Merged

tomastola merged 28 commits into
mainfrom
fix-examine-kfd-builtin

Conversation

@tomastola

@tomastola tomastola commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #448

Problem

rocm examine --json reported "amdkfd_loaded": false on machines where kfd works. The field was true only when a module named amdkfd appeared in lsmod or /proc/modules. kfd is not a module of its own: HSA_AMD is a bool Kconfig option and amdkfd is linked into amdgpu, so that name never shows up, whether amdgpu is built in or loaded as a module. A healthy system was reported as "kfd missing".

amdgpu_loaded had the same blind spot for a different reason: an amdgpu that is built into the kernel is absent from lsmod, so the field was false. rocm diagnose reads that as evidence of a DKMS failure (+30), "amdgpu module is not loaded" (+35) and a Secure Boot problem (+10), none of which is true on such a machine.

Change

This PR fixes both flags in probe_modules.

amdkfd_loaded is now true when any of these holds:

  • amdkfd is in the module list (unchanged behaviour);
  • /sys/class/kfd/kfd exists;
  • /dev/kfd exists.

amdgpu_loaded is now true when any of these holds:

  • amdgpu is in the module list (unchanged behaviour);
  • /sys/bus/pci/drivers/amdgpu exists. The kernel creates it once amdgpu_init has registered the driver, for a loaded module and a built-in driver alike. /sys/module/amdgpu is not used: a built-in driver gets it at an early initcall even when amdgpu_init then bails out (nomodeset, amdgpu.modeset=0);
  • kfd is present (/sys/class/kfd/kfd or /dev/kfd). kfd is linked into amdgpu, so kfd cannot be there without it, and a container that passes /dev/kfd through without /sys/bus/pci must not report amdkfd up and amdgpu down.

Both fields report that the driver is present, not that a GPU is bound to it. amdgpu_init creates kfd's device before it registers the PCI driver, and a built-in driver registers at boot whether or not a matching GPU is there, just as a module that has been modprobed is listed in lsmod on any machine. Whether there is an AMD GPU is has_amd_gpu. The diagnose checks read only amdgpu_loaded == Some(false), as a module that failed to load (DKMS, a blacklist, Secure Boot), and none of those can stop a driver that is present.

The module-list check and the host probes share one function, module_or_host_state, and the host probe only runs when the module list has not already answered. driver_states pairs the two fields' checks, so which host signal answers which field is testable apart from the real module list.

All paths are read through host_path, so the probes can be tested against a planted root; they are listed with the other simulated-host-root reads in hardware_root.rs and docs/release-trust.md, and the CI comment on planted-root coverage names their tests. If the module list cannot be read at all but the host shows the driver, the field is true rather than unknown; with no module list and nothing on the host it stays unknown. Both fields keep their names and types, so the JSON schema is unchanged. Each field has a doc comment saying what it means now.

Naming. The name amdkfd_loaded is kept for schema compatibility. The old module check still covers kernels from before kfd was merged into amdgpu. A separate field such as kfd_available was considered and rejected.

Tests

  • module_or_host_state: module listed; built in with an empty or unrelated module list; neither present; module list unreadable with and without the driver on the host.
  • kfd_present_on_host and amdgpu_present_on_host against a planted host root (e2e-test-hooks): sysfs, /dev/kfd, neither; for amdgpu, /sys/module/amdgpu alone does not count.
  • amdgpu_implied_by_host with only /sys/class/kfd/kfd, then only /dev/kfd: true. Dropping the kfd disjunct fails it.
  • driver_states against a planted host root, with an empty module list and with none, planting one signal at a time: kfd answers both fields, the amdgpu PCI driver answers only amdgpu_loaded, and with nothing planted the fields are false (list read) or unknown (no list). Swapping the host checks behind the two fields fails it.
  • probe_modules_from against an empty host root, handed lists naming both drivers, only amdgpu and only amdkfd: each field follows its own name. A misspelt name in driver_states or a broken module-list parse fails it.
  • probe_modules_from, handed a module list that names neither driver, against a planted host root with /dev/kfd (both fields true) and with /sys/bus/pci/drivers/amdgpu (amdgpu_loaded true, amdkfd_loaded false): dropping the host check or swapping the two fields at the call site fails them.
  • e2e scenarios examine-reports-kfd-built-into-the-kernel (examine-39) and examine-reports-kfd-absent-without-kfd-interfaces (examine-40): a planted host root and a lsmod that lists amdgpu and no amdkfd; examine --json must report amdkfd_loaded true with /dev/kfd and false without. examine-40 is a negative control (for examine-39): it passes with the fix reverted too, and fails if the kfd check reports true regardless.
  • e2e scenario examine-reports-amdgpu-built-into-the-kernel (examine-41): a planted host root with /sys/bus/pci/drivers/amdgpu, a lsmod that lists no amdgpu, a ROCm install and Secure Boot enabled. examine --json must report amdgpu_loaded true, and diagnose --json must cite none of the three evidence lines above.
  • e2e scenario examine-reports-drivers-from-kfd-without-a-module-list (examine-42): a planted host root with /dev/kfd and no /proc/modules, and a lsmod that fails. examine --json must report both amdkfd_loaded and amdgpu_loaded true.

All of them run on any Linux runner and need no GPU.

cargo test -p rocm-core --lib passes, the new tests pass with --features e2e-test-hooks, and clippy with that feature is clean. Mutating the kfd probe (hard-coding the host check off, dropping the unreadable-list fallback, dropping the host check from the decision) or swapping the host checks behind the two fields fails the unit tests. examine-31 to examine-34 pass through cargo xtask e2e. Not run on real hardware; the behaviour was checked against the reporter's examine output and the upstream kernel Kconfig/Makefile.

@tomastola
tomastola force-pushed the fix-examine-kfd-builtin branch from 0b97f7b to 88a535e Compare October 15, 2026 16:53
@tomastola tomastola changed the title fix(examine): report amdkfd_loaded when kfd is built into the kernel fix(examine): report amdkfd_loaded and amdgpu_loaded when built into the kernel Oct 16, 2026
@tomastola
tomastola marked this pull request as ready for review October 16, 2026 10:42
@tomastola
tomastola requested a review from a team as a code owner October 16, 2026 10:42

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was produced using Claude Code's review-code method (independent fan-out over the core probe logic, the e2e feature/steps, and AGENTS.md/impact compliance), at the request of @jussielo-amd. No GitHub approval is filed by this account for automated reviews — the merge decision stays with a human reviewer.

Blocking

  • CI: "E2E tests (Strix Halo, WSL2, current)" is failing on the current head (7f9c703). The suite's own reconciliation step reports: FAIL: 'comfyui-source-download-shows-live-progress' was expected to pass on this host but FAILED — a regression. That scenario lives in comfyui.feature and isn't touched by this PR's diff (examine.rs, examine.feature, examine_steps.rs). Issue #580 ("comfyui-04 times out when managed Python bootstraps inside its 30s spinner budget") describes a similar timing flake in the same area on this job class, so this looks like a pre-existing flake rather than something this PR broke. Requesting changes mainly to get a rerun/confirmation before merge rather than because the diff itself looks wrong here — happy to withdraw once the WSL2 job is green (or confirmed as an unrelated, already-tracked flake).

Non-blocking

  • crates/rocm-core/src/hardware_root.rs's module doc comment says it enumerates "every one of these reads, and only these" that go through the mockable host_path, but doesn't list the new /sys/class/kfd/kfd read added by kfd_present_on_host() in examine.rs. The read itself is correctly routed through host_path; this is just a stale enumeration. Suggest adding it alongside the existing topology-path mention.
  • amdgpu_state/amdkfd_state in examine.rs are structurally identical modulo the literal module name and the host-presence bool — could collapse into one generic helper to remove the duplication.
  • The new e2e step definitions in tests/e2e-cucumber/tests/e2e/examine_steps.rs duplicate the existing fake-executable/PATH-prepend idiom that plant_gpu_fixtures already implements (now three near-identical copies in this file). A small shared helper would remove the duplication.
  • Pre-existing, pattern-relevant: the new /sys/class/kfd/kfd, /dev/kfd, and /sys/module/amdgpu checks use Path::exists(), which doesn't distinguish "absent" from "I/O error / permission denied" — same as every other host_path(...).exists() check already in this file. Not introduced by this change, and unlikely to matter in practice since /sys//dev entries are normally world-traversable, but noting it since the new code extends the same pattern.
  • Minor: the pre-existing e.kfd field computation and the new kfd_present_on_host() each independently stat /dev/kfd through host_path — harmless but redundant.

Positive signals

  • The full tri-state branch set for amdgpu_state/amdkfd_state is covered by unit tests, and a mutation check (flipping || to &&) correctly broke 2 of 3 tests — a real regression guard, not a tautological one.
  • The e2e scenario asserting diagnose --json omits the DKMS/"module not loaded"/Secure-Boot evidence lines traces cleanly to diagnose.rs's three gates, all keyed on amdgpu_loaded == Some(false) — reverting the fix would reintroduce exactly those lines.
  • Doc comments on the new fields/helpers accurately reflect real Linux kernel Kconfig/module_param behavior for built-in vs. loadable drivers.

Full diff reviewed: 03cdbcc (base) → 7f9c703 (head), all three changed files read completely.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the fix for #448. The probe logic for amdgpu_loaded/amdkfd_loaded correctly folds in a host-presence check (sysfs class device / /dev/kfd for kfd, /sys/module/amdgpu for amdgpu) alongside the existing lsmod check, and preserves the three-state semantics: a readable module list ORs module-presence with host-presence, while an unreadable list still reports true if the host shows the driver active, and otherwise stays unknown rather than collapsing to a false negative. Cross-checked against diagnose.rs's fault-evidence checks, which gate strictly on == Some(false), so this directly fixes the false DKMS/Secure-Boot/module-missing evidence the linked issue reported, with no risk of masking a genuine negative.

Extending the same fix to amdgpu_loaded (not explicitly named in #448) is the right call — it's the identical root cause, and examine-31 adds dedicated e2e coverage that diagnose no longer cites those evidence lines for a built-in amdgpu. Keeping the field name amdkfd_loaded rather than splitting it is also reasonable: downstream consumers only care whether the driver is active, not whether it was loaded as a distinct module.

One non-blocking nit inline. CI is green aside from two unrelated, pre-existing flaky lanes (a comfyui download-timing scenario and the known Strix Halo Windows PATH-apostrophe/download-infra flake) — neither touches this diff.

Comment thread crates/rocm-core/src/examine.rs Resolved

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated code review (medium effort, recall-biased). 7 findings posted inline, ranked roughly by severity: two correctness issues in the new built-in-driver detection logic (a /sys/module/amdgpu-exists check that doesn't confirm the driver actually bound, and amdgpu_state/amdkfd_state not cross-checking each other despite kfd being linked into amdgpu), one cross-file consistency gap (the diagnose/fix remedy flows still assume lsmod-module semantics), and four cleanup items (reuse of an existing host_gpu.rs check, a doc-comment invariant in hardware_root.rs that's now stale, and two minor simplification/efficiency notes).

Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/examine.rs Resolved
@nowycondro

Copy link
Copy Markdown
Collaborator

🤖 Automated review (Claude Code) · 7f9c703

This supersedes/clarifies the review posted moments ago (pullrequestreview-5655465252) to match the required report format; the 7 inline findings on that review stand as the substantive output of this pass.

Scope read: full diff base 03cdbcc → head 7f9c7032 in crates/rocm-core/src/examine.rs (the amdgpu_state/amdkfd_state/kfd_present_on_host/amdgpu_present_on_host logic), plus the new e2e scenario/steps. No Rust toolchain available in this sandbox — I did not build or run the test suite; findings are from static reading of the diff only, cross-referenced against diagnose.rs, fix.rs, host_gpu.rs, and hardware_root.rs in the same tree.

Blocking (confidence: medium)

  • examine.rs (amdgpu_present_on_host, ~line 1788): /sys/module/amdgpu existing only proves the kernel registered the module's parameters at an early initcall, not that the driver actually bound/probed successfully (e.g. under nomodeset or a probe failure). A loadable module in that broken state is correctly reported false; the new built-in-kernel path would now report true for the same broken state. Concrete failure: built-in amdgpu fails to bind a GPU, amdgpu_loaded reports true, and diagnose.rs's == Some(false) gates (DKMS/module-missing/Secure-Boot evidence) are suppressed even though the driver genuinely isn't working.
  • examine.rs (amdgpu_state/amdkfd_state, ~line 1797): the two states are computed independently even though kfd is linked into amdgpu; a container that passes through /dev/kfd without exposing /sys/module can yield the contradictory amdgpu_loaded=false / amdkfd_loaded=true.

Non-blocking

  • examine.rs (~line 1787) duplicates presence checks already implemented in host_gpu.rs::detect_driver_summary() instead of sharing one helper.
  • diagnose.rs/fix.rs remedy flows (e.g. sudo modprobe amdgpu, lsmod | grep amdgpu verify step) still assume loadable-module semantics and can still fire on exactly the built-in-kernel machines this PR targets, producing a remedy that can never verify as succeeded.
  • hardware_root.rs's module doc comment claims an exhaustive list of paths routed through host_path; the new /sys/class/kfd/kfd read isn't listed (stale invariant comment, not a behavior bug).
  • amdgpu_state/amdkfd_state are structurally near-identical and could collapse into one parameterized helper.
  • Minor/efficiency: host-presence checks run eagerly even when the module list already resolved the state to true.

Not verified: build, unit tests, e2e suite (no toolchain in this sandbox); CI status on the current head (see the other human/automated reviews on this PR for that signal).

🤖 Generated with Claude Code

@tomastola

Copy link
Copy Markdown
Collaborator Author

Pushed 6891205 for the 16:30 review.

Fixed

  • amdgpu_loaded and amdkfd_loaded are no longer decided independently. kfd is linked into amdgpu, so kfd on the host now also answers amdgpu_loaded. A container with /dev/kfd but no /sys/module can't report amdkfd up and amdgpu down.
  • amdgpu_state and amdkfd_state are now one module_or_host_state. The unit tests are collapsed to match.
  • The host probe is a closure, so it is skipped when the module list already answers. A test pins this.
  • hardware_root.rs now lists /sys/class/kfd/kfd.

Not changed

  • /sys/module/amdgpu for a built-in driver that never bound (nomodeset, failed probe): this is a real gap, but I left it. /sys/module/amdgpu is already this codebase's signal for "amdgpu is present" (host_gpu.rs, hardware_root.rs, the version read). Switching examine alone to /sys/bus/pci/drivers/amdgpu would make the two disagree about the same machine. Better as a follow-up that moves both together.
  • Sharing the check with host_gpu::detect_driver_summary: it reads /sys/module/amdgpu for a different purpose (module metadata, amdgpu_available), so I'd do that together with the follow-up above.
  • diagnose/fix remedies (lsmod | grep amdgpu, modprobe amdgpu) still assume a loadable module. That is a separate change to the remedy recipes and out of scope for the detection fix here.

The WSL2 failure is comfyui-source-download-shows-live-progress, a scenario this diff doesn't touch (a timing flake, #580). I haven't re-run it, and the new run will show whether it recurs.

@tomastola
tomastola force-pushed the fix-examine-kfd-builtin branch from 6891205 to 47d8a2a Compare October 27, 2026 10:26

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 47d8a2a

Review

Full review of the whole change (crates/rocm-core/src/examine.rs, crates/rocm-core/src/hardware_root.rs, tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs), base a9e9aa2…head 47d8a2a. The PR explicitly fixes issue #448 (amdkfd_loaded false negative) and extends the same fix to amdgpu_loaded for the same root cause; the PR body states that extension plainly, so it is not scope creep.

Blocking

  • The one genuinely new behavior this PR adds — "kfd on the host implies amdgpu" — has no test that exercises it. crates/rocm-core/src/examine.rs:1863-1865 computes e.amdgpu_loaded as module_or_host_state(.., "amdgpu", || amdgpu_present_on_host() || kfd_present_on_host()). The kfd_present_on_host() disjunct is new and is called out in both the inline comment above it and in commit 0ae8927a's message ("kfd is linked into amdgpu, so kfd on the host now also answers amdgpu_loaded"). Every test that asserts amdgpu_loaded == Some(true) instead plants /sys/module/amdgpu directly: the unit test probe_modules_reports_amdgpu_when_it_is_on_the_host (examine.rs:6278-6300) and e2e scenario examine-32 via machine_with_built_in_amdgpu (tests/e2e-cucumber/tests/e2e/examine_steps.rs:1520-1549, asserted by assert_amdgpu_loaded at line 1574). No unit test or e2e scenario plants only a kfd signal (/dev/kfd or /sys/class/kfd/kfd) with no amdgpu module entry and no /sys/module/amdgpu, and checks that amdgpu_loaded becomes true through that path alone. Deleting || kfd_present_on_host() from that closure would not fail a single test in the suite.
    Confidence 85 · logic · Fix: add a unit test (and ideally an e2e scenario, e.g. a container-shaped host that passes through /dev/kfd but has no /sys/module/amdgpu) that plants kfd-only presence and asserts amdgpu_loaded == Some(true).

Non-blocking

  • amdgpu_loaded's doc comment understates its own logic. examine.rs:228-231 says it is true when "listed as a module, or /sys/module/amdgpu exists" — it omits the third signal the implementation now ORs in (kfd_present_on_host(), line 1864). The sibling amdkfd_loaded doc comment is fully accurate about its own logic; this one is not.
    Confidence 85 · mechanical · Fix: mention the kfd-on-host fallback in the doc comment.
  • amdgpu_present_on_host() (examine.rs:1821-1823, checking host_path("/sys/module/amdgpu").exists()) duplicates logic already in host_gpu.rs::detect_driver_summary() (crates/rocm-core/src/host_gpu.rs:2449-2456), which independently checks the same path (and already separately folds in /dev/kfd for its detail text). Two independent "is amdgpu present on this host" checks now exist in the crate.
    Confidence 70 · logic (reuse) · Fix: consider sharing one helper between the two call sites as a follow-up; not required for this PR's scope.
  • The new ~140-line e2e fixture block duplicates the file's existing "write a fake executable, prepend it onto PATH" idiom (plant_machine_without_amdkfd_module at examine_steps.rs:1463, and the equivalent block inside machine_with_built_in_amdgpu at line 1520) rather than reusing it — a pre-existing instance of the idiom plus these two new ones means it's now written out three times in this file.
    Confidence 70 · mechanical (reuse) · Fix: extract a shared "plant fake lsmod/mokutil on PATH" helper now that the pattern recurs a third time.
  • The new fixture block has no section-divider comment, unlike the rest of the file's established convention, and currently reads as though it belongs to the preceding "release file" section (gpu_machine_with_unreadable_release_file, examine_steps.rs:1445) rather than being its own section.
    Confidence 70 · mechanical · Fix: add the divider comment used elsewhere in the file.
  • Pre-existing, pattern-relevant: diagnose.rs (:853, :978) and fix.rs (:333, :369) still print user-facing remediation strings of the form "lsmod | grep amdgpu && rocminfo | head -n ..." — the exact false-negative class this PR fixes in detection (a built-in amdgpu never appears in lsmod) still exists in the remediation text a user is told to run to confirm the driver is present. Not touched by this PR's four changed files, so out of scope here, but the same bug class and worth a follow-up.
    Confidence 70 · logic · Fix: a future change to those verify strings, e.g. checking /sys/module/amdgpu instead of/in addition to lsmod.
  • Commit 0ae8927a is labeled refactor(examine): … but is not a pure refactor — it adds the kfd_present_on_host() disjunct to amdgpu_loaded's computation, which is new semantics, not a reshuffle, as its own commit message's second sentence states. The behavior is otherwise fully described in the message, documented in the doc comments, and intentional; this is a labeling nit only.
    Confidence 70 · mechanical · Fix: none needed; note for future commit-type labeling.

Decisions for the author

  • Naming tradeoff, already made and documented — the PR keeps the field name amdkfd_loaded for JSON-schema compatibility rather than splitting it into e.g. amdkfd_module_loaded / kfd_functional as issue #448 suggested as an alternative. The PR body states this explicitly and gives the reason (schema stability). Reasonable; flagging only so the choice is visible as deliberate rather than overlooked.
  • Historical continuity worth a one-line mention — PR #434 (merged earlier) explicitly observed and recorded amdkfd_loaded: false alongside has_amd_gpu: true on an MI300X host as a known, unresolved quirk at the time ("the MI300X host confirms it, reporting amdkfd_loaded: false"). This PR now fixes exactly that quirk but doesn't reference #434. Not a code defect — purely a continuity nicety for anyone searching history later.

Positive signals

  • module_or_host_state's unit tests (examine.rs ~6186-6260) cover the full tri-state matrix cleanly: module listed, built-in-and-absent-from-the-list, neither-present, unreadable-list-follows-host, and host-probe-skipped-when-the-list-already-answers (that last one uses || unreachable!() as its own proof).
  • hardware_root.rs's module doc comment, which claims to enumerate "every one of these reads, and only these" through the mockable host_path, was correctly updated to add the new /sys/class/kfd/kfd read — this invariant is accurate as of this PR.
  • E2e scenario examine-32 does real impact-analysis work: it drives both examine --json and diagnose --json against the same planted host and asserts the diagnose evidence lines are actually absent, not just that the flag reads true — closing the loop to the real downstream consumer rather than stopping at the probe.
  • Clean 7-commit history: each commit is independently meaningful, DCO-signed, and a numbering collision in the e2e scenario IDs was proactively caught and fixed in its own dedicated commit (47d8a2aa) rather than left for review to find.

Deployment notes

  • Behavioral change, not just a bugfix in the abstract: on hosts where amdgpu/kfd are compiled into the kernel (increasingly common on stock distro kernels), rocm examine --json's amdgpu_loaded/amdkfd_loaded will now report true where they previously reported false, and rocm diagnose will stop emitting its DKMS-failure / module-not-loaded / Secure-Boot evidence lines for such hosts. JSON field names and types are unchanged (confirmed by reading Examination's struct definition), so this is a value-level behavior change, not a schema change.
  • As of this review (live-checked against head 47d8a2aa), no check had failed, but several E2E lanes (MI300X, Strix Halo Ubuntu release/nightly, W7900) were still QUEUED/not finished — full CI had not completed.

What this covered

Read all four changed files in full at the PR head, plus the consumers of the changed fields (diagnose.rs, fix.rs, host_gpu.rs) and the full AGENTS.md, via three independent blind passes (core probe logic + consumers, e2e feature/steps, and a cross-cutting AGENTS.md/impact-analysis/history/commit-quality pass), then consolidated and independently re-verified the load-bearing claims directly against the source (the untested-branch claim, the doc-comment gap, the host_gpu.rs duplication, the stale lsmod remedy strings, the refactor-commit behavior change, and the PR #434 history) before scoring. Base a9e9aa2, head 47d8a2a, re-confirmed live immediately before publishing this review.

Did not run: cargo test/cargo clippy locally in this sandbox — reasoned from reading the code and test matrix directly instead, and cross-checked against the PR's own CI, where the rust/clippy/build-and-test/coverage checks are green on this head (several GPU-hardware E2E lanes were still queued, not failed, at review time). The "prior changes" pass (whether review comments on earlier, unrelated PRs touching these same files still apply here) was not separately run. Issue #448 was fetched directly (gh issue view) and checked against the diff and PR description — the PR accurately describes its own scope, including the deliberate extension beyond what #448 asked for.

Not reconciled against any prior review thread on this PR — this review was produced blind to that discussion by design.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 3266af4

Review — needs work

Full review of the whole change. The PR fixes issue #448 (amdkfd_loaded false negative when kfd is built into the kernel) and deliberately extends the same fix to amdgpu_loaded, stating that extension explicitly in its own body — not scope creep.

Blocking

  • The "kfd present on the host implies amdgpu is loaded too" branch has no test that exercises it — crates/rocm-core/src/examine.rs:1866-1868 computes e.amdgpu_loaded as module_or_host_state(modules, "amdgpu", || amdgpu_present_on_host() || kfd_present_on_host()). The kfd_present_on_host() disjunct is new, deliberate behaviour (the inline comment at examine.rs:1864-1865 and the commit introducing it both say so explicitly). I traced every test touching amdgpu_loaded: the unit tests for module_or_host_state pass synthetic || true/|| false closures, never the real composed one; amdgpu_present_on_host_reads_the_pci_driver_dir and probe_modules_reports_amdgpu_when_it_is_on_the_host (examine.rs:6249, ~6300) plant only /sys/bus/pci/drivers/amdgpu, never kfd alone; e2e scenario examine-32 (machine_with_built_in_amdgpu, tests/e2e-cucumber/tests/e2e/examine_steps.rs:1520) plants the PCI driver dir directly and never plants /dev/kfd/kfd sysfs; examine-30/31 plant a fake lsmod that already lists amdgpu as a module, so modules.contains("amdgpu") answers before the host closure is ever reached, and neither scenario asserts amdgpu_loaded at all. Deleting || kfd_present_on_host() from examine.rs:1867 would not fail a single existing test.
    Confidence 90 · logic · Fix: add a unit test and/or e2e scenario that plants kfd-only presence (/dev/kfd or /sys/class/kfd/kfd) with no amdgpu module entry and no /sys/bus/pci/drivers/amdgpu, and asserts amdgpu_loaded == Some(true) through that path alone.

  • The PR description no longer matches the shipped code. The body still says amdgpu_loaded is true when "amdgpu is in the module list or /sys/module/amdgpu exists" — the final commit on this PR (fix(examine): take a built-in amdgpu from the PCI driver, not /sys/module) deliberately replaced that check with /sys/bus/pci/drivers/amdgpu (examine.rs:1829-1833) specifically because /sys/module/amdgpu is created even when the driver's init aborted under nomodeset/amdgpu.modeset=0 — the description now describes the behaviour its own author judged unreliable. The description also never mentions the kfd_present_on_host() fallback inside the amdgpu computation (examine.rs:1867) at all. The Tests section names amdkfd_state/amdgpu_state (examine.rs, Tests paragraph) — functions that no longer exist, having been unified into module_or_host_state by an earlier commit in this same PR — and cites e2e scenario IDs examine-29/examine-30/examine-31, while the actual scenarios in tests/e2e-cucumber/features/examine.feature are numbered examine-30/examine-31/examine-32 after a later renumbering commit. A reader relying on the description to understand what shipped would get the wrong sysfs path, miss an entire code path, and be unable to find the cited tests by name or ID.
    Confidence 85 · logic · Fix: update the PR body to describe the final /sys/bus/pci/drivers/amdgpu check, mention the kfd-implies-amdgpu fallback, and correct the test/scenario names and IDs before merge.

Non-blocking

  • Pre-existing, pattern-relevant: host_gpu.rs's detect_driver_summary() (crates/rocm-core/src/host_gpu.rs:2456) still decides status: "amdgpu_available" from host_path("/sys/module/amdgpu").exists() — the exact check this PR's own investigation found unreliable for a built-in driver (see the rationale at examine.rs:1829-1833). Not touched by this diff and not called from the changed code path, so out of scope here, but it's the identical bug class sitting right next to the fix.
    Confidence 70 · logic (reuse) · Fix: consider sharing amdgpu_present_on_host() with this call site as a follow-up.
  • The fake-executable/PATH-prepend fixture idiom is now duplicated 3-4 times in tests/e2e-cucumber/tests/e2e/examine_steps.rs with no shared helper: the pre-existing plant_gpu_fixtures (:1409-1431), and the two new fixtures this PR adds, plant_machine_without_amdkfd_module (:1463-1490) and machine_with_built_in_amdgpu (:1520-1549), each write a fake executable and manually prepend it onto PATH via the same std::env::join_paths/split_paths sequence.
    Confidence 70 · mechanical (reuse) · Fix: extract a small fn prepend_fake_bin_to_path(world, bin) helper now that the pattern recurs a third time.

Decisions for the author

  • Naming tradeoff, already made and documented — the PR keeps the field name amdkfd_loaded for JSON-schema compatibility rather than splitting it into e.g. kfd_functional, which issue #448 suggested as an alternative. The body states this explicitly with its reasoning; flagging only so the choice is visible as deliberate.
  • Historical continuity worth a one-line mention — an earlier merged PR (#434) recorded amdkfd_loaded: false on an MI300X host as a known, unresolved quirk at the time. This PR fixes exactly that quirk but doesn't reference #434. Not a defect, purely a continuity nicety for anyone searching history later.

Positive signals

  • module_or_host_state's unit tests cover the full tri-state matrix cleanly (module listed, built-in-and-absent, neither present, unreadable-list-follows-host, host-probe-skipped-once-the-list-answers — the last uses || unreachable!() as its own proof).
  • diagnose.rs's fault-evidence checks all gate strictly on == Some(false), so the new tri-state semantics (module list OR host probes, falling back to None/"unknown" rather than a false negative) compose correctly with the existing consumer without any additional change needed there.
  • hardware_root.rs's module doc comment, which claims to enumerate every host-path read in the crate, was correctly updated to add both the new /sys/class/kfd/kfd class-device read and /sys/bus/pci/drivers/amdgpu — this invariant holds as of this PR.
  • Clean 8-commit history: each commit is independently meaningful and signed off, and the switch from /sys/module/amdgpu to the PCI-driver-dir check (a real correctness improvement, catching a case the earlier commits in this same PR had gotten wrong) was caught and fixed in its own dedicated commit rather than folded in silently.

Deployment notes

  • Behavioural change, not just a bugfix in the abstract: on hosts where amdgpu/kfd are compiled into the kernel, rocm examine --json's amdgpu_loaded/amdkfd_loaded will now report true where they previously reported false, and rocm diagnose will stop emitting its DKMS-failure/module-not-loaded/Secure-Boot evidence lines for such hosts. JSON field names and types are unchanged, so this is a value-level behaviour change, not a schema change.
  • As of this review (live-checked against head 3266af46), every CI check has completed and passed except the aggregate "E2E tests" check, which was still in progress; nothing had failed.

What this covered

Read both changed core-logic files (crates/rocm-core/src/examine.rs, crates/rocm-core/src/hardware_root.rs) and both changed e2e files (tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs) completely, via two independent blind fan-out passes grouped by coupling (core probe logic + its consumers; e2e feature/steps), plus my own direct reading of the full diff, the 8-commit history, the consumers in diagnose.rs/fix.rs/host_gpu.rs, and AGENTS.md. Consolidated and independently re-verified the load-bearing claims against the source (the untested-branch claim, the stale-description claim, the host_gpu.rs duplication, the e2e fixture duplication) before scoring. Base a9e9aa2103ae974b10c5f6e141c09dc6720d8c26, head 3266af461f84bff2e35a69d587f87461b6302a17, re-confirmed live immediately before publishing this review. Issue #448 was fetched directly and checked against the diff and description.

Did not run: cargo test/cargo clippy locally (no Rust toolchain in this sandbox) — reasoned from reading the code and test matrix directly instead, cross-checked against the PR's own CI, where every completed check is green on this head. The "prior changes" pass (whether review comments on earlier, unrelated PRs touching these same files still apply here) was not separately run beyond the one historical PR (#434) found by searching commit history for this exact field name.

Not reconciled against any prior review thread on this PR — this review was produced blind to that discussion by design.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Pushed 108d462 for the 3266af4 review (it had no inline threads, so replying here).

Fixed

  • Untested "kfd implies amdgpu" branch: the composed host probe is now amdgpu_implied_by_host, and kfd_alone_on_the_host_implies_amdgpu plants only /sys/class/kfd/kfd, then only /dev/kfd, and asserts true. I checked it fails with the kfd disjunct removed. I did not add an e2e scenario, because the unit test is independent of the lsmod of the machine running it.
  • The amdgpu_loaded doc comment now names the kfd fallback.
  • PR description: updated to the PCI driver directory check, the kfd fallback, module_or_host_state, and examine-30/31/32. It also now says the e2e scenarios were last run locally before the renumbering and the PCI-directory change.

Not changed

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 108d462

Review — no blocking findings

Full review of the whole change: crates/rocm-core/src/examine.rs, crates/rocm-core/src/hardware_root.rs, tests/e2e-cucumber/features/examine.feature, and tests/e2e-cucumber/tests/e2e/examine_steps.rs, base a9e9aa21…head 108d462c. The PR fixes issue #448 (amdkfd_loaded reporting false on a host where kfd is built into the kernel) and deliberately extends the same reasoning to amdgpu_loaded, stating that extension explicitly in its own description. The most recent commit on this PR adds a dedicated unit test (kfd_alone_on_the_host_implies_amdgpu) and a named helper (amdgpu_implied_by_host) that together pin down the "kfd on the host alone implies amdgpu" path, which is exactly the kind of behavior that benefits from its own direct test rather than only being exercised incidentally through other fixtures. With that in place, and with CI fully green, this round found no blocking issues — only two small, pre-existing-style documentation/maintenance notes below.

Blocking

None

Non-blocking

  • Two doc comments on this PR's own new code disagree about whether a pre-4.20 kernel's amdkfd ever shows up in lsmod. The amdkfd_loaded field doc (crates/rocm-core/src/examine.rs:234-237) says kfd is available "as a separate amdkfd module on kernels before 4.20 (where kfd was merged into amdgpu.ko)" — implying older kernels do list it as a module. The kfd_present_on_host() doc directly above it (crates/rocm-core/src/examine.rs:1813-1818) says the opposite: "amdkfd is linked into amdgpu, so no amdkfd line ever appears in the module list, built in or not." The field doc looks like the one that matches reality (and matches why probe_modules still checks the module list for "amdkfd" before falling back to the host probe) — the function doc should be narrowed to "on current kernels" rather than stated unconditionally.
  • The fake-executable/PATH-prepend fixture idiom is now written out a third time in tests/e2e-cucumber/tests/e2e/examine_steps.rs: the pre-existing plant_gpu_fixtures (:1409), and this PR's two new fixtures plant_machine_without_amdkfd_module (:1463) and machine_with_built_in_amdgpu (:1520), each independently create a fake binary and splice it onto PATH. Not a functional problem, but a good candidate for a small shared helper now that the pattern recurs.

Decisions for the author

  • Naming tradeoff, already made and documented — the PR keeps the field name amdkfd_loaded for JSON-schema compatibility rather than splitting it into separate module-vs-functional fields, which issue #448 raised as an alternative. The description states this explicitly with its reasoning; noting it here only so the choice reads as deliberate.

Positive signals

  • The tri-state logic (module_or_host_state: module-list hit, host-probe hit, list unreadable but host answers, neither) is covered by unit tests for every branch, including a test that proves the host probe is skipped once the module list has already answered.
  • The e2e scenario for a built-in amdgpu drives both examine --json and diagnose --json against the same planted host and asserts that diagnose's DKMS/module-not-loaded/Secure-Boot evidence lines are actually absent, closing the loop to the real downstream consumer rather than stopping at the probe's own output.
  • The PR description's "Tests" section accurately names the current helper functions, test names, and e2e scenario IDs as they exist in the code at this head — it reads as kept in sync through the PR's later commits rather than written once and left stale.

Deployment notes

  • Behavioral change, not just a bugfix in the abstract: on hosts where amdgpu/kfd are compiled into the kernel, rocm examine --json's amdgpu_loaded/amdkfd_loaded will now report true where they previously reported false, and rocm diagnose will stop emitting its DKMS-failure/module-not-loaded/Secure-Boot evidence lines for such hosts. JSON field names and types are unchanged, so this is a value-level behavior change, not a schema change.
  • CI on this head: all 24 checks report either success (22) or an advisory skip (2, not required for merge); nothing failing or pending at review time.

What this covered

Read both core-logic files and both e2e files completely at the PR head, via two independent passes grouped by coupling (core probe logic + its mockable host-path primitive; e2e feature + step definitions), plus a direct read of the full project contributor-guidance file and a comparison of the PR description and linked issue (#448) against the shipped code. Did not run cargo test/cargo clippy/the e2e suite locally — no Rust toolchain or package-registry network access in this environment — so the above is reasoned from reading the code and tests directly, cross-checked against this PR's own CI (fully green). Not reconciled against any existing review thread on this PR; this review was produced independently against the code at the current head.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Pushed a doc-only commit for the 108d462 review. The kfd_present_on_host doc comment now says kfd is linked into amdgpu on current kernels, and that before 4.20 it was a separate amdkfd module, so it agrees with the amdkfd_loaded field doc.

The fake-executable/PATH fixture written out three times in examine_steps.rs is a fair follow-up but out of scope here. Extracting it means touching unrelated steps in the same file.

@tomastola

Copy link
Copy Markdown
Collaborator Author

@jussielo-amd your blocking item was the "E2E tests (Strix Halo, WSL2, current)" lane. On 6891205 it passed (job).

That self-hosted run went red on two other lanes. Neither failure involves examine:

  • Strix Halo, Windows: the lane stopped in setup, before any scenario ran. rocm engines install got HTTP 403 downloading the TheRock tarball for the Lemonade backend.
  • MI300X: the failing steps are serve and engine-install scenarios. The Lemonade llama.cpp and TheRock tarball downloads failed after 6 attempts, and there is a vllm reinstall. Every examine scenario passed, including the new examine-31.

The self-hosted run on the current head (4d58eac) is still queued. Every other check on 4d58eac is green. If the WSL2 result is enough for you, could you dismiss the changes-requested review?

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 4d58eac

Review — no blocking findings

Full review of the whole change (4 files, 457 lines, two independent fan-out workers over coupled file groups plus driver-level impact analysis).
Fixes #448 (GitHub issue, retrieved): "amdkfd_loaded=false is a false negative when kfd is built into the kernel." The change implements the issue's suggested functional-check fallback for amdkfd_loaded, and additionally fixes a related amdgpu_loaded blind spot not raised in the issue itself — this extra scope is disclosed and justified in the PR description, so it is not an undisclosed-work mismatch.

Blocking

None

Non-blocking

  • assert_no_amdgpu_fault_evidence's first guarded substring doesn't verbatim-match one of the two evidence strings it's meant to guard — tests/e2e-cucumber/tests/e2e/examine_steps.rs:1593 (checked against crates/rocm-core/src/diagnose.rs:829 and :934)
    The assertion guards against "amdgpu module is not loaded". That matches check_5_amdgpu_blacklisted's evidence (diagnose.rs:934) exactly, but check_3_rocm_kernel_unsupported's evidence (diagnose.rs:829) reads "...the amdgpu kernel module is not loaded..." — the extra word "kernel" breaks the substring match. The test is not currently vacuous: a regression in check_3 would still be caught by the separate "DKMS failed" substring in the same assertion, since check_3's full evidence text also contains that phrase. But the two substrings now guard the two checks by accident of overlapping wording rather than by each substring matching its intended check, so a future independent wording tweak to check_5 (e.g. inserting "kernel" to match check_3's phrasing) would silently stop that substring from testing anything, with the assertion still passing.
    Confidence 75 · mechanical · Fix: match check_3's and check_5's evidence text precisely (two substrings, one per check), or assert on diagnosis IDs/titles rather than free-text evidence substrings.

Decisions for the author

  • kfd presence on the host is treated as sufficient to imply amdgpu_loaded = true, without independently confirming /sys/bus/pci/drivers/amdgpu — tradeoff
    This is deliberate and documented (amdgpu_implied_by_host's doc comment: "a container that passes /dev/kfd through without /sys/bus/pci must not report amdkfd up and amdgpu down"), and is the right call for the stated container-passthrough case. The flip side: a mount namespace that still exposes a stale /dev/kfd node after the host's amdgpu driver has actually unbound (rare, but not guaranteed impossible across every container runtime) would make amdgpu_loaded report true when the PCI driver is genuinely gone. Worth confirming this residual risk is acceptable versus the false-negative it replaces, which was the entire point of the fix.

Positive signals

  • The doc comments on amdgpu_present_on_host/amdgpu_implied_by_host give a precise, falsifiable reason for choosing /sys/bus/pci/drivers/amdgpu over /sys/module/amdgpu (the latter appears at an early initcall even when amdgpu_init then bails out on nomodeset), and the unit test amdgpu_present_on_host_reads_the_pci_driver_dir directly plants both directories separately to prove the distinction the comment claims.
  • the_host_probe_is_skipped_when_the_module_list_already_answers uses an || unreachable!() closure to prove the laziness claimed in module_or_host_state's doc comment, rather than merely asserting the result.
  • The PR description directly engages with the linked issue's own suggested alternative (splitting into amdkfd_module_loaded/kfd_functional) and states a specific reason for rejecting it (JSON schema compatibility) rather than silently picking one approach.

Deployment notes

None — both fields keep their existing names, types, and Option<bool> tri-state semantics (unknown stays unknown when neither the module list nor the host answers); this only makes Some(true) reachable in more true-positive cases. probe_modules remains gated to os_family == "linux", so Windows/WSL paths are unaffected.

What this covered

Every file in the diff read completely by at least one of two independent fan-out workers, grouped by coupling: (1) crates/rocm-core/src/examine.rs + crates/rocm-core/src/hardware_root.rs (core probe logic, its unit tests, and every amdgpu_loaded/amdkfd_loaded read site in crates/rocm-core/src/diagnose.rs for impact analysis), and (2) tests/e2e-cucumber/features/examine.feature + tests/e2e-cucumber/tests/e2e/examine_steps.rs (e2e scenarios and step definitions, cross-checked against the real probe_modules/probe_secure_boot parsers and the DiagnoseReport JSON schema). I additionally read the commit history on the branch (10 commits, confirming substantial iteration since the previously-reviewed commit 108d462 — notably a full switch of the amdgpu signal from /sys/module/amdgpu to /sys/bus/pci/drivers/amdgpu in commit 3266af4), checked for other consumers of these two fields and for required doc/README updates (none needed; no doc references these fields' detection semantics), retrieved the linked issue (#448) directly by its identifier via gh issue view, and read CI (gh pr view --json statusCheckRollup): every required check is SUCCESS; the two SKIPPED checks (Skill checks (skillscope), Sphinx docs build (-W)) are path-filtered and correctly skipped since this change touches neither skills/ nor docs.
Did not run: cargo/clippy locally (no cargo available in this sandbox) — relied on CI's passing clippy, build-and-test, and Coverage (rocm-dash crates, ratcheted) checks instead, per the review method's guidance to trust a passing check rather than re-derive what it covered. History pass (git log/git blame beyond the commit-message read above) and prior-changes pass were folded into the fan-out workers' context rather than run as fully separate passes. Not reconciled against any discussion on this PR — this report was produced blind to any existing review thread, per instructions, and the prior review at commit 108d462 was deliberately not read.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Pushed bc2937d for the 4d58eac review. The review had no inline threads, so I'm replying here.

Fixed: the evidence substrings in assert_no_amdgpu_fault_evidence
The assertion now has one phrase per check, each copied from that check's evidence text:

  • "the amdgpu kernel module is not loaded" for check_3
  • "amdgpu module is not loaded" and the Secure Boot line for check_5

check_3 is no longer caught only through the "DKMS failed" overlap.

To check that each phrase works, I made amdgpu_loaded report false and ran examine-32 locally. The step failed on check_5's line. With the check_5 phrases removed from the list, it still failed, this time on check_3's line. The unmodified scenarios examine-31 and examine-32 pass.

Decision: kfd on the host implies amdgpu_loaded = true
I'm keeping it. A stale /dev/kfd would also make amdkfd_loaded true, because both fields read the same host evidence. Deriving amdgpu_loaded from it keeps the two fields consistent. Reporting kfd up and amdgpu down describes a state that cannot exist, since kfd is linked into amdgpu. Before this change, the container passthrough case (with /dev/kfd but no /sys/bus/pci) always read as "amdgpu not loaded". I judge that false negative to be the bigger risk.

@tomastola
tomastola requested a review from nowycondro October 27, 2026 14:51

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · bc2937d

Review — needs work

Full review of the whole change, read against a9e9aa21…bc2937d8 (the merge-base with origin/main).
Fixes #448, and goes beyond it: the issue reports only the amdkfd_loaded false negative, the change also fixes the analogous amdgpu_loaded false negative for a built-in driver. Both are described accurately in the PR body.

Blocking

  • The new host-fallback wiring in probe_modules is under-tested: two ways it could regress would still pass every test. crates/rocm-core/src/examine.rs:1873-1874 (the call site), with the gap visible at :6294-6318 (probe_modules_reports_amdgpu_when_it_is_on_the_host), :6394-6418 (probe_modules_reports_amdkfd_when_kfd_is_on_the_host), :6237-6239 (an_unreadable_module_list_follows_the_host), and tests/e2e-cucumber/features/examine.feature:447-461 (examine-30/31).

    • (a) Swapping which closure answers which field at the call site is not caught anywhere. If line 1873 used amdgpu_present_on_host instead of amdgpu_implied_by_host (dropping "kfd on the host implies amdgpu"), or line 1874 used amdgpu_implied_by_host instead of kfd_present_on_host (making amdkfd_loaded true whenever amdgpu alone is present), no test fails:
      • probe_modules_reports_amdgpu_when_it_is_on_the_host only plants /sys/bus/pci/drivers/amdgpu and only asserts amdgpu_loaded.
      • probe_modules_reports_amdkfd_when_kfd_is_on_the_host only plants /dev/kfd and only asserts amdkfd_loaded.
      • kfd_alone_on_the_host_implies_amdgpu (examine.rs:6358-6387, per the diff) calls amdgpu_implied_by_host() directly, bypassing probe_modules's call-site wiring entirely.
      • e2e examine-30's fake lsmod already lists amdgpu, so the module list answers amdgpu_loaded before any host check runs; examine-32 plants the PCI driver dir directly, so it never isolates the kfd-only path either.
        I traced this by reading the tests against the production code, not by running a mutation (no cargo/rustc available in this sandbox).
    • (b) The "module list unreadable, host says true" path (module_or_host_state's None arm, examine.rs:1849-1858) has no scenario, only the plain helper unit test an_unreadable_module_list_follows_the_host, which calls module_or_host_state directly and asserts nothing about probe_modules's wiring or examine --json output. This is new user-observable behaviour this PR introduces (before: an unreadable module list left both fields None unconditionally; now it can resolve to Some(true)), and AGENTS.md §3 requires a scenario for CLI-observable behaviour, not only a unit test.
      Confidence 85 · logic · Fix: strengthen the two existing wiring tests rather than adding new ones — in probe_modules_reports_amdkfd_when_kfd_is_on_the_host (only /dev/kfd planted) also assert amdgpu_loaded == Some(true); in probe_modules_reports_amdgpu_when_it_is_on_the_host (only the PCI driver dir planted) also assert amdkfd_loaded == Some(false) (host-independent, since no modern kernel lists an amdkfd module). For (b), add a scenario/unit test with a failing lsmod and empty /proc/modules plus a planted /dev/kfd, asserting both fields come back true.
  • docs/release-trust.md's "Simulated Host Root" path inventory was not updated with the new routed read. docs/release-trust.md:312-326, caused by crates/rocm-core/src/examine.rs:1821-1833 (new amdgpu_present_on_host/kfd_present_on_host reads) together with crates/rocm-core/src/hardware_root.rs:11-17.

    • The change correctly updated hardware_root.rs's own "every one of these reads, and only these" doc comment to add /sys/class/kfd/kfd and /sys/bus/pci/drivers/amdgpu.
    • docs/release-trust.md's "Simulated Host Root" section is the same enumeration, restated for a different audience (what the ROCM_CLI_TEST_HOST_ROOT override can steer): it still only mentions "/sys/module/amdgpu and its version file" and "the KFD topology under /sys/class/kfd", with no mention of /sys/bus/pci/drivers/amdgpu.
    • AGENTS.md §5 names this exact pattern: "the same behavior claim… often repeats across… docs/*.md; each drifts independently, so grep for the claim's wording across all of them… and check each against the actual code path." The code is what governs here — the new read genuinely goes through host_path.
    • Confidence 85 · mechanical · Fix: add /sys/bus/pci/drivers/amdgpu (and ideally call out the bare /sys/class/kfd/kfd class device as distinct from the topology-nodes subpath) to that paragraph.

Non-blocking

  • A CI workflow comment enumerating which routed reads have planted-root unit assertions is now stale. .github/workflows/ci.yml:552-562. It lists which probes have planted-root assertions and says the rest (including /sys/module/amdgpu) have none; this change adds planted-root assertions for /sys/bus/pci/drivers/amdgpu, /sys/class/kfd/kfd, and /dev/kfd via probe_modules, which the comment doesn't mention. Confidence 70 · mechanical · Fix: add the new probe_modules/kfd_present_on_host/amdgpu_present_on_host tests to the list.

Decisions for the author

None.

Positive signals

  • module_or_host_state keeps "module list unreadable and host silent" as None rather than Some(false), so rocm diagnose cannot score a driver as missing when nothing was actually observed.
  • Choosing /sys/bus/pci/drivers/amdgpu over /sys/module/amdgpu is the right call for a built-in driver whose init bailed out (nomodeset), and it's backed by a unit test (amdgpu_present_on_host_reads_the_pci_driver_dir) pinning that /sys/module/amdgpu alone does not count.
  • The 11-commit history is clean and well-scoped (one concern per commit: helper extraction, wiring, docs, each e2e scenario), with descriptive messages — no "fix"/"wip" noise.
  • The PR description is unusually candid about what wasn't verified ("not re-run locally since [renumbering]... CI is the check for those"; "not run on real hardware"), which is exactly the kind of verified-vs-assumed distinction AGENTS.md asks for.

Deployment notes

  • amdkfd_loaded and amdgpu_loaded in rocm examine --json change meaning for existing hosts: a modern-kernel host with kfd now reports amdkfd_loaded: true where it previously always reported false, and a host with amdgpu built into the kernel flips amdgpu_loaded from false to true. Field names and types are unchanged (schema-compatible). No consumer inside this repo (only diagnose.rs, checked) needs updating; an external consumer of the JSON contract that hard-coded the old (buggy) semantics would see a behavior change.

What this covered

Read completely: crates/rocm-core/src/examine.rs (6723 lines at head) and crates/rocm-core/src/hardware_root.rs (408 lines), tests/e2e-cucumber/features/examine.feature (474 lines) and tests/e2e-cucumber/tests/e2e/examine_steps.rs (2173 lines), via two independent fan-out workers grouped by coupling (production code; e2e test code), each reading its files in full plus the production code the scenarios exercise. I separately verified the key blocking finding myself by reading the exact test bodies and fake-lsmod fixtures against the call site, and checked diagnose.rs's three amdgpu_loaded-reading evidence lines, docs/release-trust.md, .github/workflows/ci.yml, and this repo's AGENTS.md directly.

Agent-instruction adherence (AGENTS.md) and code-comments passes were run directly rather than as separate workers, given the diff's size. History pass was limited to this branch's own 11-commit sequence (read and found clean); I did not search git blame/pre-existing history of examine.rs/hardware_root.rs beyond this branch. Prior-changes pass (review comments on earlier PRs touching these files) did not run — no search performed. Step 7 design questions were investigated directly (host-path routing consistency, diagnose.rs interaction with amdgpu_blacklisted_in, the examine.py-mirroring doc comment's relevance — found non-actionable, that file lives outside this repo) rather than via separate workers, again given the diff's size.

CI: read via gh pr view --json statusCheckRollup at the reviewed head; all required checks (Analyze ×3, clippy, CodeQL, build-and-test, Test (affected crates), windows-build-and-test, E2E tests, E2E consolidated report, license/notices/lint checks, etc.) report SUCCESS. Skill checks (skillscope) and Sphinx docs build (-W) report SKIPPED, unrelated to this diff.

Issue #448 was retrieved and read; the PR's description was read after all findings above were settled. Not reconciled against any PR review-thread discussion — this is a blind pass per the review method, and a prior nowycondro review exists on this PR at an earlier head (4d58eacc) that was not read.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Pushed 672fcf4, 5474d87 and e0385fe for the bc2937d review. That review had no inline threads either, so I'm replying here.

Fixed: the probe_modules wiring could swap the host checks without any test failing (blocking, part a)
The two fields now come from driver_states(modules), which takes the module list as an argument. driver_states_answer_each_field_from_its_own_host_signal plants one signal at a time and runs it with an empty module list and with none. Swapping either closure at the call site now fails it. I applied both swaps you described and confirmed each one fails. Following your suggestion, the two probe_modules_* wiring tests now also assert the other field:

  • with only /dev/kfd planted, amdgpu_loaded == Some(true);
  • with only the PCI driver dir planted, amdkfd_loaded != Some(true).

That second assertion is != rather than == Some(false). If the runner's lsmod fails, the routed /proc/modules is empty, so None is also correct there.

Fixed: no scenario covered the unreadable module list (blocking, part b)
examine-33 (examine-reports-drivers-from-kfd-without-a-module-list) covers it. It plants /dev/kfd, no /proc/modules and a lsmod that exits 1, then requires examine --json to report both amdkfd_loaded and amdgpu_loaded true. examine-30 to examine-33 pass locally through cargo xtask e2e.

Fixed: docs/release-trust.md's path list (blocking)
The Simulated Host Root paragraph now lists the /sys/class/kfd/kfd class device separately from the topology under it, and adds /sys/bus/pci/drivers/amdgpu.

Fixed: the stale CI comment (non-blocking)
The planted-root coverage comment in ci.yml now names the /sys/class/kfd/kfd, /dev/kfd and /sys/bus/pci/drivers/amdgpu reads and the tests that assert them.

The PR description's Tests section now covers all of this.

@tomastola
tomastola requested a review from nowycondro October 27, 2026 16:00

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · e0385fe

Review

This round covers the 3 commits added since the last review at bc2937d8 (672fcf4, 5474d87, e0385fe), which were written specifically to close the two blocking findings and one non-blocking finding from that review: an untested "closures swapped at the call site" risk, an untested "module list unreadable, host still answers" path, and a stale docs/release-trust.md path inventory. All three are verified fixed below. No blocking findings; CI is fully green (24/24 checks).

Blocking

None

Non-blocking

  • The fake-executable/PATH-prepend fixture idiom is now written out a fourth time. tests/e2e-cucumber/tests/e2e/examine_steps.rs's new machine_with_kfd_and_no_module_list (added in 5474d87) repeats the same "write a fake binary, splice it onto PATH via std::env::join_paths/split_paths" sequence as plant_gpu_fixtures, plant_machine_without_amdkfd_module, and machine_with_built_in_amdgpu. Raised in every prior round on this PR as a candidate for a shared helper; still unaddressed, still low-cost to leave as is.
    Confidence 70 · mechanical (reuse) · Fix: extract a fn prepend_fake_bin_to_path(world, name, script) helper now that the pattern recurs a fourth time.

Decisions for the author

  • kfd presence alone is sufficient to imply amdgpu_loaded = true, without independently confirming /sys/bus/pci/drivers/amdgpu — tradeoff (unchanged from an earlier round, still present and still the deliberate, documented choice in amdgpu_implied_by_host's doc comment). A mount namespace that still exposes a stale /dev/kfd node after the host's amdgpu driver has actually unbound would make amdgpu_loaded report true when the PCI driver is genuinely gone. Worth re-confirming this residual risk is acceptable versus the false-negative it replaces.
  • Naming tradeoff, already made and documented — amdkfd_loaded is kept rather than split into e.g. kfd_functional, which issue #448 suggested as an alternative. Stated explicitly in the PR body with its reasoning; noting only so the choice reads as deliberate.

Positive signals

  • driver_states_answer_each_field_from_its_own_host_signal (new in 672fcf4, crates/rocm-core/src/examine.rs) directly closes the swap-detection gap from the last round: its pci_only/unread_pci_only cases assert (Some(true), Some(false))/(Some(true), None) — an asymmetric pair that a swapped driver_states tuple, or a swapped destructuring at the probe_modules call site, would actually fail. I traced the logic by hand (amdgpu_present_on_host/kfd_present_on_host/module_or_host_state against each planted-root case) and confirmed every asserted value is correct.
  • The new e2e scenario examine-33 (5474d87) plants /dev/kfd with no readable module list at all (no /proc/modules, a failing fake lsmod) and asserts both amdkfd_loaded and amdgpu_loaded come back true through examine --json — this is real CLI-observable coverage for the "list unreadable, host answers" path the prior round flagged as unit-test-only.
  • docs/release-trust.md and the .github/workflows/ci.yml planted-root coverage comment (e0385fe) were both updated to list /sys/class/kfd/kfd and /sys/bus/pci/drivers/amdgpu, matching what hardware_root.rs's own doc comment already enumerated — no drift between the three restatements of this invariant anymore.
  • The PR description's Tests section is now byte-accurate against the shipped code (function names, scenario IDs examine-30..examine-33) — a genuine improvement over an earlier round in this PR's history where that section had drifted from a mid-PR renumbering.

Deployment notes

Unchanged from prior rounds: on hosts where amdgpu/kfd are compiled into the kernel, rocm examine --json's amdgpu_loaded/amdkfd_loaded now report true where they previously reported false, and rocm diagnose stops emitting its DKMS/module-not-loaded/Secure-Boot evidence lines for such hosts. Field names and types are unchanged (schema-compatible).

What this covered

Fetched the PR head read-only (git fetch origin pull/694/head, never checked out) and diffed a9e9aa21 (merge-base with main) through the current head e0385fe8. The prior nowycondro review at bc2937d8 is attributable, covered the rest of this file set in full at a head that differs from the current one only by these 3 commits, so I narrowed to the delta: git diff --stat bc2937d8..e0385fe8 shows only crates/rocm-core/src/examine.rs, .github/workflows/ci.yml, docs/release-trust.md, tests/e2e-cucumber/features/examine.feature, and tests/e2e-cucumber/tests/e2e/examine_steps.rs touched; hardware_root.rs is untouched by this delta and was not re-read. Read the full diff for each of those 5 files, the surrounding functions it touches in examine.rs (driver_states, module_or_host_state, probe_modules, all four host-presence helpers and their doc comments), and the full current field doc comments on amdgpu_loaded/amdkfd_loaded. The delta is small (147 lines) and each of the three commits maps one-to-one onto a specific prior finding, so this was done as a single direct pass rather than a fan-out (per the method's size threshold), not because the change was large.

Read issue #448 directly (gh issue view) — the PR's scope (fixing amdkfd_loaded, and disclosing the deliberate extension to amdgpu_loaded) still matches it. Read the PR description after settling findings; it is accurate against the current head, including the previously-stale Tests section.

Did not run: cargo test/cargo clippy locally (no Rust toolchain in this environment) — relied on this PR's own CI instead, where all 24 checks report pass on the current head (gh pr checks 694), re-confirmed live immediately before publishing. History pass beyond this PR's own 14-commit sequence, and a separate "prior changes on these files" search, were not run.

Not reconciled against any discussion on this PR, including the existing review thread with inline comments from tomastola and the prior nowycondro reviews — this review was produced blind to that discussion by design.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Thanks for the review of e0385fe.

  • Fake-executable/PATH fixture repeated four times: fixed in a3911ce. put_fake_executables_on_path now writes the fake binaries and puts them on PATH for all four fixtures. examine-23, -28 and -30 to -33 pass locally.
  • kfd alone implies amdgpu_loaded: kept as is. /dev/kfd exists only while amdgpu is bound, so a stale node left after an unbind is the narrower risk. The false false it replaces affected every host with the driver built in.
  • amdkfd_loaded naming: kept as is, for the reasons given in the description.

@tomastola
tomastola requested a review from nowycondro October 27, 2026 16:39
main split examine.rs into domain modules (#829). Port this branch's
driver-presence probe and its tests into examine/linux_driver.rs, and the
host-root locks in the repo-scan tests into examine/rocm_install_probe.rs.
The Examination field docs stay in examine.rs.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · e0115c8

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (all 8 changed files, read completely).
Fixes #448 (the amdkfd_loaded false-negative for built-in kfd) and additionally extends the same approach to amdgpu_loaded for the analogous built-in-amdgpu case — not asked for by #448, but explicitly described and justified in the PR body, so this is not a scope mismatch.

Blocking

  • amdgpu_loaded can read true from a kfd/driver-object signal that proves the driver's init code ran, not that any GPU actually bound to it — crates/rocm-core/src/examine/linux_driver.rs (amdgpu_implied_by_host, kfd_present_on_host, amdgpu_present_on_host), consumed at crates/rocm-core/src/diagnose.rs:832,949,957 (check_3_rocm_kernel_unsupported, check_5_amdgpu_blacklisted).
    Upstream drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c's amdgpu_init() calls amdgpu_amdkfd_init() (which creates /dev/kfd and /sys/class/kfd/kfd) before pci_register_driver() — i.e. before the PCI subsystem has matched, probed, or bound any device. So kfd's chardev/class-device existing proves amdgpu's init code executed, not that a GPU is bound to it. Concretely reachable: a CONFIG_AMDGPU=y kernel with no AMD GPU, or one whose GPU fails to probe (unsupported ASIC, missing firmware, IOMMU/reset failure, nomodeset) — /dev/kfd exists, so amdgpu_implied_by_host() reports true, which suppresses exactly the DKMS/blacklist/Secure-Boot evidence lines meant to catch "amdgpu isn't really bound to your GPU." amdgpu_present_on_host() (the /sys/bus/pci/drivers/amdgpu check) has a structurally similar gap one level down (driver registration ≠ device bound), and the pre-PR lsmod-only check never proved binding either — so the underlying imprecision (driver-resident vs. GPU-bound) isn't new. What's new is the population it now reaches: previously a built-in, unbound amdgpu was (for the wrong specific reason) still reported false and still triggered an alert; after this PR the same machine is reported true and the alert is silently suppressed. No test in this diff plants a "kfd/driver-object present, nothing bound" fixture, and no comment acknowledges this direction of the asymmetry (only the documented "amdgpu presence doesn't imply kfd" direction is covered).
    Confidence 85 · logic · Fix: require evidence of an actually-bound device before reporting true via the host-probe path — e.g. PCI BDF-named entries (not just bind/uevent/module) under /sys/bus/pci/drivers/amdgpu/, or a kfd topology node beyond node 0 (/sys/class/kfd/kfd/topology/nodes/<N>, N>0 — node 0 is always the CPU).

  • A CI check on this head commit is red: E2E tests (Strix Halo, Windows, current) — run 41639909936, job 125930604326. The failure is 'runtime-resolve-canonical-release' was expected to pass on this host but FAILED — a regression — a wheel/runtime-resolution scenario, not anything in this PR's diff (which only touches Linux kfd/amdgpu detection). Flagging per a failing check being an objective blocking signal, but the subject-matter mismatch makes it likely an unrelated pre-existing flake on that hardware lane rather than something this change caused; worth a quick confirmation rather than treating it as caused by this PR. Two GPU hardware lanes (E2E tests (MI300X, current), E2E tests (W7900, current)) are still queued and two Strix-Halo Ubuntu lanes (nightly, release) were cancelled on this head — none of them have actually exercised this change yet.
    Confidence 100 (directly observed check state) · n/a · Fix: rerun/triage that job; not an action item against this diff's content unless it reproduces again.

Non-blocking

  • cargo xtask coverage's job doesn't pass --features e2e-test-hooks — .github/workflows/ci.yml's coverage: job, so the new kfd_present_on_host/amdgpu_present_on_host/amdgpu_implied_by_host/driver_states bodies are exercised by the separate "Test rocm-core with e2e-test-hooks" step but likely not counted toward the rocm-core coverage floor (82.14 in coverage-floors.toml). This mirrors a pre-existing gap already present for other routed reads (stat_device, is_wsl_host, probe_container), so it isn't newly introduced — just worth confirming cargo xtask coverage --bless wasn't silently needed.
    Confidence 70 · architectural · Fix: none required; confirm the floor still holds, or add the feature to the coverage job's invocation.
  • examine-32 (Cucumber scenario, tests/e2e-cucumber/features/examine.feature:478-482) would still pass with this PR's whole production change reverted. It plants no kfd signal at all, so old and new code both compute Some(false). It's a legitimate paired negative control (catches an "always-true" host-probe mutation) alongside examine-31, not a wasted test, but it doesn't by itself discharge "a new test is shown to fail when the code it covers is broken" the way examine-31/33/34 do.
    Confidence 85 · logic · Fix: note in the scenario's comment block that it's a documented negative control, not standalone proof of the fix.
  • machine_with_built_in_amdgpu (tests/e2e-cucumber/tests/e2e/examine_steps.rs:1579-1607) hand-rolls ROCm-install planting (.info/version + manual ROCM_PATH push into command_env) instead of reusing the existing E2eWorld::plant_unmanaged_rocm()/legacy_rocm_path helper, which isolate_cmd already applies automatically. This forces the paired When step (user_inspects_and_diagnoses_planted_machine, lines 1611-1624) to manually clone/restore world.command_env between the two CLI invocations — correct as written, just avoidable complexity.
    Confidence 85 · mechanical · Fix: call plant_unmanaged_rocm() instead.
  • New e2e step definitions push the ROCM_CLI_TEST_HOST_ROOT sentinel as a literal string (examine_steps.rs:1397,1420,1512) rather than referencing crate::hardware_root::TEST_HOST_ROOT_ENV (hardware_root.rs:70), which AGENTS.md §6 asks for ("a sentinel string gets one shared constant"). The same literal is already used identically at three pre-existing call sites in the same file, so this follows rather than introduces the file's established (non-compliant) convention.
    Confidence 70 · mechanical · Fix: reference the shared constant, file-wide, as a follow-up.

Decisions for the author

  • Field semantics kept broad (amdkfd_loaded/amdgpu_loaded mean "present," not "bound and working") rather than split into separate fields — tradeoff. The PR body states this was deliberated and a split (e.g. kfd_available) was rejected for JSON-schema compatibility. Given the Blocking finding above, a consumer that specifically wants "is the GPU actually usable" still has no field for that distinction — worth confirming this is still the intended long-term shape, independent of whether the Blocking finding is addressed now or later.

Positive signals

  • The old inline probe_modules body was fully replaced by probe_modules_from/driver_states/module_or_host_state with no leftover dead code from the previous logic.
  • module_or_host_state's three-way distinction (confirmed-absent vs. unknown-list vs. present) is deliberately preserved rather than collapsed into a single boolean, avoiding a silent-failure trap.
  • hardware_root.rs's doc-comment inventory, docs/release-trust.md, and the ci.yml coverage comment are kept in exact lockstep for the two new routed paths (/sys/class/kfd/kfd, /sys/bus/pci/drivers/amdgpu) — verified by direct cross-check, not left to drift.
  • The new field doc comments on Examination::amdgpu_loaded/amdkfd_loaded precisely state the new broadened "module or built-in" meaning for a future reader of examine.rs alone, closing a documentation gap this kind of change usually leaves open.
  • Three of the four new e2e scenarios (examine-31, -33, -34) are genuine, individually-traced regression tests confirmed (by reading, not running) to fail under the pre-PR code, including one (examine-33) that isolates the PCI-driver-only signal from the kfd signal.

Deployment notes

  • Purely additive/broadening for existing consumers: no code in diagnose.rs/fix.rs branches on module-vs-built-in, only on the false case, which this PR doesn't change.
  • Per the PR's own text, this has not been run on real built-in-kernel hardware; at review time none of the GPU hardware CI lanes (MI300X, W7900) had exercised this head commit, and the Strix Halo Ubuntu nightly/release lanes were cancelled.
  • Until the Blocking finding above is addressed, any real machine with amdgpu's driver code resident but not bound to a working GPU will have its DKMS/blacklist/Secure-Boot diagnose evidence silently suppressed.

What this covered

Full review of all 8 changed files (.github/workflows/ci.yml, crates/rocm-core/src/examine.rs, crates/rocm-core/src/examine/linux_driver.rs, crates/rocm-core/src/examine/rocm_install_probe.rs, crates/rocm-core/src/hardware_root.rs, docs/release-trust.md, tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs), each read completely, via three coupled-file passes (production probe logic; host-root infra/docs/CI; e2e feature+steps), three source-reading passes (AGENTS.md adherence; history + prior-changes against PR #510; code-comments adherence), and two targeted Step-7 design-review investigations (kfd/amdgpu binding-vs-residency plausibility, verified against upstream kernel source; downstream consumer semantics, verified by a repo-wide grep for every reader of the two JSON fields). Commits reviewed: d7faa84…e0115c89 (merge-base…PR head).

Did not run: cargo test/cargo clippy/cargo xtask coverage — no Rust toolchain in this environment; all test-quality and mutation-equivalence verdicts above are argued from reading the code and tracing assertions by hand, cross-checked against this PR's own green CI lane for the relevant unit/e2e-test-hooks tests. GitHub's global search_commits/search_pull_requests/search_issues failed in the history pass with a token/permission error; it substituted list_commits/pull_request_read-by-number and found PR #510 as the one directly relevant prior PR. Per the task's own instructions, this PR's existing reviews were listed once (logins/states/commit SHAs only, no review body text) solely to confirm HEAD had moved past the last review and avoid double-posting; no review body content was read before or during the findings passes, and this report was not reconciled against any discussion on this PR.


🤖 Generated with Claude Code

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this reads well and I'm happy to approve. No blocking findings.

I re-checked the open question from the last round (a built-in driver with no GPU bound now reads amdgpu_loaded=true). I agree with your answer in the thread: the module-list path already reports true for a modprobed driver with no device, and every consumer in diagnose.rs (lines 832, 949, 957) acts only on Some(false), so "driver present" is the right meaning here. The earlier WSL2 flake concern is resolved too.

Non-blocking, take or leave:

  • host_gpu.rs (detect_driver_summary) still uses /sys/module/amdgpu, which this PR argues is wrong under nomodeset. So examine text can say amdgpu_available while the JSON says amdgpu_loaded: false. Fine as the follow-up you already mentioned.
  • In rocm_install_probe.rs, merge_live_repo_paths_does_not_duplicate_an_already_seen_path is the third repo-scan test that goes through host_path without HOST_ROOT_TEST_LOCK. Harmless under nextest.
  • examine-33 copies the three diagnose evidence strings, but nothing ties them to diagnose.rs. A small unit test asserting them would keep the scenario from quietly going vacuous if one is reworded.

CI: the red E2E tests (Strix Halo, Windows) lane fails runtime-resolve-canonical-release, which this Linux-only change can't affect. The MI300X and W7900 lanes were still queued when I looked.


🤖 by agenthub on AMD AgentHub

amdgpu_init creates kfd's device before it registers the PCI driver, and
the PCI driver directory exists before any device is probed, so neither
host signal proves a GPU is bound. The field never meant that: a loaded
module is in lsmod with or without a GPU too, and diagnose reads only
Some(false), as a module that failed to load. Say so on the field and on
amdgpu_implied_by_host, and mark examine-32 as a negative control.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

tomastola commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

I merged main into the branch (e0115c8) after #829 split examine.rs into modules. The probe and its tests now live in examine/linux_driver.rs, and the repo-scan test locks are in examine/rocm_install_probe.rs. The code is the same as before. Neither review below has inline threads, so I'm answering both here. I pushed 5a14086 and 279738c.

The change request on a2ac4d1 (09:40 review). All three items are still fixed after the move:

  • Driver-name literals not covered (blocking): probe_modules_reports_each_driver_its_module_list_names gives probe_modules_from lists that name both drivers, only amdgpu, and only amdkfd. Your amdgpuX/amdkfdX mutations fail it, and so does the parse mutation.
  • Doc comment on the wrong item (blocking): the wiring paragraph is on probe_modules_reports_amdgpu_when_it_is_on_the_host. NO_AMD_MODULES keeps only its own two lines.
  • Repo-scan test race (non-blocking): both tests take HOST_ROOT_TEST_LOCK.

On e0115c8, cargo test -p rocm-core --lib --features e2e-test-hooks passes 960/960. This head also adds the module-list test to the description's test list. Re-requesting your review.

Review of e0115c8.

  • "kfd/the PCI driver directory prove the driver initialised, not that a GPU is bound" (blocking). The ordering is right: amdgpu_init creates kfd's device before pci_register_driver. I'm not narrowing the fields to binding, though. They report that the driver is present, and they always have: a modprobed amdgpu is in lsmod on a machine with no matching GPU too. The three diagnose consumers read only Some(false), and each one explains it as a module that failed to load: DKMS against an unsupported kernel, a blacklist, Secure Boot blocking an unsigned module. None of those can apply to a driver that is present, built in or loaded. So on a built-in kernel whose GPU fails to probe, reporting false would point the user at the wrong fix. Whether there is an AMD GPU is has_amd_gpu.
    You're right that the code didn't say this anywhere, and this is the second review that read the fields as "bound". 5a14086 states it on the amdgpu_loaded field doc and on amdgpu_implied_by_host, and the description now has a paragraph on it.
  • E2E tests (Strix Halo, Windows, current) red (blocking). The failure is not in this change. runtime-09 failed because the runner's environment carried a TheRock pip mirror override. The preview reported canonical_source: https://mirror-lax.techaccelerator.com/repository/pypi-therock, which is the cause test(e2e): keep the canonical-release preview from inheriting a runner mirror #869 describes. That lane has been green on main and on other PRs since this morning, and 5a14086 starts a fresh run.
  • Coverage floor (non-blocking). The coverage job passed on e0115c8, so the rocm-core floor holds as it is.
  • examine-32 passes with the fix reverted (non-blocking). Agreed. Its comment now says it is a negative control for examine-31, not proof of the fix. The description says the same.
  • machine_with_built_in_amdgpu hand-plants the ROCm install (non-blocking). Fixed in 279738c. The step now calls plant_unmanaged_rocm(). The When step still restores the scenario environment between examine and diagnose, because the host root and the fake lsmod/mokutil on PATH live in command_env, which each run consumes. examine-31 to examine-34 pass locally through cargo xtask e2e.
  • Literal ROCM_CLI_TEST_HOST_ROOT in the step file (non-blocking). It follows the three existing call sites in that file. Switching the whole file to the shared constant belongs in its own change.

The shared helper already points ROCM_PATH at a planted install for every
command, so the step need not hand-roll one. The scenario environment is
still restored between examine and diagnose, for the host root and the
fake executables.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 279738c

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (not narrowed to the delta since the last automated pass, because that earlier pass's own findings were not available to re-scope against without reading PR discussion, which this method does not do; the full diff was small enough — 8 files, +683/-26 — to review in full anyway). Implements the fix described in GitHub issue #448 (amdkfd_loaded false-negative when kfd is built into the kernel) and extends the same fix to amdgpu_loaded for a built-in amdgpu, which issue #448 did not report but which the PR explains is the same root cause. No Jira ticket is named in the title or body.

Blocking

None

Non-blocking

None

Decisions for the author

  • Six new host-root tests in linux_driver.rs hand-build a unique temp directory with std::env::temp_dir().join(format!("...-{pid}-{nanos}")) and manually remove_dir_all it, instead of using tempfile::tempdir() (already used elsewhere in the same file and in rocm_install_probe.rs's tests) — non-blocking-improvement. tempfile::tempdir() gives auto-cleanup and guaranteed-unique paths with less boilerplate (~10 lines × 6 call sites saved); nothing about these tests needs the manual form.
  • driver_states calls kfd_present_on_host() twice per invocation (once inside amdgpu_implied_by_host, once directly for the amdkfd field), costing up to 4 Path::exists() stats instead of 3 — non-blocking-improvement, negligible cost, not worth restructuring for a probe that runs once per examine invocation.
  • The field is named amdgpu_loaded/amdkfd_loaded but now means "driver present," not "driver bound to a GPU" — tradeoff, already addressed in the PR description itself ("Both fields report that the driver is present, not that a GPU is bound to it... Whether there is an AMD GPU is has_amd_gpu") and the name is kept deliberately for JSON-schema compatibility with a separate name (kfd_available) considered and rejected. Confirmed has_amd_gpu is a distinct, unaffected field and no consumer (diagnose.rs, the only in-repo consumer of amdgpu_loaded) conflates the two. Worth the author/maintainers confirming this is still the intended tradeoff, since any future downstream consumer reading the field name literally (without the doc comment) could misread "loaded" as "working."

Positive signals

  • Every new helper (module_or_host_state, driver_states, kfd_present_on_host, amdgpu_present_on_host, amdgpu_implied_by_host) is unit-tested against the exhaustive branch table (module-present, built-in-absent-from-list, confirmed-absent, unreadable-list×host-true/false, host-skipped-once-list-answers, kfd-implies-amdgpu-but-not-reverse), plus two "wiring" tests that specifically catch the helpers being correct in isolation but not actually wired into probe_modules_from — this is mutation-resistant test design, not just happy-path coverage.
  • The e2e negative control (examine-32) is explicitly documented in its own comment as "a negative control... not proof of the fix," preventing it from being mistaken for coverage it doesn't provide.
  • examine-33's e2e scenario plants a ROCm install + Secure Boot-enabled + built-in amdgpu specifically so the three diagnose evidence lines it asserts absent are actually reachable if the flag regresses (verified: plant_unmanaged_rocm makes rocm_version non-empty, which is required for one of the three gates to be reachable at all) — a real regression test, not a vacuous one.
  • diagnose.rs's three evidence checks were traced line-by-line and confirmed to gate on exactly amdgpu_loaded == Some(false) as the PR claims, with no other form used anywhere in that 8900+-line file, and amdgpu_loaded's doc comment in examine.rs was checked word-for-word against that gating logic.
  • hardware_root.rs's module-doc path allowlist, docs/release-trust.md, and a ci.yml comment were all updated together to name the two new host paths and the tests that cover them, and were cross-checked as mutually consistent.
  • JSON schema confirmed unchanged: both fields keep their Option<bool> type and name; the pre-existing examination_top_level_keys_match_examine_py_contract test (untouched by this PR) still enforces the full key set.
  • Commit history (22 commits) is disciplined: every commit is single-purpose and DCO-signed, and several commits show the author catching and fixing their own earlier mistakes (a scenario-ID renumbering collision, a miscounted CI comment, a non-matching evidence string) before this review ever ran.
  • A leak scan across the full diff for internal/proprietary terms came back clean.

Deployment notes

  • No migration, API-contract break, or new dependency; JSON schema is unchanged.
  • At review time, two hardware-specific E2E check runs (W7900, MI300X) were still queued/in-progress and the PR's mergeable_state was reported as blocked by GitHub; every check that had completed was green. Confirm those finish green before merging — they weren't a review finding since nothing had failed.

What this covered

Read completely: crates/rocm-core/src/examine.rs, crates/rocm-core/src/examine/linux_driver.rs, crates/rocm-core/src/examine/rocm_install_probe.rs, crates/rocm-core/src/hardware_root.rs, tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs, .github/workflows/ci.yml and docs/release-trust.md diffs, plus crates/rocm-core/src/diagnose.rs (targeted, around every amdgpu_loaded/amdkfd_loaded/has_amd_gpu use) and the root AGENTS.md for project-convention compliance. Commits d7faa84…279738ca (merge-base…head).

Run as independent fan-out workers: the core-logic pass (the four crates/rocm-core files above), the agent-instruction-adherence pass (against AGENTS.md), the history pass (git log/git blame), and the prior-changes pass (review feedback on earlier, separate PRs touching the same files — checked against PR #510, #675 and #829). The e2e-test-file pass (the .feature/examine_steps.rs group) and the code-comments-compliance pass could not be dispatched as independent workers: the shared subagent pool was persistently exhausted by unrelated concurrent sessions for the full duration of this review despite repeated retries, so both were instead carried out directly by this reviewing context — flagged here per the method's rule that a pass run by the driver itself is not an independent pass, even though its conclusions (no blocking findings) agreed with the independent passes'.

Not reconciled against any discussion on this PR — this review did not read PR comments, review threads, or any earlier review's content, only the PR's own description/issue (Step 9) and the code itself.

🤖 Generated with Claude Code

main added examine-31 (the unopenable /dev/kfd pre-flight), which took the number this branch's first scenario used. Keep main's examine-31 and renumber this branch's built-in kfd/amdgpu scenarios to examine-32..35, in declaration order.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Resolve the examine.feature conflict with #915: main's ROCDXG scenarios keep
examine-32..34 (referenced from docs/testing.md and model_serving.feature),
and this branch's kfd/amdgpu scenarios move to examine-35..38.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola force-pushed the fix-examine-kfd-builtin branch from f952395 to 9a03d73 Compare November 6, 2026 12:54
@tomastola
tomastola requested a review from nowycondro November 6, 2026 13:13
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

All three items from siloteemu's 10-30 review are fixed on this branch:

  • Unguarded driver-name literals in driver_states: 7e406af adds tests that name amdgpu and amdkfd in the module list and assert each field.
  • Doc comment on the wrong item: the "wiring" paragraph and the NO_AMD_MODULES doc now sit on their own items in linux_driver.rs.
  • Unit-test race: 6c8661c takes HOST_ROOT_TEST_LOCK in the repo-scan tests (50224ab updates the CI comment count).

Main is merged in (2ba18c0). The CI workflow is green; the self-hosted E2E lanes (run 42604993736) are still queued/running. Re-requesting review.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 2ba18c0

Outcome: needs work (one blocking finding). The core fix itself (treating a built-in amdgpu/kfd as "loaded" via /sys/bus/pci/drivers/amdgpu, /sys/class/kfd/kfd, /dev/kfd) was traced through by hand across every call path and all 12 new unit tests, and holds up — no defect found there.

Blocking

  • Missing test-lock guard leaves a data race on TEST_HOST_ROOT_ENV — crates/rocm-core/src/examine/rocm_install_probe.rs:309-324 (merge_live_repo_paths_does_not_duplicate_an_already_seen_path).
    This PR adds a HOST_ROOT_TEST_LOCK guard to two sibling tests in the very same file/commit (live_rocm_repo_paths_in_dir_skips_a_real_bak_file_on_disk, merge_live_repo_paths_scans_every_family_dir_including_zypp) because they call host_path indirectly and must not race against another test re-rooting TEST_HOST_ROOT_ENV. merge_live_repo_paths_does_not_duplicate_an_already_seen_path calls the identical path (merge_live_repo_paths → live_rocm_repo_paths_in_dir → crate::host_path) but was left unguarded. It's a plain #[test] (not feature-gated), so under cargo test --features e2e-test-hooks with the default multi-threaded runner it can race against any of the new host-root tests in linux_driver.rs.
    Fix: add the same HOST_ROOT_TEST_LOCK guard used in the two sibling tests.
    Confidence 85 · logic.

Non-blocking

  • PR description cites stale scenario numbers: it says the new scenarios are examine-31..examine-34, but the committed tests/e2e-cucumber/features/examine.feature numbers them examine-35..examine-38 (the @id: tags are correct and unique; only the human-readable index drifted, apparently from a main-branch rebase after the description was written). A reader following "examine-31" lands on an unrelated pre-existing scenario. Worth a quick fix in the description.
  • assert_no_amdgpu_fault_evidence (examine_steps.rs) hardcodes three evidence strings copied from diagnose/linux.rs's check_3_rocm_kernel_unsupported/check_5_amdgpu_blacklisted with no compile-time link; verified they match today, but a future wording change there would make this e2e assertion pass vacuously with no signal. Consider a shared constant, or at least a comment pointing back from the diagnose-side strings to this test.

Decisions for the author

  • Built-in driver masks a DKMS-specific failure — tradeoff. With amdgpu built into the kernel, diagnose's "DKMS failed" / "amdgpu module not loaded" / "Secure Boot blocked amdgpu" evidence (keyed on amdgpu_loaded == Some(false)) now stays silent even if a separately-installed out-of-tree amdgpu-dkms package failed, because the in-tree driver still satisfies amdgpu_loaded. This is consistent with the field's own doc comment ("reports the driver is present, not that a GPU is bound to it... none of those can stop a driver that is present"), so it reads as an intentional scope narrowing rather than an oversight — flagging so it's confirmed as accepted rather than silently assumed.

Positive signals

  • amdgpu_present_on_host's doc comment explains precisely why /sys/module/amdgpu is the wrong signal (created at an early initcall before amdgpu_init can bail out), and a dedicated test plants that directory alone to prove it doesn't count — directly guarding against re-introducing a mistake the branch's own history shows was made once already.
  • The module-list-already-answered short-circuit is tested via an unreachable!() closure, catching an accidental operand swap that would otherwise only show up as a perf regression.

Deployment notes

  • GitHub currently reports this PR as mergeable: CONFLICTING against main (likely just needs a rebase) — not a code defect, but worth resolving before merge.

What this covered: the full diff (base 17f655d3…head 2ba18c0, 8 files, +684/-27), every changed file read in full by four independent parallel passes (core probe logic; downstream diagnose/fix consumers and impact analysis; e2e feature/step-def tests; AGENTS.md adherence + docs/CI cross-check + commit history), plus my own direct verification of the highest-signal candidates. All CI checks for this commit report SUCCESS except one hardware lane still QUEUED. Not reconciled against this PR's existing review threads.

Not a human review. This is an automated code review; please verify findings independently before acting on them.

main added examine-35..38 (the rocminfo PATH scenarios), which took the numbers this branch's built-in kfd/amdgpu scenarios used. Keep main's and renumber this branch's to examine-39..42, in declaration order.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Reply to the 11-09 review (head 2ba18c0); it has no inline threads, so answering here.

  • Blocking, missing lock in merge_live_repo_paths_does_not_duplicate_an_already_seen_path: agreed, fixed in ffe1098 — it now holds HOST_ROOT_TEST_LOCK like its two siblings.
  • Stale scenario numbers in the description: fixed. Main took examine-35..38 for the rocminfo PATH scenarios, so after merging main (40e8d19) this branch's scenarios are examine-39..42, and the description now says so.
  • Evidence strings copied from diagnose: not changing here. The three strings are verbatim and examine-41 plants the ROCm install and Secure Boot so all three lines stay reachable; a shared constant would mean editing diagnose for a test-only link, so I'd rather keep that out of this PR. Happy to do it as a follow-up.
  • Built-in driver masks a DKMS-specific failure: confirmed as intended. amdgpu_loaded means the driver is present, per its doc comment; a failed out-of-tree DKMS build next to a working in-tree driver is not what examine: amdkfd_loaded=false is a false negative when kfd is built into the kernel (no module in lsmod) #448 is about.
  • Conflict with main: cleared by the merge above (only examine.feature conflicted, on scenario numbers).

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head ffe1098d

Review — needs work

Full review of the whole change (all 8 files in the PR's diff against main@2d9b5a1c, including the full new test content and production probe logic), not a narrowed follow-up: although I (this same account) reviewed an earlier commit of this PR, the gap to this head touches real content, so this pass re-derives everything from the diff/source rather than reusing prior conclusions.
Implements the fix for #448 (amdkfd_loaded/amdgpu_loaded false negatives when the driver is built into the kernel) — the PR's own description matches the linked issue and the actual diff; no scope mismatch found.

Blocking

  • assert_no_amdgpu_fault_evidence hardcodes production evidence strings with no shared constant tying test to source — tests/e2e-cucumber/tests/e2e/examine_steps.rs (the three literals "the amdgpu kernel module is not loaded", "amdgpu module is not loaded", "Secure Boot is enabled and amdgpu didn't load"), duplicated from crates/rocm-core/src/diagnose/linux.rs:27,143,151.
    The in-repo AGENTS.md states directly: "A value matched across files (a version tag, a sentinel string) gets one shared constant and a test tying the producer to the consumer." These three strings are matched across exactly two files with nothing but a comment ("each copied verbatim from that check so it keeps guarding it") holding them together. If diagnose/linux.rs's wording is edited later (a typo fix, a clarity pass), this regression guard silently stops matching anything — no compile error, no test failure anywhere flags the drift — and the examine-41 scenario this guards would pass regardless of whether amdgpu_loaded actually regressed.
    Confidence 85 · architectural · Fix: pull the three fragments into pub(crate) constants beside check_3_rocm_kernel_unsupported/check_5_amdgpu_blacklisted in diagnose/linux.rs, and have the test reference those constants instead of re-typing the text.

  • Both the PR description and a new code comment misidentify the negative-control scenario — PR description "Tests" section ("examine-32 is a negative control..."); tests/e2e-cucumber/features/examine.feature:612 ("# A negative control for examine-35, not proof of the fix: ...").
    The actual negative control added by this PR is examine-40 ("The inspection reports kfd as not loaded on a machine without it"), controlling against examine-39 ("...reports kfd on a machine whose module list has no amdkfd"), both new in this diff. examine-32 is an unrelated pre-existing scenario ("ROCDXG ready when the loader reaches it through LD_LIBRARY_PATH") and examine-35 is also unrelated and pre-existing ("Both reports find a rocminfo that sits in a PATH directory") — verified directly against examine.feature. Both wrong numbers appear to be leftovers from scenario renumbering across this PR's many revisions. This is a description/comment-accuracy defect (the PR's own account of its test coverage is incorrect), not a functional bug, but it affects a maintainer's ability to verify the claimed coverage without independently re-deriving it.
    Confidence 100 · mechanical · Fix: replace "examine-32" in the PR description and "examine-35" in the feature-file comment with "examine-39" (and clarify the comment refers to being a negative control for examine-39, proven by examine-40).

Non-blocking

  • docs/release-trust.md's "Simulated Host Root" paragraph omits the LD_LIBRARY_PATH clause of the WSL ROCDXG check — docs/release-trust.md:329-333 vs. crates/rocm-core/src/hardware_root.rs:27-29, which documents the same check as also looking for lib/librocdxg.so "again in each directory on LD_LIBRARY_PATH". This PR edits this exact paragraph (to add the new kfd/amdgpu paths) but the gap itself pre-dates this change and isn't introduced or worsened by it. Flagging because of the direct pattern relevance (same paragraph, same "exhaustive list" claim) and because the doc explicitly presents itself as describing "every one of these reads, and only these."
    Confidence 85 · mechanical · Fix: add the LD_LIBRARY_PATH clause to release-trust.md to match hardware_root.rs, ideally in the same change that next touches this paragraph.

Decisions for the author

  • Driver-presence vs. GPU-bound, under namespace/mount sharing — tradeoff. The new amdgpu_implied_by_host/amdgpu_present_on_host checks (and the pre-existing lsmod-based ones they extend) report the driver's presence, not whether a GPU is bound to it, by design (documented in both the Examination struct doc comment and the PR description). This is consistent with existing behavior for loaded modules, but widens the specific signal to /sys/bus/pci/drivers/amdgpu: in an unusual setup where a container has /sys bind-mounted (read-only) from the host but no /dev/kfd/GPU passthrough, this would read amdgpu_loaded: true for a container with no real driver access. The PR text already reasons carefully about the symmetric case (kfd passed through without /sys/bus/pci); confirm this asymmetric case was considered and is an accepted limitation consistent with the field's documented meaning, not an oversight.

Positive signals

  • module_or_host_state's Option<bool> handling is precise and consistently threaded through driver_states and probe_modules_from: None (list unreadable) only resolves to Some(true) via the host signal and otherwise stays None, never incorrectly collapsing to Some(false) — verified by reading the implementation and its unit tests together.
  • rocm_install_probe.rs's added HOST_ROOT_TEST_LOCK guards fix a real, non-cosmetic hazard: live_rocm_repo_paths_in_dir/merge_live_repo_paths pass their own tempdir paths through host_path, which a concurrently-running e2e-test-hooks test setting TEST_HOST_ROOT_ENV could otherwise re-root out from under them.
  • The asymmetric design (kfd implies amdgpu; amdgpu does not imply kfd) is correctly carried through driver_states, matches real kernel linkage (amdgpu_init creates kfd's device before registering the PCI driver), and is exercised by tests that would fail if the two checks were swapped.

Deployment notes

None — the JSON schema for amdgpu_loaded/amdkfd_loaded is unchanged (same field names and Option<bool> type), only the circumstances under which each resolves to true/false/unknown change, as stated in the PR description.

What this covered

Read in full: crates/rocm-core/src/examine.rs, crates/rocm-core/src/examine/linux_driver.rs (including every new helper and its unit tests), crates/rocm-core/src/hardware_root.rs, crates/rocm-core/src/examine/rocm_install_probe.rs, docs/release-trust.md, .github/workflows/ci.yml (the touched job/comment), tests/e2e-cucumber/features/examine.feature, and tests/e2e-cucumber/tests/e2e/examine_steps.rs (every new step function plus the helpers they call: run_rocm_with_scenario_env, parsed_json, plant_gpu_fixtures, put_fake_executables_on_path), diffed against merge-base 2d9b5a1c with main. Also read: linked issue #448, crates/rocm-core/src/diagnose/linux.rs (to verify the evidence-string claims and the matched/MIN_SCORE_FOR_MATCH semantics so the examine-41 assertion isn't vacuous — it isn't: a sub-threshold match still carries its evidence text into matched), and the project's AGENTS.md. Live CI at the reviewed head: all required checks green; two self-hosted hardware lanes (E2E tests (W7900, current), E2E tests (MI300X, current)) were still queued at review time, not failing.

Used three independent fan-out passes grouped by file coupling (core probe logic; docs/CI/install-probe test hygiene; e2e feature+steps), then verified and re-derived every candidate finding directly against the source myself before scoring — one reported finding (the docs omission) was reported by a worker with a different framing and is reflected here after independent confirmation; the other two were independently re-derived and, in the scenario-numbering case, found to be broader (also present in the PR description, not just the code comment) than initially reported. Not reconciled against the PR's review-comment discussion/history (by design — only the description and the linked issue were read, per method).


Not a human review. This is an automated code review; please verify findings independently before acting on them.

The three phrases examine-41 asserts the diagnosis does not cite were copied into
examine_steps.rs. diagnose/linux.rs now builds those lines from shared constants,
exports them as AMDGPU_MISSING_EVIDENCE for the step to use, and a unit test
checks every phrase is in what the checks produce. Also point the negative-control
comment at examine-39 and add the LD_LIBRARY_PATH clause to the simulated host
root paragraph in release-trust.md.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Reply to the 11-10 review (head ffe1098); no inline threads, so answering here. Pushed 8f678e2.

  • Blocking, evidence strings without a shared constant: you're right, AGENTS.md section 3 settles it and I was wrong to decline last round. diagnose/linux.rs now holds the three phrases as constants, builds the check_3 and check_5 evidence lines from them, and exports AMDGPU_MISSING_EVIDENCE. The assert_no_amdgpu_fault_evidence step uses that export instead of re-typing the text. Two unit tests tie producer to consumer: every phrase appears in the evidence the checks produce when amdgpu_loaded is Some(false), and none appears when it is Some(true).
  • Blocking, wrong scenario numbers: fixed. The comment in examine.feature now says examine-39, and the description says examine-40 is the negative control for examine-39.
  • Non-blocking, LD_LIBRARY_PATH clause missing from release-trust.md: added, to match hardware_root.rs.
  • Decision, a container with /sys bind-mounted from the host but no GPU passthrough: I did consider it. It reads amdgpu_loaded: true, which is consistent with the field's documented meaning (driver present, not a GPU bound to it), and the host really does run the driver. I'm accepting it as a known limitation and not adding a guard.

Locally: the 192 diagnose unit tests pass, clippy and fmt are clean, and the e2e crate compiles. I did not run the cucumber scenarios.

@github-actions

Copy link
Copy Markdown
Contributor

Caution

9 potential secret finding(s) were committed in this pull request. Treat each credential as compromised and revoke or rotate it now, even if a later commit removed it. Deleting the line or force-pushing does not undo the leak: the commits stay reachable from this pull request and from every clone or fork, and automated scanners harvest secrets pushed to public repositories within minutes.

If a finding is a test fixture or a false positive, allowlist it in the configuration of the scanner that reported it.

Security scan found 9 critical or high finding(s) on lines changed by this pull request
Scanner Severity Rule Location Finding
Gitleaks HIGH generic-api-key apps/rocm/src/runtime_install_finalize.rs:1315 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/runtime_install_finalize.rs:2685 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/runtime_install_finalize.rs:2692 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:335 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:335 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:363 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:363 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:521 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:549 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 8f678e2c

Review — no blocking findings

Review of changes since the review at ffe1098d (this same account's most recent prior pass), narrowed to the one follow-up commit (8f678e2c) and the 5 files it touches, read in full at the current head: crates/rocm-core/src/diagnose.rs, crates/rocm-core/src/diagnose/linux.rs, docs/release-trust.md, tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs. Narrowing was established by diffing ffe1098d...8f678e2c directly (one commit, 5 files, ~90 lines) and confirming both of that review's open Blocking findings, plus its one Non-blocking finding, are addressed by this diff — see below. The rest of the PR (production probe logic in examine.rs/linux_driver.rs/hardware_root.rs, .github/workflows/ci.yml) is unchanged since that full pass and was not re-read.
Implements the fix for #448 (amdkfd_loaded/amdgpu_loaded false negatives when the driver is built into the kernel) — the PR description matches both the linked issue and the diff; no scope mismatch found.

Blocking

None.

Both of the previous round's Blocking findings are resolved by this commit:

  • The hardcoded evidence-string triplet in assert_no_amdgpu_fault_evidence (tests/e2e-cucumber/tests/e2e/examine_steps.rs) now references a single shared pub const AMDGPU_MISSING_EVIDENCE: [&str; 3] defined in crates/rocm-core/src/diagnose/linux.rs and re-exported from diagnose.rs, exactly matching AGENTS.md's "a value matched across files gets one shared constant and a test tying the producer to the consumer" rule. The three named constants (AMDGPU_KERNEL_MODULE_NOT_LOADED, AMDGPU_MODULE_NOT_LOADED, SECURE_BOOT_BLOCKED_AMDGPU) are consumed directly by check_3_rocm_kernel_unsupported/check_5_amdgpu_blacklisted via format!, so the producer and the array can no longer drift independently, and two new unit tests exercise both directions (every phrase present when amdgpu_loaded == Some(false), none present when Some(true)).
  • The stale "examine-35" cross-reference in the examine.feature comment is now "examine-39", matching the PR description, which already read correctly.
  • (Non-blocking previously) docs/release-trust.md now documents the LD_LIBRARY_PATH clause of the ROCDXG check, matching hardware_root.rs.

Non-blocking

None.

Decisions for the author

None.

Positive signals

  • The new AMDGPU_MISSING_EVIDENCE array is consumed by the same named constants the checks format their evidence strings from, not a second hand-copied literal — the exact class of drift the prior finding warned about is now structurally prevented, not just patched once.
  • pub const AMDGPU_MISSING_EVIDENCE's re-export from diagnose.rs mirrors the existing dkms_driver_plan_is_supported export on the same line, keeping the module's public surface pattern consistent rather than introducing a one-off.

Deployment notes

None — no schema or behavior change in this follow-up commit; it only ties a test to production constants and corrects comment/doc text.

What this covered

Full read of the 5 files changed between ffe1098da01de304be50a9ba6bdb70236ad62874 and 8f678e2c33017575725dcc485d4ad98fb808570b, plus surrounding context (Examination struct, hardware_root.rs's documented host-path list, AGENTS.md, issue #448). Used three independent fan-out workers grouped by coupling: production code (diagnose.rs + diagnose/linux.rs), e2e/docs (examine.feature + examine_steps.rs + release-trust.md), and the 4b source-reading passes (agent-instruction adherence, history, prior-pattern, code-comments) across all 5 files. One candidate from the production-code worker (no compile-time length guard against someone shrinking AMDGPU_MISSING_EVIDENCE below 3 elements) was verified and dropped: the array's [&str; 3] type annotation already makes such a shrink a compile error unless the size is also edited, so the scenario isn't silent. No worker reported anything else scoring ≥70.

Live CI at the reviewed head: every required check is green (gh pr checks --required). Two optional, non-required jobs (Security scan / gitleaks, Security scan / Security scanners) are failing, but on inspection the flagged secrets are in apps/rocm/src/runtime_install_finalize.rs and apps/rocm/src/serve_summary.rs from commits (#752, #798) already on main well before this branch's recent work and untouched by this PR's diff — pre-existing and unrelated, not a finding here.

Did not run: no Rust toolchain was available to any fan-out worker, so no finding reached confidence 100 by execution; all were traced/verified by reading. One worker (4b passes) did a full-file read of examine.feature/the diff region but not a complete top-to-bottom read of examine_steps.rs's 2766 lines for its comment-compliance pass specifically — a second, independent worker did read that file fully for the primary correctness pass, so coverage of the file itself is not a gap, only the narrower comment-pass's scope on it. Not reconciled against the PR's review-comment discussion/history (by design — only the description and linked issue were read).


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola
tomastola dismissed siloteemu’s stale review November 10, 2026 12:42

Thanks for the review. Your points are addressed: the driver-name coverage in 7e406af, the doc comments now sitting on their own items, and the host-root test locks in 6c8661c and ffe1098. The PR will proceed with the other reviewers; a re-review is welcome any time.

@tomastola
tomastola removed the request for review from siloteemu November 10, 2026 12:42
@tomastola
tomastola added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit a09c920 Nov 10, 2026
39 of 41 checks passed
@tomastola
tomastola deleted the fix-examine-kfd-builtin branch November 10, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

examine: amdkfd_loaded=false is a false negative when kfd is built into the kernel (no module in lsmod)

5 participants