Repository navigation
fix(examine): report amdkfd_loaded and amdgpu_loaded when built into the kernel - #694
Conversation
0b97f7b to
88a535e
Compare
jussielo-amd
left a comment
There was a problem hiding this comment.
This review was produced using Claude Code's review-code method (independent fan-out over the core probe logic, the e2e feature/steps, and AGENTS.md/impact compliance), at the request of @jussielo-amd. No GitHub approval is filed by this account for automated reviews — the merge decision stays with a human reviewer.
Blocking
- CI: "E2E tests (Strix Halo, WSL2, current)" is failing on the current head (
7f9c703). The suite's own reconciliation step reports:FAIL: 'comfyui-source-download-shows-live-progress' was expected to pass on this host but FAILED — a regression.That scenario lives incomfyui.featureand isn't touched by this PR's diff (examine.rs,examine.feature,examine_steps.rs). Issue #580 ("comfyui-04 times out when managed Python bootstraps inside its 30s spinner budget") describes a similar timing flake in the same area on this job class, so this looks like a pre-existing flake rather than something this PR broke. Requesting changes mainly to get a rerun/confirmation before merge rather than because the diff itself looks wrong here — happy to withdraw once the WSL2 job is green (or confirmed as an unrelated, already-tracked flake).
Non-blocking
crates/rocm-core/src/hardware_root.rs's module doc comment says it enumerates "every one of these reads, and only these" that go through the mockablehost_path, but doesn't list the new/sys/class/kfd/kfdread added bykfd_present_on_host()inexamine.rs. The read itself is correctly routed throughhost_path; this is just a stale enumeration. Suggest adding it alongside the existing topology-path mention.amdgpu_state/amdkfd_stateinexamine.rsare structurally identical modulo the literal module name and the host-presence bool — could collapse into one generic helper to remove the duplication.- The new e2e step definitions in
tests/e2e-cucumber/tests/e2e/examine_steps.rsduplicate the existing fake-executable/PATH-prepend idiom thatplant_gpu_fixturesalready implements (now three near-identical copies in this file). A small shared helper would remove the duplication. - Pre-existing, pattern-relevant: the new
/sys/class/kfd/kfd,/dev/kfd, and/sys/module/amdgpuchecks usePath::exists(), which doesn't distinguish "absent" from "I/O error / permission denied" — same as every otherhost_path(...).exists()check already in this file. Not introduced by this change, and unlikely to matter in practice since/sys//deventries are normally world-traversable, but noting it since the new code extends the same pattern. - Minor: the pre-existing
e.kfdfield computation and the newkfd_present_on_host()each independentlystat/dev/kfdthroughhost_path— harmless but redundant.
Positive signals
- The full tri-state branch set for
amdgpu_state/amdkfd_stateis covered by unit tests, and a mutation check (flipping||to&&) correctly broke 2 of 3 tests — a real regression guard, not a tautological one. - The e2e scenario asserting
diagnose --jsonomits the DKMS/"module not loaded"/Secure-Boot evidence lines traces cleanly todiagnose.rs's three gates, all keyed onamdgpu_loaded == Some(false)— reverting the fix would reintroduce exactly those lines. - Doc comments on the new fields/helpers accurately reflect real Linux kernel Kconfig/
module_parambehavior for built-in vs. loadable drivers.
Full diff reviewed: 03cdbcc (base) → 7f9c703 (head), all three changed files read completely.
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed the fix for #448. The probe logic for amdgpu_loaded/amdkfd_loaded correctly folds in a host-presence check (sysfs class device / /dev/kfd for kfd, /sys/module/amdgpu for amdgpu) alongside the existing lsmod check, and preserves the three-state semantics: a readable module list ORs module-presence with host-presence, while an unreadable list still reports true if the host shows the driver active, and otherwise stays unknown rather than collapsing to a false negative. Cross-checked against diagnose.rs's fault-evidence checks, which gate strictly on == Some(false), so this directly fixes the false DKMS/Secure-Boot/module-missing evidence the linked issue reported, with no risk of masking a genuine negative.
Extending the same fix to amdgpu_loaded (not explicitly named in #448) is the right call — it's the identical root cause, and examine-31 adds dedicated e2e coverage that diagnose no longer cites those evidence lines for a built-in amdgpu. Keeping the field name amdkfd_loaded rather than splitting it is also reasonable: downstream consumers only care whether the driver is active, not whether it was loaded as a distinct module.
One non-blocking nit inline. CI is green aside from two unrelated, pre-existing flaky lanes (a comfyui download-timing scenario and the known Strix Halo Windows PATH-apostrophe/download-infra flake) — neither touches this diff.
nowycondro
left a comment
There was a problem hiding this comment.
Automated code review (medium effort, recall-biased). 7 findings posted inline, ranked roughly by severity: two correctness issues in the new built-in-driver detection logic (a /sys/module/amdgpu-exists check that doesn't confirm the driver actually bound, and amdgpu_state/amdkfd_state not cross-checking each other despite kfd being linked into amdgpu), one cross-file consistency gap (the diagnose/fix remedy flows still assume lsmod-module semantics), and four cleanup items (reuse of an existing host_gpu.rs check, a doc-comment invariant in hardware_root.rs that's now stale, and two minor simplification/efficiency notes).
|
🤖 Automated review (Claude Code) · 7f9c703 This supersedes/clarifies the review posted moments ago (pullrequestreview-5655465252) to match the required report format; the 7 inline findings on that review stand as the substantive output of this pass. Scope read: full diff base Blocking (confidence: medium)
Non-blocking
Not verified: build, unit tests, e2e suite (no toolchain in this sandbox); CI status on the current head (see the other human/automated reviews on this PR for that signal). 🤖 Generated with Claude Code |
|
Pushed 6891205 for the 16:30 review. Fixed
Not changed
The WSL2 failure is |
6891205 to
47d8a2a
Compare
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 47d8a2a
Review
Full review of the whole change (crates/rocm-core/src/examine.rs, crates/rocm-core/src/hardware_root.rs, tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs), base a9e9aa2…head 47d8a2a. The PR explicitly fixes issue #448 (amdkfd_loaded false negative) and extends the same fix to amdgpu_loaded for the same root cause; the PR body states that extension plainly, so it is not scope creep.
Blocking
- The one genuinely new behavior this PR adds — "kfd on the host implies amdgpu" — has no test that exercises it.
crates/rocm-core/src/examine.rs:1863-1865computese.amdgpu_loadedasmodule_or_host_state(.., "amdgpu", || amdgpu_present_on_host() || kfd_present_on_host()). Thekfd_present_on_host()disjunct is new and is called out in both the inline comment above it and in commit0ae8927a's message ("kfd is linked into amdgpu, so kfd on the host now also answers amdgpu_loaded"). Every test that assertsamdgpu_loaded == Some(true)instead plants/sys/module/amdgpudirectly: the unit testprobe_modules_reports_amdgpu_when_it_is_on_the_host(examine.rs:6278-6300) and e2e scenario examine-32 viamachine_with_built_in_amdgpu(tests/e2e-cucumber/tests/e2e/examine_steps.rs:1520-1549, asserted byassert_amdgpu_loadedat line 1574). No unit test or e2e scenario plants only a kfd signal (/dev/kfdor/sys/class/kfd/kfd) with noamdgpumodule entry and no/sys/module/amdgpu, and checks thatamdgpu_loadedbecomestruethrough that path alone. Deleting|| kfd_present_on_host()from that closure would not fail a single test in the suite.
Confidence 85 · logic · Fix: add a unit test (and ideally an e2e scenario, e.g. a container-shaped host that passes through/dev/kfdbut has no/sys/module/amdgpu) that plants kfd-only presence and assertsamdgpu_loaded == Some(true).
Non-blocking
amdgpu_loaded's doc comment understates its own logic.examine.rs:228-231says it is true when "listed as a module, or/sys/module/amdgpuexists" — it omits the third signal the implementation now ORs in (kfd_present_on_host(), line 1864). The siblingamdkfd_loadeddoc comment is fully accurate about its own logic; this one is not.
Confidence 85 · mechanical · Fix: mention the kfd-on-host fallback in the doc comment.amdgpu_present_on_host()(examine.rs:1821-1823, checkinghost_path("/sys/module/amdgpu").exists()) duplicates logic already inhost_gpu.rs::detect_driver_summary()(crates/rocm-core/src/host_gpu.rs:2449-2456), which independently checks the same path (and already separately folds in/dev/kfdfor itsdetailtext). Two independent "is amdgpu present on this host" checks now exist in the crate.
Confidence 70 · logic (reuse) · Fix: consider sharing one helper between the two call sites as a follow-up; not required for this PR's scope.- The new ~140-line e2e fixture block duplicates the file's existing "write a fake executable, prepend it onto PATH" idiom (
plant_machine_without_amdkfd_moduleatexamine_steps.rs:1463, and the equivalent block insidemachine_with_built_in_amdgpuat line 1520) rather than reusing it — a pre-existing instance of the idiom plus these two new ones means it's now written out three times in this file.
Confidence 70 · mechanical (reuse) · Fix: extract a shared "plant fake lsmod/mokutil on PATH" helper now that the pattern recurs a third time. - The new fixture block has no section-divider comment, unlike the rest of the file's established convention, and currently reads as though it belongs to the preceding "release file" section (
gpu_machine_with_unreadable_release_file,examine_steps.rs:1445) rather than being its own section.
Confidence 70 · mechanical · Fix: add the divider comment used elsewhere in the file. - Pre-existing, pattern-relevant:
diagnose.rs(:853,:978) andfix.rs(:333,:369) still print user-facing remediation strings of the form"lsmod | grep amdgpu && rocminfo | head -n ..."— the exact false-negative class this PR fixes in detection (a built-inamdgpunever appears inlsmod) still exists in the remediation text a user is told to run to confirm the driver is present. Not touched by this PR's four changed files, so out of scope here, but the same bug class and worth a follow-up.
Confidence 70 · logic · Fix: a future change to those verify strings, e.g. checking/sys/module/amdgpuinstead of/in addition tolsmod. - Commit
0ae8927ais labeledrefactor(examine): …but is not a pure refactor — it adds thekfd_present_on_host()disjunct toamdgpu_loaded's computation, which is new semantics, not a reshuffle, as its own commit message's second sentence states. The behavior is otherwise fully described in the message, documented in the doc comments, and intentional; this is a labeling nit only.
Confidence 70 · mechanical · Fix: none needed; note for future commit-type labeling.
Decisions for the author
- Naming tradeoff, already made and documented — the PR keeps the field name
amdkfd_loadedfor JSON-schema compatibility rather than splitting it into e.g.amdkfd_module_loaded/kfd_functionalas issue #448 suggested as an alternative. The PR body states this explicitly and gives the reason (schema stability). Reasonable; flagging only so the choice is visible as deliberate rather than overlooked. - Historical continuity worth a one-line mention — PR #434 (merged earlier) explicitly observed and recorded
amdkfd_loaded: falsealongsidehas_amd_gpu: trueon an MI300X host as a known, unresolved quirk at the time ("the MI300X host confirms it, reportingamdkfd_loaded: false"). This PR now fixes exactly that quirk but doesn't reference #434. Not a code defect — purely a continuity nicety for anyone searching history later.
Positive signals
module_or_host_state's unit tests (examine.rs~6186-6260) cover the full tri-state matrix cleanly: module listed, built-in-and-absent-from-the-list, neither-present, unreadable-list-follows-host, and host-probe-skipped-when-the-list-already-answers (that last one uses|| unreachable!()as its own proof).hardware_root.rs's module doc comment, which claims to enumerate "every one of these reads, and only these" through the mockablehost_path, was correctly updated to add the new/sys/class/kfd/kfdread — this invariant is accurate as of this PR.- E2e scenario examine-32 does real impact-analysis work: it drives both
examine --jsonanddiagnose --jsonagainst the same planted host and asserts the diagnose evidence lines are actually absent, not just that the flag readstrue— closing the loop to the real downstream consumer rather than stopping at the probe. - Clean 7-commit history: each commit is independently meaningful, DCO-signed, and a numbering collision in the e2e scenario IDs was proactively caught and fixed in its own dedicated commit (
47d8a2aa) rather than left for review to find.
Deployment notes
- Behavioral change, not just a bugfix in the abstract: on hosts where
amdgpu/kfd are compiled into the kernel (increasingly common on stock distro kernels),rocm examine --json'samdgpu_loaded/amdkfd_loadedwill now reporttruewhere they previously reportedfalse, androcm diagnosewill stop emitting its DKMS-failure / module-not-loaded / Secure-Boot evidence lines for such hosts. JSON field names and types are unchanged (confirmed by readingExamination's struct definition), so this is a value-level behavior change, not a schema change. - As of this review (live-checked against head
47d8a2aa), no check had failed, but several E2E lanes (MI300X, Strix Halo Ubuntu release/nightly, W7900) were stillQUEUED/not finished — full CI had not completed.
What this covered
Read all four changed files in full at the PR head, plus the consumers of the changed fields (diagnose.rs, fix.rs, host_gpu.rs) and the full AGENTS.md, via three independent blind passes (core probe logic + consumers, e2e feature/steps, and a cross-cutting AGENTS.md/impact-analysis/history/commit-quality pass), then consolidated and independently re-verified the load-bearing claims directly against the source (the untested-branch claim, the doc-comment gap, the host_gpu.rs duplication, the stale lsmod remedy strings, the refactor-commit behavior change, and the PR #434 history) before scoring. Base a9e9aa2, head 47d8a2a, re-confirmed live immediately before publishing this review.
Did not run: cargo test/cargo clippy locally in this sandbox — reasoned from reading the code and test matrix directly instead, and cross-checked against the PR's own CI, where the rust/clippy/build-and-test/coverage checks are green on this head (several GPU-hardware E2E lanes were still queued, not failed, at review time). The "prior changes" pass (whether review comments on earlier, unrelated PRs touching these same files still apply here) was not separately run. Issue #448 was fetched directly (gh issue view) and checked against the diff and PR description — the PR accurately describes its own scope, including the deliberate extension beyond what #448 asked for.
Not reconciled against any prior review thread on this PR — this review was produced blind to that discussion by design.
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 3266af4
Review — needs work
Full review of the whole change. The PR fixes issue #448 (amdkfd_loaded false negative when kfd is built into the kernel) and deliberately extends the same fix to amdgpu_loaded, stating that extension explicitly in its own body — not scope creep.
Blocking
-
The "kfd present on the host implies amdgpu is loaded too" branch has no test that exercises it —
crates/rocm-core/src/examine.rs:1866-1868computese.amdgpu_loadedasmodule_or_host_state(modules, "amdgpu", || amdgpu_present_on_host() || kfd_present_on_host()). Thekfd_present_on_host()disjunct is new, deliberate behaviour (the inline comment atexamine.rs:1864-1865and the commit introducing it both say so explicitly). I traced every test touchingamdgpu_loaded: the unit tests formodule_or_host_statepass synthetic|| true/|| falseclosures, never the real composed one;amdgpu_present_on_host_reads_the_pci_driver_dirandprobe_modules_reports_amdgpu_when_it_is_on_the_host(examine.rs:6249,~6300) plant only/sys/bus/pci/drivers/amdgpu, never kfd alone; e2e scenario examine-32 (machine_with_built_in_amdgpu,tests/e2e-cucumber/tests/e2e/examine_steps.rs:1520) plants the PCI driver dir directly and never plants/dev/kfd/kfd sysfs; examine-30/31 plant a fakelsmodthat already listsamdgpuas a module, somodules.contains("amdgpu")answers before the host closure is ever reached, and neither scenario assertsamdgpu_loadedat all. Deleting|| kfd_present_on_host()fromexamine.rs:1867would not fail a single existing test.
Confidence 90 · logic · Fix: add a unit test and/or e2e scenario that plants kfd-only presence (/dev/kfdor/sys/class/kfd/kfd) with noamdgpumodule entry and no/sys/bus/pci/drivers/amdgpu, and assertsamdgpu_loaded == Some(true)through that path alone. -
The PR description no longer matches the shipped code. The body still says
amdgpu_loadedis true when "amdgpuis in the module list or/sys/module/amdgpuexists" — the final commit on this PR (fix(examine): take a built-in amdgpu from the PCI driver, not /sys/module) deliberately replaced that check with/sys/bus/pci/drivers/amdgpu(examine.rs:1829-1833) specifically because/sys/module/amdgpuis created even when the driver's init aborted undernomodeset/amdgpu.modeset=0— the description now describes the behaviour its own author judged unreliable. The description also never mentions thekfd_present_on_host()fallback inside the amdgpu computation (examine.rs:1867) at all. The Tests section namesamdkfd_state/amdgpu_state(examine.rs, Tests paragraph) — functions that no longer exist, having been unified intomodule_or_host_stateby an earlier commit in this same PR — and cites e2e scenario IDsexamine-29/examine-30/examine-31, while the actual scenarios intests/e2e-cucumber/features/examine.featureare numberedexamine-30/examine-31/examine-32after a later renumbering commit. A reader relying on the description to understand what shipped would get the wrong sysfs path, miss an entire code path, and be unable to find the cited tests by name or ID.
Confidence 85 · logic · Fix: update the PR body to describe the final/sys/bus/pci/drivers/amdgpucheck, mention the kfd-implies-amdgpu fallback, and correct the test/scenario names and IDs before merge.
Non-blocking
- Pre-existing, pattern-relevant:
host_gpu.rs'sdetect_driver_summary()(crates/rocm-core/src/host_gpu.rs:2456) still decidesstatus: "amdgpu_available"fromhost_path("/sys/module/amdgpu").exists()— the exact check this PR's own investigation found unreliable for a built-in driver (see the rationale atexamine.rs:1829-1833). Not touched by this diff and not called from the changed code path, so out of scope here, but it's the identical bug class sitting right next to the fix.
Confidence 70 · logic (reuse) · Fix: consider sharingamdgpu_present_on_host()with this call site as a follow-up. - The fake-executable/PATH-prepend fixture idiom is now duplicated 3-4 times in
tests/e2e-cucumber/tests/e2e/examine_steps.rswith no shared helper: the pre-existingplant_gpu_fixtures(:1409-1431), and the two new fixtures this PR adds,plant_machine_without_amdkfd_module(:1463-1490) andmachine_with_built_in_amdgpu(:1520-1549), each write a fake executable and manually prepend it ontoPATHvia the samestd::env::join_paths/split_pathssequence.
Confidence 70 · mechanical (reuse) · Fix: extract a smallfn prepend_fake_bin_to_path(world, bin)helper now that the pattern recurs a third time.
Decisions for the author
- Naming tradeoff, already made and documented — the PR keeps the field name
amdkfd_loadedfor JSON-schema compatibility rather than splitting it into e.g.kfd_functional, which issue #448 suggested as an alternative. The body states this explicitly with its reasoning; flagging only so the choice is visible as deliberate. - Historical continuity worth a one-line mention — an earlier merged PR (
#434) recordedamdkfd_loaded: falseon an MI300X host as a known, unresolved quirk at the time. This PR fixes exactly that quirk but doesn't reference #434. Not a defect, purely a continuity nicety for anyone searching history later.
Positive signals
module_or_host_state's unit tests cover the full tri-state matrix cleanly (module listed, built-in-and-absent, neither present, unreadable-list-follows-host, host-probe-skipped-once-the-list-answers — the last uses|| unreachable!()as its own proof).diagnose.rs's fault-evidence checks all gate strictly on== Some(false), so the new tri-state semantics (module list OR host probes, falling back toNone/"unknown" rather than a false negative) compose correctly with the existing consumer without any additional change needed there.hardware_root.rs's module doc comment, which claims to enumerate every host-path read in the crate, was correctly updated to add both the new/sys/class/kfd/kfdclass-device read and/sys/bus/pci/drivers/amdgpu— this invariant holds as of this PR.- Clean 8-commit history: each commit is independently meaningful and signed off, and the switch from
/sys/module/amdgputo the PCI-driver-dir check (a real correctness improvement, catching a case the earlier commits in this same PR had gotten wrong) was caught and fixed in its own dedicated commit rather than folded in silently.
Deployment notes
- Behavioural change, not just a bugfix in the abstract: on hosts where
amdgpu/kfd are compiled into the kernel,rocm examine --json'samdgpu_loaded/amdkfd_loadedwill now reporttruewhere they previously reportedfalse, androcm diagnosewill stop emitting its DKMS-failure/module-not-loaded/Secure-Boot evidence lines for such hosts. JSON field names and types are unchanged, so this is a value-level behaviour change, not a schema change. - As of this review (live-checked against head
3266af46), every CI check has completed and passed except the aggregate "E2E tests" check, which was still in progress; nothing had failed.
What this covered
Read both changed core-logic files (crates/rocm-core/src/examine.rs, crates/rocm-core/src/hardware_root.rs) and both changed e2e files (tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs) completely, via two independent blind fan-out passes grouped by coupling (core probe logic + its consumers; e2e feature/steps), plus my own direct reading of the full diff, the 8-commit history, the consumers in diagnose.rs/fix.rs/host_gpu.rs, and AGENTS.md. Consolidated and independently re-verified the load-bearing claims against the source (the untested-branch claim, the stale-description claim, the host_gpu.rs duplication, the e2e fixture duplication) before scoring. Base a9e9aa2103ae974b10c5f6e141c09dc6720d8c26, head 3266af461f84bff2e35a69d587f87461b6302a17, re-confirmed live immediately before publishing this review. Issue #448 was fetched directly and checked against the diff and description.
Did not run: cargo test/cargo clippy locally (no Rust toolchain in this sandbox) — reasoned from reading the code and test matrix directly instead, cross-checked against the PR's own CI, where every completed check is green on this head. The "prior changes" pass (whether review comments on earlier, unrelated PRs touching these same files still apply here) was not separately run beyond the one historical PR (#434) found by searching commit history for this exact field name.
Not reconciled against any prior review thread on this PR — this review was produced blind to that discussion by design.
|
Pushed 108d462 for the 3266af4 review (it had no inline threads, so replying here). Fixed
Not changed
|
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 108d462
Review — no blocking findings
Full review of the whole change: crates/rocm-core/src/examine.rs, crates/rocm-core/src/hardware_root.rs, tests/e2e-cucumber/features/examine.feature, and tests/e2e-cucumber/tests/e2e/examine_steps.rs, base a9e9aa21…head 108d462c. The PR fixes issue #448 (amdkfd_loaded reporting false on a host where kfd is built into the kernel) and deliberately extends the same reasoning to amdgpu_loaded, stating that extension explicitly in its own description. The most recent commit on this PR adds a dedicated unit test (kfd_alone_on_the_host_implies_amdgpu) and a named helper (amdgpu_implied_by_host) that together pin down the "kfd on the host alone implies amdgpu" path, which is exactly the kind of behavior that benefits from its own direct test rather than only being exercised incidentally through other fixtures. With that in place, and with CI fully green, this round found no blocking issues — only two small, pre-existing-style documentation/maintenance notes below.
Blocking
None
Non-blocking
- Two doc comments on this PR's own new code disagree about whether a pre-4.20 kernel's
amdkfdever shows up inlsmod. Theamdkfd_loadedfield doc (crates/rocm-core/src/examine.rs:234-237) says kfd is available "as a separateamdkfdmodule on kernels before 4.20 (where kfd was merged intoamdgpu.ko)" — implying older kernels do list it as a module. Thekfd_present_on_host()doc directly above it (crates/rocm-core/src/examine.rs:1813-1818) says the opposite: "amdkfd is linked intoamdgpu, so noamdkfdline ever appears in the module list, built in or not." The field doc looks like the one that matches reality (and matches whyprobe_modulesstill checks the module list for"amdkfd"before falling back to the host probe) — the function doc should be narrowed to "on current kernels" rather than stated unconditionally. - The fake-executable/
PATH-prepend fixture idiom is now written out a third time intests/e2e-cucumber/tests/e2e/examine_steps.rs: the pre-existingplant_gpu_fixtures(:1409), and this PR's two new fixturesplant_machine_without_amdkfd_module(:1463) andmachine_with_built_in_amdgpu(:1520), each independently create a fake binary and splice it ontoPATH. Not a functional problem, but a good candidate for a small shared helper now that the pattern recurs.
Decisions for the author
- Naming tradeoff, already made and documented — the PR keeps the field name
amdkfd_loadedfor JSON-schema compatibility rather than splitting it into separate module-vs-functional fields, which issue #448 raised as an alternative. The description states this explicitly with its reasoning; noting it here only so the choice reads as deliberate.
Positive signals
- The tri-state logic (
module_or_host_state: module-list hit, host-probe hit, list unreadable but host answers, neither) is covered by unit tests for every branch, including a test that proves the host probe is skipped once the module list has already answered. - The e2e scenario for a built-in amdgpu drives both
examine --jsonanddiagnose --jsonagainst the same planted host and asserts thatdiagnose's DKMS/module-not-loaded/Secure-Boot evidence lines are actually absent, closing the loop to the real downstream consumer rather than stopping at the probe's own output. - The PR description's "Tests" section accurately names the current helper functions, test names, and e2e scenario IDs as they exist in the code at this head — it reads as kept in sync through the PR's later commits rather than written once and left stale.
Deployment notes
- Behavioral change, not just a bugfix in the abstract: on hosts where amdgpu/kfd are compiled into the kernel,
rocm examine --json'samdgpu_loaded/amdkfd_loadedwill now reporttruewhere they previously reportedfalse, androcm diagnosewill stop emitting its DKMS-failure/module-not-loaded/Secure-Boot evidence lines for such hosts. JSON field names and types are unchanged, so this is a value-level behavior change, not a schema change. - CI on this head: all 24 checks report either success (22) or an advisory skip (2, not required for merge); nothing failing or pending at review time.
What this covered
Read both core-logic files and both e2e files completely at the PR head, via two independent passes grouped by coupling (core probe logic + its mockable host-path primitive; e2e feature + step definitions), plus a direct read of the full project contributor-guidance file and a comparison of the PR description and linked issue (#448) against the shipped code. Did not run cargo test/cargo clippy/the e2e suite locally — no Rust toolchain or package-registry network access in this environment — so the above is reasoned from reading the code and tests directly, cross-checked against this PR's own CI (fully green). Not reconciled against any existing review thread on this PR; this review was produced independently against the code at the current head.
|
Pushed a doc-only commit for the 108d462 review. The The fake-executable/PATH fixture written out three times in |
|
@jussielo-amd your blocking item was the "E2E tests (Strix Halo, WSL2, current)" lane. On 6891205 it passed (job). That self-hosted run went red on two other lanes. Neither failure involves
The self-hosted run on the current head (4d58eac) is still queued. Every other check on 4d58eac is green. If the WSL2 result is enough for you, could you dismiss the changes-requested review? |
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 4d58eac
Review — no blocking findings
Full review of the whole change (4 files, 457 lines, two independent fan-out workers over coupled file groups plus driver-level impact analysis).
Fixes #448 (GitHub issue, retrieved): "amdkfd_loaded=false is a false negative when kfd is built into the kernel." The change implements the issue's suggested functional-check fallback for amdkfd_loaded, and additionally fixes a related amdgpu_loaded blind spot not raised in the issue itself — this extra scope is disclosed and justified in the PR description, so it is not an undisclosed-work mismatch.
Blocking
None
Non-blocking
assert_no_amdgpu_fault_evidence's first guarded substring doesn't verbatim-match one of the two evidence strings it's meant to guard —tests/e2e-cucumber/tests/e2e/examine_steps.rs:1593(checked againstcrates/rocm-core/src/diagnose.rs:829and:934)
The assertion guards against"amdgpu module is not loaded". That matchescheck_5_amdgpu_blacklisted's evidence (diagnose.rs:934) exactly, butcheck_3_rocm_kernel_unsupported's evidence (diagnose.rs:829) reads"...the amdgpu kernel module is not loaded..."— the extra word "kernel" breaks the substring match. The test is not currently vacuous: a regression incheck_3would still be caught by the separate"DKMS failed"substring in the same assertion, since check_3's full evidence text also contains that phrase. But the two substrings now guard the two checks by accident of overlapping wording rather than by each substring matching its intended check, so a future independent wording tweak to check_5 (e.g. inserting "kernel" to match check_3's phrasing) would silently stop that substring from testing anything, with the assertion still passing.
Confidence 75 · mechanical · Fix: match check_3's and check_5's evidence text precisely (two substrings, one per check), or assert on diagnosis IDs/titles rather than free-text evidence substrings.
Decisions for the author
- kfd presence on the host is treated as sufficient to imply
amdgpu_loaded = true, without independently confirming/sys/bus/pci/drivers/amdgpu— tradeoff
This is deliberate and documented (amdgpu_implied_by_host's doc comment: "a container that passes/dev/kfdthrough without/sys/bus/pcimust not report amdkfd up and amdgpu down"), and is the right call for the stated container-passthrough case. The flip side: a mount namespace that still exposes a stale/dev/kfdnode after the host's amdgpu driver has actually unbound (rare, but not guaranteed impossible across every container runtime) would makeamdgpu_loadedreporttruewhen the PCI driver is genuinely gone. Worth confirming this residual risk is acceptable versus the false-negative it replaces, which was the entire point of the fix.
Positive signals
- The doc comments on
amdgpu_present_on_host/amdgpu_implied_by_hostgive a precise, falsifiable reason for choosing/sys/bus/pci/drivers/amdgpuover/sys/module/amdgpu(the latter appears at an early initcall even whenamdgpu_initthen bails out onnomodeset), and the unit testamdgpu_present_on_host_reads_the_pci_driver_dirdirectly plants both directories separately to prove the distinction the comment claims. the_host_probe_is_skipped_when_the_module_list_already_answersuses an|| unreachable!()closure to prove the laziness claimed inmodule_or_host_state's doc comment, rather than merely asserting the result.- The PR description directly engages with the linked issue's own suggested alternative (splitting into
amdkfd_module_loaded/kfd_functional) and states a specific reason for rejecting it (JSON schema compatibility) rather than silently picking one approach.
Deployment notes
None — both fields keep their existing names, types, and Option<bool> tri-state semantics (unknown stays unknown when neither the module list nor the host answers); this only makes Some(true) reachable in more true-positive cases. probe_modules remains gated to os_family == "linux", so Windows/WSL paths are unaffected.
What this covered
Every file in the diff read completely by at least one of two independent fan-out workers, grouped by coupling: (1) crates/rocm-core/src/examine.rs + crates/rocm-core/src/hardware_root.rs (core probe logic, its unit tests, and every amdgpu_loaded/amdkfd_loaded read site in crates/rocm-core/src/diagnose.rs for impact analysis), and (2) tests/e2e-cucumber/features/examine.feature + tests/e2e-cucumber/tests/e2e/examine_steps.rs (e2e scenarios and step definitions, cross-checked against the real probe_modules/probe_secure_boot parsers and the DiagnoseReport JSON schema). I additionally read the commit history on the branch (10 commits, confirming substantial iteration since the previously-reviewed commit 108d462 — notably a full switch of the amdgpu signal from /sys/module/amdgpu to /sys/bus/pci/drivers/amdgpu in commit 3266af4), checked for other consumers of these two fields and for required doc/README updates (none needed; no doc references these fields' detection semantics), retrieved the linked issue (#448) directly by its identifier via gh issue view, and read CI (gh pr view --json statusCheckRollup): every required check is SUCCESS; the two SKIPPED checks (Skill checks (skillscope), Sphinx docs build (-W)) are path-filtered and correctly skipped since this change touches neither skills/ nor docs.
Did not run: cargo/clippy locally (no cargo available in this sandbox) — relied on CI's passing clippy, build-and-test, and Coverage (rocm-dash crates, ratcheted) checks instead, per the review method's guidance to trust a passing check rather than re-derive what it covered. History pass (git log/git blame beyond the commit-message read above) and prior-changes pass were folded into the fan-out workers' context rather than run as fully separate passes. Not reconciled against any discussion on this PR — this report was produced blind to any existing review thread, per instructions, and the prior review at commit 108d462 was deliberately not read.
|
Pushed bc2937d for the 4d58eac review. The review had no inline threads, so I'm replying here. Fixed: the evidence substrings in
check_3 is no longer caught only through the To check that each phrase works, I made Decision: kfd on the host implies |
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · bc2937d
Review — needs work
Full review of the whole change, read against a9e9aa21…bc2937d8 (the merge-base with origin/main).
Fixes #448, and goes beyond it: the issue reports only the amdkfd_loaded false negative, the change also fixes the analogous amdgpu_loaded false negative for a built-in driver. Both are described accurately in the PR body.
Blocking
-
The new host-fallback wiring in
probe_modulesis under-tested: two ways it could regress would still pass every test.crates/rocm-core/src/examine.rs:1873-1874(the call site), with the gap visible at:6294-6318(probe_modules_reports_amdgpu_when_it_is_on_the_host),:6394-6418(probe_modules_reports_amdkfd_when_kfd_is_on_the_host),:6237-6239(an_unreadable_module_list_follows_the_host), andtests/e2e-cucumber/features/examine.feature:447-461(examine-30/31).- (a) Swapping which closure answers which field at the call site is not caught anywhere. If line 1873 used
amdgpu_present_on_hostinstead ofamdgpu_implied_by_host(dropping "kfd on the host implies amdgpu"), or line 1874 usedamdgpu_implied_by_hostinstead ofkfd_present_on_host(makingamdkfd_loadedtrue whenever amdgpu alone is present), no test fails:probe_modules_reports_amdgpu_when_it_is_on_the_hostonly plants/sys/bus/pci/drivers/amdgpuand only assertsamdgpu_loaded.probe_modules_reports_amdkfd_when_kfd_is_on_the_hostonly plants/dev/kfdand only assertsamdkfd_loaded.kfd_alone_on_the_host_implies_amdgpu(examine.rs:6358-6387, per the diff) callsamdgpu_implied_by_host()directly, bypassingprobe_modules's call-site wiring entirely.- e2e examine-30's fake
lsmodalready listsamdgpu, so the module list answersamdgpu_loadedbefore any host check runs; examine-32 plants the PCI driver dir directly, so it never isolates the kfd-only path either.
I traced this by reading the tests against the production code, not by running a mutation (nocargo/rustcavailable in this sandbox).
- (b) The "module list unreadable, host says true" path (
module_or_host_state'sNonearm, examine.rs:1849-1858) has no scenario, only the plain helper unit testan_unreadable_module_list_follows_the_host, which callsmodule_or_host_statedirectly and asserts nothing aboutprobe_modules's wiring orexamine --jsonoutput. This is new user-observable behaviour this PR introduces (before: an unreadable module list left both fieldsNoneunconditionally; now it can resolve toSome(true)), and AGENTS.md §3 requires a scenario for CLI-observable behaviour, not only a unit test.
Confidence 85 · logic · Fix: strengthen the two existing wiring tests rather than adding new ones — inprobe_modules_reports_amdkfd_when_kfd_is_on_the_host(only/dev/kfdplanted) also assertamdgpu_loaded == Some(true); inprobe_modules_reports_amdgpu_when_it_is_on_the_host(only the PCI driver dir planted) also assertamdkfd_loaded == Some(false)(host-independent, since no modern kernel lists anamdkfdmodule). For (b), add a scenario/unit test with a failinglsmodand empty/proc/modulesplus a planted/dev/kfd, asserting both fields come backtrue.
- (a) Swapping which closure answers which field at the call site is not caught anywhere. If line 1873 used
-
docs/release-trust.md's "Simulated Host Root" path inventory was not updated with the new routed read.docs/release-trust.md:312-326, caused bycrates/rocm-core/src/examine.rs:1821-1833(newamdgpu_present_on_host/kfd_present_on_hostreads) together withcrates/rocm-core/src/hardware_root.rs:11-17.- The change correctly updated
hardware_root.rs's own "every one of these reads, and only these" doc comment to add/sys/class/kfd/kfdand/sys/bus/pci/drivers/amdgpu. docs/release-trust.md's "Simulated Host Root" section is the same enumeration, restated for a different audience (what theROCM_CLI_TEST_HOST_ROOToverride can steer): it still only mentions "/sys/module/amdgpuand itsversionfile" and "the KFD topology under/sys/class/kfd", with no mention of/sys/bus/pci/drivers/amdgpu.- AGENTS.md §5 names this exact pattern: "the same behavior claim… often repeats across… docs/*.md; each drifts independently, so grep for the claim's wording across all of them… and check each against the actual code path." The code is what governs here — the new read genuinely goes through
host_path. - Confidence 85 · mechanical · Fix: add
/sys/bus/pci/drivers/amdgpu(and ideally call out the bare/sys/class/kfd/kfdclass device as distinct from the topology-nodes subpath) to that paragraph.
- The change correctly updated
Non-blocking
- A CI workflow comment enumerating which routed reads have planted-root unit assertions is now stale.
.github/workflows/ci.yml:552-562. It lists which probes have planted-root assertions and says the rest (including/sys/module/amdgpu) have none; this change adds planted-root assertions for/sys/bus/pci/drivers/amdgpu,/sys/class/kfd/kfd, and/dev/kfdviaprobe_modules, which the comment doesn't mention. Confidence 70 · mechanical · Fix: add the newprobe_modules/kfd_present_on_host/amdgpu_present_on_hosttests to the list.
Decisions for the author
None.
Positive signals
module_or_host_statekeeps "module list unreadable and host silent" asNonerather thanSome(false), sorocm diagnosecannot score a driver as missing when nothing was actually observed.- Choosing
/sys/bus/pci/drivers/amdgpuover/sys/module/amdgpuis the right call for a built-in driver whose init bailed out (nomodeset), and it's backed by a unit test (amdgpu_present_on_host_reads_the_pci_driver_dir) pinning that/sys/module/amdgpualone does not count. - The 11-commit history is clean and well-scoped (one concern per commit: helper extraction, wiring, docs, each e2e scenario), with descriptive messages — no "fix"/"wip" noise.
- The PR description is unusually candid about what wasn't verified ("not re-run locally since [renumbering]... CI is the check for those"; "not run on real hardware"), which is exactly the kind of verified-vs-assumed distinction AGENTS.md asks for.
Deployment notes
amdkfd_loadedandamdgpu_loadedinrocm examine --jsonchange meaning for existing hosts: a modern-kernel host with kfd now reportsamdkfd_loaded: truewhere it previously always reportedfalse, and a host with amdgpu built into the kernel flipsamdgpu_loadedfromfalsetotrue. Field names and types are unchanged (schema-compatible). No consumer inside this repo (onlydiagnose.rs, checked) needs updating; an external consumer of the JSON contract that hard-coded the old (buggy) semantics would see a behavior change.
What this covered
Read completely: crates/rocm-core/src/examine.rs (6723 lines at head) and crates/rocm-core/src/hardware_root.rs (408 lines), tests/e2e-cucumber/features/examine.feature (474 lines) and tests/e2e-cucumber/tests/e2e/examine_steps.rs (2173 lines), via two independent fan-out workers grouped by coupling (production code; e2e test code), each reading its files in full plus the production code the scenarios exercise. I separately verified the key blocking finding myself by reading the exact test bodies and fake-lsmod fixtures against the call site, and checked diagnose.rs's three amdgpu_loaded-reading evidence lines, docs/release-trust.md, .github/workflows/ci.yml, and this repo's AGENTS.md directly.
Agent-instruction adherence (AGENTS.md) and code-comments passes were run directly rather than as separate workers, given the diff's size. History pass was limited to this branch's own 11-commit sequence (read and found clean); I did not search git blame/pre-existing history of examine.rs/hardware_root.rs beyond this branch. Prior-changes pass (review comments on earlier PRs touching these files) did not run — no search performed. Step 7 design questions were investigated directly (host-path routing consistency, diagnose.rs interaction with amdgpu_blacklisted_in, the examine.py-mirroring doc comment's relevance — found non-actionable, that file lives outside this repo) rather than via separate workers, again given the diff's size.
CI: read via gh pr view --json statusCheckRollup at the reviewed head; all required checks (Analyze ×3, clippy, CodeQL, build-and-test, Test (affected crates), windows-build-and-test, E2E tests, E2E consolidated report, license/notices/lint checks, etc.) report SUCCESS. Skill checks (skillscope) and Sphinx docs build (-W) report SKIPPED, unrelated to this diff.
Issue #448 was retrieved and read; the PR's description was read after all findings above were settled. Not reconciled against any PR review-thread discussion — this is a blind pass per the review method, and a prior nowycondro review exists on this PR at an earlier head (4d58eacc) that was not read.
|
Pushed 672fcf4, 5474d87 and e0385fe for the bc2937d review. That review had no inline threads either, so I'm replying here. Fixed: the
That second assertion is Fixed: no scenario covered the unreadable module list (blocking, part b) Fixed: Fixed: the stale CI comment (non-blocking) The PR description's Tests section now covers all of this. |
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · e0385fe
Review
This round covers the 3 commits added since the last review at bc2937d8 (672fcf4, 5474d87, e0385fe), which were written specifically to close the two blocking findings and one non-blocking finding from that review: an untested "closures swapped at the call site" risk, an untested "module list unreadable, host still answers" path, and a stale docs/release-trust.md path inventory. All three are verified fixed below. No blocking findings; CI is fully green (24/24 checks).
Blocking
None
Non-blocking
- The fake-executable/
PATH-prepend fixture idiom is now written out a fourth time.tests/e2e-cucumber/tests/e2e/examine_steps.rs's newmachine_with_kfd_and_no_module_list(added in5474d87) repeats the same "write a fake binary, splice it ontoPATHviastd::env::join_paths/split_paths" sequence asplant_gpu_fixtures,plant_machine_without_amdkfd_module, andmachine_with_built_in_amdgpu. Raised in every prior round on this PR as a candidate for a shared helper; still unaddressed, still low-cost to leave as is.
Confidence 70 · mechanical (reuse) · Fix: extract afn prepend_fake_bin_to_path(world, name, script)helper now that the pattern recurs a fourth time.
Decisions for the author
- kfd presence alone is sufficient to imply
amdgpu_loaded = true, without independently confirming/sys/bus/pci/drivers/amdgpu— tradeoff (unchanged from an earlier round, still present and still the deliberate, documented choice inamdgpu_implied_by_host's doc comment). A mount namespace that still exposes a stale/dev/kfdnode after the host's amdgpu driver has actually unbound would makeamdgpu_loadedreporttruewhen the PCI driver is genuinely gone. Worth re-confirming this residual risk is acceptable versus the false-negative it replaces. - Naming tradeoff, already made and documented —
amdkfd_loadedis kept rather than split into e.g.kfd_functional, which issue #448 suggested as an alternative. Stated explicitly in the PR body with its reasoning; noting only so the choice reads as deliberate.
Positive signals
driver_states_answer_each_field_from_its_own_host_signal(new in672fcf4,crates/rocm-core/src/examine.rs) directly closes the swap-detection gap from the last round: itspci_only/unread_pci_onlycases assert(Some(true), Some(false))/(Some(true), None)— an asymmetric pair that a swappeddriver_statestuple, or a swapped destructuring at theprobe_modulescall site, would actually fail. I traced the logic by hand (amdgpu_present_on_host/kfd_present_on_host/module_or_host_stateagainst each planted-root case) and confirmed every asserted value is correct.- The new e2e scenario
examine-33(5474d87) plants/dev/kfdwith no readable module list at all (no/proc/modules, a failing fakelsmod) and asserts bothamdkfd_loadedandamdgpu_loadedcome backtruethroughexamine --json— this is real CLI-observable coverage for the "list unreadable, host answers" path the prior round flagged as unit-test-only. docs/release-trust.mdand the.github/workflows/ci.ymlplanted-root coverage comment (e0385fe) were both updated to list/sys/class/kfd/kfdand/sys/bus/pci/drivers/amdgpu, matching whathardware_root.rs's own doc comment already enumerated — no drift between the three restatements of this invariant anymore.- The PR description's Tests section is now byte-accurate against the shipped code (function names, scenario IDs
examine-30..examine-33) — a genuine improvement over an earlier round in this PR's history where that section had drifted from a mid-PR renumbering.
Deployment notes
Unchanged from prior rounds: on hosts where amdgpu/kfd are compiled into the kernel, rocm examine --json's amdgpu_loaded/amdkfd_loaded now report true where they previously reported false, and rocm diagnose stops emitting its DKMS/module-not-loaded/Secure-Boot evidence lines for such hosts. Field names and types are unchanged (schema-compatible).
What this covered
Fetched the PR head read-only (git fetch origin pull/694/head, never checked out) and diffed a9e9aa21 (merge-base with main) through the current head e0385fe8. The prior nowycondro review at bc2937d8 is attributable, covered the rest of this file set in full at a head that differs from the current one only by these 3 commits, so I narrowed to the delta: git diff --stat bc2937d8..e0385fe8 shows only crates/rocm-core/src/examine.rs, .github/workflows/ci.yml, docs/release-trust.md, tests/e2e-cucumber/features/examine.feature, and tests/e2e-cucumber/tests/e2e/examine_steps.rs touched; hardware_root.rs is untouched by this delta and was not re-read. Read the full diff for each of those 5 files, the surrounding functions it touches in examine.rs (driver_states, module_or_host_state, probe_modules, all four host-presence helpers and their doc comments), and the full current field doc comments on amdgpu_loaded/amdkfd_loaded. The delta is small (147 lines) and each of the three commits maps one-to-one onto a specific prior finding, so this was done as a single direct pass rather than a fan-out (per the method's size threshold), not because the change was large.
Read issue #448 directly (gh issue view) — the PR's scope (fixing amdkfd_loaded, and disclosing the deliberate extension to amdgpu_loaded) still matches it. Read the PR description after settling findings; it is accurate against the current head, including the previously-stale Tests section.
Did not run: cargo test/cargo clippy locally (no Rust toolchain in this environment) — relied on this PR's own CI instead, where all 24 checks report pass on the current head (gh pr checks 694), re-confirmed live immediately before publishing. History pass beyond this PR's own 14-commit sequence, and a separate "prior changes on these files" search, were not run.
Not reconciled against any discussion on this PR, including the existing review thread with inline comments from tomastola and the prior nowycondro reviews — this review was produced blind to that discussion by design.
|
Thanks for the review of e0385fe.
|
main split examine.rs into domain modules (#829). Port this branch's driver-presence probe and its tests into examine/linux_driver.rs, and the host-root locks in the repo-scan tests into examine/rocm_install_probe.rs. The Examination field docs stay in examine.rs. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · e0115c8
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (all 8 changed files, read completely).
Fixes #448 (the amdkfd_loaded false-negative for built-in kfd) and additionally extends the same approach to amdgpu_loaded for the analogous built-in-amdgpu case — not asked for by #448, but explicitly described and justified in the PR body, so this is not a scope mismatch.
Blocking
-
amdgpu_loadedcan readtruefrom akfd/driver-object signal that proves the driver's init code ran, not that any GPU actually bound to it —crates/rocm-core/src/examine/linux_driver.rs(amdgpu_implied_by_host,kfd_present_on_host,amdgpu_present_on_host), consumed atcrates/rocm-core/src/diagnose.rs:832,949,957(check_3_rocm_kernel_unsupported,check_5_amdgpu_blacklisted).
Upstreamdrivers/gpu/drm/amd/amdgpu/amdgpu_drv.c'samdgpu_init()callsamdgpu_amdkfd_init()(which creates/dev/kfdand/sys/class/kfd/kfd) beforepci_register_driver()— i.e. before the PCI subsystem has matched, probed, or bound any device. So kfd's chardev/class-device existing proves amdgpu's init code executed, not that a GPU is bound to it. Concretely reachable: aCONFIG_AMDGPU=ykernel with no AMD GPU, or one whose GPU fails to probe (unsupported ASIC, missing firmware, IOMMU/reset failure,nomodeset) —/dev/kfdexists, soamdgpu_implied_by_host()reportstrue, which suppresses exactly the DKMS/blacklist/Secure-Boot evidence lines meant to catch "amdgpu isn't really bound to your GPU."amdgpu_present_on_host()(the/sys/bus/pci/drivers/amdgpucheck) has a structurally similar gap one level down (driver registration ≠ device bound), and the pre-PRlsmod-only check never proved binding either — so the underlying imprecision (driver-resident vs. GPU-bound) isn't new. What's new is the population it now reaches: previously a built-in, unbound amdgpu was (for the wrong specific reason) still reportedfalseand still triggered an alert; after this PR the same machine is reportedtrueand the alert is silently suppressed. No test in this diff plants a "kfd/driver-object present, nothing bound" fixture, and no comment acknowledges this direction of the asymmetry (only the documented "amdgpu presence doesn't imply kfd" direction is covered).
Confidence 85 · logic · Fix: require evidence of an actually-bound device before reportingtruevia the host-probe path — e.g. PCI BDF-named entries (not justbind/uevent/module) under/sys/bus/pci/drivers/amdgpu/, or a kfd topology node beyond node 0 (/sys/class/kfd/kfd/topology/nodes/<N>, N>0 — node 0 is always the CPU). -
A CI check on this head commit is red:
E2E tests (Strix Halo, Windows, current)— run 41639909936, job 125930604326. The failure is'runtime-resolve-canonical-release' was expected to pass on this host but FAILED — a regression— a wheel/runtime-resolution scenario, not anything in this PR's diff (which only touches Linux kfd/amdgpu detection). Flagging per a failing check being an objective blocking signal, but the subject-matter mismatch makes it likely an unrelated pre-existing flake on that hardware lane rather than something this change caused; worth a quick confirmation rather than treating it as caused by this PR. Two GPU hardware lanes (E2E tests (MI300X, current),E2E tests (W7900, current)) are still queued and two Strix-Halo Ubuntu lanes (nightly, release) were cancelled on this head — none of them have actually exercised this change yet.
Confidence 100 (directly observed check state) · n/a · Fix: rerun/triage that job; not an action item against this diff's content unless it reproduces again.
Non-blocking
cargo xtask coverage's job doesn't pass--features e2e-test-hooks—.github/workflows/ci.yml'scoverage:job, so the newkfd_present_on_host/amdgpu_present_on_host/amdgpu_implied_by_host/driver_statesbodies are exercised by the separate "Test rocm-core with e2e-test-hooks" step but likely not counted toward therocm-corecoverage floor (82.14 incoverage-floors.toml). This mirrors a pre-existing gap already present for other routed reads (stat_device,is_wsl_host,probe_container), so it isn't newly introduced — just worth confirmingcargo xtask coverage --blesswasn't silently needed.
Confidence 70 · architectural · Fix: none required; confirm the floor still holds, or add the feature to the coverage job's invocation.examine-32(Cucumber scenario,tests/e2e-cucumber/features/examine.feature:478-482) would still pass with this PR's whole production change reverted. It plants no kfd signal at all, so old and new code both computeSome(false). It's a legitimate paired negative control (catches an "always-true" host-probe mutation) alongsideexamine-31, not a wasted test, but it doesn't by itself discharge "a new test is shown to fail when the code it covers is broken" the wayexamine-31/33/34do.
Confidence 85 · logic · Fix: note in the scenario's comment block that it's a documented negative control, not standalone proof of the fix.machine_with_built_in_amdgpu(tests/e2e-cucumber/tests/e2e/examine_steps.rs:1579-1607) hand-rolls ROCm-install planting (.info/version+ manualROCM_PATHpush intocommand_env) instead of reusing the existingE2eWorld::plant_unmanaged_rocm()/legacy_rocm_pathhelper, whichisolate_cmdalready applies automatically. This forces the pairedWhenstep (user_inspects_and_diagnoses_planted_machine, lines 1611-1624) to manually clone/restoreworld.command_envbetween the two CLI invocations — correct as written, just avoidable complexity.
Confidence 85 · mechanical · Fix: callplant_unmanaged_rocm()instead.- New e2e step definitions push the
ROCM_CLI_TEST_HOST_ROOTsentinel as a literal string (examine_steps.rs:1397,1420,1512) rather than referencingcrate::hardware_root::TEST_HOST_ROOT_ENV(hardware_root.rs:70), which AGENTS.md §6 asks for ("a sentinel string gets one shared constant"). The same literal is already used identically at three pre-existing call sites in the same file, so this follows rather than introduces the file's established (non-compliant) convention.
Confidence 70 · mechanical · Fix: reference the shared constant, file-wide, as a follow-up.
Decisions for the author
- Field semantics kept broad (
amdkfd_loaded/amdgpu_loadedmean "present," not "bound and working") rather than split into separate fields — tradeoff. The PR body states this was deliberated and a split (e.g.kfd_available) was rejected for JSON-schema compatibility. Given the Blocking finding above, a consumer that specifically wants "is the GPU actually usable" still has no field for that distinction — worth confirming this is still the intended long-term shape, independent of whether the Blocking finding is addressed now or later.
Positive signals
- The old inline
probe_modulesbody was fully replaced byprobe_modules_from/driver_states/module_or_host_statewith no leftover dead code from the previous logic. module_or_host_state's three-way distinction (confirmed-absent vs. unknown-list vs. present) is deliberately preserved rather than collapsed into a single boolean, avoiding a silent-failure trap.hardware_root.rs's doc-comment inventory,docs/release-trust.md, and theci.ymlcoverage comment are kept in exact lockstep for the two new routed paths (/sys/class/kfd/kfd,/sys/bus/pci/drivers/amdgpu) — verified by direct cross-check, not left to drift.- The new field doc comments on
Examination::amdgpu_loaded/amdkfd_loadedprecisely state the new broadened "module or built-in" meaning for a future reader ofexamine.rsalone, closing a documentation gap this kind of change usually leaves open. - Three of the four new e2e scenarios (
examine-31,-33,-34) are genuine, individually-traced regression tests confirmed (by reading, not running) to fail under the pre-PR code, including one (examine-33) that isolates the PCI-driver-only signal from the kfd signal.
Deployment notes
- Purely additive/broadening for existing consumers: no code in
diagnose.rs/fix.rsbranches on module-vs-built-in, only on thefalsecase, which this PR doesn't change. - Per the PR's own text, this has not been run on real built-in-kernel hardware; at review time none of the GPU hardware CI lanes (MI300X, W7900) had exercised this head commit, and the Strix Halo Ubuntu nightly/release lanes were cancelled.
- Until the Blocking finding above is addressed, any real machine with amdgpu's driver code resident but not bound to a working GPU will have its DKMS/blacklist/Secure-Boot diagnose evidence silently suppressed.
What this covered
Full review of all 8 changed files (.github/workflows/ci.yml, crates/rocm-core/src/examine.rs, crates/rocm-core/src/examine/linux_driver.rs, crates/rocm-core/src/examine/rocm_install_probe.rs, crates/rocm-core/src/hardware_root.rs, docs/release-trust.md, tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs), each read completely, via three coupled-file passes (production probe logic; host-root infra/docs/CI; e2e feature+steps), three source-reading passes (AGENTS.md adherence; history + prior-changes against PR #510; code-comments adherence), and two targeted Step-7 design-review investigations (kfd/amdgpu binding-vs-residency plausibility, verified against upstream kernel source; downstream consumer semantics, verified by a repo-wide grep for every reader of the two JSON fields). Commits reviewed: d7faa84…e0115c89 (merge-base…PR head).
Did not run: cargo test/cargo clippy/cargo xtask coverage — no Rust toolchain in this environment; all test-quality and mutation-equivalence verdicts above are argued from reading the code and tracing assertions by hand, cross-checked against this PR's own green CI lane for the relevant unit/e2e-test-hooks tests. GitHub's global search_commits/search_pull_requests/search_issues failed in the history pass with a token/permission error; it substituted list_commits/pull_request_read-by-number and found PR #510 as the one directly relevant prior PR. Per the task's own instructions, this PR's existing reviews were listed once (logins/states/commit SHAs only, no review body text) solely to confirm HEAD had moved past the last review and avoid double-posting; no review body content was read before or during the findings passes, and this report was not reconciled against any discussion on this PR.
🤖 Generated with Claude Code
jussielo-amd
left a comment
There was a problem hiding this comment.
Thanks, this reads well and I'm happy to approve. No blocking findings.
I re-checked the open question from the last round (a built-in driver with no GPU bound now reads amdgpu_loaded=true). I agree with your answer in the thread: the module-list path already reports true for a modprobed driver with no device, and every consumer in diagnose.rs (lines 832, 949, 957) acts only on Some(false), so "driver present" is the right meaning here. The earlier WSL2 flake concern is resolved too.
Non-blocking, take or leave:
host_gpu.rs(detect_driver_summary) still uses/sys/module/amdgpu, which this PR argues is wrong undernomodeset. Soexaminetext can sayamdgpu_availablewhile the JSON saysamdgpu_loaded: false. Fine as the follow-up you already mentioned.- In
rocm_install_probe.rs,merge_live_repo_paths_does_not_duplicate_an_already_seen_pathis the third repo-scan test that goes throughhost_pathwithoutHOST_ROOT_TEST_LOCK. Harmless under nextest. - examine-33 copies the three diagnose evidence strings, but nothing ties them to
diagnose.rs. A small unit test asserting them would keep the scenario from quietly going vacuous if one is reworded.
CI: the red E2E tests (Strix Halo, Windows) lane fails runtime-resolve-canonical-release, which this Linux-only change can't affect. The MI300X and W7900 lanes were still queued when I looked.
🤖 by agenthub on AMD AgentHub
amdgpu_init creates kfd's device before it registers the PCI driver, and the PCI driver directory exists before any device is probed, so neither host signal proves a GPU is bound. The field never meant that: a loaded module is in lsmod with or without a GPU too, and diagnose reads only Some(false), as a module that failed to load. Say so on the field and on amdgpu_implied_by_host, and mark examine-32 as a negative control. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
I merged main into the branch (e0115c8) after #829 split The change request on a2ac4d1 (09:40 review). All three items are still fixed after the move:
On e0115c8, Review of e0115c8.
|
The shared helper already points ROCM_PATH at a planted install for every command, so the step need not hand-roll one. The scenario environment is still restored between examine and diagnose, for the host root and the fake executables. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 279738c
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (not narrowed to the delta since the last automated pass, because that earlier pass's own findings were not available to re-scope against without reading PR discussion, which this method does not do; the full diff was small enough — 8 files, +683/-26 — to review in full anyway). Implements the fix described in GitHub issue #448 (amdkfd_loaded false-negative when kfd is built into the kernel) and extends the same fix to amdgpu_loaded for a built-in amdgpu, which issue #448 did not report but which the PR explains is the same root cause. No Jira ticket is named in the title or body.
Blocking
None
Non-blocking
None
Decisions for the author
- Six new host-root tests in
linux_driver.rshand-build a unique temp directory withstd::env::temp_dir().join(format!("...-{pid}-{nanos}"))and manuallyremove_dir_allit, instead of usingtempfile::tempdir()(already used elsewhere in the same file and inrocm_install_probe.rs's tests) — non-blocking-improvement.tempfile::tempdir()gives auto-cleanup and guaranteed-unique paths with less boilerplate (~10 lines × 6 call sites saved); nothing about these tests needs the manual form. driver_statescallskfd_present_on_host()twice per invocation (once insideamdgpu_implied_by_host, once directly for theamdkfdfield), costing up to 4Path::exists()stats instead of 3 — non-blocking-improvement, negligible cost, not worth restructuring for a probe that runs once perexamineinvocation.- The field is named
amdgpu_loaded/amdkfd_loadedbut now means "driver present," not "driver bound to a GPU" — tradeoff, already addressed in the PR description itself ("Both fields report that the driver is present, not that a GPU is bound to it... Whether there is an AMD GPU ishas_amd_gpu") and the name is kept deliberately for JSON-schema compatibility with a separate name (kfd_available) considered and rejected. Confirmedhas_amd_gpuis a distinct, unaffected field and no consumer (diagnose.rs, the only in-repo consumer ofamdgpu_loaded) conflates the two. Worth the author/maintainers confirming this is still the intended tradeoff, since any future downstream consumer reading the field name literally (without the doc comment) could misread "loaded" as "working."
Positive signals
- Every new helper (
module_or_host_state,driver_states,kfd_present_on_host,amdgpu_present_on_host,amdgpu_implied_by_host) is unit-tested against the exhaustive branch table (module-present, built-in-absent-from-list, confirmed-absent, unreadable-list×host-true/false, host-skipped-once-list-answers, kfd-implies-amdgpu-but-not-reverse), plus two "wiring" tests that specifically catch the helpers being correct in isolation but not actually wired intoprobe_modules_from— this is mutation-resistant test design, not just happy-path coverage. - The e2e negative control (
examine-32) is explicitly documented in its own comment as "a negative control... not proof of the fix," preventing it from being mistaken for coverage it doesn't provide. examine-33's e2e scenario plants a ROCm install + Secure Boot-enabled + built-in amdgpu specifically so the threediagnoseevidence lines it asserts absent are actually reachable if the flag regresses (verified:plant_unmanaged_rocmmakesrocm_versionnon-empty, which is required for one of the three gates to be reachable at all) — a real regression test, not a vacuous one.diagnose.rs's three evidence checks were traced line-by-line and confirmed to gate on exactlyamdgpu_loaded == Some(false)as the PR claims, with no other form used anywhere in that 8900+-line file, andamdgpu_loaded's doc comment inexamine.rswas checked word-for-word against that gating logic.hardware_root.rs's module-doc path allowlist,docs/release-trust.md, and aci.ymlcomment were all updated together to name the two new host paths and the tests that cover them, and were cross-checked as mutually consistent.- JSON schema confirmed unchanged: both fields keep their
Option<bool>type and name; the pre-existingexamination_top_level_keys_match_examine_py_contracttest (untouched by this PR) still enforces the full key set. - Commit history (22 commits) is disciplined: every commit is single-purpose and DCO-signed, and several commits show the author catching and fixing their own earlier mistakes (a scenario-ID renumbering collision, a miscounted CI comment, a non-matching evidence string) before this review ever ran.
- A leak scan across the full diff for internal/proprietary terms came back clean.
Deployment notes
- No migration, API-contract break, or new dependency; JSON schema is unchanged.
- At review time, two hardware-specific E2E check runs (W7900, MI300X) were still queued/in-progress and the PR's
mergeable_statewas reported asblockedby GitHub; every check that had completed was green. Confirm those finish green before merging — they weren't a review finding since nothing had failed.
What this covered
Read completely: crates/rocm-core/src/examine.rs, crates/rocm-core/src/examine/linux_driver.rs, crates/rocm-core/src/examine/rocm_install_probe.rs, crates/rocm-core/src/hardware_root.rs, tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs, .github/workflows/ci.yml and docs/release-trust.md diffs, plus crates/rocm-core/src/diagnose.rs (targeted, around every amdgpu_loaded/amdkfd_loaded/has_amd_gpu use) and the root AGENTS.md for project-convention compliance. Commits d7faa84…279738ca (merge-base…head).
Run as independent fan-out workers: the core-logic pass (the four crates/rocm-core files above), the agent-instruction-adherence pass (against AGENTS.md), the history pass (git log/git blame), and the prior-changes pass (review feedback on earlier, separate PRs touching the same files — checked against PR #510, #675 and #829). The e2e-test-file pass (the .feature/examine_steps.rs group) and the code-comments-compliance pass could not be dispatched as independent workers: the shared subagent pool was persistently exhausted by unrelated concurrent sessions for the full duration of this review despite repeated retries, so both were instead carried out directly by this reviewing context — flagged here per the method's rule that a pass run by the driver itself is not an independent pass, even though its conclusions (no blocking findings) agreed with the independent passes'.
Not reconciled against any discussion on this PR — this review did not read PR comments, review threads, or any earlier review's content, only the PR's own description/issue (Step 9) and the code itself.
🤖 Generated with Claude Code
main added examine-31 (the unopenable /dev/kfd pre-flight), which took the number this branch's first scenario used. Keep main's examine-31 and renumber this branch's built-in kfd/amdgpu scenarios to examine-32..35, in declaration order. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Resolve the examine.feature conflict with #915: main's ROCDXG scenarios keep examine-32..34 (referenced from docs/testing.md and model_serving.feature), and this branch's kfd/amdgpu scenarios move to examine-35..38. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
f952395 to
9a03d73
Compare
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
All three items from siloteemu's 10-30 review are fixed on this branch:
Main is merged in (2ba18c0). The CI workflow is green; the self-hosted E2E lanes (run 42604993736) are still queued/running. Re-requesting review. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 2ba18c0
Outcome: needs work (one blocking finding). The core fix itself (treating a built-in amdgpu/kfd as "loaded" via /sys/bus/pci/drivers/amdgpu, /sys/class/kfd/kfd, /dev/kfd) was traced through by hand across every call path and all 12 new unit tests, and holds up — no defect found there.
Blocking
- Missing test-lock guard leaves a data race on
TEST_HOST_ROOT_ENV—crates/rocm-core/src/examine/rocm_install_probe.rs:309-324(merge_live_repo_paths_does_not_duplicate_an_already_seen_path).
This PR adds aHOST_ROOT_TEST_LOCKguard to two sibling tests in the very same file/commit (live_rocm_repo_paths_in_dir_skips_a_real_bak_file_on_disk,merge_live_repo_paths_scans_every_family_dir_including_zypp) because they callhost_pathindirectly and must not race against another test re-rootingTEST_HOST_ROOT_ENV.merge_live_repo_paths_does_not_duplicate_an_already_seen_pathcalls the identical path (merge_live_repo_paths→live_rocm_repo_paths_in_dir→crate::host_path) but was left unguarded. It's a plain#[test](not feature-gated), so undercargo test --features e2e-test-hookswith the default multi-threaded runner it can race against any of the new host-root tests inlinux_driver.rs.
Fix: add the sameHOST_ROOT_TEST_LOCKguard used in the two sibling tests.
Confidence 85 · logic.
Non-blocking
- PR description cites stale scenario numbers: it says the new scenarios are
examine-31..examine-34, but the committedtests/e2e-cucumber/features/examine.featurenumbers themexamine-35..examine-38(the@id:tags are correct and unique; only the human-readable index drifted, apparently from a main-branch rebase after the description was written). A reader following "examine-31" lands on an unrelated pre-existing scenario. Worth a quick fix in the description. assert_no_amdgpu_fault_evidence(examine_steps.rs) hardcodes three evidence strings copied fromdiagnose/linux.rs'scheck_3_rocm_kernel_unsupported/check_5_amdgpu_blacklistedwith no compile-time link; verified they match today, but a future wording change there would make this e2e assertion pass vacuously with no signal. Consider a shared constant, or at least a comment pointing back from the diagnose-side strings to this test.
Decisions for the author
- Built-in driver masks a DKMS-specific failure — tradeoff. With
amdgpubuilt into the kernel,diagnose's "DKMS failed" / "amdgpu module not loaded" / "Secure Boot blocked amdgpu" evidence (keyed onamdgpu_loaded == Some(false)) now stays silent even if a separately-installed out-of-treeamdgpu-dkmspackage failed, because the in-tree driver still satisfiesamdgpu_loaded. This is consistent with the field's own doc comment ("reports the driver is present, not that a GPU is bound to it... none of those can stop a driver that is present"), so it reads as an intentional scope narrowing rather than an oversight — flagging so it's confirmed as accepted rather than silently assumed.
Positive signals
amdgpu_present_on_host's doc comment explains precisely why/sys/module/amdgpuis the wrong signal (created at an early initcall beforeamdgpu_initcan bail out), and a dedicated test plants that directory alone to prove it doesn't count — directly guarding against re-introducing a mistake the branch's own history shows was made once already.- The module-list-already-answered short-circuit is tested via an
unreachable!()closure, catching an accidental operand swap that would otherwise only show up as a perf regression.
Deployment notes
- GitHub currently reports this PR as
mergeable: CONFLICTINGagainstmain(likely just needs a rebase) — not a code defect, but worth resolving before merge.
What this covered: the full diff (base 17f655d3…head 2ba18c0, 8 files, +684/-27), every changed file read in full by four independent parallel passes (core probe logic; downstream diagnose/fix consumers and impact analysis; e2e feature/step-def tests; AGENTS.md adherence + docs/CI cross-check + commit history), plus my own direct verification of the highest-signal candidates. All CI checks for this commit report SUCCESS except one hardware lane still QUEUED. Not reconciled against this PR's existing review threads.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
main added examine-35..38 (the rocminfo PATH scenarios), which took the numbers this branch's built-in kfd/amdgpu scenarios used. Keep main's and renumber this branch's to examine-39..42, in declaration order. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Reply to the 11-09 review (head 2ba18c0); it has no inline threads, so answering here.
|
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head ffe1098d
Review — needs work
Full review of the whole change (all 8 files in the PR's diff against main@2d9b5a1c, including the full new test content and production probe logic), not a narrowed follow-up: although I (this same account) reviewed an earlier commit of this PR, the gap to this head touches real content, so this pass re-derives everything from the diff/source rather than reusing prior conclusions.
Implements the fix for #448 (amdkfd_loaded/amdgpu_loaded false negatives when the driver is built into the kernel) — the PR's own description matches the linked issue and the actual diff; no scope mismatch found.
Blocking
-
assert_no_amdgpu_fault_evidencehardcodes production evidence strings with no shared constant tying test to source —tests/e2e-cucumber/tests/e2e/examine_steps.rs(the three literals"the amdgpu kernel module is not loaded","amdgpu module is not loaded","Secure Boot is enabled and amdgpu didn't load"), duplicated fromcrates/rocm-core/src/diagnose/linux.rs:27,143,151.
The in-repoAGENTS.mdstates directly: "A value matched across files (a version tag, a sentinel string) gets one shared constant and a test tying the producer to the consumer." These three strings are matched across exactly two files with nothing but a comment ("each copied verbatim from that check so it keeps guarding it") holding them together. Ifdiagnose/linux.rs's wording is edited later (a typo fix, a clarity pass), this regression guard silently stops matching anything — no compile error, no test failure anywhere flags the drift — and the examine-41 scenario this guards would pass regardless of whetheramdgpu_loadedactually regressed.
Confidence 85 · architectural · Fix: pull the three fragments intopub(crate)constants besidecheck_3_rocm_kernel_unsupported/check_5_amdgpu_blacklistedindiagnose/linux.rs, and have the test reference those constants instead of re-typing the text. -
Both the PR description and a new code comment misidentify the negative-control scenario — PR description "Tests" section ("examine-32 is a negative control...");
tests/e2e-cucumber/features/examine.feature:612("# A negative control for examine-35, not proof of the fix: ...").
The actual negative control added by this PR is examine-40 ("The inspection reports kfd as not loaded on a machine without it"), controlling against examine-39 ("...reports kfd on a machine whose module list has no amdkfd"), both new in this diff.examine-32is an unrelated pre-existing scenario ("ROCDXG ready when the loader reaches it through LD_LIBRARY_PATH") andexamine-35is also unrelated and pre-existing ("Both reports find a rocminfo that sits in a PATH directory") — verified directly againstexamine.feature. Both wrong numbers appear to be leftovers from scenario renumbering across this PR's many revisions. This is a description/comment-accuracy defect (the PR's own account of its test coverage is incorrect), not a functional bug, but it affects a maintainer's ability to verify the claimed coverage without independently re-deriving it.
Confidence 100 · mechanical · Fix: replace "examine-32" in the PR description and "examine-35" in the feature-file comment with "examine-39" (and clarify the comment refers to being a negative control for examine-39, proven by examine-40).
Non-blocking
docs/release-trust.md's "Simulated Host Root" paragraph omits theLD_LIBRARY_PATHclause of the WSL ROCDXG check —docs/release-trust.md:329-333vs.crates/rocm-core/src/hardware_root.rs:27-29, which documents the same check as also looking forlib/librocdxg.so"again in each directory onLD_LIBRARY_PATH". This PR edits this exact paragraph (to add the new kfd/amdgpu paths) but the gap itself pre-dates this change and isn't introduced or worsened by it. Flagging because of the direct pattern relevance (same paragraph, same "exhaustive list" claim) and because the doc explicitly presents itself as describing "every one of these reads, and only these."
Confidence 85 · mechanical · Fix: add theLD_LIBRARY_PATHclause torelease-trust.mdto matchhardware_root.rs, ideally in the same change that next touches this paragraph.
Decisions for the author
- Driver-presence vs. GPU-bound, under namespace/mount sharing — tradeoff. The new
amdgpu_implied_by_host/amdgpu_present_on_hostchecks (and the pre-existinglsmod-based ones they extend) report the driver's presence, not whether a GPU is bound to it, by design (documented in both theExaminationstruct doc comment and the PR description). This is consistent with existing behavior for loaded modules, but widens the specific signal to/sys/bus/pci/drivers/amdgpu: in an unusual setup where a container has/sysbind-mounted (read-only) from the host but no/dev/kfd/GPU passthrough, this would readamdgpu_loaded: truefor a container with no real driver access. The PR text already reasons carefully about the symmetric case (kfd passed through without/sys/bus/pci); confirm this asymmetric case was considered and is an accepted limitation consistent with the field's documented meaning, not an oversight.
Positive signals
module_or_host_state'sOption<bool>handling is precise and consistently threaded throughdriver_statesandprobe_modules_from:None(list unreadable) only resolves toSome(true)via the host signal and otherwise staysNone, never incorrectly collapsing toSome(false)— verified by reading the implementation and its unit tests together.rocm_install_probe.rs's addedHOST_ROOT_TEST_LOCKguards fix a real, non-cosmetic hazard:live_rocm_repo_paths_in_dir/merge_live_repo_pathspass their own tempdir paths throughhost_path, which a concurrently-runninge2e-test-hookstest settingTEST_HOST_ROOT_ENVcould otherwise re-root out from under them.- The asymmetric design (kfd implies amdgpu; amdgpu does not imply kfd) is correctly carried through
driver_states, matches real kernel linkage (amdgpu_initcreates kfd's device before registering the PCI driver), and is exercised by tests that would fail if the two checks were swapped.
Deployment notes
None — the JSON schema for amdgpu_loaded/amdkfd_loaded is unchanged (same field names and Option<bool> type), only the circumstances under which each resolves to true/false/unknown change, as stated in the PR description.
What this covered
Read in full: crates/rocm-core/src/examine.rs, crates/rocm-core/src/examine/linux_driver.rs (including every new helper and its unit tests), crates/rocm-core/src/hardware_root.rs, crates/rocm-core/src/examine/rocm_install_probe.rs, docs/release-trust.md, .github/workflows/ci.yml (the touched job/comment), tests/e2e-cucumber/features/examine.feature, and tests/e2e-cucumber/tests/e2e/examine_steps.rs (every new step function plus the helpers they call: run_rocm_with_scenario_env, parsed_json, plant_gpu_fixtures, put_fake_executables_on_path), diffed against merge-base 2d9b5a1c with main. Also read: linked issue #448, crates/rocm-core/src/diagnose/linux.rs (to verify the evidence-string claims and the matched/MIN_SCORE_FOR_MATCH semantics so the examine-41 assertion isn't vacuous — it isn't: a sub-threshold match still carries its evidence text into matched), and the project's AGENTS.md. Live CI at the reviewed head: all required checks green; two self-hosted hardware lanes (E2E tests (W7900, current), E2E tests (MI300X, current)) were still queued at review time, not failing.
Used three independent fan-out passes grouped by file coupling (core probe logic; docs/CI/install-probe test hygiene; e2e feature+steps), then verified and re-derived every candidate finding directly against the source myself before scoring — one reported finding (the docs omission) was reported by a worker with a different framing and is reflected here after independent confirmation; the other two were independently re-derived and, in the scenario-numbering case, found to be broader (also present in the PR description, not just the code comment) than initially reported. Not reconciled against the PR's review-comment discussion/history (by design — only the description and the linked issue were read, per method).
Not a human review. This is an automated code review; please verify findings independently before acting on them.
The three phrases examine-41 asserts the diagnosis does not cite were copied into examine_steps.rs. diagnose/linux.rs now builds those lines from shared constants, exports them as AMDGPU_MISSING_EVIDENCE for the step to use, and a unit test checks every phrase is in what the checks produce. Also point the negative-control comment at examine-39 and add the LD_LIBRARY_PATH clause to the simulated host root paragraph in release-trust.md. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Reply to the 11-10 review (head ffe1098); no inline threads, so answering here. Pushed 8f678e2.
Locally: the 192 |
|
Caution 9 potential secret finding(s) were committed in this pull request. Treat each credential as compromised and revoke or rotate it now, even if a later commit removed it. Deleting the line or force-pushing does not undo the leak: the commits stay reachable from this pull request and from every clone or fork, and automated scanners harvest secrets pushed to public repositories within minutes. If a finding is a test fixture or a false positive, allowlist it in the configuration of the scanner that reported it. Security scan found 9 critical or high finding(s) on lines changed by this pull request
View the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 8f678e2c
Review — no blocking findings
Review of changes since the review at ffe1098d (this same account's most recent prior pass), narrowed to the one follow-up commit (8f678e2c) and the 5 files it touches, read in full at the current head: crates/rocm-core/src/diagnose.rs, crates/rocm-core/src/diagnose/linux.rs, docs/release-trust.md, tests/e2e-cucumber/features/examine.feature, tests/e2e-cucumber/tests/e2e/examine_steps.rs. Narrowing was established by diffing ffe1098d...8f678e2c directly (one commit, 5 files, ~90 lines) and confirming both of that review's open Blocking findings, plus its one Non-blocking finding, are addressed by this diff — see below. The rest of the PR (production probe logic in examine.rs/linux_driver.rs/hardware_root.rs, .github/workflows/ci.yml) is unchanged since that full pass and was not re-read.
Implements the fix for #448 (amdkfd_loaded/amdgpu_loaded false negatives when the driver is built into the kernel) — the PR description matches both the linked issue and the diff; no scope mismatch found.
Blocking
None.
Both of the previous round's Blocking findings are resolved by this commit:
- The hardcoded evidence-string triplet in
assert_no_amdgpu_fault_evidence(tests/e2e-cucumber/tests/e2e/examine_steps.rs) now references a single sharedpub const AMDGPU_MISSING_EVIDENCE: [&str; 3]defined incrates/rocm-core/src/diagnose/linux.rsand re-exported fromdiagnose.rs, exactly matching AGENTS.md's "a value matched across files gets one shared constant and a test tying the producer to the consumer" rule. The three named constants (AMDGPU_KERNEL_MODULE_NOT_LOADED,AMDGPU_MODULE_NOT_LOADED,SECURE_BOOT_BLOCKED_AMDGPU) are consumed directly bycheck_3_rocm_kernel_unsupported/check_5_amdgpu_blacklistedviaformat!, so the producer and the array can no longer drift independently, and two new unit tests exercise both directions (every phrase present whenamdgpu_loaded == Some(false), none present whenSome(true)). - The stale "examine-35" cross-reference in the
examine.featurecomment is now "examine-39", matching the PR description, which already read correctly. - (Non-blocking previously)
docs/release-trust.mdnow documents theLD_LIBRARY_PATHclause of the ROCDXG check, matchinghardware_root.rs.
Non-blocking
None.
Decisions for the author
None.
Positive signals
- The new
AMDGPU_MISSING_EVIDENCEarray is consumed by the same named constants the checks format their evidence strings from, not a second hand-copied literal — the exact class of drift the prior finding warned about is now structurally prevented, not just patched once. pub const AMDGPU_MISSING_EVIDENCE's re-export fromdiagnose.rsmirrors the existingdkms_driver_plan_is_supportedexport on the same line, keeping the module's public surface pattern consistent rather than introducing a one-off.
Deployment notes
None — no schema or behavior change in this follow-up commit; it only ties a test to production constants and corrects comment/doc text.
What this covered
Full read of the 5 files changed between ffe1098da01de304be50a9ba6bdb70236ad62874 and 8f678e2c33017575725dcc485d4ad98fb808570b, plus surrounding context (Examination struct, hardware_root.rs's documented host-path list, AGENTS.md, issue #448). Used three independent fan-out workers grouped by coupling: production code (diagnose.rs + diagnose/linux.rs), e2e/docs (examine.feature + examine_steps.rs + release-trust.md), and the 4b source-reading passes (agent-instruction adherence, history, prior-pattern, code-comments) across all 5 files. One candidate from the production-code worker (no compile-time length guard against someone shrinking AMDGPU_MISSING_EVIDENCE below 3 elements) was verified and dropped: the array's [&str; 3] type annotation already makes such a shrink a compile error unless the size is also edited, so the scenario isn't silent. No worker reported anything else scoring ≥70.
Live CI at the reviewed head: every required check is green (gh pr checks --required). Two optional, non-required jobs (Security scan / gitleaks, Security scan / Security scanners) are failing, but on inspection the flagged secrets are in apps/rocm/src/runtime_install_finalize.rs and apps/rocm/src/serve_summary.rs from commits (#752, #798) already on main well before this branch's recent work and untouched by this PR's diff — pre-existing and unrelated, not a finding here.
Did not run: no Rust toolchain was available to any fan-out worker, so no finding reached confidence 100 by execution; all were traced/verified by reading. One worker (4b passes) did a full-file read of examine.feature/the diff region but not a complete top-to-bottom read of examine_steps.rs's 2766 lines for its comment-compliance pass specifically — a second, independent worker did read that file fully for the primary correctness pass, so coverage of the file itself is not a gap, only the narrower comment-pass's scope on it. Not reconciled against the PR's review-comment discussion/history (by design — only the description and linked issue were read).
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Fixes #448
Problem
rocm examine --jsonreported"amdkfd_loaded": falseon machines where kfd works. The field was true only when a module namedamdkfdappeared inlsmodor/proc/modules. kfd is not a module of its own:HSA_AMDis aboolKconfig option and amdkfd is linked intoamdgpu, so that name never shows up, whetheramdgpuis built in or loaded as a module. A healthy system was reported as "kfd missing".amdgpu_loadedhad the same blind spot for a different reason: anamdgputhat is built into the kernel is absent fromlsmod, so the field wasfalse.rocm diagnosereads that as evidence of a DKMS failure (+30), "amdgpu module is not loaded" (+35) and a Secure Boot problem (+10), none of which is true on such a machine.Change
This PR fixes both flags in
probe_modules.amdkfd_loadedis now true when any of these holds:amdkfdis in the module list (unchanged behaviour);/sys/class/kfd/kfdexists;/dev/kfdexists.amdgpu_loadedis now true when any of these holds:amdgpuis in the module list (unchanged behaviour);/sys/bus/pci/drivers/amdgpuexists. The kernel creates it onceamdgpu_inithas registered the driver, for a loaded module and a built-in driver alike./sys/module/amdgpuis not used: a built-in driver gets it at an early initcall even whenamdgpu_initthen bails out (nomodeset,amdgpu.modeset=0);/sys/class/kfd/kfdor/dev/kfd). kfd is linked intoamdgpu, so kfd cannot be there without it, and a container that passes/dev/kfdthrough without/sys/bus/pcimust not report amdkfd up and amdgpu down.Both fields report that the driver is present, not that a GPU is bound to it.
amdgpu_initcreates kfd's device before it registers the PCI driver, and a built-in driver registers at boot whether or not a matching GPU is there, just as a module that has beenmodprobed is listed inlsmodon any machine. Whether there is an AMD GPU ishas_amd_gpu. Thediagnosechecks read onlyamdgpu_loaded == Some(false), as a module that failed to load (DKMS, a blacklist, Secure Boot), and none of those can stop a driver that is present.The module-list check and the host probes share one function,
module_or_host_state, and the host probe only runs when the module list has not already answered.driver_statespairs the two fields' checks, so which host signal answers which field is testable apart from the real module list.All paths are read through
host_path, so the probes can be tested against a planted root; they are listed with the other simulated-host-root reads inhardware_root.rsanddocs/release-trust.md, and the CI comment on planted-root coverage names their tests. If the module list cannot be read at all but the host shows the driver, the field istruerather than unknown; with no module list and nothing on the host it stays unknown. Both fields keep their names and types, so the JSON schema is unchanged. Each field has a doc comment saying what it means now.Naming. The name
amdkfd_loadedis kept for schema compatibility. The old module check still covers kernels from before kfd was merged intoamdgpu. A separate field such askfd_availablewas considered and rejected.Tests
module_or_host_state: module listed; built in with an empty or unrelated module list; neither present; module list unreadable with and without the driver on the host.kfd_present_on_hostandamdgpu_present_on_hostagainst a planted host root (e2e-test-hooks): sysfs,/dev/kfd, neither; for amdgpu,/sys/module/amdgpualone does not count.amdgpu_implied_by_hostwith only/sys/class/kfd/kfd, then only/dev/kfd: true. Dropping the kfd disjunct fails it.driver_statesagainst a planted host root, with an empty module list and with none, planting one signal at a time: kfd answers both fields, the amdgpu PCI driver answers onlyamdgpu_loaded, and with nothing planted the fields are false (list read) or unknown (no list). Swapping the host checks behind the two fields fails it.probe_modules_fromagainst an empty host root, handed lists naming both drivers, onlyamdgpuand onlyamdkfd: each field follows its own name. A misspelt name indriver_statesor a broken module-list parse fails it.probe_modules_from, handed a module list that names neither driver, against a planted host root with/dev/kfd(both fields true) and with/sys/bus/pci/drivers/amdgpu(amdgpu_loadedtrue,amdkfd_loadedfalse): dropping the host check or swapping the two fields at the call site fails them.examine-reports-kfd-built-into-the-kernel(examine-39) andexamine-reports-kfd-absent-without-kfd-interfaces(examine-40): a planted host root and alsmodthat listsamdgpuand noamdkfd;examine --jsonmust reportamdkfd_loadedtrue with/dev/kfdand false without. examine-40 is a negative control (for examine-39): it passes with the fix reverted too, and fails if the kfd check reports true regardless.examine-reports-amdgpu-built-into-the-kernel(examine-41): a planted host root with/sys/bus/pci/drivers/amdgpu, alsmodthat lists noamdgpu, a ROCm install and Secure Boot enabled.examine --jsonmust reportamdgpu_loadedtrue, anddiagnose --jsonmust cite none of the three evidence lines above.examine-reports-drivers-from-kfd-without-a-module-list(examine-42): a planted host root with/dev/kfdand no/proc/modules, and alsmodthat fails.examine --jsonmust report bothamdkfd_loadedandamdgpu_loadedtrue.All of them run on any Linux runner and need no GPU.
cargo test -p rocm-core --libpasses, the new tests pass with--features e2e-test-hooks, and clippy with that feature is clean. Mutating the kfd probe (hard-coding the host check off, dropping the unreadable-list fallback, dropping the host check from the decision) or swapping the host checks behind the two fields fails the unit tests. examine-31 to examine-34 pass throughcargo xtask e2e. Not run on real hardware; the behaviour was checked against the reporter'sexamineoutput and the upstream kernel Kconfig/Makefile.