Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 56 additions & 17 deletions crates/rocm-core/src/diagnose/catalog.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,17 @@ use crate::examine::Examination;
/// A checker plus the OS families it applies to.
type Checker = (fn(&Examination, &str) -> Diagnosis, &'static [&'static str]);

// The `wsl` entries below run on less evidence than on bare metal: WSL collects
// no GPU topology (`gpus` is empty, because the probes that fill it read KFD and
// DRM), so the parts of those checks that compare a detected gfx target against a
// wheel's arch list cannot contribute. They still fire on the evidence WSL does
// The `wsl` entries below run on different evidence than on bare metal. WSL has
// no KFD or DRM topology, so its `gpus` are the adapters the Windows host lists,
// which are not guaranteed to be devices the guest runtime can open. A check that
// compares a detected gfx target against a wheel's arch list can therefore name
// an adapter the runtime never sees. They also fire on the evidence WSL does
// have -- environment, ROCm install, symptom keywords -- which is strictly better
// than the previous behaviour of not running at all.
//
// The degradation is one-directional and must stay that way: less evidence means
// a check may miss a real fault, never that it invents one. Pinned by
// When the host inventory could not be read, `gpus` is empty and the degradation
// is one-directional: less evidence means a check may miss a real fault, never
// that it invents one. Pinned by
// `the_shared_checks_under_report_on_wsl_rather_than_over_report`.
const CHECKERS: &[Checker] = &[
(check_1_arch_not_in_wheel, &["linux", "windows", "wsl"]),
Expand All @@ -67,8 +69,7 @@ const CHECKERS: &[Checker] = &[
(check_6_path_missing, &["linux", "windows", "wsl"]),
(check_7_stale_repos, &["linux"]),
(check_8_wheel_rocm_mismatch, &["linux", "windows", "wsl"]),
// Not "wsl": WSL2 exposes no per-device topology to collide over.
(check_9_igpu_dgpu_collision, &["linux", "windows"]),
(check_9_igpu_dgpu_collision, &["linux", "windows", "wsl"]),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enabling check 9 on wsl needs the generated outputs regenerated (cargo xtask catalog for doctor-catalog.json, cargo xtask eval-fixtures) and skills/rocm-doctor/reference.md updated (table row and the "Linux and Windows" prose); clippy and the Windows xtask test fail on these today. The header comment above CHECKERS and the check_20 comment in this file still say WSL has no GPU list.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done on all four. cargo xtask catalog and cargo xtask eval-fixtures are regenerated and committed.

Updated all stale references to WSL having no GPU list.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check_9 now runs on WSL on evidence from the Windows host inventory, while usable_amd_gpu_indices still reports one device there. What shows the guest runtime enumerates both adapters, and that a host-order --device-index N matches the HIP ordinal? Please back the "runtime sees BOTH GPUs" finding with guest-side evidence or soften it on WSL.

(check_10_container_devices, &["linux"]),
(check_11_iommu_hang, &["linux"]),
(check_12_repo_native_broken, &["linux"]),
Expand Down Expand Up @@ -101,8 +102,9 @@ const CHECKERS: &[Checker] = &[
// same 64 MB default a container does, so leaving this tagged `linux` alone
// would silence it on one of the two platforms most likely to have it.
(check_19_shm_too_small, &["linux", "wsl"]),
// Not "wsl": WSL2 reports no GPU topology (`gpus` is empty), so this check
// has nothing to read there and would never fire.
// Not "wsl": this makes a claim about the whole machine's ROCm support, and
// WSL's `gpus` come from the Windows host inventory rather than from what the
// guest runtime can open.
(check_20_gpu_unsupported_by_rocm, &["linux", "windows"]),
// Windows only: the runtime is read from the Windows System32 folder. A WSL
// distro is examined as a distro, so it has no such runtime to read.
Expand Down Expand Up @@ -219,6 +221,43 @@ mod tests {
use crate::diagnose::{MIN_SCORE_FOR_MATCH, diagnose, render_report_text};
use crate::examine::Gpu;

/// An APU paired with a discrete card is a finding on WSL2, not silence.
///
/// Both halves are asserted together because either alone still reports
/// nothing useful: the checker has to run, *and* the catalog has to name the
/// WSL family, or applicability falls back to print-only and the fix the
/// finding offers cannot be run at all.
#[test]
fn an_apu_beside_a_discrete_card_is_reported_on_wsl_as_it_is_on_bare_metal() {
let mut e = igpu_dgpu_host("linux");
e.is_wsl = true;
assert_eq!(platform_family(&e), "wsl", "the fixture must read as WSL");

let report = diagnose(&e, "");
let found = report
.matched
.iter()
.find(|d| d.id == "fix-9-igpu-dgpu")
.expect("the iGPU/dGPU pairing is reported on WSL");

assert!(
found.score >= MIN_SCORE_FOR_MATCH,
"a reported pairing has to clear the match threshold, not merely exist: {}",
found.score
);
let fix = found.fix.as_ref().expect("the finding carries its fix");
assert_eq!(
fix.class,
crate::fix::FixClass::NeedsArgument,
"WSL is its own catalog family; without an entry the fix degrades to print-only"
);
assert_eq!(
fix.needs.as_deref(),
Some("--device-index"),
"and the argument that makes it runnable must be named"
);
}

/// A pasted HIP-loading error still opens fix-13 when no AMD GPU was
/// detected at all — deliberately, and this is the test that says so.
///
Expand Down Expand Up @@ -460,15 +499,15 @@ mod tests {

#[test]
fn the_shared_checks_under_report_on_wsl_rather_than_over_report() {
// WSL does not collect GPU topology: `gpus` stays empty because the
// probes that fill it read KFD and DRM, which do not exist there. The
// cross-platform checks enabled on WSL read those fields, so they run on
// less evidence here than on bare metal.
// WSL enumerates from the Windows host inventory rather than KFD and
// DRM, so a host whose inventory could not be read records no adapters
// at all. The cross-platform checks enabled on WSL then run on less
// evidence than they would on bare metal.
//
// That degradation has to be one-directional. Missing topology must mean
// That degradation has to be one-directional. Missing evidence must mean
// a check cannot reach its threshold on structure alone -- never that it
// invents a fault. This pins the direction: on a healthy host with a
// framework installed, no shared check may fire at all.
// framework installed and no adapters recorded, no shared check may fire.
let mut e = wsl_base();
e.framework = "pytorch".to_owned();
e.framework_version = "2.6.0".to_owned();
Expand All @@ -478,7 +517,7 @@ mod tests {
e.rocm_path = "/opt/rocm".to_owned();
e.env
.insert("PATH".to_owned(), "/opt/rocm/bin:/usr/bin".to_owned());
assert!(e.gpus.is_empty(), "WSL collects no GPU topology");
assert!(e.gpus.is_empty(), "this fixture records no adapters");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test now only covers the empty-gpus case. With gpus populated from the host inventory the shared wsl checks (check_1, check_8, ...) can name an adapter the runtime never sees; please add a populated-gpus healthy-host test so the under-report-not-over-report invariant stays pinned.


let report = diagnose(&e, "");
assert!(
Expand Down
9 changes: 5 additions & 4 deletions crates/rocm-core/src/examine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -660,9 +660,10 @@ impl Examination {
/// healthy machine. The caller already holds the detection (it is the
/// `summary` of `rocm examine --json`), so this takes it rather than probing
/// again: the Windows query is a PowerShell call with its own timeout. Does
/// nothing off WSL, where the probes own the field, and leaves `gpus` empty
/// (there is no PCI listing to build entries from) and `status` alone
/// (`compute_status` answers `wsl` before it asks about a GPU).
/// nothing off WSL, where the probes own the field, and leaves `gpus` and
/// `status` alone — the WSL probe fills the GPU list from the host adapter
/// inventory (`record_host_adapters`), and `compute_status` answers `wsl`
/// before it asks about a GPU.
pub const fn adopt_wsl_host_gpu(&mut self, host_has_amd_gpu: bool) {
if self.is_wsl {
self.has_amd_gpu = host_has_amd_gpu;
Expand Down Expand Up @@ -924,7 +925,7 @@ mod tests {
assert!(e.has_amd_gpu, "a detected gfx target means a GPU is there");
assert!(
e.gpus.is_empty(),
"there is no PCI listing on WSL to build gpu entries from"
"adopting the host verdict must not invent gpu entries of its own; the WSL probe fills the list from the host adapter inventory"
);
assert!(
!e.has_apu && !e.has_discrete_amd,
Expand Down
141 changes: 139 additions & 2 deletions crates/rocm-core/src/examine/wsl.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
//! same facts from a named WSL distribution when `rocm examine` runs on the Windows
//! host.

use super::{Examination, WSL_MIN_UBUNTU, WSL_PLATFORM_NOTE, WslFacts};
use super::{Examination, Gpu, WSL_MIN_UBUNTU, WSL_PLATFORM_NOTE, WslFacts};
use crate::probe_exec::{MEDIUM, run, run_utf16le, which};
use std::collections::BTreeMap;
use std::time::Duration;
Expand All @@ -19,7 +19,9 @@ use std::time::Duration;
/// the Windows host driver version.
pub(super) fn probe_wsl(e: &mut Examination) {
let summary = crate::detect_wsl_summary();
let (host_reachable, host_driver_version) = host_driver_fields(crate::detect_wsl_host_driver());
let (host_driver, host_adapters) = crate::detect_wsl_host_displays();
let (host_reachable, host_driver_version) = host_driver_fields(host_driver);
record_host_adapters(e, &host_adapters);
let rocminfo = which("rocminfo");
e.wsl = Some(WslFacts {
version: if crate::is_wsl1_kernel(&e.kernel_release) {
Expand Down Expand Up @@ -50,6 +52,45 @@ pub(super) fn probe_wsl(e: &mut Examination) {
e.notes.push(WSL_PLATFORM_NOTE.to_owned());
}

/// Record the Windows host's AMD adapters as this examination's GPU list.
///
/// WSL has no `lspci` scan and no KFD topology to enumerate from, but the host
/// inventory the driver probe already reads names every AMD adapter; keeping
/// only the driver version from it is why `examine` reported one gfx target on
/// a two-GPU machine.
///
/// The target is resolved in three tiers: PCI device id, then the shared
/// marketing-name table, then the keyword heuristic. The middle tier is not
/// optional — the id table misses parts the name table knows (`DEV_7448`, the
/// PRO W7900), which without it get no target at all.
///
/// An adapter nothing identified keeps `is_apu` unset rather than `Some(false)`,
/// unlike `probe_gpus_windows`: `check_9_igpu_dgpu_collision` pairs an APU with a
/// discrete card, so calling an unidentified part discrete would invent that
/// pairing. An APU keyword match still counts as integrated with no target.
fn record_host_adapters(e: &mut Examination, adapters: &[crate::WslHostDisplayAdapter]) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These entries come from the Windows host inventory, not from what the WSL guest runtime enumerates, yet fix-9 now says the runtime sees both GPUs and --device-index is applied against host order. Can a host iGPU that is not exposed to the distro occur, and does wsl_usable_indices (still one device) now disagree with gpus[]? Consider gating on guest evidence or noting the limitation.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can a host iGPU that is not exposed to the distro occur

Yes, though this was more of an issue in older (<10) ROCm releases. WSL used to only enumerate nodes that were present in a pre-defined hardware mapping so there could be cases where Windows list != WSL list.

That's no longer the case on newer TheRock builds where WSL runtime will get GPU info directly from the driver instead of the intermediary mapping.

From what I understand, rocm-cli reports gpu lists both prior and post rocm install. In this case, we could,

  1. No ROCm SDK - report Windows list
  2. ROCm SDK installed - compare existing Windows list with runtime list - overwrite with the latter in case of discrepancy.

This seems to be what's already done on Ubuntu with lspci (pre ROCm) then KFD (post ROCm). Let me know what you think.

for adapter in adapters {
let (gfx_target, is_apu) =
match crate::host_gpu::gfx_target_from_pnp_device_id(&adapter.pnp_device_id)
.or_else(|| crate::gfx_target::gfx_target_from_amd_marketing_name(&adapter.name))
{
Some(target) => (target.to_owned(), Some(super::gfx_is_apu_family(target))),
None => match super::linux_gpus::classify_amd_marketing_name(&adapter.name) {
(target, false) if target.is_empty() => (target, None),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test where the inventory lists the same physical adapter twice (or a virtual AMD-named display); as written each row becomes a gpus[] entry and can fabricate an APU+dGPU pair.

(target, is_apu) => (target, Some(is_apu)),
},
};
e.gpus.push(Gpu {
name: adapter.name.clone(),
gfx_target,
pci_id: adapter.pnp_device_id.clone(),
is_apu,
is_amd: true,
});
}
super::linux_gpus::summarise_gpu_categories(e);
}

/// Flatten a host-driver probe into the two [`WslFacts`] fields that carry it.
///
/// One place, so the in-guest and host-side probes cannot drift on the point
Expand Down Expand Up @@ -338,6 +379,102 @@ mod tests {
use super::*;
use crate::probe_exec::decode_utf16le;

fn host_adapter(name: &str, device_id: &str) -> crate::WslHostDisplayAdapter {
crate::WslHostDisplayAdapter {
name: name.to_owned(),
// The whole string `Win32_VideoController` reports, instance path
// included, so the id parser is exercised on what a host actually
// sends rather than on a tidied prefix.
pnp_device_id: format!(
"PCI\\VEN_1002&DEV_{device_id}&SUBSYS_0E0D1002&REV_00\\6&3666587B&0&00000008"
),
}
}

/// `DEV_7448`, the PRO W7900, is not in the PCI id table, so this adapter
/// is identifiable only by its name. Measured on a real WSL2 host: reading
/// the id and then dropping straight to the APU keyword list left this card
/// with no target at all, while the scalar probe answered `gfx1100` from
/// the same row by consulting the shared marketing-name table.
#[test]
fn an_adapter_missing_from_the_id_table_is_identified_by_its_name() {
let mut e = Examination::default();
record_host_adapters(&mut e, &[host_adapter("AMD Radeon PRO W7900", "7448")]);

assert_eq!(
e.gpus[0].gfx_target, "gfx1100",
"the name table names this part even though the id table does not"
);
assert_eq!(
e.gpus[0].is_apu,
Some(false),
"and the target says it is discrete"
);
assert!(
e.has_discrete_amd && !e.has_apu,
"a lone discrete card is exactly one kind"
);
}

/// A Ryzen desktop whose integrated Radeon is listed by Windows *ahead* of
/// the discrete card. Both adapters have to survive the probe — keeping only
/// the first is what reported gfx1036 on a host whose card is a gfx1201 —
/// and the pair has to read as one APU and one discrete part, which is what
/// `check_9_igpu_dgpu_collision` fires on.
#[test]
fn both_host_adapters_are_recorded_when_the_igpu_is_listed_first() {
let mut e = Examination::default();
record_host_adapters(
&mut e,
&[
host_adapter("AMD Radeon(TM) Graphics", "164E"),
host_adapter("AMD Radeon RX 9070 XT", "7550"),
],
);

let targets: Vec<&str> = e.gpus.iter().map(|g| g.gfx_target.as_str()).collect();
assert_eq!(
targets,
vec!["gfx1036", "gfx1201"],
"every AMD adapter the host listed must be recorded, in the host's order"
);
assert_eq!(
e.gpus[0].is_apu,
Some(true),
"the Raphael iGPU is an APU by its device id"
);
assert_eq!(
e.gpus[1].is_apu,
Some(false),
"the discrete card must not be folded in with it"
);
assert!(
e.has_apu && e.has_discrete_amd,
"the host summary must show both kinds, or the iGPU/dGPU check has nothing to fire on"
);
}

/// An AMD adapter neither table identifies is recorded with no APU verdict.
/// `Some(false)` would read as "discrete" and could invent the second half
/// of an iGPU/dGPU pair out of a part nothing recognized.
#[test]
fn an_unidentified_adapter_is_recorded_without_an_apu_verdict() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This depends on DEV_73DF staying absent from the id table (it is an RX 6700 XT id). Use an obviously synthetic id (e.g. DEV_FFFF) for the unidentified case.

let mut e = Examination::default();
record_host_adapters(&mut e, &[host_adapter("AMD Radeon(TM) Graphics", "73DF")]);

assert_eq!(e.gpus.len(), 1, "the adapter is still reported");
assert!(
e.gpus[0].gfx_target.is_empty(),
"no table names this device id, so no target may be claimed"
);
assert_eq!(e.gpus[0].is_apu, None, "and no kind may be claimed either");
assert!(
!e.has_apu && !e.has_discrete_amd,
"an unidentified part counts as neither kind"
);
assert!(e.has_amd_gpu, "it is still an AMD GPU that is present");
}

/// Ported from the Python preflight this replaced, whose `wsl.exe` parser
/// was covered by a self-test that CI ran on both lanes. That coverage has to
/// land here, or deleting the script quietly drops it.
Expand Down
20 changes: 13 additions & 7 deletions crates/rocm-core/src/fix/catalog.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,10 @@ const PRINT_ON_LINUX_AND_WSL_AUTO_ON_WINDOWS: &[Platform] = &[
on("wsl", FixClass::PrintOnly),
];

/// `fix-9-igpu-dgpu`. Both arms short-circuit when no device index is given:
/// they print the query that identifies the discrete GPU and return, so there
/// is nothing for `--dry-run` to preview and no prompt to refuse.
/// `fix-9-igpu-dgpu`. Every arm short-circuits without `--device-index`, printing
/// the query that identifies the discrete GPU. WSL needs its own entry despite
/// running the Linux arm: applicability is keyed on platform family, so without
/// one the fix degrades to print-only.
const NEEDS_DEVICE_INDEX: &[Platform] = &[
Platform {
os: "linux",
Expand All @@ -112,11 +113,16 @@ const NEEDS_DEVICE_INDEX: &[Platform] = &[
class: FixClass::NeedsArgument,
needs: Some("--device-index"),
},
Platform {
os: "wsl",
class: FixClass::NeedsArgument,
needs: Some("--device-index"),
},
];

/// `info-20-gpu-no-rocm-support`. Not "wsl": WSL2 reports no GPU topology, so
/// the checker that produces this id never fires there and `rocm fix` would be
/// claiming an answer for a host it was never asked about.
/// `info-20-gpu-no-rocm-support`. Not "wsl": the checker that produces this id
/// does not run there, so `rocm fix` would be claiming an answer for a host it
/// was never asked about.
const DIAGNOSE_ONLY_ON_LINUX_AND_WINDOWS: &[Platform] = &[
on("linux", FixClass::DiagnoseOnly),
on("windows", FixClass::DiagnoseOnly),
Expand Down Expand Up @@ -2295,9 +2301,9 @@ mod tests {
("fix-6-path", "linux", FixClass::Auto),
("fix-6-path", "windows", FixClass::Auto),
("fix-6-path", "wsl", FixClass::Auto),
// Both arms short-circuit until `--device-index` names a target.
("fix-9-igpu-dgpu", "linux", FixClass::NeedsArgument),
("fix-9-igpu-dgpu", "windows", FixClass::NeedsArgument),
("fix-9-igpu-dgpu", "wsl", FixClass::NeedsArgument),
]
);
// fix-16-vllm-oom carries a workaround the user must weigh (lowering the
Expand Down
6 changes: 3 additions & 3 deletions crates/rocm-core/src/host_gpu.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,10 @@ use windows_inventory::{
use wsl::detect_wsl_windows_display_probe_text;
#[cfg(test)]
use wsl::local_windows_host_driver_probe;
pub use wsl::{WSL_ROCDXG_READY, WslHostDriverProbe, is_wsl_host};
pub use wsl::{WSL_ROCDXG_READY, WslHostDisplayAdapter, WslHostDriverProbe, is_wsl_host};
pub(crate) use wsl::{
detect_local_windows_host_driver, detect_wsl_host_driver, detect_wsl_summary, is_wsl1_kernel,
ldconfig_cache, ldconfig_lists_librocdxg,
detect_local_windows_host_driver, detect_wsl_host_displays, detect_wsl_host_driver,
detect_wsl_summary, is_wsl1_kernel, ldconfig_cache, ldconfig_lists_librocdxg,
};
use wsl::{detect_wsl_windows_display_gfx_target_fast, wsl_driver_summary};

Expand Down
7 changes: 4 additions & 3 deletions crates/rocm-core/src/host_gpu/gpu_visibility.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,10 @@ fn probe_usable_amd_gpu_indices() -> Option<Vec<u32>> {

/// The usable ordinals on a WSL host, from the plumbing `examine` reports on.
///
/// One device: WSL2 exposes no per-device topology to enumerate, and the
/// visibility mask still applies on top so an explicit `HIP_VISIBLE_DEVICES=""`
/// is honoured. Takes the summary rather than reading the machine so the
/// One device: this reads the guest's ROCDXG readiness, not the host adapter list
/// `examine` records in `gpus`, so on a multi-adapter host the two can disagree.
/// The visibility mask still applies on top so an explicit
/// `HIP_VISIBLE_DEVICES=""` is honoured. Takes the summary rather than reading the machine so the
/// decision -- ready means a device, anything else means none -- is testable
/// without a WSL host.
#[cfg(any(target_os = "linux", test))]
Expand Down
Loading
Loading