Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 131 additions & 0 deletions crates/rocm-core/src/diagnose.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3759,6 +3759,137 @@ mod tests {
);
}

/// ROCMAI-830: the other host ROCMAI-546's gate has to hold on. A native
/// Windows Strix Halo (gfx1151) whose default engine is Lemonade, with no
/// system HIP SDK, diagnosed twice: before any runtime was installed, and
/// after `rocm install sdk` put a managed TheRock wheel runtime in place.
///
/// Unlike the gfx90c laptop above, gfx1151 is a supported ROCm target, so
/// `info-20` does not fire and nothing outranks fix-13 here: on v0.2.1,
/// which was cut before the gate landed, fix-13 was the *only* entry and
/// ranked #1 at 35 on a machine serving models fine. A managed runtime is
/// not a HIP-based setup that needs the system SDK either -- its wheels
/// bundle their own HIP runtime -- whether the torch probe through it
/// answered (a `hip=` build) or timed out (`unknown`, as on the reporting
/// machine under load).
///
/// The fields mirror that machine's `rocm examine --json`. The final
/// positive-control arm keeps the gate honest: the same host with an
/// ambient HIP-built torch on PATH does need the SDK, so fix-13 still
/// ranks first there.
#[test]
fn fix_13_does_not_fire_on_a_lemonade_strix_halo_host_with_a_managed_runtime() {
let strix_halo = |framework: &str, rocm_version: &str, source: &str| Examination {
os_family: "windows".to_owned(),
cpu_vendor: "amd".to_owned(),
gpus: vec![Gpu {
name: "AMD Radeon(TM) 8060S Graphics".to_owned(),
gfx_target: "gfx1151".to_owned(),
is_apu: Some(true),
is_amd: true,
..Gpu::default()
}],
has_amd_gpu: true,
has_apu: true,
// No system HIP SDK: the arm that used to score 35 on its own.
hip_sdk_path: String::new(),
hipinfo_present: false,
adrenalin_version: "32.0.32015.7029".to_owned(),
msvc_redist_present: Some(true),
framework: framework.to_owned(),
framework_rocm_version: rocm_version.to_owned(),
framework_source: source.to_owned(),
..Examination::default()
};
let summary = |report: &DiagnoseReport| {
report
.matched
.iter()
.map(|d| (d.id.clone(), d.score))
.collect::<Vec<_>>()
};

for (label, host) in [
(
"before any runtime was installed",
strix_halo("unknown", "", ""),
),
// Framework "unknown" means nothing HIP-based is detected, so
// `needs_hip` is false whatever `framework_source` says: this arm
// documents the reporting machine's shape but does not exercise
// the managed-runtime exemption. The next arm does.
(
"managed runtime whose torch probe timed out (no HIP evidence)",
strix_halo("unknown", "", "managed-runtime"),
),
(
"managed runtime whose torch is a HIP build",
strix_halo("pytorch", "hip=7.14.60850", "managed-runtime"),
),
] {
assert_eq!(
check_13_hip_sdk_missing(&host, "").score,
0,
"{label}: nothing on this host calls into the system HIP SDK"
);
let report = diagnose(&host, "");
assert!(
report.matched.is_empty() && !report.has_match,
"{label}: a healthy Lemonade host must report no match, not a WEAK \
fix-13 at #1: {:?}",
summary(&report)
);
}

let ambient = diagnose(&strix_halo("pytorch", "hip=7.14.60850", "path"), "");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Positive arm is good. Please confirm in the PR text that you broke the gate (e.g. removed the managed-runtime/Lemonade exemption) and saw this test go red, per AGENTS §3.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, and shown in the PR description: with needs_hip forced on (removing the managed-runtime / Lemonade exemption), this test fails and diagnose-48 fails too; with the gate intact both pass. Pushed as 864aef2.

assert_eq!(
ambient.matched.first().map(|d| d.id.as_str()),
Some("fix-13-hip-sdk-missing"),
"the same host with a HIP-built torch on PATH really needs the SDK: {:?}",
summary(&ambient)
);
}

/// Pins what production does today for an ambient HIP-built PyTorch beside
/// a healthy system SDK: the +25 "PyTorch needs a working SDK" bonus is not
/// gated on the SDK being unusable, so fix-13 still lists (score 25, below
/// `MIN_SCORE_FOR_MATCH`, so advice only). The diagnose-48 oracle mirrors
/// this. If the bonus is later gated on `sdk_unusable`, change both.
#[test]
fn fix_13_lists_weakly_for_ambient_hip_torch_beside_a_healthy_system_sdk() {
let host = |hip_sdk_path: &str, hipinfo_present: bool, source: &str| Examination {
os_family: "windows".to_owned(),
has_amd_gpu: true,
hip_sdk_path: hip_sdk_path.to_owned(),
hipinfo_present,
framework: "pytorch".to_owned(),
framework_rocm_version: "hip=7.14.60850".to_owned(),
framework_source: source.to_owned(),
..Examination::default()
};
let sdk = r"C:\Program Files\AMD\ROCm\6.4";

let healthy = host(sdk, true, "path");
assert_eq!(check_13_hip_sdk_missing(&healthy, "").score, 25);
let report = diagnose(&healthy, "");
let row = report
.matched
.iter()
.find(|d| d.id == "fix-13-hip-sdk-missing")
.expect("ambient HIP torch lists fix-13 even with a healthy SDK");
assert!(
row.score < MIN_SCORE_FOR_MATCH,
"advice only: {}",
row.score
);

// Control: the same healthy SDK under a managed runtime is silent.
assert_eq!(
check_13_hip_sdk_missing(&host(sdk, true, "managed-runtime"), "").score,
0
);
}

/// The textbook fix-13 host: Windows, an AMD GPU, no HIP SDK, no framework
/// the probe recognises — and a pasted error that is HIP failing to load.
///
Expand Down
27 changes: 23 additions & 4 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -2368,16 +2368,35 @@ cargo xtask e2e -- -n '(diagnose-42|driver-install-08)'
```

The other half of the same change — `fix-13-hip-sdk-missing` no longer firing
just because no HIP SDK is installed — has **no** end-to-end coverage and cannot
have any. `check_13_hip_sdk_missing` returns early on every non-Windows host, so
just because no HIP SDK is installed — has no Linux end-to-end coverage.
`check_13_hip_sdk_missing` returns early on every non-Windows host, so
on a Linux lane that early return alone satisfies "no cause tells this user to
install the HIP SDK"; the assertion would still hold with `check_20` deleted.
The gate itself is held by unit tests in
[diagnose.rs](../crates/rocm-core/src/diagnose.rs)
(`fix_13_does_not_fire_on_a_lemonade_vulkan_host_with_no_hip_evidence` and its
neighbours) and by the Windows manual check in
(`fix_13_does_not_fire_on_a_lemonade_vulkan_host_with_no_hip_evidence`,
`fix_13_does_not_fire_on_a_lemonade_strix_halo_host_with_a_managed_runtime`
and their neighbours) and by the Windows manual check in
[manual-testing.md](manual-testing.md), in the Ryzen 4000/5000 section.

On the Windows GPU lane (`e2e-gpu-strix-windows`), `diagnose-48` in
[diagnose.feature](../tests/e2e-cucumber/features/diagnose.feature) runs the
gate on a real host, like `diagnose-41`: it names no host shape and holds
`rocm diagnose --json` to `rocm examine --json` on the same machine.
`fix-13-hip-sdk-missing` has to appear exactly when the inspection reports an
AMD GPU and either a HIP-built llama.cpp with no usable system HIP SDK, or an
ambient (not managed-runtime) HIP-built PyTorch: the latter scores a further
25 whether or not the SDK is healthy, so it lists the entry (weakly, below the
match threshold) even beside a working SDK. ROCMAI-837 considers that +25
listing wrong, so whoever changes the bonus must update the scenario's oracle
too. On that lane's Strix Halo hosts (Lemonade, no system SDK) it asserts the
cause is absent, which is the host shape of ROCMAI-830. Run it only on
Windows:

```bash
cargo xtask e2e -- -n '(diagnose-48)'
```

## Model Fit Preflight

`rocm diagnose --model <ref>` answers whether a curated model will run on this
Expand Down
21 changes: 21 additions & 0 deletions tests/e2e-cucumber/features/diagnose.feature
Original file line number Diff line number Diff line change
Expand Up @@ -727,3 +727,24 @@ Feature: Diagnosing failures and listing fixes
Given a simulated Linux machine with ROCm missing from PATH
When the user diagnoses that machine with a rocminfo not found symptom
Then the PATH cause prints a verify line that shows the GPU with rocminfo alone

# A missing system HIP SDK is a fault only when something on the machine calls
# into it: a HIP-built PyTorch from outside a managed runtime, or a HIP-built
# llama.cpp on PATH. Lemonade's bundled llama.cpp does not, and a managed
# TheRock runtime brings its own HIP. On v0.2.1 a healthy Windows Strix Halo
# serving through Lemonade was still told "install the HIP SDK" as its #1
# cause, before and after `rocm install sdk`. Like diagnose-41 the scenario
# names no host shape: it holds `diagnose` to what `rocm examine --json`
# reports on the same machine, so on the self-hosted Windows GPU lanes (Strix
# Halo, Lemonade, no system SDK) it asserts the cause is absent. The arm where
# an ambient HIP framework needs the SDK is held by the unit test
# `fix_13_does_not_fire_on_a_lemonade_strix_halo_host_with_a_managed_runtime`
# in crates/rocm-core/src/diagnose.rs. The oracle also counts a weak listing
# (below the match threshold) as present, which is what production does for
# an ambient HIP torch beside a healthy SDK; ROCMAI-837 considers that +25
# bonus wrong, so whoever changes it must update this scenario's oracle too.
@id:diagnose-windows-hip-sdk-advice-follows-the-inspection @requires-os:windows @requires-gpu
Scenario: diagnose-48 - On Windows the diagnosis lists the HIP SDK cause only when the inspection finds something that needs it
Given a machine with an AMD GPU
When the user diagnoses this machine in machine-readable form, with no symptom
Then the diagnosis lists the HIP SDK cause only when the inspection finds something that needs it
63 changes: 62 additions & 1 deletion tests/e2e-cucumber/tests/e2e/diagnose_steps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3222,7 +3222,9 @@ async fn assert_no_hip_sdk_advice(world: &mut E2eWorld) {
// The stated substitute is the unit test
// `fix_13_does_not_fire_on_a_lemonade_vulkan_host_with_no_hip_evidence` in
// `rocm-core::diagnose`, which drives the whole `diagnose()` pipeline on
// the ticket's exact Windows/gfx90c host shape.
// the ticket's exact Windows/gfx90c host shape. The gate on a supported
// Windows GPU (the Strix Halo shape of ROCMAI-830) does run end to end, as
// `diagnose-48` on the Windows GPU lane.
let output = world.cli_output.as_ref().expect("no diagnose output");
assert!(
!output.contains("fix-13-hip-sdk-missing"),
Expand Down Expand Up @@ -3270,3 +3272,62 @@ async fn assert_old_driver_advice_follows_the_inspection(world: &mut E2eWorld) {
assert!(score < threshold, "{row:#}");
}
}

#[then(
"the diagnosis lists the HIP SDK cause only when the inspection finds something that needs it"
)]
async fn assert_hip_sdk_advice_follows_the_inspection(world: &mut E2eWorld) {
const FIX_ID: &str = "fix-13-hip-sdk-missing";
let diagnosis: serde_json::Value =
serde_json::from_str(world.cli_output.as_ref().expect("no diagnosis"))
.expect("`diagnose --json` did not emit valid JSON");
let (examine_json, _, _) = crate::run_rocm(world, &["examine", "--json"]);
let examine: serde_json::Value =
serde_json::from_str(&examine_json).expect("`examine --json` did not emit valid JSON");
let field = |name: &str| {
examine
.get(name)
.unwrap_or_else(|| panic!("the inspection carries no `{name}`:\n{examine:#}"))
};
let text = |name: &str| field(name).as_str().unwrap_or_default().to_owned();
// Something on this host that calls into the system HIP SDK, read from the
// inspection the same way `check_13_hip_sdk_missing` reads it with no
// symptom: a managed runtime's HIP build brings its own HIP runtime, and
// Lemonade's bundled llama.cpp is not on PATH as a HIP build.
let has_amd_gpu = field("has_amd_gpu").as_bool() == Some(true);
let framework = text("framework");
let ambient_hip_torch = framework == "pytorch"
&& text("framework_rocm_version").starts_with("hip=")
&& text("framework_source") != "managed-runtime";
let hip_llama_cpp = framework == "llama-cpp" && text("framework_rocm_version") == "GGML_HIP=ON";
let ambient_hip_torch = has_amd_gpu && ambient_hip_torch;
let hip_llama_cpp = has_amd_gpu && hip_llama_cpp;
let needs_hip = ambient_hip_torch || hip_llama_cpp;
let sdk_unusable =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This oracle restates check_13's predicate (SDK path empty or hipinfo absent). Please confirm it matches the production condition exactly. On the target Strix Halo lane expected is always false, so the step only proves absence; note which break of the gate turns diagnose-48 red.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and nowycondro found the same thing: check_13_hip_sdk_missing also adds 25 for an ambient HIP-built PyTorch whether or not the SDK is usable, so the oracle was wrong for a healthy-SDK host with ambient torch. The oracle now mirrors the scoring exactly (an AMD GPU plus either a HIP-built llama.cpp with no usable SDK or a non-managed HIP-built PyTorch), and a new unit test pins that host. On this lane the expected value is always false, so diagnose-48 only proves absence here; it goes red when a regression makes fix-13 score for a host with no HIP evidence (verified by forcing needs_hip on). The weak listing itself looks wrong and is tracked as ROCMAI-837; production scoring is unchanged in this test-only PR.

text("hip_sdk_path").is_empty() || field("hipinfo_present").as_bool() != Some(true);
// ROCMAI-837 considers the ambient-torch +25 listing beside a healthy SDK
// wrong. This oracle copies that rule on purpose, so whoever changes the
// +25 bonus in `check_13_hip_sdk_missing` must update this copy too.
// Mirrors the scoring in `check_13_hip_sdk_missing`, not only its
// headline: inside the `needs_hip` gate an unusable SDK scores 35 or 30,
// and an ambient HIP-built torch adds a further 25 that does not look at
// the SDK at all. `run_all_checks` keeps any score above zero in
// `matched`, so a host with ambient HIP torch and a healthy SDK still
// lists the entry (weakly, 25).
let expected = needs_hip && (sdk_unusable || ambient_hip_torch);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This oracle copies check_13's gating and the +25 rule, and pins the weak listing that ROCMAI-837 considers wrong. Please name ROCMAI-837 in the comment here (and in the feature comment) so whoever changes the +25 bonus finds this copy too.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the oracle comment and the feature comment now name ROCMAI-837 and say that whoever changes the +25 bonus must update this copy.

let present = diagnosis["matched"]
.as_array()
.expect("matched is a list")
.iter()
.any(|entry| entry["id"] == FIX_ID);
assert_eq!(
present,
expected,
"{FIX_ID} present: {present}; inspection: framework {framework:?} (source {:?}, \
rocm {:?}), HIP SDK {:?}, hipInfo {}\n{diagnosis:#}",
text("framework_source"),
text("framework_rocm_version"),
text("hip_sdk_path"),
field("hipinfo_present"),
);
}
Loading