From 196b95d0049376f8094665628716f80cb8911ede Mon Sep 17 00:00:00 2001 From: Eugene Volen Date: Tue, 27 Oct 2026 14:40:55 +0200 Subject: [PATCH] fix(doctor): make the lines printed for Windows run in PowerShell Several catalog entries that apply on Windows printed lines written for one shell. Windows PowerShell 5.1 rejects them or runs them wrongly: - The verify line for fix-15-msvc-redist was `where vcruntime140.dll && where vcruntime140_1.dll`. In Windows PowerShell 5.1 `&&` is a syntax error, and `where` is the alias for Where-Object, not where.exe. fix-6-path's verify line also chained two commands with `&&`. - The verify line for fix-2-unset-override was `env | grep HSA_OVERRIDE_GFX_VERSION || echo OK_UNSET`. `||` is a syntax error in Windows PowerShell 5.1, and `env` and `grep` do not exist in cmd. - fix-2 printed `setx HSA_OVERRIDE_GFX_VERSION ""` for Windows. Windows PowerShell 5.1 drops an empty-string argument when it calls a native program, so setx received only the variable name, failed with "Invalid syntax" and left the override set. The catalog and the diagnose fix now print the .NET call `[Environment]::SetEnvironmentVariable('HSA_OVERRIDE_GFX_VERSION',$null,'User')`, labelled as a PowerShell line. `rocm fix` itself is unchanged: it spawns setx directly, with no shell, and passes the empty argument intact. - Four verify lines wrapped in `powershell -Command "..."` could not be pasted into PowerShell. fix-6 (diagnose, Windows) and fix-13 put `\"` inside the body, and PowerShell does not take a backslash as an escape, so the string ends early. fix-14, fix-22 and fix-9 (diagnose, Windows) used `$_` or `$env:NAME=...`, which an outer PowerShell expands before the inner one starts. The fix-6 PATH write is not changed here. It keeps the bare PowerShell `[Environment]::SetEnvironmentVariable` line, its two Windows notes and the shared `ps_single_quote_escape` that are already on main. New verify lines. Each is meant to run in both cmd and Windows PowerShell 5.1; the Windows-only tests below run the catalog ones, and a few lines that only `rocm diagnose` prints are PowerShell-only by design: - fix-15 and fix-21: the DLL is found with `[Environment]::GetFolderPath('System')` inside the body, so the body holds no `$` and an outer PowerShell has nothing to expand (a `$env:SystemRoot` expanded there would split the command at a space in the path). fix-15 prints `FileName,FileVersion` for vcruntime140.dll and vcruntime140_1.dll; fix-21 prints the FileVersion of msvcp140.dll. Get-Item reports a missing path as a non-terminating error, which leaves the exit code at 0, so both lines use `-ErrorAction Stop` to keep the old behavior that a missing DLL fails the verify. - fix-2: `python -c "import os; print(os.environ.get('HSA_OVERRIDE_GFX_VERSION') or 'OK_UNSET')"`. The program is one double-quoted argument with no `$`, `%` or inner double quote, so no shell changes it. A stock Ubuntu has `python3` and no `python`, so Linux and WSL get a second line, `Verify (Linux, WSL): echo "${HSA_OVERRIDE_GFX_VERSION:-OK_UNSET}"`, which needs no Python and no external command and has no `||`; a note says which line to run. `rocm diagnose` on Linux prints that line and then `python3 -c "import torch; print(torch.cuda.is_available())"`, so the user can tell whether the GPU still works after the override is cleared. That arm only builds for a bash host; the shared catalog line cannot assume torch. The Windows commands in the catalog now say that the change only affects new shells. - fix-6: the catalog prints `hipcc --version` as the line every OS can run (it checks what the fix does: the bin directory is on PATH), and a second line, `Verify (Linux, WSL): rocminfo | head -n 5`, which shows the GPU and, unlike hipcc, exists on a runtime-only install. This is a new optional field on the recipe, `verify_linux`, printed after `verify`, with a note that names the runtime-only case; the plain `Verify:` line and the manifest are unchanged. `rocm diagnose` on Linux prints the catalog's Linux line alone, so it passes on a runtime-only install with PATH fixed (`rocminfo` lives in the bin directory this fix puts on PATH). The Windows arm of the checker verifies with `& '\hipInfo.exe' | Select-Object -First 5`, with a `'` in the folder doubled by the shared escape. - fix-13: `& (Join-Path ([Environment]::GetEnvironmentVariable('HIP_PATH')) 'bin\hipInfo.exe') | Select-Object -First 5`. It uses single quotes and no `$`. With HIP_PATH unset, Join-Path stops with a parameter-binding error and exit code 1. - fix-14 and fix-22: `Where-Object Name -match 'AMD|Radeon'` replaces the script block that used `$_`. - fix-9 (diagnose, Windows): sets the variable with `[Environment]::SetEnvironmentVariable('HIP_VISIBLE_DEVICES','1','Process')` and runs python with a single-quoted program. The catalog recipes and the copies in the diagnose checkers are kept in step by the shared-id tests in diagnose.rs, which now also cover fix-13, fix-14, fix-2 and fix-6 (the Linux arms are compared with the catalog's Linux lines). fix-9's Windows line exists only in the diagnose checker and has its own tests. Tests: - `fails_in_windows_powershell` is the rule for a line shown on Windows: no `&&` or `||`, no bare `where` (a command starts after `|`, `;`, `(`, `{`, a lone `&`, at the start of a `-Command` body and after `cmd /c`), no `%NAME%`, no empty-string argument outside a `-Command` body, no single-quoted `-Command` argument (`-c` and the other prefixes of `-Command` count), in a double-quoted `-Command` body no `\"` and no `$` at all, and no `Get-Item` there without `-ErrorAction Stop`. The `where` and `-Command` matches ignore case. It is a text rule over known-bad constructs and does not prove a line runs. It is applied to every printed command, the verify line and the Linux verify line of each catalog recipe that applies on Windows (derived from `applies_on`, so a new recipe is covered), and to the `Fix` each Windows checker builds. - The rule exists twice, because the e2e crate cannot depend on rocm-core. Both copies read one table of cases, `tests/e2e-cucumber/fixtures/powershell_text_rule_cases.txt`, so a case that one rule gets wrong fails on that side. - Unit tests take the fix-15 and fix-2 lines apart (one quoted argument, the cmdlets, no `$`). Others assert the two verify lines of fix-6 (the plain one has no rocminfo, the Linux one has no hipcc and shows the GPU) and of fix-2 (the Linux one has no python), and that the notes name the runtime-only and the python3 cases. The diagnose tests assert that the Linux fix-2 line is the catalog's Linux line plus a `python3` torch check and that the Linux fix-6 line is the catalog's Linux line alone. On Unix hosts they also run those lines in /bin/sh on a PATH of stand-in executables with no hipcc, or with `python3` and no `python`, and run the catalog's fix-2 line with an empty PATH. - Windows-only unit tests type the printed lines into cmd and into Windows PowerShell (on stdin, as if typed), from a helper in fix.rs that starts PowerShell 5.1 without an inherited `PSModulePath`. Each asserts exit 0 and an empty stderr, plus what is specific to its line: `the_msvc_verify_runs_in_cmd_and_powershell` (fix-15: one row per DLL, each with a dotted version), `the_msvc_verify_exits_nonzero_when_a_dll_is_missing`, `the_outdated_msvc_verify_prints_a_version_and_fails_when_the_dll_is_missing` (fix-21), `the_override_verify_runs_in_cmd_and_powershell` (fix-2), `the_hip_sdk_verify_runs_the_exe_under_hip_path` (fix-13: HIP_PATH points at a throwaway directory whose name has a space), `the_adrenalin_verify_runs_and_its_filter_keeps_only_amd` (fix-14), `the_display_driver_verify_runs_in_cmd_and_powershell` (fix-22) and `the_igpu_verify_pins_the_variable_it_names` (fix-9, in diagnose.rs). - Scenarios diagnose-43, diagnose-44 and diagnose-45 preview fix-15, fix-2 and fix-6 through `rocm fix --dry-run` and apply the rule to the printed lines. diagnose-43 also checks that the verify line is a PowerShell command; diagnose-44 checks the Linux verify line reads the override without Python and that a note names the python3 case; diagnose-45 checks the Linux verify line shows the GPU without hipcc and that a note names the runtime-only install. The plan prints before the platform gate, so they run on the Linux lanes too. They do not run the lines. Verified on Windows 11 with Windows PowerShell 5.1: the Windows-only tests above, run locally with `cargo test -p rocm-core --lib`, and the scenarios with `cargo xtask e2e`. fix-13 runs against a stand-in executable, not a real HIP SDK, and fix-14 and fix-9 not against a real torch. The fix-6 PATH write is not run by any test, because it changes the user's Path. Signed-off-by: Eugene Volen --- crates/rocm-core/src/diagnose.rs | 444 ++++++- crates/rocm-core/src/fix.rs | 1057 ++++++++++++++++- docs/testing.md | 72 ++ tests/e2e-cucumber/features/diagnose.feature | 74 ++ .../fixtures/powershell_text_rule_cases.txt | 92 ++ tests/e2e-cucumber/src/lib.rs | 4 +- .../e2e-cucumber/src/powershell_text_rule.rs | 128 ++ .../e2e-cucumber/tests/e2e/diagnose_steps.rs | 295 +++++ 8 files changed, 2137 insertions(+), 29 deletions(-) create mode 100644 tests/e2e-cucumber/fixtures/powershell_text_rule_cases.txt create mode 100644 tests/e2e-cucumber/src/powershell_text_rule.rs diff --git a/crates/rocm-core/src/diagnose.rs b/crates/rocm-core/src/diagnose.rs index 79a8a70f18..940fb1b6a1 100644 --- a/crates/rocm-core/src/diagnose.rs +++ b/crates/rocm-core/src/diagnose.rs @@ -786,7 +786,7 @@ fn check_2_hsa_override_unneeded(e: &Examination, symptom: &str) -> Diagnosis { "[Environment]::GetEnvironmentVariable('HSA_OVERRIDE_GFX_VERSION','User')".to_owned(), "[Environment]::GetEnvironmentVariable('HSA_OVERRIDE_GFX_VERSION','Machine')".to_owned(), "# Clear from the User scope (does NOT affect already-open shells):".to_owned(), - "setx HSA_OVERRIDE_GFX_VERSION \"\"".to_owned(), + "[Environment]::SetEnvironmentVariable('HSA_OVERRIDE_GFX_VERSION',$null,'User')".to_owned(), "# Or remove via System Properties -> Environment Variables.".to_owned(), ], fix_id: "fix-2-unset-override".to_owned(), @@ -801,7 +801,14 @@ fn check_2_hsa_override_unneeded(e: &Examination, symptom: &str) -> Diagnosis { "# Also remove it from ~/.bashrc / ~/.zshrc / ~/.profile if persisted.".to_owned(), ], fix_id: "fix-2-unset-override".to_owned(), - verify: "env | grep HSA_OVERRIDE_GFX_VERSION || echo OK_UNSET; python -c \"import torch; print(torch.cuda.is_available())\"".to_owned(), + // The `rocm fix` catalog's Linux line (needs no Python, which a + // stock Ubuntu lacks), then the check that clearing the override + // did not break the GPU. The catalog lines are shared with Windows + // and cannot assume torch; this arm only builds for a bash host, + // so it can run a second command after the first. `python3`, not + // `python`: whatever interpreter has torch, a stock `python3`, a + // virtual environment and a conda environment all provide it. + verify: "echo \"${HSA_OVERRIDE_GFX_VERSION:-OK_UNSET}\"; python3 -c \"import torch; print(torch.cuda.is_available())\"".to_owned(), ..Fix::default() } }; @@ -1075,7 +1082,7 @@ fn check_6_path_missing(e: &Examination, symptom: &str) -> Diagnosis { ], fix_id: "fix-6-path".to_owned(), verify: format!( - "powershell -NoProfile -Command \"& \\\"{bin_dir}\\hipInfo.exe\\\" | Select-Object -First 5\"" + "powershell -NoProfile -Command \"& '{escaped_bin_dir}\\hipInfo.exe' | Select-Object -First 5\"" ), ..Fix::default() } @@ -1087,7 +1094,12 @@ fn check_6_path_missing(e: &Examination, symptom: &str) -> Diagnosis { format!("echo 'export PATH={bin_dir}:$PATH' >> ~/.bashrc # or ~/.zshrc"), ], fix_id: "fix-6-path".to_owned(), - verify: "rocminfo | head -n 5 && hipcc --version".to_owned(), + // The `rocm fix` catalog's Linux line. `rocminfo` lives in the bin + // directory this fix puts on PATH, so it printing the GPU shows + // both that PATH is fixed and that the GPU is visible. It does + // not chain `hipcc --version`: a runtime-only install has no + // `hipcc`, and the chain would fail there with PATH fixed. + verify: "rocminfo | head -n 5".to_owned(), ..Fix::default() } }; @@ -1366,7 +1378,7 @@ fn check_9_igpu_dgpu_collision(e: &Examination, symptom: &str) -> Diagnosis { "# `setx` only takes effect in NEW shells; reopen the terminal.".to_owned(), ], fix_id: "fix-9-igpu-dgpu".to_owned(), - verify: "powershell -NoProfile -Command \"$env:HIP_VISIBLE_DEVICES=1; python -c \\\"import torch; print(torch.cuda.device_count())\\\"\"".to_owned(), + verify: "powershell -NoProfile -Command \"[Environment]::SetEnvironmentVariable('HIP_VISIBLE_DEVICES','1','Process'); python -c 'import torch; print(torch.cuda.device_count())'\"".to_owned(), notes: vec![note], ..Fix::default() } @@ -1832,7 +1844,7 @@ fn check_13_hip_sdk_missing(e: &Examination, symptom: &str) -> Diagnosis { "# After install, reopen the shell so HIP_PATH and PATH pick up the new install.".to_owned(), ], fix_id: "fix-13-hip-sdk-missing".to_owned(), - verify: "powershell -NoProfile -Command \"& \\\"$env:HIP_PATH\\bin\\hipInfo.exe\\\" | Select-Object -First 5\"".to_owned(), + verify: "powershell -NoProfile -Command \"& (Join-Path ([Environment]::GetEnvironmentVariable('HIP_PATH')) 'bin\\hipInfo.exe') | Select-Object -First 5\"".to_owned(), notes: vec!["If you only need PyTorch on Windows AMD and don't need the C/C++ HIP toolchain, the TheRock wheels bundle their own HIP runtime and may not require a system HIP SDK install.".to_owned()], ..Fix::default() }; @@ -1889,7 +1901,7 @@ fn check_14_adrenalin_too_old(e: &Examination, symptom: &str) -> Diagnosis { ], needs_reboot: true, fix_id: "fix-14-adrenalin-too-old".to_owned(), - verify: "powershell -NoProfile -Command \"(Get-CimInstance Win32_VideoController | Where-Object { $_.Name -like '*AMD*' -or $_.Name -like '*Radeon*' } | Select-Object -First 1).DriverVersion\"".to_owned(), + verify: "powershell -NoProfile -Command \"(Get-CimInstance Win32_VideoController | Where-Object Name -match 'AMD|Radeon' | Select-Object -First 1).DriverVersion\"".to_owned(), ..Fix::default() }; finalize( @@ -1926,7 +1938,7 @@ fn check_15_msvc_redist(e: &Examination, symptom: &str) -> Diagnosis { "# After the install, reopen the shell and re-run your import / hipInfo check.".to_owned(), ], fix_id: "fix-15-msvc-redist".to_owned(), - verify: "where vcruntime140.dll && where vcruntime140_1.dll".to_owned(), + verify: "powershell -NoProfile -Command \"(Get-Item ([Environment]::GetFolderPath('System') + '\\vcruntime140.dll'),([Environment]::GetFolderPath('System') + '\\vcruntime140_1.dll') -ErrorAction Stop).VersionInfo | Select-Object FileName,FileVersion\"".to_owned(), notes: vec!["If installing the redistributable still leaves a missing-DLL error, the failing DLL is probably amdhip64_X.dll itself; that points at fix-13-hip-sdk-missing rather than this fix. If the runtime is present but the program crashes with an access violation, see fix-21-msvc-redist-outdated.".to_owned()], ..Fix::default() }; @@ -2592,7 +2604,7 @@ fn check_21_msvc_redist_outdated(e: &Examination, symptom: &str) -> Diagnosis { "# Then close and restart the program (or Lemonade) so it loads the new runtime.".to_owned(), ], fix_id: ID.to_owned(), - verify: "powershell -NoProfile -Command \"(Get-Item $env:SystemRoot\\System32\\msvcp140.dll).VersionInfo.FileVersion\"".to_owned(), + verify: "powershell -NoProfile -Command \"(Get-Item ([Environment]::GetFolderPath('System') + '\\msvcp140.dll') -ErrorAction Stop).VersionInfo.FileVersion\"".to_owned(), notes: vec![ "The verify command prints the runtime version. It should read 14.40 or higher.".to_owned(), "`rocm fix` only prints this plan. It does not run the install: the install needs the Windows administrator prompt and your consent.".to_owned(), @@ -2641,7 +2653,7 @@ fn check_22_display_driver_old(e: &Examination, _symptom: &str) -> Diagnosis { ], needs_reboot: true, fix_id: id.to_owned(), - verify: "powershell -NoProfile -Command \"Get-CimInstance Win32_VideoController | Where-Object { $_.Name -like '*AMD*' -or $_.Name -like '*Radeon*' } | Select-Object Name,DriverVersion,DriverDate\"".to_owned(), + verify: "powershell -NoProfile -Command \"Get-CimInstance Win32_VideoController | Where-Object Name -match 'AMD|Radeon' | Select-Object Name,DriverVersion,DriverDate\"".to_owned(), notes: vec![ crate::windows_driver::NO_MINIMUM_EXPLANATION.to_owned(), crate::windows_driver::FLOOR_NOTE.to_owned(), @@ -6991,7 +7003,7 @@ mod tests { #[test] fn diagnosis_remediation_matches_the_fix_catalog_for_shared_fix_ids() { - // fix-15, fix-16, fix-21, info-20 and fix-22 live in two places: the `rocm fix` + // fix-13, fix-14, fix-15, fix-16, fix-21, info-20 and fix-22 live in two places: the `rocm fix` // catalog (crate::fix::RECIPES) and the Fix these checks embed in a // Diagnosis. The prose *framing* legitimately differs by surface (a // catalog rationale vs a diagnosis summary), but the actionable @@ -7041,6 +7053,25 @@ mod tests { }, "", ); + // fix-13 and fix-14 print a Windows verify that the Windows-only tests + // in `fix.rs` run from the catalog copy, so this copy has to match it. + let hip_sdk_missing = check_13_hip_sdk_missing( + &Examination { + os_family: "windows".to_owned(), + ..Examination::default() + }, + "ImportError: DLL load failed: amdhip64_6.dll was not found", + ); + let adrenalin_old = check_14_adrenalin_too_old( + &Examination { + os_family: "windows".to_owned(), + hip_sdk_path: r"C:\Program Files\AMD\ROCm\6.4".to_owned(), + hipinfo_present: true, + hipinfo_status: "error".to_owned(), + ..Examination::default() + }, + "driver too old", + ); // fix-22 is advice only: it scores below the match threshold on purpose, // yet a user meets both of its copies, so it is held to the same guard. let display_driver_old = @@ -7050,6 +7081,8 @@ mod tests { msvc, msvc_outdated, unsupported_gpu, + hip_sdk_missing, + adrenalin_old, display_driver_old, ] { if diag.id == crate::windows_driver::DISPLAY_DRIVER_OLD_FIX_ID { @@ -7086,6 +7119,81 @@ mod tests { } } + /// The Windows `Fix` for fix-9, which only `diagnose.rs` prints: its verify + /// differs from the catalog's because it has to run in PowerShell. + fn windows_igpu_fix() -> Fix { + check_9_igpu_dgpu_collision(&igpu_dgpu_host("windows"), "torch crashes with a segfault") + .fix + .expect("an APU beside a discrete GPU on Windows gets a fix") + } + + /// Runs on every host. The Windows fix-9 verify sets `HIP_VISIBLE_DEVICES` + /// to `1` for the process, then runs one single-quoted python program, all + /// in one double-quoted `-Command` argument. + /// `the_igpu_verify_pins_the_variable_it_names` runs it, on Windows. + #[test] + fn the_windows_igpu_verify_is_one_quoted_powershell_argument() { + let verify = windows_igpu_fix().verify; + let body = verify + .strip_prefix("powershell -NoProfile -Command \"") + .and_then(|rest| rest.strip_suffix('"')) + .unwrap_or_else(|| panic!("not a quoted `powershell -Command` argument: {verify}")); + assert!( + !body.contains('"'), + "an inner quote ends the argument: {body}" + ); + assert!( + body.starts_with("[Environment]::SetEnvironmentVariable('HIP_VISIBLE_DEVICES','1','Process'); python -c '"), + "{body}" + ); + } + + /// Runs the fix-9 Windows verify line in cmd and in Windows PowerShell with + /// a stand-in `torch` module first on `PYTHONPATH`. The module's + /// `cuda.device_count()` returns what `HIP_VISIBLE_DEVICES` holds, so the + /// line, unchanged, prints `1` only if it set that variable, to that value, + /// before python started. The variable is removed from the child's + /// environment first, so a value left on the host cannot satisfy it. Python + /// is on PATH on the Windows CI hosts. Only compiled on Windows. + #[cfg(windows)] + #[test] + fn the_igpu_verify_pins_the_variable_it_names() { + let verify = windows_igpu_fix().verify; + let dir = std::env::temp_dir().join(format!("rocm igpu torch {}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("the stand-in module directory is created"); + std::fs::write( + dir.join("torch.py"), + "import os\n\nclass cuda:\n @staticmethod\n def device_count():\n \ + return os.environ.get('HIP_VISIBLE_DEVICES', 'UNSET')\n", + ) + .expect("the stand-in torch is written"); + let python_path = dir.to_string_lossy().into_owned(); + let runs = crate::fix::run_typed_in_cmd_and_powershell( + &verify, + &[ + ("HIP_VISIBLE_DEVICES", None), + ("PYTHONPATH", Some(python_path.as_str())), + ], + ); + let _ = std::fs::remove_dir_all(&dir); + for run in runs { + let crate::fix::ShellRun { + shell, + success, + status, + stdout, + stderr, + } = run; + assert!(success, "{shell}: {status}\n{stdout}\n{stderr}"); + assert!( + stderr.trim().is_empty(), + "{shell} wrote to stderr: {stderr}" + ); + assert_eq!(stdout.trim(), "1", "{shell}, stderr: {stderr}"); + } + } + /// What the Windows event log and Lemonade print when the llama.cpp server /// dies on a Visual C++ runtime that is too old. const MSVC_OUTDATED_CRASH_TEXT: &str = "llama-server.exe exited with code 3221225477. \ @@ -7335,6 +7443,320 @@ mod tests { assert!(!hit.title.contains("HIP DLLs cannot load"), "{}", hit.title); } + /// The `Fix` that each checker builds for itself is shown to users of + /// Windows PowerShell 5.1 as well as cmd, just like the `rocm fix` catalog. + /// This runs every checker that answers on Windows and applies the same + /// rule as the catalog test in `fix.rs` (`fails_in_windows_powershell`), so a + /// hand-built verify string or command that breaks one of the rule's known + /// constructs fails here. A construct the rule does not list passes. + /// + /// Like that test it pins the text only. Nothing here runs a command in a + /// shell. The Windows-only tests named on `fails_in_windows_powershell` in + /// `fix.rs` run the fix-2, fix-13, fix-14, fix-15, fix-21 and fix-22 lines, + /// and `the_igpu_verify_pins_the_variable_it_names` runs fix-9's, all on + /// Windows. + #[test] + fn windows_diagnosis_remediation_passes_the_powershell_text_rule() { + // Host state that wakes the checkers which answer from the machine and + // not from the symptom: an override set, a HIP SDK with no `hipInfo`, + // an APU beside a discrete GPU, and a runtime that is missing. + let e = Examination { + os_family: "windows".to_owned(), + msvc_redist_present: Some(false), + hip_sdk_path: r"C:\Program Files\AMD\ROCm\6.4".to_owned(), + hipinfo_present: false, + has_apu: true, + has_discrete_amd: true, + env: [ + ("HSA_OVERRIDE_GFX_VERSION".to_owned(), "11.0.0".to_owned()), + ("PATH".to_owned(), r"C:\Windows".to_owned()), + ] + .into_iter() + .collect(), + ..Examination::default() + }; + // Wide enough to wake the keyword-driven checkers, so the test sees + // the Fix they build rather than an empty answer. fix-13 only scores + // when the paste names HIP failing to load, hence `was not found`. + let symptom = "hipErrorNoBinaryForGpu HSA_STATUS_ERROR_INVALID_ISA invalid device function \ + HSA_OVERRIDE_GFX_VERSION vcruntime140_1.dll amdhip64_6.dll was not found \ + hipblas.dll driver too old"; + let mut seen: Vec = Vec::new(); + for (check, families) in CHECKERS { + if !families.contains(&"windows") { + continue; + } + let Some(fix) = check(&e, symptom).fix else { + continue; + }; + let printed = fix + .commands + .iter() + .map(String::as_str) + .filter(|c| !c.trim_start().starts_with('#')) + .chain(std::iter::once(fix.verify.as_str())); + for line in printed { + if let Some(why) = crate::fix::fails_in_windows_powershell(line) { + panic!("{}: `{line}` uses {why}", fix.fix_id); + } + } + seen.push(fix.fix_id); + } + // Guard the test itself: it must have looked at the Windows checkers. + for expected in [ + "fix-2-unset-override", + "fix-6-path", + "fix-9-igpu-dgpu", + "fix-13-hip-sdk-missing", + "fix-14-adrenalin-too-old", + "fix-15-msvc-redist", + ] { + assert!( + seen.iter().any(|id| id == expected), + "the sweep never saw {expected}; it saw {seen:?}" + ); + } + } + + /// fix-6's verify line lives in the `rocm fix` catalog and in the Fix that + /// `check_6_path_missing` builds. The shared-id guard above covers fix-13, + /// fix-14, fix-15 and fix-16; this one keeps fix-6's two copies from drifting. + /// + /// The Windows arm of the checker is not compared with the catalog on + /// purpose. It names the SDK's own `hipInfo.exe` in a PowerShell command, + /// which the single catalog line cannot do, so it is checked for being a + /// PowerShell command instead. Its text is also covered by the Windows sweep + /// above. + #[test] + fn the_path_fix_verify_matches_the_catalog_copy() { + let (verify, _, _) = crate::fix::recipe_verify_notes_and_run_commands("fix-6-path") + .expect("fix-6-path is in the recipe catalog"); + // A ROCm install whose bin directory has no `rocminfo` or `hipInfo`. + let host = |os_family: &str| Examination { + os_family: os_family.to_owned(), + rocm_path: "/opt/rocm".to_owned(), + hip_sdk_path: r"C:\Program Files\AMD\ROCm\6.4".to_owned(), + ..Examination::default() + }; + let linux = check_6_path_missing(&host("linux"), "hipcc: command not found") + .fix + .expect("check 6 builds a Fix for its own symptom"); + // The Linux arm runs the catalog's Linux line and nothing else. The + // line shared with Windows (`hipcc --version`) is not chained on: a + // runtime-only install has no `hipcc`, and the catalog note says that + // line fails there even with PATH fixed. + let catalog_linux = crate::fix::recipe_linux_verify("fix-6-path") + .expect("fix-6-path has a Linux verify line in the recipe catalog"); + assert_eq!( + linux.verify, catalog_linux, + "the verify line for fix-6-path diverged between `diagnose` and `rocm fix`" + ); + assert!( + !linux.verify.contains("hipcc") && !linux.verify.contains(verify), + "the Linux verify must pass on a runtime-only install, which has no hipcc: {}", + linux.verify + ); + let windows = check_6_path_missing(&host("windows"), "hipcc: command not found") + .fix + .expect("check 6 builds a Fix for its own symptom"); + assert!( + windows + .verify + .starts_with("powershell -NoProfile -Command "), + "the Windows verify line must run in PowerShell: {}", + windows.verify + ); + } + + /// A folder with an apostrophe in it, such as `C:\Users\O'Neil\ROCm\bin`, + /// must not end a PowerShell string literal early in either printed + /// fix-6 line: the PATH write or the verify. + #[test] + fn the_windows_path_fix_doubles_an_apostrophe_in_the_sdk_folder() { + let e = Examination { + os_family: "windows".to_owned(), + hip_sdk_path: r"C:\Users\O'Neil\ROCm".to_owned(), + hipinfo_present: false, + ..Examination::default() + }; + let fix = check_6_path_missing(&e, "hipInfo.exe not found") + .fix + .expect("check 6 builds a Fix for its own symptom"); + let write = fix + .commands + .iter() + .find(|c| c.contains("SetEnvironmentVariable")) + .expect("the PATH write is printed"); + assert!( + write.contains(r"'C:\Users\O''Neil\ROCm\bin;'"), + "the apostrophe must be doubled: {write}" + ); + assert!( + fix.verify + .contains(r"& 'C:\Users\O''Neil\ROCm\bin\hipInfo.exe'"), + "the apostrophe must be doubled: {}", + fix.verify + ); + } + + /// fix-2's Linux verify line is in the `rocm fix` catalog (`Verify (Linux, + /// WSL)`) and in the Linux arm of `check_2_hsa_override_unneeded`, which + /// runs it and then the GPU check the catalog line cannot carry (it is + /// shared with Windows and cannot assume torch). Neither may need `python`: + /// a stock Ubuntu has `python3` only. The Windows arm names the User scope + /// in a PowerShell command and is covered by the Windows sweep above. + #[test] + fn the_override_fix_verify_matches_the_catalog_copy() { + let catalog_linux = crate::fix::recipe_linux_verify("fix-2-unset-override") + .expect("fix-2-unset-override has a Linux verify line in the recipe catalog"); + let e = Examination { + os_family: "linux".to_owned(), + env: std::iter::once(("HSA_OVERRIDE_GFX_VERSION".to_owned(), "11.0.0".to_owned())) + .collect(), + ..Examination::default() + }; + let linux = check_2_hsa_override_unneeded(&e, "") + .fix + .expect("check 2 builds a Fix when the override is set"); + assert_eq!( + linux.verify, + format!( + "{catalog_linux}; python3 -c \"import torch; print(torch.cuda.is_available())\"" + ), + "the verify line for fix-2-unset-override diverged between `diagnose` and `rocm fix`" + ); + // A stock Ubuntu has `python3` and no `python`. + assert!( + !linux.verify.split_whitespace().any(|word| word == "python"), + "{}", + linux.verify + ); + // Clearing the override must leave the GPU usable; the printed line is + // what tells the user whether it did. + assert!( + linux.verify.contains("torch.cuda.is_available()"), + "{}", + linux.verify + ); + } + + /// Writes each `(name, script)` as an executable in a new directory, to be + /// the whole `PATH` of a shell that runs a Linux verify line. + #[cfg(unix)] + fn stub_bin_dir(stubs: &[(&str, &str)]) -> tempfile::TempDir { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().expect("the stub directory is created"); + for (name, script) in stubs { + let path = dir.path().join(name); + std::fs::write(&path, format!("#!/bin/sh\n{script}\n")).expect("a stub is written"); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o755)) + .expect("a stub is made executable"); + } + dir + } + + /// Runs one printed line in `/bin/sh` whose `PATH` is only `bin` and whose + /// environment is only `PATH` and `env`. Returns whether it exited 0, and + /// its stdout and stderr. + #[cfg(unix)] + fn run_in_sh( + line: &str, + bin: &std::path::Path, + env: &[(&str, &str)], + ) -> (bool, String, String) { + let out = std::process::Command::new("/bin/sh") + .arg("-c") + .arg(line) + .env_clear() + .env("PATH", bin) + .envs(env.iter().copied()) + .output() + .expect("/bin/sh runs"); + ( + out.status.success(), + String::from_utf8_lossy(&out.stdout).into_owned(), + String::from_utf8_lossy(&out.stderr).into_owned(), + ) + } + + /// The fix-6 line `rocm diagnose` prints on Linux, run on a runtime-only + /// install: `rocminfo` is on PATH once the fix has worked, and there is no + /// `hipcc`. The line must succeed and show the GPU. The old line, with + /// `&& hipcc --version` chained on, fails on the same PATH, which is what + /// the catalog note says about the plain verify line. + #[cfg(unix)] + #[test] + fn the_path_fix_verify_passes_on_a_runtime_only_linux_install() { + let real_head = ["/usr/bin/head", "/bin/head"] + .into_iter() + .find(|p| std::path::Path::new(p).exists()) + .expect("head is installed"); + let bin = stub_bin_dir(&[ + ( + "rocminfo", + "for n in 1 2 3 4 5 6 7 8; do echo \"agent $n\"; done", + ), + ("head", &format!("exec {real_head} \"$@\"")), + ]); + let fix = check_6_path_missing( + &Examination { + os_family: "linux".to_owned(), + rocm_path: "/opt/rocm".to_owned(), + ..Examination::default() + }, + "rocminfo: command not found", + ) + .fix + .expect("check 6 builds a Fix for its own symptom"); + let (ok, stdout, stderr) = run_in_sh(&fix.verify, bin.path(), &[]); + assert!(ok, "{}: {stdout}\n{stderr}", fix.verify); + assert_eq!( + stdout.lines().collect::>(), + ["agent 1", "agent 2", "agent 3", "agent 4", "agent 5"] + ); + // The control: the chain this arm used to print fails on this PATH. + let (old_ok, _, old_err) = + run_in_sh("rocminfo | head -n 5 && hipcc --version", bin.path(), &[]); + assert!(!old_ok && old_err.contains("hipcc"), "{old_err}"); + } + + /// The fix-2 line `rocm diagnose` prints on Linux, run on a system that has + /// `python3` and no `python`, with the override set and with it cleared. It + /// must print the override's value (or OK_UNSET) and then the torch check, + /// and exit 0. The line this arm used to print, which called `python`, + /// fails on the same PATH. + #[cfg(unix)] + #[test] + fn the_override_fix_verify_runs_where_only_python3_exists() { + let bin = stub_bin_dir(&[("python3", "echo True")]); + let fix = check_2_hsa_override_unneeded( + &Examination { + os_family: "linux".to_owned(), + env: std::iter::once(("HSA_OVERRIDE_GFX_VERSION".to_owned(), "11.0.0".to_owned())) + .collect(), + ..Examination::default() + }, + "", + ) + .fix + .expect("check 2 builds a Fix when the override is set"); + for (env, shown) in [ + (&[("HSA_OVERRIDE_GFX_VERSION", "11.0.0")][..], "11.0.0"), + (&[][..], "OK_UNSET"), + ] { + let (ok, stdout, stderr) = run_in_sh(&fix.verify, bin.path(), env); + assert!(ok, "{}: {stdout}\n{stderr}", fix.verify); + assert_eq!( + stdout.lines().collect::>(), + [shown, "True"], + "{stderr}" + ); + } + // The control: calling `python` on this PATH is what the reviewer saw. + let (old_ok, _, old_err) = run_in_sh("python -c 'print(1)'", bin.path(), &[]); + assert!(!old_ok && old_err.contains("python"), "{old_err}"); + } + #[test] fn keyword_score_takes_top_two() { // INVALID_ISA: two hits (50 + 40) -> 90, not the sum of all. diff --git a/crates/rocm-core/src/fix.rs b/crates/rocm-core/src/fix.rs index 4ce1b830ac..fda7ac1e1d 100644 --- a/crates/rocm-core/src/fix.rs +++ b/crates/rocm-core/src/fix.rs @@ -236,6 +236,14 @@ struct FixRecipe { needs_reboot: bool, needs_relogin: bool, verify: &'static str, + /// A second verify line for Linux and WSL, printed as `Verify (Linux, WSL):` + /// after `verify`. `None` when `verify` runs the same on every OS the + /// recipe applies on. `verify` is shown to Windows users, so it cannot + /// name a tool that only exists on Linux (and cannot chain with `&&`); + /// this field carries the Linux check without that limit. It is still + /// printed on Windows, labelled, as the `# Linux:` command lines are, so + /// the text rule for Windows lines applies to it too. + verify_linux: Option<&'static str>, notes: &'static [&'static str], /// Where to read more about this failure. `None` for most entries today -- /// an empty answer beats a guessed one, which would send a user somewhere @@ -288,6 +296,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "python -c \"import torch; print(torch.cuda.is_available(), torch.cuda.get_arch_list())\"", + verify_linux: None, notes: &[ "TheRock per-gfx wheels are the recommended fallback when the official pytorch index does not yet cover your gfx (and the only first-party option on Windows AMD).", "HSA_OVERRIDE_GFX_VERSION is NOT the right fix here -- it papers over the mismatch and risks page faults at runtime.", @@ -306,15 +315,27 @@ const RECIPES: &[FixRecipe] = &[ "# Linux:", "unset HSA_OVERRIDE_GFX_VERSION", "# Then remove the line from ~/.bashrc / ~/.zshrc / ~/.profile.", - "# Windows:", - "setx HSA_OVERRIDE_GFX_VERSION \"\"", + "# Windows (PowerShell):", + "[Environment]::SetEnvironmentVariable('HSA_OVERRIDE_GFX_VERSION',$null,'User')", "# Or remove via System Properties -> Environment Variables.", + "# This only affects NEW shells; close and reopen this terminal before running the verify command.", ], needs_sudo: false, needs_reboot: false, needs_relogin: false, - verify: "env | grep HSA_OVERRIDE_GFX_VERSION || echo OK_UNSET", - notes: &[], + // One line for cmd and Windows PowerShell (bash runs it too): it + // prints the override's value, or OK_UNSET. It needs `python`, which + // a Windows Python install and a virtual environment provide but a + // stock Ubuntu does not (that has `python3` only), so Linux and WSL get + // their own line below. + verify: "python -c \"import os; print(os.environ.get('HSA_OVERRIDE_GFX_VERSION') or 'OK_UNSET')\"", + // The same answer from the shell's own variable, with no Python and + // no external command, so it runs on a bare system. It has no `||`: + // this line is printed on Windows too, where the text rule refuses it. + verify_linux: Some("echo \"${HSA_OVERRIDE_GFX_VERSION:-OK_UNSET}\""), + notes: &[ + "Linux, WSL: run the `Verify (Linux, WSL)` line. It reads the shell's own variable and needs no Python. The plain `Verify` line runs `python`, which a Windows Python install and a virtual environment provide, but a stock Ubuntu install has only `python3`, so the plain line fails there with `command not found` even after the override is cleared.", + ], doc_url: None, applies_on: PRINT_ON_LINUX_AND_WSL_AUTO_ON_WINDOWS, runner: Some(run_unset_override), @@ -332,6 +353,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: true, needs_relogin: false, verify: "lsmod | grep amdgpu && rocminfo >/dev/null 2>&1", + verify_linux: None, notes: &[], doc_url: Some( "https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html", @@ -348,6 +370,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: true, verify: "groups | tr ' ' '\\n' | grep -E '^(render|video)$' && rocminfo | head -n 5", + verify_linux: None, notes: &[], doc_url: None, applies_on: AUTO_ON_LINUX, @@ -368,6 +391,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: true, needs_relogin: false, verify: "lsmod | grep amdgpu && rocminfo | head -n 5", + verify_linux: None, notes: &[ "If Secure Boot is enabled and amdgpu still won't load, the DKMS module isn't signed. Either sign it with mokutil or disable Secure Boot in firmware.", ], @@ -388,7 +412,12 @@ const RECIPES: &[FixRecipe] = &[ needs_sudo: false, needs_reboot: false, needs_relogin: false, - verify: "rocminfo | head -n 5 && hipcc --version", + // The line every OS can run, Windows included: `hipcc` ships with the + // HIP SDK there. Linux gets its own line below. + verify: "hipcc --version", + // `rocminfo` prints the GPU agents and needs no compiler, so it also + // passes on a runtime-only Linux install, which has no `hipcc`. + verify_linux: Some("rocminfo | head -n 5"), // The Windows command above is the manual-paste twin of what // `path_export_windows` actually runs -- same scope, same prepend // ordering -- so a user who pastes it and a user who lets `rocm fix` @@ -396,7 +425,8 @@ const RECIPES: &[FixRecipe] = &[ // the `setx` one-liner a Windows user would otherwise reach for. notes: &[ "Windows: the PowerShell line reads and rewrites the User scope only, and puts the SDK's bin dir first -- the same write `rocm fix fix-6-path` performs. Do not substitute `setx PATH \"%PATH%;...\"`: setx silently truncates the value at 1024 characters while still exiting 0, and %PATH% expands to the combined user+system PATH, so that form would copy every system entry into your User PATH.", - "Windows: a PATH change only takes effect in NEW shells -- close this terminal and reopen it before re-running hipInfo.", + "Windows: a PATH change only takes effect in NEW shells -- close this terminal and reopen it before running the verify command.", + "Linux, WSL: run the `Verify (Linux, WSL)` line. `rocminfo` prints the GPU and needs no compiler, so it passes on a runtime-only install. The plain `Verify` line runs `hipcc --version`, which exists on Windows (HIP SDK) and on a Linux install that has the compiler, and fails on a runtime-only Linux install even when PATH is fixed, because that install has no `hipcc`.", ], doc_url: None, applies_on: AUTO_ON_LINUX_WINDOWS_AND_WSL, @@ -425,6 +455,7 @@ const RECIPES: &[FixRecipe] = &[ bash -c 'set -o pipefail; sudo apt update 2>&1 | tail -n 20' # Debian/Ubuntu\n\ bash -c 'set -o pipefail; sudo dnf makecache 2>&1 | tail -n 20' # RHEL/Fedora/Rocky/Alma/Oracle\n\ bash -c 'set -o pipefail; sudo zypper refresh 2>&1 | tail -n 20' # SLES/openSUSE", + verify_linux: None, notes: &[], doc_url: None, applies_on: PRINT_ON_LINUX, @@ -447,6 +478,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "python -c \"import torch; print(torch.__version__, torch.version.hip, torch.cuda.is_available())\"", + verify_linux: None, notes: &[], doc_url: Some( "https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html", @@ -470,6 +502,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "python -c \"import torch; print(torch.cuda.device_count(), torch.cuda.get_device_name(0))\"", + verify_linux: None, notes: &[ "Pass --device-index N to persist the env var; without it, this fix only prints the rocminfo / hipInfo query so you can identify N.", ], @@ -494,6 +527,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "rocminfo | head -n 5", + verify_linux: None, notes: &[ "Rootless podman additionally needs `--userns=keep-id` and a host user that is in the render group; podman maps it through.", ], @@ -516,6 +550,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: true, needs_relogin: false, verify: "cat /proc/cmdline | grep -o 'iommu=\\w*'", + verify_linux: None, notes: &[], doc_url: None, applies_on: PRINT_ON_LINUX, @@ -556,6 +591,7 @@ const RECIPES: &[FixRecipe] = &[ || rpm -q amdgpu-dkms >/dev/null 2>&1) \ && dkms status amdgpu 2>/dev/null | grep -q installed \ && rocminfo >/dev/null 2>&1", + verify_linux: None, notes: &[ "If the purge/remove step warns it will take unrelated packages with it, stop and resolve those by hand before continuing.", "The `rocm install driver --dkms --yes` line and the verify below apply only on AMD-documented distros; `rocm install driver --dkms --yes` exits non-zero where it plans nothing (Ubuntu 26.04, undocumented distros). There, verify with `rocminfo` alone, or `dpkg-query -W rocm` after `sudo apt install rocm` on Ubuntu 26.04.", @@ -576,7 +612,8 @@ const RECIPES: &[FixRecipe] = &[ needs_sudo: false, needs_reboot: false, needs_relogin: false, - verify: "powershell -NoProfile -Command \"& \\\"$env:HIP_PATH\\bin\\hipInfo.exe\\\" | Select-Object -First 5\"", + verify: "powershell -NoProfile -Command \"& (Join-Path ([Environment]::GetEnvironmentVariable('HIP_PATH')) 'bin\\hipInfo.exe') | Select-Object -First 5\"", + verify_linux: None, notes: &[ "If you only need PyTorch on Windows AMD and don't need the C/C++ HIP toolchain, the TheRock wheels bundle their own HIP runtime and may not require a system HIP SDK install.", ], @@ -598,7 +635,8 @@ const RECIPES: &[FixRecipe] = &[ needs_sudo: false, needs_reboot: true, needs_relogin: false, - verify: "powershell -NoProfile -Command \"(Get-CimInstance Win32_VideoController | Where-Object { $_.Name -like '*AMD*' -or $_.Name -like '*Radeon*' } | Select-Object -First 1).DriverVersion\"", + verify: "powershell -NoProfile -Command \"(Get-CimInstance Win32_VideoController | Where-Object Name -match 'AMD|Radeon' | Select-Object -First 1).DriverVersion\"", + verify_linux: None, notes: &[], doc_url: Some( "https://rocm.docs.amd.com/projects/install-on-windows/en/latest/install/install.html", @@ -618,7 +656,8 @@ const RECIPES: &[FixRecipe] = &[ needs_sudo: false, needs_reboot: false, needs_relogin: false, - verify: "where vcruntime140.dll && where vcruntime140_1.dll", + verify: "powershell -NoProfile -Command \"(Get-Item ([Environment]::GetFolderPath('System') + '\\vcruntime140.dll'),([Environment]::GetFolderPath('System') + '\\vcruntime140_1.dll') -ErrorAction Stop).VersionInfo | Select-Object FileName,FileVersion\"", + verify_linux: None, notes: &[ "If installing the redistributable still leaves a missing-DLL error, the failing DLL is probably amdhip64_X.dll itself; that points at fix-13-hip-sdk-missing rather than this fix. If the runtime is present but the program crashes with an access violation, see fix-21-msvc-redist-outdated.", ], @@ -642,6 +681,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "rocm serve # re-run and watch for a clean startup", + verify_linux: None, notes: &[ "Only lower --gpu-memory-utilization when the GPU is shared or already busy; on a GPU dedicated to this server it cannot create the room a too-large model needs.", "This entry is keyword-matched from the error text: `rocm diagnose` cannot see per-GPU VRAM or tenancy, so pass the failure with --symptom (or arrive from the `rocm serve` failure note).", @@ -694,6 +734,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "rocm serve --engine vllm # then `rocm services list --all` and `rocm services logs ` to confirm the import no longer aborts", + verify_linux: None, notes: &[ "Running vLLM on ROCm is not by itself a reason to apply this. torch-c-dlpack-ext arrives as a transitive dependency of tilelang, which vLLM pins, and it only misbehaves on the torch versions it ships prebuilts for.", "The usual way a runtime lands in the failing range is `rocm install sdk` being re-run after the engine was installed, which overwrites the engine's pinned torch. Reinstalling the engine puts the pin back.", @@ -723,6 +764,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "ls -l /dev/dxg", + verify_linux: None, notes: &[ "A container running on WSL2 reports itself as WSL but sees /dev/dxg only when it was started with the device. Check that before touching the Windows driver.", ], @@ -746,6 +788,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "ls -l /usr/lib/wsl/lib/libdxcore.so && ldconfig -p | grep libdxcore", + verify_linux: None, notes: &["apt cannot repair /usr/lib/wsl: it is a mount supplied by WSL, not a package."], doc_url: None, applies_on: PRINT_ON_WSL, @@ -764,6 +807,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "ldconfig -p | grep librocdxg", + verify_linux: None, notes: &[ "Print-only on purpose: this downloads a .deb from a release page and installs it with sudo. `rocm install driver` prints the plan first so the URL and the package are reviewable before anything runs.", "The download is checked against a digest pinned for that ROCDXG release. To install a release rocm-cli has no digest for, set ROCM_CLI_ROCDXG_SHA256 to the one published with it.", @@ -781,6 +825,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "ldconfig -p | grep librocdxg", + verify_linux: None, notes: &[ "If ldconfig alone does not do it, the library landed outside the linker's search path: add that directory under /etc/ld.so.conf.d/ and re-run.", ], @@ -800,6 +845,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "f=/etc/os-release; [ -e \"$f\" ] || f=/usr/lib/os-release; grep VERSION_ID \"$f\" 2>/dev/null | head -1", + verify_linux: None, notes: &[ "Distros install side by side, so the current one can stay until the new one is set up.", ], @@ -819,6 +865,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "rocminfo | head -n 20", + verify_linux: None, notes: &[ "Nothing inside the distro can carry this out, which is why it prints rather than runs.", "The ROCm release and the Adrenalin release are paired; check the WSL install guide for the version that matches your ROCm.", @@ -840,6 +887,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "uname -r", + verify_linux: None, notes: &[ "Converting rewrites the distro's filesystem and can take a long time on a large install. Back up anything you cannot lose first.", ], @@ -870,6 +918,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "python -c \"import torch; torch.zeros(1, device='cuda')\"", + verify_linux: None, notes: &[ COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED, "This describes the environment outside the CLI's managed runtimes. `rocm serve` puts a managed runtime's libraries first on purpose, so inside one the wheel copy wins by design and that is correct.", @@ -902,6 +951,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "df -h /dev/shm", + verify_linux: None, notes: &[ "A running container cannot have its allowance changed. It has to be started again with the larger value.", "This is reported below 1 GiB. Silence is not proof of enough: a container given 2 GiB clears that bar and can still be too small for a large model.", @@ -933,7 +983,8 @@ const RECIPES: &[FixRecipe] = &[ needs_sudo: false, needs_reboot: false, needs_relogin: false, - verify: "powershell -NoProfile -Command \"(Get-Item $env:SystemRoot\\System32\\msvcp140.dll).VersionInfo.FileVersion\"", + verify: "powershell -NoProfile -Command \"(Get-Item ([Environment]::GetFolderPath('System') + '\\msvcp140.dll') -ErrorAction Stop).VersionInfo.FileVersion\"", + verify_linux: None, notes: &[ "The verify command prints the runtime version. It should read 14.40 or higher.", "`rocm fix` only prints this plan. It does not run the install: the install needs the Windows administrator prompt and your consent.", @@ -958,7 +1009,8 @@ const RECIPES: &[FixRecipe] = &[ needs_sudo: false, needs_reboot: true, needs_relogin: false, - verify: "powershell -NoProfile -Command \"Get-CimInstance Win32_VideoController | Where-Object { $_.Name -like '*AMD*' -or $_.Name -like '*Radeon*' } | Select-Object Name,DriverVersion,DriverDate\"", + verify: "powershell -NoProfile -Command \"Get-CimInstance Win32_VideoController | Where-Object Name -match 'AMD|Radeon' | Select-Object Name,DriverVersion,DriverDate\"", + verify_linux: None, notes: &[ crate::windows_driver::NO_MINIMUM_EXPLANATION, crate::windows_driver::FLOOR_NOTE, @@ -983,6 +1035,7 @@ const RECIPES: &[FixRecipe] = &[ needs_reboot: false, needs_relogin: false, verify: "", + verify_linux: None, notes: &[ "HSA_OVERRIDE_GFX_VERSION is NOT a workaround here: it makes the runtime load kernels built for another architecture, which faults at runtime rather than adding support.", // Not "`rocm serve` already picks a Vulkan engine on this host": @@ -1166,6 +1219,268 @@ pub(crate) fn recipe_verify_notes_and_run_commands( }) } +/// Test-only: the Linux and WSL verify line of a recipe, when it has one. +#[cfg(test)] +pub(crate) fn recipe_linux_verify(fix_id: &str) -> Option<&'static str> { + find_recipe(fix_id).and_then(|recipe| recipe.verify_linux) +} + +/// What one Windows shell did with a line typed into it. +#[cfg(all(test, windows))] +pub(crate) struct ShellRun { + pub shell: &'static str, + pub success: bool, + pub status: String, + pub stdout: String, + pub stderr: String, +} + +/// Runs `text` the way a user would type it into Windows PowerShell 5.1. +/// +/// The text goes on stdin (`-Command -`), which parses it as typed. Passing it +/// as an argument of an outer `-Command` is not the same thing: PowerShell +/// joins its arguments after removing their quotes, so the inner command text +/// would be cut at the first space and the pipe would run in the outer shell. +/// +/// `env` sets (`Some`) or removes (`None`) variables for the child only. +/// +/// `PSModulePath` is removed first, so Windows PowerShell 5.1 starts with its own +/// default module path. A test run from a PowerShell 7 session inherits that +/// session's path, which lists the PowerShell 7 modules first. 5.1 then picks +/// the PSReadLine built for 7, cannot load it, and writes "Cannot load PSReadline +/// module" to stderr, which the tests below treat as a failure of the line. +#[cfg(all(test, windows))] +pub(crate) fn run_typed_in_powershell(text: &str, env: &[(&str, Option<&str>)]) -> ShellRun { + use std::io::Write; + use std::process::{Command, Stdio}; + let mut command = Command::new("powershell"); + command.args(["-NoProfile", "-Command", "-"]); + command.env_remove("PSModulePath"); + for (name, value) in env { + match value { + Some(v) => command.env(name, v), + None => command.env_remove(name), + }; + } + command + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + let mut child = command.spawn().expect("powershell starts"); + let mut stdin = child.stdin.take().expect("stdin is piped"); + writeln!(stdin, "{text}").expect("the line is written"); + drop(stdin); + shell_run( + "powershell", + child.wait_with_output().expect("powershell finishes"), + ) +} + +/// Runs `line` the way a user would type it into cmd (`cmd /C`, with the line +/// passed unquoted and unescaped). `env` is as for [`run_typed_in_powershell`]. +#[cfg(all(test, windows))] +pub(crate) fn run_typed_in_cmd(line: &str, env: &[(&str, Option<&str>)]) -> ShellRun { + use std::os::windows::process::CommandExt; + use std::process::{Command, Stdio}; + let mut command = Command::new("cmd"); + command.arg("/C").raw_arg(line); + command.env_remove("PSModulePath"); + for (name, value) in env { + match value { + Some(v) => command.env(name, v), + None => command.env_remove(name), + }; + } + command + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + let out = command + .spawn() + .expect("cmd starts") + .wait_with_output() + .expect("cmd finishes"); + shell_run("cmd", out) +} + +/// Runs one printed line in cmd and in Windows PowerShell, in that order. +#[cfg(all(test, windows))] +pub(crate) fn run_typed_in_cmd_and_powershell( + line: &str, + env: &[(&str, Option<&str>)], +) -> Vec { + vec![ + run_typed_in_cmd(line, env), + run_typed_in_powershell(line, env), + ] +} + +#[cfg(all(test, windows))] +fn shell_run(shell: &'static str, out: std::process::Output) -> ShellRun { + ShellRun { + shell, + success: out.status.success(), + status: format!("{:?}", out.status), + stdout: String::from_utf8_lossy(&out.stdout).into_owned(), + stderr: String::from_utf8_lossy(&out.stderr).into_owned(), + } +} + +/// Where a `powershell ... -Command "..."` argument starts in `lowered` (the +/// line, lowercased): the byte offset of the flag, the quote that opens the +/// argument, and the text after that quote. +/// +/// PowerShell accepts any prefix of `-Command` down to `-c`, so `-c` and +/// `-comm` are found as well. +#[cfg(test)] +fn command_argument(lowered: &str) -> Option<(usize, char, &str)> { + lowered.match_indices('-').find_map(|(at, _)| { + let head = &lowered[..at]; + // Only a PowerShell line: `python -c "..."` has the same flag. + if head.chars().next_back().is_some_and(|c| !c.is_whitespace()) + || !(head.contains("powershell") || head.contains("pwsh")) + { + return None; + } + let rest = &lowered[at..]; + let flag = rest.split_whitespace().next()?; + if flag.len() < 2 || !"-command".starts_with(flag) { + return None; + } + let after = rest[flag.len()..].trim_start(); + let quote = after.chars().next().filter(|c| matches!(c, '"' | '\''))?; + Some((at, quote, &after[1..])) + }) +} + +/// Test-only rule for what a printed command must not contain when it is shown +/// to a user of Windows PowerShell 5.1 as well as cmd: +/// +/// - `&&` and `||`, which are syntax errors in Windows PowerShell 5.1; +/// - a bare `where`, which starts a command and is the `Where-Object` alias in +/// PowerShell, not `where.exe`. A command starts after `|`, `;`, `(`, `{` and +/// a lone `&` (cmd's unconditional separator), at the start of a +/// `-Command "..."` body, and after `cmd /c`, so `echo x & where foo` is +/// caught. The `&` of a call operator (`& 'path\exe'`) is split the same way +/// and stays legal, because the word after it is not `where`; +/// - a `%NAME%` reference, which PowerShell passes on as literal text. For +/// `setx PATH "%PATH%;..."` that writes `%PATH%` into the user's Path; +/// - a `-Command` (or `-c`, or any longer prefix of it) argument in single +/// quotes. cmd does not treat `'` as a quote, so it splits the argument at +/// its spaces; +/// - inside the double-quoted body of a `powershell ... -Command "..."` line, a +/// backslash-escaped quote (`\"`) or any `$`. PowerShell does not take `\` as +/// an escape, so typed into PowerShell the `\"` ends the string early; and +/// an outer PowerShell expands `$_`, `$env:NAME` and the like before the inner +/// one starts, so they would be empty or hold the outer value, and a value +/// with a space in it would split the inner command. Use single quotes inside +/// the body, `[Environment]::GetEnvironmentVariable('NAME')` for a variable, +/// `[Environment]::GetFolderPath('System')` for the system directory, and a +/// cmdlet that needs no `$_`. There is no exemption. The `-Command` match +/// ignores case; +/// - outside such a body, an empty-string argument (`""` or `''`). Windows +/// PowerShell 5.1 drops it when it calls a native program, so +/// `setx NAME ""` reaches setx as `setx NAME`, which fails with "Invalid +/// syntax" and clears nothing. Use a .NET call such as +/// `[Environment]::SetEnvironmentVariable('NAME',$null,'User')`; +/// - inside a `-Command` body, a `Get-Item` that does not carry +/// `-ErrorAction Stop`. `Get-Item` reports a missing path as a +/// non-terminating error and PowerShell exits 0 after one, so a verify that +/// checks a file this way would pass with the file missing. Other cmdlets +/// that behave this way are not listed. +/// +/// It is a text rule over those known-bad constructs. It is not a parser: it +/// does not prove a line is valid or that it runs, and a construct it does not +/// list passes. On Windows, the verify lines of fix-2, fix-13, fix-14, fix-15, +/// fix-21 and fix-22 are run by `the_override_verify_runs_in_cmd_and_powershell`, +/// `the_hip_sdk_verify_runs_the_exe_under_hip_path`, +/// `the_adrenalin_verify_runs_and_its_filter_keeps_only_amd`, +/// `the_msvc_verify_runs_in_cmd_and_powershell` (fix-15 again in +/// `the_msvc_verify_exits_nonzero_when_a_dll_is_missing`), +/// `the_outdated_msvc_verify_prints_a_version_and_fails_when_the_dll_is_missing` +/// and `the_display_driver_verify_runs_in_cmd_and_powershell`; fix-9's Windows +/// line, which only `diagnose.rs` prints, by +/// `the_igpu_verify_pins_the_variable_it_names`. +/// Some lines `diagnose.rs` prints are PowerShell-only by design, and are +/// checked as such where they are built. +/// +/// Shared by the checks on the `RECIPES` catalog here and on the `Fix` values +/// that `diagnose.rs` builds. A copy of this rule lives in the e2e crate +/// (`powershell_text_rule::refusal`), which cannot depend on rocm-core; +/// `the_powershell_text_rule_agrees_with_the_shared_case_table` and the test +/// next to that copy both read `tests/e2e-cucumber/fixtures/powershell_text_rule_cases.txt`, +/// so a case one rule gets wrong fails on that side. +#[cfg(test)] +pub(crate) fn fails_in_windows_powershell(line: &str) -> Option<&'static str> { + if line.contains("&&") || line.contains("||") { + return Some("`&&` or `||`, a syntax error in Windows PowerShell 5.1"); + } + // `-Command` is not case-sensitive, and neither is this match. + let lowered = line.to_ascii_lowercase(); + let command = command_argument(&lowered); + // An opening brace is written as an escape on purpose: the advised-commands + // scanner skips a cfg(test) item by counting braces, and a lone one throws it off. + let starts_bare_where = |text: &str| { + text.split(['|', ';', '&', '(', '\u{7b}']).any(|part| { + let mut words = part.split_whitespace().map(|w| w.trim_matches('"')); + let mut first = words.next(); + if first == Some("cmd") { + first = words.find(|w| !w.starts_with('/')); + } + first == Some("where") + }) + }; + if starts_bare_where(&lowered) + || command.is_some_and(|(_, _, body)| starts_bare_where(body.trim_start())) + { + return Some("a bare `where`, which is `Where-Object` in PowerShell"); + } + // The text outside a `-Command "..."` body, which is typed at the outer + // shell and so is where an argument can be dropped. + let outer = command.map_or(lowered.as_str(), |(at, _, _)| &lowered[..at]); + if outer + .split_whitespace() + .any(|word| word == "\"\"" || word == "''") + { + return Some( + "an empty-string argument (`\"\"` or `''`), which Windows PowerShell 5.1 drops when it calls a native program", + ); + } + if let Some((_, quote, body)) = command { + if quote == '\'' { + return Some( + "a single-quoted `-Command` argument, which cmd does not take as one argument", + ); + } + if body.contains("\\\"") { + return Some( + "a backslash-escaped quote inside a `powershell -Command \"...\"` body, which ends the string early when typed into PowerShell", + ); + } + if body.contains('$') { + return Some( + "a `$` inside a `powershell -Command \"...\"` body (`$_`, `$env:NAME`), which an outer PowerShell expands before the inner one starts", + ); + } + if body.contains("get-item") && !body.contains("-erroraction stop") { + return Some( + "a `Get-Item` without `-ErrorAction Stop`: a missing path is a non-terminating error, so the line prints it and still exits 0", + ); + } + } + // The text between a `%` and the next one. The last piece has no closing + // `%`, so it is never a name. + let pieces: Vec<&str> = line.split('%').collect(); + let names_a_cmd_variable = pieces + .get(1..pieces.len().saturating_sub(1)) + .unwrap_or_default() + .iter() + .any(|name| { + !name.is_empty() && name.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'_') + }); + names_a_cmd_variable.then_some("a `%NAME%` reference, which PowerShell does not expand") +} + /// Test-only view of the platform families a recipe is runnable on. /// /// `None` when no recipe carries `fix_id`. Lets the diagnosis catalog assert @@ -1449,12 +1764,45 @@ fn print_recipe(r: &FixRecipe) { for n in r.notes { println!("Note: {n}"); } + for line in verify_lines(r) { + println!("{line}"); + } +} + +/// The `Verify` lines of the plan `print_recipe` prints, in print order: the +/// line every OS can run, then the Linux and WSL line when the recipe has one. +/// Split out so the labels are testable, as `format_flags` is. A multi-line +/// line keeps its continuation under the label. +fn verify_lines(r: &FixRecipe) -> Vec { + let mut lines = Vec::new(); if !r.verify.is_empty() { - println!( + lines.push(format!( "Verify: {}", indent_continuation(r.verify, " ") - ); + )); + } + if let Some(linux) = r.verify_linux { + lines.push(format!( + "Verify (Linux, WSL): {}", + indent_continuation(linux, " ") + )); + } + lines +} + +/// The commands a print-only plan tells the user to run to verify, as printed. +fn verify_steps(r: &FixRecipe) -> Vec { + let mut steps = Vec::new(); + if !r.verify.is_empty() { + steps.push(format!(" $ {}", indent_continuation(r.verify, " "))); + } + if let Some(linux) = r.verify_linux { + steps.push(format!( + " $ {} # Linux, WSL", + indent_continuation(linux, " ") + )); } + steps } /// Indent every line after the first so a multi-line `verify` (fix-7 lists one @@ -1590,8 +1938,8 @@ fn act_on(recipe: &FixRecipe, opts: &FixOptions) -> i32 { Plan::PrintSteps => { println!("This fix is print-only (manual change required)."); println!("Copy the commands above, run them yourself, then verify with:"); - if !recipe.verify.is_empty() { - println!(" $ {}", indent_continuation(recipe.verify, " ")); + for step in verify_steps(recipe) { + println!("{step}"); } exit::OK } @@ -1863,11 +2211,21 @@ fn report_and_clear_override_windows( let mut declined = false; if !user_val.is_empty() { println!("\nClear from the User scope (no admin needed):"); - println!(" Will run: setx HSA_OVERRIDE_GFX_VERSION \"\""); + // What the CLI runs, not a line to paste: the process is started + // directly with these arguments, empty one included. Windows PowerShell + // 5.1 drops an empty argument, so pasting the line would not clear + // anything; the plan above carries the line to paste. The text is built + // from the argument list that is run, so the two cannot differ. + let setx_args = ["HSA_OVERRIDE_GFX_VERSION", ""]; + let shown: Vec = setx_args.iter().map(|a| format!("\"{a}\"")).collect(); + println!( + " The CLI runs (arguments, not a line to paste): setx {}", + shown.join(" ") + ); if opts.dry_run { println!(" (dry-run; not executed)"); } else if consent("Clear HSA_OVERRIDE_GFX_VERSION from User scope?") { - let (rc, out, err) = run("setx", &["HSA_OVERRIDE_GFX_VERSION", ""], RUN_TIMEOUT); + let (rc, out, err) = run("setx", &setx_args, RUN_TIMEOUT); relay_output(&out, &err); if rc != 0 { fail!("setx exited {rc}; User scope NOT changed."); @@ -2498,6 +2856,7 @@ mod tests { needs_reboot: false, needs_relogin: false, verify: "", + verify_linux: None, notes: &[], doc_url: None, applies_on: Box::leak(Box::new([on(current_os(), class)])), @@ -2793,6 +3152,670 @@ mod tests { } } + /// The commands and the verify line of a recipe that applies on Windows + /// are shown to users of Windows PowerShell and of cmd, so none may use + /// what `fails_in_windows_powershell` lists. The set of recipes is derived + /// from `applies_on`, so a new Windows recipe is covered without editing + /// this test. + /// + /// This pins the text only. The Windows-only tests named on + /// `fails_in_windows_powershell` run the verify lines of fix-2, fix-13, + /// fix-14, fix-15, fix-21 and fix-22 in a shell; the other catalog lines + /// are not run. The same + /// rule is applied to the `Fix` values that `diagnose.rs` builds, in + /// `windows_diagnosis_remediation_passes_the_powershell_text_rule`. + #[test] + fn windows_recipes_pass_the_powershell_text_rule() { + let windows: Vec<&FixRecipe> = RECIPES + .iter() + .filter(|r| r.platform("windows").is_some()) + .collect(); + assert!( + windows.iter().any(|r| r.fix_id == "fix-15-msvc-redist"), + "the derivation found no Windows recipes: {windows:?}", + windows = windows.iter().map(|r| r.fix_id).collect::>() + ); + for r in windows { + let printed = r + .commands + .iter() + .copied() + .filter(|c| !c.trim_start().starts_with('#')) + .chain(std::iter::once(r.verify)) + .chain(r.verify_linux); + for line in printed { + if let Some(why) = fails_in_windows_powershell(line) { + panic!("{}: `{line}` uses {why}", r.fix_id); + } + } + } + } + + /// fix-6's plan carries one verify line that runs on Windows and one for + /// Linux and WSL that shows the GPU. The plain line is what Windows runs, so + /// it cannot need `rocminfo`; the Linux line must not need `hipcc`, which a + /// runtime-only install lacks. The note says exactly that, so the two + /// claims in it are asserted against the lines it names. + #[test] + fn the_path_fix_verifies_the_gpu_on_linux_without_needing_hipcc() { + let recipe = find_recipe("fix-6-path").expect("fix-6 is in the catalog"); + let linux = recipe.verify_linux.expect("fix-6 has a Linux verify line"); + assert!( + linux.starts_with("rocminfo") && !linux.contains("hipcc"), + "the Linux line must show the GPU and not need hipcc: {linux}" + ); + assert!( + !recipe.verify.contains("rocminfo"), + "the plain line is shown to Windows users, where rocminfo does not exist: {}", + recipe.verify + ); + assert_eq!( + verify_lines(recipe), + [ + "Verify: hipcc --version", + "Verify (Linux, WSL): rocminfo | head -n 5" + ] + ); + let note = recipe + .notes + .iter() + .find(|n| n.contains("runtime-only")) + .expect("a note names the runtime-only install"); + assert!( + note.contains("hipcc") && note.contains("Verify (Linux, WSL)"), + "{note}" + ); + } + + /// fix-2's plan carries the line every OS can run (`python`, as cmd and + /// PowerShell run it) and a Linux and WSL line that does not need Python, + /// because a stock Ubuntu has `python3` only. The note names both lines, so + /// the claims in it are asserted against what is printed: the Linux line + /// really needs no `python` and no external command, and it reports the + /// same answer as the plain line (the override's value, or OK_UNSET). + // The `${...}` in the expected line is shell text, not a format argument. + #[allow(clippy::literal_string_with_formatting_args)] + #[test] + fn the_override_fix_has_a_linux_verify_line_that_needs_no_python() { + let recipe = find_recipe("fix-2-unset-override").expect("fix-2 is in the catalog"); + let linux = recipe.verify_linux.expect("fix-2 has a Linux verify line"); + assert_eq!( + verify_lines(recipe), + [ + format!("Verify: {}", recipe.verify), + "Verify (Linux, WSL): echo \"${HSA_OVERRIDE_GFX_VERSION:-OK_UNSET}\"".to_owned() + ] + ); + assert!( + recipe.verify.starts_with("python -c "), + "the plain line is the one that runs in cmd and PowerShell: {}", + recipe.verify + ); + assert!(!linux.contains("python"), "{linux}"); + let note = recipe + .notes + .iter() + .find(|n| n.contains("python3")) + .expect("a note names the stock-Ubuntu python3 case"); + assert!( + note.contains("Verify (Linux, WSL)") && note.contains("`python`"), + "{note}" + ); + } + + /// The fix-2 Linux line, run in `/bin/sh` with a `PATH` that holds nothing + /// (no `python`, no `env`, no `grep`): it must still report the override's + /// value, or OK_UNSET once it is cleared, and exit 0. The plain `python` + /// line fails on this PATH. + #[cfg(unix)] + #[test] + fn the_override_linux_verify_runs_with_nothing_on_path() { + let linux = find_recipe("fix-2-unset-override") + .and_then(|r| r.verify_linux) + .expect("fix-2 has a Linux verify line"); + let empty = tempfile::tempdir().expect("an empty PATH directory is created"); + let run = |line: &str, value: Option<&str>| { + let mut command = std::process::Command::new("/bin/sh"); + command + .arg("-c") + .arg(line) + .env_clear() + .env("PATH", empty.path()); + if let Some(value) = value { + command.env("HSA_OVERRIDE_GFX_VERSION", value); + } + let out = command.output().expect("/bin/sh runs"); + ( + out.status.success(), + String::from_utf8_lossy(&out.stdout).trim().to_owned(), + ) + }; + assert_eq!(run(linux, Some("11.0.0")), (true, "11.0.0".to_owned())); + assert_eq!(run(linux, None), (true, "OK_UNSET".to_owned())); + let plain = find_recipe("fix-2-unset-override").unwrap().verify; + assert!(!run(plain, None).0, "the plain line needs python on PATH"); + } + + /// A Linux verify line is only worth having on a recipe that applies on + /// Linux or WSL; on any other it could never be run. + #[test] + fn a_linux_verify_line_belongs_to_a_recipe_that_applies_on_linux_or_wsl() { + let with_one: Vec<&FixRecipe> = RECIPES + .iter() + .filter(|r| r.verify_linux.is_some()) + .collect(); + assert!( + !with_one.is_empty(), + "no recipe carries a Linux verify line" + ); + for r in with_one { + assert!( + r.platform("linux").is_some() || r.platform("wsl").is_some(), + "{} has a Linux verify line but does not apply on Linux or WSL", + r.fix_id + ); + } + } + + #[test] + fn a_print_only_plan_labels_its_linux_verify_step() { + let mut recipe = unfixable_recipe(FixClass::PrintOnly); + recipe.verify = "hipcc --version"; + assert_eq!(verify_steps(&recipe), [" $ hipcc --version"]); + recipe.verify_linux = Some("rocminfo | head -n 5"); + assert_eq!( + verify_steps(&recipe), + [ + " $ hipcc --version", + " $ rocminfo | head -n 5 # Linux, WSL" + ] + ); + recipe.verify = ""; + recipe.verify_linux = None; + assert!(verify_steps(&recipe).is_empty() && verify_lines(&recipe).is_empty()); + } + + /// The fix-15 verify line, taken apart. This is a structural check on the + /// text, so it runs on every host: a misspelled cmdlet or environment + /// variable fails here even where no Windows shell is available. It does not + /// replace running the line. `the_msvc_verify_runs_in_cmd_and_powershell` + /// does that, and only on Windows. + #[test] + fn the_msvc_verify_is_a_wellformed_powershell_command() { + let verify = find_recipe("fix-15-msvc-redist") + .expect("fix-15 is in the catalog") + .verify; + // Starts with `powershell` so cmd runs it too. The body is one + // double-quoted argument with no `$`, so an outer PowerShell has + // nothing to expand, whatever the system directory's path holds. + let body = verify + .strip_prefix("powershell -NoProfile -Command \"") + .and_then(|rest| rest.strip_suffix('"')) + .unwrap_or_else(|| panic!("not a quoted `powershell -Command` argument: {verify}")); + assert!( + !body.contains('"'), + "an inner quote would end the argument: {body}" + ); + // Every command word is a real cmdlet, and the Windows directory is + // read inside the body by .NET rather than from a variable. + let cmdlets: Vec<&str> = body + .split('|') + .map(|stage| { + stage + .trim() + .trim_start_matches('(') + .split_whitespace() + .next() + .unwrap_or("") + }) + .collect(); + assert_eq!(cmdlets, ["Get-Item", "Select-Object"], "{body}"); + assert!( + !body.contains('$'), + "an outer PowerShell would expand it: {body}" + ); + assert_eq!( + body.matches("[Environment]::GetFolderPath('System')") + .count(), + 2, + "{body}" + ); + for dll in ["vcruntime140.dll", "vcruntime140_1.dll"] { + assert!(body.contains(dll), "{dll} is not checked: {body}"); + } + assert!( + body.contains(".VersionInfo") && body.ends_with("Select-Object FileName,FileVersion"), + "{body}" + ); + } + + /// Runs the fix-15 verify line the way a user would: typed into cmd, and + /// typed into Windows PowerShell. Both must exit 0 and print each DLL on its own + /// row with a dotted version. The runtime is installed on the Windows CI hosts, so a + /// failure here means the line is broken. + /// + /// The PowerShell leg feeds the line on stdin (`-Command -`), which parses + /// it as typed text. Passing it as an argument of an outer `-Command` is + /// not the same thing: PowerShell joins its arguments after removing their + /// quotes, so the inner command text would be cut at the first space and + /// the pipe would run in the outer shell. + /// + /// Only compiled on Windows. Linux and macOS lanes do not run it, which is + /// why `the_msvc_verify_is_a_wellformed_powershell_command` also checks the + /// text on every host. + #[cfg(windows)] + #[test] + fn the_msvc_verify_runs_in_cmd_and_powershell() { + let verify = find_recipe("fix-15-msvc-redist") + .expect("fix-15 is in the catalog") + .verify; + for run in run_typed_in_cmd_and_powershell(verify, &[]) { + let ShellRun { + shell, + success, + status, + stdout: text, + stderr: err, + } = run; + assert!(success, "{shell}: {status}\n{text}\n{err}"); + assert!(err.trim().is_empty(), "{shell} wrote to stderr: {err}"); + // `Select-Object FileName,FileVersion` prints a table with one row + // per DLL: the path, then the version, on one line. Check each DLL + // on its own row, so a pairing that lost or crossed a version (a + // row with a name and no version, or both names on one row) fails. + // `vcruntime140.dll` is not a substring of `vcruntime140_1.dll`, + // so each name picks out exactly one row. This proves each listed + // DLL has a dotted version beside it; it cannot prove the version + // is the right one for that file. A path wrapped onto a second + // line would fail here, and then the test is wrong, not the line. + let is_version = |w: &str| { + let parts: Vec<&str> = w.split('.').collect(); + parts.len() >= 3 + && parts + .iter() + .all(|p| !p.is_empty() && p.bytes().all(|b| b.is_ascii_digit())) + }; + for dll in ["vcruntime140.dll", "vcruntime140_1.dll"] { + let rows: Vec<&str> = text.lines().filter(|l| l.contains(dll)).collect(); + assert_eq!( + rows.len(), + 1, + "{shell}: expected one row for {dll}:\n{text}" + ); + let row = rows[0]; + assert!( + row.split_whitespace().any(is_version), + "{shell}: the {dll} row has no dotted version:\n{text}" + ); + let other = if dll == "vcruntime140.dll" { + "vcruntime140_1.dll" + } else { + "vcruntime140.dll" + }; + assert!( + !row.contains(other), + "{shell}: {dll} and {other} share a row:\n{text}" + ); + } + } + } + + /// A DLL that is missing must fail the fix-15 verify. `Get-Item` reports a + /// missing path as a non-terminating error, and PowerShell exits 0 after one, + /// so without `-ErrorAction Stop` the line printed an error for the missing + /// DLL and still exited 0. The line is run with the name of the second DLL + /// changed to one that does not exist; the system directory cannot be + /// redirected to one without it, because PowerShell does not start under a + /// `SystemRoot` that is not the real one. Only compiled on Windows. + #[cfg(windows)] + #[test] + fn the_msvc_verify_exits_nonzero_when_a_dll_is_missing() { + let verify = find_recipe("fix-15-msvc-redist") + .expect("fix-15 is in the catalog") + .verify; + let broken = verify.replace("vcruntime140_1.dll", "vcruntime140_absent.dll"); + assert_ne!( + broken, verify, + "the recipe no longer names vcruntime140_1.dll" + ); + for run in run_typed_in_cmd_and_powershell(&broken, &[]) { + let ShellRun { + shell, + success, + stdout, + stderr, + .. + } = run; + assert!( + !success, + "{shell} exited 0 with a DLL missing:\n{stdout}\n{stderr}" + ); + assert!( + stderr.contains("vcruntime140_absent.dll"), + "{shell} did not name the missing DLL on stderr:\n{stderr}" + ); + } + } + + /// fix-21's verify line, run typed into cmd and into Windows PowerShell: + /// it prints the dotted version of `msvcp140.dll`. With the DLL's name + /// changed to one that does not exist, both shells must exit non-zero and + /// name the file: `Get-Item` reports a missing path as a non-terminating + /// error, so without `-ErrorAction Stop` the line exits 0 with no output + /// (the case `the_msvc_verify_exits_nonzero_when_a_dll_is_missing` covers + /// for fix-15). `diagnose.rs` holds its copy of the line equal to this one. + /// Only compiled on Windows. + #[cfg(windows)] + #[test] + fn the_outdated_msvc_verify_prints_a_version_and_fails_when_the_dll_is_missing() { + let verify = find_recipe("fix-21-msvc-redist-outdated") + .expect("fix-21 is in the catalog") + .verify; + for run in run_typed_in_cmd_and_powershell(verify, &[]) { + let ShellRun { + shell, + success, + status, + stdout, + stderr, + } = run; + assert!(success, "{shell}: {status}\n{stdout}\n{stderr}"); + assert!( + stderr.trim().is_empty(), + "{shell} wrote to stderr: {stderr}" + ); + let version = stdout.trim(); + let parts: Vec<&str> = version.split('.').collect(); + assert!( + parts.len() >= 3 + && parts + .iter() + .all(|p| !p.is_empty() && p.bytes().all(|b| b.is_ascii_digit())), + "{shell}: not a dotted version: {stdout}" + ); + } + let broken = verify.replace("msvcp140.dll", "msvcp140_absent.dll"); + assert_ne!(broken, verify, "the recipe no longer names msvcp140.dll"); + for run in run_typed_in_cmd_and_powershell(&broken, &[]) { + let ShellRun { + shell, + success, + stdout, + stderr, + .. + } = run; + assert!( + !success, + "{shell} exited 0 with the DLL missing:\n{stdout}\n{stderr}" + ); + assert!( + stderr.contains("msvcp140_absent.dll"), + "{shell} did not name the missing DLL on stderr:\n{stderr}" + ); + } + } + + /// Runs the fix-2 verify line typed into cmd and into Windows PowerShell, + /// once with the override unset and once with it set. Python is on PATH on + /// the Windows CI hosts. Only compiled on Windows; the line's shape is + /// checked on every host by `the_override_verify_is_one_line_for_every_shell`. + #[cfg(windows)] + #[test] + fn the_override_verify_runs_in_cmd_and_powershell() { + let verify = find_recipe("fix-2-unset-override").unwrap().verify; + for (value, expected) in [(None, "OK_UNSET"), (Some("11.0.0"), "11.0.0")] { + for run in + run_typed_in_cmd_and_powershell(verify, &[("HSA_OVERRIDE_GFX_VERSION", value)]) + { + let ShellRun { + shell, + success, + status, + stdout: text, + stderr: err, + } = run; + assert!(success, "{shell}: {status}\n{text}\n{err}"); + assert_eq!(text.trim(), expected, "{shell}, stderr: {err}"); + } + } + } + + /// The text of fix-13's verify, taken apart. Runs on every host. It reads + /// `HIP_PATH`, joins `bin\hipInfo.exe` onto it as a single-quoted string, + /// and keeps the first lines; `the_hip_sdk_verify_runs_the_exe_under_hip_path` + /// runs it, on Windows. + #[test] + fn the_hip_sdk_verify_reads_hip_path_and_runs_hipinfo() { + let verify = find_recipe("fix-13-hip-sdk-missing").unwrap().verify; + let body = verify + .strip_prefix("powershell -NoProfile -Command \"") + .and_then(|rest| rest.strip_suffix('"')) + .unwrap_or_else(|| panic!("not a quoted `powershell -Command` argument: {verify}")); + assert!( + !body.contains('"'), + "an inner quote ends the argument: {body}" + ); + assert!( + body.starts_with("& (Join-Path ([Environment]::GetEnvironmentVariable('HIP_PATH')) 'bin\\hipInfo.exe')") + && body.ends_with("| Select-Object -First 5"), + "{body}" + ); + } + + /// The Where-Object stage of fix-14's verify, as written in the recipe. + /// The Windows test and the shape test both take the filter from here, so + /// neither keeps a copy of it. + fn adrenalin_filter(verify: &str) -> &str { + verify + .split(" | ") + .find(|stage| stage.starts_with("Where-Object")) + .unwrap_or_else(|| panic!("fix-14's verify has no Where-Object stage: {verify}")) + } + + /// Runs on every host: the filter that picks the AMD adapter is the one + /// the Windows test exercises, so it has to be found and be the AMD match. + #[test] + fn the_adrenalin_verify_filters_on_amd_or_radeon() { + let verify = find_recipe("fix-14-adrenalin-too-old").unwrap().verify; + assert_eq!( + adrenalin_filter(verify), + "Where-Object Name -match 'AMD|Radeon'" + ); + } + + /// Runs the fix-13 verify line in cmd and in Windows PowerShell against a + /// throwaway SDK directory whose name contains a space: the single-quoted + /// Join-Path form exists to survive one. `whoami.exe` stands in for + /// `hipInfo.exe`; nothing else is reachable through `HIP_PATH`, so output + /// means the line ran the file it built from `HIP_PATH`. With `HIP_PATH` + /// pointing at a directory that has no `hipInfo.exe`, the line must fail + /// and say so, which shows the variable is the one it reads. Only compiled + /// on Windows. + #[cfg(windows)] + #[test] + fn the_hip_sdk_verify_runs_the_exe_under_hip_path() { + let verify = find_recipe("fix-13-hip-sdk-missing").unwrap().verify; + let root = std::env::temp_dir().join(format!("rocm hip sdk {}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(root.join("bin")).expect("the SDK directory is created"); + let system_root = std::env::var("SystemRoot").expect("SystemRoot is set on Windows"); + std::fs::copy( + std::path::Path::new(&system_root).join("System32\\whoami.exe"), + root.join("bin").join("hipInfo.exe"), + ) + .expect("a stand-in hipInfo.exe is copied"); + let empty = root.join("empty"); + std::fs::create_dir_all(&empty).unwrap(); + let hip_path = root.to_string_lossy().into_owned(); + let empty_path = empty.to_string_lossy().into_owned(); + + let present = + run_typed_in_cmd_and_powershell(verify, &[("HIP_PATH", Some(hip_path.as_str()))]); + let missing = + run_typed_in_cmd_and_powershell(verify, &[("HIP_PATH", Some(empty_path.as_str()))]); + let _ = std::fs::remove_dir_all(&root); + + for run in present { + let ShellRun { + shell, + success, + status, + stdout, + stderr, + } = run; + assert!(success, "{shell}: {status}\n{stdout}\n{stderr}"); + assert!( + stderr.trim().is_empty(), + "{shell} wrote to stderr: {stderr}" + ); + assert!( + !stdout.trim().is_empty(), + "{shell} printed nothing from the stand-in hipInfo.exe" + ); + } + for run in missing { + let ShellRun { + shell, + success, + stdout, + stderr, + .. + } = run; + assert!( + !(success && stderr.trim().is_empty()) && stdout.trim().is_empty(), + "{shell} ran something with no hipInfo.exe under HIP_PATH:\n{stdout}\n{stderr}" + ); + } + } + + /// Runs the fix-22 verify line typed into cmd and into Windows PowerShell: + /// both must parse it and exit 0 with nothing on stderr. The line used a + /// `{ $_.Name ... }` filter, and `$_` inside the double-quoted `-Command` + /// body is expanded by an outer PowerShell to nothing, so it failed with + /// "The term '.Name' is not recognized" on every host. Only compiled on + /// Windows. + #[cfg(windows)] + #[test] + fn the_display_driver_verify_runs_in_cmd_and_powershell() { + let verify = find_recipe("fix-22-display-driver-old").unwrap().verify; + for run in run_typed_in_cmd_and_powershell(verify, &[]) { + let ShellRun { + shell, + success, + status, + stdout, + stderr, + } = run; + assert!(success, "{shell}: {status}\n{stdout}\n{stderr}"); + assert!( + stderr.trim().is_empty(), + "{shell} wrote to stderr: {stderr}" + ); + } + } + + /// Runs the fix-14 verify line in cmd and in Windows PowerShell: both must + /// parse it and exit 0 with nothing on stderr. Its output is empty on a + /// host with no AMD adapter, so that alone would pass for a filter that + /// matches nothing. The filter text is therefore taken from the recipe and + /// run over made-up adapters; only the AMD ones may come out. Only compiled + /// on Windows. + #[cfg(windows)] + #[test] + fn the_adrenalin_verify_runs_and_its_filter_keeps_only_amd() { + let verify = find_recipe("fix-14-adrenalin-too-old").unwrap().verify; + for run in run_typed_in_cmd_and_powershell(verify, &[]) { + let ShellRun { + shell, + success, + status, + stdout, + stderr, + } = run; + assert!(success, "{shell}: {status}\n{stdout}\n{stderr}"); + assert!( + stderr.trim().is_empty(), + "{shell} wrote to stderr: {stderr}" + ); + } + let filter = adrenalin_filter(verify); + let synthetic = format!( + "@([pscustomobject]@{{Name='AMD Radeon X'}},[pscustomobject]@{{Name='Other'}},\ + [pscustomobject]@{{Name='Intel UHD'}},[pscustomobject]@{{Name='Radeon Pro W'}}) \ + | {filter} | ForEach-Object {{ $_.Name }}" + ); + let run = run_typed_in_powershell(&synthetic, &[]); + assert!( + run.success, + "{}\n{}\n{}", + run.status, run.stdout, run.stderr + ); + let names: Vec<&str> = run.stdout.lines().map(str::trim).collect(); + assert_eq!(names, ["AMD Radeon X", "Radeon Pro W"], "{}", run.stderr); + } + + /// The cases the rule is held to live in one file that the e2e crate's copy + /// of the rule reads too, so the two cannot disagree on a listed line. + const RULE_CASES: &str = + include_str!("../../../tests/e2e-cucumber/fixtures/powershell_text_rule_cases.txt"); + + #[test] + fn the_powershell_text_rule_agrees_with_the_shared_case_table() { + let mut flagged = 0; + let mut passed = 0; + for (number, raw) in RULE_CASES.lines().enumerate() { + let raw = raw.trim_end(); + if raw.is_empty() || raw.starts_with('#') { + continue; + } + if let Some(line) = raw.strip_prefix("flag ") { + assert!( + fails_in_windows_powershell(line).is_some(), + "line {}: `{line}`", + number + 1 + ); + flagged += 1; + } else if let Some(line) = raw.strip_prefix("pass ") { + assert_eq!( + fails_in_windows_powershell(line), + None, + "line {}: `{line}`", + number + 1 + ); + passed += 1; + } else { + panic!("line {}: neither `flag` nor `pass`: {raw}", number + 1); + } + } + // The table was read, not skipped. + assert!( + flagged >= 20 && passed >= 15, + "{flagged} flag, {passed} pass" + ); + // The catalog's own fix-2 line is one of the lines it must let through. + let fix_2 = find_recipe("fix-2-unset-override").unwrap().verify; + assert_eq!(fails_in_windows_powershell(fix_2), None, "{fix_2}"); + } + + /// The fix-2 verify line is shown on Linux, WSL and Windows, so it must be + /// one command that bash, cmd and PowerShell all run the same way: a + /// `python -c` whose program is one double-quoted argument with no inner + /// double quote and no `$`, so no shell expands anything in it. + #[test] + fn the_override_verify_is_one_line_for_every_shell() { + let verify = find_recipe("fix-2-unset-override").unwrap().verify; + let program = verify + .strip_prefix("python -c \"") + .and_then(|rest| rest.strip_suffix('"')) + .unwrap_or_else(|| panic!("not a quoted `python -c` argument: {verify}")); + assert!(!program.contains(['"', '$', '`', '%']), "{program}"); + assert!( + program.contains("os.environ.get('HSA_OVERRIDE_GFX_VERSION')") + && program.contains("'OK_UNSET'"), + "{program}" + ); + } + #[test] fn a_needs_argument_platform_names_the_argument() { for (fix_id, os, class) in classes() { diff --git a/docs/testing.md b/docs/testing.md index 1a241280cf..56443be2e8 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1295,6 +1295,78 @@ install steps also require a native Windows build before any CLI invocation. that ordering without running an installer. Do not run these scenarios on a non-Windows host. +Lines that `rocm fix` and `rocm diagnose` print for Windows are meant to be +pasted into Windows PowerShell 5.1 or cmd, so they are held to a text rule: no +`&&` or `||`, no bare `where`, no `%NAME%`, no empty-string argument, and no +`$` or `\"` inside a double-quoted `powershell -Command "..."` body, and no +`Get-Item` there without `-ErrorAction Stop` (a missing path is a +non-terminating error, so the line would print it and still exit 0). The `where` +and `-Command` matches ignore case. +`fails_in_windows_powershell` in +[fix.rs](../crates/rocm-core/src/fix.rs) applies it to every catalog recipe that +applies on Windows (commands, the `Verify` line and the `Verify (Linux, WSL)` +line) and to the `Fix` values `rocm diagnose` builds for Windows. Scenarios +`diagnose-43`, `diagnose-44` and `diagnose-45` in +[diagnose.feature](../tests/e2e-cucumber/features/diagnose.feature) apply it to +what `rocm fix --dry-run` prints; the plan prints before the platform +gate, so they run on the Linux lanes too. The e2e crate cannot depend on +rocm-core, so it carries its own copy of the rule, +[powershell_text_rule.rs](../tests/e2e-cucumber/src/powershell_text_rule.rs). +Both copies are tested against one table of cases, +[powershell_text_rule_cases.txt](../tests/e2e-cucumber/fixtures/powershell_text_rule_cases.txt): +add a case there, and a rule that gets it wrong fails on its own side. + +The rule reads text; it does not prove a line runs. On Windows, +`cargo test -p rocm-core --lib` types the `Verify` lines of fix-2, fix-13, fix-14, fix-15, fix-21 and +fix-22 into cmd and into Windows PowerShell and checks the exit code and +output (the tests in [fix.rs](../crates/rocm-core/src/fix.rs) named +`the_override_verify_runs_in_cmd_and_powershell`, +`the_hip_sdk_verify_runs_the_exe_under_hip_path`, +`the_adrenalin_verify_runs_and_its_filter_keeps_only_amd`, +`the_msvc_verify_runs_in_cmd_and_powershell`, +`the_outdated_msvc_verify_prints_a_version_and_fails_when_the_dll_is_missing` +and `the_display_driver_verify_runs_in_cmd_and_powershell`; fix-15 and fix-21 +also must exit non-zero when their DLL is missing, fix-15 in +`the_msvc_verify_exits_nonzero_when_a_dll_is_missing`), and runs the fix-9 line +that only `rocm diagnose` prints +(`the_igpu_verify_pins_the_variable_it_names` in +[diagnose.rs](../crates/rocm-core/src/diagnose.rs)). These tests are compiled on +Windows only, so the Linux lanes do not run them, and no Windows scenario covers +the Windows lines `rocm diagnose` prints: those are held by the text rule +(`windows_diagnosis_remediation_passes_the_powershell_text_rule`) and the tests +above. The fix-6 PATH write is not run by any test because it changes the +user's PATH. + +A catalog entry can carry a second line for Linux and WSL, printed as +`Verify (Linux, WSL):`, when the line every OS can run cannot show what Linux +needs. fix-6 does: `hipcc --version` for every OS, and `rocminfo | head -n 5` +for Linux and WSL, which shows the GPU and passes on a runtime-only install +that has no `hipcc`. `diagnose-45` asserts both lines and the note that names +the runtime-only case. fix-2 does too: the `python` line for every OS (it is +what cmd and PowerShell run), and `echo "${HSA_OVERRIDE_GFX_VERSION:-OK_UNSET}"` +for Linux and WSL, which needs no Python (a stock Ubuntu has `python3` only) +and no `||`. `diagnose-44` asserts both lines and the note. + +The `rocm diagnose` Linux arms print the catalog's Linux line and nothing from +the plain line: fix-6 prints `rocminfo | head -n 5` alone (chaining `hipcc +--version` fails on a runtime-only install), and fix-2 prints its Linux line +followed by `python3 -c "import torch; ..."`, the check that clearing the +override left the GPU usable. Scenarios `diagnose-46` (fix-2, run with +`HSA_OVERRIDE_GFX_VERSION` set) and `diagnose-47` (fix-6, on a simulated host) +in [diagnose.feature](../tests/e2e-cucumber/features/diagnose.feature) assert +the line `rocm diagnose` prints, and that it names `python3` and no `hipcc`. +Both are `@requires-os:linux`, so they run on every Linux lane (WSL2 included), +GPU or not, and are skipped on Windows. The arms are also checked against the +catalog in [diagnose.rs](../crates/rocm-core/src/diagnose.rs) +(`the_override_fix_verify_matches_the_catalog_copy`, +`the_path_fix_verify_matches_the_catalog_copy`). On Unix hosts, +`the_path_fix_verify_passes_on_a_runtime_only_linux_install` and +`the_override_fix_verify_runs_where_only_python3_exists` run the lines +`rocm diagnose` prints in `/bin/sh` on a `PATH` of stand-in executables with no +`hipcc` or no `python`. `the_override_linux_verify_runs_with_nothing_on_path` +([fix.rs](../crates/rocm-core/src/fix.rs)) runs the catalog's `verify_linux` +line for fix-2, not the `rocm diagnose` one, on an empty `PATH`. + Provider audit focused test: ```bash diff --git a/tests/e2e-cucumber/features/diagnose.feature b/tests/e2e-cucumber/features/diagnose.feature index 0d06e88dea..1a350c71d3 100644 --- a/tests/e2e-cucumber/features/diagnose.feature +++ b/tests/e2e-cucumber/features/diagnose.feature @@ -653,3 +653,77 @@ Feature: Diagnosing failures and listing fixes When the user diagnoses that machine with a stale repo symptom Then the stale repo cause names both planted repo files and the driver reinstall command And the advised driver reinstall command accepts the flags it is advised with + + # The verify line of the MSVC runtime fix is shown to users of Windows + # PowerShell 5.1 and of cmd. `&&` and `||` are syntax errors in the first, a + # bare `where` is `Where-Object` there, it does not expand `%NAME%`, and in a + # `powershell -Command "..."` body it takes neither `\"` as an escaped quote + # nor `$_` (the outer shell expands it). + # Deliberately not OS-gated: the plan is printed before the fix's platform + # gate, so the text is the same on every lane, and the scenario asserts the + # printed lines and not the exit code. It does not run the line. The test + # `the_msvc_verify_runs_in_cmd_and_powershell` in + # crates/rocm-core/src/fix.rs does that, and only on Windows. + @id:diagnose-msvc-verify-line-text-is-powershell-safe + Scenario: diagnose-43 - The lines printed for the MSVC runtime fix avoid the constructs known to break in Windows PowerShell, and the verify line is a PowerShell command + Given a user who has chosen the fix for the missing MSVC runtime + When the user previews that fix without applying it + Then the printed commands and verify line avoid the constructs known to break in Windows PowerShell + And the printed verify command is a PowerShell command that selects FileVersion + + # The same check for the two other fixes whose lines are shown on Windows + # and were written for one shell: the override fix's verify line used + # `env | grep ... ||`, and the PATH fix's verify line used `rocminfo | head + # && ...`. The override fix's Linux line is `echo "${HSA_OVERRIDE_GFX_VERSION:-OK_UNSET}"`: + # it has no `||`, which the rule refuses on a line a Windows user sees, and it + # needs no `python`, which a stock Ubuntu lacks. The override fix also used to print `setx HSA_OVERRIDE_GFX_VERSION ""`, + # and Windows PowerShell 5.1 drops the empty argument, so setx failed with + # "Invalid syntax" and cleared nothing; the step refuses an empty-string + # argument too. Text only, like diagnose-43. `the_override_verify_runs_in_cmd_and_powershell` + # in crates/rocm-core/src/fix.rs runs the override verify line, on Windows. + # These three scenarios see what `rocm fix --dry-run` prints from the + # catalog. The `Fix` values `rocm diagnose` builds for Windows are held to the + # same text rule by `windows_diagnosis_remediation_passes_the_powershell_text_rule` + # in crates/rocm-core/src/diagnose.rs, and a printed line the rule passes is + # still not proven to run: the Windows-only tests named above do that. + @id:diagnose-override-fix-lines-are-powershell-safe + Scenario: diagnose-44 - The lines printed for the GPU override fix avoid the constructs known to break in Windows PowerShell + Given a user who has chosen the fix for the GPU override + When the user previews that fix without applying it + Then the printed commands and verify line avoid the constructs known to break in Windows PowerShell + And the printed Linux verify line reads the override without needing Python + And the printed notes say a stock Linux install has python3 and not python + + @id:diagnose-path-fix-lines-are-powershell-safe + Scenario: diagnose-45 - The lines printed for the PATH fix avoid the constructs known to break in Windows PowerShell + Given a user who has chosen the fix for ROCm missing from PATH + When the user previews that fix without applying it + Then the printed commands and verify line avoid the constructs known to break in Windows PowerShell + And the printed Linux verify line shows the GPU without needing a compiler + And the printed notes say a runtime-only Linux install has no hipcc + + # The three scenarios above see `rocm fix --dry-run`, which prints the + # catalog's lines. `rocm diagnose` builds its own `Fix` values and prints the + # verify line from them, so the line a user reads after a diagnosis is checked + # here, on Linux, where it can be told apart from the catalog's by running + # the binary. The exact line is asserted, and that it names `python3`: a stock + # Ubuntu has no `python`. Text only: the line is run by + # `the_override_fix_verify_runs_where_only_python3_exists` in + # crates/rocm-core/src/diagnose.rs, on Unix hosts. The Windows arms of these + # printed lines are held by `windows_diagnosis_remediation_passes_the_powershell_text_rule` + # (text) and the Windows-only `*_verify_runs_*` tests in + # crates/rocm-core/src/fix.rs (run); there is no Windows scenario for them. + @id:diagnose-override-verify-line-names-python3 @requires-os:linux + Scenario: diagnose-46 - The verify line printed for the GPU override cause reads the override and uses python3 + Given a Linux machine whose shell has a GPU override set + When the user diagnoses that machine with a page fault symptom + Then the override cause prints a verify line that reads the override and checks torch with python3 + + # The fix-6 Linux arm used to chain `hipcc --version` after `rocminfo`, which + # fails on a runtime-only install even with PATH fixed. The host is simulated + # as in diagnose-40 and the symptom is what raises the cause. + @id:diagnose-path-verify-line-is-rocminfo-alone @requires-os:linux + Scenario: diagnose-47 - The verify line printed for the PATH cause is rocminfo alone, with no hipcc + Given a simulated Linux machine with ROCm missing from PATH + When the user diagnoses that machine with a rocminfo not found symptom + Then the PATH cause prints a verify line that shows the GPU with rocminfo alone diff --git a/tests/e2e-cucumber/fixtures/powershell_text_rule_cases.txt b/tests/e2e-cucumber/fixtures/powershell_text_rule_cases.txt new file mode 100644 index 0000000000..2a7cc11693 --- /dev/null +++ b/tests/e2e-cucumber/fixtures/powershell_text_rule_cases.txt @@ -0,0 +1,92 @@ +# Cases for the text rule over the lines `rocm fix` and `rocm diagnose` print for +# Windows: what it must refuse (`flag`) and what it must let through (`pass`). +# +# The rule exists twice. `fails_in_windows_powershell` in +# crates/rocm-core/src/fix.rs checks the catalog and the diagnosis fixes; the +# diagnose-43..45 steps use `powershell_text_rule::refusal` in this crate, which +# does not depend on rocm-core. Both read this file and must agree on every +# line, so a change to one rule that the other does not follow fails a test on +# whichever side was left behind. Add a case here, not in either test. +# +# One case per line: `flag ` or `pass `. Blank lines and `#` lines +# are ignored. + +# `&&` and `||` are syntax errors in Windows PowerShell 5.1. +flag a && b +flag a || b +flag env | grep HSA_OVERRIDE_GFX_VERSION || echo OK_UNSET + +# A bare `where` is `Where-Object` in PowerShell. A lone `&` is cmd's command +# separator, so `where` starts a command after it. +flag where foo.dll +flag echo x | where foo +flag echo x & where foo +flag where foo & echo x +flag (where foo.dll) +flag cmd /c where foo.dll +flag cmd /c "where foo.dll" +flag powershell -NoProfile -Command "where foo.dll" +pass where.exe foo.dll +pass echo x & where.exe foo +pass cmd /c where.exe foo.dll +pass echo where is it + +# PowerShell does not expand `%NAME%`. +flag setx PATH "%PATH%;C:\bin" +flag echo %HIP_PATH% +pass echo 100% done +pass echo 50%x +pass echo % % +pass a | b + +# An empty-string argument to a native program is dropped by Windows +# PowerShell 5.1, so `setx` gets one argument and fails. +flag setx HSA_OVERRIDE_GFX_VERSION "" +flag setx HSA_OVERRIDE_GFX_VERSION '' +flag tool a "" b +# An empty string that is not a whole argument, or that sits in a body. +pass python -c "print('')" +pass powershell -NoProfile -Command "Get-Item a -ErrorAction Stop | Where-Object Name -ne ''" +pass [Environment]::SetEnvironmentVariable('HSA_OVERRIDE_GFX_VERSION',$null,'User') + +# `-Command` is not case-sensitive, and `-c` and the other prefixes of it are +# the same flag. `\"` inside the body ends the string early in PowerShell, and +# `$_` in it is expanded by the outer shell. +flag powershell -NoProfile -command "(Get-Item a | Where-Object { $_.Name -like 'x' })" +flag powershell -NoProfile -Command "(Get-Item a | Where-Object { $_.Name -like 'x' })" +flag powershell -NoProfile -Command "& \"C:\x\hipInfo.exe\" | Select-Object -First 5" +flag powershell -NoProfile -c "& \"C:\x\hipInfo.exe\" | Select-Object -First 5" +flag powershell -NoProfile -comm "(Get-Item a | Where-Object { $_.Name -like 'x' })" + +# cmd does not take a single-quoted argument as one argument. +flag powershell -NoProfile -Command 'Get-Item a | Select-Object Name' +flag powershell -NoProfile -c 'Get-Item a' + +# An outer PowerShell expands any `$` in a double-quoted body before the inner +# one starts, `$env:SystemRoot` included: with a space in its value it would +# split the inner command. +flag powershell -NoProfile -Command "& (Join-Path $env:HIP_PATH 'bin\hipInfo.exe')" +flag powershell -NoProfile -Command "Get-Item $env:SystemRoot\a.dll, $env:HIP_PATH" +flag powershell -NoProfile -Command "(Get-Item $env:SystemRoot\System32\a.dll).VersionInfo" +flag powershell -NoProfile -c "(Get-Item $env:SystemRoot\System32\a.dll).VersionInfo" +flag powershell -NoProfile -Command "$null" + +# What the catalog prints instead. +pass powershell -NoProfile -Command "& 'C:\x\hipInfo.exe' | Select-Object -First 5" +pass powershell -NoProfile -Command "Get-Item a -ErrorAction Stop | Where-Object Name -match 'x'" +pass powershell -NoProfile -Command "(Get-Item ([Environment]::GetFolderPath('System') + '\a.dll') -ErrorAction Stop).VersionInfo" +pass powershell -NoProfile -c "(Get-Item ([Environment]::GetFolderPath('System') + '\a.dll') -ErrorAction Stop).VersionInfo" + +# `Get-Item` on a missing path is a non-terminating error, and the line exits 0. +flag powershell -NoProfile -Command "(Get-Item ([Environment]::GetFolderPath('System') + '\a.dll')).VersionInfo" +flag POWERSHELL -command "(GET-ITEM 'x').Length" + +# The bare `where` match ignores case, like the rest of the rule. +flag CMD /C WHERE foo.dll +flag WHERE foo.dll + +# Not a PowerShell `-Command` line, so the body rule does not apply. +pass python -c "print('a')" +pass python -c "import os; print(os.environ.get('A') or 'B')" +pass rocminfo | head -n 5 +pass echo "${HSA_OVERRIDE_GFX_VERSION:-OK_UNSET}" diff --git a/tests/e2e-cucumber/src/lib.rs b/tests/e2e-cucumber/src/lib.rs index dbb7a22920..e83321cd3f 100644 --- a/tests/e2e-cucumber/src/lib.rs +++ b/tests/e2e-cucumber/src/lib.rs @@ -16,6 +16,7 @@ pub mod monotonic_clock; pub mod paced_download; pub mod panic_capture; pub mod polled_deadline; +pub mod powershell_text_rule; pub mod quit_and_wait; pub mod reader_failure; pub mod send_until; @@ -181,7 +182,8 @@ mod tests { "Flags: rocm fix can run it\n", "Note: Windows: the PowerShell line reads and rewrites the User scope only. Do not substitute `setx PATH \"%PATH%;...\"`: setx silently truncates the value at 1024 characters.\n", "Note: Windows: a PATH change only takes effect in NEW shells.\n", - "Verify: rocminfo | head -n 5 && hipcc --version\n", + "Verify: hipcc --version\n", + "Verify (Linux, WSL): rocminfo | head -n 5\n", "\n", ); diff --git a/tests/e2e-cucumber/src/powershell_text_rule.rs b/tests/e2e-cucumber/src/powershell_text_rule.rs new file mode 100644 index 0000000000..f3e385aab2 --- /dev/null +++ b/tests/e2e-cucumber/src/powershell_text_rule.rs @@ -0,0 +1,128 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The text rule for the lines `rocm fix` and `rocm diagnose` print for Windows. +//! +//! This is a copy of `fails_in_windows_powershell` in `crates/rocm-core/src/fix.rs` +//! (test-only there; this crate does not depend on rocm-core, and +//! `cargo xtask check-crate-edges` is not to be bypassed). Two copies drift, so +//! both are tested against the one table in +//! `fixtures/powershell_text_rule_cases.txt`: the test below reads it here, and a +//! rocm-core test reads the same file. A case added there that one rule does not +//! handle fails that side's test. +//! +//! The rule is a text check over known-bad constructs. It is not a parser and +//! does not prove a line runs. + +/// Why `line` would break in Windows PowerShell 5.1 or cmd, or `None` when the +/// rule finds nothing wrong with it. +#[must_use] +pub fn refusal(line: &str) -> Option<&'static str> { + if line.contains("&&") || line.contains("||") { + return Some("`&&` or `||`, a syntax error in Windows PowerShell 5.1"); + } + let lowered = line.to_ascii_lowercase(); + let command = command_argument(&lowered); + let starts_bare_where = |text: &str| { + text.split(['|', ';', '&', '(', '\u{7b}']).any(|part| { + let mut words = part.split_whitespace().map(|w| w.trim_matches('"')); + let mut first = words.next(); + if first == Some("cmd") { + first = words.find(|w| !w.starts_with('/')); + } + first == Some("where") + }) + }; + if starts_bare_where(&lowered) + || command.is_some_and(|(_, _, body)| starts_bare_where(body.trim_start())) + { + return Some("a bare `where`, which is `Where-Object` in PowerShell"); + } + let outer = command.map_or(lowered.as_str(), |(at, _, _)| &lowered[..at]); + if outer + .split_whitespace() + .any(|word| word == "\"\"" || word == "''") + { + return Some("an empty-string argument, which Windows PowerShell 5.1 drops"); + } + if let Some((_, quote, body)) = command { + if quote == '\'' { + return Some("a single-quoted `-Command` argument, which cmd does not take"); + } + if body.contains("\\\"") { + return Some("a backslash-escaped quote inside a `-Command` body"); + } + if body.contains('$') { + return Some("a `$` inside a double-quoted `-Command` body"); + } + if body.contains("get-item") && !body.contains("-erroraction stop") { + return Some("a `Get-Item` without `-ErrorAction Stop`"); + } + } + let pieces: Vec<&str> = line.split('%').collect(); + let names_a_cmd_variable = pieces + .get(1..pieces.len().saturating_sub(1)) + .unwrap_or_default() + .iter() + .any(|name| { + !name.is_empty() && name.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'_') + }); + names_a_cmd_variable.then_some("a `%NAME%` reference, which PowerShell does not expand") +} + +/// Where a `powershell ... -Command "..."` argument starts in `lowered`: the +/// offset of the flag, the quote that opens the argument, and the text after +/// it. PowerShell takes any prefix of `-Command` down to `-c`. +fn command_argument(lowered: &str) -> Option<(usize, char, &str)> { + lowered.match_indices('-').find_map(|(at, _)| { + let head = &lowered[..at]; + if head.chars().next_back().is_some_and(|c| !c.is_whitespace()) + || !(head.contains("powershell") || head.contains("pwsh")) + { + return None; + } + let rest = &lowered[at..]; + let flag = rest.split_whitespace().next()?; + if flag.len() < 2 || !"-command".starts_with(flag) { + return None; + } + let after = rest[flag.len()..].trim_start(); + let quote = after.chars().next().filter(|c| matches!(c, '"' | '\''))?; + Some((at, quote, &after[1..])) + }) +} + +#[cfg(test)] +mod tests { + use super::refusal; + + /// The cases both copies of the rule are held to. + const CASES: &str = include_str!("../fixtures/powershell_text_rule_cases.txt"); + + #[test] + fn the_rule_agrees_with_the_shared_case_table() { + let mut flagged = 0; + let mut passed = 0; + for (number, raw) in CASES.lines().enumerate() { + let raw = raw.trim_end(); + if raw.is_empty() || raw.starts_with('#') { + continue; + } + if let Some(line) = raw.strip_prefix("flag ") { + assert!(refusal(line).is_some(), "line {}: `{line}`", number + 1); + flagged += 1; + } else if let Some(line) = raw.strip_prefix("pass ") { + assert_eq!(refusal(line), None, "line {}: `{line}`", number + 1); + passed += 1; + } else { + panic!("line {}: neither `flag` nor `pass`: {raw}", number + 1); + } + } + // The table was read, not skipped. + assert!( + flagged >= 20 && passed >= 15, + "{flagged} flag, {passed} pass" + ); + } +} diff --git a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs index 6175594b78..6ab80d521c 100644 --- a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs @@ -316,6 +316,163 @@ async fn user_chose_engine_import_fix(world: &mut E2eWorld) { world.model_name = Some(ENGINE_IMPORT_FIX_ID.to_string()); } +#[given("a user who has chosen the fix for the missing MSVC runtime")] +async fn user_chose_msvc_runtime_fix(world: &mut E2eWorld) { + world.model_name = Some("fix-15-msvc-redist".to_string()); +} + +#[given("a user who has chosen the fix for the GPU override")] +async fn user_chose_override_fix(world: &mut E2eWorld) { + world.model_name = Some("fix-2-unset-override".to_string()); +} + +#[given("a user who has chosen the fix for ROCm missing from PATH")] +async fn user_chose_path_fix(world: &mut E2eWorld) { + world.model_name = Some("fix-6-path".to_string()); +} + +#[then( + "the printed commands and verify line avoid the constructs known to break in Windows PowerShell" +)] +async fn assert_printed_lines_avoid_powershell_hostile_syntax(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no fix preview output"); + let mut lines: Vec<&str> = output + .lines() + .filter_map(|line| line.strip_prefix(" $ ")) + .filter(|c| !c.trim_start().starts_with('#')) + .collect(); + // Every verify line is shown on Windows, the Linux one included (labelled). + lines.extend(printed_verify_lines(output)); + // The rule lives in this crate because it does not depend on rocm-core. The + // table in `fixtures/powershell_text_rule_cases.txt` holds it and the + // rocm-core rule to the same answers. + for line in lines { + if let Some(why) = e2e_cucumber::powershell_text_rule::refusal(line) { + panic!("`{line}` uses {why}"); + } + } +} + +#[then("the printed verify command is a PowerShell command that selects FileVersion")] +async fn assert_verify_is_powershell_selecting_version(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no fix preview output"); + let verify = printed_verify_line(output); + // Text only. `the_msvc_verify_runs_in_cmd_and_powershell` (rocm-core, + // Windows only) runs the line. + assert!( + verify.starts_with("powershell -NoProfile -Command ") && verify.contains("FileVersion"), + "expected a PowerShell command that prints FileVersion: {verify}" + ); +} + +#[then("the printed Linux verify line shows the GPU without needing a compiler")] +async fn assert_linux_verify_shows_the_gpu(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no fix preview output"); + let linux = output + .lines() + .find_map(|line| line.strip_prefix("Verify (Linux, WSL):")) + .unwrap_or_else(|| panic!("the preview printed no Linux verify line:\n{output}")) + .trim(); + // `rocminfo` lists the GPU agents. `hipcc` is the compiler a runtime-only + // Linux install does not have, so a line that runs it fails there even + // with PATH fixed. Text only: the line is not run here, because a Linux + // lane without ROCm has no `rocminfo`. + assert!( + linux.starts_with("rocminfo") && !linux.contains("hipcc"), + "the Linux verify line must show the GPU and not need hipcc: {linux}" + ); + // The line every OS can run stays free of anything only Linux has. + let shared = printed_verify_line(output); + assert!( + !shared.contains("rocminfo"), + "the Windows verify line must not need rocminfo: {shared}" + ); +} + +#[then("the printed Linux verify line reads the override without needing Python")] +async fn assert_linux_verify_reads_the_override_without_python(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no fix preview output"); + let linux = output + .lines() + .find_map(|line| line.strip_prefix("Verify (Linux, WSL):")) + .unwrap_or_else(|| panic!("the preview printed no Linux verify line:\n{output}")) + .trim(); + // A stock Ubuntu has `python3` and no `python`, so a line that runs either + // fails there after a correct fix. Text only: the line is run, with an + // empty PATH, by `the_override_linux_verify_runs_with_nothing_on_path` in + // rocm-core (Unix hosts). + assert!( + linux.contains("HSA_OVERRIDE_GFX_VERSION") && !linux.contains("python"), + "the Linux verify line must read the override without Python: {linux}" + ); + // The line every OS can run is the one that needs `python`. + let shared = printed_verify_line(output); + assert!( + shared.starts_with("python -c "), + "the plain verify line is the python one: {shared}" + ); +} + +#[then("the printed notes say a stock Linux install has python3 and not python")] +async fn assert_notes_name_the_python3_case(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no fix preview output"); + // The claim is about the two verify lines printed above: the plain one runs + // `python` and fails on a stock Ubuntu, the Linux one does not. The + // assertion in `assert_linux_verify_reads_the_override_without_python` is + // what makes the note's statement about the Linux line true. + assert!( + output + .lines() + .any(|line| line.strip_prefix("Note:").is_some_and(|note| { + note.contains("python3") + && note.contains("`python`") + && note.contains("Verify (Linux, WSL)") + })), + "no note names the python3 case:\n{output}" + ); +} + +#[then("the printed notes say a runtime-only Linux install has no hipcc")] +async fn assert_notes_name_the_runtime_only_case(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no fix preview output"); + // The claim is about the two verify lines printed above: the one that runs + // `hipcc` fails on a runtime-only install, the Linux one does not. The + // assertion in `assert_linux_verify_shows_the_gpu` is what makes the note's + // statement about the Linux line true. + assert!( + output + .lines() + .any(|line| line.strip_prefix("Note:").is_some_and(|note| { + note.contains("runtime-only") + && note.contains("hipcc") + && note.contains("Verify (Linux, WSL)") + })), + "no note names the runtime-only case:\n{output}" + ); +} + +/// Every `Verify` line the preview printed: the plain one and the Linux one. +fn printed_verify_lines(output: &str) -> Vec<&str> { + output + .lines() + .filter_map(|line| { + line.strip_prefix("Verify:") + .or_else(|| line.strip_prefix("Verify (Linux, WSL):")) + }) + .map(str::trim) + .collect() +} + +fn printed_verify_line(output: &str) -> &str { + output + .lines() + .find_map(|line| line.strip_prefix("Verify:")) + .map_or_else( + || panic!("the preview printed no Verify: line:\n{output}"), + str::trim, + ) +} + #[given("a user who has chosen a known fix")] async fn user_chose_known_fix(world: &mut E2eWorld) { world.model_name = Some(PREVIEW_FIX_ID.to_string()); @@ -2800,6 +2957,144 @@ async fn assert_advised_driver_command_exists(world: &mut E2eWorld) { } } +// ── The verify line `rocm diagnose` prints on Linux ──────────────── + +/// The reported cause block for `fix_id`: the `#N (id)` header line and +/// everything under it up to the blank line that ends it. +fn cause_block<'a>(output: &'a str, fix_id: &str) -> Vec<&'a str> { + let marker = format!("({fix_id})"); + output + .lines() + .skip_while(|line| !line.contains(&marker)) + .take_while(|line| !line.trim().is_empty()) + .collect() +} + +/// The text after `verify after fix:` in the cause block for `fix_id`. +fn diagnose_verify_line(output: &str, fix_id: &str) -> String { + let block = cause_block(output, fix_id); + assert!( + !block.is_empty(), + "no `{fix_id}` cause was reported: +{output}" + ); + block + .iter() + .find_map(|line| line.trim().strip_prefix("verify after fix:")) + .unwrap_or_else(|| { + panic!( + "`{fix_id}` printed no verify line: +{}", + block.join( + " +" + ) + ) + }) + .trim() + .to_owned() +} + +#[given("a Linux machine whose shell has a GPU override set")] +async fn linux_machine_with_a_gpu_override(world: &mut E2eWorld) { + // `rocm diagnose` reads the override from the environment it examines, + // which is the CLI child's, so setting it there is the whole setup. + world + .command_env + .push(("HSA_OVERRIDE_GFX_VERSION", "11.0.0".into())); +} + +#[when("the user diagnoses that machine with a page fault symptom")] +async fn user_diagnoses_page_fault_symptom(world: &mut E2eWorld) { + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env( + world, + &["diagnose", "--symptom", "page fault", "--top", "20"], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then( + "the override cause prints a verify line that reads the override and checks torch with python3" +)] +async fn assert_diagnose_override_verify_uses_python3(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no diagnose output"); + let verify = diagnose_verify_line(output, "fix-2-unset-override"); + // A stock Ubuntu has `python3` and no `python`, so a printed line that runs + // `python` fails there after a correct fix. Text only: the line is run by + // `the_override_fix_verify_runs_where_only_python3_exists` in rocm-core + // (Unix hosts), and this machine may have no torch. + assert_eq!( + verify, + "echo \"${HSA_OVERRIDE_GFX_VERSION:-OK_UNSET}\"; python3 -c \"import torch; print(torch.cuda.is_available())\"", + "the printed verify line changed" + ); + assert!( + !verify.split_whitespace().any(|word| word == "python"), + "the verify line must not run bare `python`: {verify}" + ); +} + +#[given("a simulated Linux machine with ROCm missing from PATH")] +async fn simulated_linux_machine_without_rocm_on_path(world: &mut E2eWorld) { + // Planted host root, as in diagnose-40/42, so the runner's own ROCm state + // is not what is described. The symptom below is what makes the cause fire. + let host = world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .join("host-root"); + std::fs::create_dir_all(host.join("etc")).expect("failed to create planted etc"); + std::fs::write( + host.join("etc/os-release"), + "ID=ubuntu +VERSION_ID=\"24.04\" +VERSION_CODENAME=noble +PRETTY_NAME=\"Ubuntu 24.04 LTS\" +", + ) + .expect("failed to write planted os-release"); + world + .command_env + .push(("ROCM_CLI_TEST_HOST_ROOT", host.into_os_string())); +} + +#[when("the user diagnoses that machine with a rocminfo not found symptom")] +async fn user_diagnoses_rocminfo_not_found(world: &mut E2eWorld) { + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env( + world, + &[ + "diagnose", + "--symptom", + "rocminfo: command not found", + "--top", + "20", + ], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the PATH cause prints a verify line that shows the GPU with rocminfo alone")] +async fn assert_diagnose_path_verify_is_rocminfo_alone(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no diagnose output"); + let verify = diagnose_verify_line(output, "fix-6-path"); + // `hipcc` is the compiler a runtime-only install does not have, so a line + // that chains it fails there with PATH fixed. Text only: running `rocminfo` + // needs ROCm on the lane. + assert_eq!( + verify, "rocminfo | head -n 5", + "the printed verify line changed" + ); + assert!( + !verify.contains("hipcc"), + "the verify line must not need hipcc: {verify}" + ); +} + // ── Windows display driver advice ────────────────────────────────── #[when("the user diagnoses this machine in machine-readable form, with no symptom")]