From 0bcdda67da0f0dac78f64b30c7addb4442917cce Mon Sep 17 00:00:00 2001 From: Eugene Volen Date: Wed, 11 Nov 2026 14:45:38 +0200 Subject: [PATCH] feat(diagnose): name the engine a symptom came from with --engine A vLLM error is often an ordinary PyTorch line that does not say vLLM, and the vLLM out-of-memory entry only scores lines that name vLLM, so a symptom has to be edited (a `vllm: ` prefix) before it matches. The rocm-doctor skill tells an agent to do that to the user's paste; it is to move to the new flag in a later change. `rocm diagnose --engine vllm` states where the symptom came from instead, whether it was passed with `--symptom` or read with `--symptom-file`. Every non-blank line of the symptom then counts as vLLM's, as if `vllm` were written on each; no other entry reads the engine. The JSON diagnosis records the engine and the symptom, verbatim (from a file: as read, minus the one trailing line break), under a new optional `provenance` field that is absent when no engine is named. The flag takes only the engines the catalog tells apart (today `vllm`), under the names `rocm serve --engine` uses, and needs a symptom from either flag (a clap group over the two); anything else is a usage error. Writing `vllm` into the symptom still works. The paste can hold non-ASCII text, and `rocm diagnose --json` must stay ASCII (ROCMAI-543): a Windows console on the OEM code page shows raw UTF-8 as mojibake. The JSON is therefore written with every non-ASCII character as a `\uXXXX` escape, on every run and for every field, host paths quoted in evidence included. Decoded, every value is unchanged; the text report never echoes the paste. Apart from that escaping and the optional field, the output without `--engine` is unchanged, and the skill eval replays, which are ASCII, stay as they are. The text report opens with a line naming the engine and saying the symptom was read as that engine's, which is how it was read, not a claim that anything matched. Where no entry on the examined platform reads it (native Windows has no vLLM entry), the line says the flag changed nothing. The answer comes from the catalog table the checks run from, where each engine-reading entry declares the engine it reads, and is not serialized, so it adds nothing to the JSON. Like the symptom flags, `--engine` is accepted with `--report` and `--distro`. The `--report` document, the one printed for mailing, is built from the examination and the entry the diagnosis established, never from the paste, so neither the symptom nor `provenance` reaches it or its mail link; a test on the real payload pins that. Covered by rocm-core unit tests (same verdict as the prefixed form on Linux and WSL2, every non-blank line anchored, nothing else changed, provenance verbatim and round-tripping through the escaped JSON, the ASCII corpus with a non-ASCII paste, the text line agreeing with what the flag did, each engine-reading entry reading the engine it declares, engine names matching their serialized form), a payload test on the shared report, CLI tests (both symptom sources, the ASCII `--json` rendering, a failing render hook surfacing as an error), and scenarios diagnose-71 to -79. ROCMAI-955 Signed-off-by: Eugene Volen --- README.md | 19 +- apps/rocm/src/cli.rs | 105 ++++- apps/rocm/src/cli_args.rs | 30 +- apps/rocm/src/cli_report.rs | 44 +- apps/rocm/src/diagnose_cmd.rs | 154 ++++++- apps/rocm/src/main.rs | 5 + apps/rocm/src/remote/doctor.rs | 1 + crates/rocm-core/src/diagnose.rs | 108 ++++- crates/rocm-core/src/diagnose/catalog.rs | 260 +++++++++--- crates/rocm-core/src/diagnose/engine.rs | 355 ++++++++++++++++- crates/rocm-core/src/diagnose/host.rs | 1 + crates/rocm-core/src/diagnose/report.rs | 77 ++++ crates/rocm-core/src/diagnose/scoring.rs | 4 +- crates/rocm-core/src/diagnose/test_support.rs | 43 +- crates/rocm-core/src/lib.rs | 5 +- docs/architecture.md | 10 +- docs/manual-testing.md | 62 +++ docs/testing.md | 115 ++++++ docs/vllm.md | 5 + tests/e2e-cucumber/features/diagnose.feature | 113 ++++++ .../e2e-cucumber/tests/e2e/diagnose_steps.rs | 377 ++++++++++++++++++ xtask/src/eval_fixtures.rs | 3 +- 22 files changed, 1801 insertions(+), 95 deletions(-) diff --git a/README.md b/README.md index 1af112fe7a..9a05b81908 100644 --- a/README.md +++ b/README.md @@ -458,7 +458,7 @@ Windows. The Vulkan driver version is not shown. ### Diagnose and fix ``` -rocm diagnose [--symptom TEXT | --symptom-file PATH] [--top N] [--json] [--distro [NAME] | --report [--send]] +rocm diagnose [--symptom TEXT [--engine vllm] | --symptom-file PATH [--engine vllm]] [--top N] [--json] [--distro [NAME] | --report [--send]] rocm diagnose --model [--context-size N] [--kv-cache-type f16|q8_0|q4_0] [--json] rocm fix [] [--yes] [--dry-run] [--device-index N] [--json] ``` @@ -518,6 +518,23 @@ position. contribute to, once the text contributed to any match in the report. That includes one past `--top`, so all the rows shown can carry the line when the only match the text contributed to was cut off. +- `--engine` names the engine the symptom came from (only `vllm` is + accepted; it needs `--symptom` or `--symptom-file`). An engine's own error + is often a line any program could print, such as a PyTorch out-of-memory + error, and the vLLM entry claims only lines that name vLLM. With + `--engine vllm`, every non-blank line of the symptom is taken as vLLM's, so + the error is matched without editing it. The text report opens with a line + naming the engine; where no entry on this OS reads it (native Windows has + no vLLM entry), that line says the flag changed nothing. The `--json` + diagnosis records the engine and the symptom, exactly as given once the + JSON is decoded (for `--symptom-file`, the text read, minus the one + trailing line break), under `provenance`. On the wire `--json` is ASCII: + any non-ASCII character in the paste is written as a `\uXXXX` escape, so a + Windows console shows no mojibake and any JSON parser reads the paste back + unchanged. A `--report` never carries either: it names only the cause + the diagnosis established, so a pasted log's paths or host names stay out + of what you might mail. Writing `vllm` into the symptom text still works + too. - On Windows, `diagnose` also checks the age of the Visual C++ runtime, not only whether it is there. The runtime in `System32` must be 14.40 or newer (`fix-21-msvc-redist-outdated`). Programs built with a recent toolset, such diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 55f8eb1579..c50a1af201 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -13,8 +13,8 @@ //! keeps reaching them as `crate::Command`, `crate::Provider` and so on. pub(crate) use crate::cli_args::{ - DevicePolicyArg, FrameworkArg, InstallFormat, PermissionsModeArg, Provider, SandboxToolArg, - TelemetryModeArg, WatcherModeArg, + DevicePolicyArg, DiagnoseEngineArg, FrameworkArg, InstallFormat, PermissionsModeArg, Provider, + SandboxToolArg, TelemetryModeArg, WatcherModeArg, }; use crate::service_records::DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS; use crate::{bootstrap, chat, comfyui, remote, storage}; @@ -23,8 +23,13 @@ use rocm_core::DEFAULT_LOCAL_HOST; use std::path::PathBuf; /// User-selectable serving engines, in the order shown in `--help` and shell -/// completions. Used as the clap `value_parser` for every user-facing `--engine` -/// argument so the possible values stay in sync across the CLI surface. +/// completions. Used as the clap `value_parser` for every user-facing argument +/// that names an engine to serve, install, open a shell in or set as a +/// preference (`serve --engine`, `engines install`/`shell`, and the +/// engine-setting `config` commands), so the possible values stay in sync +/// across those commands. +/// `diagnose --engine` is narrower on purpose: it takes `DiagnoseEngineArg`, +/// only the engines the diagnosis catalog tells apart. pub(crate) const SUPPORTED_ENGINES: [&str; 3] = ["lemonade", "vllm", "aim"]; #[derive(Parser, Debug)] @@ -103,7 +108,14 @@ pub(crate) enum Command { /// /// Each match then says whether this text contributed to it /// (`symptom_matched` in --json). - #[arg(long, value_name = "TEXT")] + // + // `symptom_source` groups the two symptom flags for `--engine`, which + // needs a symptom from either. A plain `requires = "symptom"` also let + // `--symptom-file` through, but only because clap waives a requirement + // on an argument that conflicts with one present; the group says what + // is meant. It does not change how the two flags refuse each other: + // `--symptom-file`'s own `conflicts_with` still does that. + #[arg(long, value_name = "TEXT", group = "symptom_source")] symptom: Option, /// Read the error text from a file instead, or from standard input /// when PATH is `-`, so no shell quoting is needed. A file named `-` @@ -121,8 +133,33 @@ pub(crate) enum Command { /// text to a program as ASCII, turning any other character into `?`. /// With nothing piped in, paste the text at the terminal and end it /// with Ctrl-D at the start of a line (Ctrl-Z then Enter on Windows). - #[arg(long, value_name = "PATH", conflicts_with = "symptom")] + #[arg( + long, + value_name = "PATH", + conflicts_with = "symptom", + group = "symptom_source" + )] symptom_file: Option, + /// The engine the symptom came from (`--symptom` or `--symptom-file`), + /// so it can be matched as that engine's error without editing it. + /// + /// Some failure modes belong to one engine, and the error it prints + /// (a PyTorch out-of-memory error, for one) is the same text any other + /// program could print. Without this, such an entry matches only a + /// line that names the engine itself. With it, every non-blank line of + /// the symptom is taken as that engine's, the symptom is used exactly + /// as given (from a file, as read: one trailing line break removed), + /// and the JSON diagnosis records both under `provenance` (exactly, + /// once the JSON is decoded: on the wire `--json` is ASCII, with any + /// non-ASCII character written as a `\uXXXX` escape). The text + /// report opens with a line naming the engine, and says the flag + /// changed nothing where no entry on this OS reads that engine (native + /// Windows has no vLLM entry). + /// A `--report` never carries either: it names only the cause the + /// diagnosis established. Only engines the catalog tells apart are + /// accepted; writing `vllm` in the symptom text still works as before. + #[arg(long, value_enum, value_name = "ENGINE", requires = "symptom_source")] + engine: Option, /// Show at most this many matches (default 5). #[arg(long, default_value_t = 5)] top: usize, @@ -2021,6 +2058,62 @@ mod tests { assert_eq!(error.kind(), clap::error::ErrorKind::ArgumentConflict); } + // `--engine` names where a symptom came from (ROCMAI-955). It takes only + // the engines the catalog tells apart, so an engine `serve` knows but the + // catalog does not is a usage error rather than a value silently ignored, + // and without a symptom it would describe nothing. The symptom may come + // from either `--symptom` or `--symptom-file`. + #[test] + fn diagnose_engine_takes_the_engines_the_catalog_tells_apart() { + let cli = parse_diagnose(&["--symptom", "torch.OutOfMemoryError", "--engine", "vllm"]) + .expect("vllm parses"); + match cli.command { + Some(Command::Diagnose { + engine, symptom, .. + }) => { + assert_eq!( + engine.map(rocm_core::SymptomEngine::from), + Some(rocm_core::SymptomEngine::Vllm) + ); + assert_eq!(symptom.as_deref(), Some("torch.OutOfMemoryError")); + } + other => panic!("expected diagnose, got {other:?}"), + } + for unknown in ["lemonade", "aim", "llama.cpp", "VLLM-ish"] { + let error = parse_diagnose(&["--symptom", "x", "--engine", unknown]) + .expect_err("an engine the catalog does not tell apart must not parse"); + assert_eq!( + error.kind(), + clap::error::ErrorKind::InvalidValue, + "{unknown}" + ); + assert!(error.to_string().contains("vllm"), "{error}"); + } + for path in ["-", "error.txt"] { + match parse_diagnose(&["--symptom-file", path, "--engine", "vllm"]) { + Ok(Cli { + command: + Some(Command::Diagnose { + engine, + symptom_file, + .. + }), + .. + }) => { + assert_eq!(engine, Some(DiagnoseEngineArg::Vllm)); + assert_eq!(symptom_file, Some(PathBuf::from(path))); + } + other => panic!("--symptom-file {path} --engine vllm must parse, got {other:?}"), + } + } + let error = parse_diagnose(&["--engine", "vllm"]) + .expect_err("an engine with no symptom describes nothing"); + assert_eq!( + error.kind(), + clap::error::ErrorKind::MissingRequiredArgument + ); + } + // The context options describe the model being asked about, so without // `--model` they would be accepted and silently do nothing. #[test] diff --git a/apps/rocm/src/cli_args.rs b/apps/rocm/src/cli_args.rs index a11fd1b520..64bc65a7a1 100644 --- a/apps/rocm/src/cli_args.rs +++ b/apps/rocm/src/cli_args.rs @@ -2,14 +2,16 @@ // // SPDX-License-Identifier: MIT -//! The clap value enums behind the `--framework`, `--format`, `--provider`, -//! `--mode`, `--device`, `--telemetry` and `--permissions` style arguments, and -//! the conversions from them to the library types they stand for. +//! The clap value enums behind the `--framework`, `diagnose --engine`, +//! `--format`, `--provider`, `--mode`, `--device`, `--telemetry` and +//! `--permissions` style arguments, and the conversions from them to the +//! library types they stand for. //! //! Mechanically relocated from `cli.rs` with no behavior change: attributes, //! help text, aliases and declaration order are unchanged. `cli.rs` re-exports //! each type, so every module keeps reaching them as `crate::cli::Provider` and -//! so on. +//! so on. `DiagnoseEngineArg` (`diagnose --engine`) is the exception: it was +//! added here after the move rather than relocated. use clap::ValueEnum; use rocm_core::{ @@ -48,6 +50,26 @@ impl From for rocm_core::FrameworkProbe { } } +/// Which engine a `rocm diagnose --symptom` came from. +/// +/// The CLI-facing mirror of [`rocm_core::diagnose::SymptomEngine`], for the +/// same reason as [`FrameworkArg`]. It lists only the engines the diagnosis +/// catalog tells apart, under the names `rocm serve --engine` uses; the others +/// are rejected as a usage error rather than accepted and silently ignored. +#[derive(Clone, Copy, Debug, Eq, PartialEq, ValueEnum)] +pub(crate) enum DiagnoseEngineArg { + /// The vLLM serving engine (`rocm serve --engine vllm`). + Vllm, +} + +impl From for rocm_core::diagnose::SymptomEngine { + fn from(value: DiagnoseEngineArg) -> Self { + match value { + DiagnoseEngineArg::Vllm => Self::Vllm, + } + } +} + #[derive(Clone, Copy, Debug, Eq, PartialEq, ValueEnum)] pub(crate) enum InstallFormat { /// Python wheel packages (recommended; smaller, pip-installable). diff --git a/apps/rocm/src/cli_report.rs b/apps/rocm/src/cli_report.rs index 72b695cf9a..93175b8896 100644 --- a/apps/rocm/src/cli_report.rs +++ b/apps/rocm/src/cli_report.rs @@ -84,6 +84,14 @@ pub(crate) fn write_marked_row( /// (side effects, audit events) with the printing stays hand-written. pub(crate) trait JsonOrText: Serialize { fn render_text(&self) -> String; + + /// The JSON form, without the trailing newline. Pretty `serde_json` by + /// default; a command whose output can carry text it did not write + /// overrides this to keep the wire form ASCII (`rocm diagnose`, see + /// `rocm_core::render_diagnose_json`). + fn render_json(&self) -> serde_json::Result { + serde_json::to_string_pretty(self) + } } /// Writes `value` to `writer` as pretty JSON followed by one newline when @@ -98,7 +106,8 @@ fn write_json_or_text( value: &impl JsonOrText, ) -> Result<()> { let rendered = if json { - let mut document = serde_json::to_string_pretty(value) + let mut document = value + .render_json() .context("failed to serialize the command output as JSON")?; document.push('\n'); document @@ -234,6 +243,39 @@ mod tests { ); } + /// Serializes fine through serde but overrides `render_json` with one that + /// fails, the way `rocm diagnose`'s ASCII rendering can: the hook's own + /// error is what has to reach the caller, and nothing may be written. + struct RenderJsonFails; + + impl Serialize for RenderJsonFails { + fn serialize(&self, serializer: S) -> Result { + serializer.serialize_str("serializes fine") + } + } + + impl JsonOrText for RenderJsonFails { + fn render_text(&self) -> String { + panic!("JSON mode must not render the text form"); + } + + fn render_json(&self) -> serde_json::Result { + Err(serde::ser::Error::custom("the render_json hook failed")) + } + } + + #[test] + fn a_failing_render_json_hook_is_an_error_and_writes_nothing() { + let mut output = Vec::new(); + let error = write_json_or_text(&mut output, true, &RenderJsonFails) + .expect_err("the hook's failure must surface"); + assert!(output.is_empty(), "nothing is written on failure"); + assert!( + format!("{error:#}").contains("the render_json hook failed"), + "unexpected error: {error:#}" + ); + } + struct BrokenPipe; impl std::io::Write for BrokenPipe { diff --git a/apps/rocm/src/diagnose_cmd.rs b/apps/rocm/src/diagnose_cmd.rs index 7558cadeda..ff5dd14455 100644 --- a/apps/rocm/src/diagnose_cmd.rs +++ b/apps/rocm/src/diagnose_cmd.rs @@ -36,8 +36,16 @@ use crate::cli_report::{JsonOrText, print_json_or_text}; use crate::diagnose_model; use crate::model_fit::assess_model_on_this_host; +/// What `--symptom` and `--engine` asked for: the error text, exactly as the +/// user gave it, and the engine they said it came from. One value because +/// `--engine` cannot be given without `--symptom`. +pub(crate) struct SymptomQuery { + pub(crate) text: String, + pub(crate) engine: Option, +} + pub(crate) fn diagnose( - symptom: Option, + symptom: Option, top: usize, json: bool, distro: Option, @@ -97,7 +105,9 @@ pub(crate) fn diagnose( .wsl .as_ref() .is_some_and(|wsl| !wsl.locally_probed); - let mut report = rocm_core::run_diagnose(&examination, &symptom.unwrap_or_default()); + let (symptom, engine) = + symptom.map_or((String::new(), None), |query| (query.text, query.engine)); + let mut report = rocm_core::run_diagnose_with_engine(&examination, &symptom, engine); // A prepared report carries no model verdict, and a GGUF estimate may read // the model's header from the network -- which `--report` promises it never // does -- so under `--report` the model is not assessed at all. @@ -142,6 +152,14 @@ impl Serialize for DiagnoseOutput<'_> { } impl JsonOrText for DiagnoseOutput<'_> { + // ASCII on the wire: the report can carry the user's paste verbatim + // (`provenance.symptom`) and host paths in evidence, and a Windows console + // on the OEM code page shows raw UTF-8 as mojibake (ROCMAI-543). The value + // is unchanged once the JSON is decoded. + fn render_json(&self) -> serde_json::Result { + rocm_core::render_diagnose_json(self.report) + } + fn render_text(&self) -> String { let mut text = rocm_core::render_diagnose_text(self.report, self.top); if let Some(readiness) = &self.report.model { @@ -240,6 +258,23 @@ fn prepare_cli_report( ) } +/// The report this machine would contribute for `diagnose`, the one document +/// meant to be shared. +/// +/// It takes from the diagnosis only the entry it established and whether a fix +/// was offered. Nothing the user typed reaches it: not the `--symptom` text, and +/// not the `provenance` that `--engine` adds to the diagnosis, which repeats +/// that text verbatim. A pasted log can carry paths, host names or tokens, and +/// this document is printed to be mailed. Pinned by +/// `a_shared_report_never_carries_the_pasted_symptom_or_its_provenance`. +fn shared_report( + examination: &rocm_core::Examination, + report: &rocm_core::DiagnoseReport, +) -> std::result::Result { + let (entry, fix_offered) = established_entry(report); + prepare_cli_report(examination, entry, fix_offered) +} + /// Print the report this machine would contribute, and send nothing. fn show_prepared_report( examination: &rocm_core::Examination, @@ -247,13 +282,12 @@ fn show_prepared_report( json: bool, send: bool, ) -> Result<()> { - let (entry, fix_offered) = established_entry(report); // Exit 0 either way. A refusal is this command working, not failing: it // decided correctly and said why, and a nonzero code would send a caller // looking for a fault. Anything scripting this reads the outcome from // `--json` rather than from the exit code, exactly as `rocm diagnose` itself // already asks callers to do. - match prepare_cli_report(examination, entry, fix_offered) { + match shared_report(examination, report) { Ok(prepared) => { if json { println!("{}", serde_json::to_string_pretty(&prepared)?); @@ -388,6 +422,64 @@ mod tests { ); } + /// ROCMAI-955: `--engine` makes the diagnosis repeat the user's paste + /// verbatim (`provenance`), and `--report` is built from that same + /// diagnosis. A paste can carry paths, host names or tokens, and the report + /// is the document printed for mailing, so neither the paste nor the + /// provenance may reach it -- not the JSON, and not the mail link `--send` + /// offers. + #[test] + fn a_shared_report_never_carries_the_pasted_symptom_or_its_provenance() { + let marker = "SECRET-TOKEN-0f3c"; + let paste = format!( + "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 2.00 GiB\n\ + at /home/someone/{marker}/serve.py on build-host-7" + ); + let examination = rocm_core::Examination { + os_family: "linux".to_owned(), + distro_id: "ubuntu".to_owned(), + distro_version: "22.04".to_owned(), + has_amd_gpu: true, + gpus: vec![rocm_core::examine::Gpu { + name: "test".to_owned(), + gfx_target: "gfx1100".to_owned(), + pci_id: String::new(), + is_apu: Some(false), + is_amd: true, + }], + ..rocm_core::Examination::default() + }; + let diagnosis = rocm_core::run_diagnose_with_engine( + &examination, + &paste, + Some(rocm_core::SymptomEngine::Vllm), + ); + assert_eq!( + diagnosis.provenance.as_ref().map(|p| p.symptom.as_str()), + Some(paste.as_str()), + "the premise: the diagnosis does carry the paste" + ); + let shared = + shared_report(&examination, &diagnosis).expect("a released GPU must produce a report"); + // Non-vacuity: the paste decided the entry, so it did flow into this + // report -- as a catalog id, and as nothing else. + assert_eq!(shared.entry, "fix-16-vllm-oom"); + + let json = serde_json::to_string_pretty(&shared).expect("a report serializes"); + let Delivery::Show(link) = rocm_core::report_delivery::deliver(&shared, false, &|_| None) + else { + panic!("a report that is not being sent is only shown"); + }; + for (surface, text) in [("report", &json), ("mail link", &link)] { + for leaked in [marker, "someone", "build-host-7", "provenance", "symptom"] { + assert!( + !text.contains(leaked), + "the {surface} carries {leaked:?} from the user's paste:\n{text}" + ); + } + } + } + /// An opener that records rather than opens, and can be told to fail. /// /// The whole reason the opener is a trait: the real one spawns a browser @@ -546,6 +638,7 @@ mod tests { }, out_of_scope: None, model: None, + provenance: None, } } @@ -601,6 +694,59 @@ mod tests { ); } + /// What `--json` prints is the ASCII rendering, not serde's raw UTF-8. + /// + /// The report carries a non-ASCII `provenance.symptom` (an accented path, a + /// box-drawing character, an emoji), so the two renderings differ: an + /// all-ASCII report would render the same either way and could not tell + /// whether `DiagnoseOutput` overrides `render_json` at all. A serialization + /// error comes back through the same `Result` rather than a panic (see + /// `a_failing_render_json_hook_is_an_error_and_writes_nothing`). + #[test] + fn diagnose_json_is_the_ascii_rendering_of_the_report() { + use std::fmt::Write as _; + let paste = "torch.OutOfMemoryError: HIP out of memory\n\u{2502} \ + /home/jos\u{e9}/serve.py \u{1f525}"; + let examination = rocm_core::Examination { + os_family: "linux".to_owned(), + ..rocm_core::Examination::default() + }; + let report = rocm_core::run_diagnose_with_engine( + &examination, + paste, + Some(rocm_core::SymptomEngine::Vllm), + ); + + let printed = output(&report, false) + .render_json() + .expect("the report renders"); + assert!( + printed.is_ascii(), + "--json must be ASCII on the wire:\n{printed}" + ); + + // Exactly the plain rendering with each non-ASCII character escaped, + // nothing more: the same document, only spelled in ASCII. + let plain = serde_json::to_string_pretty(&report).expect("the report serializes"); + assert!(!plain.is_ascii(), "the fixture must carry non-ASCII text"); + let backslash = char::from(92_u8); + let mut expected = String::new(); + for c in plain.chars() { + if c.is_ascii() { + expected.push(c); + } else { + for unit in c.encode_utf16(&mut [0_u16; 2]) { + let _ = write!(expected, "{backslash}u{unit:04x}"); + } + } + } + assert_eq!(printed, expected); + + let parsed: serde_json::Value = + serde_json::from_str(&printed).expect("the printed document is valid JSON"); + assert_eq!(parsed["provenance"]["symptom"].as_str(), Some(paste)); + } + #[test] fn diagnose_text_is_the_report_text_when_nothing_is_appended() { let report = host_report(); diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index 5d385cf235..712208d805 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -661,6 +661,7 @@ fn dispatch(cli: Cli) -> Result<()> { Some(Command::Diagnose { symptom, symptom_file, + engine, top, json, distro, @@ -675,6 +676,10 @@ fn dispatch(cli: Cli) -> Result<()> { let model = model.map(|model_ref| { diagnose_model::ModelQuery::new(model_ref, context_size, kv_cache_type) }); + let symptom = symptom.map(|text| diagnose_cmd::SymptomQuery { + text, + engine: engine.map(Into::into), + }); diagnose(symptom, top, json, distro, model, report, send) } // Keep this error chained rather than discarding it into a fresh diff --git a/apps/rocm/src/remote/doctor.rs b/apps/rocm/src/remote/doctor.rs index 33379fa766..58c3d85c61 100644 --- a/apps/rocm/src/remote/doctor.rs +++ b/apps/rocm/src/remote/doctor.rs @@ -365,6 +365,7 @@ mod tests { // that half rather than replacing it, so there is nothing for these // to say about one. model: None, + provenance: None, } } diff --git a/crates/rocm-core/src/diagnose.rs b/crates/rocm-core/src/diagnose.rs index 83f5533ad9..4b150b4753 100644 --- a/crates/rocm-core/src/diagnose.rs +++ b/crates/rocm-core/src/diagnose.rs @@ -23,7 +23,7 @@ use crate::examine::Examination; use catalog::{ - platform_family, run_all_checks, take_applicability_from_the_catalog, + platform_family, reads_engine_here, run_all_checks, take_applicability_from_the_catalog, uncovered_platform_message, }; use engine::scores_on_every_engine_host; @@ -43,7 +43,9 @@ mod wsl; pub use engine::{VLLM_IMPORT_CANONICAL_SYMPTOM, VLLM_OOM_CANONICAL_SYMPTOM}; pub use linux::{AMDGPU_MISSING_EVIDENCE, dkms_driver_plan_is_supported}; -pub use report::{SYMPTOM_DID_NOT_CONTRIBUTE_NOTE, render_report_text, upstream_route}; +pub use report::{ + SYMPTOM_DID_NOT_CONTRIBUTE_NOTE, render_report_json, render_report_text, upstream_route, +}; /// At/above this score a diagnosis is treated as a real match. pub const MIN_SCORE_FOR_MATCH: i32 = 50; @@ -51,6 +53,59 @@ pub const MIN_SCORE_FOR_MATCH: i32 = 50; /// At/above this score the agent may propose the fix immediately. pub const HIGH_CONFIDENCE: i32 = 75; +/// The engine a `--symptom` came from, when the user says so. +/// +/// Engine-specific catalog entries must not claim a generic framework error for +/// their engine, so they look for the engine's name in the pasted text (see +/// `check_16_vllm_oom`). A paste that is the engine's own error line often does +/// not carry the name, and editing the user's text to add one -- what the +/// rocm-doctor skill tells an agent to do today, with a `vllm: ` prefix -- +/// changes what the user said. Naming the engine instead states where the text +/// came from and leaves the text alone; the skill is to move to `--engine` in a +/// later change. +/// +/// Only the engines the catalog tells apart are listed: naming one changes +/// matching, so a value that changed nothing would read as if it had been +/// taken into account. The serialized names are the ones `rocm serve --engine` +/// takes. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum SymptomEngine { + /// The vLLM serving engine. + Vllm, +} + +impl SymptomEngine { + /// The name `--engine` takes and the JSON report records. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::Vllm => "vllm", + } + } +} + +/// Where a diagnosed symptom came from: the engine the user named, and the +/// text they passed, exactly as given. +/// +/// Recorded so a reader of the report can see that the engine was stated by +/// the user rather than found in the text, and that the text was not edited +/// to make it match. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub struct SymptomProvenance { + pub engine: SymptomEngine, + pub symptom: String, + /// Whether an entry that ran on the examined platform reads `engine`, that + /// is, whether naming it could change anything. The text report says so + /// when it could not (native Windows has no vLLM entry today), so the flag + /// is not silently ignored. + /// + /// Not serialized, so it adds nothing to the JSON report; `None` for a report + /// read back from JSON, where it is not known. + #[serde(skip)] + pub read_here: Option, +} + /// A proposed remediation for a [`Diagnosis`]. #[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)] pub struct Fix { @@ -166,6 +221,15 @@ pub struct DiagnoseReport { /// driver is also misconfigured is two findings, not one. #[serde(default)] pub model: Option, + /// The engine the symptom came from and the symptom itself, verbatim, when + /// the caller named an engine (`rocm diagnose --engine`). + /// + /// Absent otherwise, so a report diagnosed without an engine is unchanged. + /// Additive and optional: a consumer that does not know the field ignores + /// it, and `#[serde(default)]` lets a payload written before it existed + /// still deserialize. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub provenance: Option, } /// Whether any diagnosis cleared [`MIN_SCORE_FOR_MATCH`]. @@ -198,7 +262,9 @@ impl DiagnoseReport { /// being printed as a runnable lookalike of the user's error. #[must_use] pub fn vllm_oom_symptom_is_diagnosable(symptom: &str) -> bool { - scores_on_every_engine_host(symptom, "fix-16-vllm-oom", run_all_checks) + scores_on_every_engine_host(symptom, "fix-16-vllm-oom", |e, s| { + run_all_checks(e, s, None) + }) } /// Whether `symptom` clears [`MIN_SCORE_FOR_MATCH`] for the engine-startup @@ -216,7 +282,9 @@ pub fn vllm_oom_symptom_is_diagnosable(symptom: &str) -> bool { /// name a command that finds nothing. #[must_use] pub fn vllm_import_symptom_is_diagnosable(symptom: &str) -> bool { - scores_on_every_engine_host(symptom, "fix-17-torch-dlpack", run_all_checks) + scores_on_every_engine_host(symptom, "fix-17-torch-dlpack", |e, s| { + run_all_checks(e, s, None) + }) } /// Set [`Diagnosis::symptom_matched`] on every entry in `matched`. @@ -237,7 +305,9 @@ fn mark_symptom_contribution(matched: &mut [Diagnosis], e: &Examination, symptom if symptom.is_empty() { return; } - let host_only = run_all_checks(e, ""); + // No symptom, so no engine either: with nothing pasted the engine has + // nothing to read, and the run must be the host-only baseline. + let host_only = run_all_checks(e, "", None); for d in matched { d.symptom_matched = contributed(d, &host_only); } @@ -259,6 +329,24 @@ fn contributed(d: &Diagnosis, host_only: &[Diagnosis]) -> bool { /// Diagnose an examination against the closed catalog. #[must_use] pub fn diagnose(e: &Examination, symptom: &str) -> DiagnoseReport { + diagnose_with_engine(e, symptom, None) +} + +/// [`diagnose`], with the engine the symptom came from when the user named it. +/// +/// `engine` changes only what an engine-specific entry may score: with +/// [`SymptomEngine::Vllm`], every non-blank line of `symptom` counts as +/// vLLM's, as if each of those lines carried the word `vllm` (a blank line is +/// dropped, where a prefix would keep it; it scores nothing either way). The +/// symptom itself is never +/// edited, and it is recorded verbatim in [`DiagnoseReport::provenance`] +/// beside the engine. +#[must_use] +pub fn diagnose_with_engine( + e: &Examination, + symptom: &str, + engine: Option, +) -> DiagnoseReport { // WSL2 used to be short-circuited here as out of scope. It is a real platform // in the catalog now, with its own entries; what keeps the bare-metal checks // off it is `platform_family`, not a special case at this level. @@ -275,10 +363,17 @@ pub fn diagnose(e: &Examination, symptom: &str) -> DiagnoseReport { let mut matched = if out_of_scope.is_some() { Vec::new() } else { - run_all_checks(e, symptom) + run_all_checks(e, symptom, engine) }; mark_symptom_contribution(&mut matched, e, symptom); take_applicability_from_the_catalog(&mut matched, platform_family(e)); + // An out-of-scope platform ran nothing, so nothing read the engine either. + let engine_read_here = |engine| out_of_scope.is_none() && reads_engine_here(e, engine); + let provenance = engine.map(|engine| SymptomProvenance { + engine, + symptom: symptom.to_owned(), + read_here: Some(engine_read_here(engine)), + }); DiagnoseReport { has_match: any_cleared_threshold(&matched), matched, @@ -287,6 +382,7 @@ pub fn diagnose(e: &Examination, symptom: &str) -> DiagnoseReport { route_when_no_match: route_when_no_match(e), out_of_scope, model: None, + provenance, } } diff --git a/crates/rocm-core/src/diagnose/catalog.rs b/crates/rocm-core/src/diagnose/catalog.rs index 4635d4923b..73897f0499 100644 --- a/crates/rocm-core/src/diagnose/catalog.rs +++ b/crates/rocm-core/src/diagnose/catalog.rs @@ -5,7 +5,6 @@ //! The catalog: which check runs on which platform family, in what order, and how //! the family is resolved from an [`Examination`]. -use super::Diagnosis; use super::engine::{ check_16_vllm_oom, check_17_torch_dlpack_cuda_variant, check_18_comgr_conflict, check_19_shm_too_small, @@ -28,6 +27,7 @@ use super::wsl::{ check_wsl_4_rocdxg_not_linked, check_wsl_5_distro_too_old, check_wsl_6_host_driver_too_old, check_wsl_7_wsl1, }; +use super::{Diagnosis, SymptomEngine}; use crate::examine::Examination; // --------------------------------------------------------------------------- @@ -45,8 +45,48 @@ use crate::examine::Examination; // left with none. That compiles, and clippy, rustfmt and every test pass. // --------------------------------------------------------------------------- +/// How a checker reads its input. +#[derive(Clone, Copy)] +enum Check { + /// The examination and the symptom text: every entry but the + /// engine-specific ones. + Text(fn(&Examination, &str) -> Diagnosis), + /// Also the engine the user said the symptom came from, for an entry that + /// otherwise has to find the engine's name in the text. The first field is + /// the engine the entry reads; [`reads_engine_here`] answers from it. + WithEngine( + SymptomEngine, + fn(&Examination, &str, Option) -> Diagnosis, + ), +} + +impl Check { + fn run(self, e: &Examination, symptom: &str, engine: Option) -> Diagnosis { + match self { + Self::Text(check) => check(e, symptom), + Self::WithEngine(_, check) => check(e, symptom, engine), + } + } +} + +/// Whether any entry that runs on `e`'s platform reads `engine`: whether naming +/// it with `rocm diagnose --engine` can change anything there. +/// +/// Read from the same table, and filtered on the same platform family, as +/// [`run_all_checks`], so the text report's line about the named engine is +/// computed from the decision that ran rather than from a list of operating +/// systems kept beside it. False on a platform the catalog does not cover at +/// all, where nothing runs. +pub(super) fn reads_engine_here(e: &Examination, engine: SymptomEngine) -> bool { + let family = platform_family(e); + CHECKERS.iter().any(|(check, applicable)| { + applicable.contains(&family) + && matches!(check, Check::WithEngine(read, _) if *read == engine) + }) +} + /// A checker plus the OS families it applies to. -type Checker = (fn(&Examination, &str) -> Diagnosis, &'static [&'static str]); +type Checker = (Check, &'static [&'static str]); // The `wsl` entries below run on less evidence than on bare metal: WSL collects // no GPU topology (`gpus` is empty, because the probes that fill it read KFD and @@ -59,57 +99,81 @@ type Checker = (fn(&Examination, &str) -> Diagnosis, &'static [&'static str]); // a check may miss a real fault, never that it invents one. Pinned by // `the_shared_checks_under_report_on_wsl_rather_than_over_report`. const CHECKERS: &[Checker] = &[ - (check_1_arch_not_in_wheel, &["linux", "windows", "wsl"]), - (check_2_hsa_override_unneeded, &["linux", "windows", "wsl"]), - (check_3_rocm_kernel_unsupported, &["linux"]), - (check_4_render_group, &["linux"]), - (check_5_amdgpu_blacklisted, &["linux"]), - (check_6_path_missing, &["linux", "windows", "wsl"]), - (check_7_stale_repos, &["linux"]), - (check_8_wheel_rocm_mismatch, &["linux", "windows", "wsl"]), + ( + Check::Text(check_1_arch_not_in_wheel), + &["linux", "windows", "wsl"], + ), + ( + Check::Text(check_2_hsa_override_unneeded), + &["linux", "windows", "wsl"], + ), + (Check::Text(check_3_rocm_kernel_unsupported), &["linux"]), + (Check::Text(check_4_render_group), &["linux"]), + (Check::Text(check_5_amdgpu_blacklisted), &["linux"]), + ( + Check::Text(check_6_path_missing), + &["linux", "windows", "wsl"], + ), + (Check::Text(check_7_stale_repos), &["linux"]), + ( + Check::Text(check_8_wheel_rocm_mismatch), + &["linux", "windows", "wsl"], + ), // Not "wsl": WSL2 exposes no per-device topology to collide over. - (check_9_igpu_dgpu_collision, &["linux", "windows"]), - (check_10_container_devices, &["linux"]), - (check_11_iommu_hang, &["linux"]), - (check_12_repo_native_broken, &["linux"]), - (check_13_hip_sdk_missing, &["windows"]), - (check_14_adrenalin_too_old, &["windows"]), - (check_15_msvc_redist, &["windows"]), - (check_16_vllm_oom, &["linux", "wsl"]), + ( + Check::Text(check_9_igpu_dgpu_collision), + &["linux", "windows"], + ), + (Check::Text(check_10_container_devices), &["linux"]), + (Check::Text(check_11_iommu_hang), &["linux"]), + (Check::Text(check_12_repo_native_broken), &["linux"]), + (Check::Text(check_13_hip_sdk_missing), &["windows"]), + (Check::Text(check_14_adrenalin_too_old), &["windows"]), + (Check::Text(check_15_msvc_redist), &["windows"]), + ( + Check::WithEngine(SymptomEngine::Vllm, check_16_vllm_oom), + &["linux", "wsl"], + ), // Linux and WSL2: `libtorch_cuda.so` is an ELF name, and the vLLM engine is // gated off native Windows but runs on WSL2. The engine's startup-failure // hint fires on both, so the entry it points at must be reachable on both. // 17 rather than 16 because the vLLM out-of-memory entry above already // holds 16; the number is a stable handle, so the two do not get to share // one. - (check_17_torch_dlpack_cuda_variant, &["linux", "wsl"]), - (check_wsl_1_gpu_not_exposed, WSL_ONLY), - (check_wsl_2_dxcore_missing, WSL_ONLY), - (check_wsl_3_rocdxg_missing, WSL_ONLY), - (check_wsl_4_rocdxg_not_linked, WSL_ONLY), - (check_wsl_5_distro_too_old, WSL_ONLY), - (check_wsl_6_host_driver_too_old, WSL_ONLY), - (check_wsl_7_wsl1, WSL_ONLY), + ( + Check::Text(check_17_torch_dlpack_cuda_variant), + &["linux", "wsl"], + ), + (Check::Text(check_wsl_1_gpu_not_exposed), WSL_ONLY), + (Check::Text(check_wsl_2_dxcore_missing), WSL_ONLY), + (Check::Text(check_wsl_3_rocdxg_missing), WSL_ONLY), + (Check::Text(check_wsl_4_rocdxg_not_linked), WSL_ONLY), + (Check::Text(check_wsl_5_distro_too_old), WSL_ONLY), + (Check::Text(check_wsl_6_host_driver_too_old), WSL_ONLY), + (Check::Text(check_wsl_7_wsl1), WSL_ONLY), // Both families, opting in explicitly as the platform split requires. Which // copy of a library the loader picks is not a question about the amdgpu // module or the Windows host driver -- a wheel copy and a system copy // collide on WSL2 exactly as they do on bare metal. Windows is deferred // until this has proved the approach. - (check_18_comgr_conflict, &["linux", "wsl"]), + (Check::Text(check_18_comgr_conflict), &["linux", "wsl"]), // Both families, for the same reason. The shortage has nothing to do with // `amdgpu` or `/dev/kfd` -- it is the size of a tmpfs -- and WSL2 ships the // same 64 MB default a container does, so leaving this tagged `linux` alone // would silence it on one of the two platforms most likely to have it. - (check_19_shm_too_small, &["linux", "wsl"]), + (Check::Text(check_19_shm_too_small), &["linux", "wsl"]), // Not "wsl": WSL2 reports no GPU topology (`gpus` is empty), so this check // has nothing to read there and would never fire. - (check_20_gpu_unsupported_by_rocm, &["linux", "windows"]), + ( + Check::Text(check_20_gpu_unsupported_by_rocm), + &["linux", "windows"], + ), // Windows only: the runtime is read from the Windows System32 folder. A WSL // distro is examined as a distro, so it has no such runtime to read. - (check_21_msvc_redist_outdated, &["windows"]), + (Check::Text(check_21_msvc_redist_outdated), &["windows"]), // Windows only: the driver date comes from the Windows display adapter. // WSL2 reads the host driver through its own entries. - (check_22_display_driver_old, &["windows"]), + (Check::Text(check_22_display_driver_old), &["windows"]), ]; const WSL_ONLY: &[&str] = &["wsl"]; @@ -140,12 +204,16 @@ pub(super) const fn platform_family(e: &Examination) -> &str { /// Run every applicable checker, drop zero-score results, sort by score /// descending (stable, so ties keep catalog order). -pub(super) fn run_all_checks(e: &Examination, symptom: &str) -> Vec { +pub(super) fn run_all_checks( + e: &Examination, + symptom: &str, + engine: Option, +) -> Vec { let family = platform_family(e); let mut results: Vec = CHECKERS .iter() .filter(|(_, applicable)| applicable.contains(&family)) - .map(|(check, _)| check(e, symptom)) + .map(|(check, _)| check.run(e, symptom, engine)) .filter(|d| d.score > 0) .collect(); // Stable sort by score descending: ties keep catalog order. @@ -213,10 +281,13 @@ mod tests { use super::*; use crate::diagnose::engine::VLLM_OOM_CANONICAL_SYMPTOM; use crate::diagnose::test_support::{ - igpu_dgpu_host, linux_base, windows_driver_exam, wsl_base, + NON_ASCII_PASTE, every_engine, host_of_family, igpu_dgpu_host, linux_base, + windows_driver_exam, wsl_base, }; use crate::diagnose::windows::DISPLAY_DRIVER_OLD_SCORE; - use crate::diagnose::{MIN_SCORE_FOR_MATCH, diagnose, render_report_text}; + use crate::diagnose::{ + MIN_SCORE_FOR_MATCH, diagnose, diagnose_with_engine, render_report_json, render_report_text, + }; use crate::examine::Gpu; /// A pasted HIP-loading error still opens fix-13 when no AMD GPU was @@ -330,6 +401,51 @@ mod tests { } } + /// The engine a `Check::WithEngine` entry declares is the engine it reads. + /// + /// `reads_engine_here` answers the text report's "changed nothing" line + /// from that declaration alone, so a declaration that disagreed with the + /// checker would make the line lie. For every such entry, on every + /// platform family it applies to, and for every engine variant: a paste + /// that only the declared engine can claim (a torch out-of-memory error + /// naming no engine) must change the entry's verdict when exactly the + /// declared engine is named, and must not change it for any other. + #[test] + fn every_engine_reading_entry_reads_the_engine_it_declares() { + const NEEDS_THE_ENGINE: &str = + "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 2.00 GiB"; + let mut checked = 0; + for (check, applicable) in CHECKERS { + let Check::WithEngine(declared, _) = check else { + continue; + }; + for family in *applicable { + let host = host_of_family(family); + let unnamed = check.run(&host, NEEDS_THE_ENGINE, None); + for engine in every_engine() { + let named = check.run(&host, NEEDS_THE_ENGINE, Some(engine)); + let changed = + (named.score, &named.evidence) != (unnamed.score, &unnamed.evidence); + assert_eq!( + changed, + engine == *declared, + "{} on {family}: declares {declared:?}, but naming {engine:?} {} its \ + verdict ({} -> {})", + unnamed.id, + if changed { "changed" } else { "did not change" }, + unnamed.score, + named.score + ); + checked += 1; + } + } + } + assert!( + checked > 0, + "no Check::WithEngine entry was checked, so a mis-declared engine would go unseen" + ); + } + /// ROCMAI-543: every string `rocm diagnose` can put on a redirectable /// stream has to be plain ASCII, or a Windows console on the OEM code page /// (437) renders UTF-8 punctuation as mojibake like `ΓÇö`. @@ -339,62 +455,107 @@ mod tests { /// check added later is covered the day it lands and no single sentence /// needs a pinned copy here -- a test that restates the constant it checks /// protects very little. Both surfaces are asserted because both are - /// redirectable: the rendered text report, and the `--json` payload, which - /// `serde_json` emits as raw UTF-8 rather than `\u` escapes. + /// redirectable: the rendered text report, and the `--json` payload as the + /// CLI prints it ([`render_report_json`]). /// /// Two of this fix's conversions sit on branches a default `Examination` /// never reaches -- the dGPU/APU pairing in `check_9_igpu_dgpu_collision` /// and `check_16_vllm_oom`'s "does not fit" wording -- so the corpus lights /// them deliberately and the coverage guard at the end fails if it stops. + /// + /// The `--engine` rows paste text this crate did not write: an accented + /// path, a box-drawing character and an emoji. `provenance.symptom` repeats + /// it verbatim, so the JSON is ASCII only because the CLI escapes it, and + /// the text report is ASCII only because nothing in it echoes the paste. #[test] fn every_message_the_diagnose_catalog_can_emit_is_ascii() { - let corpus: Vec<(&str, Examination, &str)> = vec![ - ("linux, no symptom", linux_base(), ""), - ("wsl, no symptom", wsl_base(), ""), + let no_engine = None; + let vllm = Some(SymptomEngine::Vllm); + let windows = Examination { + os_family: "windows".to_owned(), + ..Examination::default() + }; + let corpus: Vec<(&str, Examination, &str, Option)> = vec![ + ("linux, no symptom", linux_base(), "", no_engine), + ("wsl, no symptom", wsl_base(), "", no_engine), ( "linux apu+dgpu, crash", igpu_dgpu_host("linux"), "torch crashes with a segfault", + no_engine, ), ( "windows apu+dgpu, crash", igpu_dgpu_host("windows"), "torch crashes with a segfault", + no_engine, + ), + ( + "linux vllm oom", + linux_base(), + VLLM_OOM_CANONICAL_SYMPTOM, + no_engine, + ), + ( + "wsl vllm oom", + wsl_base(), + VLLM_OOM_CANONICAL_SYMPTOM, + no_engine, + ), + ( + "linux, non-ASCII paste, --engine", + linux_base(), + NON_ASCII_PASTE, + vllm, + ), + ( + "wsl, non-ASCII paste, --engine", + wsl_base(), + NON_ASCII_PASTE, + vllm, + ), + ( + "windows, non-ASCII paste, --engine", + windows, + NON_ASCII_PASTE, + vllm, ), - ("linux vllm oom", linux_base(), VLLM_OOM_CANONICAL_SYMPTOM), - ("wsl vllm oom", wsl_base(), VLLM_OOM_CANONICAL_SYMPTOM), ]; let mut everything_seen = String::new(); - for (label, examination, symptom) in &corpus { + for (label, examination, symptom, engine) in &corpus { // Every catalog entry, not only the ones that scored high enough to // be reported on this host: a message is still shipped text when the // host it describes is not the one under test. for (check, _) in CHECKERS { - let diagnosis = check(examination, symptom); + let diagnosis = check.run(examination, symptom, *engine); let json = serde_json::to_string(&diagnosis).expect("a Diagnosis serializes"); assert!( json.is_ascii(), - "{label}: diagnosis `{}` carries non-ASCII punctuation, which renders as \ + "{label}: diagnosis `{}` carries non-ASCII text, which renders as \ mojibake on a Windows console using the OEM code page (ROCMAI-543): {json}", diagnosis.id ); everything_seen.push_str(&json); } - let report = diagnose(examination, symptom); + let report = diagnose_with_engine(examination, symptom, *engine); let text = render_report_text(&report, report.matched.len()); assert!( text.is_ascii(), "{label}: `rocm diagnose` text output must be ASCII-only (ROCMAI-543):\n{text}" ); - let json = serde_json::to_string(&report).expect("a DiagnoseReport serializes"); + let json = render_report_json(&report).expect("the report renders"); assert!( json.is_ascii(), "{label}: `rocm diagnose --json` output must be ASCII-only (ROCMAI-543): {json}" ); everything_seen.push_str(&text); } + assert!( + everything_seen.contains("Engine: vllm (named with --engine; no entry"), + "the corpus no longer reaches the --engine rows, so the escaping is unguarded" + ); // An invariant test that quietly stops reaching the branches it was // written for passes for the wrong reason. These markers sit beside the @@ -586,7 +747,7 @@ mod tests { // for it on both platforms its checker is registered for, and the // `apply with:` line diagnose prints for it is live like every other. for (check, families) in CHECKERS { - let id = check(&Examination::default(), "").id; + let id = check.run(&Examination::default(), "", None).id; let applies_on = crate::fix::recipe_applies_on(&id).unwrap_or_else(|| { panic!( "checker id `{id}` has no recipe in the fix catalog, so the \ @@ -627,6 +788,7 @@ mod tests { ..Examination::default() }, VLLM_OOM_CANONICAL_SYMPTOM, + None, ); let msvc = check_15_msvc_redist( &Examination { @@ -764,7 +926,7 @@ mod tests { if !families.contains(&"windows") { continue; } - let Some(fix) = check(&e, symptom).fix else { + let Some(fix) = check.run(&e, symptom, None).fix else { continue; }; let printed = fix diff --git a/crates/rocm-core/src/diagnose/engine.rs b/crates/rocm-core/src/diagnose/engine.rs index e476b8358c..1c32f3a64d 100644 --- a/crates/rocm-core/src/diagnose/engine.rs +++ b/crates/rocm-core/src/diagnose/engine.rs @@ -13,7 +13,7 @@ use super::scoring::{ KEYWORDS_COMGR_CONFLICT, KEYWORDS_SHM_TOO_SMALL, KEYWORDS_TORCH_DLPACK_CUDA_VARIANT, KEYWORDS_VLLM_OOM, finalize, keyword_score, keyword_score_collapsing_overlaps, zero, }; -use super::{Diagnosis, Fix, MIN_SCORE_FOR_MATCH}; +use super::{Diagnosis, Fix, MIN_SCORE_FOR_MATCH, SymptomEngine}; use crate::examine::Examination; use regex::Regex; @@ -34,6 +34,9 @@ use regex::Regex; /// `matched` but can never carry a verdict, and it can no longer lend its anchor /// to an OOM elsewhere in the paste. It stays an anchor because the flag is /// vLLM-specific, and because a real vLLM OOM often names it on the failing line. +/// None of this applies when the user names vLLM with `rocm diagnose --engine`: +/// every non-blank line is then vLLM's by their account, and the anchor plays +/// no part. const VLLM_ANCHOR_PATTERN: &str = r"vllm|gpu[-_]memory[-_]utilization"; /// A canonical vLLM-OOM `--symptom` string guaranteed to clear @@ -52,7 +55,8 @@ const VLLM_ANCHOR_PATTERN: &str = r"vllm|gpu[-_]memory[-_]utilization"; pub const VLLM_OOM_CANONICAL_SYMPTOM: &str = "vllm: torch.OutOfMemoryError: HIP out of memory"; /// The lines of `symptom` that carry the vLLM OOM anchor (`vllm` or -/// `gpu_memory_utilization`), joined by newlines — the only text +/// `gpu_memory_utilization`), joined by newlines — or every non-blank line when +/// the user named vLLM as the engine (`engine`, below). That is the only text /// [`check_16_vllm_oom`] is allowed to score. /// /// Co-occurrence anywhere in a pasted log is too loose -- a stray `vllm` mention @@ -114,13 +118,29 @@ pub const VLLM_OOM_CANONICAL_SYMPTOM: &str = "vllm: torch.OutOfMemoryError: HIP /// silently stops diagnosing it (measured: 95 → 0). /// /// `\r\n` yields an empty segment, which carries no anchor and is dropped. -fn vllm_anchored_lines(symptom: &str) -> String { +/// +/// `engine` is the engine the user said the paste came from. With +/// [`SymptomEngine::Vllm`] the whole paste is vLLM's by the user's own +/// account, so every non-blank rendered line counts as anchored, as if the +/// user had written `vllm` at the start of each of those lines, without +/// editing their text. Blank lines are dropped, where a prefix would keep them; +/// a blank line scores nothing either way, so the verdict is the same. +/// The split still applies then, so a pattern cannot straddle two rendered +/// lines either way. +fn vllm_anchored_lines(symptom: &str, engine: Option) -> String { let Ok(anchor) = Regex::new(VLLM_ANCHOR_PATTERN) else { return String::new(); }; + let stated_vllm = engine == Some(SymptomEngine::Vllm); crate::terminal::rendered_lines(symptom) .into_iter() - .filter(|line| anchor.is_match(&line.to_lowercase())) + .filter(|line| { + if stated_vllm { + !line.trim().is_empty() + } else { + anchor.is_match(&line.to_lowercase()) + } + }) .collect::>() .join("\n") } @@ -129,15 +149,27 @@ fn vllm_anchored_lines(symptom: &str) -> String { /// per-GPU VRAM or tenancy fields, so nothing structural can corroborate this — /// the user must pass the error via `--symptom`, or arrive from the serve /// failure note that points here. `e` is therefore unused. -pub(super) fn check_16_vllm_oom(_e: &Examination, symptom: &str) -> Diagnosis { - // Score the anchored lines only, never the whole paste. The anchor is the - // *only* thing separating this failure mode from any other framework's - // PyTorch/HIP OOM, so a boolean gate over the whole symptom is not enough: - // keyword hits from unanchored lines would still count at full weight. Any - // future change here must keep the scored text restricted to anchored lines. - let anchored = vllm_anchored_lines(symptom); +/// +/// `engine` is the engine the user named with `rocm diagnose --engine`. Naming +/// vLLM stands in for the anchor on every non-blank line (see +/// [`vllm_anchored_lines`]). +pub(super) fn check_16_vllm_oom( + _e: &Examination, + symptom: &str, + engine: Option, +) -> Diagnosis { + // Score the anchored lines only, never the whole paste -- unless the user + // named vLLM with `--engine`, when every non-blank line is vLLM's by their + // account and all of them are scored. Otherwise the anchor is the *only* thing + // separating this failure mode from any other framework's PyTorch/HIP OOM, + // so a boolean gate over the whole symptom is not enough: keyword hits from + // unanchored lines would still count at full weight. Any future change here + // must keep the scored text restricted to anchored lines whenever the user + // did not name the engine. + let anchored = vllm_anchored_lines(symptom, engine); if anchored.is_empty() { - // No vLLM anchor anywhere: don't attribute a bare framework OOM to vLLM. + // No vLLM anchor anywhere, and the user did not say the paste was + // vLLM's: don't attribute a bare framework OOM to vLLM. return zero("fix-16-vllm-oom", "vLLM GPU out of memory"); } let (score, evidence) = keyword_score_collapsing_overlaps(&anchored, KEYWORDS_VLLM_OOM); @@ -535,7 +567,9 @@ pub(super) fn check_19_shm_too_small(e: &Examination, symptom: &str) -> Diagnosi mod tests { use super::*; use crate::diagnose::test_support::{linux_base, wsl_base}; - use crate::diagnose::{DiagnoseReport, HIGH_CONFIDENCE, diagnose}; + use crate::diagnose::{ + DiagnoseReport, HIGH_CONFIDENCE, diagnose, diagnose_with_engine, render_report_text, + }; fn shm_finding(report: &DiagnoseReport) -> Option<&Diagnosis> { report @@ -1212,6 +1246,301 @@ mod tests { assert!(!report.has_match()); } + /// The scored part of a report: which entries matched, how strongly, and on + /// what evidence. Everything a caller acts on, without the provenance. + fn verdict(report: &DiagnoseReport) -> Vec<(String, i32, Vec)> { + report + .matched + .iter() + .map(|d| (d.id.clone(), d.score, d.evidence.clone())) + .collect() + } + + /// ROCMAI-955: naming the engine is the replacement for the `vllm: ` prefix + /// the rocm-doctor skill tells an agent to write into the user's paste (the + /// skill moves to `--engine` in a later change). It has to match exactly as + /// the prefixed text does, on both families vLLM runs on, while the + /// unprefixed paste on its own still does not match. + #[test] + fn naming_vllm_matches_the_paste_as_the_vllm_prefix_did() { + let paste = "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 2.00 GiB"; + for host in [linux_base(), wsl_base()] { + let named = diagnose_with_engine(&host, paste, Some(SymptomEngine::Vllm)); + let prefixed = diagnose(&host, &format!("vllm: {paste}")); + assert_eq!(verdict(&named), verdict(&prefixed)); + assert_eq!(named.has_match, prefixed.has_match); + let oom = named + .matched + .iter() + .find(|d| d.id == "fix-16-vllm-oom") + .expect("naming vLLM must reach the vLLM OOM entry"); + assert!(oom.score >= HIGH_CONFIDENCE, "score was {}", oom.score); + assert!( + diagnose(&host, paste) + .matched + .iter() + .all(|d| d.id != "fix-16-vllm-oom"), + "without the engine the bare paste must still not be attributed to vLLM" + ); + } + } + + /// A pasted log, not one line: with vLLM named, every non-blank line is + /// vLLM's, so the result is that of `vllm` written on each of them -- not + /// only on the first, which is all a prefix on the paste ever anchored. + #[test] + fn naming_vllm_anchors_every_line_of_a_multi_line_paste() { + let lines = [ + "INFO 11-11 10:00:00 Loading model weights took 14.9 GiB", + "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 7.21 GiB.", + ]; + let named = + diagnose_with_engine(&linux_base(), &lines.join("\n"), Some(SymptomEngine::Vllm)); + let every_line_prefixed = diagnose( + &linux_base(), + &lines + .iter() + .map(|line| format!("vllm: {line}")) + .collect::>() + .join("\n"), + ); + assert_eq!(verdict(&named), verdict(&every_line_prefixed)); + assert!( + named + .matched + .iter() + .any(|d| d.id == "fix-16-vllm-oom" && d.score >= HIGH_CONFIDENCE) + ); + } + + /// Naming the engine is provenance for the vLLM entry only: an unrelated + /// failure is diagnosed exactly as without it, and on native Windows, where + /// the vLLM engine does not run, it still produces no vLLM verdict. + #[test] + fn naming_vllm_changes_nothing_but_the_vllm_entry() { + let isa = "HSA_STATUS_ERROR_INVALID_ISA"; + assert_eq!( + verdict(&diagnose_with_engine( + &linux_base(), + isa, + Some(SymptomEngine::Vllm) + )), + verdict(&diagnose(&linux_base(), isa)) + ); + let windows = Examination { + os_family: "windows".to_owned(), + ..Examination::default() + }; + let oom = "torch.OutOfMemoryError: HIP out of memory"; + let named = diagnose_with_engine(&windows, oom, Some(SymptomEngine::Vllm)); + assert!(named.matched.iter().all(|d| d.id != "fix-16-vllm-oom")); + assert_eq!(verdict(&named), verdict(&diagnose(&windows, oom))); + } + + /// The report says which engine the user named and carries the symptom + /// byte for byte -- control bytes and all -- and says nothing of either + /// when no engine was named, so a report without one gains no field. + #[test] + fn the_report_records_the_named_engine_and_the_symptom_verbatim() { + let paste = " torch.OutOfMemoryError: HIP out of memory\r\n\u{1b}[31mTried\u{1b}[0m "; + let named = diagnose_with_engine(&linux_base(), paste, Some(SymptomEngine::Vllm)); + let json = serde_json::to_value(&named).expect("a report serializes"); + assert_eq!(json["provenance"]["engine"], "vllm"); + assert_eq!(json["provenance"]["symptom"], paste); + + let unnamed = serde_json::to_value(diagnose(&linux_base(), paste)).unwrap(); + assert!( + unnamed.get("provenance").is_none(), + "a report with no engine named must not grow the field: {unnamed}" + ); + } + + /// ROCMAI-955 review: the `--json` the CLI prints is ASCII on the wire, yet + /// the paste in it is the paste, exactly. A JSON parser reading the escaped + /// document back gets the original bytes; nothing was stripped or + /// transliterated. And a document with nothing to escape is byte-for-byte + /// what plain pretty `serde_json` would have printed. + #[test] + fn a_non_ascii_paste_round_trips_through_the_json_exactly() { + use crate::diagnose::render_report_json; + use crate::diagnose::test_support::NON_ASCII_PASTE; + let windows = Examination { + os_family: "windows".to_owned(), + ..Examination::default() + }; + for host in [linux_base(), wsl_base(), windows] { + let report = diagnose_with_engine(&host, NON_ASCII_PASTE, Some(SymptomEngine::Vllm)); + let json = render_report_json(&report).expect("the report renders"); + assert!(json.is_ascii(), "{json}"); + // The emoji is beyond the Basic Multilingual Plane, so it travels + // as a UTF-16 surrogate pair, the form every JSON parser accepts. + let backslash = char::from(92_u8); + let surrogate_pair = format!("{backslash}ud83d{backslash}udd25"); + assert!(json.contains(&surrogate_pair), "{json}"); + let parsed: serde_json::Value = + serde_json::from_str(&json).expect("the escaped document is valid JSON"); + assert_eq!( + parsed["provenance"]["symptom"].as_str().map(str::as_bytes), + Some(NON_ASCII_PASTE.as_bytes()), + "the paste must come back byte for byte" + ); + assert_eq!( + serde_json::from_str::(&json) + .expect("it reads back as a report") + .provenance + .map(|p| p.symptom), + Some(NON_ASCII_PASTE.to_owned()) + ); + } + + let ascii_only = diagnose(&linux_base(), "torch.OutOfMemoryError"); + assert_eq!( + render_report_json(&ascii_only).expect("the report renders"), + serde_json::to_string_pretty(&ascii_only).unwrap(), + "escaping must not change a document that has nothing to escape" + ); + } + + /// `SymptomEngine::name()` (what the text report prints) and the serde name + /// (what `provenance.engine` holds) are one contract, for every variant. + /// `every_engine` is checked against the enum by an exhaustive `match`, so + /// a new variant fails to compile until it is listed there. + #[test] + fn every_engine_name_is_its_serialized_name() { + for engine in crate::diagnose::test_support::every_engine() { + assert_eq!( + serde_json::to_value(engine).expect("an engine serializes"), + serde_json::Value::from(engine.name()), + "{engine:?}" + ); + } + } + + /// The text report opens with the engine `--engine` named, and on a + /// platform where no entry reads it (native Windows, and a platform the + /// catalog does not cover) says the flag changed nothing. The line is + /// checked against the behaviour it describes: where it says nothing + /// changed, the diagnosis equals the one without the flag, and where it + /// says the symptom was read as vLLM's, a vLLM paste does differ. Read as, + /// not matched as: an unrelated or empty paste is read the same way and + /// matches nothing for vLLM, which the last rows pin. + #[test] + fn the_text_report_says_which_engine_was_named_and_whether_it_counted() { + let oom = "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 2.00 GiB"; + let windows = Examination { + os_family: "windows".to_owned(), + ..Examination::default() + }; + let uncovered = Examination { + os_family: "freebsd".to_owned(), + ..Examination::default() + }; + let nothing_changed = "no entry for vllm exists on this OS, so the flag changed nothing"; + for (label, host, read) in [ + ("linux", linux_base(), true), + ("wsl", wsl_base(), true), + ("windows", windows, false), + ("uncovered", uncovered, false), + ] { + let named = diagnose_with_engine(&host, oom, Some(SymptomEngine::Vllm)); + let text = render_report_text(&named, 5); + let first = text.lines().next().unwrap_or_default(); + assert!( + first.starts_with("Engine: vllm (named with --engine;"), + "{label}: the text report must open with the named engine:\n{text}" + ); + assert_eq!( + first.contains(nothing_changed), + !read, + "{label}: wrong claim about whether the flag counted: {first}" + ); + assert_eq!( + verdict(&named) == verdict(&diagnose(&host, oom)), + !read, + "{label}: the line must agree with what the flag did to the diagnosis" + ); + // The line is added above the report and takes nothing from it: + // the per-cause `source:` notes and everything else render as they + // would for the same diagnosis with no engine line. + let without_line = DiagnoseReport { + provenance: None, + ..named.clone() + }; + assert_eq!( + text.strip_prefix(first) + .and_then(|rest| rest.strip_prefix("\n\n")), + Some(render_report_text(&without_line, 5).as_str()), + "{label}: the engine line must sit above the report and change nothing in it" + ); + } + // A Linux host with a host-state finding (both GPU groups missing) that + // the paste does not touch: the `source:` note marks that cause, and + // sits in the same report as the engine line. + let mut groups_missing = linux_base(); + groups_missing.in_render_group = Some(false); + groups_missing.in_video_group = Some(false); + let linux_text = render_report_text( + &diagnose_with_engine(&groups_missing, oom, Some(SymptomEngine::Vllm)), + 50, + ); + assert!( + linux_text.starts_with("Engine: vllm") + && linux_text.contains(crate::diagnose::SYMPTOM_DID_NOT_CONTRIBUTE_NOTE), + "the engine line and the per-cause source note must both appear:\n{linux_text}" + ); + + assert!( + !render_report_text(&diagnose(&linux_base(), oom), 5).contains("Engine:"), + "no engine named, no line" + ); + + // The missing row of the matrix: the engine is named on a host whose + // catalog reads it, but nothing for vLLM is in the paste. The line says + // how the symptom was read, which is still true, and claims no match -- + // and the report holds no vLLM cause beside it. + let read_line = "Engine: vllm (named with --engine; every non-blank line of the symptom \ + was read as vllm's)."; + for (label, paste) in [ + ( + "unrelated paste", + "Traceback (most recent call last):\n File \"train.py\", line 3, in \n\ + ZeroDivisionError: division by zero", + ), + ("empty paste", ""), + ] { + let named = diagnose_with_engine(&linux_base(), paste, Some(SymptomEngine::Vllm)); + let text = render_report_text(&named, 50); + assert_eq!(text.lines().next(), Some(read_line), "{label}:\n{text}"); + assert!( + named.matched.iter().all(|d| d.id != "fix-16-vllm-oom") + && !text.contains("fix-16-vllm-oom"), + "{label}: nothing for vLLM is in the paste, so no vLLM cause may be reported \ + beside the line:\n{text}" + ); + } + + // The line adds nothing to the JSON report: `provenance` holds exactly the + // engine and the symptom, and a report read back from it does not know + // whether the engine was read, so it claims neither. + let json = serde_json::to_value(diagnose_with_engine( + &linux_base(), + oom, + Some(SymptomEngine::Vllm), + )) + .expect("a report serializes"); + let keys: Vec<&String> = json["provenance"] + .as_object() + .expect("provenance is an object") + .keys() + .collect(); + assert_eq!(keys, ["engine", "symptom"]); + let read_back: DiagnoseReport = serde_json::from_value(json).expect("it reads back"); + assert_eq!( + render_report_text(&read_back, 5).lines().next(), + Some("Engine: vllm (named with --engine).") + ); + } + #[test] fn torch_oom_class_alone_is_not_a_vllm_match() { // `outofmemory` is nested inside the exception class name. It must not diff --git a/crates/rocm-core/src/diagnose/host.rs b/crates/rocm-core/src/diagnose/host.rs index 32d43db2b7..23a2aee85d 100644 --- a/crates/rocm-core/src/diagnose/host.rs +++ b/crates/rocm-core/src/diagnose/host.rs @@ -911,6 +911,7 @@ mod tests { }, out_of_scope: None, model: None, + provenance: None, }; let out = render_report_text(&report, 5); assert!( diff --git a/crates/rocm-core/src/diagnose/report.rs b/crates/rocm-core/src/diagnose/report.rs index 0ea6614080..569b2c5952 100644 --- a/crates/rocm-core/src/diagnose/report.rs +++ b/crates/rocm-core/src/diagnose/report.rs @@ -81,11 +81,86 @@ fn is_advice_only(fix_id: &str) -> bool { pub const SYMPTOM_DID_NOT_CONTRIBUTE_NOTE: &str = "source: the examined host's state alone; the symptom text did not contribute to it"; +/// The `rocm diagnose --json` document: `report` as pretty JSON, ASCII on the +/// wire. +/// +/// `serde_json` writes non-ASCII characters raw, as UTF-8, and a Windows +/// console on the OEM code page shows those bytes as mojibake (ROCMAI-543). +/// The catalog's own messages are ASCII, but a report can carry text it did +/// not write: `provenance.symptom` is the user's paste, verbatim, and some +/// evidence lines quote paths and versions read from the examined host. That +/// text must not be edited -- `--engine` exists so the paste is not -- so it is +/// escaped instead: every non-ASCII character becomes `\uXXXX` (a surrogate +/// pair beyond the Basic Multilingual Plane), and any JSON parser reads back +/// the identical string. +/// +/// Escaping after serialization is exact: in valid JSON a non-ASCII character +/// can only occur inside a string, and every JSON escape sequence is ASCII, so +/// no character this rewrites is part of one. Pinned by +/// `every_message_the_diagnose_catalog_can_emit_is_ascii` (the wire form) and +/// `a_non_ascii_paste_round_trips_through_the_json_exactly` (the value). +/// +/// # Errors +/// +/// Whatever `serde_json` returns when `report` does not serialize. A +/// `DiagnoseReport` holds only strings, numbers, booleans and maps with string +/// keys, so none is expected, but the caller decides what a failure means +/// rather than this function panicking. +pub fn render_report_json(report: &DiagnoseReport) -> serde_json::Result { + serde_json::to_string_pretty(report).map(|pretty| escape_non_ascii(&pretty)) +} + +/// `json` with every non-ASCII character replaced by its `\uXXXX` escape, in +/// UTF-16 code units. +fn escape_non_ascii(json: &str) -> String { + use std::fmt::Write as _; + let mut out = String::with_capacity(json.len()); + let mut units = [0_u16; 2]; + for c in json.chars() { + if c.is_ascii() { + out.push(c); + } else { + for unit in c.encode_utf16(&mut units) { + let _ = write!(out, "\\u{unit:04x}"); + } + } + } + out +} + +/// The line the text report opens with when `--engine` named an engine. +/// +/// Computed from `read_here`, which `diagnose_with_engine` takes from the +/// catalog table the checks ran from, so the line cannot claim the engine was +/// taken into account on a platform where no entry read it (native Windows has +/// no vLLM entry today) or the reverse. A report read back from JSON does not +/// know, and says only which engine was named. The JSON report carries no such +/// line: `provenance` already names the engine there. +fn named_engine_line(report: &DiagnoseReport) -> Option { + let provenance = report.provenance.as_ref()?; + let engine = provenance.engine.name(); + Some(match provenance.read_here { + Some(true) => format!( + "Engine: {engine} (named with --engine; every non-blank line of the symptom was \ + read as {engine}'s)." + ), + Some(false) => format!( + "Engine: {engine} (named with --engine; no entry for {engine} exists on this OS, \ + so the flag changed nothing)." + ), + None => format!("Engine: {engine} (named with --engine)."), + }) +} + /// Render the human-facing diagnosis view (mirrors `diagnose.py`'s text output). #[must_use] pub fn render_report_text(report: &DiagnoseReport, top: usize) -> String { use std::fmt::Write as _; let mut out = String::new(); + if let Some(line) = named_engine_line(report) { + out.push_str(&line); + out.push_str("\n\n"); + } if let Some(reason) = &report.out_of_scope { out.push_str("rocm diagnose: out of scope for this platform.\n\n"); out.push_str(reason); @@ -265,6 +340,7 @@ mod tests { }, out_of_scope: None, model: None, + provenance: None, }; let out = render_report_text(&report, 5); assert!( @@ -496,6 +572,7 @@ mod tests { route_when_no_match: upstream_route("rocm-core"), out_of_scope: None, model: None, + provenance: None, }; let out = render_report_text(&report, 1); assert!( diff --git a/crates/rocm-core/src/diagnose/scoring.rs b/crates/rocm-core/src/diagnose/scoring.rs index 2473403416..f325432ac9 100644 --- a/crates/rocm-core/src/diagnose/scoring.rs +++ b/crates/rocm-core/src/diagnose/scoring.rs @@ -264,7 +264,9 @@ pub(super) const KEYWORDS_PAGE_FAULT: KeywordTable = &[ // vLLM's OOM apart from an arbitrary training script hitting the same // allocator error. `check_16_vllm_oom` therefore scores *only* the lines // carrying `VLLM_ANCHOR_PATTERN` (see `vllm_anchored_lines`), never the whole -// pasted log: an OOM on an unanchored line contributes nothing at all. +// pasted log: an OOM on an unanchored line contributes nothing at all. The one +// exception is a symptom the user says came from vLLM (`rocm diagnose --engine +// vllm`), where every non-blank line is vLLM's by their account and scores. // // Weighting splits the *allocator message* from the *exception class name*: // - The HIP allocator's own messages (`HIP out of memory`, the diff --git a/crates/rocm-core/src/diagnose/test_support.rs b/crates/rocm-core/src/diagnose/test_support.rs index cf13f8a33a..0d7aa0e430 100644 --- a/crates/rocm-core/src/diagnose/test_support.rs +++ b/crates/rocm-core/src/diagnose/test_support.rs @@ -2,11 +2,50 @@ // // SPDX-License-Identifier: MIT -//! `Examination` builders and shell stubs shared by the tests of more than one -//! `diagnose` module. +//! `Examination` builders, shell stubs and a symptom, shared by the tests of +//! more than one `diagnose` module. +use crate::diagnose::SymptomEngine; use crate::examine::{Examination, Gpu, WslFacts}; +/// Every `SymptomEngine` variant, for tests that must cover each one. The +/// exhaustive `match` in [`every_engine`] stops compiling when a variant is +/// added and not listed here. +pub(super) const ALL_ENGINES: [SymptomEngine; 1] = [SymptomEngine::Vllm]; + +/// [`ALL_ENGINES`], checked against the enum by an exhaustive `match`. +pub(super) fn every_engine() -> [SymptomEngine; 1] { + for engine in ALL_ENGINES { + match engine { + SymptomEngine::Vllm => {} + } + } + ALL_ENGINES +} + +/// An `Examination` of the given catalog platform family (`linux`, `wsl` or +/// `windows`), with nothing else set. +pub(super) fn host_of_family(family: &str) -> Examination { + match family { + "linux" => linux_base(), + "wsl" => wsl_base(), + "windows" => Examination { + os_family: "windows".to_owned(), + ..Examination::default() + }, + other => panic!("no test host for platform family {other:?}"), + } +} + +/// A paste carrying text no catalog message contains: an accented path +/// (two-byte UTF-8), a box-drawing character (three bytes, what a `rich` +/// traceback frames itself with) and an emoji beyond the Basic Multilingual +/// Plane (four bytes, a surrogate pair once escaped). Used where a test needs +/// to know the `--json` output is ASCII because it was escaped, not because the +/// input happened to be. +pub(super) const NON_ASCII_PASTE: &str = "torch.OutOfMemoryError: HIP out of memory. \ + Tried to allocate 2.00 GiB\n\u{2502} File \"/home/jos\u{e9}/caf\u{e9}/serve.py\" \u{1f525}"; + pub(super) fn linux_base() -> Examination { Examination { os_family: "linux".to_owned(), diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index 90385d208a..67f48850fa 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -105,8 +105,9 @@ pub use crypto::{ verify_rsa_pkcs1_sha256_signature, }; pub use diagnose::{ - DiagnoseReport, Diagnosis, Fix, Route, VLLM_IMPORT_CANONICAL_SYMPTOM, - VLLM_OOM_CANONICAL_SYMPTOM, diagnose as run_diagnose, + DiagnoseReport, Diagnosis, Fix, Route, SymptomEngine, SymptomProvenance, + VLLM_IMPORT_CANONICAL_SYMPTOM, VLLM_OOM_CANONICAL_SYMPTOM, diagnose as run_diagnose, + diagnose_with_engine as run_diagnose_with_engine, render_report_json as render_diagnose_json, render_report_text as render_diagnose_text, vllm_import_symptom_is_diagnosable, vllm_oom_symptom_is_diagnosable, }; diff --git a/docs/architecture.md b/docs/architecture.md index ae418380a5..d2d498bf75 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -49,7 +49,7 @@ The `rocm` binary: the clap command tree, a handler module for most commands, an | [`chat_cmd.rs`](../apps/rocm/src/chat_cmd.rs) | `rocm chat` (`run_chat_command`): hands off to the dashboard chat in an interactive terminal with no `--prompt`, otherwise prints the one-shot reply or the status screen. | relocation | | [`chat_host_facts.rs`](../apps/rocm/src/chat_host_facts.rs) | The OS, WSL and GPU facts the binary detects cheaply and renders as a plain-text block for the chat assistant's system prompt. | domain | | [`cli.rs`](../apps/rocm/src/cli.rs) | The `rocm` clap definition: `Cli`, `Command`, the nested subcommand enums and `SUPPORTED_ENGINES`, and a re-export of the value enums behind flags such as `--provider` and `--device`, which [`cli_args.rs`](../apps/rocm/src/cli_args.rs) defines. | flat | -| [`cli_args.rs`](../apps/rocm/src/cli_args.rs) | The clap value enums behind `--provider`, `--device`, `--framework` and similar arguments (`Provider`, `DevicePolicyArg`, `FrameworkArg`, `InstallFormat`, `WatcherModeArg`, `TelemetryModeArg`, `SandboxToolArg`, `PermissionsModeArg`) and the conversions from a value enum to its domain type. [`cli.rs`](../apps/rocm/src/cli.rs) re-exports every one, so other modules keep reaching them as `crate::cli::Provider`; moved so `cli.rs`, which every new subcommand grows, stays under the module size limit. | relocation | +| [`cli_args.rs`](../apps/rocm/src/cli_args.rs) | The clap value enums behind `--provider`, `--device`, `--framework` and similar arguments (`Provider`, `DevicePolicyArg`, `FrameworkArg`, `InstallFormat`, `WatcherModeArg`, `TelemetryModeArg`, `SandboxToolArg`, `PermissionsModeArg`, `DiagnoseEngineArg`) and the conversions from a value enum to its domain type. [`cli.rs`](../apps/rocm/src/cli.rs) re-exports every one, so other modules keep reaching them as `crate::cli::Provider`; moved so `cli.rs`, which every new subcommand grows, stays under the module size limit. `DiagnoseEngineArg` (`diagnose --engine`) was added here after the move, not relocated; the class stays `relocation` because it records how the file came to be, and new value enums belong here whichever way the file arose. | relocation | | [`cli_audit.rs`](../apps/rocm/src/cli_audit.rs) | The CLI audit and lifecycle log (`record_cli_audit_event`): one audit event, mirrored as a line in `logs/cli-lifecycle.log` and in a per-action file under `logs/cli/`. | relocation | | [`cli_progress.rs`](../apps/rocm/src/cli_progress.rs) | Shared TTY-gated progress indicators on stderr: `Spinner`, advanced by the caller's loop, and `AnimatedSpinner`, which keeps animating between caller updates. | flat | | [`cli_report.rs`](../apps/rocm/src/cli_report.rs) | Shared CLI output formatters: `ActionReport` (headline plus indented details), `write_marked_row` for marked listings, and `JsonOrText`/`print_json_or_text` for `--json` commands. | flat | @@ -62,7 +62,7 @@ The `rocm` binary: the clap command tree, a handler module for most commands, an | [`dash.rs`](../apps/rocm/src/dash.rs) | `rocm dash` and the other ways into the dashboard TUI (launcher, chat, bench, setup): builds the telemetry daemon options and starts an embedded daemon when none is listening. | domain | | [`dash_seam.rs`](../apps/rocm/src/dash_seam.rs) | The binary's implementation of the dashboard's `RocmToolExecutor` seam, reusing the CLI's own command functions with plain data in and out. The seam exists because `rocm-dash-tui` (like `rocm-dash-core`) must not depend on `rocm-core`; `rocm-dash-collectors` and `rocm-dash-daemon` do depend on it. | domain | | [`deletion_properties.rs`](../apps/rocm/src/deletion_properties.rs) | Unix-only property tests for the recursive deletions behind `rocm uninstall` and `rocm storage remove-downloads`, checked against a real sandbox filesystem. | flat | -| [`diagnose_cmd.rs`](../apps/rocm/src/diagnose_cmd.rs) | `rocm diagnose` and `rocm fix` (`diagnose`, `fix`), and the `--report` step that builds a report naming this build's leading version (`prepare_cli_report`), shows it and offers to deliver it. | relocation | +| [`diagnose_cmd.rs`](../apps/rocm/src/diagnose_cmd.rs) | `rocm diagnose` and `rocm fix` (`diagnose`, `fix`), and the `--report` step that builds a report naming this build's leading version (`prepare_cli_report`), shows it and offers to deliver it. `SymptomQuery` carries the symptom text and the `--engine` it came from into `diagnose`; `DiagnoseOutput` overrides the `render_json` hook so `--json` is the ASCII rendering (`rocm_core::render_diagnose_json`); and `shared_report` builds the `--report` payload from the examination and the established entry only, so neither the paste nor its `provenance` reaches it. | relocation | | [`diagnose_model.rs`](../apps/rocm/src/diagnose_model.rs) | The binary's side of `rocm diagnose --model`'s GGUF context estimate: the `--kv-cache-type` and model flags, the `--report` note, and the device-visibility-mask caveat. | domain | | [`driver_install.rs`](../apps/rocm/src/driver_install.rs) | `rocm install driver` (`install_driver`, `reconcile_driver_install`): runs an approved plan's commands and persists the driver install state. It re-exports `host_install_driver_only_reports` from [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs) for its callers, [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) and [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs). | relocation | | [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs) | The `rocm install driver` plan: its model, its printed text, and the per-distro builders (apt, dnf, zypper) from the host's examination and os-release. | domain | @@ -276,12 +276,12 @@ The library every first-party app and engine crate depends on. It holds host exa | [`config_proptests.rs`](../crates/rocm-core/src/config_proptests.rs) | Property tests for the persisted `config.json` (`RocmCliConfig`) and its one-shot migration from the legacy rocm-dash `config.toml`: save and load round trips, and a save that never writes a file load rejects. Each property counts how often the generator reached the edge classes (absent optionals, empty and non-ASCII strings, non-finite floats) and fails a run that never did. Declared from [`config.rs`](../crates/rocm-core/src/config.rs). | flat | | [`confirm.rs`](../crates/rocm-core/src/confirm.rs) | The shared `[y/N]` prompt (`print_prompt`, `read_answer`), `interactive_terminal`, and the `require_yes_or_interactive` guard for `--yes`-gated commands. | domain | | [`crypto.rs`](../crates/rocm-core/src/crypto.rs) | Generic RSASSA-PKCS#1 v1.5 over SHA-256 sign and verify, with no external `openssl` process. | domain | -| [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) | Root of `rocm diagnose`: the public result types, the score thresholds, `diagnose` itself (with `mark_symptom_contribution`, which sets each finding's `symptom_matched` by running the catalog again with no symptom and comparing), and the vLLM symptom classifiers that run the whole catalog. | domain | -| [`diagnose/catalog.rs`](../crates/rocm-core/src/diagnose/catalog.rs) | The checker table: which check runs on which platform family and in what order, and how the family is resolved from an `Examination`. | domain | +| [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) | Root of `rocm diagnose`: the public result types (with `SymptomEngine` and `SymptomProvenance` for `--engine`), the score thresholds, `diagnose` itself (with `mark_symptom_contribution`, which sets each finding's `symptom_matched` by running the catalog again with no symptom and comparing), `diagnose_with_engine`, which also takes the engine a symptom came from, and the vLLM symptom classifiers that run the whole catalog. | domain | +| [`diagnose/catalog.rs`](../crates/rocm-core/src/diagnose/catalog.rs) | The checker table: which check runs on which platform family and in what order, which one check reads the engine a symptom came from (and `reads_engine_here`, which asks the same table whether any check on a platform reads a named engine, for the text report's `--engine` line), and how the family is resolved from an `Examination`. | domain | | [`diagnose/engine.rs`](../crates/rocm-core/src/diagnose/engine.rs) | Engine-runtime checks on Linux and WSL2: vLLM out-of-memory and import failures, a second code object manager, an undersized `/dev/shm`. | domain | | [`diagnose/host.rs`](../crates/rocm-core/src/diagnose/host.rs) | Host-configuration checks that run on more than one platform: wheel architecture, HSA override, `PATH`, wheel against system ROCm, mixed integrated and discrete GPUs, unsupported GPUs. | domain | | [`diagnose/linux.rs`](../crates/rocm-core/src/diagnose/linux.rs) | Linux-only checks (kernel and amdgpu module, device-node access, package repositories, repo-native install) and the package-manager command helpers. | domain | -| [`diagnose/report.rs`](../crates/rocm-core/src/diagnose/report.rs) | Upstream routing for a failure nothing recognised (`upstream_route`) and the human-readable report (`render_report_text`), including the per-entry note under a cause the symptom did not contribute to, whose text is the exported `SYMPTOM_DID_NOT_CONTRIBUTE_NOTE` the e2e steps read too. | domain | +| [`diagnose/report.rs`](../crates/rocm-core/src/diagnose/report.rs) | Upstream routing for a failure nothing recognised (`upstream_route`) and the human-readable report (`render_report_text`), including the per-entry note under a cause the symptom did not contribute to, whose text is the exported `SYMPTOM_DID_NOT_CONTRIBUTE_NOTE` the e2e steps read too, and the `--json` document (`render_report_json`), ASCII on the wire with non-ASCII text escaped rather than edited. | domain | | [`diagnose/scoring.rs`](../crates/rocm-core/src/diagnose/scoring.rs) | The keyword tables shared across platforms and the scoring over them, plus helpers that read an `Examination` or build a `Diagnosis`. | domain | | [`diagnose/test_support.rs`](../crates/rocm-core/src/diagnose/test_support.rs) | Test-only `Examination` builders shared by the tests of several `diagnose` modules. | flat | | [`diagnose/windows.rs`](../crates/rocm-core/src/diagnose/windows.rs) | Windows-only checks: the HIP SDK, the Adrenalin driver, the Visual C++ runtime and the display driver date. | domain | diff --git a/docs/manual-testing.md b/docs/manual-testing.md index ca491f3a26..d7a49c7b8d 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -1538,6 +1538,68 @@ On some GPU families `rocm install sdk` installs vLLM itself the `rocm engines install vllm` that follows then shows the block without the warning, and nothing more needs doing. +### 6d. Naming the engine instead of editing the error + +Take the failing line from 6a without any `vllm` in it (the +`torch.OutOfMemoryError: HIP out of memory ...` line from +`rocm services logs `), and diagnose it as is, then with the engine +named: + +```bash +rocm diagnose --symptom '' --json +rocm diagnose --symptom '' --engine vllm --json +``` + +Expected result: + +- The first reports no vLLM out-of-memory cause: the line alone could be any + PyTorch program's. +- The second reports `fix-16-vllm-oom`, with the same score and evidence as + `rocm diagnose --symptom 'vllm: ' --json`, and its `provenance` + holds `"engine": "vllm"` and your line exactly as you passed it once the + JSON is decoded. A line with non-ASCII text in it (an accented path, a + box-drawing character) prints as `\uXXXX` escapes, and every byte of the + output is ASCII; `python -c "import json,sys; + print(json.load(sys.stdin)['provenance']['symptom'])"` prints it back as + you pasted it. +- `--engine lemonade` (or any other value) is refused with a usage error that + lists `vllm`, and nothing is diagnosed. `--engine` with neither `--symptom` + nor `--symptom-file` is refused too. +- In bash, `rocm diagnose --symptom-file - --engine vllm --json <<'EOF'`, then + the line and `EOF`, gives the same result as the second command, and its + `provenance.symptom` is your line without the heredoc's final line break. + +Then ask what a report would carry, with a line that names something of yours +(a home-directory path or the host name) appended to the paste: + +```bash +rocm diagnose --engine vllm --report \ + --symptom ' at /home//serve.py on ' +``` + +Expected result: the report (or, on hardware not on the compatibility matrix, +the refusal) and the mail link under it carry no part of your paste and no +`provenance`. On bare-metal Linux with a GPU on the matrix, its `entry` is +`fix-16-vllm-oom`, which is the only thing the paste contributes (WSL is not +inspected for a report yet, and native Windows has no vLLM entry). It says +nothing has been sent. Do not add `--send` for this check unless you mean to +open a mail. + +Last, the text report, with no `--json`: + +```bash +rocm diagnose --symptom '' --engine vllm +``` + +Expected result: the first line names the engine. On Linux and WSL it reads +`Engine: vllm (named with --engine; every non-blank line of the symptom was +read as vllm's).` and the vLLM out-of-memory cause follows. The line says how +the symptom was read, not that anything matched: with an unrelated paste (a +plain Python traceback) it is the same line and no vLLM cause follows. On native +Windows it reads `Engine: vllm (named with --engine; no entry for vllm exists +on this OS, so the flag changed nothing).`, and the rest of the report is the +same as without `--engine`. + ## 7. vLLM Endpoint Key Coverage Needs a Linux or WSL host where the vLLM engine can start. Check that an diff --git a/docs/testing.md b/docs/testing.md index d5d2027da3..5435b6db64 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -4871,6 +4871,121 @@ others assert on the outcome the smoke test returns. cargo xtask e2e -- -n "serve-60" ``` +## Diagnose Engine Provenance + +`rocm diagnose --engine vllm` (ROCMAI-955) says which engine a symptom came +from, whether it was passed with `--symptom` or read with `--symptom-file`, so +a vLLM error that does not itself name vLLM is matched as vLLM's without anyone +writing `vllm: ` into the user's text. The vLLM out-of-memory +entry (`fix-16-vllm-oom`) only scores lines that name vLLM; with the engine +named, every non-blank line of the symptom counts as vLLM's, as if `vllm` were +written on each of them (a blank line is dropped, where a prefix would keep it; +it scores nothing either way). No other entry reads the engine. The JSON +diagnosis records the engine and the symptom, verbatim once decoded, under +`provenance`; without `--engine` the field is absent. The output is otherwise +unchanged, except that non-ASCII text in `--json` is now written as `\uXXXX` +escapes on every run, host-derived evidence included (see below); the skill +eval replays (`cargo xtask eval-fixtures --check`) are ASCII and stay as they +are. +`--engine` requires a symptom from either flag (a clap group over the two), and +`--engine` alone is a usage error. With `--symptom-file` the recorded symptom +is the text as read, with the one trailing line break that flag removes. + +`--engine` is accepted with `--report` and `--distro`, as plain `--symptom` is. +`--distro` prints the diagnosis on the user's own screen, `provenance` +included. The `--report` document, the one printed for mailing, is built from +the examination, the entry the diagnosis established and whether a fix was +offered, and from nothing the user typed, so neither the paste nor +`provenance` reaches it or the mail link `--send` offers. + +`rocm diagnose --json` is ASCII on the wire (ROCMAI-543): the paste in +`provenance.symptom`, and any host path an evidence line quotes, can hold +non-ASCII text, so `render_report_json` in +[report.rs](../crates/rocm-core/src/diagnose/report.rs) writes every non-ASCII +character as a `\uXXXX` escape (a surrogate pair beyond the Basic Multilingual +Plane). A JSON parser reads back the identical string. The ROCMAI-543 corpus +test runs `diagnose_with_engine` with an accented path, a box-drawing +character and an emoji, and fails if the escaping is dropped. + +The text report (not `--json`) opens with a line naming the engine. Where an +entry on the examined platform reads it, the line says every non-blank line of +the symptom was read as that engine's; that is how it was read, not a claim +that anything matched, so an unrelated or empty paste gets the same line and no +vLLM cause. Where no entry on the examined platform reads that engine, the line +says the flag +changed nothing; that is native Windows today, and a platform the catalog does +not cover. The answer comes from `reads_engine_here` in +[catalog.rs](../crates/rocm-core/src/diagnose/catalog.rs), which reads the same +`CHECKERS` table and platform family the checks run from, so the line follows +the catalog rather than a list of operating systems. The answer is not +serialized, so it adds nothing to the JSON report. + +Unit coverage — the match equals the prefixed form on Linux and WSL2, a +multi-line paste is anchored on every non-blank line, nothing else changes +(native Windows included), the diagnosis keeps the symptom byte for byte, the +shared report and its mail link carry none of it, the text line agrees with +what the flag did on Linux, WSL2, native Windows and an uncovered platform, and +the flag takes only `vllm`: + +```bash +cargo test -p rocm-core naming_vllm_ +cargo test -p rocm-core the_report_records_the_named_engine_and_the_symptom_verbatim +cargo test -p rocm-core the_text_report_says_which_engine_was_named_and_whether_it_counted +cargo test -p rocm-core a_non_ascii_paste_round_trips_through_the_json_exactly +cargo test -p rocm-core every_message_the_diagnose_catalog_can_emit_is_ascii +cargo test -p rocm-core every_engine_name_is_its_serialized_name +cargo test -p rocm --bin rocm a_failing_render_json_hook_is_an_error_and_writes_nothing +cargo test -p rocm --bin rocm diagnose_json_is_the_ascii_rendering_of_the_report +cargo test -p rocm --bin rocm a_shared_report_never_carries_the_pasted_symptom_or_its_provenance +cargo test -p rocm --bin rocm diagnose_engine_takes_the_engines_the_catalog_tells_apart +``` + +Scenarios in +[diagnose.feature](../tests/e2e-cucumber/features/diagnose.feature): +`diagnose-71` (`@requires-os:linux`, as `diagnose-36`, because the vLLM entry is +not in the native Windows catalog) diagnoses an unprefixed two-line vLLM OOM +with the engine named and checks the match and the verbatim symptom; +`diagnose-72` checks the recorded engine and symptom on every lane; +`diagnose-73` checks that an engine `rocm serve` knows but the catalog does not +tell apart (`lemonade`) is a usage error that diagnoses nothing; +`diagnose-74` (`@requires-os:linux`) simulates a Linux machine whose only GPU +is on the compatibility matrix, as `diagnose-58` does, so a real report is +prepared on every Linux lane, WSL2 and GPU-less ones included; it asks for that +`--report` with the engine named and a paste naming a user, host and token, and +checks that a report was printed and that none of them, and no `provenance`, +appears in it or in the mail link, and that nothing was sent (native Windows +cannot be simulated this way; the payload unit test above covers it there); +`diagnose-75` pipes the paste in with `--symptom-file -` and a final line +break, names vLLM, and checks the recorded engine and that the recorded +symptom is the paste without that line break; `diagnose-76` +(`@requires-os:linux`, so Linux and WSL2) asks for the text report with the +engine named and checks that it opens by saying the symptom was read as vLLM's +and that the vLLM cause is there, and was not there without the flag; +`diagnose-77` (`@requires-os:windows`) checks that the line says no entry for +vLLM exists on this OS and that the rest of the report is the one without the +flag; and `diagnose-78`, on every lane, saves a paste with an accented path, a +box-drawing character and an emoji to a file, diagnoses it with +`--symptom-file` and `--engine vllm --json`, and checks that every byte printed +is ASCII and that the decoded symptom is the paste byte for byte; and +`diagnose-79` (`@requires-os:linux`) names vLLM for a plain Python traceback +and checks that the text report still says the symptom was read as vLLM's and +reports no vLLM cause: the line says how the symptom was read, never that it +matched. (`diagnose-59` to `diagnose-70` belong to other changes; see their own +sections.) + +```bash +cargo xtask e2e -- -n 'diagnose-(7[1-689]) ' +``` + +A name filter bypasses the suite's host filter, so on native Windows select +`diagnose-72`, `diagnose-73`, `diagnose-75`, `diagnose-77` and `diagnose-78` +instead (Linux and WSL2 run `diagnose-71` to `diagnose-76`, `diagnose-78` and +`diagnose-79`): + +```bash +cargo xtask e2e -- -n 'diagnose-(7[23578]) ' +``` + ## Windows Tool Notes The TheRock SDK wheel install path should not require users to install global diff --git a/docs/vllm.md b/docs/vllm.md index 7893b5007e..04c8ea698c 100644 --- a/docs/vllm.md +++ b/docs/vllm.md @@ -438,6 +438,11 @@ rocm serve --engine vllm --gpu 1 --gpu-memory-utilization 0.5 rocm diagnose --symptom 'vllm: torch.OutOfMemoryError: HIP out of memory' ``` +The `vllm: ` in that symptom is what lets the diagnosis claim the line for +vLLM: the line on its own is what any PyTorch program prints. To diagnose your +own failing line without editing it, name the engine instead: +`rocm diagnose --symptom '' --engine vllm`. + ## Engine import failures A startup failure can also come from the engine failing to import, with no diff --git a/tests/e2e-cucumber/features/diagnose.feature b/tests/e2e-cucumber/features/diagnose.feature index 94962ec519..b2f76c3631 100644 --- a/tests/e2e-cucumber/features/diagnose.feature +++ b/tests/e2e-cucumber/features/diagnose.feature @@ -1054,3 +1054,116 @@ Feature: Diagnosing failures and listing fixes When the user previews that fix for the second GPU Then the CLI stops with exit code 4, saying it could not read the saved device setting and that nothing was changed And the fix's catalog entry is printed, but not the device setting it would save + + # ROCMAI-955. A vLLM error is often an ordinary PyTorch line that does not say + # vLLM, and the vLLM entry will not claim a line that does not, so the + # rocm-doctor skill tells an agent to write `vllm: ` in front of the user's + # paste (it moves to naming the engine in a later change). The user can now + # say where the paste came from instead, and the paste is used, and reported, + # exactly as given. @requires-os:linux for the reason given on + # diagnose-36: the vLLM entry is not in the native Windows catalog. Writing + # `vllm` into the paste still matches as before; diagnose-36 covers that form. + @id:diagnose-named-engine-matches-the-paste-unedited @requires-os:linux + Scenario: diagnose-71 - A pasted vLLM error is matched as vLLM's when the user names the engine, and not edited + Given a user who pasted a vLLM out-of-memory error that does not name vLLM + When the user asks the CLI to diagnose that symptom, naming vLLM as the engine it came from + Then the diagnosis identifies the vLLM startup OOM + And the symptom in the report is exactly what the user pasted + + # The record of where the paste came from does not depend on whether this + # machine's catalog has an entry for that engine, so it holds on every lane, + # native Windows included. + @id:diagnose-named-engine-is-recorded-in-the-report + Scenario: diagnose-72 - The report says which engine the user named and keeps the paste as given + Given a user who pasted a vLLM out-of-memory error that does not name vLLM + When the user asks the CLI to diagnose that symptom, naming vLLM as the engine it came from + Then the report says the user named vLLM as the source of the symptom + And the symptom in the report is exactly what the user pasted + + # Only the engines the catalog tells apart are accepted. An engine `rocm + # serve` knows but the catalog does not would change nothing, so it is + # refused rather than accepted and silently ignored. + @id:diagnose-unknown-engine-is-refused + Scenario: diagnose-73 - Naming an engine the diagnosis cannot tell apart is refused + Given a user who pasted a vLLM out-of-memory error that does not name vLLM + When the user asks the CLI to diagnose that symptom, naming an engine the diagnosis does not know + Then the CLI refuses the engine, names the ones it accepts, and diagnoses nothing + + # `--engine` makes the diagnosis repeat the user's paste, verbatim, and + # `--report` is built from that diagnosis. A paste can carry the user's own + # paths and host names, and the report is the document printed for mailing, + # so the paste must not reach it: the report names the cause it established + # and nothing the user typed. The machine is simulated as in diagnose-58, a + # Linux host whose only GPU is on the compatibility matrix, so a real report + # is prepared on every Linux lane (WSL2 included, GPU or not) and the check + # runs against it, not against a refusal that never quotes the paste. Native + # Windows cannot be simulated this way; the payload unit test + # `a_shared_report_never_carries_the_pasted_symptom_or_its_provenance` holds + # the same property on every platform. + @id:diagnose-named-engine-paste-stays-out-of-the-report @requires-os:linux + Scenario: diagnose-74 - A report asked for with a named engine carries nothing the user pasted + Given a simulated Linux machine whose only GPU is a released one + And a user who pasted a vLLM error that names their own machine + When the user asks the CLI what a report would carry, naming vLLM as the engine the error came from + Then the CLI either shows the whole report or says why it will not prepare one + And the CLI states that nothing has been sent + And nothing the user pasted appears in what the CLI printed + + # `--engine` needs a symptom, and `--symptom-file` supplies one as much as + # `--symptom` does, so the two are accepted together. The paste is piped in + # with the line break a shell or editor adds at the end; the symptom the + # report records is the text as `--symptom-file` reads it, with that one line + # break removed and nothing else changed. Holds on every lane: the record of + # where the paste came from does not depend on this machine's catalog. + @id:diagnose-named-engine-takes-a-piped-paste + Scenario: diagnose-75 - A vLLM error piped to the diagnosis is taken as vLLM's when the user names the engine + Given a user who pasted a vLLM out-of-memory error that does not name vLLM + When the user pipes it to the diagnosis, naming vLLM as the engine it came from + Then the report says the user named vLLM as the source of the symptom + And the symptom in the report is exactly what the user pasted + + # The text report says which engine `--engine` named, and whether any entry on + # this machine reads it. Linux and WSL2 have the vLLM out-of-memory entry, so + # the line says the symptom was read as vLLM's, and the step checks that + # the vLLM entry is in the report. Native Windows has no vLLM entry, so the + # line says the flag changed nothing (diagnose-77), and that step checks that + # the report is otherwise the one the same paste gets without the flag. The + # line is derived from the catalog table the checks run from, not from a list + # of operating systems, so each half runs on the lanes its claim is about. + @id:diagnose-named-engine-is-stated-in-the-text-report @requires-os:linux + Scenario: diagnose-76 - The text report names the engine and says the symptom was read as that engine's + Given a user who pasted a vLLM out-of-memory error that does not name vLLM + When the user asks the CLI to diagnose that symptom as text, naming vLLM as the engine it came from + Then the report says the symptom was read as vLLM's, and the vLLM out-of-memory cause is reported + + @id:diagnose-named-engine-without-an-entry-says-it-changed-nothing @requires-os:windows + Scenario: diagnose-77 - On a machine with no entry for the named engine, the text report says the flag changed nothing + Given a user who pasted a vLLM out-of-memory error that does not name vLLM + When the user asks the CLI to diagnose that symptom as text, naming vLLM as the engine it came from + Then the report says no entry for vLLM exists on this OS, and the diagnosis is the one without the flag + + # `provenance.symptom` repeats the paste verbatim, and a paste can hold text + # no catalog message does: an accented path, the box-drawing characters a + # `rich` traceback frames itself with, an emoji. On a Windows console using + # the OEM code page raw UTF-8 shows as mojibake, so `--json` is ASCII on the + # wire: each non-ASCII character is written as a `\u` escape, which every JSON + # parser decodes back to the identical text. Holds on every lane. The paste + # goes in through a file because a shell would get to it first otherwise. + @id:diagnose-named-engine-json-is-ascii-and-keeps-the-paste + Scenario: diagnose-78 - A pasted error with non-ASCII text is printed as ASCII JSON and reads back unchanged + Given a user whose pasted vLLM error contains non-ASCII text, saved to a file + When the user asks the CLI to diagnose that file in machine-readable form, naming vLLM as the engine + Then every byte the CLI printed is ASCII + And the symptom in the report reads back exactly as the file held it + + # The line in diagnose-76 says how the symptom was read, not that it matched. + # With an error that has nothing to do with vLLM (a plain Python traceback) + # the same line appears, and the report holds no vLLM cause: naming the engine + # never makes a paste match on its own. Linux and WSL2, where the vLLM entry + # exists and so reads the engine; native Windows says the flag changed nothing + # instead (diagnose-77). + @id:diagnose-named-engine-claims-no-match-for-an-unrelated-paste @requires-os:linux + Scenario: diagnose-79 - Naming vLLM for an error that is not vLLM's says how it was read and reports no vLLM cause + Given a user who pasted a Python error that has nothing to do with vLLM + When the user asks the CLI to diagnose that symptom as text, naming vLLM as the engine it came from + Then the report says the symptom was read as vLLM's, and no vLLM cause is reported diff --git a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs index c7fac668d6..ac875466b2 100644 --- a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs @@ -331,6 +331,126 @@ async fn user_pasted_a_capture_with_a_foreign_oom(world: &mut E2eWorld) { ); } +/// A vLLM startup OOM as a user pastes it from the engine log: two lines, and +/// neither names vLLM, which is what made the old prefix necessary. The leading +/// line and the trailing spaces are there so "exactly as pasted" has something +/// to lose: a paste trimmed, split or re-joined would no longer compare equal. +const UNNAMED_VLLM_OOM_PASTE: &str = "INFO Loading model weights took 14.9 GiB\n\ + torch.OutOfMemoryError: HIP out of memory. Tried to allocate 2.00 GiB "; + +#[given("a user who pasted a vLLM out-of-memory error that does not name vLLM")] +async fn user_pasted_unnamed_vllm_oom(world: &mut E2eWorld) { + world.model_name = Some(UNNAMED_VLLM_OOM_PASTE.to_string()); +} + +/// The parts of [`PASTE_NAMING_THE_MACHINE`] that belong to the user's own +/// machine. None of them is a catalog word, so none can appear in a report +/// except by being copied from the paste. +const PASTED_MACHINE_DETAILS: [&str; 3] = ["e2e-paste-user", "e2e-paste-host", "e2e-paste-token"]; + +/// A vLLM startup OOM as pasted from a log that also names the user's home +/// directory, host and a token: the shape of paste that must not reach a +/// report meant for sharing. +const PASTE_NAMING_THE_MACHINE: &str = "torch.OutOfMemoryError: HIP out of memory. \ + Tried to allocate 2.00 GiB\n\ + File \"/home/e2e-paste-user/serve.py\" on e2e-paste-host, HF_TOKEN=e2e-paste-token"; + +/// A vLLM out-of-memory paste with non-ASCII text in it: an accented path +/// (two UTF-8 bytes), a box-drawing character (three) and an emoji beyond the +/// Basic Multilingual Plane (four, a surrogate pair once escaped). +const NON_ASCII_VLLM_PASTE: &str = "torch.OutOfMemoryError: HIP out of memory. \ + Tried to allocate 2.00 GiB\n\u{2502} File \"/home/jos\u{e9}/caf\u{e9}/serve.py\" \u{1f525}"; + +/// Where [`NON_ASCII_VLLM_PASTE`] is saved, inside the scenario's own root. +fn non_ascii_paste_file(world: &E2eWorld) -> std::path::PathBuf { + world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .join("engine-paste.txt") +} + +/// Saved with the final line break an editor adds, which `--symptom-file` +/// removes, so the symptom the report records is the paste itself. +#[given("a user whose pasted vLLM error contains non-ASCII text, saved to a file")] +async fn user_saved_non_ascii_vllm_paste(world: &mut E2eWorld) { + std::fs::write( + non_ascii_paste_file(world), + format!("{NON_ASCII_VLLM_PASTE}\n"), + ) + .expect("failed to save the paste"); + world.model_name = Some(NON_ASCII_VLLM_PASTE.to_string()); +} + +#[when( + "the user asks the CLI to diagnose that file in machine-readable form, naming vLLM as the engine" +)] +async fn user_diagnoses_non_ascii_file_naming_vllm(world: &mut E2eWorld) { + let path = non_ascii_paste_file(world).display().to_string(); + let (stdout, stderr, rc) = crate::run_rocm( + world, + &[ + "diagnose", + "--symptom-file", + &path, + "--engine", + "vllm", + "--json", + ], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("every byte the CLI printed is ASCII")] +async fn assert_stdout_is_ascii(world: &mut E2eWorld) { + let stdout = world.cli_output.clone().expect("no diagnose output"); + assert_eq!( + world.cli_rc, + Some(0), + "diagnose should exit 0:\n{stdout}\n{:?}", + world.cli_stderr + ); + let stray: Vec = stdout.chars().filter(|c| !c.is_ascii()).collect(); + assert!( + stray.is_empty(), + "--json must be ASCII on the wire, so a Windows console on the OEM code page shows \ + no mojibake; found {stray:?} in:\n{stdout}" + ); +} + +#[then("the symptom in the report reads back exactly as the file held it")] +async fn assert_symptom_reads_back_exactly(world: &mut E2eWorld) { + let pasted = world.model_name.clone().expect("no symptom set"); + let (report, output) = parsed_diagnosis(world); + assert_eq!( + report + .pointer("/provenance/symptom") + .and_then(serde_json::Value::as_str) + .map(str::as_bytes), + Some(pasted.as_bytes()), + "decoded, the recorded symptom must be the paste byte for byte:\n{output}" + ); +} + +/// A plain Python failure with no out-of-memory text in it and nothing to do +/// with vLLM. +#[given("a user who pasted a Python error that has nothing to do with vLLM")] +async fn user_pasted_unrelated_python_error(world: &mut E2eWorld) { + world.model_name = Some( + "Traceback (most recent call last):\n File \"train.py\", line 3, in \n\ + ZeroDivisionError: division by zero" + .to_string(), + ); +} + +#[given("a user who pasted a vLLM error that names their own machine")] +async fn user_pasted_vllm_error_naming_their_machine(world: &mut E2eWorld) { + world.model_name = Some(PASTE_NAMING_THE_MACHINE.to_string()); +} + #[given("a user who hit the vLLM engine-startup import failure")] async fn user_hit_engine_import_failure(world: &mut E2eWorld) { world.model_name = Some(ENGINE_IMPORT_SYMPTOM.to_string()); @@ -742,6 +862,164 @@ async fn user_diagnoses_json(world: &mut E2eWorld) { world.cli_rc = Some(rc); } +/// Diagnose the scenario's symptom, naming `engine` as where it came from. +/// Keeps stdout, stderr and the exit code apart: a refusal is judged on all +/// three. +fn diagnose_naming_engine(world: &mut E2eWorld, engine: &str) { + let symptom = world.model_name.clone().expect("no symptom set"); + let (stdout, stderr, rc) = crate::run_rocm( + world, + &[ + "diagnose", + "--symptom", + &symptom, + "--engine", + engine, + "--json", + ], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[when("the user asks the CLI to diagnose that symptom, naming vLLM as the engine it came from")] +async fn user_diagnoses_naming_vllm(world: &mut E2eWorld) { + diagnose_naming_engine(world, "vllm"); +} + +/// Text mode, twice: with `--engine vllm`, and without it as the baseline the +/// "changed nothing" claim is checked against. `cli_outputs` holds the two +/// stdouts in that order. +#[when( + "the user asks the CLI to diagnose that symptom as text, naming vLLM as the engine it came from" +)] +async fn user_diagnoses_as_text_naming_vllm(world: &mut E2eWorld) { + let symptom = world.model_name.clone().expect("no symptom set"); + let (named, named_stderr, rc) = crate::run_rocm( + world, + &["diagnose", "--symptom", &symptom, "--engine", "vllm"], + ); + let (unnamed, _, unnamed_rc) = crate::run_rocm(world, &["diagnose", "--symptom", &symptom]); + assert_eq!(unnamed_rc, 0, "the baseline diagnosis must run:\n{unnamed}"); + world.cli_outputs = Some(vec![named.clone(), unnamed]); + world.cli_output = Some(named); + world.cli_stderr = Some(named_stderr); + world.cli_rc = Some(rc); +} + +/// The text report with `--engine` and the one without it, after checking the +/// named run exited 0. +fn named_and_unnamed_text(world: &E2eWorld) -> (String, String) { + let outputs = world.cli_outputs.clone().expect("no diagnose outputs"); + assert_eq!( + world.cli_rc, + Some(0), + "diagnose should exit 0 (it is a query):\n{}", + outputs[0] + ); + (outputs[0].clone(), outputs[1].clone()) +} + +#[then( + "the report says the symptom was read as vLLM's, and the vLLM out-of-memory cause is reported" +)] +async fn assert_text_says_matched_as_vllm(world: &mut E2eWorld) { + let (named, unnamed) = named_and_unnamed_text(world); + assert_eq!( + named.lines().next(), + Some( + "Engine: vllm (named with --engine; every non-blank line of the symptom was \ + read as vllm's)." + ), + "the text report must open by naming the engine it took the symptom as:\n{named}" + ); + assert!( + named.contains("fix-16-vllm-oom"), + "the line says the symptom was read as vLLM's, and this paste is a vLLM OOM, so the vLLM cause must be there:\n{named}" + ); + assert!( + !unnamed.contains("fix-16-vllm-oom"), + "without the flag the unnamed paste must not reach the vLLM cause, or the flag was not \ + what made the difference:\n{unnamed}" + ); +} + +/// The line diagnose-79 expects is the one diagnose-76 expects: it says how +/// the symptom was read. What differs is the outcome beside it. +#[then("the report says the symptom was read as vLLM's, and no vLLM cause is reported")] +async fn assert_text_says_read_as_vllm_without_a_match(world: &mut E2eWorld) { + let (named, _) = named_and_unnamed_text(world); + assert_eq!( + named.lines().next(), + Some( + "Engine: vllm (named with --engine; every non-blank line of the symptom was \ + read as vllm's)." + ), + "the text report must open by saying how it read the symptom:\n{named}" + ); + assert!( + !named.contains("fix-16-vllm-oom"), + "nothing in this paste is vLLM's, so naming the engine must not produce a vLLM cause:\n{named}" + ); +} + +#[then( + "the report says no entry for vLLM exists on this OS, and the diagnosis is the one without the flag" +)] +async fn assert_text_says_flag_changed_nothing(world: &mut E2eWorld) { + let (named, unnamed) = named_and_unnamed_text(world); + let first = named.lines().next().unwrap_or_default(); + assert_eq!( + first, + "Engine: vllm (named with --engine; no entry for vllm exists on this OS, so the flag \ + changed nothing).", + "the text report must say the named engine is read by nothing here:\n{named}" + ); + let rest = named + .strip_prefix(first) + .and_then(|rest| rest.strip_prefix("\n\n")) + .unwrap_or_else(|| panic!("the engine line must be followed by a blank line:\n{named}")); + assert_eq!( + rest, unnamed, + "the line says the flag changed nothing, so the rest of the report must be the one \ + without it" + ); +} + +/// Piped on standard input with the `\n` a shell or editor adds at the end, +/// which `--symptom-file` removes; the paste itself goes in unchanged, so the +/// recorded symptom has to equal it exactly. +#[when("the user pipes it to the diagnosis, naming vLLM as the engine it came from")] +async fn user_pipes_symptom_naming_vllm(world: &mut E2eWorld) { + let symptom = world.model_name.clone().expect("no symptom set"); + let (stdout, stderr, rc) = crate::run_rocm_with_stdin( + world, + &[ + "diagnose", + "--symptom-file", + "-", + "--engine", + "vllm", + "--json", + ], + &format!("{symptom}\n"), + &[], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +/// `lemonade` because it is an engine `rocm serve --engine` accepts: the +/// refusal is about what the diagnosis tells apart, not about a typo. +#[when( + "the user asks the CLI to diagnose that symptom, naming an engine the diagnosis does not know" +)] +async fn user_diagnoses_naming_unknown_engine(world: &mut E2eWorld) { + diagnose_naming_engine(world, "lemonade"); +} + #[when("the user asks the CLI which fixes it offers")] async fn user_lists_fixes(world: &mut E2eWorld) { let (stdout, _, rc) = crate::run_rocm(world, &["fix"]); @@ -1059,6 +1337,56 @@ async fn assert_no_vllm_oom_reported(world: &mut E2eWorld) { ); } +#[then("the symptom in the report is exactly what the user pasted")] +async fn assert_report_keeps_the_paste_verbatim(world: &mut E2eWorld) { + let pasted = world.model_name.clone().expect("no symptom set"); + let (report, output) = parsed_diagnosis(world); + assert_eq!( + report + .pointer("/provenance/symptom") + .and_then(serde_json::Value::as_str), + Some(pasted.as_str()), + "the report must carry the paste byte for byte, with nothing added:\n{output}" + ); +} + +#[then("the report says the user named vLLM as the source of the symptom")] +async fn assert_report_names_vllm_as_source(world: &mut E2eWorld) { + assert_eq!( + world.cli_rc, + Some(0), + "diagnose should exit 0 (it is a query)" + ); + let (report, output) = parsed_diagnosis(world); + assert_eq!( + report + .pointer("/provenance/engine") + .and_then(serde_json::Value::as_str), + Some("vllm"), + "the report must record the engine the user named:\n{output}" + ); +} + +#[then("the CLI refuses the engine, names the ones it accepts, and diagnoses nothing")] +async fn assert_unknown_engine_refused(world: &mut E2eWorld) { + let stderr = world.cli_stderr.clone().unwrap_or_default(); + let stdout = world.cli_output.clone().unwrap_or_default(); + // clap's usage-error exit code. + assert_eq!( + world.cli_rc, + Some(2), + "an unknown engine is a usage error:\nstdout:\n{stdout}\nstderr:\n{stderr}" + ); + assert!( + stderr.contains("lemonade") && stderr.contains("vllm"), + "the refusal must name the rejected value and the accepted one:\n{stderr}" + ); + assert!( + stdout.trim().is_empty(), + "nothing may be diagnosed when the engine is refused:\n{stdout}" + ); +} + #[then("the diagnosis identifies the vLLM startup OOM")] async fn assert_diagnosis_identifies_vllm_oom(world: &mut E2eWorld) { assert_eq!(world.cli_rc, Some(0), "diagnose should exit 0"); @@ -2750,6 +3078,55 @@ async fn user_asks_what_a_report_would_carry_json(world: &mut E2eWorld) { world.cli_rc = Some(rc); } +/// Text mode, not `--json`: it prints the report and the mail link that +/// `--send` would offer, so both surfaces a paste could leak into are read. +/// No `--send`, so nothing is opened or sent. +#[when( + "the user asks the CLI what a report would carry, naming vLLM as the engine the error came from" +)] +async fn user_asks_for_a_report_naming_vllm(world: &mut E2eWorld) { + let symptom = world.model_name.clone().expect("no symptom set"); + // With the scenario's environment, so a simulated machine planted by a + // Given step is the one examined. + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env( + world, + &[ + "diagnose", + "--symptom", + &symptom, + "--engine", + "vllm", + "--report", + ], + ); + world.cli_output = Some(format!("{stdout}\n{stderr}")); + world.cli_rc = Some(rc); +} + +#[then("nothing the user pasted appears in what the CLI printed")] +async fn assert_paste_absent_from_report(world: &mut E2eWorld) { + let out = world.cli_output.clone().expect("no CLI output"); + // A refusal is fixed text that never quotes the paste, so sweeping one + // proves nothing. The scenario plants a machine whose report is prepared; + // require that report (`cli_version` is a field only a `Report` carries, + // see `answer_names_nothing_identifying`) before sweeping it. + assert!( + out.contains("a report would carry") && out.contains("cli_version"), + "no report was prepared, so there is nothing to check the paste against; the \ + simulated released GPU should have produced one:\n{out}" + ); + for detail in PASTED_MACHINE_DETAILS { + assert!( + !out.contains(detail), + "{detail:?} from the user's paste reached what a report would carry:\n{out}" + ); + } + assert!( + !out.contains("provenance"), + "the diagnosis's provenance reached what a report would carry:\n{out}" + ); +} + #[then("the CLI either shows the whole report or says why it will not prepare one")] async fn report_is_shown_or_refused(world: &mut E2eWorld) { let out = world.cli_output.clone().expect("no CLI output"); diff --git a/xtask/src/eval_fixtures.rs b/xtask/src/eval_fixtures.rs index dbc335d0ef..ef9645d894 100644 --- a/xtask/src/eval_fixtures.rs +++ b/xtask/src/eval_fixtures.rs @@ -261,8 +261,9 @@ fn replies_for(root: &Path, case: &Case, recorded: &rocm_core::Examination) -> R .host .map_or_else(|| recorded.clone(), |host| examination_for(recorded, host)); let report = rocm_core::run_diagnose(examination, &case.symptom); + // The same rendering `rocm diagnose --json` prints, ASCII on the wire. let diagnose_json = Reply::ok( - serde_json::to_string_pretty(&report).context("failed to render the diagnosis")? + "\n", + rocm_core::render_diagnose_json(&report).context("failed to render the diagnosis")? + "\n", ); let diagnose_text = Reply::ok(rocm_core::render_diagnose_text(&report, 5)); let (version, fix_json) = match case.cli {