Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -458,7 +458,7 @@ Windows. The Vulkan driver version is not shown.
### Diagnose and fix

```
rocm diagnose [--symptom TEXT | --symptom-file PATH] [--top N] [--json] [--distro [NAME] | --report [--send]]
rocm diagnose [--symptom TEXT [--engine vllm] | --symptom-file PATH [--engine vllm]] [--top N] [--json] [--distro [NAME] | --report [--send]]
rocm diagnose --model <model> [--context-size N] [--kv-cache-type f16|q8_0|q4_0] [--json]
rocm fix [<fix-id>] [--yes] [--dry-run] [--device-index N] [--json]
```
Expand Down Expand Up @@ -518,6 +518,23 @@ position.
contribute to, once the text contributed to any match in the report. That
includes one past `--top`, so all the rows shown can carry the line when the
only match the text contributed to was cut off.
- `--engine` names the engine the symptom came from (only `vllm` is
accepted; it needs `--symptom` or `--symptom-file`). An engine's own error
is often a line any program could print, such as a PyTorch out-of-memory
error, and the vLLM entry claims only lines that name vLLM. With
`--engine vllm`, every non-blank line of the symptom is taken as vLLM's, so
the error is matched without editing it. The text report opens with a line
naming the engine; where no entry on this OS reads it (native Windows has
no vLLM entry), that line says the flag changed nothing. The `--json`
diagnosis records the engine and the symptom, exactly as given once the
JSON is decoded (for `--symptom-file`, the text read, minus the one
trailing line break), under `provenance`. On the wire `--json` is ASCII:
any non-ASCII character in the paste is written as a `\uXXXX` escape, so a
Windows console shows no mojibake and any JSON parser reads the paste back
unchanged. A `--report` never carries either: it names only the cause
the diagnosis established, so a pasted log's paths or host names stay out
of what you might mail. Writing `vllm` into the symptom text still works
too.
- On Windows, `diagnose` also checks the age of the Visual C++ runtime, not
only whether it is there. The runtime in `System32` must be 14.40 or newer
(`fix-21-msvc-redist-outdated`). Programs built with a recent toolset, such
Expand Down
105 changes: 99 additions & 6 deletions apps/rocm/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@
//! keeps reaching them as `crate::Command`, `crate::Provider` and so on.

pub(crate) use crate::cli_args::{
DevicePolicyArg, FrameworkArg, InstallFormat, PermissionsModeArg, Provider, SandboxToolArg,
TelemetryModeArg, WatcherModeArg,
DevicePolicyArg, DiagnoseEngineArg, FrameworkArg, InstallFormat, PermissionsModeArg, Provider,
SandboxToolArg, TelemetryModeArg, WatcherModeArg,
};
use crate::service_records::DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS;
use crate::{bootstrap, chat, comfyui, remote, storage};
Expand All @@ -23,8 +23,13 @@ use rocm_core::DEFAULT_LOCAL_HOST;
use std::path::PathBuf;

/// User-selectable serving engines, in the order shown in `--help` and shell
/// completions. Used as the clap `value_parser` for every user-facing `--engine`
/// argument so the possible values stay in sync across the CLI surface.
/// completions. Used as the clap `value_parser` for every user-facing argument
/// that names an engine to serve, install, open a shell in or set as a
/// preference (`serve --engine`, `engines install`/`shell`, and the
/// engine-setting `config` commands), so the possible values stay in sync
/// across those commands.
/// `diagnose --engine` is narrower on purpose: it takes `DiagnoseEngineArg`,
/// only the engines the diagnosis catalog tells apart.
pub(crate) const SUPPORTED_ENGINES: [&str; 3] = ["lemonade", "vllm", "aim"];

#[derive(Parser, Debug)]
Expand Down Expand Up @@ -103,7 +108,14 @@ pub(crate) enum Command {
///
/// Each match then says whether this text contributed to it
/// (`symptom_matched` in --json).
#[arg(long, value_name = "TEXT")]
//
// `symptom_source` groups the two symptom flags for `--engine`, which
// needs a symptom from either. A plain `requires = "symptom"` also let
// `--symptom-file` through, but only because clap waives a requirement
// on an argument that conflicts with one present; the group says what
// is meant. It does not change how the two flags refuse each other:
// `--symptom-file`'s own `conflicts_with` still does that.
#[arg(long, value_name = "TEXT", group = "symptom_source")]
symptom: Option<String>,
/// Read the error text from a file instead, or from standard input
/// when PATH is `-`, so no shell quoting is needed. A file named `-`
Expand All @@ -121,8 +133,33 @@ pub(crate) enum Command {
/// text to a program as ASCII, turning any other character into `?`.
/// With nothing piped in, paste the text at the terminal and end it
/// with Ctrl-D at the start of a line (Ctrl-Z then Enter on Windows).
#[arg(long, value_name = "PATH", conflicts_with = "symptom")]
#[arg(
long,
value_name = "PATH",
conflicts_with = "symptom",
group = "symptom_source"
)]
symptom_file: Option<PathBuf>,
/// The engine the symptom came from (`--symptom` or `--symptom-file`),
/// so it can be matched as that engine's error without editing it.
///
/// Some failure modes belong to one engine, and the error it prints
/// (a PyTorch out-of-memory error, for one) is the same text any other
/// program could print. Without this, such an entry matches only a
/// line that names the engine itself. With it, every non-blank line of
/// the symptom is taken as that engine's, the symptom is used exactly
/// as given (from a file, as read: one trailing line break removed),
/// and the JSON diagnosis records both under `provenance` (exactly,
/// once the JSON is decoded: on the wire `--json` is ASCII, with any
/// non-ASCII character written as a `\uXXXX` escape). The text
/// report opens with a line naming the engine, and says the flag
/// changed nothing where no entry on this OS reads that engine (native
/// Windows has no vLLM entry).
/// A `--report` never carries either: it names only the cause the
/// diagnosis established. Only engines the catalog tells apart are
/// accepted; writing `vllm` in the symptom text still works as before.
#[arg(long, value_enum, value_name = "ENGINE", requires = "symptom_source")]
engine: Option<DiagnoseEngineArg>,
/// Show at most this many matches (default 5).
#[arg(long, default_value_t = 5)]
top: usize,
Expand Down Expand Up @@ -2021,6 +2058,62 @@ mod tests {
assert_eq!(error.kind(), clap::error::ErrorKind::ArgumentConflict);
}

// `--engine` names where a symptom came from (ROCMAI-955). It takes only
// the engines the catalog tells apart, so an engine `serve` knows but the
// catalog does not is a usage error rather than a value silently ignored,
// and without a symptom it would describe nothing. The symptom may come
// from either `--symptom` or `--symptom-file`.
#[test]
fn diagnose_engine_takes_the_engines_the_catalog_tells_apart() {
let cli = parse_diagnose(&["--symptom", "torch.OutOfMemoryError", "--engine", "vllm"])
.expect("vllm parses");
match cli.command {
Some(Command::Diagnose {
engine, symptom, ..
}) => {
assert_eq!(
engine.map(rocm_core::SymptomEngine::from),
Some(rocm_core::SymptomEngine::Vllm)
);
assert_eq!(symptom.as_deref(), Some("torch.OutOfMemoryError"));
}
other => panic!("expected diagnose, got {other:?}"),
}
for unknown in ["lemonade", "aim", "llama.cpp", "VLLM-ish"] {
let error = parse_diagnose(&["--symptom", "x", "--engine", unknown])
.expect_err("an engine the catalog does not tell apart must not parse");
assert_eq!(
error.kind(),
clap::error::ErrorKind::InvalidValue,
"{unknown}"
);
assert!(error.to_string().contains("vllm"), "{error}");
}
for path in ["-", "error.txt"] {
match parse_diagnose(&["--symptom-file", path, "--engine", "vllm"]) {
Ok(Cli {
command:
Some(Command::Diagnose {
engine,
symptom_file,
..
}),
..
}) => {
assert_eq!(engine, Some(DiagnoseEngineArg::Vllm));
assert_eq!(symptom_file, Some(PathBuf::from(path)));
}
other => panic!("--symptom-file {path} --engine vllm must parse, got {other:?}"),
}
}
let error = parse_diagnose(&["--engine", "vllm"])
.expect_err("an engine with no symptom describes nothing");
assert_eq!(
error.kind(),
clap::error::ErrorKind::MissingRequiredArgument
);
}

// The context options describe the model being asked about, so without
// `--model` they would be accepted and silently do nothing.
#[test]
Expand Down
30 changes: 26 additions & 4 deletions apps/rocm/src/cli_args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,16 @@
//
// SPDX-License-Identifier: MIT

//! The clap value enums behind the `--framework`, `--format`, `--provider`,
//! `--mode`, `--device`, `--telemetry` and `--permissions` style arguments, and
//! the conversions from them to the library types they stand for.
//! The clap value enums behind the `--framework`, `diagnose --engine`,
//! `--format`, `--provider`, `--mode`, `--device`, `--telemetry` and
//! `--permissions` style arguments, and the conversions from them to the
//! library types they stand for.
//!
//! Mechanically relocated from `cli.rs` with no behavior change: attributes,
//! help text, aliases and declaration order are unchanged. `cli.rs` re-exports
//! each type, so every module keeps reaching them as `crate::cli::Provider` and
//! so on.
//! so on. `DiagnoseEngineArg` (`diagnose --engine`) is the exception: it was
//! added here after the move rather than relocated.

use clap::ValueEnum;
use rocm_core::{
Expand Down Expand Up @@ -48,6 +50,26 @@ impl From<FrameworkArg> for rocm_core::FrameworkProbe {
}
}

/// Which engine a `rocm diagnose --symptom` came from.
///
/// The CLI-facing mirror of [`rocm_core::diagnose::SymptomEngine`], for the
/// same reason as [`FrameworkArg`]. It lists only the engines the diagnosis
/// catalog tells apart, under the names `rocm serve --engine` uses; the others
/// are rejected as a usage error rather than accepted and silently ignored.
#[derive(Clone, Copy, Debug, Eq, PartialEq, ValueEnum)]
pub(crate) enum DiagnoseEngineArg {
/// The vLLM serving engine (`rocm serve --engine vllm`).
Vllm,
}

impl From<DiagnoseEngineArg> for rocm_core::diagnose::SymptomEngine {
fn from(value: DiagnoseEngineArg) -> Self {
match value {
DiagnoseEngineArg::Vllm => Self::Vllm,
}
}
}

#[derive(Clone, Copy, Debug, Eq, PartialEq, ValueEnum)]
pub(crate) enum InstallFormat {
/// Python wheel packages (recommended; smaller, pip-installable).
Expand Down
44 changes: 43 additions & 1 deletion apps/rocm/src/cli_report.rs
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,14 @@ pub(crate) fn write_marked_row(
/// (side effects, audit events) with the printing stays hand-written.
pub(crate) trait JsonOrText: Serialize {
fn render_text(&self) -> String;

/// The JSON form, without the trailing newline. Pretty `serde_json` by
/// default; a command whose output can carry text it did not write
/// overrides this to keep the wire form ASCII (`rocm diagnose`, see
/// `rocm_core::render_diagnose_json`).
fn render_json(&self) -> serde_json::Result<String> {
serde_json::to_string_pretty(self)
}
}

/// Writes `value` to `writer` as pretty JSON followed by one newline when
Expand All @@ -98,7 +106,8 @@ fn write_json_or_text(
value: &impl JsonOrText,
) -> Result<()> {
let rendered = if json {
let mut document = serde_json::to_string_pretty(value)
let mut document = value
.render_json()
.context("failed to serialize the command output as JSON")?;
document.push('\n');
document
Expand Down Expand Up @@ -234,6 +243,39 @@ mod tests {
);
}

/// Serializes fine through serde but overrides `render_json` with one that
/// fails, the way `rocm diagnose`'s ASCII rendering can: the hook's own
/// error is what has to reach the caller, and nothing may be written.
struct RenderJsonFails;

impl Serialize for RenderJsonFails {
fn serialize<S: serde::Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
serializer.serialize_str("serializes fine")
}
}

impl JsonOrText for RenderJsonFails {
fn render_text(&self) -> String {
panic!("JSON mode must not render the text form");
}

fn render_json(&self) -> serde_json::Result<String> {
Err(serde::ser::Error::custom("the render_json hook failed"))
}
}

#[test]
fn a_failing_render_json_hook_is_an_error_and_writes_nothing() {
let mut output = Vec::new();
let error = write_json_or_text(&mut output, true, &RenderJsonFails)
.expect_err("the hook's failure must surface");
assert!(output.is_empty(), "nothing is written on failure");
assert!(
format!("{error:#}").contains("the render_json hook failed"),
"unexpected error: {error:#}"
);
}

struct BrokenPipe;

impl std::io::Write for BrokenPipe {
Expand Down
Loading
Loading