Skip to content

fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it - #1029

Draft
siloteemu wants to merge 2 commits into
mainfrom
docs/rocm-doctor-skill-review
Draft

siloteemu wants to merge 2 commits into
mainfrom
docs/rocm-doctor-skill-review

Conversation

@siloteemu

@siloteemu siloteemu commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Why

The rocm-doctor skill had drifted from what the rocm CLI actually does, and an agent following it literally could go wrong in ways found by reading it against the CLI source and running the read-only commands:

  • rocm fix <id> was assumed to prompt. In an agent's shell with no terminal it refuses and exits 5, which the skill's text invited reporting as "you declined" when nobody had been asked.
  • A pasted symptom went into a double-quoted --symptom argument, so $(...) or a backtick in the paste ran as a command, and a paste starting with - (for example -1073741819) was read by clap as a flag.
  • Step 2 acted on the top match only. On a container host, fix-10 and fix-19 (host-state matches) outrank fix-17-torch-dlpack for a libtorch_cuda.so paste, so the cause the user pasted was dropped and an unrelated host finding was presented as the remedy.
  • reference.md carried a hand-kept exit-code table that was wrong in several places and that no test parsed, and SKILL.md and reference.md restated the same rules with drifting copies.
  • The installer was offered with an incomplete account of what it changes, and the skill said nothing about CPU fallback although AGENTS.md §7 requires it to stay consistent with the assistant skill.

What changes

  • The skill reads what the CLI publishes instead of restating it: exit outcomes come by name from the manifest's exit_codes, thresholds by name (min_score_for_match, high_confidence_threshold), and the apply decision from fix.class. The hand-kept exit-code table is gone.
  • Step 2 is one ordered decision list (out of scope, no match and where to route it, a match that does not explain the user's described problem, otherwise the best match that does). A host-state match that does not explain the complaint is offered as a separate finding, never as the remedy.
  • Applying a fix is dry run, then the user's explicit OK, then --yes; each exit is read by manifest name together with the output, and a declined exit without --yes means nobody was asked.
  • --symptom='<text>' is single-quoted with '\'' for an apostrophe in POSIX shells; in any other shell the user is handed the command to run.
  • The installer disclosure, the full-path re-check after an install, native-Windows vLLM, the rocm serve --engine lemonade route, the LM Studio tracker, and a no-CPU-fallback statement are rewritten or added; SKILL.md and reference.md each own their rules once.
  • Evals (same change, skills/rocm-doctor/evals/): new replayed cases for the apply sequence, the "match does not explain the paste" and "host-only match" decisions, Ollama / LM Studio / rocm serve routing, shell metacharacters in a paste, the no-fix control, an installer-disclosure expectation, and a clean-machine should-not-trigger case. The apply answers are generated by cargo xtask eval-fixtures from the catalog and the replayed diagnosis, so --check covers them, and the stub never answers --yes.
  • Comments and assertion messages in the e2e suite and rocm-core that cited the removed exit-code table or described the old text were corrected; no CLI behaviour changes.

Verification

AGENTS.md §3: this PR changes the skill's instructions to an agent, not CLI output, exit codes, files or engine selection, so the behavioural coverage is the evals above. The existing scenarios that pin the skill's documents to the CLI still pass: skill-catalog-ids-match-cli, skill-auto-fix-set-matches-cli, skill-os-scope-matches-cli, skill-diagnosis-matches-what-the-skill-documents, skill-examine-verdicts-are-documented, skill-escalation-target-matches-cli and skill-required-contract-is-the-one-the-cli-publishes.

Run locally on this head: cargo fmt --all --check; python3 -B -m unittest on evals/test_hooks.py (14 pass, 1 Windows-only skipped); cargo test -p xtask eval_fixtures (24 pass); cargo test -p e2e-cucumber --test skill_reference (3 pass); the skill scenarios through cargo xtask e2e against a prebuilt rocm (7 scenarios, 25 steps pass). Each new stub and replay test was broken on purpose (an apply branch widened to accept --yes, a replayed flag dropped, a premise changed) and watched fail before being restored. The change was reviewed independently against its base until no blocking findings remained.

Not verified:

  • The new or changed evals and the rewritten description have not been run through the model-graded behavioral and routing grading described in docs/skill-quality.md; those make paid model calls and are not part of this repository's CI.
  • The license-headers and link-check hooks are not installed on the machine this was prepared on, so CI is the check for them.
  • Needs a Windows or WSL2 host, or a Linux host with password-protected sudo: a --symptom quoting form for PowerShell and cmd.exe (the skill hands the user the command there); the irm | iex install updating PATH in the same session and the Windows full-path form & "$env:USERPROFILE\.local\bin\rocm.exe" --version; rocm diagnose --distro from a Windows host followed by rocm fix acting on the host (the skill tells the user to run fixes inside the distribution); whether sudo prompts or fails fast when rocm fix fix-4-render-group --yes runs from a shell with no terminal.

For maintainers to decide

  • A failure of rocm serve itself with no catalog match, any engine, now goes to this project's issues rather than to the tracker for the host's detected framework; CONTRIBUTING.md names this project's issues for bugs here, and route_when_no_match follows the host framework, which says nothing about whose bug it is.
  • Step 2 offers a host-state match that does not explain the user's problem as "worth doing anyway", with consent, including auto fixes such as fix-4.
  • Relaying the CLI's --n-gpu-layers suggestion (some layers in system RAM) is allowed alongside the no-CPU-fallback statement; confirm that is within the policy.
  • The LM Studio route points at lmstudio-ai/lmstudio-bug-tracker, a public repository; it was not confirmed as LM Studio's documented place for bug reports.
  • fix-8-wheel-rocm's title says "ROCm major" but the code compares X.Y; the skill uses the CLI's wording. A --symptom-file option and a per-match flag saying whether the symptom contributed would remove the quoting rules and the "does it explain the paste" judgement from the skill.

Known gaps in this change

  • The apply eval is one turn: it expects the dry run and the sudo and re-login warnings before the user's OK, while step 2 says to propose only when the user is on board, so an agent that asks first fails it.
  • The generated apply replay omits the OS scope:, Rationale:, Will run:, Note: and Verify (Linux, WSL): lines the CLI prints, which is harmless for fix-4 (the only apply case; the generator refuses other ids) but is not fully listed in docs/testing.md.
  • The installer disclosure does not name the fish completions file the Linux installer writes when fish is present.
  • A comment in tests/e2e-cucumber/tests/e2e/skill_steps.rs still says reference.md's per-host auto prose is what the agent reads before offering a fix.

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable: no CLI bug is fixed here and no xfail rows concern the skill.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. Not applicable: no subcommand or subsystem is added.
  • If this PR adds, renames or removes a production source file, docs/architecture.md has the matching entry. Not applicable: no production source file is added, renamed or removed (the new files are eval fixtures).
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state, per AGENTS.md §3. Not applicable: no CLI message changed; the only Rust edits are comments and test assertion messages.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the whole change: every file in the diff, plus the CLI code the skill drives (apps/rocm, and the diagnose, fix and examine code in rocm-core). I checked each command, flag, status and JSON field the skill names against the code, and they all exist and behave as the skill says. That includes --yes exiting 5 when there is no terminal, the fix-4 preconditions, and the installer defaults. Single-quoting with --symptom='...' closes a real command-injection path. The = form also stops a paste like -1073741819 from being read as a flag. Replacing the hand-kept exit-code table with the manifest's exit_codes is the right call.

I've requested changes for the reasons in the inline comments below. None of them is a deep problem. The one that changes what an agent does is the vLLM startup case on SKILL.md: it sends a symptom the catalog covers to an upstream tracker. The rest are test and comment accuracy. Some of the comment fixes overlap items already filed as ROCMAI-955 follow-ups. I flag them here anyway because they sit in the comment block this PR rewrites, so fixing them now costs almost nothing.

One check outside the diff: on 8610f5e, E2E tests (Strix Halo, WSL2, current) is red. It failed in the "Install Ubuntu-24.04 distro" step, before any test ran, so it looks like infrastructure. The same check is green on main's tip, though, so please rerun it. The MI300X and W7900 lanes were still queued when I looked.

What I ran on this head: test_hooks.py (14 pass), cargo nextest run -p xtask eval_fixtures (24 pass), cargo xtask eval-fixtures --check (clean), and skillscope structural on the skill (OK). I did not run the skill e2e scenarios locally (CI's mock lane is green), and not the model-graded routing or behavioral evals.

One question rather than a request: when the user names no vendor, Phase 0 asks to install the CLI before anything has shown there is an AMD GPU (rocm examine reports no-amd-gpu only afterwards). I see why: nothing can probe without the CLI, and the scope gate catches a vendor the user names. Is that ordering deliberate?

space instead, a paste that begins with `-` (an exit code such as `-1073741819`)
is read as a flag and the command fails.

If the user pasted no error text, run `rocm diagnose --json` without

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A vLLM startup failure described in words, with no error line pasted, gets routed upstream even though the catalog covers it. Step 1 says to run rocm diagnose --json without --symptom when nothing was pasted. But both vLLM checkers ignore the examination and score only the symptom text. check_16_vllm_oom (crates/rocm-core/src/diagnose/engine.rs, around line 132) returns zero(...) when no anchored vLLM line is present. So "vLLM runs out of memory at startup" gets has_match: false, and step 2 sends it to a tracker, though the description lists that symptom as a trigger.

Suggest: for a vLLM startup complaint, ask for the exact error line first (or the rocm diagnose --symptom '...' command rocm serve printed), then diagnose.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f78bbc2. You're right: both vLLM checkers score only the symptom text, so a description in words can't match either entry. The vLLM section now says that for a vLLM startup failure on Linux or WSL2 described in words, with no error line pasted, the agent asks for the exact error line (or the rocm diagnose --symptom '...' command rocm serve printed) before diagnosing. Step 1's "no error text pasted" paragraph points there instead of repeating it.

df9c981 adds an eval case for it, rocm-vllm-failure-in-words-asks-for-the-error-line. Its premise test pins that the replayed diagnosis for the description has has_match false and lists neither vLLM entry. Letting a described failure match would need a change to the matcher, which is out of scope for this PR.

beside a discrete GPU), so a healthy host still returns a non-empty
`matched` of sub-threshold entries. Gating on emptiness proposes a fix for a
machine with nothing wrong.
- `matched[]` — ranked causes, each with `id`, `title`, `score`, `evidence[]`,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing ties the exit_codes names and fix.* field names quoted here to the CLI. the_published_exit_codes_are_the_ones_apply_returns (crates/rocm-core/src/fix.rs) pins the manifest's keys, not the names this file quotes. skill-04 deliberately skips the fix sub-fields. Not a regression, since the old table wasn't parsed either, and the agent reads the map at runtime. But a small unit test would be cheap: every backticked exit_codes name in SKILL.md must be a manifest key.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f78bbc2, with two tests in tests/e2e-cucumber/tests/skill_reference.rs:

  • exit_code_names_listed_in_the_skill_are_the_manifest_failure_exits: the exit-code list in step 3 (the bullets after "by its exit_codes name") must equal the manifest's non-zero exit_codes. That's stricter than you suggested: a published exit the skill leaves out fails too.
  • fix_fields_quoted_in_the_skill_exist_in_the_diagnosis_json: every field in step 1's fix field list must be a key of a serialised rocm_core::Fix.

Both read the whitespace-collapsed text, so re-wrapping doesn't matter, and both fail loudly if their anchor phrase disappears. One limit: only those two lists are parsed. The other fix.<field> mentions in the prose (fix.class, fix.needs, fix.commands) are fields from that same list, but nothing parses them where they appear.

Comment thread xtask/src/eval_fixtures.rs Outdated
fn render_plan(fix_id: &str, title: &str, fix: &rocm_core::Fix) -> Result<String> {
// No `auto` entry needs a reboot today, so the plan has no line for one: refuse
// rather than print a plan that leaves the requirement out.
if fix.class != rocm_core::fix::FixClass::Auto || fix.needs_reboot {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

render_plan re-implements the CLI's plan rendering: the flag strings ("requires sudo", "requires re-login", "rocm fix can run it") and their order, from format_flags in crates/rocm-core/src/fix/render.rs. It also takes Verify: from the diagnosis, not the recipe. For fix-4 that gives ... && ls -l /dev/kfd && rocminfo | head -n 5, which the recipe in fix/catalog.rs doesn't have. --check compares the recordings to this generator, not to the CLI. So a wording change in format_flags would leave rocm-apply-auto-fix-needs-consent grading the agent against text the CLI no longer prints. The dropped Note: and other lines are disclosed; the duplicated flag logic isn't.

Suggest: make format_flags reachable from xtask and call it, or add a test that renders fix-4 both ways and compares the Flags: line.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in df9c981, and the fix goes further than format_flags. render_plan and its tests are gone. rocm-core now builds the whole plan in one function that print_recipe uses, with unchanged output, and exposes it as a #[doc(hidden)] rocm_core::fix_plan_text(id, os). The generator replays that function's output. So the flags line, the Note: lines and Verify: all come from the recipe, as rocm fix prints them. The fix-4 replay's Verify: no longer has the diagnosis's ls -l /dev/kfd. The refusal message and the dry-run notice are now constants the CLI prints and the generator reuses.

What's still not the CLI's: the runner's Will run: line is left out because it names the user and the machine. The layout around the plan (a blank line, then the notice) is written by hand in the generator and checked only against the committed replay. The module doc and docs/testing.md both say this.

Comment thread docs/testing.md Outdated
stub then answers `rocm fix <id> --dry-run` (the flag before or after the id) and
`rocm fix <id>` with no flag, and never `--yes`, so an agent that applies before the
user said OK meets the stub's refusal and the transcript shows the command. The
generator accepts only an `auto` entry that needs no reboot, since the plan has no line

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This misses one requirement. Besides being an auto entry with no reboot, the apply fix id has to be one the replayed diagnosis actually matches, given the case's symptom and host. Otherwise plan_text fails with "the replayed diagnosis carries no match for {fix_id}". The error is loud, but nothing here says so. Suggest adding "and the case's symptom and host must make the replayed diagnosis match it" (same for the module doc in xtask/src/eval_fixtures.rs).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in df9c981. docs/testing.md and the module doc in xtask/src/eval_fixtures.rs now say that the fix must be auto on the host's platform (wsl for the WSL host, otherwise linux) and that the case's symptom and host must make the replayed diagnosis list it at or above min_score_for_match. Otherwise the generator fails with "the replayed diagnosis carries no match for ...". A test pins that error.

The "no reboot" condition is gone: the plan now comes from the CLI's own renderer, so the generator no longer has a reboot line it can't print.

Host::AmdFound | Host::ContainerSmallShm => ("ok", true),
};
let value: serde_json::Value =
serde_json::from_str(&examine.stdout).expect("an examination is JSON");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just below, around line 612: assert_eq!(examination.status, examination.compute_status()) can't fail. examination_for ends with examination.status = examination.compute_status(); (line 292), so it compares a value with itself, while the comment says it checks "the status in the file". The assertion predates this PR, but the new ContainerSmallShm host now relies on it. Suggest deleting it, or deserializing examine.stdout and comparing its status with compute_status().

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was fixed on main by your #1047, which removed that assertion. The test now checks the status and has_amd_gpu in the replayed examine JSON against fixed values for each host. This branch is rebased onto it. The new ContainerSmallShm host is held to the same check (ok, true), and it's also in the test that runs each host over a contradicting base. Nothing in this PR changed that assertion.


#[test]
fn a_case_id_cannot_escape_the_fixtures_directory() {
let (_tmp, root) = scratch();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On pin_problems (around lines 1077-1084, outside this diff): only the "pin quoted in skill text" check has a test (a_pin_the_skill_text_quotes_is_caught). The other two, "pins nothing" and "pin not in its replay", have none. Deleting either keeps every test green, because the committed data satisfies both. The 8 new cases rely on them, so two scratch-copy tests like the existing one would be worth adding: one blanks a logs_contain, one pins a string absent from the replay.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in df9c981. There are two scratch-copy tests, built like a_pin_the_skill_text_quotes_is_caught. a_replayed_case_that_pins_nothing_is_caught empties a case's logs_contain. a_pin_the_replay_does_not_print_is_caught pins text the replay never prints. Each one asserts the problem message for its check.

@@ -15,8 +15,8 @@ Feature: The ROCm Doctor skill's instructions match the CLI they drive
#

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this header block (lines 11 and 34-36): the two tags cited from diagnose.feature don't exist as written. The real ones are @id:diagnose-fix-inapplicable-here-is-declined-not-attempted and @id:diagnose-fix-catalog-is-complete. Line 11 also says "a 16th failure mode", and the catalog now has 29. All three predate this PR, but this PR rewrites the block, and AGENTS.md §3 relies on cited @id:s being real. Suggest adding the diagnose- prefix and saying "a new failure mode".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f78bbc2. The header now cites @id:diagnose-fix-inapplicable-here-is-declined-not-attempted and @id:diagnose-fix-catalog-is-complete, which both exist in diagnose.feature. The third tag it cites, @id:diagnose-json-states-when-nothing-matched, was already correct. "A 16th failure mode" is now "a new failure mode".

@@ -718,8 +718,10 @@ async fn assert_documented_fields_present(world: &mut E2eWorld) {
// offered is fully readable by an agent following the skill.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment at lines 642-645 (outside this diff) is now stale. It says "marker cells above, per-host prose below ... the prose is what an agent reads before it decides whether to offer to run a fix". In the rewritten reference.md the per-host bullets sit above the table, and SKILL.md now tells the agent to read fix.class. The PR's known-gaps list mentions this; suggest fixing it here. Say the prose is checked so it can't drift from the table, and drop "what an agent reads".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f78bbc2. The comment now says the reference states it twice, as per-host bullets above the table and as marker cells in it, and that the bullets are checked too so they can't drift from the table. "What an agent reads" is gone.

@volen-silo

Copy link
Copy Markdown
Collaborator

I ran the Windows, WSL2 and sudo claims in this skill on real machines (Windows 11 and WSL2 on one AMD Strix Halo box, CLI built from main at 8f086ad9, skill text at 8610f5e). Most of it holds. Three places need a correction, and I've written the replacement text for each.

1. Quoting for PowerShell and cmd.exe (SKILL.md, step 1). The sentence "hand the user the command, written in the quoting of their own shell" gives no form, and the safe form differs per shell. I tested a paste with a leading -1073741819, single and double quotes, $(New-Item marker) and a backtick, and checked that no marker file appeared:

  • PowerShell 7.3+: rocm diagnose '--symptom=<text>' --json with every ' doubled. The typographic ‘ ’ ‚ ‛ count as single quotes too. An undoubled quote ends the quote early, and with an even number of them the $(…) runs.
  • Windows PowerShell 5.1: the same form fails (the text splits at a space after a "). It works when every " in the text is also written \". On 7.3+ that \" arrives as a literal backslash.
  • cmd.exe: rocm diagnose "--symptom=<text>" --json with every " doubled. With \" and an odd number of quotes, &, | and > run. %NAME% is expanded even inside quotes, so for text with % hand over the PowerShell form.

Suggested replacement for "In any other shell (PowerShell, cmd.exe), … written in the quoting of their own shell":

In any other shell, do not run it; hand the user the command in their shell's form, never the bash-escaped one, and ask them to paste the JSON back:

  • PowerShell 7.3 or later: rocm diagnose '--symptom=<text>' --json, with every single quote in the text doubled by repeating it. The typographic ‘ ’ ‚ ‛ are single quotes there too (it’s → it’’s). One left undoubled ends the quote early, and a $(…) after it runs.
  • Windows PowerShell 5.1: the same, and also write every " in the text as \". Without that the text splits at a space after a " and the command fails. Never use \" on 7.3+, where it arrives as a literal backslash.
  • cmd.exe: rocm diagnose "--symptom=<text>" --json with every " doubled (""), never \". With \", an odd number of quotes leaves &, | and > outside cmd's quotes, and they run. cmd expands %NAME% even inside quotes, so for text containing % hand the PowerShell form instead.

The "keep the =" claim is confirmed: with a space, all three shells fail with unexpected argument '-1' (exit 2). Not tested: PowerShell 7.0 to 7.2, a backslash before a ", multi-line pastes. A --symptom-file option (#1048) would remove the need for all of this.

2. /dev/shm on WSL2 (reference.md, the fix-19-shm-too-small row). "…which needs gigabytes where a container and WSL2 both default to 64 MB" is not true on current WSL2. On WSL 3.0.3.0 (kernel 6.18) with no .wslconfig, df -h /dev/shm shows 24G, half the VM's 49 GB of memory, and fix-19 does not fire. Suggested: "…which needs gigabytes where a container defaults to 64 MB; WSL2, like a bare-metal host, normally gives it half the VM's memory, so the entry fires there only when the allowance has been shrunk". Caveat: I used a copy of an existing Ubuntu's system tree, not a fresh install, and did not test older WSL versions. The same claim is in the CLI source (the SHM_MIN_BYTES comment in crates/rocm-core/src/diagnose/engine.rs), which I'll fix separately.

3. sudo with no terminal (SKILL.md, step 3.2). "…it may fail (the failed exit)" is right but incomplete. For rocm fix fix-4-render-group --yes with password-protected sudo (tested with sudo-rs 0.2.13 and sudo 1.9.17, on WSL with the WSL markers hidden so the sudo path ran):

  • no controlling terminal: sudo fails at once ("a terminal is required"), exit 4 in about 0.1 s;
  • shell attached to the user's terminal, nobody typing: the password prompt appears there, and rocm kills it at 60 s (exit 4).

Suggested replacement: "If the fix has needs_sudo, say too that rocm runs sudo with no input and kills it after one minute. Where sudo needs a password, a shell with no terminal fails at once; a shell attached to the user's terminal shows a password prompt there and fails after the minute if nobody answers. Either way it ends in the failed exit, and the user runs fix.commands themselves." Separately, rocm's message says "usermod exited 1" or "usermod exited 124", though it was sudo that failed or was killed; that is a CLI fix, not a skill one.

Confirmed as written: on native Windows, rocm serve --engine vllm refuses with exit 1 before any download, and its message matches the skill's wording. Not run: the installer's PATH behaviour in the same session, and diagnose --distro from a Windows host.

An agent following the skill literally would have acted on things the CLI
does not do. This brings SKILL.md and reference.md in line with the CLI
source and with what `rocm` prints.

Deciding what a diagnosis supports: step 2 is one ordered list that covers
out_of_scope, a native-Windows vLLM symptom, a symptom naming an app,
PyTorch or llama.cpp, and a match whose evidence does not explain what the
user described. Many entries score from host state alone, so every match
that clears the threshold is checked, not only the top one, and the one that
explains the problem is acted on; the check is keyed on the problem the user
described, not on whether text was pasted. Any host-only match that clears the
threshold is offered as worth doing anyway and never as the remedy; routing
the problem upstream applies only where no match explains it. A `--model` question is answered
from its `model` object and is not routed upstream on has_match. The
no-match routing decision is stated once: a failure of `rocm serve` itself,
with any engine, goes to this project's tracker (it is rocm's own run of the
engine, and `route_when_no_match` follows the host's framework) and is checked
before the app-named-in-the-symptom rule, and the CLI's trailing note on the
pytorch route is relayed as advice.

The `fix` fields the skill acts on are named once, in step 1, and `notes` are
relayed with the plan and with a diagnose-only explanation, since they carry
warnings the user must see.

Applying a fix: `rocm fix` refuses in a non-interactive shell without
`--yes`, so the skill goes dry-run, the user's OK, then `--yes`, and names
the argument a needs-argument fix takes. A need for a new session (re-login,
reboot, new terminal) is said with the proposal, before the OK, and so is that
a `sudo` fix may fail under `--yes` from an agent's shell (no input, one-minute
limit). A non-zero
exit is read by its manifest `exit_codes` name together with the output, with
a remedy for each and the same treatment for an exit the map does not name;
`--dry-run` is not offered after `not_applicable`, which it exits with too,
and the fix commands are handed over when the cause is a precondition.
For a `--distro` diagnosis the skill does not run `rocm fix` on the Windows
host and hands the user the fix commands to run inside the distribution. The
exit-code table in reference.md, which nothing pinned, is dropped.

Install: the consent text says the installer installs the CLI, edits the
user's PATH configuration, writes its own config under the home directory and
prints what it changed, with its opt-out; `nightly` is described as a moving
tag, and the CLI is re-checked by the full path the installer reports.

Symptoms: the pasted text is passed as `--symptom='<text>'`, which keeps a
leading `-` from being read as a flag. For bash and zsh it is single-quoted so
a paste cannot run as a command, with a `'` in the paste written `'\''`; for
any other shell the user is handed the command and asked to paste the JSON
back. The
`vllm: ` prefix is added only when vLLM is known to have printed the line, a
`rocm diagnose` command printed by `rocm serve` is run with `--json`, and
fix-17 relays the CLI's confidence.

Reference: `model.warnings[]` is named as the field that carries the note on
which options `rocm serve` does not take; Typical-signal cells are short handles with no score arithmetic
and the CLI's score and evidence decide; the diagnose section documents the
`--distro`, `--report`, `--send` and `--top` flags (`--distro` and `--report`
as alternatives, `--send` not with `--json`) and the `model` object, and the
`fix` and `model` field lists are cut to the fields the skill acts on; the
Marker spelling is said to be lower-cased; rules stated in both files are
kept in one, with a pointer from the other; the Out of scope section and the
Linux-only/Windows-only recap, which restated SKILL.md and the table, are
removed.

Routing: a symptom naming both an app in Framework routing and PyTorch or
llama.cpp goes to the app's tracker, and the published skill cites no source
file or line.

vLLM: a startup failure described in words matches neither vLLM entry, so
the skill asks for the exact error line (or the diagnose command `rocm serve`
printed) first on Linux or WSL2 instead of diagnosing without a symptom.

Tests: two tests in skill_reference.rs hold the exit-code list in SKILL.md equal
to the manifest's non-zero `exit_codes` and every `fix` field it names to the
diagnosis JSON. Both read only the list after their marker and match on the
whitespace-collapsed text. A missing marker, a `fix` field list that names no
field, and a field name that is not lower-case snake_case fail loudly; a wrong
exit-code list fails the comparison. The fixture sentences SKILL.md must keep are
listed once, in skill_reference.rs.

Rules: a `suggestions[]` entry of a model estimate is relayed as the llama.cpp
option it names and never written into a `rocm serve` command, which takes no
context size, KV cache type or GPU layer count; "never invent a fix" is scoped
to what the CLI returns, a diagnosis `fix` or the steps in `model.fix`; and
`model.warnings[]` is described as carrying the standing note only for a GGUF
estimate that was compared with the GPU.

Also: the advice-only entry is named where step 1 describes advice, the
description gains a boundary against first-time setup, the
CPU-fallback ban from the assistant policy is carried over, LM
Studio is routed to its public bug tracker, a failed install is in scope and
a first-time setup request is not.

AGENTS.md no longer counts the SKILL.md fixture sentences; it points at the
module doc of skill_reference.rs, which lists them all.

Comments and messages in the CLI crate and the e2e suite that pointed at the
removed exit-code table, the removed recap, or `auto_applicable` as what the
skill reads now point at the manifest's `exit_codes`, at a paragraph that
still exists, or at `fix.class`, and the runner comments speak of the `auto`
class and of fix-9 as `needs-argument`. No behaviour changes outside the skill.

Corrections from a hardware run and its review: a pasted symptom is quoted for
PowerShell and cmd.exe only when it is one line with no `"`, `\` or `%`, and
otherwise the user is asked for a single line or to quote it themselves; the
fix-19 row says only that the entry fires under 1 GiB of `/dev/shm` (a container
defaults to 64 MB); the `needs_sudo` disclosure says what happens with and
without a controlling terminal (sudo is killed after one minute, and an
unanswered prompt ends in the `failed` exit). A diagnose-only match is relayed
with its `summary` and `notes` and offers no fix, the apply sub-steps are
lettered so step 4 is unambiguous, the routing-follows-the-host rule is stated
once, and the guard test and READMEs name the fixture sentences from one place.

Signed-off-by: Teemu Kuusisto <teemu.kuusisto@amd.com>
The skill text carries rules no eval reached: asking before applying an auto
fix, telling a match that explains the pasted error from one that only
reflects the host, routing Lemonade, Ollama, LM Studio and PyTorch-named
failures and a failure of `rocm serve` itself, passing a paste as one literal
argument, never suggesting a CPU route, asking for the error line when a vLLM
startup failure is described in words, disclosing a sudo fix and what the
installer changes before asking for consent, and declining a first-time setup
on a clean machine.

Add replayed cases for each, with judged expectations rather than pins the
skill text itself satisfies. The routing prompts name the vendor (AMD), so the
agent does not stop at `rocm examine --json`, which those replays refuse. The
vLLM case's premise, that the description matches nothing, is pinned. A container
host with the default shared-memory size lets the replay hold a host-only match
above the one that explains the paste, and a second case holds a host-only match
as the only match that clears the threshold; every host states its container and
`/dev/shm` fields, and a test with a contradicting base asserts them. A test
pins the premise each judged case rests on, so a catalog change cannot silently
void it.

The stub answers the dry run and the unflagged `rocm fix <id>` from the replayed
catalog, rendered by `cargo xtask eval-fixtures` like every other recording, so
the existing check covers it, and it takes `--dry-run` before or after the id as
the real command does, never answering `--yes`. The plan it prints is the CLI's
own: rocm-core exposes `fix_plan_text`, a shared plan builder extracted from
`print_recipe`, and the refusal and dry-run strings and the `declined` exit as
constants the CLI prints from its own code paths and the generator reuses, so
the replay cannot drift from the CLI. `print_recipe`'s output is unchanged, and
every new export is `#[doc(hidden)]`, like the crate's other test seams. Only
the runner's `Will run:` line, which names the user and the machine, is left
out. The print-only fallback has one home on the recipe,
`FixRecipe::class_on_os`, instead of copies. The platform comes from the host,
so the WSL host, where `rocm fix` reports a Linux-only fix as not applicable, is
refused with a clear message, and an unknown id is reported as unknown. Only an
`auto` entry is modelled, the apply fix must be one the replayed diagnosis
matches and must clear the match threshold, and a fix that is not `auto` on the
host's platform is refused, with a test. The apply-match guard is tested with a
paste that makes a different fix clear the threshold. The needs-argument wording
of the plan is pinned. The apply items about logging out and `sudo` rest on
`needs_relogin` and `needs_sudo` being set for the replayed fix, which the
premise test asserts, and the two vLLM cases that rested on a recorded score pin
their premises. The two `pin_problems` checks that nothing exercised (a case
that pins nothing, a pin its replay does not print) are now tested.

The testing and skill-quality docs are brought in line with what is replayed.
They say the layout around the plan is written by hand and checked only against
the committed replay.

Signed-off-by: Teemu Kuusisto <teemu.kuusisto@amd.com>
@siloteemu
siloteemu force-pushed the docs/rocm-doctor-skill-review branch from 8610f5e to df9c981 Compare November 12, 2026 13:55
@siloteemu

Copy link
Copy Markdown
Contributor Author

I ran the Windows, WSL2 and sudo claims in this skill on real machines

Thanks for running these on real hardware. All three corrections are in f78bbc2, two of them narrower than your text:

  1. Quoting (SKILL.md step 1). bash/zsh keep the single-quote form. For any other shell the agent hands the user a written command only when the paste is one line with no ", no \ and no %. In that case PowerShell gets rocm diagnose '--symptom=<text>' --json with every single quote doubled, typographic ones included. Without " or \, your 5.1 and 7.3+ forms come out the same, so there's one PowerShell form. cmd.exe gets rocm diagnose "--symptom=<text>" --json. For any other paste the skill writes no form: it asks for a single line without those characters, and if the user can't give one, has them quote their text for their own shell. I didn't take the per-version \" and "" rules because the cases you didn't test (7.0–7.2, a backslash before a ", multi-line) fall exactly there. The "keep the =" sentence now covers every form.
  2. /dev/shm (reference.md, fix-19 row). It now says only that the entry fires wherever /dev/shm is under 1 GiB and that a container defaults to 64 MB. It makes no WSL2 claim either way, because the CLI's own fix-19 rationale and docs/wsl.md still say WSL2 ships the 64 MB default. That source claim is yours to correct, as you said: "The same claim is in the CLI source (the SHM_MIN_BYTES comment in crates/rocm-core/src/diagnose/engine.rs), which I'll fix separately." The fix-19 rationale in fix/catalog.rs and docs/wsl.md carry it too.
  3. sudo (SKILL.md step 3). Your two outcomes are in: with no controlling terminal it fails at once, and an unanswered prompt fails after the minute, both with the failed exit. I also added two things you didn't run: the fix applies if the user answers the prompt in time, and cached credentials, a NOPASSWD rule or running as root skip the prompt. The "usermod exited N" wording when sudo was the step that failed is a CLI message, so it isn't in this PR.

@siloteemu

Copy link
Copy Markdown
Contributor Author

I rebased onto main and force-pushed (now df9c981). The changes address the review and the hardware-run comment:

  • SKILL.md: a vLLM startup failure described in words now gets a request for the exact error line before diagnosing. An eval case covers it.
  • Two new tests tie SKILL.md's exit-code list to the manifest's non-zero exit_codes and its fix field list to the diagnosis JSON.
  • The eval generator replays the CLI's own plan text (rocm_core::fix_plan_text) and its refusal and dry-run strings instead of a copy. docs/testing.md states every condition for an apply case. The two untested pin_problems checks now have tests.
  • From the hardware run: PowerShell and cmd.exe forms for a one-line paste with no ", \ or %; a fix-19 row with no WSL2 default claim; and what sudo does with and without a terminal. My reply to that comment says where these differ from the suggested text.

Known gaps:

  • The model-graded routing and behavioural evals were not run, including the new judged cases.
  • No eval case covers a paste outside that one-line set.
  • Windows, WSL2 and GPU items beyond that run are tracked separately.
  • This PR changes no rocm fix behaviour; fix-9 and the other entries are as on main.

On the two points in the review body:

Phase 0 offering to install before anything shows an AMD GPU. Yes, that's deliberate. Nothing can probe the machine without the CLI. If the user names NVIDIA, Intel or Apple, the scope gate declines before Phase 0 runs. If they name no vendor, ROCm or HIP wording is treated as AMD, and the install only runs with the user's consent, after the skill has said what the installer changes. Once the CLI is in place, step 1 runs rocm examine --json first. On no-amd-gpu, or has_amd_gpu: false on WSL2, the skill says the CLI found no AMD GPU, asks which GPU the machine has, and proposes no fix until that's settled.

E2E tests (Strix Halo, WSL2, current) on 8610f5e. I re-ran it once and it passed. The new head runs the full set again.

Ready for another look once CI is green on the new head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants