Skip to content

ROCMAI-955: name the engine a diagnose symptom came from with --engine - #1052

Open
volen-silo wants to merge 1 commit into
mainfrom
rocmai-955-diagnose-engine
Open

volen-silo wants to merge 1 commit into
mainfrom
rocmai-955-diagnose-engine

Conversation

@volen-silo

@volen-silo volen-silo commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Why

The vLLM out-of-memory entry (fix-16-vllm-oom) scores only lines that mention vLLM (vllm or gpu-memory-utilization). That is deliberate: the error vLLM prints at startup is an ordinary torch.OutOfMemoryError, and without the anchor any PyTorch program's out-of-memory error would be blamed on vLLM. But a user's paste often does not contain the word. To get a match, the rocm-doctor skill has been editing the user's text, writing vllm: in front of it, so what the CLI diagnosed and reported was not what the user pasted.

What changes

  • New rocm diagnose --symptom <TEXT> --engine vllm says which engine the symptom came from. With it, every non-blank line of the symptom counts as vLLM's for the vLLM entry. That gives the same match as prefixing each line with vllm, and the text is not changed.
  • Only engines the catalog tells apart are accepted, which today is vllm, the name rocm serve --engine uses. Any other value, lemonade included, is a clap usage error (exit 2) that lists the accepted values. --engine needs --symptom or --symptom-file and is refused alone, because it would describe nothing. A file's text is recorded without its final line break, as the reader strips it.
  • No catalog entry other than the vLLM out-of-memory one reads the engine. Native Windows still has no vLLM entry and still gets no vLLM match.
  • The JSON report gains an optional provenance: {engine, symptom}, with the symptom exactly as given once the JSON is decoded. On the wire, rocm diagnose --json writes every non-ASCII character as a \uXXXX escape (UTF-16 units), so the output stays plain ASCII as the existing rule for redirectable output requires (ROCMAI-543) and any JSON parser recovers the identical paste. It is absent when no engine is named, so reports without --engine are unchanged. The diagnose JSON has no version number and the field is additive, so nothing is bumped.
  • With --engine, the text report opens with one line saying which engine was named and whether it counted on this OS (it says the flag changed nothing where no entry reads the engine, native Windows today). The answer comes from the catalog table that records which entry reads the engine, through the same platform filter the checks use; JSON is unchanged.
  • --report is unaffected: the shareable report is a fixed struct built without the diagnosis, so neither the paste nor provenance can reach it or its mail link. --engine stays accepted with --report and --distro; with --distro the diagnosis, provenance included, is shown on the user's own screen, and the docs say so.
  • Writing vllm in the symptom keeps working: the vLLM engine's own startup hint and rocm serve's out-of-memory note still print that form.
  • In rocm-core, the catalog marks which checks read the engine, and diagnose_with_engine is the new entry point. diagnose calls it with no engine.
  • --help, README.md, docs/vllm.md, docs/testing.md, docs/manual-testing.md and docs/architecture.md describe the flag.

Verification

New scenarios in tests/e2e-cucumber/features/diagnose.feature, numbered 68 to 76 after #1048, #1051 and #1053 took 59 to 67:

  • diagnose-68 (@requires-os:linux): a two-line vLLM out-of-memory paste that never names vLLM, diagnosed with --engine vllm, is matched as the vLLM startup out-of-memory failure, and the report's symptom equals the paste byte for byte.

  • diagnose-69 (every lane): the report records vLLM as the named engine and keeps the paste as given.

  • diagnose-70 (every lane): --engine lemonade is refused as a usage error, and nothing is diagnosed.

  • diagnose-71 (@requires-os:linux): on a simulated Linux machine whose only GPU is a released one (the same Given as diagnose-58, so a real report is prepared on every Linux lane, with or without a GPU), --report with the engine named and a paste carrying a user, host and token prints none of them and no provenance, and nothing is sent. The sweep first requires that a real report was printed. Native Windows cannot be simulated this way; the payload unit test covers it.

  • diagnose-72 (every lane): a paste piped through --symptom-file - with --engine vllm records the engine and the exact symptom without its final line break.

  • diagnose-73 (@requires-os:linux, so Linux and WSL): the text report opens with Engine: vllm (named with --engine; ...) and shows the vLLM cause, which is absent without the flag.

  • diagnose-74 (@requires-os:windows): the text report opens with the line saying no entry for vllm exists on this OS, so the flag changed nothing, and the rest of the report is byte-identical to the run without the flag.

  • diagnose-76 (@requires-os:linux): an unrelated paste (a plain Python traceback) with --engine vllm in text mode: the report opens with the line saying the lines were read as vllm's, and the vLLM cause is absent, so the line is asserted together with the nothing-scored outcome.

  • diagnose-75 (every lane): a non-ASCII paste (an accented path, a box-drawing character, an emoji) saved to a file and diagnosed with --symptom-file <file> --engine vllm --json prints only ASCII, and the decoded provenance.symptom equals the paste.

The ROCMAI-543 ASCII test now also runs --engine rows with a non-ASCII paste on Linux, WSL and Windows; a_non_ascii_paste_round_trips_through_the_json_exactly checks the parsed symptom equals the paste byte for byte; every_engine_name_is_its_serialized_name ties the printed engine name to its JSON name for every variant. Dropping the escaping fails the first two; removing the CLI's escaping hook fails diagnose-75.

the_text_report_says_which_engine_was_named_and_whether_it_counted also has a row for an engine named on a host that reads it with an unrelated paste and with an empty one: the line says the lines were read as vllm's and no vLLM entry is in the report. every_engine_reading_entry_reads_the_engine_it_declares walks the catalog table and checks, for every engine-reading entry, every platform family it applies to and every engine variant, that an engine-free paste changes verdict when the declared engine is named and does not for any other; it fails if it finds no such entry. render_report_json returns a Result through JsonOrText::render_json; a test checks that a failing hook surfaces an error and writes nothing.

Unit tests check that naming vLLM matches as the vllm: prefix did, that every non-blank line of a multi-line paste is anchored, that no entry other than vLLM's changes, and that the report keeps the symptom verbatim. a_shared_report_never_carries_the_pasted_symptom_or_its_provenance plants a token, a path and a host in the paste and checks the report JSON and the mail link; leaking the symptom into the entry, and separately into the architecture matrix, makes it fail. The clap parse test covers --engine with both symptom sources and alone.

Platform Run Result
Windows cargo test --workspace --all-targets --no-fail-fast (AGENTS.md §8) 47 suites, 6963 pass, 0 fail, 17 ignored
Windows cargo xtask smoke (AGENTS.md §8) smoke: ok
Windows cargo test -p xtask 833 pass
Windows cargo clippy --workspace --all-targets -D warnings, cargo clippy -p e2e-cucumber --test e2e -D warnings, cargo fmt --check, feature_naming, skill_reference, eval-fixtures --check, catalog --check, check-architecture-doc pass
Windows cargo nextest run on the 12 crates cargo xtask affected selects 6731 pass, 0 fail
Windows cargo xtask e2e for diagnose-59, 61 to 67, 69, 70, 72, 74, 75 13 scenarios, 48 steps pass (diagnose-60, -68, -71 and -73 are Linux-only)
WSL2 (Ubuntu, Strix Halo) same nextest set 7292 pass, 2 fail (below)
WSL2 cargo xtask e2e for diagnose-58 to -73, -75 and -76 18 scenarios, 68 steps pass

These rows were taken on the final commit on top of main 9e50eac5, on Windows and in WSL2. The pre-push hook failed once on atomic_write::tests::a_reader_never_finds_the_destination_absent_while_it_is_replaced, a known Windows timing test in code this change does not touch; it passed alone 3 of 3 and the hook passed on the retry.

Failures and instability outside this change:

  • WSL2: build_metadata::build_script::tests::git_watch_paths_* fail because that account's git config signs commits with an SSH key that is absent there. They pass on Windows.
  • WSL2, once under load: process::tests::loopback_lease_blocks_a_second_bind_until_released (rocm-core) failed in the full-workspace run and passed alone 3 of 3, in code this change does not touch.
  • Windows, intermittent on this machine: the two rocm-engine-lemonade archive_cache tests, interprocess_lock_serializes_poisoned_validation_and_recovery (times out waiting for its child process) and archive_digest_check_rejects_a_single_bit_of_tampering (can hang for tens of minutes). Both also fail or hang on main under load, and both passed in the run recorded above.

Scope

  • The ASCII escaping of rocm diagnose --json (and its render_json hook and the fixture recorder using it) lands in this change on purpose: echoing the paste verbatim in provenance.symptom is what adds the first raw user text to that output, and shipping --engine without the escaping would break the ROCMAI-543 rule for redirectable output on the day it lands. The text report and other --json commands can still print host-derived non-ASCII.
  • The ASCII escaping covers provenance.symptom, the only field that holds the user's paste. Host-derived evidence lines can quote an install path read from the examined machine; a non-ASCII install path is now escaped in --json, but the text report still prints it raw. That predates this change and is left alone.
  • skills/rocm-doctor/ is not changed here. fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029 rewrites it; replacing its vllm: instruction with --engine vllm follows once that lands.
  • The vLLM engine's startup hint and rocm serve's out-of-memory note keep printing the vllm: … symptom form, which still works.
  • --engine gains another value only when the catalog has an entry specific to that engine. Until then, lemonade is a usage error here even though rocm serve accepts it.
  • rocm remote doctor --symptom has no --engine; remote users still need the vllm: prefix. That is a follow-up.

Part of ROCMAI-955.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable: no bug fix.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. Not applicable: a flag on an existing command.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state, per AGENTS.md §3. The usage error for an unknown engine is asserted together with nothing being diagnosed (diagnose-70), and the claim that --report never carries the paste is asserted against the printed report (diagnose-71) and the payload (unit test).

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Merge note: #1048, #1051, #1052 and #1053 each add e2e scenarios numbered from diagnose-59 in diagnose.feature, and feature_naming requires the numbers to be sequential. Whichever of the four merges first keeps its numbers. Each of the others renumbers on rebase, and updates the numbers cited in its docs/testing.md section. Their @id: tags are distinct, so only the numbers collide. Their skill-doc follow-ups wait for #1029 to merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@volen-silo
volen-silo force-pushed the rocmai-955-diagnose-engine branch from 584db61 to c205556 Compare November 12, 2026 14:01
@volen-silo
volen-silo marked this pull request as ready for review November 13, 2026 06:59
@volen-silo
volen-silo requested a review from a team as a code owner November 13, 2026 06:59

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head c205556d

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change. Implements ROCMAI-955 ("name the engine a diagnose symptom came from with --engine"); the ticket itself was not retrievable from here (no tracker access available to this review), so it is judged only against the PR description, which matches the diff in substance.

Blocking

None

Non-blocking

  • docs/architecture.md's cli_args.rs row is stale — docs/architecture.md (the cli_args.rs entry), apps/rocm/src/cli_args.rs:1-17
    The PR adds a new clap enum, DiagnoseEngineArg, to apps/rocm/src/cli_args.rs, and even updates that file's own module doc comment to mention it ("DiagnoseEngineArg (diagnose --engine) is the exception: it was added here after the move rather than relocated."). The corresponding row in docs/architecture.md still enumerates only the pre-existing enums (Provider, DevicePolicyArg, FrameworkArg, InstallFormat, WatcherModeArg, TelemetryModeArg, SandboxToolArg, PermissionsModeArg) and isn't in this PR's diff, so it no longer fully describes the file. cargo xtask check-architecture-doc only fails on a missing entry, not a stale one, so this slips past CI. The PR's own description claims "docs/architecture.md describe the flag," which is true for the diagnose.rs/diagnose/catalog.rs rows it did touch, just not this one.
    Confidence 85 · mechanical · Fix: add DiagnoseEngineArg and diagnose --engine to that row's enum/flag list.
  • Stale doc comment on SUPPORTED_ENGINES — apps/rocm/src/cli.rs:25-27
    SUPPORTED_ENGINES's comment says it's "used as the clap value_parser for every user-facing --engine argument so the possible values stay in sync across the CLI surface." This PR adds a second user-facing --engine argument (diagnose --engine) that deliberately does not use SUPPORTED_ENGINES — it has its own DiagnoseEngineArg enum listing only vllm, by design (the catalog only tells vllm apart today). The comment predates this PR and is no longer accurate now that there are two independently-scoped --engine flags.
    Confidence 75 · mechanical · Fix: reword to exclude diagnose --engine, e.g. "...every --engine argument that serves or installs an engine (not diagnose --engine, which is narrower)."

Decisions for the author

None

Positive signals

  • The Check::Text / Check::WithEngine wrapper enum in catalog.rs lets one checker (check_16_vllm_oom) take the new engine parameter without touching the other 21 checkers' signatures or threading an unused Option through them.
  • Strong, layered test coverage: equivalence tests proving --engine vllm produces identical verdicts to the old vllm: prefix convention (single- and multi-line), a "changes nothing but the vLLM entry" test that includes a Windows host where vLLM doesn't run, a byte-exact provenance test using control bytes and CRLF, a clap-level unit test for the parse/error-kind, and e2e scenarios (diagnose-59/60/61) exercising the same behavior through the real CLI binary. The e2e fixture for the "verbatim" claim is deliberately engineered (leading blank-ish line, trailing spaces) so the assertion isn't vacuous against a trim/rejoin bug.
  • The refusal path (--engine lemonade) is asserted together with its effect (exit code 2 and that nothing was diagnosed), matching this repo's own "a message is asserted together with the resulting state" convention.
  • provenance is wired as genuinely additive (#[serde(default, skip_serializing_if = "Option::is_none")], field absent rather than null when no engine is named), so existing JSON consumers are unaffected.

Deployment notes

None — the new provenance field is additive/optional on the rocm diagnose --json report, and no existing command's default behavior changes.

What this covered

Reviewed the whole diff between f536619a08a3b50dd26aac5c21a3bb9f6bd266d2 and c205556dccd1ea04965f179d15c1255ab6ce996a (19 files, +617/-68) via four independent parallel passes grouped by coupling — core engine/catalog (crates/rocm-core/src/diagnose.rs, diagnose/catalog.rs, diagnose/engine.rs, diagnose/scoring.rs, diagnose/host.rs, diagnose/report.rs, lib.rs), the CLI layer (apps/rocm/src/cli.rs, cli_args.rs, diagnose_cmd.rs, main.rs, remote/doctor.rs), the e2e cucumber tests (tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs), and the docs (README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, docs/vllm.md) — plus my own direct read of the core and CLI diffs as a cross-check. Each pass read its files completely at the PR head plus surrounding context, and verified claims against the actual code rather than taking the PR description at face value until findings were settled.

CI at this head is fully green (clippy, both E2E consolidated reports, Linux/Windows build-and-test, coverage ratchet, commit-signatures, security scans); the only non-success entry is "Skill checks (skillscope)" showing skipped, which this repo's own docs mark as advisory, not a required gate.

Did not run: no separate "prior review comments" pass, since this PR has no prior review history to reconcile against; the 2 existing PR comments were not read, per this method's rule that review findings are reached blind from the code and discussion is not reconciled here.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing here breaks the diagnosis itself, and the tests and scenarios for the new flag look solid. I'm requesting changes on one open question: what --report does with the pasted symptom now that it is stored.

To address before merge

provenance.symptom copies the user's raw paste into DiagnoseReport. The symptom was never stored in the report before, and the report is built through the same call the --report / --send path uses. If --symptom ... --engine vllm --report --send is accepted, an arbitrary pasted log (paths, hostnames, tokens) travels in a report meant to be shared. I couldn't tell from the diff what --report does here. Please either drop provenance on that path, refuse the combination, or state and test that the text is redacted. Right now nothing covers --engine together with --report or --distro.

Smaller notes, non-blocking

  • On native Windows --engine vllm is accepted but there is no vLLM entry, so the user gets a plain no-match. Text output never echoes the engine; only --json shows provenance. A one-line note in the text rendering would make that visible.
  • The doc on vllm_anchored_lines says the paste is treated "exactly as" if each line started with vllm, but blank lines are dropped here while a vllm: prefix would keep them. Harmless for scoring; "every non-blank line" would be more accurate.
  • The rocm-doctor skill still tells agents to write the vllm: prefix, and rocm remote doctor --symptom has no --engine. Both are disclosed as follow-ups in the PR body, so fine to leave.
  • CI: the only failure is the MI300X E2E job (12 vLLM serve/bench/chat scenarios, torchvision::nms does not exist); diagnose-59/60/61 pass there. It looks environmental, but worth confirming it also fails on main.

Comment thread crates/rocm-core/src/diagnose.rs Outdated
route_when_no_match: route_when_no_match(e),
out_of_scope,
model: None,
provenance: engine.map(|engine| SymptomProvenance {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This stores the raw paste in the report. Is --symptom ... --engine ... --report [--send] accepted? If so, pasted logs (paths, hostnames, tokens) now travel in a report meant to be shared. Please confirm what --report does with it (redact, drop, or refuse) and add a test or doc line.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7c2d1f3. --symptom ... --engine ... --report stays accepted, and the report is the fixed Report struct (schema, matrix, entry id, OS, distro, ROCm and engine versions, fix_offered). It never receives the diagnosis, so neither provenance nor the paste can reach it or the mail link. That is now stated in the README and --help and proved by a_shared_report_never_carries_the_pasted_symptom_or_its_provenance, which plants a token, a path and a host in the paste and fails if the symptom leaks into the entry or the matrix, plus scenario diagnose-65 against the real binary. --distro shows the diagnosis on the user's own screen, provenance included, and the docs say so.

let mut report = rocm_core::run_diagnose(&examination, &symptom.unwrap_or_default());
let (symptom, engine) =
symptom.map_or((String::new(), None), |query| (query.text, query.engine));
let mut report = rocm_core::run_diagnose_with_engine(&examination, &symptom, engine);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same call builds the report for --report / --distro, and no test covers --engine combined with them. Same question as the provenance comment in diagnose.rs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered in 7c2d1f3, see the reply on the provenance comment: diagnose-65 runs --engine with --report and asserts none of the paste, host or provenance appears and nothing is sent; the unit test covers the payload and the mail link.

@volen-silo
volen-silo force-pushed the rocmai-955-diagnose-engine branch from c205556 to 7c2d1f3 Compare November 16, 2026 09:29
@volen-silo

Copy link
Copy Markdown
Collaborator Author

The fixes for the changes-requested review are in 7c2d1f3, rebased onto main after #1067.

  • --report and the pasted symptom (the blocking point): --engine stays accepted with --report and --distro. The shareable report is a fixed struct that is built without the diagnosis, so neither the paste nor provenance reaches it or its mail link. This is documented, covered by a_shared_report_never_carries_the_pasted_symptom_or_its_provenance (with mutation checks) and by scenario diagnose-65 against the real binary. See the inline replies.
  • "every line" wording: now "every non-blank line" in the code docs, --help, README and docs/testing.md.
  • --engine with --symptom-file: now accepted through a clap group on both symptom flags, with scenario diagnose-66. While doing this I found the group had to sit on the fields, not on the command, for clap to register it.
  • Windows no-match and text output not echoing the engine: not changed here. Both are noted in the description as follow-ups.
  • rocm-doctor skill still writing the vllm: prefix, and rocm remote doctor --symptom having no --engine: not changed here. The skill is rewritten in fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029, and the remote form is listed under Scope.

The description is updated, with the scenarios renumbered to 62 to 66.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 16, 2026 09:31

Addressed in 7c2d1f3: --report is built from a fixed struct that never receives the paste or provenance, now documented and tested (unit test + diagnose-65). Replies: #1052 (comment) and the two inline threads.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this diff and found nothing that blocks merge on correctness. Requesting changes on the points below, which I'd like addressed or answered before this lands.

What I checked and found sound: --engine vllm only feeds check_16, and every non-blank line counts as anchored, which matches a per-line vllm: prefix (the unit test compares score and evidence on linux and wsl). provenance is additive and omitted when no engine is named. The --report document and mail link are built from the examination plus the established entry only, so neither the pasted text nor provenance reaches them (unit test plus diagnose-65). Scenarios diagnose-62..66 cover the new behaviour and assert the refusal with empty stdout.

To address:

  1. Text output never says which engine was named. On native Windows, where there is no vLLM entry, --engine vllm is accepted and silently has no effect. A one-line note in text mode, or a warning on Windows, would stop a user-supplied flag from being ignored without a trace.
  2. rocm remote doctor --symptom has no --engine, and the doctor skill still needs the vllm: prefix there. The PR body acknowledges this, but the ticket item is only partly delivered until the follow-up lands. Please link the follow-up so it is not lost.
  3. The cli_args.rs row in docs/architecture.md is still tagged relocation although the file now holds a new enum.
  4. Two doc lines run longer than the text around them (inline comment on README.md; the last paragraph of manual-testing 6c is similar).

CI: nothing has failed. The E2E lanes, affected-crates tests and Windows debug test are still pending, so this reflects a diff read only. I did not re-run the claim that the new tests fail when their target is broken.

Comment thread README.md Outdated
- `--engine` names the engine the symptom came from (only `vllm` is
accepted; it needs `--symptom` or `--symptom-file`). An engine's own error
is often a line any program could print, such as a PyTorch out-of-memory
error, and the vLLM entry claims only lines that name vLLM. With `--engine vllm`, every non-blank line

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this line is longer than its neighbours in the new bullet; re-wrap for consistency.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 37e1085: the new --engine bullet is re-wrapped to the width of its neighbours, and the same fix is applied to the long paragraph and command in manual-testing 6c.

@volen-silo
volen-silo force-pushed the rocmai-955-diagnose-engine branch from 7c2d1f3 to 37e1085 Compare November 16, 2026 12:06
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Round 4 is in 37e1085, rebased onto main after #1051 and #1093 (scenarios renumbered to 65 to 71).

  1. Text output never says which engine was named: done. With --engine, the text report now opens with one line. On Linux and WSL it says the engine was named and that every non-blank line was matched as its own. On native Windows it says no entry for that engine exists on this OS, so the flag changed nothing. A report read back from JSON prints only the named engine. The answer comes from the catalog table that records which entry reads the engine, through the same platform filter the checks use, so no OS is hardcoded, and the JSON is unchanged. Unit test the_text_report_says_which_engine_was_named_and_whether_it_counted covers Linux, WSL, Windows and an uncovered platform and fails if the decision is always true; scenarios diagnose-70 (Linux and WSL) and diagnose-71 (Windows) run the real binary.
  2. Link the follow-up for rocm remote doctor --engine and the skill's vllm: prefix: not done as a separate ticket. Both stay out of this change and are listed under Scope as follow-up work under ROCMAI-955; the skill change waits for fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029.
  3. cli_args.rs row tagged relocation: the tag stays. The architecture doc's legend says the class records how a file came to be, not a rule a new file is judged by, and this file came from a mechanical relocation. I added a sentence to the row saying so; check-architecture-doc passes.
  4. Long doc lines: the README bullet and the manual-testing 6c paragraph and command are re-wrapped.

The description is updated with the new scenarios and the rebased results.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 16, 2026 12:07

Addressed in 37e1085: the text report now says which engine was named and whether it counted on this OS, architecture row explained, long lines re-wrapped; remote doctor --engine and the skill prefix stay listed as follow-ups. Replies: #1052 (comment) and the README thread.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 37e10855

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change. Implements ROCMAI-955 ("name the engine a diagnose symptom came from with --engine"); the ticket itself was not retrievable from here (no tracker access available to this review), so it is judged only against the PR description, which matches the diff in substance (new --engine flag, scenarios diagnose-65..71, the additive provenance field, --report exclusion) and explicitly discloses the rocm-doctor skill and rocm remote doctor --symptom as deferred follow-ups rather than part of this change's scope.

Blocking

  • provenance.symptom can put non-ASCII user text on a redirectable stream, contradicting this codebase's own ROCMAI-543 invariant — crates/rocm-core/src/diagnose.rs:93-99,230, crates/rocm-core/src/diagnose/catalog.rs:400-419
    catalog.rs's dedicated regression test states the contract explicitly: "every string rocm diagnose can put on a redirectable stream has to be plain ASCII, or a Windows console on the OEM code page (437) renders UTF-8 punctuation as mojibake," and asserts it against both the rendered text report and the --json payload. Before this PR, the only paste-derived content reaching output was evidence, built from a closed, static, ASCII keyword vocabulary (e.g. KEYWORDS_VLLM_OOM in check_16_vllm_oom) — never a raw copy of user input. provenance.symptom is the first path that serializes the user's full paste verbatim, byte-for-byte (confirmed by this PR's own the_report_records_the_named_engine_and_the_symptom_verbatim test, which uses control bytes and CRLF to prove no edits occur), with no ASCII filtering — and the invariant test's corpus is never run through diagnose_with_engine, so this new surface is unguarded.
    Confidence 85 · logic · Fix: extend the ROCMAI-543 corpus with a diagnose_with_engine case using a non-ASCII symptom and document provenance.symptom as a deliberate, narrow exception (with a reason) if that's the intent — or constrain/escape the field so the invariant holds end-to-end.

Non-blocking

  • SymptomEngine's CLI-printed name and its JSON wire name are two independent, never-cross-checked representations — crates/rocm-core/src/diagnose.rs:79-83 (name()) vs. the enum's #[serde(rename_all = "lowercase")] derive (line 70)
    Both happen to read "vllm" today with a single variant, so nothing can yet disagree, but nothing asserts they must. Confidence 70 · architectural · Fix: a small test asserting name() matches the serialized value for every variant, so a second engine can't silently desync the two.

Decisions for the author

  • skills/rocm-doctor/SKILL.md/reference.md still instruct writing the vllm: prefix rather than --engine vllm — answered-by-intent
    diagnose.rs's own doc comment on SymptomEngine says: "the skill is to move to --engine in a later change", and the PR description's "Scope" section discloses this explicitly, tied to #1029. Would otherwise have been a non-blocking impact-analysis gap; confirm this sequencing is still the plan.
  • rocm remote doctor --symptom has no --engine — answered-by-intent
    Same "Scope" section discloses this as a follow-up. Confirm it's tracked.

Positive signals

  • Check::WithEngine's tag cleanly separates "which catalog entries read the engine" from "which engine the user named for this invocation" — reads_engine_here/read_here are derived structurally off the same CHECKERS table and platform filter the checks themselves run from, not a hand-maintained OS list, and this is directly exercised across linux/wsl/windows/uncovered in the_text_report_says_which_engine_was_named_and_whether_it_counted.
  • Backward compatibility is verified, not just claimed: diagnose() delegates to diagnose_with_engine(e, s, None) unchanged, provenance is #[serde(default, skip_serializing_if = "Option::is_none")], and a dedicated test pins that a report without --engine grows no provenance key at all.
  • --report's exclusion of the pasted symptom and provenance is independently verified by a_shared_report_never_carries_the_pasted_symptom_or_its_provenance, which plants a token, a path and a host in the paste and checks both the JSON and the mail link.
  • apps/rocm/src/symptom_input.rs (file/stdin reading, trailing-newline stripping, BOM/UTF-16 decoding established in #1048) is reused unchanged rather than duplicated for the --engine path — no divergence from established convention.
  • clap wiring for --engine (an ArgGroup-based requires rather than a plain single-arg requires, with the reasoning given in a code comment) is precise and directly unit-tested for all three relevant paths: valid value, an engine the catalog doesn't know (InvalidValue, message lists accepted values), and --engine with no symptom source (MissingRequiredArgument).
  • The new Gherkin scenarios (diagnose-65..71) read well: behavioral language, one behavior per scenario, and message+state combined in the same assertion where it matters (the refusal scenario checks both the error and that nothing was diagnosed; the --report privacy scenario checks the report/refusal, "nothing sent," and "nothing pasted appears" together). diagnose-70/71 form a deliberate Linux/Windows contrast pair for the new text-report line.
  • AGENTS.md compliance is clean across the board: docs sync, module-size limits, and "new user-facing message asserted together with resulting state" are all satisfied, including for the new tri-state read_here (Some(true)/Some(false)/None) matrix.

Deployment notes

provenance is additive/optional on rocm diagnose --json; no version bump, no migration, and old consumers are unaffected (verified, not just asserted). At review time, every CI check that had reported was green (clippy, Linux/Windows build-and-test, both security-scan suites, CodeQL, coverage ratchet, commit-signatures, affected-crates tests, docs build) except the aggregate "E2E tests" check, which was still in_progress; confirm it finishes green before merging.

What this covered

Full diff between merge-base 3e00084c and PR head 37e10855 (19 files, +1210/-77). Code passes: core diagnose engine/catalog group (independent worker, full read); CLI layer group (cli.rs, cli_args.rs, diagnose_cmd.rs, main.rs, remote/doctor.rs — a dedicated independent worker confirmed its reads were sound, but its final itemized findings did not arrive before this review was finalized under time pressure, so the driver performed a direct supplementary read of this group's full diff and traced the --engine/--symptom-file dispatch path itself); e2e cucumber tests group (an independent worker confirmed its reads were sound but its final itemized findings likewise did not arrive in time; the driver read the new scenarios directly as a supplementary check); docs group (independent worker, full pass). Source-reading passes: agent-instruction adherence (independent worker, full AGENTS.md read, no violations ≥70 found); history/prior-changes (driver performed this directly — catalog.rs/engine.rs were extracted fresh in #890, diagnose.rs's prior --symptom-file work in #1048 was checked for convention consistency and found consistent; a dedicated independent worker was also launched for this pass but did not complete before this review was finalized).

Mid-review hazard: the shared working directory used for most of this review (/home/agent/workspace/repo) was found to be concurrently mutated by other unrelated processes partway through (other branches appeared, HEAD moved). Every finding above was verified against the PR head via git show <exact 40-char SHA>:<path> in a separate, isolated clone, not against the contended working tree; nothing here rests on unverified content.

Did not run: no cargo test/cargo nextest/cargo clippy/cargo xtask e2e execution (CI's own results were read instead, see Deployment notes); the test-quality "would it pass reverted" / "per-branch mutation" checks were done by reading, not by executing a revert. Not reconciled against any prior review discussion on this PR — conducted blind per this method's rules.


🤖 Automated by an AI agent

/// it, and `#[serde(default)]` lets a payload written before it existed
/// still deserialize.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub provenance: Option<SymptomProvenance>,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: provenance.symptom can put non-ASCII user text on a redirectable stream, which contradicts this codebase's own ROCMAI-543 invariant.

crates/rocm-core/src/diagnose/catalog.rs's every_message_the_diagnose_catalog_can_emit_is_ascii test states the contract explicitly: "every string rocm diagnose can put on a redirectable stream has to be plain ASCII, or a Windows console on the OEM code page (437) renders UTF-8 punctuation as mojibake" — and it asserts this for both the rendered text report and the --json payload.

Before this PR, the only paste-derived content reaching output was evidence, built from a closed, static, ASCII keyword vocabulary (e.g. KEYWORDS_VLLM_OOM in check_16_vllm_oom) — never a raw copy of user input. provenance.symptom breaks that: it stores and serializes the user's full paste verbatim, byte-for-byte (confirmed by the PR's own the_report_records_the_named_engine_and_the_symptom_verbatim test, which uses control bytes and CRLF to prove no edits occur), with no ASCII filtering. The invariant test's corpus is never run through diagnose_with_engine, so this new path is unguarded: a user pasting a non-ASCII error (an accented path, a non-English log line, box-drawing characters from a progress bar) and naming --engine vllm would now get non-ASCII bytes in rocm diagnose --json, on a redirectable stream, which ROCMAI-543 exists specifically to prevent.

Confidence 85 (traced: confirmed evidence strings are keyword-literal-only pre-PR, confirmed provenance.symptom is the first raw-verbatim-to-JSON path, confirmed the invariant test's corpus doesn't exercise it).

Suggested fix: either extend the ROCMAI-543 corpus (catalog.rs) with a diagnose_with_engine case using a non-ASCII symptom, and decide/document that provenance.symptom is a deliberate, narrow exception to the invariant (with a reason) — or constrain/escape the field so the invariant actually holds end-to-end.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bab5fb9. The paste is still carried exactly, but rocm diagnose --json now writes every non-ASCII character as a \uXXXX escape, so the output is plain ASCII and any JSON parser reads back the identical text. The ROCMAI-543 corpus test now runs --engine rows with a non-ASCII paste, a round-trip test compares the parsed symptom byte for byte, and scenario diagnose-75 checks the bytes the real binary prints. Details in the general comment.

impl SymptomEngine {
/// The name `--engine` takes and the JSON report records.
#[must_use]
pub const fn name(self) -> &'static str {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: SymptomEngine's CLI-printed name and its JSON wire name are two independent, never-cross-checked representations.

name() (here) is hand-written and used only by report.rs's text-report line; the JSON wire name comes separately from #[serde(rename_all = "lowercase")] on the enum. With a single variant (Vllm) today both happen to read "vllm", so nothing can yet disagree — but nothing asserts they must stay in sync.

Confidence 70 (reasoned from the code; can't yet manifest with one variant). Suggested fix: a small test asserting name() matches the serde-serialized value for every variant, so adding a second engine can't silently desync the printed line from the JSON contract.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bab5fb9: every_engine_name_is_its_serialized_name checks name() against the serde value for every variant, and an exhaustive match makes a new variant fail to compile until it is listed.

@volen-silo
volen-silo force-pushed the rocmai-955-diagnose-engine branch from 37e1085 to bab5fb9 Compare November 16, 2026 14:22
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Round 5 is in bab5fb9, rebased onto main after #1053 (the scenarios are now 68 to 75).

  • provenance.symptom and the ASCII rule (blocking): confirmed and fixed. The ROCMAI-543 corpus test only ran diagnose(), so the paste, which provenance.symptom copies verbatim, was never covered, and the CLI printed it with plain serde_json. rocm diagnose --json now writes every non-ASCII character as a \uXXXX escape (UTF-16 units, so astral characters become surrogate pairs). In valid JSON non-ASCII can only occur inside strings, so this is exact: any JSON parser recovers the identical paste, and a document with nothing to escape is byte-identical to before. The eval-fixture recorder uses the same function. This is the only place a DiagnoseReport reaches a stream; --report prints its own fixed struct, which never carries the paste.
    • Tests: the corpus test gains --engine rows for Linux, WSL and Windows with an accented path, a box-drawing character and an emoji, and checks the text report and the JSON are both ASCII. a_non_ascii_paste_round_trips_through_the_json_exactly parses the output back and compares the symptom byte for byte. Dropping the escaping fails both. Scenario diagnose-75 runs the real binary with --symptom-file ... --engine vllm --json and checks the bytes printed are ASCII and the decoded symptom equals the paste; with the CLI's escaping hook removed it fails.
    • Other fields: the paste cannot reach any other field. Host-derived evidence lines can quote an install path, so a non-ASCII install path is now escaped in --json, but the text report still prints it raw. That predates this change and is left alone, and the description says so.
  • name() and the serde name (non-blocking): added every_engine_name_is_its_serialized_name, with an exhaustive match so a new variant does not compile until it is listed.
  • skills/rocm-doctor still writing vllm: and rocm remote doctor --engine: unchanged, as before; both stay listed under Scope as follow-ups, the skill change waiting on fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029.

The description is updated, including the wire-form wording of "exactly as given".

@volen-silo
volen-silo dismissed juhovainio’s stale review November 16, 2026 14:23

Addressed in bab5fb9: provenance.symptom is now written ASCII-escaped in --json (exact round trip), the ROCMAI-543 corpus covers --engine, name()/serde test added. Replies: #1052 (comment) and the two inline threads.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on this. The flag itself matches the ticket item and the docs and scenarios are in good shape, but I'd like these changed before merge:

  1. The text line overclaims. "every non-blank line ... was matched as vllm's" prints whenever the engine is named on a host that reads it, even when nothing for vLLM scored (an unrelated paste, or an empty --symptom). "Matched" reads as "a match was found". Please reword it to say the lines were read as vllm's, and add a row for "engine named, nothing for vLLM scored" to the state matrix (unit test and a scenario), so the line is asserted together with that outcome.
  2. render_report_json panics via expect while the only caller sits behind JsonOrText::render_json, which already returns a Result. Please return the error through it instead of discarding that channel.
  3. Scope. ASCII-escaping all rocm diagnose --json output (plus the new render_json hook and the eval_fixtures edit) is a separable behavior change that the ticket item doesn't mention. It's disclosed and follows from echoing the paste verbatim, so I'm fine with it if you keep it, but please either split it into its own PR or state in the PR text why it has to land together. Note it's only partly enforced: the text report and other --json commands still emit raw non-ASCII.

CI was still running on build-and-test, the Windows jobs and coverage when I looked; nothing had failed. I didn't build or run the tests myself, so I couldn't re-run the mutation checks described in the PR body.

let provenance = report.provenance.as_ref()?;
let engine = provenance.engine.name();
Some(match provenance.read_here {
Some(true) => format!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"was matched as {engine}'s" reads as a match was found, but this prints even when no vLLM entry scored (unrelated or empty symptom). Please say "was read as {engine}'s" and add a test row for an engine named with a non-vLLM symptom.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a0e8914: the line now says the lines were read as the engine's, there is a test row and scenario diagnose-76 for an engine named with nothing scored, and the line is asserted together with the absence of the vLLM cause. See the general comment.

Comment thread crates/rocm-core/src/diagnose/report.rs Outdated
/// and maps with string keys, which always serialize.
#[must_use]
pub fn render_report_json(report: &DiagnoseReport) -> String {
let pretty = serde_json::to_string_pretty(report).expect("a DiagnoseReport serializes");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This panics via expect, but its caller is a trait method that returns serde_json::Result. Please return a Result from here so the error channel JsonOrText::render_json provides is used.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a0e8914: render_report_json returns serde_json::Result<String>, propagated through JsonOrText::render_json, DiagnoseOutput and the eval-fixture recorder, with no expect left on that path and a test that a failing hook surfaces an error and writes nothing.

// (`provenance.symptom`) and host paths in evidence, and a Windows console
// on the OEM code page shows raw UTF-8 as mojibake (ROCMAI-543). The value
// is unchanged once the JSON is decoded.
fn render_json(&self) -> serde_json::Result<String> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This signature returns a Result, but the body wraps the render in Ok(..), so a serialization failure can't reach it (see the expect in render_report_json).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a0e8914: render_report_json returns serde_json::Result<String>, propagated through JsonOrText::render_json, DiagnoseOutput and the eval-fixture recorder, with no expect left on that path and a test that a failing hook surfaces an error and writes nothing.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at the current head (bab5fb98), which is the same commit the last round on this PR was already requesting changes on. No commits have landed since then, so I traced both of that round's open points against the code myself rather than taking them on faith, and both still hold:

  1. The new "Engine: ..." line overclaims when nothing actually scored. render_report_text calls named_engine_line unconditionally, before it checks whether anything matched. So --engine vllm on Linux/WSL with a paste that doesn't score against the vLLM entry still opens the report with "... every non-blank line of the symptom was matched as vllm's)." directly above "no known misconfiguration matched" — two contradictory claims in the same output. The only test for this line always uses the real vLLM OOM text, so it never exercises "named and read, but nothing scored." Inline comment with specifics.

  2. render_report_json still panics via .expect(...) instead of using the Result path already wired for it. Its only caller, JsonOrText::render_json, returns serde_json::Result<String> and the write path already propagates that with ?. The override here just wraps the panicking call in Ok(...), discarding the fallible channel that exists end-to-end. Low real-world risk ("never in practice" per the doc comment), but there's no reason to keep the panic next to a Result-returning caller. Inline comment with specifics.

Separately, worth a look before merging: as of this review, E2E tests (MI300X, current) and the self-hosted consolidated E2E report are failing on this exact commit, with the vLLM-installed torch reporting 0 GPU devices for the installed SDK. Nothing in this diff touches engine install or torch selection, so this reads like a runner/provisioning issue rather than something this change caused — but it wasn't red at the time of the previous review, so I'd confirm it's a known flake (or also fails on main) rather than assume that.

Everything else here looks solid: read_here is derived from the same catalog table and platform filter the checks use rather than a hand-maintained OS list, provenance is genuinely additive (skip_serializing_if, a dedicated test pins no key when --engine is absent), the --report/mail path is verified not to carry the pasted symptom or provenance, and the ASCII-escaping of provenance.symptom for the JSON wire form is exact (escaping happens after serialization, and every JSON escape sequence is itself ASCII) and covered by a round-trip test with real non-ASCII input.

🤖 Automated by an AI agent

let engine = provenance.engine.name();
Some(match provenance.read_here {
Some(true) => format!(
"Engine: {engine} (named with --engine; every non-blank line of the symptom was \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is unconditional once read_here == Some(true): render_report_text calls named_engine_line first and only afterwards checks whether anything actually matched (the "no known misconfiguration matched" branch a few lines below). So --engine vllm on Linux/WSL with a paste that doesn't score against the vLLM entry at all still prints "... every non-blank line of the symptom was matched as vllm's)." directly above "no known misconfiguration matched" — a contradictory pair of claims in the same report.

the_text_report_says_which_engine_was_named_and_whether_it_counted (engine.rs) only varies the host OS; every case uses the real vLLM OOM text, so a "named, read, but nothing scored" row is never exercised.

Suggest rewording to something like "was read as {engine}'s" (describes what the flag did, not that a match resulted), and adding that state-matrix row with a non-matching paste.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a0e8914: the line now says the lines were read as the engine's, there is a test row and scenario diagnose-76 for an engine named with nothing scored, and the line is asserted together with the absence of the vLLM cause. See the general comment.

Comment thread crates/rocm-core/src/diagnose/report.rs Outdated
/// and maps with string keys, which always serialize.
#[must_use]
pub fn render_report_json(report: &DiagnoseReport) -> String {
let pretty = serde_json::to_string_pretty(report).expect("a DiagnoseReport serializes");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still panics via .expect("a DiagnoseReport serializes"), but the only caller no longer needs it to: JsonOrText::render_json (apps/rocm/src/cli_report.rs) returns serde_json::Result<String> and write_json_or_text already propagates it with .context(...)?. DiagnoseOutput's override (apps/rocm/src/diagnose_cmd.rs) just wraps this panicking call in Ok(...) instead of using the Result channel that's already wired end-to-end.

The doc comment's "never in practice" is accurate today, but since the fallible path already exists all the way up, it seems better to return the serialization error through it (e.g. have this function return serde_json::Result<String>) rather than keep a panic alongside a Result-returning caller.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a0e8914: render_report_json returns serde_json::Result<String>, propagated through JsonOrText::render_json, DiagnoseOutput and the eval-fixture recorder, with no expect left on that path and a test that a failing hook surfaces an error and writes nothing.

@volen-silo
volen-silo force-pushed the rocmai-955-diagnose-engine branch from bab5fb9 to a0e8914 Compare November 17, 2026 08:17
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Round 6 is in a0e8914, rebased onto main (9101286a).

  1. The "Engine:" line overclaimed: it now says every non-blank line of the symptom was read as the engine's, not that a match was found. I added the missing state-matrix row, an engine named on a host that reads it with an unrelated paste and with an empty one, asserting the line and that no vLLM entry is in the report together, and scenario diagnose-76 (Linux and WSL) against the real binary with a plain Python traceback. Making the line claim a match again, or making check_16 score an unrelated paste when an engine is named, fails the unit test and diagnose-76. Three "matched as" phrases remain on purpose, in the --engine help line, the docs/testing.md intro and the diagnose-68 name, because they describe a vLLM error actually matching.
  2. render_report_json panicked via expect: it now returns serde_json::Result<String>, propagated through JsonOrText::render_json, DiagnoseOutput (no Ok(..) wrap left) and the eval-fixture recorder. No panic remains on that path. A test checks that a failing render_json hook surfaces an error and writes nothing.
  3. Scope of the ASCII escaping: kept in this change, and the description now says why. Echoing the paste verbatim in provenance.symptom is what adds the first raw user text to the --json output, so shipping --engine without the escaping would break the ROCMAI-543 rule on the day it lands. It is only enforced for this field: host-derived text in the text report and other --json commands can still print non-ASCII, which the description also says.
  4. E2E tests (MI300X, current) red on bab5fb98: the log shows the vLLM runtime install ending with device_check: no_devices (torch 2.11.0+rocm7.14.1 imports but reports 0 devices), in the install path this change does not touch. The same lane failed in main's merge-queue run for 9101286a and on another PR's recent run, and passed on other runs, so it is intermittent on that runner. It is not a required check. I did not rerun it; it will run again on this head.

The description is updated with scenario 76 and the new results.

@volen-silo
volen-silo dismissed stale reviews from juhovainio and juhovainio November 17, 2026 08:20

Addressed in a0e8914: the Engine line says 'read as' with a nothing-scored test row and scenario 76, render_report_json returns a Result, ASCII-escaping scope explained in the PR text. Replies: #1052 (comment) and the inline threads.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change. Implements the --engine flag described by ROCMAI-955 ("name the engine a diagnose symptom came from with --engine"); the ticket itself is not retrievable from here (no tracker access available to this review), so it is judged only against the PR description, which matches the diff in substance and explicitly discloses skills/rocm-doctor/ and rocm remote doctor --symptom as deferred follow-ups rather than part of this change's scope — confirmed accurate: neither is touched by this diff.

Blocking

  • diagnose-71's "nothing the user pasted appears in what the CLI printed" check passes vacuously on the common no-GPU CI lane — tests/e2e-cucumber/tests/e2e/diagnose_steps.rs:2860-2872 (assert_paste_absent_from_report), backing tests/e2e-cucumber/features/diagnose.feature:1042-1047 (scenario diagnose-71)
    This step unconditionally sweeps world.cli_output for the pasted markers and the word "provenance", with no branch on whether --report actually produced a report or a refusal. On a lane with no AMD GPU (the common case), rocm diagnose --report hits one of the static ReportRefusal strings (apps/rocm/src/diagnose_cmd.rs), none of which ever interpolates the symptom text — so the assertion passes whether or not shared_report's paste-exclusion logic (the thing this scenario exists to protect) works at all. Two sibling steps in the exact same file explicitly guard against this: nothing_has_been_sent only asserts inside if out.contains("a report would carry"), and answer_names_nothing_identifying's own comment states the vacuous-pass risk verbatim ("on a lane whose hardware is not on the allowlist... every sweep would pass without a report ever having existed to sweep. Branch on the outcome...") and branches on cli_version accordingly. assert_paste_absent_from_report has neither guard.
    Confidence 85 · logic · Fix: branch the same way answer_names_nothing_identifying does (confirm a genuine report via cli_version before sweeping, and assert a recognizable refusal shape otherwise), or give diagnose-71 a Given step that forces the shown-report branch deterministically (as diagnose-40 does).

  • diagnose_json_is_the_ascii_rendering_of_the_report would still pass if the ASCII-escaping wiring it exists to pin were deleted — apps/rocm/src/diagnose_cmd.rs:702-709 (test), apps/rocm/src/diagnose_cmd.rs:159-161 (the DiagnoseOutput::render_json override being tested)
    The override is new in this diff and exists to route --json through rocm_core::render_diagnose_json, which ASCII-escapes non-ASCII bytes that a plain serde_json::to_string_pretty would not (ROCMAI-543). The test's fixture, host_report() = rocm_core::run_diagnose(&Examination::default(), ""), is all-ASCII: no symptom, no provenance, and nothing non-ASCII reachable from a default Examination anywhere in the catalog. For all-ASCII input, escape_non_ascii is the identity function, so rocm_core::render_diagnose_json(report) is byte-identical to the trait's default serde_json::to_string_pretty(self) (since DiagnoseOutput's Serialize impl just delegates to self.report). Deleting the override entirely — falling back to the default trait method — would leave both of this test's assertions passing. DiagnoseOutput is private to apps/rocm, so no rocm-core-side test (which does cover the escaping logic itself thoroughly) can cover this specific wiring; this is the only test that could, and it doesn't.
    Confidence 85 · logic · Fix: rebuild the fixture with a report carrying a non-ASCII provenance.symptom (or any other field escape_non_ascii would touch) and assert the rendered JSON actually differs from plain serde_json::to_string_pretty in the way escaping predicts, rather than asserting equality with rocm_core::render_diagnose_json's output on an input where both functions coincide.

Non-blocking

  • docs/architecture.md's diagnose_cmd.rs row is stale relative to this diff — docs/architecture.md:65
    This diff adds substantial new surface to diagnose_cmd.rs — the SymptomQuery struct threading --engine into diagnose(), the ASCII-safe render_json() override, and shared_report()'s provenance-stripping — none of which the row mentions. The same commit updated the equivalent rows for cli_args.rs, diagnose.rs, diagnose/catalog.rs and diagnose/report.rs to name their new items; diagnose_cmd.rs is the one comparably-affected file left as-is. cargo xtask check-architecture-doc only checks presence, not accuracy, so this does not fail CI, but AGENTS.md itself says "the check reads no prose, so keeping an entry's description true is still on you."
    Confidence 70 · mechanical · Fix: add a clause naming SymptomQuery, the ASCII render_json hook, and shared_report's provenance-stripping, matching the treatment the sibling rows got in this commit.

Decisions for the author

  • provenance.symptom duplicates the user's full pasted text, verbatim, into rocm diagnose --json — tradeoff
    This is clearly intentional — proving the paste was not silently edited is the entire point of --engine — and --report/--send is independently verified to exclude both the paste and provenance from anything that leaves the machine. But it is a new surface for a large or sensitive pasted log (paths, hostnames, tokens) to end up wherever a user redirects --json output, e.g. into a bug report they file themselves. Worth confirming this is the intended boundary (local --json may carry it, nothing that leaves the machine does) rather than an oversight.
  • Check::WithEngine's tagged engine field is not used to filter which engine value reaches a checker; each checker self-gates internally — non-blocking-improvement
    Safe today with exactly one such checker (check_16_vllm_oom) and one engine variant (Vllm), since there's nothing to misroute to. Worth tightening when a second engine or a second engine-reading checker is added, so a forgotten self-check in a new checker can't silently read the wrong engine's symptom rather than failing closed.

Positive signals

  • The ASCII-invariant regression test (every_message_the_diagnose_catalog_can_emit_is_ascii, extended in this diff) is driven off the CHECKERS table itself rather than a hand-maintained list of messages, and its own corpus now includes --engine rows with a real non-ASCII paste on Linux/WSL/Windows, with a self-guard (everything_seen.contains(...)) that fails if the corpus ever stops reaching those rows — a test that protects itself against silently rotting into a vacuous pass, the same failure mode the two blocking findings above found elsewhere.
  • the_text_report_says_which_engine_was_named_and_whether_it_counted enumerates and tests all three read_here states (Some(true)/Some(false)/None) across linux/wsl/windows/uncovered hosts, including the specific "engine named and read, but nothing scored" row (unrelated and empty paste) — present and correctly asserts the line reads "read as vllm's" without a vLLM cause appearing beside it.
  • a_shared_report_never_carries_the_pasted_symptom_or_its_provenance plants a distinctive token/path/host in the paste, confirms the paste genuinely influenced the diagnosis first (so the test isn't vacuously passing because nothing matched), and then checks both the JSON report and the mail link don't carry it.
  • reads_engine_here is derived from the same CHECKERS table and platform filter the checks themselves run from, rather than a second hand-maintained OS list, avoiding future drift between what actually reads the engine and what the text report claims does.
  • every_engine_name_is_its_serialized_name ties the text-report name and the JSON wire name to each other via an exhaustive match, so a second SymptomEngine variant cannot silently desync the two.

Deployment notes

provenance is additive and optional on rocm diagnose --json (#[serde(default, skip_serializing_if = "Option::is_none")]), and a dedicated test pins that a report built without --engine grows no provenance key at all, so existing JSON consumers (including the skill eval fixtures, confirmed to carry no engine field) are unaffected. No migration, no version bump needed.

What this covered

Full diff between 9101286acfe1d071b2d1ae43739f937593589ad7 (merge-base with origin/main, confirmed identical to origin/main's tip) and a0e8914e69e7c90a63d52368246f54f4d9567e77 (confirmed PR head), 22 files, +1668/-94, a single commit. Reviewed as four independent parallel passes grouped by coupling — core diagnose engine/catalog/report (crates/rocm-core/src/diagnose.rs, diagnose/{catalog,engine,host,report,scoring,test_support}.rs, lib.rs), the CLI layer (cli.rs, cli_args.rs, cli_report.rs, diagnose_cmd.rs, main.rs, remote/doctor.rs), the e2e cucumber tests (diagnose.feature, diagnose_steps.rs), and docs plus xtask/src/eval_fixtures.rs — each pass reading its group's full diff and files completely, plus an agent-instruction-adherence check against the repo's AGENTS.md (564 lines) folded into each pass. I independently re-verified both blocking findings and the non-blocking finding directly against the source myself before including them, rather than taking any worker's report at face value.

Mid-review hazard: the shared working directory used by the fan-out workers was being concurrently mutated by other, unrelated processes — its checked-out commit was observed to silently move to an unrelated branch/commit more than once during the review (this environment appears to run several concurrent automated reviewers sharing one checkout). Every worker caught this independently or was alerted to it, and re-verified its reads against the exact pinned commit SHAs (git show <sha>:<path> / git diff <sha> <sha>) rather than trusting the working tree; one worker explicitly retracted a near-miss finding produced against wrong, silently-swapped content before re-verifying. I did all of my own direct verification in a separate, isolated clone never shared with any other process. I'm not aware of any finding above that rests on unverified content, but the hazard itself is worth flagging as a real risk to any concurrent review run in this environment.

CI at this exact head is fully green: clippy, both build-and-test lanes (Linux/Windows), all E2E lanes including the self-hosted MI300X one, coverage ratchet, commit-signatures, CodeQL, both security-scan suites, and the Read the Docs build. The only non-success entries are "Skill checks (skillscope)" (skipped — the repo's own AGENTS.md states this is advisory, not yet a required check) and one Strix-Halo matrix entry (skipped, not applicable to this change).

Every blocking and non-blocking finding above is new relative to the prior review discussion visible on this PR (reached blind, then checked afterward only to avoid duplicating an already-settled point): previously-raised points from earlier rounds on this PR that I'm aware of (the cli_args.rs/SUPPORTED_ENGINES doc staleness, the text-report line overclaiming a match, render_report_json panicking via .expect, the ASCII-invariant corpus not reaching diagnose_with_engine, the engine-name/serde-name cross-check) are all independently confirmed fixed at this exact head by direct reading of the current source, not by trusting that discussion. I'm not aware of any human reviewer holding an unresolved change-request review at this exact commit.

Did not run: no cargo test/cargo clippy/cargo xtask e2e execution (CI's own results were read instead); no fan-out worker attempted a build given the shared-checkout contention risk described above, so the two blocking test-vacuity findings are argued from reading the code and its exact behavior under a hypothetical revert, not from actually reverting and running it — scored at 85 (traced, not executed) accordingly rather than 100. Step 7 (design review) was folded into the fan-out passes and this consolidation rather than run as a separate set of spawned workers, given the depth of design-level tracing each pass already did; the two "Decisions for the author" above are its output. Not reconciled against standalone PR issue comments outside full reviews.


🤖 Automated by an AI agent

Comment on lines +2860 to +2872
#[then("nothing the user pasted appears in what the CLI printed")]
async fn assert_paste_absent_from_report(world: &mut E2eWorld) {
let out = world.cli_output.clone().expect("no CLI output");
for detail in PASTED_MACHINE_DETAILS {
assert!(
!out.contains(detail),
"{detail:?} from the user's paste reached what a report would carry:\n{out}"
);
}
assert!(
!out.contains("provenance"),
"the diagnosis's provenance reached what a report would carry:\n{out}"
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: assert_paste_absent_from_report (backing diagnose-71) passes vacuously on the common no-GPU CI lane.

This step unconditionally sweeps world.cli_output for the pasted markers and "provenance", with no branch on whether --report produced a genuine report or a refusal. On a lane with no AMD GPU (the common case), rocm diagnose --report hits one of the static ReportRefusal strings in apps/rocm/src/diagnose_cmd.rs, none of which ever interpolates the symptom text — so this assertion passes whether or not shared_report's paste-exclusion logic (the thing diagnose-71 exists to protect) works at all.

Two sibling steps in this same file explicitly guard against exactly this failure mode: nothing_has_been_sent only asserts inside if out.contains("a report would carry"), and answer_names_nothing_identifying's own comment states the risk verbatim ("on a lane whose hardware is not on the allowlist... every sweep would pass without a report ever having existed to sweep. Branch on the outcome...") and branches on cli_version accordingly. This step has neither guard.

Confidence 85 · logic · Fix: branch the same way answer_names_nothing_identifying does (confirm a genuine report via cli_version before sweeping, assert a recognizable refusal shape otherwise), or give diagnose-71 a Given step that forces the shown-report branch deterministically (as diagnose-40 does).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d7798f2: diagnose-71 now opens with the same Given as diagnose-58 (a simulated Linux machine whose only GPU is a released one), so a real report is prepared on every Linux lane, and the sweep first requires that a real report was printed. It is now @requires-os:linux; the payload unit test covers Windows. Making shared_report leak the paste fails it in WSL. See the general comment.

Comment on lines +700 to +709
/// `a_failing_render_json_hook_is_an_error_and_writes_nothing`).
#[test]
fn diagnose_json_is_the_ascii_rendering_of_the_report() {
let report = host_report();
assert_eq!(
output(&report, false).render_json().ok(),
rocm_core::render_diagnose_json(&report).ok()
);
assert!(output(&report, false).render_json().is_ok());
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: diagnose_json_is_the_ascii_rendering_of_the_report would still pass if the ASCII-escaping wiring it exists to pin (the new DiagnoseOutput::render_json override at line ~159) were deleted.

The override is new in this diff and routes --json through rocm_core::render_diagnose_json, which ASCII-escapes non-ASCII bytes a plain serde_json::to_string_pretty would not (ROCMAI-543). This test's fixture, host_report() = rocm_core::run_diagnose(&Examination::default(), ""), is all-ASCII: no symptom, no provenance, nothing non-ASCII reachable from a default Examination. For all-ASCII input, escape_non_ascii is the identity function, so rocm_core::render_diagnose_json(report) is byte-identical to the trait's default serde_json::to_string_pretty(self) (since DiagnoseOutput's Serialize impl just delegates to self.report). Deleting the override entirely — falling back to the default trait method — would leave both of this test's assertions passing. DiagnoseOutput is private to apps/rocm, so no rocm-core-side test (which does thoroughly cover the escaping logic itself) can cover this specific wiring; this is the only test that could, and it doesn't.

Confidence 85 · logic · Fix: rebuild the fixture with a report carrying a non-ASCII provenance.symptom (or any field escape_non_ascii would touch) and assert the rendered JSON actually differs from plain serde_json::to_string_pretty in the way escaping predicts, rather than asserting equality with rocm_core::render_diagnose_json's output on an input where both functions coincide.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d7798f2: the fixture is now a report with a non-ASCII provenance.symptom (accented path, box-drawing character, emoji), and the test asserts the output is ASCII, differs from plain serde_json::to_string_pretty by exactly the predicted \uXXXX escapes, and parses back to the paste. Deleting the DiagnoseOutput::render_json override makes it fail.

Comment thread docs/architecture.md
| [`crypto.rs`](../crates/rocm-core/src/crypto.rs) | Generic RSASSA-PKCS#1 v1.5 over SHA-256 sign and verify, with no external `openssl` process. | domain |
| [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) | Root of `rocm diagnose`: the public result types, the score thresholds, `diagnose` itself (with `mark_symptom_contribution`, which sets each finding's `symptom_matched` by running the catalog again with no symptom and comparing), and the vLLM symptom classifiers that run the whole catalog. | domain |
| [`diagnose/catalog.rs`](../crates/rocm-core/src/diagnose/catalog.rs) | The checker table: which check runs on which platform family and in what order, and how the family is resolved from an `Examination`. | domain |
| [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) | Root of `rocm diagnose`: the public result types (with `SymptomEngine` and `SymptomProvenance` for `--engine`), the score thresholds, `diagnose` itself (with `mark_symptom_contribution`, which sets each finding's `symptom_matched` by running the catalog again with no symptom and comparing), `diagnose_with_engine`, which also takes the engine a symptom came from, and the vLLM symptom classifiers that run the whole catalog. | domain |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the diagnose_cmd.rs row (line 65, not touched by this diff and so not directly commentable) is stale relative to this diff, unlike this row.

This diff updated this diagnose.rs row (and diagnose/catalog.rs's and diagnose/report.rs's) to name the new SymptomEngine/SymptomProvenance/diagnose_with_engine items it added there. But diagnose_cmd.rs got comparable new surface in this same diff — the SymptomQuery struct threading --engine into diagnose(), the ASCII-safe render_json() override, and shared_report()'s provenance-stripping — and its row (line 65) still only describes prepare_cli_report/shared report build/show/deliver, none of the new items.

cargo xtask check-architecture-doc only checks presence, not accuracy, so this won't fail CI, but AGENTS.md itself says "the check reads no prose, so keeping an entry's description true is still on you."

Confidence 70 · mechanical · Fix: add a clause to the diagnose_cmd.rs row (line 65) naming SymptomQuery, the ASCII render_json hook, and shared_report's provenance-stripping, matching the treatment this row and its siblings got in this same commit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d7798f2: the diagnose_cmd.rs row now names SymptomQuery, the ASCII render_json override and shared_report building the payload without the paste or provenance.

@volen-silo
volen-silo force-pushed the rocmai-955-diagnose-engine branch from a0e8914 to d7798f2 Compare November 17, 2026 11:03
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Round 7 is in d7798f2, rebased onto main (c21942f5).

  1. diagnose-71 passed vacuously on a lane with no GPU (blocking): confirmed and fixed. The scenario now opens with the same Given as diagnose-58 (a simulated Linux machine whose only GPU is a released one), and its When step runs with the scenario environment, so a real report is prepared on every Linux lane, GPU or not. The planted root also replaces /dev/dxg and /proc/version, so WSL produces a real report too. The sweep step first requires that a real report was printed (a report would carry and cli_version) and fails if none was. Native Windows cannot be simulated this way, so the scenario is now @requires-os:linux; a_shared_report_never_carries_the_pasted_symptom_or_its_provenance still covers the payload on every platform. With shared_report changed to leak the paste, diagnose-71 fails in WSL on "nothing the user pasted appears in what the CLI printed".
  2. diagnose_json_is_the_ascii_rendering_of_the_report stayed green without the override (blocking): confirmed and fixed. The fixture is now a run_diagnose_with_engine report whose paste has an accented path, a box-drawing character and an emoji. The test asserts the rendered output is ASCII, differs from plain serde_json::to_string_pretty by exactly the predicted \uXXXX escapes, and parses back to the original paste. Deleting the DiagnoseOutput::render_json override makes it fail.
  3. docs/architecture.md diagnose_cmd.rs row (non-blocking): the row now names SymptomQuery, the ASCII render_json override and shared_report building the --report payload without the paste or provenance.

On the two decisions:

  • provenance.symptom in local --json: that is the intended boundary. The local --json output may carry the paste, since proving it was not edited is the point of --engine, and nothing that leaves the machine does: --report and its mail link are built from a fixed struct that never receives the diagnosis. The README and --help say so.
  • Check::WithEngine self-gating: left as is. With one engine and one reading checker there is nothing to misroute. I will tighten it when a second engine or a second reading checker is added.

The description is updated for diagnose-71 and the rebase. The Windows and WSL2 e2e runs in it are from the commit before this rebase, which conflicted only in docs/testing.md; the description says what was re-run after it.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 17, 2026 11:05

Addressed in d7798f2: diagnose-71 now runs against a real report on every Linux lane, the ASCII rendering test uses a non-ASCII fixture and fails without the override, architecture row updated; the two decisions are answered in the reply. Replies: #1052 (comment) and the inline threads.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found. Requesting changes for the items below, which I'd like addressed or answered before merge.

Non-blocking, but please address

  1. Check::WithEngine(engine, fn) is not tied to what the checker actually reads (check_16_vllm_oom hard-codes vLLM), so a future entry could declare one engine and read another, making the "read as X's" / "changed nothing" line wrong with no test failing. See inline comment.
  2. docs/testing.md and the commit message say output is unchanged without --engine, but every diagnose --json now escapes non-ASCII (including host-derived evidence). See inline comment.
  3. One commit-message paragraph is mis-wrapped and will land in the squash body; please reword it.
  4. The PR bundles three separable changes (the flag with provenance, the ASCII-JSON render_json hook, the text-report "Engine:" line). Splitting is optional, but worth considering for reviewability.

Checked and fine: docs (README, --help, testing, manual-testing, vllm, architecture) are updated; scenarios diagnose-68 to -76 assert both message and state; the leak-guard test is non-vacuous; ROCMAI-955 scope matches, with the extras disclosed in the PR body. Required CI is green; the MI300X lane is pending on the known-down runner pool.

/// Also the engine the user said the symptom came from, for an entry that
/// otherwise has to find the engine's name in the text. The first field is
/// the engine the entry reads; [`reads_engine_here`] answers from it.
WithEngine(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing ties this declared engine to what the function reads: check_16_vllm_oom hard-codes vLLM, and reads_engine_here trusts this field. A future entry that declares one engine and reads another would make the text line ("read as X's" / "changed nothing") wrong with no test failing. Consider a catalog test that every WithEngine(engine, f) changes its verdict for a canned paste when engine is passed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 076591e: every_engine_reading_entry_reads_the_engine_it_declares checks, for every engine-reading entry, every platform family it applies to and every engine variant, that an engine-free paste changes verdict when the declared engine is named and does not for any other, and it fails if it finds no such entry. A checker that ignores the engine fails it.

Comment thread docs/testing.md Outdated
it scores nothing either way). No other entry reads the engine. The JSON
diagnosis records the engine and the symptom, verbatim once decoded, under
`provenance`;
without `--engine` the field is absent and the output is unchanged, so the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the output is unchanged" is not strictly true: render_report_json now escapes non-ASCII in all diagnose --json output, including host-derived evidence (the PR body says so). Suggest "unchanged except that non-ASCII text is written as \uXXXX escapes".

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 076591e: docs/testing.md now says the output is unchanged except that non-ASCII text in --json is written as \uXXXX escapes on every run, host-derived evidence included, and the same correction is made in the code comments and the commit message.

A vLLM error is often an ordinary PyTorch line that does not say vLLM,
and the vLLM out-of-memory entry only scores lines that name vLLM, so a
symptom has to be edited (a `vllm: ` prefix) before it matches. The
rocm-doctor skill tells an agent to do that to the user's paste; it is
to move to the new flag in a later change.

`rocm diagnose --engine vllm` states where the symptom came from
instead, whether it was passed with `--symptom` or read with
`--symptom-file`. Every non-blank line of the symptom then counts as
vLLM's, as if `vllm` were written on each; no other entry reads the
engine. The JSON diagnosis records the engine and the symptom, verbatim
(from a file: as read, minus the one trailing line break), under a new
optional `provenance` field that is absent when no engine is named. The
flag takes only the engines the catalog tells apart (today `vllm`),
under the names `rocm serve --engine` uses, and needs a symptom from
either flag (a clap group over the two); anything else is a usage error.
Writing `vllm` into the symptom still works.

The paste can hold non-ASCII text, and `rocm diagnose --json` must stay
ASCII (ROCMAI-543): a Windows console on the OEM code page shows raw
UTF-8 as mojibake. The JSON is therefore written with every non-ASCII
character as a `\uXXXX` escape, on every run and for every field, host
paths quoted in evidence included. Decoded, every value is unchanged;
the text report never echoes the paste. Apart from that escaping and
the optional field, the output without `--engine` is unchanged, and the
skill eval replays, which are ASCII, stay as they are.

The text report opens with a line naming the engine and saying the
symptom was read as that engine's, which is how it was read, not a
claim that anything matched. Where no entry on the examined platform
reads it (native Windows has no vLLM entry), the line says the flag
changed nothing. The answer comes from the catalog table the checks run
from, where each engine-reading entry declares the engine it reads, and
is not serialized, so it adds nothing to the JSON.

Like the symptom flags, `--engine` is accepted with `--report` and
`--distro`. The `--report` document, the one printed for mailing, is
built from the examination and the entry the diagnosis established,
never from the paste, so neither the symptom nor `provenance` reaches
it or its mail link; a test on the real payload pins that.

Covered by rocm-core unit tests (same verdict as the prefixed form on
Linux and WSL2, every non-blank line anchored, nothing else changed,
provenance verbatim and round-tripping through the escaped JSON, the
ASCII corpus with a non-ASCII paste, the text line agreeing with what
the flag did, each engine-reading entry reading the engine it declares,
engine names matching their serialized form), a payload test on the
shared report, CLI tests (both symptom sources, the ASCII `--json`
rendering, a failing render hook surfacing as an error), and scenarios
diagnose-68 to -76.

ROCMAI-955

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
volen-silo force-pushed the rocmai-955-diagnose-engine branch from d7798f2 to 076591e Compare November 17, 2026 13:29
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Round 8 is in 076591e, on top of main (9e50eac5).

  1. Check::WithEngine not tied to what the checker reads: added every_engine_reading_entry_reads_the_engine_it_declares. It walks the catalog table and, for every engine-reading entry, every platform family that entry applies to and every SymptomEngine variant, uses a torch out-of-memory paste that names no engine. The verdict (score and evidence) must change when the declared engine is named, and must not change for any other variant. It fails if it finds no engine-reading entry. A checker that ignores the engine, or fix-16 registered as an ordinary entry, fails it. A mutation that tags a different engine cannot be written while the enum has one variant; the loop covers it as soon as a second exists.
  2. "Output unchanged" overstated: corrected. docs/testing.md (two sentences), three code comments and the commit message now say the output is unchanged except that non-ASCII text in --json is written as \uXXXX escapes on every run, host-derived evidence included. The README and --help never made the claim.
  3. Mis-wrapped commit-message paragraph: the message is rewritten and reflowed, with no line over 72 characters and no paragraph broken mid-sentence.
  4. Bundled changes: I am keeping them together. The ASCII-JSON hook exists because provenance.symptom echoes the paste verbatim, so shipping the flag without it would break the ROCMAI-543 rule for redirectable output on the day it lands, and the text-report line reads the same catalog data the flag adds. The description says this under Scope.

The description is updated with the new test and the rebased results.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 17, 2026 13:32

Addressed in 076591e: new catalog test ties each declared engine to what its checker reads, the unchanged-output claim is corrected, commit message reflowed; the PR stays unsplit for the reason given in the reply. Replies: #1052 (comment) and the two inline threads.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found at this head. The --engine provenance item from the ticket is met, docs/architecture/scenarios are updated, the help/README/docs claims match the code, and the text-report Engine line is derived from the same table the checks use.

Non-blocking:

  • Scope: rocm diagnose --json now escapes all non-ASCII as \uXXXX for every run, and the PR adds a JsonOrText::render_json hook and touches the eval-fixture recorder. It is disclosed and justified in the PR body, but separable if maintainers prefer a split.
  • Two small inline notes (docs readability, one test half that is vacuous with a single engine variant).

Not verified by me: consumers of DiagnoseReport outside the diff, and local build/tests. Several CI jobs (Linux/Windows build-and-test, coverage, CodeQL) were still pending at review time, so this is conditional on those going green.

Comment thread docs/testing.md
`--report` with the engine named and a paste naming a user, host and token, and
checks that a report was printed and that none of them, and no `provenance`,
appears in it or in the mail link, and that nothing was sent (native Windows
cannot be simulated this way; the payload unit test above covers it there); `diagnose-72` pipes the paste in with `--symptom-file -` and a final

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this sentence runs diagnose-71's description straight into diagnose-72 and on through diagnose-76 in one paragraph; consider a list so each scenario id reads on its own.

/// naming no engine) must change the entry's verdict when exactly the
/// declared engine is named, and must not change it for any other.
#[test]
fn every_engine_reading_entry_reads_the_engine_it_declares() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: with a single SymptomEngine variant the "does not change for any other engine" half of this test cannot fail today; it only starts guarding once a second engine exists. Fine to keep, just don't read it as proof of that half yet.

@volen-silo
volen-silo added this pull request to the merge queue Nov 17, 2026

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review of the whole diff against main (merge-base 9e50eac5, head 076591e1), run blind (no prior review discussion read until findings were settled), then reconciled against the PR's existing review history and live CI.

Scope reviewed: every changed file read in full — crates/rocm-core/src/diagnose.rs, diagnose/{catalog,engine,host,report,scoring,test_support}.rs, lib.rs (core matching/scoring/report-rendering); apps/rocm/src/{cli.rs,cli_args.rs,cli_report.rs,diagnose_cmd.rs,main.rs,remote/doctor.rs} (CLI wiring, --report/--send path); README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, docs/vllm.md, tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs, xtask/src/eval_fixtures.rs (docs/test completeness).

Verification of the previously-flagged risk areas, done independently rather than taken on faith:

  • Text-report "Engine: ..." line: gated strictly on report.provenance.is_some(), worded "read as X's" (not "matched"), and a dedicated test/scenario (diagnose-76) exercises the "named and read, but nothing scored" row together with the absence of the vLLM cause — no overclaiming path found.
  • render_report_json: returns serde_json::Result<String> end-to-end through JsonOrText::render_json and DiagnoseOutput; no .expect()/.unwrap()/panic found on this or any other path reachable from pasted --symptom text.
  • Non-ASCII/JSON escaping: escape_non_ascii walks the pretty-printed JSON char-by-char and is proven exact (round-trip, including an astral-plane emoji) by a_non_ascii_paste_round_trips_through_the_json_exactly; the ROCMAI-543 invariant test's corpus now includes --engine rows, closing the gap a prior round found.
  • --report/--send: shared_report is built only from the examination plus an established-entry id/flag, never from report.provenance or the raw symptom; a_shared_report_never_carries_the_pasted_symptom_or_its_provenance plants a token/host/path in the paste and proves it doesn't leak into the JSON or the mail link.
  • vLLM anchoring: --engine vllm is a strict, explicit opt-in that widens matching only for check_16_vllm_oom's own entry (proven equivalent to the old per-line vllm: prefix), not an inference heuristic — so there's no over-broad/unanchored matching surface introduced here.

All of the above check out at the current head; I did not find any of them regressed or incompletely fixed.

New (non-blocking) observations from this pass, posted inline where the diff allows an anchor:

  1. crates/rocm-core/src/diagnose.rs:95-107 — SymptomProvenance.symptom is the first rocm-core field to carry a user's raw paste verbatim into a serializable report; the --report/--send exclusion that keeps it from leaving the machine lives entirely in the CLI layer (diagnose_cmd.rs::shared_report), not as a structural guard in rocm-core itself. This is a disclosed, intentional, well-tested tradeoff (confirmed by the author in this PR's thread), not a defect — flagging only as a suggestion for discoverability (e.g. a DiagnoseReport::redacted() helper) for the next consumer of DiagnoseReport.
  2. crates/rocm-core/src/diagnose.rs:18-22 (module doc comment, not itself touched by this diff, so no inline anchor available) — stale relative to this PR's new public surface: doesn't mention diagnose_with_engine/SymptomEngine/SymptomProvenance. Nitpick.

Not independently verified by me: a local cargo test/cargo clippy run (no toolchain available in this environment) and any consumer of DiagnoseReport outside this diff's tree.

CI at head 076591e1: all 39 check runs completed, all success except CodeQL (neutral, standard for that job) and Skill checks (skillscope) (skipped, documented in this repo as advisory-only, not yet a required gate). No failing required check.

This matches the conclusion of the latest human reviewer's approval at this same head. No blocking findings from this independent pass.

🤖 Automated by an AI agent

Comment on lines +95 to +107
pub struct SymptomProvenance {
pub engine: SymptomEngine,
pub symptom: String,
/// Whether an entry that ran on the examined platform reads `engine`, that
/// is, whether naming it could change anything. The text report says so
/// when it could not (native Windows has no vLLM entry today), so the flag
/// is not silently ignored.
///
/// Not serialized, so it adds nothing to the JSON report; `None` for a report
/// read back from JSON, where it is not known.
#[serde(skip)]
pub read_here: Option<bool>,
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking/informational: SymptomProvenance.symptom is the first field in rocm-core that carries a user's raw pasted text verbatim into a serializable DiagnoseReport. I can see from the PR thread this is a deliberate, disclosed tradeoff (the whole point of --engine is proving the paste wasn't silently edited), and that the --report/--send exclusion is independently tested (a_shared_report_never_carries_the_pasted_symptom_or_its_provenance) — so I'm not re-litigating that decision. The one thing I'd still suggest: that exclusion currently lives entirely in the CLI layer (apps/rocm/src/diagnose_cmd.rs's shared_report), with nothing in rocm-core itself (e.g. a DiagnoseReport::redacted() helper dropping provenance) making "build the safe-to-share form" a one-line, discoverable operation co-located with the type that carries the risk. Worth considering for the next consumer of DiagnoseReport outside this diff, not blocking here.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants