Repository navigation
ROCMAI-955: name the engine a diagnose symptom came from with --engine - #1052
volen-silo wants to merge 1 commit into
Conversation
|
Merge note: #1048, #1051, #1052 and #1053 each add e2e scenarios numbered from diagnose-59 in |
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
584db61 to
c205556
Compare
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head c205556d
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change. Implements ROCMAI-955 ("name the engine a diagnose symptom came from with --engine"); the ticket itself was not retrievable from here (no tracker access available to this review), so it is judged only against the PR description, which matches the diff in substance.
Blocking
None
Non-blocking
docs/architecture.md'scli_args.rsrow is stale —docs/architecture.md(thecli_args.rsentry),apps/rocm/src/cli_args.rs:1-17
The PR adds a new clap enum,DiagnoseEngineArg, toapps/rocm/src/cli_args.rs, and even updates that file's own module doc comment to mention it ("DiagnoseEngineArg(diagnose --engine) is the exception: it was added here after the move rather than relocated."). The corresponding row indocs/architecture.mdstill enumerates only the pre-existing enums (Provider,DevicePolicyArg,FrameworkArg,InstallFormat,WatcherModeArg,TelemetryModeArg,SandboxToolArg,PermissionsModeArg) and isn't in this PR's diff, so it no longer fully describes the file.cargo xtask check-architecture-doconly fails on a missing entry, not a stale one, so this slips past CI. The PR's own description claims "docs/architecture.md describe the flag," which is true for thediagnose.rs/diagnose/catalog.rsrows it did touch, just not this one.
Confidence 85 · mechanical · Fix: addDiagnoseEngineArganddiagnose --engineto that row's enum/flag list.- Stale doc comment on
SUPPORTED_ENGINES—apps/rocm/src/cli.rs:25-27
SUPPORTED_ENGINES's comment says it's "used as the clapvalue_parserfor every user-facing--engineargument so the possible values stay in sync across the CLI surface." This PR adds a second user-facing--engineargument (diagnose --engine) that deliberately does not useSUPPORTED_ENGINES— it has its ownDiagnoseEngineArgenum listing onlyvllm, by design (the catalog only tellsvllmapart today). The comment predates this PR and is no longer accurate now that there are two independently-scoped--engineflags.
Confidence 75 · mechanical · Fix: reword to excludediagnose --engine, e.g. "...every--engineargument that serves or installs an engine (notdiagnose --engine, which is narrower)."
Decisions for the author
None
Positive signals
- The
Check::Text/Check::WithEnginewrapper enum incatalog.rslets one checker (check_16_vllm_oom) take the newengineparameter without touching the other 21 checkers' signatures or threading an unusedOptionthrough them. - Strong, layered test coverage: equivalence tests proving
--engine vllmproduces identical verdicts to the oldvllm:prefix convention (single- and multi-line), a "changes nothing but the vLLM entry" test that includes a Windows host where vLLM doesn't run, a byte-exact provenance test using control bytes and CRLF, a clap-level unit test for the parse/error-kind, and e2e scenarios (diagnose-59/60/61) exercising the same behavior through the real CLI binary. The e2e fixture for the "verbatim" claim is deliberately engineered (leading blank-ish line, trailing spaces) so the assertion isn't vacuous against a trim/rejoin bug. - The refusal path (
--engine lemonade) is asserted together with its effect (exit code 2 and that nothing was diagnosed), matching this repo's own "a message is asserted together with the resulting state" convention. provenanceis wired as genuinely additive (#[serde(default, skip_serializing_if = "Option::is_none")], field absent rather than null when no engine is named), so existing JSON consumers are unaffected.
Deployment notes
None — the new provenance field is additive/optional on the rocm diagnose --json report, and no existing command's default behavior changes.
What this covered
Reviewed the whole diff between f536619a08a3b50dd26aac5c21a3bb9f6bd266d2 and c205556dccd1ea04965f179d15c1255ab6ce996a (19 files, +617/-68) via four independent parallel passes grouped by coupling — core engine/catalog (crates/rocm-core/src/diagnose.rs, diagnose/catalog.rs, diagnose/engine.rs, diagnose/scoring.rs, diagnose/host.rs, diagnose/report.rs, lib.rs), the CLI layer (apps/rocm/src/cli.rs, cli_args.rs, diagnose_cmd.rs, main.rs, remote/doctor.rs), the e2e cucumber tests (tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs), and the docs (README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, docs/vllm.md) — plus my own direct read of the core and CLI diffs as a cross-check. Each pass read its files completely at the PR head plus surrounding context, and verified claims against the actual code rather than taking the PR description at face value until findings were settled.
CI at this head is fully green (clippy, both E2E consolidated reports, Linux/Windows build-and-test, coverage ratchet, commit-signatures, security scans); the only non-success entry is "Skill checks (skillscope)" showing skipped, which this repo's own docs mark as advisory, not a required gate.
Did not run: no separate "prior review comments" pass, since this PR has no prior review history to reconcile against; the 2 existing PR comments were not read, per this method's rule that review findings are reached blind from the code and discussion is not reconciled here.
juhovainio
left a comment
There was a problem hiding this comment.
Nothing here breaks the diagnosis itself, and the tests and scenarios for the new flag look solid. I'm requesting changes on one open question: what --report does with the pasted symptom now that it is stored.
To address before merge
provenance.symptom copies the user's raw paste into DiagnoseReport. The symptom was never stored in the report before, and the report is built through the same call the --report / --send path uses. If --symptom ... --engine vllm --report --send is accepted, an arbitrary pasted log (paths, hostnames, tokens) travels in a report meant to be shared. I couldn't tell from the diff what --report does here. Please either drop provenance on that path, refuse the combination, or state and test that the text is redacted. Right now nothing covers --engine together with --report or --distro.
Smaller notes, non-blocking
- On native Windows
--engine vllmis accepted but there is no vLLM entry, so the user gets a plain no-match. Text output never echoes the engine; only--jsonshowsprovenance. A one-line note in the text rendering would make that visible. - The doc on
vllm_anchored_linessays the paste is treated "exactly as" if each line started withvllm, but blank lines are dropped here while avllm:prefix would keep them. Harmless for scoring; "every non-blank line" would be more accurate. - The
rocm-doctorskill still tells agents to write thevllm:prefix, androcm remote doctor --symptomhas no--engine. Both are disclosed as follow-ups in the PR body, so fine to leave. - CI: the only failure is the MI300X E2E job (12 vLLM serve/bench/chat scenarios,
torchvision::nms does not exist); diagnose-59/60/61 pass there. It looks environmental, but worth confirming it also fails on main.
| route_when_no_match: route_when_no_match(e), | ||
| out_of_scope, | ||
| model: None, | ||
| provenance: engine.map(|engine| SymptomProvenance { |
There was a problem hiding this comment.
This stores the raw paste in the report. Is --symptom ... --engine ... --report [--send] accepted? If so, pasted logs (paths, hostnames, tokens) now travel in a report meant to be shared. Please confirm what --report does with it (redact, drop, or refuse) and add a test or doc line.
There was a problem hiding this comment.
Fixed in 7c2d1f3. --symptom ... --engine ... --report stays accepted, and the report is the fixed Report struct (schema, matrix, entry id, OS, distro, ROCm and engine versions, fix_offered). It never receives the diagnosis, so neither provenance nor the paste can reach it or the mail link. That is now stated in the README and --help and proved by a_shared_report_never_carries_the_pasted_symptom_or_its_provenance, which plants a token, a path and a host in the paste and fails if the symptom leaks into the entry or the matrix, plus scenario diagnose-65 against the real binary. --distro shows the diagnosis on the user's own screen, provenance included, and the docs say so.
| let mut report = rocm_core::run_diagnose(&examination, &symptom.unwrap_or_default()); | ||
| let (symptom, engine) = | ||
| symptom.map_or((String::new(), None), |query| (query.text, query.engine)); | ||
| let mut report = rocm_core::run_diagnose_with_engine(&examination, &symptom, engine); |
There was a problem hiding this comment.
The same call builds the report for --report / --distro, and no test covers --engine combined with them. Same question as the provenance comment in diagnose.rs.
There was a problem hiding this comment.
Covered in 7c2d1f3, see the reply on the provenance comment: diagnose-65 runs --engine with --report and asserts none of the paste, host or provenance appears and nothing is sent; the unit test covers the payload and the mail link.
c205556 to
7c2d1f3
Compare
|
The fixes for the changes-requested review are in 7c2d1f3, rebased onto main after #1067.
The description is updated, with the scenarios renumbered to 62 to 66. |
Addressed in 7c2d1f3: --report is built from a fixed struct that never receives the paste or provenance, now documented and tested (unit test + diagnose-65). Replies: #1052 (comment) and the two inline threads.
juhovainio
left a comment
There was a problem hiding this comment.
I went through this diff and found nothing that blocks merge on correctness. Requesting changes on the points below, which I'd like addressed or answered before this lands.
What I checked and found sound: --engine vllm only feeds check_16, and every non-blank line counts as anchored, which matches a per-line vllm: prefix (the unit test compares score and evidence on linux and wsl). provenance is additive and omitted when no engine is named. The --report document and mail link are built from the examination plus the established entry only, so neither the pasted text nor provenance reaches them (unit test plus diagnose-65). Scenarios diagnose-62..66 cover the new behaviour and assert the refusal with empty stdout.
To address:
- Text output never says which engine was named. On native Windows, where there is no vLLM entry,
--engine vllmis accepted and silently has no effect. A one-line note in text mode, or a warning on Windows, would stop a user-supplied flag from being ignored without a trace. rocm remote doctor --symptomhas no--engine, and the doctor skill still needs thevllm:prefix there. The PR body acknowledges this, but the ticket item is only partly delivered until the follow-up lands. Please link the follow-up so it is not lost.- The
cli_args.rsrow in docs/architecture.md is still taggedrelocationalthough the file now holds a new enum. - Two doc lines run longer than the text around them (inline comment on README.md; the last paragraph of manual-testing 6c is similar).
CI: nothing has failed. The E2E lanes, affected-crates tests and Windows debug test are still pending, so this reflects a diff read only. I did not re-run the claim that the new tests fail when their target is broken.
| - `--engine` names the engine the symptom came from (only `vllm` is | ||
| accepted; it needs `--symptom` or `--symptom-file`). An engine's own error | ||
| is often a line any program could print, such as a PyTorch out-of-memory | ||
| error, and the vLLM entry claims only lines that name vLLM. With `--engine vllm`, every non-blank line |
There was a problem hiding this comment.
Nit: this line is longer than its neighbours in the new bullet; re-wrap for consistency.
There was a problem hiding this comment.
Fixed in 37e1085: the new --engine bullet is re-wrapped to the width of its neighbours, and the same fix is applied to the long paragraph and command in manual-testing 6c.
7c2d1f3 to
37e1085
Compare
|
Round 4 is in 37e1085, rebased onto main after #1051 and #1093 (scenarios renumbered to 65 to 71).
The description is updated with the new scenarios and the rebased results. |
Addressed in 37e1085: the text report now says which engine was named and whether it counted on this OS, architecture row explained, long lines re-wrapped; remote doctor --engine and the skill prefix stay listed as follow-ups. Replies: #1052 (comment) and the README thread.
juhovainio
left a comment
There was a problem hiding this comment.
🤖 Automated review · head 37e10855
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change. Implements ROCMAI-955 ("name the engine a diagnose symptom came from with --engine"); the ticket itself was not retrievable from here (no tracker access available to this review), so it is judged only against the PR description, which matches the diff in substance (new --engine flag, scenarios diagnose-65..71, the additive provenance field, --report exclusion) and explicitly discloses the rocm-doctor skill and rocm remote doctor --symptom as deferred follow-ups rather than part of this change's scope.
Blocking
provenance.symptomcan put non-ASCII user text on a redirectable stream, contradicting this codebase's own ROCMAI-543 invariant —crates/rocm-core/src/diagnose.rs:93-99,230,crates/rocm-core/src/diagnose/catalog.rs:400-419
catalog.rs's dedicated regression test states the contract explicitly: "every stringrocm diagnosecan put on a redirectable stream has to be plain ASCII, or a Windows console on the OEM code page (437) renders UTF-8 punctuation as mojibake," and asserts it against both the rendered text report and the--jsonpayload. Before this PR, the only paste-derived content reaching output wasevidence, built from a closed, static, ASCII keyword vocabulary (e.g.KEYWORDS_VLLM_OOMincheck_16_vllm_oom) — never a raw copy of user input.provenance.symptomis the first path that serializes the user's full paste verbatim, byte-for-byte (confirmed by this PR's ownthe_report_records_the_named_engine_and_the_symptom_verbatimtest, which uses control bytes and CRLF to prove no edits occur), with no ASCII filtering — and the invariant test's corpus is never run throughdiagnose_with_engine, so this new surface is unguarded.
Confidence 85 · logic · Fix: extend the ROCMAI-543 corpus with adiagnose_with_enginecase using a non-ASCII symptom and documentprovenance.symptomas a deliberate, narrow exception (with a reason) if that's the intent — or constrain/escape the field so the invariant holds end-to-end.
Non-blocking
SymptomEngine's CLI-printed name and its JSON wire name are two independent, never-cross-checked representations —crates/rocm-core/src/diagnose.rs:79-83(name()) vs. the enum's#[serde(rename_all = "lowercase")]derive (line 70)
Both happen to read"vllm"today with a single variant, so nothing can yet disagree, but nothing asserts they must. Confidence 70 · architectural · Fix: a small test assertingname()matches the serialized value for every variant, so a second engine can't silently desync the two.
Decisions for the author
skills/rocm-doctor/SKILL.md/reference.mdstill instruct writing thevllm:prefix rather than--engine vllm— answered-by-intent
diagnose.rs's own doc comment onSymptomEnginesays: "the skill is to move to--enginein a later change", and the PR description's "Scope" section discloses this explicitly, tied to #1029. Would otherwise have been a non-blocking impact-analysis gap; confirm this sequencing is still the plan.rocm remote doctor --symptomhas no--engine— answered-by-intent
Same "Scope" section discloses this as a follow-up. Confirm it's tracked.
Positive signals
Check::WithEngine's tag cleanly separates "which catalog entries read the engine" from "which engine the user named for this invocation" —reads_engine_here/read_hereare derived structurally off the sameCHECKERStable and platform filter the checks themselves run from, not a hand-maintained OS list, and this is directly exercised across linux/wsl/windows/uncovered inthe_text_report_says_which_engine_was_named_and_whether_it_counted.- Backward compatibility is verified, not just claimed:
diagnose()delegates todiagnose_with_engine(e, s, None)unchanged,provenanceis#[serde(default, skip_serializing_if = "Option::is_none")], and a dedicated test pins that a report without--enginegrows noprovenancekey at all. --report's exclusion of the pasted symptom andprovenanceis independently verified bya_shared_report_never_carries_the_pasted_symptom_or_its_provenance, which plants a token, a path and a host in the paste and checks both the JSON and the mail link.apps/rocm/src/symptom_input.rs(file/stdin reading, trailing-newline stripping, BOM/UTF-16 decoding established in #1048) is reused unchanged rather than duplicated for the--enginepath — no divergence from established convention.- clap wiring for
--engine(anArgGroup-basedrequiresrather than a plain single-argrequires, with the reasoning given in a code comment) is precise and directly unit-tested for all three relevant paths: valid value, an engine the catalog doesn't know (InvalidValue, message lists accepted values), and--enginewith no symptom source (MissingRequiredArgument). - The new Gherkin scenarios (diagnose-65..71) read well: behavioral language, one behavior per scenario, and message+state combined in the same assertion where it matters (the refusal scenario checks both the error and that nothing was diagnosed; the
--reportprivacy scenario checks the report/refusal, "nothing sent," and "nothing pasted appears" together). diagnose-70/71 form a deliberate Linux/Windows contrast pair for the new text-report line. - AGENTS.md compliance is clean across the board: docs sync, module-size limits, and "new user-facing message asserted together with resulting state" are all satisfied, including for the new tri-state
read_here(Some(true)/Some(false)/None) matrix.
Deployment notes
provenance is additive/optional on rocm diagnose --json; no version bump, no migration, and old consumers are unaffected (verified, not just asserted). At review time, every CI check that had reported was green (clippy, Linux/Windows build-and-test, both security-scan suites, CodeQL, coverage ratchet, commit-signatures, affected-crates tests, docs build) except the aggregate "E2E tests" check, which was still in_progress; confirm it finishes green before merging.
What this covered
Full diff between merge-base 3e00084c and PR head 37e10855 (19 files, +1210/-77). Code passes: core diagnose engine/catalog group (independent worker, full read); CLI layer group (cli.rs, cli_args.rs, diagnose_cmd.rs, main.rs, remote/doctor.rs — a dedicated independent worker confirmed its reads were sound, but its final itemized findings did not arrive before this review was finalized under time pressure, so the driver performed a direct supplementary read of this group's full diff and traced the --engine/--symptom-file dispatch path itself); e2e cucumber tests group (an independent worker confirmed its reads were sound but its final itemized findings likewise did not arrive in time; the driver read the new scenarios directly as a supplementary check); docs group (independent worker, full pass). Source-reading passes: agent-instruction adherence (independent worker, full AGENTS.md read, no violations ≥70 found); history/prior-changes (driver performed this directly — catalog.rs/engine.rs were extracted fresh in #890, diagnose.rs's prior --symptom-file work in #1048 was checked for convention consistency and found consistent; a dedicated independent worker was also launched for this pass but did not complete before this review was finalized).
Mid-review hazard: the shared working directory used for most of this review (/home/agent/workspace/repo) was found to be concurrently mutated by other unrelated processes partway through (other branches appeared, HEAD moved). Every finding above was verified against the PR head via git show <exact 40-char SHA>:<path> in a separate, isolated clone, not against the contended working tree; nothing here rests on unverified content.
Did not run: no cargo test/cargo nextest/cargo clippy/cargo xtask e2e execution (CI's own results were read instead, see Deployment notes); the test-quality "would it pass reverted" / "per-branch mutation" checks were done by reading, not by executing a revert. Not reconciled against any prior review discussion on this PR — conducted blind per this method's rules.
🤖 Automated by an AI agent
| /// it, and `#[serde(default)]` lets a payload written before it existed | ||
| /// still deserialize. | ||
| #[serde(default, skip_serializing_if = "Option::is_none")] | ||
| pub provenance: Option<SymptomProvenance>, |
There was a problem hiding this comment.
Blocking: provenance.symptom can put non-ASCII user text on a redirectable stream, which contradicts this codebase's own ROCMAI-543 invariant.
crates/rocm-core/src/diagnose/catalog.rs's every_message_the_diagnose_catalog_can_emit_is_ascii test states the contract explicitly: "every string rocm diagnose can put on a redirectable stream has to be plain ASCII, or a Windows console on the OEM code page (437) renders UTF-8 punctuation as mojibake" — and it asserts this for both the rendered text report and the --json payload.
Before this PR, the only paste-derived content reaching output was evidence, built from a closed, static, ASCII keyword vocabulary (e.g. KEYWORDS_VLLM_OOM in check_16_vllm_oom) — never a raw copy of user input. provenance.symptom breaks that: it stores and serializes the user's full paste verbatim, byte-for-byte (confirmed by the PR's own the_report_records_the_named_engine_and_the_symptom_verbatim test, which uses control bytes and CRLF to prove no edits occur), with no ASCII filtering. The invariant test's corpus is never run through diagnose_with_engine, so this new path is unguarded: a user pasting a non-ASCII error (an accented path, a non-English log line, box-drawing characters from a progress bar) and naming --engine vllm would now get non-ASCII bytes in rocm diagnose --json, on a redirectable stream, which ROCMAI-543 exists specifically to prevent.
Confidence 85 (traced: confirmed evidence strings are keyword-literal-only pre-PR, confirmed provenance.symptom is the first raw-verbatim-to-JSON path, confirmed the invariant test's corpus doesn't exercise it).
Suggested fix: either extend the ROCMAI-543 corpus (catalog.rs) with a diagnose_with_engine case using a non-ASCII symptom, and decide/document that provenance.symptom is a deliberate, narrow exception to the invariant (with a reason) — or constrain/escape the field so the invariant actually holds end-to-end.
There was a problem hiding this comment.
Fixed in bab5fb9. The paste is still carried exactly, but rocm diagnose --json now writes every non-ASCII character as a \uXXXX escape, so the output is plain ASCII and any JSON parser reads back the identical text. The ROCMAI-543 corpus test now runs --engine rows with a non-ASCII paste, a round-trip test compares the parsed symptom byte for byte, and scenario diagnose-75 checks the bytes the real binary prints. Details in the general comment.
| impl SymptomEngine { | ||
| /// The name `--engine` takes and the JSON report records. | ||
| #[must_use] | ||
| pub const fn name(self) -> &'static str { |
There was a problem hiding this comment.
Non-blocking: SymptomEngine's CLI-printed name and its JSON wire name are two independent, never-cross-checked representations.
name() (here) is hand-written and used only by report.rs's text-report line; the JSON wire name comes separately from #[serde(rename_all = "lowercase")] on the enum. With a single variant (Vllm) today both happen to read "vllm", so nothing can yet disagree — but nothing asserts they must stay in sync.
Confidence 70 (reasoned from the code; can't yet manifest with one variant). Suggested fix: a small test asserting name() matches the serde-serialized value for every variant, so adding a second engine can't silently desync the printed line from the JSON contract.
There was a problem hiding this comment.
Fixed in bab5fb9: every_engine_name_is_its_serialized_name checks name() against the serde value for every variant, and an exhaustive match makes a new variant fail to compile until it is listed.
37e1085 to
bab5fb9
Compare
|
Round 5 is in bab5fb9, rebased onto main after #1053 (the scenarios are now 68 to 75).
The description is updated, including the wire-form wording of "exactly as given". |
Addressed in bab5fb9: provenance.symptom is now written ASCII-escaped in --json (exact round trip), the ROCMAI-543 corpus covers --engine, name()/serde test added. Replies: #1052 (comment) and the two inline threads.
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for the work on this. The flag itself matches the ticket item and the docs and scenarios are in good shape, but I'd like these changed before merge:
- The text line overclaims. "every non-blank line ... was matched as vllm's" prints whenever the engine is named on a host that reads it, even when nothing for vLLM scored (an unrelated paste, or an empty
--symptom). "Matched" reads as "a match was found". Please reword it to say the lines were read as vllm's, and add a row for "engine named, nothing for vLLM scored" to the state matrix (unit test and a scenario), so the line is asserted together with that outcome. render_report_jsonpanics viaexpectwhile the only caller sits behindJsonOrText::render_json, which already returns aResult. Please return the error through it instead of discarding that channel.- Scope. ASCII-escaping all
rocm diagnose --jsonoutput (plus the newrender_jsonhook and the eval_fixtures edit) is a separable behavior change that the ticket item doesn't mention. It's disclosed and follows from echoing the paste verbatim, so I'm fine with it if you keep it, but please either split it into its own PR or state in the PR text why it has to land together. Note it's only partly enforced: the text report and other--jsoncommands still emit raw non-ASCII.
CI was still running on build-and-test, the Windows jobs and coverage when I looked; nothing had failed. I didn't build or run the tests myself, so I couldn't re-run the mutation checks described in the PR body.
| let provenance = report.provenance.as_ref()?; | ||
| let engine = provenance.engine.name(); | ||
| Some(match provenance.read_here { | ||
| Some(true) => format!( |
There was a problem hiding this comment.
"was matched as {engine}'s" reads as a match was found, but this prints even when no vLLM entry scored (unrelated or empty symptom). Please say "was read as {engine}'s" and add a test row for an engine named with a non-vLLM symptom.
There was a problem hiding this comment.
Fixed in a0e8914: the line now says the lines were read as the engine's, there is a test row and scenario diagnose-76 for an engine named with nothing scored, and the line is asserted together with the absence of the vLLM cause. See the general comment.
| /// and maps with string keys, which always serialize. | ||
| #[must_use] | ||
| pub fn render_report_json(report: &DiagnoseReport) -> String { | ||
| let pretty = serde_json::to_string_pretty(report).expect("a DiagnoseReport serializes"); |
There was a problem hiding this comment.
This panics via expect, but its caller is a trait method that returns serde_json::Result. Please return a Result from here so the error channel JsonOrText::render_json provides is used.
There was a problem hiding this comment.
Fixed in a0e8914: render_report_json returns serde_json::Result<String>, propagated through JsonOrText::render_json, DiagnoseOutput and the eval-fixture recorder, with no expect left on that path and a test that a failing hook surfaces an error and writes nothing.
| // (`provenance.symptom`) and host paths in evidence, and a Windows console | ||
| // on the OEM code page shows raw UTF-8 as mojibake (ROCMAI-543). The value | ||
| // is unchanged once the JSON is decoded. | ||
| fn render_json(&self) -> serde_json::Result<String> { |
There was a problem hiding this comment.
This signature returns a Result, but the body wraps the render in Ok(..), so a serialization failure can't reach it (see the expect in render_report_json).
There was a problem hiding this comment.
Fixed in a0e8914: render_report_json returns serde_json::Result<String>, propagated through JsonOrText::render_json, DiagnoseOutput and the eval-fixture recorder, with no expect left on that path and a test that a failing hook surfaces an error and writes nothing.
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed at the current head (bab5fb98), which is the same commit the last round on this PR was already requesting changes on. No commits have landed since then, so I traced both of that round's open points against the code myself rather than taking them on faith, and both still hold:
-
The new "Engine: ..." line overclaims when nothing actually scored.
render_report_textcallsnamed_engine_lineunconditionally, before it checks whether anything matched. So--engine vllmon Linux/WSL with a paste that doesn't score against the vLLM entry still opens the report with "... every non-blank line of the symptom was matched as vllm's)." directly above "no known misconfiguration matched" — two contradictory claims in the same output. The only test for this line always uses the real vLLM OOM text, so it never exercises "named and read, but nothing scored." Inline comment with specifics. -
render_report_jsonstill panics via.expect(...)instead of using the Result path already wired for it. Its only caller,JsonOrText::render_json, returnsserde_json::Result<String>and the write path already propagates that with?. The override here just wraps the panicking call inOk(...), discarding the fallible channel that exists end-to-end. Low real-world risk ("never in practice" per the doc comment), but there's no reason to keep the panic next to a Result-returning caller. Inline comment with specifics.
Separately, worth a look before merging: as of this review, E2E tests (MI300X, current) and the self-hosted consolidated E2E report are failing on this exact commit, with the vLLM-installed torch reporting 0 GPU devices for the installed SDK. Nothing in this diff touches engine install or torch selection, so this reads like a runner/provisioning issue rather than something this change caused — but it wasn't red at the time of the previous review, so I'd confirm it's a known flake (or also fails on main) rather than assume that.
Everything else here looks solid: read_here is derived from the same catalog table and platform filter the checks use rather than a hand-maintained OS list, provenance is genuinely additive (skip_serializing_if, a dedicated test pins no key when --engine is absent), the --report/mail path is verified not to carry the pasted symptom or provenance, and the ASCII-escaping of provenance.symptom for the JSON wire form is exact (escaping happens after serialization, and every JSON escape sequence is itself ASCII) and covered by a round-trip test with real non-ASCII input.
🤖 Automated by an AI agent
| let engine = provenance.engine.name(); | ||
| Some(match provenance.read_here { | ||
| Some(true) => format!( | ||
| "Engine: {engine} (named with --engine; every non-blank line of the symptom was \ |
There was a problem hiding this comment.
This line is unconditional once read_here == Some(true): render_report_text calls named_engine_line first and only afterwards checks whether anything actually matched (the "no known misconfiguration matched" branch a few lines below). So --engine vllm on Linux/WSL with a paste that doesn't score against the vLLM entry at all still prints "... every non-blank line of the symptom was matched as vllm's)." directly above "no known misconfiguration matched" — a contradictory pair of claims in the same report.
the_text_report_says_which_engine_was_named_and_whether_it_counted (engine.rs) only varies the host OS; every case uses the real vLLM OOM text, so a "named, read, but nothing scored" row is never exercised.
Suggest rewording to something like "was read as {engine}'s" (describes what the flag did, not that a match resulted), and adding that state-matrix row with a non-matching paste.
There was a problem hiding this comment.
Fixed in a0e8914: the line now says the lines were read as the engine's, there is a test row and scenario diagnose-76 for an engine named with nothing scored, and the line is asserted together with the absence of the vLLM cause. See the general comment.
| /// and maps with string keys, which always serialize. | ||
| #[must_use] | ||
| pub fn render_report_json(report: &DiagnoseReport) -> String { | ||
| let pretty = serde_json::to_string_pretty(report).expect("a DiagnoseReport serializes"); |
There was a problem hiding this comment.
This still panics via .expect("a DiagnoseReport serializes"), but the only caller no longer needs it to: JsonOrText::render_json (apps/rocm/src/cli_report.rs) returns serde_json::Result<String> and write_json_or_text already propagates it with .context(...)?. DiagnoseOutput's override (apps/rocm/src/diagnose_cmd.rs) just wraps this panicking call in Ok(...) instead of using the Result channel that's already wired end-to-end.
The doc comment's "never in practice" is accurate today, but since the fallible path already exists all the way up, it seems better to return the serialization error through it (e.g. have this function return serde_json::Result<String>) rather than keep a panic alongside a Result-returning caller.
There was a problem hiding this comment.
Fixed in a0e8914: render_report_json returns serde_json::Result<String>, propagated through JsonOrText::render_json, DiagnoseOutput and the eval-fixture recorder, with no expect left on that path and a test that a failing hook surfaces an error and writes nothing.
bab5fb9 to
a0e8914
Compare
|
Round 6 is in a0e8914, rebased onto main (
The description is updated with scenario 76 and the new results. |
Addressed in a0e8914: the Engine line says 'read as' with a nothing-scored test row and scenario 76, render_report_json returns a Result, ASCII-escaping scope explained in the PR text. Replies: #1052 (comment) and the inline threads.
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change. Implements the --engine flag described by ROCMAI-955 ("name the engine a diagnose symptom came from with --engine"); the ticket itself is not retrievable from here (no tracker access available to this review), so it is judged only against the PR description, which matches the diff in substance and explicitly discloses skills/rocm-doctor/ and rocm remote doctor --symptom as deferred follow-ups rather than part of this change's scope — confirmed accurate: neither is touched by this diff.
Blocking
-
diagnose-71's "nothing the user pasted appears in what the CLI printed" check passes vacuously on the common no-GPU CI lane —tests/e2e-cucumber/tests/e2e/diagnose_steps.rs:2860-2872(assert_paste_absent_from_report), backingtests/e2e-cucumber/features/diagnose.feature:1042-1047(scenario diagnose-71)
This step unconditionally sweepsworld.cli_outputfor the pasted markers and the word"provenance", with no branch on whether--reportactually produced a report or a refusal. On a lane with no AMD GPU (the common case),rocm diagnose --reporthits one of the staticReportRefusalstrings (apps/rocm/src/diagnose_cmd.rs), none of which ever interpolates the symptom text — so the assertion passes whether or notshared_report's paste-exclusion logic (the thing this scenario exists to protect) works at all. Two sibling steps in the exact same file explicitly guard against this:nothing_has_been_sentonly asserts insideif out.contains("a report would carry"), andanswer_names_nothing_identifying's own comment states the vacuous-pass risk verbatim ("on a lane whose hardware is not on the allowlist... every sweep would pass without a report ever having existed to sweep. Branch on the outcome...") and branches oncli_versionaccordingly.assert_paste_absent_from_reporthas neither guard.
Confidence 85 · logic · Fix: branch the same wayanswer_names_nothing_identifyingdoes (confirm a genuine report viacli_versionbefore sweeping, and assert a recognizable refusal shape otherwise), or give diagnose-71 a Given step that forces the shown-report branch deterministically (as diagnose-40 does). -
diagnose_json_is_the_ascii_rendering_of_the_reportwould still pass if the ASCII-escaping wiring it exists to pin were deleted —apps/rocm/src/diagnose_cmd.rs:702-709(test),apps/rocm/src/diagnose_cmd.rs:159-161(theDiagnoseOutput::render_jsonoverride being tested)
The override is new in this diff and exists to route--jsonthroughrocm_core::render_diagnose_json, which ASCII-escapes non-ASCII bytes that a plainserde_json::to_string_prettywould not (ROCMAI-543). The test's fixture,host_report()=rocm_core::run_diagnose(&Examination::default(), ""), is all-ASCII: no symptom, noprovenance, and nothing non-ASCII reachable from a defaultExaminationanywhere in the catalog. For all-ASCII input,escape_non_asciiis the identity function, sorocm_core::render_diagnose_json(report)is byte-identical to the trait's defaultserde_json::to_string_pretty(self)(sinceDiagnoseOutput'sSerializeimpl just delegates toself.report). Deleting the override entirely — falling back to the default trait method — would leave both of this test's assertions passing.DiagnoseOutputis private toapps/rocm, so norocm-core-side test (which does cover the escaping logic itself thoroughly) can cover this specific wiring; this is the only test that could, and it doesn't.
Confidence 85 · logic · Fix: rebuild the fixture with a report carrying a non-ASCIIprovenance.symptom(or any other fieldescape_non_asciiwould touch) and assert the rendered JSON actually differs from plainserde_json::to_string_prettyin the way escaping predicts, rather than asserting equality withrocm_core::render_diagnose_json's output on an input where both functions coincide.
Non-blocking
docs/architecture.md'sdiagnose_cmd.rsrow is stale relative to this diff —docs/architecture.md:65
This diff adds substantial new surface todiagnose_cmd.rs— theSymptomQuerystruct threading--engineintodiagnose(), the ASCII-saferender_json()override, andshared_report()'s provenance-stripping — none of which the row mentions. The same commit updated the equivalent rows forcli_args.rs,diagnose.rs,diagnose/catalog.rsanddiagnose/report.rsto name their new items;diagnose_cmd.rsis the one comparably-affected file left as-is.cargo xtask check-architecture-doconly checks presence, not accuracy, so this does not fail CI, but AGENTS.md itself says "the check reads no prose, so keeping an entry's description true is still on you."
Confidence 70 · mechanical · Fix: add a clause namingSymptomQuery, the ASCIIrender_jsonhook, andshared_report's provenance-stripping, matching the treatment the sibling rows got in this commit.
Decisions for the author
provenance.symptomduplicates the user's full pasted text, verbatim, intorocm diagnose --json— tradeoff
This is clearly intentional — proving the paste was not silently edited is the entire point of--engine— and--report/--sendis independently verified to exclude both the paste andprovenancefrom anything that leaves the machine. But it is a new surface for a large or sensitive pasted log (paths, hostnames, tokens) to end up wherever a user redirects--jsonoutput, e.g. into a bug report they file themselves. Worth confirming this is the intended boundary (local--jsonmay carry it, nothing that leaves the machine does) rather than an oversight.Check::WithEngine's tagged engine field is not used to filter which engine value reaches a checker; each checker self-gates internally — non-blocking-improvement
Safe today with exactly one such checker (check_16_vllm_oom) and one engine variant (Vllm), since there's nothing to misroute to. Worth tightening when a second engine or a second engine-reading checker is added, so a forgotten self-check in a new checker can't silently read the wrong engine's symptom rather than failing closed.
Positive signals
- The ASCII-invariant regression test (
every_message_the_diagnose_catalog_can_emit_is_ascii, extended in this diff) is driven off theCHECKERStable itself rather than a hand-maintained list of messages, and its own corpus now includes--enginerows with a real non-ASCII paste on Linux/WSL/Windows, with a self-guard (everything_seen.contains(...)) that fails if the corpus ever stops reaching those rows — a test that protects itself against silently rotting into a vacuous pass, the same failure mode the two blocking findings above found elsewhere. the_text_report_says_which_engine_was_named_and_whether_it_countedenumerates and tests all threeread_herestates (Some(true)/Some(false)/None) across linux/wsl/windows/uncovered hosts, including the specific "engine named and read, but nothing scored" row (unrelated and empty paste) — present and correctly asserts the line reads "read as vllm's" without a vLLM cause appearing beside it.a_shared_report_never_carries_the_pasted_symptom_or_its_provenanceplants a distinctive token/path/host in the paste, confirms the paste genuinely influenced the diagnosis first (so the test isn't vacuously passing because nothing matched), and then checks both the JSON report and the mail link don't carry it.reads_engine_hereis derived from the sameCHECKERStable and platform filter the checks themselves run from, rather than a second hand-maintained OS list, avoiding future drift between what actually reads the engine and what the text report claims does.every_engine_name_is_its_serialized_nameties the text-report name and the JSON wire name to each other via an exhaustive match, so a secondSymptomEnginevariant cannot silently desync the two.
Deployment notes
provenance is additive and optional on rocm diagnose --json (#[serde(default, skip_serializing_if = "Option::is_none")]), and a dedicated test pins that a report built without --engine grows no provenance key at all, so existing JSON consumers (including the skill eval fixtures, confirmed to carry no engine field) are unaffected. No migration, no version bump needed.
What this covered
Full diff between 9101286acfe1d071b2d1ae43739f937593589ad7 (merge-base with origin/main, confirmed identical to origin/main's tip) and a0e8914e69e7c90a63d52368246f54f4d9567e77 (confirmed PR head), 22 files, +1668/-94, a single commit. Reviewed as four independent parallel passes grouped by coupling — core diagnose engine/catalog/report (crates/rocm-core/src/diagnose.rs, diagnose/{catalog,engine,host,report,scoring,test_support}.rs, lib.rs), the CLI layer (cli.rs, cli_args.rs, cli_report.rs, diagnose_cmd.rs, main.rs, remote/doctor.rs), the e2e cucumber tests (diagnose.feature, diagnose_steps.rs), and docs plus xtask/src/eval_fixtures.rs — each pass reading its group's full diff and files completely, plus an agent-instruction-adherence check against the repo's AGENTS.md (564 lines) folded into each pass. I independently re-verified both blocking findings and the non-blocking finding directly against the source myself before including them, rather than taking any worker's report at face value.
Mid-review hazard: the shared working directory used by the fan-out workers was being concurrently mutated by other, unrelated processes — its checked-out commit was observed to silently move to an unrelated branch/commit more than once during the review (this environment appears to run several concurrent automated reviewers sharing one checkout). Every worker caught this independently or was alerted to it, and re-verified its reads against the exact pinned commit SHAs (git show <sha>:<path> / git diff <sha> <sha>) rather than trusting the working tree; one worker explicitly retracted a near-miss finding produced against wrong, silently-swapped content before re-verifying. I did all of my own direct verification in a separate, isolated clone never shared with any other process. I'm not aware of any finding above that rests on unverified content, but the hazard itself is worth flagging as a real risk to any concurrent review run in this environment.
CI at this exact head is fully green: clippy, both build-and-test lanes (Linux/Windows), all E2E lanes including the self-hosted MI300X one, coverage ratchet, commit-signatures, CodeQL, both security-scan suites, and the Read the Docs build. The only non-success entries are "Skill checks (skillscope)" (skipped — the repo's own AGENTS.md states this is advisory, not yet a required check) and one Strix-Halo matrix entry (skipped, not applicable to this change).
Every blocking and non-blocking finding above is new relative to the prior review discussion visible on this PR (reached blind, then checked afterward only to avoid duplicating an already-settled point): previously-raised points from earlier rounds on this PR that I'm aware of (the cli_args.rs/SUPPORTED_ENGINES doc staleness, the text-report line overclaiming a match, render_report_json panicking via .expect, the ASCII-invariant corpus not reaching diagnose_with_engine, the engine-name/serde-name cross-check) are all independently confirmed fixed at this exact head by direct reading of the current source, not by trusting that discussion. I'm not aware of any human reviewer holding an unresolved change-request review at this exact commit.
Did not run: no cargo test/cargo clippy/cargo xtask e2e execution (CI's own results were read instead); no fan-out worker attempted a build given the shared-checkout contention risk described above, so the two blocking test-vacuity findings are argued from reading the code and its exact behavior under a hypothetical revert, not from actually reverting and running it — scored at 85 (traced, not executed) accordingly rather than 100. Step 7 (design review) was folded into the fan-out passes and this consolidation rather than run as a separate set of spawned workers, given the depth of design-level tracing each pass already did; the two "Decisions for the author" above are its output. Not reconciled against standalone PR issue comments outside full reviews.
🤖 Automated by an AI agent
| #[then("nothing the user pasted appears in what the CLI printed")] | ||
| async fn assert_paste_absent_from_report(world: &mut E2eWorld) { | ||
| let out = world.cli_output.clone().expect("no CLI output"); | ||
| for detail in PASTED_MACHINE_DETAILS { | ||
| assert!( | ||
| !out.contains(detail), | ||
| "{detail:?} from the user's paste reached what a report would carry:\n{out}" | ||
| ); | ||
| } | ||
| assert!( | ||
| !out.contains("provenance"), | ||
| "the diagnosis's provenance reached what a report would carry:\n{out}" | ||
| ); |
There was a problem hiding this comment.
Blocking: assert_paste_absent_from_report (backing diagnose-71) passes vacuously on the common no-GPU CI lane.
This step unconditionally sweeps world.cli_output for the pasted markers and "provenance", with no branch on whether --report produced a genuine report or a refusal. On a lane with no AMD GPU (the common case), rocm diagnose --report hits one of the static ReportRefusal strings in apps/rocm/src/diagnose_cmd.rs, none of which ever interpolates the symptom text — so this assertion passes whether or not shared_report's paste-exclusion logic (the thing diagnose-71 exists to protect) works at all.
Two sibling steps in this same file explicitly guard against exactly this failure mode: nothing_has_been_sent only asserts inside if out.contains("a report would carry"), and answer_names_nothing_identifying's own comment states the risk verbatim ("on a lane whose hardware is not on the allowlist... every sweep would pass without a report ever having existed to sweep. Branch on the outcome...") and branches on cli_version accordingly. This step has neither guard.
Confidence 85 · logic · Fix: branch the same way answer_names_nothing_identifying does (confirm a genuine report via cli_version before sweeping, assert a recognizable refusal shape otherwise), or give diagnose-71 a Given step that forces the shown-report branch deterministically (as diagnose-40 does).
There was a problem hiding this comment.
Fixed in d7798f2: diagnose-71 now opens with the same Given as diagnose-58 (a simulated Linux machine whose only GPU is a released one), so a real report is prepared on every Linux lane, and the sweep first requires that a real report was printed. It is now @requires-os:linux; the payload unit test covers Windows. Making shared_report leak the paste fails it in WSL. See the general comment.
| /// `a_failing_render_json_hook_is_an_error_and_writes_nothing`). | ||
| #[test] | ||
| fn diagnose_json_is_the_ascii_rendering_of_the_report() { | ||
| let report = host_report(); | ||
| assert_eq!( | ||
| output(&report, false).render_json().ok(), | ||
| rocm_core::render_diagnose_json(&report).ok() | ||
| ); | ||
| assert!(output(&report, false).render_json().is_ok()); | ||
| } |
There was a problem hiding this comment.
Blocking: diagnose_json_is_the_ascii_rendering_of_the_report would still pass if the ASCII-escaping wiring it exists to pin (the new DiagnoseOutput::render_json override at line ~159) were deleted.
The override is new in this diff and routes --json through rocm_core::render_diagnose_json, which ASCII-escapes non-ASCII bytes a plain serde_json::to_string_pretty would not (ROCMAI-543). This test's fixture, host_report() = rocm_core::run_diagnose(&Examination::default(), ""), is all-ASCII: no symptom, no provenance, nothing non-ASCII reachable from a default Examination. For all-ASCII input, escape_non_ascii is the identity function, so rocm_core::render_diagnose_json(report) is byte-identical to the trait's default serde_json::to_string_pretty(self) (since DiagnoseOutput's Serialize impl just delegates to self.report). Deleting the override entirely — falling back to the default trait method — would leave both of this test's assertions passing. DiagnoseOutput is private to apps/rocm, so no rocm-core-side test (which does thoroughly cover the escaping logic itself) can cover this specific wiring; this is the only test that could, and it doesn't.
Confidence 85 · logic · Fix: rebuild the fixture with a report carrying a non-ASCII provenance.symptom (or any field escape_non_ascii would touch) and assert the rendered JSON actually differs from plain serde_json::to_string_pretty in the way escaping predicts, rather than asserting equality with rocm_core::render_diagnose_json's output on an input where both functions coincide.
There was a problem hiding this comment.
Fixed in d7798f2: the fixture is now a report with a non-ASCII provenance.symptom (accented path, box-drawing character, emoji), and the test asserts the output is ASCII, differs from plain serde_json::to_string_pretty by exactly the predicted \uXXXX escapes, and parses back to the paste. Deleting the DiagnoseOutput::render_json override makes it fail.
| | [`crypto.rs`](../crates/rocm-core/src/crypto.rs) | Generic RSASSA-PKCS#1 v1.5 over SHA-256 sign and verify, with no external `openssl` process. | domain | | ||
| | [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) | Root of `rocm diagnose`: the public result types, the score thresholds, `diagnose` itself (with `mark_symptom_contribution`, which sets each finding's `symptom_matched` by running the catalog again with no symptom and comparing), and the vLLM symptom classifiers that run the whole catalog. | domain | | ||
| | [`diagnose/catalog.rs`](../crates/rocm-core/src/diagnose/catalog.rs) | The checker table: which check runs on which platform family and in what order, and how the family is resolved from an `Examination`. | domain | | ||
| | [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) | Root of `rocm diagnose`: the public result types (with `SymptomEngine` and `SymptomProvenance` for `--engine`), the score thresholds, `diagnose` itself (with `mark_symptom_contribution`, which sets each finding's `symptom_matched` by running the catalog again with no symptom and comparing), `diagnose_with_engine`, which also takes the engine a symptom came from, and the vLLM symptom classifiers that run the whole catalog. | domain | |
There was a problem hiding this comment.
Non-blocking: the diagnose_cmd.rs row (line 65, not touched by this diff and so not directly commentable) is stale relative to this diff, unlike this row.
This diff updated this diagnose.rs row (and diagnose/catalog.rs's and diagnose/report.rs's) to name the new SymptomEngine/SymptomProvenance/diagnose_with_engine items it added there. But diagnose_cmd.rs got comparable new surface in this same diff — the SymptomQuery struct threading --engine into diagnose(), the ASCII-safe render_json() override, and shared_report()'s provenance-stripping — and its row (line 65) still only describes prepare_cli_report/shared report build/show/deliver, none of the new items.
cargo xtask check-architecture-doc only checks presence, not accuracy, so this won't fail CI, but AGENTS.md itself says "the check reads no prose, so keeping an entry's description true is still on you."
Confidence 70 · mechanical · Fix: add a clause to the diagnose_cmd.rs row (line 65) naming SymptomQuery, the ASCII render_json hook, and shared_report's provenance-stripping, matching the treatment this row and its siblings got in this same commit.
There was a problem hiding this comment.
Fixed in d7798f2: the diagnose_cmd.rs row now names SymptomQuery, the ASCII render_json override and shared_report building the payload without the paste or provenance.
a0e8914 to
d7798f2
Compare
|
Round 7 is in d7798f2, rebased onto main (
On the two decisions:
The description is updated for diagnose-71 and the rebase. The Windows and WSL2 e2e runs in it are from the commit before this rebase, which conflicted only in |
Addressed in d7798f2: diagnose-71 now runs against a real report on every Linux lane, the ASCII rendering test uses a non-ASCII fixture and fails without the override, architecture row updated; the two decisions are answered in the reply. Replies: #1052 (comment) and the inline threads.
juhovainio
left a comment
There was a problem hiding this comment.
No blocking issues found. Requesting changes for the items below, which I'd like addressed or answered before merge.
Non-blocking, but please address
Check::WithEngine(engine, fn)is not tied to what the checker actually reads (check_16_vllm_oomhard-codes vLLM), so a future entry could declare one engine and read another, making the "read as X's" / "changed nothing" line wrong with no test failing. See inline comment.docs/testing.mdand the commit message say output is unchanged without--engine, but everydiagnose --jsonnow escapes non-ASCII (including host-derived evidence). See inline comment.- One commit-message paragraph is mis-wrapped and will land in the squash body; please reword it.
- The PR bundles three separable changes (the flag with provenance, the ASCII-JSON
render_jsonhook, the text-report "Engine:" line). Splitting is optional, but worth considering for reviewability.
Checked and fine: docs (README, --help, testing, manual-testing, vllm, architecture) are updated; scenarios diagnose-68 to -76 assert both message and state; the leak-guard test is non-vacuous; ROCMAI-955 scope matches, with the extras disclosed in the PR body. Required CI is green; the MI300X lane is pending on the known-down runner pool.
| /// Also the engine the user said the symptom came from, for an entry that | ||
| /// otherwise has to find the engine's name in the text. The first field is | ||
| /// the engine the entry reads; [`reads_engine_here`] answers from it. | ||
| WithEngine( |
There was a problem hiding this comment.
Nothing ties this declared engine to what the function reads: check_16_vllm_oom hard-codes vLLM, and reads_engine_here trusts this field. A future entry that declares one engine and reads another would make the text line ("read as X's" / "changed nothing") wrong with no test failing. Consider a catalog test that every WithEngine(engine, f) changes its verdict for a canned paste when engine is passed.
There was a problem hiding this comment.
Fixed in 076591e: every_engine_reading_entry_reads_the_engine_it_declares checks, for every engine-reading entry, every platform family it applies to and every engine variant, that an engine-free paste changes verdict when the declared engine is named and does not for any other, and it fails if it finds no such entry. A checker that ignores the engine fails it.
| it scores nothing either way). No other entry reads the engine. The JSON | ||
| diagnosis records the engine and the symptom, verbatim once decoded, under | ||
| `provenance`; | ||
| without `--engine` the field is absent and the output is unchanged, so the |
There was a problem hiding this comment.
"the output is unchanged" is not strictly true: render_report_json now escapes non-ASCII in all diagnose --json output, including host-derived evidence (the PR body says so). Suggest "unchanged except that non-ASCII text is written as \uXXXX escapes".
There was a problem hiding this comment.
Fixed in 076591e: docs/testing.md now says the output is unchanged except that non-ASCII text in --json is written as \uXXXX escapes on every run, host-derived evidence included, and the same correction is made in the code comments and the commit message.
A vLLM error is often an ordinary PyTorch line that does not say vLLM, and the vLLM out-of-memory entry only scores lines that name vLLM, so a symptom has to be edited (a `vllm: ` prefix) before it matches. The rocm-doctor skill tells an agent to do that to the user's paste; it is to move to the new flag in a later change. `rocm diagnose --engine vllm` states where the symptom came from instead, whether it was passed with `--symptom` or read with `--symptom-file`. Every non-blank line of the symptom then counts as vLLM's, as if `vllm` were written on each; no other entry reads the engine. The JSON diagnosis records the engine and the symptom, verbatim (from a file: as read, minus the one trailing line break), under a new optional `provenance` field that is absent when no engine is named. The flag takes only the engines the catalog tells apart (today `vllm`), under the names `rocm serve --engine` uses, and needs a symptom from either flag (a clap group over the two); anything else is a usage error. Writing `vllm` into the symptom still works. The paste can hold non-ASCII text, and `rocm diagnose --json` must stay ASCII (ROCMAI-543): a Windows console on the OEM code page shows raw UTF-8 as mojibake. The JSON is therefore written with every non-ASCII character as a `\uXXXX` escape, on every run and for every field, host paths quoted in evidence included. Decoded, every value is unchanged; the text report never echoes the paste. Apart from that escaping and the optional field, the output without `--engine` is unchanged, and the skill eval replays, which are ASCII, stay as they are. The text report opens with a line naming the engine and saying the symptom was read as that engine's, which is how it was read, not a claim that anything matched. Where no entry on the examined platform reads it (native Windows has no vLLM entry), the line says the flag changed nothing. The answer comes from the catalog table the checks run from, where each engine-reading entry declares the engine it reads, and is not serialized, so it adds nothing to the JSON. Like the symptom flags, `--engine` is accepted with `--report` and `--distro`. The `--report` document, the one printed for mailing, is built from the examination and the entry the diagnosis established, never from the paste, so neither the symptom nor `provenance` reaches it or its mail link; a test on the real payload pins that. Covered by rocm-core unit tests (same verdict as the prefixed form on Linux and WSL2, every non-blank line anchored, nothing else changed, provenance verbatim and round-tripping through the escaped JSON, the ASCII corpus with a non-ASCII paste, the text line agreeing with what the flag did, each engine-reading entry reading the engine it declares, engine names matching their serialized form), a payload test on the shared report, CLI tests (both symptom sources, the ASCII `--json` rendering, a failing render hook surfacing as an error), and scenarios diagnose-68 to -76. ROCMAI-955 Signed-off-by: Eugene Volen <eugene.volen@amd.com>
d7798f2 to
076591e
Compare
|
Round 8 is in 076591e, on top of main (
The description is updated with the new test and the rebased results. |
Addressed in 076591e: new catalog test ties each declared engine to what its checker reads, the unchanged-output claim is corrected, commit message reflowed; the PR stays unsplit for the reason given in the reply. Replies: #1052 (comment) and the two inline threads.
juhovainio
left a comment
There was a problem hiding this comment.
No blocking issues found at this head. The --engine provenance item from the ticket is met, docs/architecture/scenarios are updated, the help/README/docs claims match the code, and the text-report Engine line is derived from the same table the checks use.
Non-blocking:
- Scope:
rocm diagnose --jsonnow escapes all non-ASCII as\uXXXXfor every run, and the PR adds aJsonOrText::render_jsonhook and touches the eval-fixture recorder. It is disclosed and justified in the PR body, but separable if maintainers prefer a split. - Two small inline notes (docs readability, one test half that is vacuous with a single engine variant).
Not verified by me: consumers of DiagnoseReport outside the diff, and local build/tests. Several CI jobs (Linux/Windows build-and-test, coverage, CodeQL) were still pending at review time, so this is conditional on those going green.
| `--report` with the engine named and a paste naming a user, host and token, and | ||
| checks that a report was printed and that none of them, and no `provenance`, | ||
| appears in it or in the mail link, and that nothing was sent (native Windows | ||
| cannot be simulated this way; the payload unit test above covers it there); `diagnose-72` pipes the paste in with `--symptom-file -` and a final |
There was a problem hiding this comment.
Nit: this sentence runs diagnose-71's description straight into diagnose-72 and on through diagnose-76 in one paragraph; consider a list so each scenario id reads on its own.
| /// naming no engine) must change the entry's verdict when exactly the | ||
| /// declared engine is named, and must not change it for any other. | ||
| #[test] | ||
| fn every_engine_reading_entry_reads_the_engine_it_declares() { |
There was a problem hiding this comment.
Note: with a single SymptomEngine variant the "does not change for any other engine" half of this test cannot fail today; it only starts guarding once a second engine exists. Fine to keep, just don't read it as proof of that half yet.
juhovainio
left a comment
There was a problem hiding this comment.
Independent review of the whole diff against main (merge-base 9e50eac5, head 076591e1), run blind (no prior review discussion read until findings were settled), then reconciled against the PR's existing review history and live CI.
Scope reviewed: every changed file read in full — crates/rocm-core/src/diagnose.rs, diagnose/{catalog,engine,host,report,scoring,test_support}.rs, lib.rs (core matching/scoring/report-rendering); apps/rocm/src/{cli.rs,cli_args.rs,cli_report.rs,diagnose_cmd.rs,main.rs,remote/doctor.rs} (CLI wiring, --report/--send path); README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, docs/vllm.md, tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs, xtask/src/eval_fixtures.rs (docs/test completeness).
Verification of the previously-flagged risk areas, done independently rather than taken on faith:
- Text-report "Engine: ..." line: gated strictly on
report.provenance.is_some(), worded "read as X's" (not "matched"), and a dedicated test/scenario (diagnose-76) exercises the "named and read, but nothing scored" row together with the absence of the vLLM cause — no overclaiming path found. render_report_json: returnsserde_json::Result<String>end-to-end throughJsonOrText::render_jsonandDiagnoseOutput; no.expect()/.unwrap()/panic found on this or any other path reachable from pasted--symptomtext.- Non-ASCII/JSON escaping:
escape_non_asciiwalks the pretty-printed JSON char-by-char and is proven exact (round-trip, including an astral-plane emoji) bya_non_ascii_paste_round_trips_through_the_json_exactly; the ROCMAI-543 invariant test's corpus now includes--enginerows, closing the gap a prior round found. --report/--send:shared_reportis built only from the examination plus an established-entry id/flag, never fromreport.provenanceor the raw symptom;a_shared_report_never_carries_the_pasted_symptom_or_its_provenanceplants a token/host/path in the paste and proves it doesn't leak into the JSON or the mail link.- vLLM anchoring:
--engine vllmis a strict, explicit opt-in that widens matching only forcheck_16_vllm_oom's own entry (proven equivalent to the old per-linevllm:prefix), not an inference heuristic — so there's no over-broad/unanchored matching surface introduced here.
All of the above check out at the current head; I did not find any of them regressed or incompletely fixed.
New (non-blocking) observations from this pass, posted inline where the diff allows an anchor:
crates/rocm-core/src/diagnose.rs:95-107—SymptomProvenance.symptomis the firstrocm-corefield to carry a user's raw paste verbatim into a serializable report; the--report/--sendexclusion that keeps it from leaving the machine lives entirely in the CLI layer (diagnose_cmd.rs::shared_report), not as a structural guard inrocm-coreitself. This is a disclosed, intentional, well-tested tradeoff (confirmed by the author in this PR's thread), not a defect — flagging only as a suggestion for discoverability (e.g. aDiagnoseReport::redacted()helper) for the next consumer ofDiagnoseReport.crates/rocm-core/src/diagnose.rs:18-22(module doc comment, not itself touched by this diff, so no inline anchor available) — stale relative to this PR's new public surface: doesn't mentiondiagnose_with_engine/SymptomEngine/SymptomProvenance. Nitpick.
Not independently verified by me: a local cargo test/cargo clippy run (no toolchain available in this environment) and any consumer of DiagnoseReport outside this diff's tree.
CI at head 076591e1: all 39 check runs completed, all success except CodeQL (neutral, standard for that job) and Skill checks (skillscope) (skipped, documented in this repo as advisory-only, not yet a required gate). No failing required check.
This matches the conclusion of the latest human reviewer's approval at this same head. No blocking findings from this independent pass.
🤖 Automated by an AI agent
| pub struct SymptomProvenance { | ||
| pub engine: SymptomEngine, | ||
| pub symptom: String, | ||
| /// Whether an entry that ran on the examined platform reads `engine`, that | ||
| /// is, whether naming it could change anything. The text report says so | ||
| /// when it could not (native Windows has no vLLM entry today), so the flag | ||
| /// is not silently ignored. | ||
| /// | ||
| /// Not serialized, so it adds nothing to the JSON report; `None` for a report | ||
| /// read back from JSON, where it is not known. | ||
| #[serde(skip)] | ||
| pub read_here: Option<bool>, | ||
| } |
There was a problem hiding this comment.
Non-blocking/informational: SymptomProvenance.symptom is the first field in rocm-core that carries a user's raw pasted text verbatim into a serializable DiagnoseReport. I can see from the PR thread this is a deliberate, disclosed tradeoff (the whole point of --engine is proving the paste wasn't silently edited), and that the --report/--send exclusion is independently tested (a_shared_report_never_carries_the_pasted_symptom_or_its_provenance) — so I'm not re-litigating that decision. The one thing I'd still suggest: that exclusion currently lives entirely in the CLI layer (apps/rocm/src/diagnose_cmd.rs's shared_report), with nothing in rocm-core itself (e.g. a DiagnoseReport::redacted() helper dropping provenance) making "build the safe-to-share form" a one-line, discoverable operation co-located with the type that carries the risk. Worth considering for the next consumer of DiagnoseReport outside this diff, not blocking here.
Why
The vLLM out-of-memory entry (
fix-16-vllm-oom) scores only lines that mention vLLM (vllmorgpu-memory-utilization). That is deliberate: the error vLLM prints at startup is an ordinarytorch.OutOfMemoryError, and without the anchor any PyTorch program's out-of-memory error would be blamed on vLLM. But a user's paste often does not contain the word. To get a match, the rocm-doctor skill has been editing the user's text, writingvllm:in front of it, so what the CLI diagnosed and reported was not what the user pasted.What changes
rocm diagnose --symptom <TEXT> --engine vllmsays which engine the symptom came from. With it, every non-blank line of the symptom counts as vLLM's for the vLLM entry. That gives the same match as prefixing each line withvllm, and the text is not changed.vllm, the namerocm serve --engineuses. Any other value,lemonadeincluded, is a clap usage error (exit 2) that lists the accepted values.--engineneeds--symptomor--symptom-fileand is refused alone, because it would describe nothing. A file's text is recorded without its final line break, as the reader strips it.provenance: {engine, symptom}, with the symptom exactly as given once the JSON is decoded. On the wire,rocm diagnose --jsonwrites every non-ASCII character as a\uXXXXescape (UTF-16 units), so the output stays plain ASCII as the existing rule for redirectable output requires (ROCMAI-543) and any JSON parser recovers the identical paste. It is absent when no engine is named, so reports without--engineare unchanged. The diagnose JSON has no version number and the field is additive, so nothing is bumped.--engine, the text report opens with one line saying which engine was named and whether it counted on this OS (it says the flag changed nothing where no entry reads the engine, native Windows today). The answer comes from the catalog table that records which entry reads the engine, through the same platform filter the checks use; JSON is unchanged.--reportis unaffected: the shareable report is a fixed struct built without the diagnosis, so neither the paste norprovenancecan reach it or its mail link.--enginestays accepted with--reportand--distro; with--distrothe diagnosis,provenanceincluded, is shown on the user's own screen, and the docs say so.vllmin the symptom keeps working: the vLLM engine's own startup hint androcm serve's out-of-memory note still print that form.rocm-core, the catalog marks which checks read the engine, anddiagnose_with_engineis the new entry point.diagnosecalls it with no engine.--help, README.md, docs/vllm.md, docs/testing.md, docs/manual-testing.md and docs/architecture.md describe the flag.Verification
New scenarios in
tests/e2e-cucumber/features/diagnose.feature, numbered 68 to 76 after #1048, #1051 and #1053 took 59 to 67:diagnose-68 (
@requires-os:linux): a two-line vLLM out-of-memory paste that never names vLLM, diagnosed with--engine vllm, is matched as the vLLM startup out-of-memory failure, and the report's symptom equals the paste byte for byte.diagnose-69 (every lane): the report records vLLM as the named engine and keeps the paste as given.
diagnose-70 (every lane):
--engine lemonadeis refused as a usage error, and nothing is diagnosed.diagnose-71 (
@requires-os:linux): on a simulated Linux machine whose only GPU is a released one (the same Given as diagnose-58, so a real report is prepared on every Linux lane, with or without a GPU),--reportwith the engine named and a paste carrying a user, host and token prints none of them and noprovenance, and nothing is sent. The sweep first requires that a real report was printed. Native Windows cannot be simulated this way; the payload unit test covers it.diagnose-72 (every lane): a paste piped through
--symptom-file -with--engine vllmrecords the engine and the exact symptom without its final line break.diagnose-73 (
@requires-os:linux, so Linux and WSL): the text report opens withEngine: vllm (named with --engine; ...)and shows the vLLM cause, which is absent without the flag.diagnose-74 (
@requires-os:windows): the text report opens with the line saying no entry for vllm exists on this OS, so the flag changed nothing, and the rest of the report is byte-identical to the run without the flag.diagnose-76 (
@requires-os:linux): an unrelated paste (a plain Python traceback) with--engine vllmin text mode: the report opens with the line saying the lines were read as vllm's, and the vLLM cause is absent, so the line is asserted together with the nothing-scored outcome.diagnose-75 (every lane): a non-ASCII paste (an accented path, a box-drawing character, an emoji) saved to a file and diagnosed with
--symptom-file <file> --engine vllm --jsonprints only ASCII, and the decodedprovenance.symptomequals the paste.The ROCMAI-543 ASCII test now also runs
--enginerows with a non-ASCII paste on Linux, WSL and Windows;a_non_ascii_paste_round_trips_through_the_json_exactlychecks the parsed symptom equals the paste byte for byte;every_engine_name_is_its_serialized_nameties the printed engine name to its JSON name for every variant. Dropping the escaping fails the first two; removing the CLI's escaping hook fails diagnose-75.the_text_report_says_which_engine_was_named_and_whether_it_countedalso has a row for an engine named on a host that reads it with an unrelated paste and with an empty one: the line says the lines were read as vllm's and no vLLM entry is in the report.every_engine_reading_entry_reads_the_engine_it_declareswalks the catalog table and checks, for every engine-reading entry, every platform family it applies to and every engine variant, that an engine-free paste changes verdict when the declared engine is named and does not for any other; it fails if it finds no such entry.render_report_jsonreturns aResultthroughJsonOrText::render_json; a test checks that a failing hook surfaces an error and writes nothing.Unit tests check that naming vLLM matches as the
vllm:prefix did, that every non-blank line of a multi-line paste is anchored, that no entry other than vLLM's changes, and that the report keeps the symptom verbatim.a_shared_report_never_carries_the_pasted_symptom_or_its_provenanceplants a token, a path and a host in the paste and checks the report JSON and the mail link; leaking the symptom into the entry, and separately into the architecture matrix, makes it fail. The clap parse test covers--enginewith both symptom sources and alone.cargo test --workspace --all-targets --no-fail-fast(AGENTS.md §8)cargo xtask smoke(AGENTS.md §8)smoke: okcargo test -p xtaskcargo clippy --workspace --all-targets -D warnings,cargo clippy -p e2e-cucumber --test e2e -D warnings,cargo fmt --check,feature_naming,skill_reference,eval-fixtures --check,catalog --check,check-architecture-doccargo nextest runon the 12 cratescargo xtask affectedselectscargo xtask e2efor diagnose-59, 61 to 67, 69, 70, 72, 74, 75cargo xtask e2efor diagnose-58 to -73, -75 and -76These rows were taken on the final commit on top of main
9e50eac5, on Windows and in WSL2. The pre-push hook failed once onatomic_write::tests::a_reader_never_finds_the_destination_absent_while_it_is_replaced, a known Windows timing test in code this change does not touch; it passed alone 3 of 3 and the hook passed on the retry.Failures and instability outside this change:
build_metadata::build_script::tests::git_watch_paths_*fail because that account's git config signs commits with an SSH key that is absent there. They pass on Windows.process::tests::loopback_lease_blocks_a_second_bind_until_released(rocm-core) failed in the full-workspace run and passed alone 3 of 3, in code this change does not touch.rocm-engine-lemonadearchive_cachetests,interprocess_lock_serializes_poisoned_validation_and_recovery(times out waiting for its child process) andarchive_digest_check_rejects_a_single_bit_of_tampering(can hang for tens of minutes). Both also fail or hang on main under load, and both passed in the run recorded above.Scope
rocm diagnose --json(and itsrender_jsonhook and the fixture recorder using it) lands in this change on purpose: echoing the paste verbatim inprovenance.symptomis what adds the first raw user text to that output, and shipping--enginewithout the escaping would break the ROCMAI-543 rule for redirectable output on the day it lands. The text report and other--jsoncommands can still print host-derived non-ASCII.provenance.symptom, the only field that holds the user's paste. Host-derivedevidencelines can quote an install path read from the examined machine; a non-ASCII install path is now escaped in--json, but the text report still prints it raw. That predates this change and is left alone.skills/rocm-doctor/is not changed here. fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029 rewrites it; replacing itsvllm:instruction with--engine vllmfollows once that lands.rocm serve's out-of-memory note keep printing thevllm: …symptom form, which still works.--enginegains another value only when the catalog has an entry specific to that engine. Until then,lemonadeis a usage error here even thoughrocm serveaccepts it.rocm remote doctor --symptomhas no--engine; remote users still need thevllm:prefix. That is a follow-up.Part of ROCMAI-955.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable: no bug fix.docs/architecture.md. Not applicable: a flag on an existing command.--reportnever carries the paste is asserted against the printed report (diagnose-71) and the payload (unit test).