Skip to content

ROCMAI-955: say per match in diagnose --json whether the symptom contributed - #1051

Merged
volen-silo merged 1 commit into
mainfrom
rocmai-955-symptom-matched
Nov 16, 2026
Merged

volen-silo merged 1 commit into
mainfrom
rocmai-955-symptom-matched

Conversation

@volen-silo

@volen-silo volen-silo commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Why

rocm diagnose --symptom <TEXT> scores every catalog entry from two sources together: the machine's examined state and the pasted text. A host-state finding can therefore outrank the entry the paste actually describes. On a host with HSA_OVERRIDE_GFX_VERSION set, for example, the override finding sits above the HSA_STATUS_ERROR_INVALID_ISA entry the user pasted. Nothing in the output said which source an entry came from. A consumer such as the rocm-doctor skill had to guess from the evidence strings, and could offer an unrelated host finding as the fix for the user's error.

What changes

  • Each entry in rocm diagnose --json's matched carries symptom_matched. It is true when the symptom text contributed to that entry's score or evidence, and false when the entry comes from the machine's state alone.
  • "Contributed" is all it says. Any keyword hit counts, however weak (for example a broad hsa_status_error hit adds 10 points to the module-not-loaded entry), so true does not mean the entry explains the pasted error. The README, --help, the field's doc comment and the scenarios all say so.
  • It is decided once, in diagnose(), by scoring the catalog again with no symptom and comparing each entry's score and evidence. The comparison is a pure function, contributed(d, host_only). Individual checks set no flag, so a new check cannot forget to. The second pass reads only the examination already collected and probes nothing.
  • Evidence is compared as well as score because scores cap at 100: a keyword hit on an entry already at the cap changes only its evidence.
  • The field is always written. With no symptom it is false on every entry, because nothing was pasted to contribute. So a missing field means only that the CLI predates it, and #[serde(default)] keeps older payloads readable.
  • The text report prints source: the examined host's state alone; the symptom text did not contribute to it under an entry the paste did not contribute to. It appears only when the paste contributed to some other entry in the report, because with no symptom the line would sit on every row and say nothing. The wording says "the examined host" because rocm remote doctor renders the same report about a remote machine.
  • The "contributed to some other entry" test looks at the whole report, not only the rows shown. If the only entry the paste contributed to is past --top, every row shown carries the line, and each is still true of its own row. The README states this and a --top 1 test pins it.
  • The diagnose JSON has no version number. The repo's versioned documents keep their number for an added field, and this one is additive, so nothing is bumped.
  • The five recorded rocm diagnose --json replies under skills/rocm-doctor/evals/fixtures/ were regenerated with cargo xtask eval-fixtures. The only change in them is the new field on each match.
  • README.md, --symptom's help, docs/testing.md, docs/manual-testing.md and docs/architecture.md describe the field.

Verification

New scenarios in tests/e2e-cucumber/features/diagnose.feature, all three on every OS lane. They run the CLI with HSA_OVERRIDE_GFX_VERSION=11.0.0, an ISA error paste that no host state scores and that names nothing the override entry reads, so the answers do not depend on the runner:

  • diagnose-59 (@id:diagnose-json-says-which-causes-the-paste-contributed-to): in the JSON, the override entry has symptom_matched: false and the ISA entry has true.
  • diagnose-60 (@id:diagnose-text-marks-causes-the-paste-did-not-contribute-to): in the text report, the override entry carries the source line and the ISA entry does not.
  • diagnose-61 (@id:diagnose-without-a-paste-credits-no-cause-to-one): with no paste, every listed entry has symptom_matched: false in the JSON and no cause carries the source line in the text. With the symptom_matched_any && guard removed, diagnose-59 and -60 stay green and diagnose-61 fails.

Unit tests cover:

  • a host finding the paste does not explain;
  • a paste that raises a host finding's score, and one that lists an entry the host alone ruled out;
  • contribution_is_any_difference_from_the_run_without_the_symptom: score alone differs, evidence alone differs, entry absent without the paste, identical. Deleting either half of the comparison fails it, and the earlier tests stayed green under either deletion;
  • no symptom (the field is written as false) and reading a payload from before the field existed;
  • the --top 1 rule, and rocm remote doctor's rendering with one true and one false entry.
Platform Run Result
Windows cargo nextest run --workspace (cargo xtask affected selects the workspace) 6468 run, 6466 pass, 17 skipped; the 2 failures are below
Windows cargo test --workspace --all-targets --no-fail-fast (AGENTS.md §8), skipping one test below 46 test binaries, 6471 pass, 0 fail, 17 ignored
Windows cargo clippy --workspace --all-targets -D warnings, python scripts/smoke_local.py (AGENTS.md §8) clean; smoke: ok
Windows cargo xtask e2e for diagnose-59, -60, -61 3 scenarios, 15 steps pass
WSL2 (Ubuntu, Strix Halo) same nextest set 6933 run, 6931 pass; the 2 failures are below
WSL2 cargo xtask e2e for diagnose-59, -60, -61 3 scenarios, 15 steps pass
Windows cargo fmt --check, cargo test -p xtask, cargo xtask eval-fixtures --check, cargo xtask catalog --check, feature_naming, skill_reference pass

The failures are outside this change:

  • Windows: two rocm-engine-lemonade archive_cache tests. interprocess_lock_serializes_poisoned_validation_and_recovery fails alone with "timed out waiting for child process" (it also fails on main, and passed in the §8 single-process run above). archive_digest_check_rejects_a_single_bit_of_tampering hangs on this machine, alone as well, so it was skipped in the §8 run and has no pass result on this tree.
  • WSL2: build_metadata::build_script::tests::git_watch_paths_* fail because that account's git config signs commits with an SSH key that is absent there.

Not verified: rocm remote doctor --symptom is tested only at the rendering level, with a hand-built report. Nothing checks that the remote path passes the symptom into the comparison; it does so by calling the same diagnose().

Scope

  • skills/rocm-doctor/SKILL.md and reference.md are not changed here. fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029 rewrites them; the skill reads symptom_matched in step 2 in a follow-up once that lands.
  • When a symptom was given and matched nothing, the text report prints no source line. Telling "no symptom" from "symptom matched nothing" there would need a report-level flag, which this change leaves out.
  • No materiality floor: a weak keyword hit still counts as contributing.

Part of ROCMAI-955.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable: no bug fix.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. Not applicable: a field on an existing report.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state, per AGENTS.md §3. The source line is asserted together with which entry the paste was scored into (diagnose-60), and its absence together with the no-symptom report (diagnose-61).

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Merge note: #1048, #1051, #1052 and #1053 each add e2e scenarios numbered from diagnose-59 in diagnose.feature, and feature_naming requires the numbers to be sequential. Whichever of the four merges first keeps its numbers. Each of the others renumbers on rebase, and updates the numbers cited in its docs/testing.md section. Their @id: tags are distinct, so only the numbers collide. Their skill-doc follow-ups wait for #1029 to merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@volen-silo
volen-silo force-pushed the rocmai-955-symptom-matched branch from 96fc975 to e19a84f Compare November 13, 2026 09:24
@volen-silo
volen-silo marked this pull request as ready for review November 13, 2026 09:47
@volen-silo
volen-silo requested a review from a team as a code owner November 13, 2026 09:47
@volen-silo
volen-silo requested a review from tomastola November 13, 2026 09:47

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. symptom_matched is always serialized and serde(default) keeps older payloads readable. The text-form note and the rocm remote doctor wording are tied to the field by tests, and scenarios diagnose-59/60/61 cover the observable behavior. README, --help, docs/manual-testing.md, docs/testing.md and docs/architecture.md are updated. CI is green apart from the still-pending GPU lane and the CodeQL job. I reviewed the diff and CI status only and did not build or run it locally.

Non-blocking:

  1. mark_symptom_contribution compares against run_all_checks(e, ""), while the symptom run is chosen by an if/else just above. Please confirm both branches use the same checker set. A test for the branch other than run_all_checks would pin it. (inline)
  2. The catalog now runs twice whenever --symptom is non-empty. That is fine if checkers are pure, but that is a comment, not something enforced.
  3. The whitespace-symptom test only passes if diagnose trims earlier, since the guard here is is_empty(). Consider trimming inside or noting the dependency.
  4. The e2e step duplicates the note string instead of importing the now-public SYMPTOM_DID_NOT_CONTRIBUTE_NOTE. (inline)
  5. docs/testing.md uses E2E_ONLY_LIFECYCLE=1 for the diagnose scenarios. Please double-check that is the right gate for them.

Follow-up: SKILL.md and reference.md don't mention symptom_matched yet. The ticket says that waits for #1029, so this is just to track.

Comment thread crates/rocm-core/src/diagnose.rs Resolved
const PASTE_FIX_ID: &str = "fix-1-arch";

/// What the text report prints under a cause the paste did not contribute to.
const HOST_STATE_ONLY_MARK: &str =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates rocm_core::diagnose::SYMPTOM_DID_NOT_CONTRIBUTE_NOTE (now pub). Importing it would tie producer and consumer by construction instead of by a matching literal.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is deliberate. The step drives the real binary and asserts the wording a user sees, so it keeps its own copy of the line. If it imported SYMPTOM_DID_NOT_CONTRIBUTE_NOTE, a change to the wording would move the producer and the assertion together and nothing would fail. The unit tests in rocm-core use the constant, so producer and renderer cannot drift. The e2e crate could import it (it already depends on rocm-core), so this is a choice, not a limit. Say if you would rather have the import.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 31523d8, and my earlier "deliberate" was wrong: the repo rule is that a sentinel string shared between producer and consumer is one constant. The e2e step now imports rocm_core::diagnose::SYMPTOM_DID_NOT_CONTRIBUTE_NOTE in all three places and the local copy is gone. The scenarios were renumbered to diagnose-62 to 64 after #1048 took 59 to 61; no scenario logic changed.

@volen-silo
volen-silo added this pull request to the merge queue Nov 13, 2026
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for the approval and the careful read. I have not pushed anything for these points, since none is a defect. In order:

  1. Same checker set: yes, answered in the inline thread on diagnose.rs.
  2. The catalog runs twice with a symptom: the doc comment says the checkers are pure functions of the examination and the symptom, and by signature they are: they take (&Examination, &str) and read no machine state. That is not enforced by the type system, so a checker that read the clock or the file system would break the claim. I kept it as a comment.
  3. The whitespace test: it does not depend on trimming elsewhere. " " is not empty, so it takes the whole path, and the symptom run and the baseline give identical entries because whitespace matches no keyword in any checker, so every entry is false. A checker that treated whitespace as a hit would fail the test.
  4. The duplicated note string: answered in the inline thread on diagnose_steps.rs.
  5. E2E_ONLY_LIFECYCLE=1 for the diagnose scenarios: that is deliberate. E2E_ONLY_LIFECYCLE=1 together with E2E_ALSO_RUN_IDS=<ids> is the route docs/testing.md already documents for selecting non-lifecycle scenarios by id, and unlike -n it keeps each scenario's OS tags, so a Linux-only scenario still skips on Windows. All three scenarios here run on every OS lane, so the same command works everywhere.
  6. Follow-up: SKILL.md and reference.md will read symptom_matched once the skill rewrite in fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029 lands. That is tracked on the ticket.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 13, 2026

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head e19a84f

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (18 files, +698/-9), diffed against the merge-base with main (f536619a). Part of ROCMAI-955: this is one item (symptom_matched) from a multi-PR follow-up ticket spanning #1048/#1051/#1052/#1053 from an earlier skill review; the PR does not claim to close the ticket, which matches the ticket's own umbrella scope.

Blocking

  • New e2e test duplicates a sentinel string instead of importing the shared constant AGENTS.md requires — tests/e2e-cucumber/tests/e2e/diagnose_steps.rs:4116-4118 (definition), used at :4258, :4272, :4286
    This PR adds const HOST_STATE_ONLY_MARK: &str = "source: the examined host's state alone; the symptom text did not contribute to it"; — a verbatim, hand-typed copy of crates/rocm-core/src/diagnose/report.rs:81's pub const SYMPTOM_DID_NOT_CONTRIBUTE_NOTE (re-exported at crates/rocm-core/src/diagnose.rs:46). AGENTS.md states explicitly: "A value matched across files (a version tag, a sentinel string) gets one shared constant and a test tying the producer to the consumer." This same PR shows the correct pattern is available and already used elsewhere: apps/rocm/src/remote/doctor.rs:753 does use rocm_core::diagnose::{Diagnosis, SYMPTOM_DID_NOT_CONTRIBUTE_NOTE}; directly, and tests/e2e-cucumber/tests/e2e/diagnose_steps.rs itself already calls rocm_core::fake_executable::write_fake_executable directly elsewhere in the same file (e.g. line 528), so the crate is linked and importing the real constant was available, not blocked by a crate-boundary issue. A pre-existing (unrelated, not introduced by this PR) comment nearby at :2551-2553 asserts this test crate "does not link the library behind it," which is demonstrably false given the rocm_core:: calls already in the file — plausibly why the new code reached for a literal instead of an import.
    Confidence 85 · mechanical · Fix: use rocm_core::diagnose::SYMPTOM_DID_NOT_CONTRIBUTE_NOTE; and drop the local HOST_STATE_ONLY_MARK const, matching the pattern already used in doctor.rs.

Non-blocking

  • docs/architecture.md's entry for diagnose/report.rs is now incomplete — docs/architecture.md:281
    The row still reads "Upstream routing for a failure nothing recognised (upstream_route) and the human-readable report (render_report_text)," with no mention of the new SYMPTOM_DID_NOT_CONTRIBUTE_NOTE export or the per-entry note logic this PR adds to render_report_text. AGENTS.md's architecture-doc rule is enforced by a source-scan check that only verifies an entry exists, and says so itself: "The check reads no prose, so keeping an entry's description true is still on you." The entry isn't false, just stale relative to the file's new public surface.
    Confidence 70 · mechanical · Fix: add a clause noting the new source-line/constant to the diagnose/report.rs row.

Decisions for the author

None.

Positive signals

  • contributed() (crates/rocm-core/src/diagnose.rs) compares both score and evidence, not score alone — correctly catches a keyword hit on an entry already at the 100-point score cap, which would otherwise be invisible to a score-only diff.
  • mark_symptom_contribution's doc comment states its own load-bearing assumption plainly ("The checkers are pure functions of the examination and the symptom, so the second run reads no more of the machine than the first"), which is currently true (verified: no env::var/fs::/Command::new/interior-mutability in any checker) and gives a future checker author fair warning if that stops holding.
  • apps/rocm/src/remote/doctor.rs reuses the exact same render_report_text function and SYMPTOM_DID_NOT_CONTRIBUTE_NOTE constant as the local path (use rocm_core::diagnose::{Diagnosis, SYMPTOM_DID_NOT_CONTRIBUTE_NOTE};), with a dedicated test pinning the wording stays host-neutral — zero drift risk there, in contrast to the e2e duplication above.

Deployment notes

Additive and backward compatible: the field is always serialized, and #[serde(default)] on the deserialize side means older recorded CLI payloads lacking it still parse (reading as false), so no migration is needed for consumers of recorded JSON.

What this covered

Read the full diff (f536619a…e19a84f5) and every changed file completely via four independent coupled-file workers — (1) core logic: crates/rocm-core/src/diagnose.rs, diagnose/report.rs, diagnose/scoring.rs, diagnose/windows.rs; (2) CLI/consumers: apps/rocm/src/cli.rs, apps/rocm/src/diagnose_cmd.rs, apps/rocm/src/remote/doctor.rs; (3) e2e tests: tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs; (4) docs/fixtures: README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, the five regenerated skills/rocm-doctor/evals/fixtures/*/replies.json — plus a fifth worker for the agent-instruction-adherence, history and code-comments passes against AGENTS.md. All candidate findings were re-verified directly against the source by the consolidating pass (direct grep/read, not taken on a worker's word). Read CI via the forge API: 56 check runs, all completed and green except one still-queued GPU lane (MI300X), matching the PR's own description of a pending GPU lane. Retrieved ROCMAI-955 from the tracker by its identifier (it is reachable from here): it's an umbrella ticket for several follow-up PRs from an earlier skill review; this PR implements one listed item and does not claim closure.

Did not run: the "prior changes" source-reading pass (whether review comments on earlier changes to these files apply here) was not performed — no prior-PR review history was pulled for these specific files beyond git log/blame, which the history pass did cover. Did not execute the Rust/e2e test suite myself (no cargo toolchain in this sandbox); relied on the green CI run and the workers' static tracing of the logic instead of a fresh run. Not reconciled against this PR's existing review thread or comments.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

`rocm diagnose --symptom` (or `--symptom-file`) scores the catalog from
the examined host's state and the pasted text together, so a host-state
finding can outrank the entry the paste added to. A consumer had to
judge from the evidence whether the paste lay behind a match. Each entry
in `matched` now carries `symptom_matched`: true when the text
contributed to the entry's score or evidence, false when it came from
the host's state alone. Any keyword hit counts, however weak, so it
records a contribution, not that the entry explains the error.

It is decided once, in `diagnose`, by running the catalog again with no
symptom and comparing each entry with its counterpart (absent, a
different score, or different evidence, since a hit on an entry already
at the 100 cap changes only its evidence), rather than by every checker
setting a flag it could forget. The field is always serialized and is
false when no symptom was given, so its absence means only an older CLI.
It is additive: the diagnose JSON carries no version, and
`#[serde(default)]` keeps older payloads readable.

The text report prints a `source: the examined host's state alone` line
under each shown cause the text did not contribute to, once the text
contributed to any cause in the report, including one cut off by
`--top`. The wording names the examined host because `rocm remote
doctor` renders the same report for a machine it reached over the
network.

ROCMAI-955

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
volen-silo force-pushed the rocmai-955-symptom-matched branch from e19a84f to 31523d8 Compare November 16, 2026 09:22
@volen-silo
volen-silo enabled auto-merge November 16, 2026 09:36
@volen-silo
volen-silo added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit ee60506 Nov 16, 2026
45 of 46 checks passed
@volen-silo
volen-silo deleted the rocmai-955-symptom-matched branch November 16, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants