Repository navigation
ROCMAI-955: say per match in diagnose --json whether the symptom contributed - #1051
Conversation
|
Merge note: #1048, #1051, #1052 and #1053 each add e2e scenarios numbered from diagnose-59 in |
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
96fc975 to
e19a84f
Compare
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. symptom_matched is always serialized and serde(default) keeps older payloads readable. The text-form note and the rocm remote doctor wording are tied to the field by tests, and scenarios diagnose-59/60/61 cover the observable behavior. README, --help, docs/manual-testing.md, docs/testing.md and docs/architecture.md are updated. CI is green apart from the still-pending GPU lane and the CodeQL job. I reviewed the diff and CI status only and did not build or run it locally.
Non-blocking:
mark_symptom_contributioncompares againstrun_all_checks(e, ""), while the symptom run is chosen by an if/else just above. Please confirm both branches use the same checker set. A test for the branch other thanrun_all_checkswould pin it. (inline)- The catalog now runs twice whenever
--symptomis non-empty. That is fine if checkers are pure, but that is a comment, not something enforced. - The whitespace-symptom test only passes if
diagnosetrims earlier, since the guard here isis_empty(). Consider trimming inside or noting the dependency. - The e2e step duplicates the note string instead of importing the now-public
SYMPTOM_DID_NOT_CONTRIBUTE_NOTE. (inline) docs/testing.mdusesE2E_ONLY_LIFECYCLE=1for the diagnose scenarios. Please double-check that is the right gate for them.
Follow-up: SKILL.md and reference.md don't mention symptom_matched yet. The ticket says that waits for #1029, so this is just to track.
| const PASTE_FIX_ID: &str = "fix-1-arch"; | ||
|
|
||
| /// What the text report prints under a cause the paste did not contribute to. | ||
| const HOST_STATE_ONLY_MARK: &str = |
There was a problem hiding this comment.
This duplicates rocm_core::diagnose::SYMPTOM_DID_NOT_CONTRIBUTE_NOTE (now pub). Importing it would tie producer and consumer by construction instead of by a matching literal.
There was a problem hiding this comment.
This one is deliberate. The step drives the real binary and asserts the wording a user sees, so it keeps its own copy of the line. If it imported SYMPTOM_DID_NOT_CONTRIBUTE_NOTE, a change to the wording would move the producer and the assertion together and nothing would fail. The unit tests in rocm-core use the constant, so producer and renderer cannot drift. The e2e crate could import it (it already depends on rocm-core), so this is a choice, not a limit. Say if you would rather have the import.
There was a problem hiding this comment.
Fixed in 31523d8, and my earlier "deliberate" was wrong: the repo rule is that a sentinel string shared between producer and consumer is one constant. The e2e step now imports rocm_core::diagnose::SYMPTOM_DID_NOT_CONTRIBUTE_NOTE in all three places and the local copy is gone. The scenarios were renumbered to diagnose-62 to 64 after #1048 took 59 to 61; no scenario logic changed.
|
Thanks for the approval and the careful read. I have not pushed anything for these points, since none is a defect. In order:
|
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head e19a84f
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (18 files, +698/-9), diffed against the merge-base with main (f536619a). Part of ROCMAI-955: this is one item (symptom_matched) from a multi-PR follow-up ticket spanning #1048/#1051/#1052/#1053 from an earlier skill review; the PR does not claim to close the ticket, which matches the ticket's own umbrella scope.
Blocking
- New e2e test duplicates a sentinel string instead of importing the shared constant AGENTS.md requires —
tests/e2e-cucumber/tests/e2e/diagnose_steps.rs:4116-4118(definition), used at:4258,:4272,:4286
This PR addsconst HOST_STATE_ONLY_MARK: &str = "source: the examined host's state alone; the symptom text did not contribute to it";— a verbatim, hand-typed copy ofcrates/rocm-core/src/diagnose/report.rs:81'spub const SYMPTOM_DID_NOT_CONTRIBUTE_NOTE(re-exported atcrates/rocm-core/src/diagnose.rs:46). AGENTS.md states explicitly: "A value matched across files (a version tag, a sentinel string) gets one shared constant and a test tying the producer to the consumer." This same PR shows the correct pattern is available and already used elsewhere:apps/rocm/src/remote/doctor.rs:753doesuse rocm_core::diagnose::{Diagnosis, SYMPTOM_DID_NOT_CONTRIBUTE_NOTE};directly, andtests/e2e-cucumber/tests/e2e/diagnose_steps.rsitself already callsrocm_core::fake_executable::write_fake_executabledirectly elsewhere in the same file (e.g. line 528), so the crate is linked and importing the real constant was available, not blocked by a crate-boundary issue. A pre-existing (unrelated, not introduced by this PR) comment nearby at:2551-2553asserts this test crate "does not link the library behind it," which is demonstrably false given therocm_core::calls already in the file — plausibly why the new code reached for a literal instead of an import.
Confidence 85 · mechanical · Fix:use rocm_core::diagnose::SYMPTOM_DID_NOT_CONTRIBUTE_NOTE;and drop the localHOST_STATE_ONLY_MARKconst, matching the pattern already used indoctor.rs.
Non-blocking
docs/architecture.md's entry fordiagnose/report.rsis now incomplete —docs/architecture.md:281
The row still reads "Upstream routing for a failure nothing recognised (upstream_route) and the human-readable report (render_report_text)," with no mention of the newSYMPTOM_DID_NOT_CONTRIBUTE_NOTEexport or the per-entry note logic this PR adds torender_report_text. AGENTS.md's architecture-doc rule is enforced by a source-scan check that only verifies an entry exists, and says so itself: "The check reads no prose, so keeping an entry's description true is still on you." The entry isn't false, just stale relative to the file's new public surface.
Confidence 70 · mechanical · Fix: add a clause noting the new source-line/constant to thediagnose/report.rsrow.
Decisions for the author
None.
Positive signals
contributed()(crates/rocm-core/src/diagnose.rs) compares bothscoreandevidence, not score alone — correctly catches a keyword hit on an entry already at the 100-point score cap, which would otherwise be invisible to a score-only diff.mark_symptom_contribution's doc comment states its own load-bearing assumption plainly ("The checkers are pure functions of the examination and the symptom, so the second run reads no more of the machine than the first"), which is currently true (verified: noenv::var/fs::/Command::new/interior-mutability in any checker) and gives a future checker author fair warning if that stops holding.apps/rocm/src/remote/doctor.rsreuses the exact samerender_report_textfunction andSYMPTOM_DID_NOT_CONTRIBUTE_NOTEconstant as the local path (use rocm_core::diagnose::{Diagnosis, SYMPTOM_DID_NOT_CONTRIBUTE_NOTE};), with a dedicated test pinning the wording stays host-neutral — zero drift risk there, in contrast to the e2e duplication above.
Deployment notes
Additive and backward compatible: the field is always serialized, and #[serde(default)] on the deserialize side means older recorded CLI payloads lacking it still parse (reading as false), so no migration is needed for consumers of recorded JSON.
What this covered
Read the full diff (f536619a…e19a84f5) and every changed file completely via four independent coupled-file workers — (1) core logic: crates/rocm-core/src/diagnose.rs, diagnose/report.rs, diagnose/scoring.rs, diagnose/windows.rs; (2) CLI/consumers: apps/rocm/src/cli.rs, apps/rocm/src/diagnose_cmd.rs, apps/rocm/src/remote/doctor.rs; (3) e2e tests: tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs; (4) docs/fixtures: README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, the five regenerated skills/rocm-doctor/evals/fixtures/*/replies.json — plus a fifth worker for the agent-instruction-adherence, history and code-comments passes against AGENTS.md. All candidate findings were re-verified directly against the source by the consolidating pass (direct grep/read, not taken on a worker's word). Read CI via the forge API: 56 check runs, all completed and green except one still-queued GPU lane (MI300X), matching the PR's own description of a pending GPU lane. Retrieved ROCMAI-955 from the tracker by its identifier (it is reachable from here): it's an umbrella ticket for several follow-up PRs from an earlier skill review; this PR implements one listed item and does not claim closure.
Did not run: the "prior changes" source-reading pass (whether review comments on earlier changes to these files apply here) was not performed — no prior-PR review history was pulled for these specific files beyond git log/blame, which the history pass did cover. Did not execute the Rust/e2e test suite myself (no cargo toolchain in this sandbox); relied on the green CI run and the workers' static tracing of the logic instead of a fresh run. Not reconciled against this PR's existing review thread or comments.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
`rocm diagnose --symptom` (or `--symptom-file`) scores the catalog from the examined host's state and the pasted text together, so a host-state finding can outrank the entry the paste added to. A consumer had to judge from the evidence whether the paste lay behind a match. Each entry in `matched` now carries `symptom_matched`: true when the text contributed to the entry's score or evidence, false when it came from the host's state alone. Any keyword hit counts, however weak, so it records a contribution, not that the entry explains the error. It is decided once, in `diagnose`, by running the catalog again with no symptom and comparing each entry with its counterpart (absent, a different score, or different evidence, since a hit on an entry already at the 100 cap changes only its evidence), rather than by every checker setting a flag it could forget. The field is always serialized and is false when no symptom was given, so its absence means only an older CLI. It is additive: the diagnose JSON carries no version, and `#[serde(default)]` keeps older payloads readable. The text report prints a `source: the examined host's state alone` line under each shown cause the text did not contribute to, once the text contributed to any cause in the report, including one cut off by `--top`. The wording names the examined host because `rocm remote doctor` renders the same report for a machine it reached over the network. ROCMAI-955 Signed-off-by: Eugene Volen <eugene.volen@amd.com>
e19a84f to
31523d8
Compare
Why
rocm diagnose --symptom <TEXT>scores every catalog entry from two sources together: the machine's examined state and the pasted text. A host-state finding can therefore outrank the entry the paste actually describes. On a host withHSA_OVERRIDE_GFX_VERSIONset, for example, the override finding sits above theHSA_STATUS_ERROR_INVALID_ISAentry the user pasted. Nothing in the output said which source an entry came from. A consumer such as the rocm-doctor skill had to guess from theevidencestrings, and could offer an unrelated host finding as the fix for the user's error.What changes
rocm diagnose --json'smatchedcarriessymptom_matched. It istruewhen the symptom text contributed to that entry's score or evidence, andfalsewhen the entry comes from the machine's state alone.hsa_status_errorhit adds 10 points to the module-not-loaded entry), sotruedoes not mean the entry explains the pasted error. The README,--help, the field's doc comment and the scenarios all say so.diagnose(), by scoring the catalog again with no symptom and comparing each entry's score and evidence. The comparison is a pure function,contributed(d, host_only). Individual checks set no flag, so a new check cannot forget to. The second pass reads only the examination already collected and probes nothing.falseon every entry, because nothing was pasted to contribute. So a missing field means only that the CLI predates it, and#[serde(default)]keeps older payloads readable.source: the examined host's state alone; the symptom text did not contribute to itunder an entry the paste did not contribute to. It appears only when the paste contributed to some other entry in the report, because with no symptom the line would sit on every row and say nothing. The wording says "the examined host" becauserocm remote doctorrenders the same report about a remote machine.--top, every row shown carries the line, and each is still true of its own row. The README states this and a--top 1test pins it.rocm diagnose --jsonreplies underskills/rocm-doctor/evals/fixtures/were regenerated withcargo xtask eval-fixtures. The only change in them is the new field on each match.--symptom's help, docs/testing.md, docs/manual-testing.md and docs/architecture.md describe the field.Verification
New scenarios in
tests/e2e-cucumber/features/diagnose.feature, all three on every OS lane. They run the CLI withHSA_OVERRIDE_GFX_VERSION=11.0.0, an ISA error paste that no host state scores and that names nothing the override entry reads, so the answers do not depend on the runner:@id:diagnose-json-says-which-causes-the-paste-contributed-to): in the JSON, the override entry hassymptom_matched: falseand the ISA entry hastrue.@id:diagnose-text-marks-causes-the-paste-did-not-contribute-to): in the text report, the override entry carries the source line and the ISA entry does not.@id:diagnose-without-a-paste-credits-no-cause-to-one): with no paste, every listed entry hassymptom_matched: falsein the JSON and no cause carries the source line in the text. With thesymptom_matched_any &&guard removed, diagnose-59 and -60 stay green and diagnose-61 fails.Unit tests cover:
contribution_is_any_difference_from_the_run_without_the_symptom: score alone differs, evidence alone differs, entry absent without the paste, identical. Deleting either half of the comparison fails it, and the earlier tests stayed green under either deletion;false) and reading a payload from before the field existed;--top 1rule, androcm remote doctor's rendering with one true and one false entry.cargo nextest run --workspace(cargo xtask affectedselects the workspace)cargo test --workspace --all-targets --no-fail-fast(AGENTS.md §8), skipping one test belowcargo clippy --workspace --all-targets -D warnings,python scripts/smoke_local.py(AGENTS.md §8)smoke: okcargo xtask e2efor diagnose-59, -60, -61cargo xtask e2efor diagnose-59, -60, -61cargo fmt --check,cargo test -p xtask,cargo xtask eval-fixtures --check,cargo xtask catalog --check,feature_naming,skill_referenceThe failures are outside this change:
rocm-engine-lemonadearchive_cachetests.interprocess_lock_serializes_poisoned_validation_and_recoveryfails alone with "timed out waiting for child process" (it also fails on main, and passed in the §8 single-process run above).archive_digest_check_rejects_a_single_bit_of_tamperinghangs on this machine, alone as well, so it was skipped in the §8 run and has no pass result on this tree.build_metadata::build_script::tests::git_watch_paths_*fail because that account's git config signs commits with an SSH key that is absent there.Not verified:
rocm remote doctor --symptomis tested only at the rendering level, with a hand-built report. Nothing checks that the remote path passes the symptom into the comparison; it does so by calling the samediagnose().Scope
skills/rocm-doctor/SKILL.mdandreference.mdare not changed here. fix(skill): make rocm-doctor follow what the rocm CLI does, and grade it #1029 rewrites them; the skill readssymptom_matchedin step 2 in a follow-up once that lands.Part of ROCMAI-955.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable: no bug fix.docs/architecture.md. Not applicable: a field on an existing report.