Skip to content

ROCMAI-955: say X.Y for fix-8 and give fix-4 one verify line - #1053

Merged
volen-silo merged 1 commit into
mainfrom
rocmai-955-catalog-consistency
Nov 16, 2026
Merged

volen-silo merged 1 commit into
mainfrom
rocmai-955-catalog-consistency

Conversation

@volen-silo

@volen-silo volen-silo commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Two catalog entries said something different from what the code does.

  • fix-8 (fix-8-wheel-rocm) is titled "a different ROCm major", but its check compares the framework's ROCm version with the system's at X.Y: major_version() in diagnose/host.rs returns X.Y. The comparison is the right one. PyTorch publishes its ROCm wheel indexes per X.Y release (whl/rocmX.Y), the function's own doc comment has said X.Y since it was written, and the first remedy offered rocm6.4 and rocm6.3 side by side. A user told "major" would expect 6.3 against 6.4 to be fine, and it is exactly what fires.
  • fix-4 (fix-4-render-group) kept its verify command in two copies. The one rocm diagnose printed included ls -l /dev/kfd; the one rocm fix printed did not. So the fix told the user to check less than the diagnosis did, and the copies could drift further with nothing to catch it.

What changes

  • fix-8: the wording now names what is compared. The diagnosis title is "Framework wheel built for a different ROCm X.Y release than the system", the fix title is "Reinstall the framework against the system ROCm/HIP X.Y release", and the remedy comments say to match the X.Y release rocm examine reports. major_version is renamed release_line, with a doc comment saying why X.Y is the unit, how it differs from the unrelated major_minor in report.rs, and that it returns None when the string has no minor (a framework reporting only hip=7), so check 8 skips the version comparison as before. Behaviour is unchanged.
  • fix-13 and fix-1: the same drift sat next to it. fix-13 said "matched to your framework's HIP major", but on Windows the check compares the framework's HIP with the installed SDK at X.Y, so an SDK matched on the major alone is diagnosed with fix-8. fix-1 said "the nightly for the ROCm major you have", but the nightly indexes are per X.Y (nightly/rocm7.14). Both now say "X.Y release". The "ROCm 10 has no released PyTorch index yet" wording stays, because there it really means the major.
  • fix-4: one constant, RENDER_GROUP_VERIFY in fix/catalog.rs, now feeds both the diagnosis and the fix recipe. It keeps the longer line and widens its group check: the line now accepts the group that owns /dev/kfd (stat -c %G /dev/kfd) as well as render and video, because the diagnosis builds its usermod command around that owning group, and a user fixed through a third group used to fail the verify. It is still one static string. Its doc comment says what the exit status checks (membership in the owning group, render or video, then that /dev/kfd exists) and what it does not (the device's mode, and rocminfo, whose status head hides). a_library_load_error_alone_raises_the_wheel_mismatch_on_every_family, the_render_group_verify_accepts_the_group_that_owns_kfd (Unix only: it runs the line in /bin/sh against stand-ins) and diagnose-65 hold it.
  • The fix-8 title cell in skills/rocm-doctor/reference.md is updated. That table is an e2e fixture compared with the CLI's catalog.
  • doctor-catalog.json and the recorded replies under skills/rocm-doctor/evals/fixtures/ are regenerated (cargo xtask catalog, cargo xtask eval-fixtures). contract_version stays at 1 because a title change removes nothing. Beyond the fix-8 title, two recordings carry more. rocm-vllm-import-error also records the fix-8 diagnosis, so its title, summary and remedy comment line change. rocm-current-cli-carries-on and rocm-older-cli-stops-before-diagnosing record the fix-1 diagnosis, so the fix-1 nightly comment line changes.

Verification

New scenarios in tests/e2e-cucumber/features/diagnose.feature. The first two preview a fix with rocm fix <id> --dry-run, which prints the plan before the platform gate, so they run on every lane:

  • diagnose-65: the fix-4 verify line checks the user's groups and shows who owns the GPU compute device.
  • diagnose-66: the fix-8 recipe title names the X.Y release, not the major version.
  • diagnose-67: rocm diagnose --symptom "libamdhip64.so.6: cannot open shared object file" prints fix-8 on every lane, and its title and plan name the X.Y release with no "major" anywhere in the cause block. With the diagnosis wording reverted to "major", the scenario fails on its first check.

Each new test was checked by breaking what it protects, running it, and restoring the tree (every restore was verified with an empty git diff):

Mutation Test It fails with
release_line returns the major only the_wheel_mismatch_compares_the_x_y_release_and_says_so a 6.3 torch beside a 6.4 system no longer reports "Framework links HIP 6.3 but system ROCm is 6.4" (left [])
the fix-8 diagnosis title goes back to "major" the same test "the title must name what is compared"
RENDER_GROUP_VERIFY goes back to the render|video-only grep the_render_group_verify_accepts_the_group_that_owns_kfd (WSL) and diagnose-65 "a member of the group that owns /dev/kfd must pass"; "the verify line must check the group that owns /dev/kfd and list it"
the fix-4 diagnosis uses its own old verify literal the_render_group_fix_verify_matches_the_catalog_copy "the verify line for fix-4-render-group diverged between diagnose and rocm fix"
RENDER_GROUP_VERIFY set back to the recipe's old line diagnose-65 "the verify line must check the groups and list /dev/kfd"
the fix-8 recipe title goes back to "…ROCm/HIP major" diagnose-66 "the title must name the X.Y release it compares"
the fix-8 diagnosis wording goes back to "major" diagnose-67 "the cause's title must name the X.Y release"

The mutations were run on Windows and repeated in WSL2 (the fix-4 code is Linux-only). The unit tests were not run against the code from before this change as a whole, only against each of the mutations above. The tests that cover the comparison and the wording are:

  • the_wheel_mismatch_compares_the_x_y_release_and_says_so: 6.3 against 6.4 fires, 6.4.x against 6.4.1 does not, and neither title says "major".
  • the_render_group_fix_verify_matches_the_catalog_copy: the diagnosis and the recipe carry the same verify line.
Platform Run Result
Windows cargo test --workspace --all-targets (AGENTS.md §8) 6782 pass, 0 fail, 17 ignored
Windows cargo xtask smoke (AGENTS.md §8) smoke: ok
Windows cargo nextest run --workspace --no-fail-fast 6778 pass, 17 skipped
Windows cargo xtask e2e for skill-01 to -07 and diagnose-12, -43, -44, -45, -65, -66, -67 14 of 14 scenarios pass
WSL2 (Ubuntu, Strix Halo) cargo nextest run -p rocm-core 1546 pass
WSL2 cargo xtask e2e, same 14 scenarios 14 of 14 pass
Windows cargo xtask verify-commits, check-architecture-doc, cargo xtask catalog --check, cargo xtask eval-fixtures --check, cargo clippy --workspace --all-targets -D warnings, cargo fmt --check, feature_naming, skill_reference pass

These rows were taken on the rebased commit. Everything above passed in the final run, including the rocm-engine-lemonade archive_cache tests, which fail or hang intermittently on this machine under load, also on main. An earlier run of this branch saw a timing-sensitive rocm-core test (host_gpu::capture::subprocess_runners_pair::output_written_within_the_drain_grace_after_the_child_exits_is_kept_by_all_three) fail once under load and pass alone three times.

Scope

  • fix-1 and fix-13 are reworded here on purpose, although the ticket lists the catalog items separately. They carry the same drift as fix-8 (wording that says "major" where the code compares or selects at X.Y), the new diagnose-67 scenario sits on the same diagnosis text, and splitting them out would leave the catalog saying X.Y in one entry and "major" in its neighbours.
  • The new release_line in diagnose/host.rs is named differently from the older private major_minor in report.rs on purpose: they parse differently (the first X.Y anywhere in the string, against a leading all-digit component), and its doc comment says so.
  • fix-4's diagnosis still carries a newgrp render / re-login note that its recipe lacks, and fix-8's diagnosis and recipe still print different Python verify commands. They are the same kind of drift, outside this change, and are a follow-up.
  • fix-8's Linux remedy still names one wheel index. A user on another X.Y release adapts the URL, as the remedy text now says.
  • Whether fix-8 should fire on a minor-version difference alone is not changed here.

Part of ROCMAI-955.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. No rows reference ROCMAI-955.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. Not applicable: no subcommand or subsystem added.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state, per AGENTS.md §3. The fix-8 wording is held against the X.Y comparison by the_wheel_mismatch_compares_the_x_y_release_and_says_so and by diagnose-67 on the diagnosis side. The fix-4 verify line is one constant for both paths, with a unit test tying the two together and a comment that says what its exit status does and does not check.

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Merge note: #1048, #1051, #1052 and #1053 each add e2e scenarios numbered from diagnose-59 in diagnose.feature, and feature_naming requires the numbers to be sequential. Whichever of the four merges first keeps its numbers. Each of the others renumbers on rebase, and updates the numbers cited in its docs/testing.md section. Their @id: tags are distinct, so only the numbers collide. Their skill-doc follow-ups wait for #1029 to merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@volen-silo
volen-silo force-pushed the rocmai-955-catalog-consistency branch from 01e5fbb to e11f9e1 Compare November 12, 2026 14:31
@volen-silo
volen-silo marked this pull request as ready for review November 13, 2026 06:59
@volen-silo
volen-silo requested a review from a team as a code owner November 13, 2026 06:59
@volen-silo
volen-silo requested a review from rominf November 13, 2026 06:59

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head e11f9e11

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change.
Implements wording/dedup consistency fixes for fix-1 / fix-4 / fix-8 / fix-13, tracked internally as "Part of ROCMAI-955" (ticket content not retrieved here — see What this covered).

Blocking

  • E2E tests (MI300X, current) is red on this head, and it isn't this diff — no single file:line; CI check on commit e11f9e11.
    The job reports 8 "unexpected regressions," all in GPU-serving/runtime scenarios (serve-vllm-inference, serve-anthropic-endpoint-advertised, serve-anthropic-messages-inference, serve-anthropic-endpoint-readiness, serve-anthropic-messages-streaming, chat-end-to-end-local-model, bench-load-real-serve, runtime-lifecycle-activate-restart-services-succeeds) — none of which touch the diagnose/fix-catalog code this PR changes. The same check failed the same way, around the same time window on the same self-hosted runner, on at least four other unrelated branches that day (rocmai-955-exit-codes, rocmai-955-diagnose-engine, docs/rocm-doctor-skill-review, worktree-lemonade-serving-discovery). That pattern points to a shared-runner/infra flake rather than something this diff introduced. Every code-relevant check (clippy, build-and-test, affected-crate tests, the other E2E lanes) is green.
    Confidence 85 (verified via the check-run API and by diffing which scenarios failed against the PR's own file list). Fix: re-run the job; if it fails the same way again, escalate with this evidence rather than changing code.

Non-blocking

None.

Decisions for the author

  • Two differently-behaving major_minor helpers now share a name in this crate — non-blocking-improvement.
    diagnose/host.rs's new fn major_minor(s: &str) -> Option<String> (regex (\d+)\.(\d+), matches anywhere in the string) is not the same function as the pre-existing fn major_minor(version: &str) -> String in report.rs (anchored split on ['.', '-', '+', '_'], requires the leading component to be pure digits). They're correctly scoped to separate, private, module-local functions, and they have to differ — report.rs's version returns empty for inputs like "hip=7.14.60850", which host.rs's depends on parsing — so this isn't a bug. But the identical name in one crate invites a future reader (or a future "let's dedupe these" pass) to assume they're interchangeable. A distinguishing name or a short cross-reference comment on one of them would close that gap; not required for this change.

Positive signals

  • RENDER_GROUP_VERIFY is a clean instance of "one shared constant, one test tying producer to consumer": the doc comment on the constant states precisely what the exit status does and does not check (group membership vs. which group owns /dev/kfd), which stops a future reader from over-trusting the check.
  • The X.Y wording fix is verified at the behavior, not just the string: the_wheel_mismatch_compares_the_x_y_release_and_says_so asserts the 6.3-vs-6.4 mismatch and the 6.4.x-vs-6.4.1 non-mismatch and the absence of "major" in the title in one test, so the comparison and its wording can't quietly drift apart again.

Deployment notes

None beyond the CI item above — this is a text/dedup change with no behavior, API, or migration impact, and that is independently verified true (regex powering the comparison is unchanged from before this PR; only its name and the surrounding prose changed).

What this covered

Full review of all 18 changed files, read completely at head e11f9e11 against merge-base f536619a. Two independent fan-out passes each read a coupled file group completely plus its supporting code: (1) diagnose/host.rs, diagnose/linux.rs, diagnose/windows.rs, fix.rs, fix/catalog.rs, plus the Examination struct and finalize/zero/keyword_score helpers; (2) tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs, docs/testing.md, skills/rocm-doctor/reference.md, doctor-catalog.json, and the 8 touched skills/rocm-doctor/evals/fixtures/*/replies.json files, plus print_recipe/report.rs's header formatting. I additionally read AGENTS.md in full (agent-instruction-adherence pass), checked history on the touched check_8 region (the X.Y-granularity regex predates this PR — introduced in #890 — so this change fixes a stale name/title, not stale behavior), grepped the whole repo for leftover "major" wording and for the renamed major_version symbol (none found outside unrelated contexts), and read live CI check-run status plus cross-branch comparison for the one failing check.

Did not run: the "prior changes" pass (no prior review comments exist to check against) and the ticket-retrieval step — ROCMAI-955 is referenced back only because it already appears in the PR's own title ("Part of ROCMAI-955"); its content was deliberately not pulled into this public review.
Not reconciled against this PR's existing discussion (2 issue comments were not read, per this method's blind-review rule).


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for the review.

E2E tests (MI300X, current). Agreed that it is not this diff. The failing scenarios are all GPU serving and runtime-lifecycle ones, and the same job fails on main's own runs with the same scenarios. It is not a required check. I am not re-running it, since a rerun on the same runner would spend another hour of GPU time without changing what it tests. Every required check is green on e11f9e11.

Two major_minor helpers. You are right that the name is shared: this change renames the existing major_version in diagnose/host.rs to major_minor, and report.rs already has an unrelated private major_minor with different parsing rules. I have not changed it in this PR. Both are private to their modules, they have to differ, and the review confirms nothing is wrong. If this PR gets another push, I will add a one-line cross-reference comment on each so a future dedupe pass does not treat them as interchangeable.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking correctness problems found: the major_version -> major_minor rename keeps the regex and output identical, the new unit test covers 6.3 vs 6.4 firing and 6.4.x vs 6.4.1 not firing, and the fix-4 verify line now comes from a single RENDER_GROUP_VERIFY constant tied to both copies by a test and scenario diagnose-59. Fixture/doc updates (reference.md, catalog JSON, evals, docs/testing.md) look in sync.

Requesting changes on process grounds before merge:

  1. Show the new tests fail when their target is broken. The PR notes the new tests were not run against pre-change code. AGENTS.md §3 asks for a break-the-branch check; diagnose-61 is reportedly red with the wording reverted, but please confirm the same for the X.Y comparison unit test and diagnose-59/60, and say so in the PR text.
  2. CI: E2E tests (MI300X, current) is red. It ran for 1h20m and failed rather than being cancelled, so it is not obviously a runner-pool problem. Please check the log and confirm it is not a diagnose/skill scenario failure, or rerun.
  3. Stale wording (nit, inline). One doc comment still says "major".
  4. Scope. The ticket asks for one PR per item; this also rewords fix-1 and fix-13. It is disclosed and the same drift class, so fine if you prefer to keep it, but worth a note in the PR body that it is deliberate.

The fix-8 Linux remedy still names a single wheel index; agreed that is out of scope here.

Comment thread crates/rocm-core/src/diagnose/host.rs Resolved
@volen-silo
volen-silo force-pushed the rocmai-955-catalog-consistency branch from e11f9e1 to d3e4dc6 Compare November 13, 2026 11:06
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. Taking the four points in order. The change for point 3 is in d3e4dc6; the rest is evidence and description.

  1. Break-the-branch evidence. Done, and recorded in the PR description under Verification. Each of these was broken, the test run, and the tree restored (an empty git diff after each):
    • major_minor returns only the major: the_wheel_mismatch_compares_the_x_y_release_and_says_so fails (a 6.3 torch beside a 6.4 system no longer reports the mismatch).
    • The fix-8 diagnosis title back to "major": the same test fails ("the title must name what is compared"), and diagnose-61 fails on its first check.
    • The fix-4 diagnosis using its own old verify literal: the_render_group_fix_verify_matches_the_catalog_copy fails.
    • RENDER_GROUP_VERIFY set back to the recipe's old line: diagnose-59 fails.
    • The fix-8 recipe title back to "major": diagnose-60 fails.
  2. E2E tests (MI300X, current). It is not a diagnose or skill failure. In the run for e11f9e11, 414 scenarios ran, 402 passed and 12 failed, with 8 unexpected failures. Every failing step is a model-serving, background-serve or runtime scenario. This PR's own scenarios, diagnose-59, -60 and -61, passed in that same run. The same lane also fails on main's own runs. I have not re-run it: it is not a required check, and a rerun would spend another hour or more of GPU time without testing anything this PR changes. All required checks are green.
  3. Stale wording. Fixed at host.rs:580, see the inline reply. I also added a one-line comment on each of the two major_minor helpers (diagnose/host.rs and report.rs) saying they differ on purpose, which an earlier review of this PR suggested.
  4. Scope. Deliberate, and now said so in the PR description's Scope section: fix-1 and fix-13 carry the same "major" versus X.Y drift as fix-8, diagnose-61 sits on the same diagnosis text, and splitting them out would leave the catalog saying X.Y in one entry and "major" in its neighbours.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 13, 2026 11:09

Addressed in d3e4dc6: each point is answered in #1053 (comment) (break-the-branch evidence is in the PR description, the MI300X failure is not a diagnose or skill scenario, the stale wording is fixed, and the scope is stated). Dismissing so the review can be re-requested on the new head.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the single RENDER_GROUP_VERIFY constant feeding both diagnose and the recipe, and the unit test pinning the X.Y comparison next to the title text, are the right way to stop this drifting. I found no correctness problems in the logic. I'm requesting changes for a few things to sort out before merge:

To address

  • Rebase. The PR currently reports as conflicting with main (one commit behind). Please rebase and resolve.
  • diagnose-61 assertion is broader than the claim. It rejects any line in the whole fix-8 cause block containing "major", so an unrelated future sentence would fail the scenario. See the inline comment.
  • No CI evidence for this head. Only the docs build has reported; the Rust/e2e/commit-signature checks haven't run on d3e4dc62. The Windows/WSL2 runs in the description can't be verified from here, so please make sure the full checks go green after the rebase.

Worth considering

  • major_minor now exists in rocm-core twice (here and in report.rs) with different semantics. The cross-reference comments help, but a distinct name here (e.g. release_line) would remove the need for them.
  • RENDER_GROUP_VERIFY still ends in rocminfo | head, so its exit status reflects head, not GPU enumeration. It's pre-existing and the new doc comment says so honestly; just noting the verify line proves a bit less than it may read as.
  • The mutation checks in the description were run on Windows only, while fix-4 is Linux-only code (diagnose-59 passes everywhere only because of --dry-run). Fine as disclosed, but a Linux mutation run would be reassuring.
  • The fix-1/fix-13 rewording goes beyond the fix-8/fix-4 items in the ticket. The description discloses it and the theme is coherent, so I'm fine with it staying in this PR.

Note for consumers: titles and recipe comments in rocm diagnose --json / rocm fix --json change for fix-1, 4, 8 and 13 (ids unchanged), so anything matching on the old fix-8 title string would notice.

plan.contains("X.Y release"),
"the plan must name the X.Y release: {plan}"
);
let says_major: Vec<&str> = block

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rejects any line in the whole cause block containing "major", so an unrelated future sentence (e.g. a Python-major remark) fails the scenario. Consider asserting only on the title and plan lines, or on the phrase "ROCm major"/"HIP major".

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 76ff987. diagnose-64 (this scenario, renumbered after rebase) no longer scans the whole cause block for "major". It now checks only the cause title line and the plan: lines, so an unrelated sentence elsewhere in the block cannot fail it. I kept the title check because that is the line that carried the wrong word.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head d3e4dc6

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change. Implements two specific follow-up items from ROCMAI-955 (fix-8's "major" vs X.Y wording, and fix-4's duplicated verify-line), not the rest of that tracker.

Blocking

None

Non-blocking

None

Decisions for the author

  • Bundling fix-1/fix-13's wording fix alongside fix-8/fix-4 in one PR — tradeoff. fix-1, fix-13 and fix-8 carry the same "major" vs "X.Y" wording drift, and the new diagnose-61 scenario sits on the same diagnosis text, so bundling catches the whole pattern in one pass. Splitting per catalog entry would make each change smaller and independently revertable. Confirm this grouping is the one you want reviewed and merged as a unit.
  • major_minor is defined twice under the same name in different modules — non-blocking-improvement. crates/rocm-core/src/diagnose/host.rs and crates/rocm-core/src/report.rs each have a private major_minor with a different algorithm (first X.Y anywhere in the string vs. a leading all-digit component), and each now carries a doc comment pointing at the other. That resolves the ambiguity for a reader who opens either doc comment, but not for one who greps the bare symbol name and reads only the first hit. A more distinguishing name on one of them would remove the ambiguity without depending on the comment being read.

Positive signals

  • RENDER_GROUP_VERIFY (crates/rocm-core/src/fix/catalog.rs) is now a single constant feeding both the static recipe table and the live diagnosis Fix, backed by a unit test (the_render_group_fix_verify_matches_the_catalog_copy) that builds a real Fix via check_4_render_group and compares it against the catalog's own independently-assembled verify string — exactly the kind of test that would have caught the original drift (the catalog copy was missing ls -l /dev/kfd) and will catch it again if it recurs.
  • The doc comment on host.rs's major_minor explicitly names the algorithmic difference from report.rs's same-named function, rather than leaving two private functions with an identical name for a future reader to puzzle over unassisted.

Deployment notes

None — this only changes CLI-printed wording, one shared string constant, and test/fixture text; no schema, config, or API-surface changes.

What this covered

Read the full diff and all six touched Rust source files (crates/rocm-core/src/diagnose/{host,linux,windows}.rs, crates/rocm-core/src/fix.rs, crates/rocm-core/src/fix/catalog.rs, crates/rocm-core/src/report.rs) plus the test/doc surface (tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs, docs/testing.md, skills/rocm-doctor/reference.md, doctor-catalog.json, and the 7 regenerated skills/rocm-doctor/evals/fixtures/*/replies.json) completely, against f536619a…d3e4dc62. Two independent fan-out passes covered the same two groupings in parallel and were each separately re-confirmed directly by the workers after being handed only file paths — never this review's conclusions or the PR's own description. Also grepped the rest of the repo (README, docs/architecture.md, engines/, apps/*/therock) for any other "ROCm major"/"HIP major" wording this sweep might have missed; everything found there is a genuinely different, out-of-scope "major" concept (SDK layout selection, vLLM build-table matching, OS version grouping).

Did not run: no local Rust toolchain was available in this environment, so no build or test suite was executed directly; findings instead rely on tracing the code paths and the literal strings each test asserts against. CI on this PR currently reports only a docs build (Read the Docs, passing); no workflow runs the Rust test/lint suite as a GitHub check here, so that gap is noted rather than assumed covered.

Ticket ROCMAI-955 was retrievable and is a multi-item follow-up tracker from an earlier review; this PR implements two of its items (the fix-8 wording and the fix-4 verify-line dedup), consistent with its own "Part of ROCMAI-955" framing in the PR description — it does not claim to close the ticket, and does not need to.

Not reconciled against any discussion on this PR (existing review threads, including an earlier review from this same automated identity at a prior commit, were deliberately not read before reaching these findings, per this method's blind-review rule).


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@volen-silo
volen-silo force-pushed the rocmai-955-catalog-consistency branch 2 times, most recently from cbae336 to 76ff987 Compare November 16, 2026 09:20
@volen-silo
volen-silo dismissed juhovainio’s stale review November 16, 2026 09:21

Addressed in 76ff987: rebased, diagnose-64 narrowed to title and plan lines, major_minor renamed to release_line, mutations repeated in WSL2. Replies: #1053 (comment) and the inline thread.

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Round 2 fixes are in 76ff987, rebased onto main after #1067 (the generated fixtures were regenerated; catalog --check and eval-fixtures --check pass).

  • Rebase: done.
  • Broad "major" assertion: narrowed to the title and plan: lines, see the inline reply.
  • CI evidence: checks can start on this head now that the branch no longer conflicts; I am following them to green.
  • major_minor name clash: renamed to release_line, with a doc comment saying why X.Y is the unit and how it differs from the one in report.rs.
  • Linux mutation run: the mutations were repeated in WSL2; the description lists them.
  • rocminfo | head exit status: not changed. It is pre-existing and the constant's doc comment says what the status does and does not check; changing the command is a separate behaviour change.
  • fix-1/fix-13 rewording: stays in this PR, as you said.

The dismissal message of review 5890553017 links an earlier comment by mistake; this comment is the reply to that review.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix-8 rename and the single fix-4 verify constant look correct, and the scenarios, reference.md, catalog JSON and fixtures are updated together. Requesting changes on these points before merge:

  1. fix-4 verify is narrower than the diagnosis. The diagnosis builds its usermod command around whichever group owns /dev/kfd (it can be something other than render/video), but the shared verify line greps only ^(render|video)$. A user fixed through a third group still fails verify. This predates the PR, but it now becomes the one shared behavior for both surfaces, so either make the verify cover the owning group or state the limit in the user-facing text.
  2. Scope beyond ROCMAI-955. The ticket lists only fix-8 and fix-4; this also rewords fix-1 and fix-13 and regenerates three fix-1 skill fixtures. The PR body discloses it, but please confirm the extra churn in recorded eval replies is intended, or split it out.
  3. Unverified doc claim. docs/testing.md says a library-load error in the symptom raises fix-8 on its own on Linux, Windows and WSL alike. That depends on the keyword score alone reaching the match threshold with --top 20; the E2E lanes were still pending when I looked, so I could not confirm it.

Two smaller notes are inline. This is based on reading the diff and CI status, not a local run.

/// Unlike `report.rs`'s `major_minor`, which reads only a leading all-digit
/// `X[.Y]` and keeps a bare major, this takes the first `X.Y` anywhere in the
/// string and returns `None` when there is no minor.
fn release_line(s: &str) -> Option<String> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rename and doc comment are right. Note it returns None when there is no minor, so a framework reporting only hip=7 silently skips check 8 (same as before). Fine to keep; consider saying so in the comment if that is intended.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 041675d: the release_line doc comment now says it returns None when there is no minor, so a framework reporting only hip=7 skips check 8 as before, and the_wheel_mismatch_compares_the_x_y_release_and_says_so pins that case.

Comment thread crates/rocm-core/src/fix/catalog.rs Outdated
/// `head`'s status. `ls -l /dev/kfd` is there for the user to read: the
/// diagnosis also fires when `/dev/kfd` belongs to a group other than `render`,
/// and its output names the owning group and the mode.
pub(crate) const RENDER_GROUP_VERIFY: &str =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This greps only render|video, while the diagnosis may add the user to another kfd owner group (kfd_group). The comment is honest about it, but a user fixed via that group still fails verify; consider covering the owning group.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 041675d: the shared verify line now also accepts the group that owns /dev/kfd (stat -c %G /dev/kfd), so a user fixed through that group passes. It is tested by running the line in /bin/sh against stand-ins (member of the owning group passes, none fails, an UNKNOWN owner or failing stat does not let a user in none pass) and by diagnose-65.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (base 20914c735464c8b3f60a7bbcee4e0432732af563 → head 76ff9874b0e7b486d9726888b03173505934dafa). Part of ROCMAI-955 (a multi-PR follow-up ticket); the PR implements two of its bullets (fix-8 X.Y wording, fix-4's single verify line) plus the same wording drift in fix-1/fix-13, and says so — it does not claim to close the ticket.

Blocking

None

Non-blocking

  • The PR description's verification table still cites the wrong scenario IDs — PR body, "Verification" section. The three new scenarios this PR actually adds are diagnose-62, diagnose-63, diagnose-64 (confirmed in tests/e2e-cucumber/features/diagnose.feature, and the body's own "What changes"/mutation-table rows now correctly name them). However the final Platform | Run | Result table still reads: cargo xtask e2e for skill-01 to -07 and diagnose-12, -43, -44, -45, -59, -60, -61 → "14 of 14 scenarios pass" (two occurrences, Windows and WSL2 rows). diagnose-59/-60/-61 are pre-existing scenarios about --symptom-file paste handling (added by #1048, unrelated to fix-4/fix-8), not this change's new scenarios — the arithmetic (7+4+3=14) only works by substituting the wrong IDs. As written, the cited run's "14 of 14 pass" evidence does not name a run of this PR's own new behavior. Confidence 85 (direct text comparison against the diff and against diagnose.feature's actual @id: tags). Fix: update the two table rows to name diagnose-62, -63, -64 (or re-run and requote if a different scenario set was actually exercised).

Decisions for the author

  • The "ROCm 10 has no released PyTorch index yet" line keeps saying "major" beside text that now consistently says "X.Y release" — tradeoff. The PR's own "Scope" section explains this is deliberate (there it means the major, not an X.Y release). Readable as intended, but worth confirming the juxtaposition (X.Y release / major / X.Y release) still reads clearly to a user scanning the remedy quickly.

Positive signals

  • release_line's doc comment gives a concrete worked example and a one-line pointer to the sibling major_minor in report.rs explaining exactly how the two differ in parsing rule — the kind of comment that stops a future reader re-deriving the distinction from scratch.
  • Consolidating fix-4's two divergent verify-line copies into one RENDER_GROUP_VERIFY constant, with a unit test tying the diagnosis and recipe together, directly closes the class of drift the PR is about rather than just the one instance.

Deployment notes

None — wording-only change to diagnosis/fix text and one internal constant; doctor-catalog.json's contract_version is unchanged.

What this covered

Full change (18 files, +286/-49) read completely: the core logic group (diagnose/host.rs, diagnose/linux.rs, diagnose/windows.rs, fix.rs, fix/catalog.rs) and its callers/siblings; the consumer/fixture group (doctor-catalog.json, skills/rocm-doctor/reference.md, all 8 touched evals/fixtures/*/replies.json, docs/testing.md, tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs) as independent passes, plus a history pass (git log/blame) on the five core-logic files. The rename (major_version→release_line) is complete repo-wide with no stale references; the RENDER_GROUP_VERIFY constant has exactly one definition and both call sites use it; all fixture/catalog text matches the source it's generated from field-by-field; the new scenarios' step definitions are real assertions, not stubs. Verified this PR's content is unchanged from an earlier push reviewed at commit cbae336875c3611befa86c3c7f57f2a742de4a78 (this branch was rebased onto a newer main since; byte-for-byte diff-of-diffs confirms only blob hashes/line offsets moved, no logical content changed, aside from the scenario numbers shifting from 59-61 to 62-64 because main had meanwhile merged an unrelated feature that claimed those numbers first — the PR description was partly, but not fully, updated for that, per the finding above).

Checked CI for the current head: all non-GPU checks (clippy, fmt-equivalent, build-and-test, coverage, security scans, skill checks) are green. The self-hosted GPU E2E workflow (Strix Halo Windows/WSL2, MI300X) was still "pending" for this exact head at review time and had not produced a result; the immediately-prior push's run of that same suite (commit cbae336..., same content) reported one unexpected (non-xfail) failure, serve-anthropic-endpoint-readiness on the Windows Strix Halo lane — a scenario about Anthropic-endpoint serve readiness, unconnected to this PR's diagnose/fix catalog wording changes, and not yet covered by an expectations.toml entry for that OS. Not scored as a blocking finding here since no failure is recorded against the actual reviewed commit, but worth watching once that run completes.

Did not run: an actual cargo test/cargo xtask e2e execution myself (relied on reading CI's own results, per method, plus tracing the regex/constant logic by hand). Prior-changes pass (do earlier review comments on these files apply here) not run as a separate pass — its ground is covered by the history pass's git-log read of these same five files. Not reconciled against this PR's review-comment discussion; this report is blind to that discussion. Earlier automated-review rounds exist on this PR at older commits (e11f9e11, d3e4dc6) — this round re-derived everything fresh.


🤖 Automated by an AI agent

fix-8-wheel-rocm compares the framework's HIP version with the system
ROCm (or HIP SDK) at X.Y, while both of its titles and its remedy text
said "major". The comparison is the intended one: ROCm PyTorch wheels
are built and indexed per X.Y (whl/rocm6.4), the function's own doc said
it extracts X.Y, and the first remedy offered rocm6.3 and rocm6.4 side
by side. So the wording changes, not the behaviour, and major_version
becomes release_line. The same wording is corrected next to it where the
advice is also per X.Y: fix-13's "HIP SDK matched to your framework's
HIP" (the SDK check_8 compares against on Windows) and fix-1's "nightly
for the ROCm you have" (nightly indexes are per X.Y too).

fix-4-render-group printed two different verify lines: the one from
`rocm diagnose` listed /dev/kfd and the `rocm fix` recipe did not. Both
now read one constant. It also accepts the group that owns /dev/kfd,
which the diagnosis tells the user to join and which need not be
render: the line passes when the user is in that group, render or video
and /dev/kfd exists, and `ls -l /dev/kfd` shows the owner and mode.

Unit tests hold the diagnosis copy to the recipe for fix-4 and run its
line in /bin/sh with stand-in tools, and pin the X.Y comparison and the
titles for fix-8. diagnose-65 and diagnose-66 assert what
`rocm fix --dry-run` prints for each, and diagnose-67 the fix-8 cause
`rocm diagnose` prints for a library-load failure. The catalog manifest
and the skill eval recordings are regenerated.

ROCMAI-955

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
volen-silo force-pushed the rocmai-955-catalog-consistency branch from 76ff987 to 041675d Compare November 16, 2026 11:33
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Round 3 is in 041675d, rebased onto main after #1051 (scenarios renumbered to diagnose-65 to 67).

  1. fix-4 verify narrower than the diagnosis: fixed. The shared RENDER_GROUP_VERIFY now also accepts the group that owns /dev/kfd (stat -c %G /dev/kfd), next to render and video, still one static string for both rocm fix and rocm diagnose. A new Unix-only test runs the line in /bin/sh against stand-ins: a member of the owning group passes, a render member passes, a user in none fails, and an UNKNOWN owner or a failing stat does not let a user in none pass. Reverting to the old grep makes it and diagnose-65 fail. The doc comment says what the status now checks and still does not check (the device mode, and rocminfo, whose status head hides).
  2. Scope beyond the ticket (fix-1/fix-13 and their fixtures): intended, and kept in this PR. The two entries carry the same "major" vs X.Y drift as fix-8, the new scenario sits on the same diagnosis text, and the three fix-1 recordings change only by the regenerated nightly line. Splitting them out would leave the catalog saying X.Y in one entry and "major" in its neighbours.
  3. Unverified docs/testing.md claim: checked against the code and rewritten to the facts. The library-load error alone scores 75 from keywords (libamdhip64.so 50, the loader failure 25) against a match threshold of 50, and check 8 is registered for Linux, Windows and WSL. A new test, a_library_load_error_alone_raises_the_wheel_mismatch_on_every_family, runs the real diagnose for all three families, and diagnose-67 passed on Windows and in WSL (no bare-metal Linux run).

Inline notes: release_line now documents that it returns None when there is no minor, so a framework reporting only hip=7 skips check 8 as before, and a test pins it. The description is updated.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 16, 2026 11:34

Addressed in 041675d: fix-4 verify now covers the /dev/kfd owning group (tested), docs/testing.md claim checked against code and rewritten, release_line None case documented; fix-1/fix-13 stay in this PR as disclosed. Replies: #1053 (comment) and the two inline threads.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I read the diff and found nothing that should block the merge.

Two small things you may want to tidy up, neither blocking:

  • docs/architecture.md still describes diagnose/test_support.rs as "Examination builders", but this PR moves the shell stubs (stub_bin_dir, run_in_sh) into it. check-architecture-doc only checks that an entry exists, so it won't catch the stale description.
  • A nit on the RENDER_GROUP_VERIFY doc comment, left inline.

One behavior note: the fix-4 verify now accepts any group that owns /dev/kfd, not just render/video. The PR body already calls this out, and the shell test covers it on unix.

This is a read of the diff only. CI was still running when I looked, so please make sure it ends up green before merging.

/// GID with no name) adds only a literal `UNKNOWN`. It checks membership, not
/// the device's mode, and it does not check `rocminfo`: `rocminfo | head`
/// reports `head`'s status. `ls -l /dev/kfd` prints the owning group and mode
/// for the user to read. The `\$` keeps the anchor literal in sh, bash and fish.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this says the \$ anchor works in "sh, bash and fish", but fish only supports $(...) command substitution from 3.4 onward. Either say "fish 3.4+" or drop fish from the claim.

@volen-silo
volen-silo enabled auto-merge November 16, 2026 11:52
@volen-silo
volen-silo added this pull request to the merge queue Nov 16, 2026

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (base ee605065115a24958d1b92e1f54df87ccbe4a5b7 → head 041675d57927e78ac169e1f26cb0e03ab83bf398). Part of ROCMAI-955 (ticket named but not retrievable from here); the PR implements two of that tracker's items (fix-8's "major" vs X.Y wording, fix-4's duplicated verify-line) plus the same wording drift in fix-1/fix-13, and says so — it does not claim to close the ticket.

Blocking

None

Non-blocking

None

Decisions for the author

  • Bundling fix-1/fix-13's wording fix alongside fix-8/fix-4 in one PR — tradeoff. The ticket lists catalog items separately; fix-1/fix-13 carry the identical "major" vs "X.Y" wording drift and the new diagnose-67 scenario sits on the same diagnosis text, so bundling catches the whole pattern in one pass at the cost of a larger diff than the ticket strictly asks for. The PR's own "Scope" section already discloses and justifies this; noted here only so a merging maintainer sees it was deliberate, not scope creep.
  • A major_minor-style helper exists under two different names with genuinely different parsing rules (release_line in diagnose/host.rs vs major_minor in report.rs) — non-blocking-improvement. Each carries a doc comment pointing at the other's differing algorithm (first X.Y anywhere in the string vs. a leading all-digit component), which resolves the ambiguity for a reader who opens either comment, but not for one who greps the bare symbol and reads only the first hit. Not required; a more distinguishing name on one side would close the remaining gap without depending on the comment being read.

Positive signals

  • RENDER_GROUP_VERIFY (crates/rocm-core/src/fix/catalog.rs) now feeds both the static recipe table and the live diagnosis from one constant, backed by a test that builds a real Fix and compares it against the catalog's independently-assembled copy — exactly the test that would have caught the original drift and will catch it again if it recurs.
  • The same change fixes a real pre-existing bug while deduplicating: the verify line now accepts whichever group actually owns /dev/kfd, not just render/video, matching what the diagnosis's own usermod recipe already builds around — a user fixed through a third group used to fail verify even after a correct fix.
  • release_line's doc comment gives a concrete worked example and explains why X.Y (not major) is the right unit (PyTorch publishes wheel indexes per X.Y). Checked by history (git blame): the regex itself is unchanged since the function's introduction, so the rename and wording fix carry no behavior change, as the PR claims.
  • Every new or changed user-facing string (fix-1, fix-4, fix-8, fix-13 titles/remedies) is consistent end-to-end: source → doctor-catalog.json → skills/rocm-doctor/reference.md → all 8 touched eval fixtures → the new Cucumber scenarios' step assertions. No stale "major" wording or stale major_version symbol name found anywhere else in the repo, including in docs/wsl.md and docs/manual-testing.md, which reference these fix IDs but not their reworded text.

Deployment notes

  • rocm diagnose --json / rocm fix --json consumers: titles and recipe/remedy text for fix-1, fix-4, fix-8 and fix-13 change (ids unchanged); anything matching on the old fix-8 title string will need updating. Already called out in the PR body.
  • No schema change: doctor-catalog.json's contract_version correctly stays at 1 — a title/wording change is neither an addition nor a removal under the project's own version-bump rule.

What this covered

Full change (19 files, +440/-87) read completely across two independent fan-out workers grouped by coupling: (1) the core logic group — diagnose/{host,linux,windows,test_support}.rs, fix.rs, fix/catalog.rs, plus report.rs and the call sites of every renamed/changed symbol; (2) the consumer/fixture group — docs/testing.md, doctor-catalog.json, skills/rocm-doctor/reference.md, all 8 touched evals/fixtures/*/replies.json, tests/e2e-cucumber/features/diagnose.feature, tests/e2e-cucumber/tests/e2e/diagnose_steps.rs — cross-checked field-by-field against the source behavior each describes. Each worker also ran a code-comments-compliance pass over its own files, and the core-logic worker ran a history pass (git log/blame) on the touched regions. I separately ran a repo-wide grep for stale major_version/"ROCm major" wording and for other docs referencing the reworded catalog entries, and read CI check-run status for this exact head: every check (clippy, build-and-test across all lanes, E2E tests and the E2E consolidated report, security scans, commit-signature check, docs build) reports success, skipped, or neutral — none failing.

Did not run: a full re-read of AGENTS.md myself beyond its module-size and doc-sync rules (both fan-out workers independently checked compliance against the sections relevant to their files). Prior-changes pass not run — no mechanism available to check earlier review comments on these specific files outside this PR's own (excluded) discussion. Did not execute cargo test/cargo xtask e2e myself; relied on tracing the code paths plus the green CI run at this head, which already ran them.

Not reconciled against this PR's existing discussion — prior review rounds, including several from this same account at older commits, were deliberately not read before reaching these findings, per this method's blind-review rule.


🤖 Automated by an AI agent

Merged via the queue into main with commit f15f9b4 Nov 16, 2026
40 checks passed
@volen-silo
volen-silo deleted the rocmai-955-catalog-consistency branch November 16, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants