Skip to content

fix(sdk): treat a two-component version pin as that release line, and suggest a ROCm 10 release only when one can help - #819

Open
rominf wants to merge 15 commits into
mainfrom
fix/sdk-two-component-version-pin
Open

rominf wants to merge 15 commits into
mainfrom
fix/sdk-two-component-version-pin

Conversation

@rominf

@rominf rominf commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes #808. rocm install sdk --version 10.1 stayed on the canonical 7.x index, because 10.1 does not parse as major.minor.patch. It then suggested --channel nightly, which refuses ROCm 10. This PR makes two related changes: it accepts a major.minor pin as that release line (the fix #808 asks for), and it reworks what a failed ROCm 10 resolution prints, so the recovery advice names a release that exists and only appears when another release can help. The second part is what replaced the --channel nightly advice #808 reports; it grew through review into a typed failure marker and its own module (therock/resolve/next_hint.rs).

Since #1033, main routes every request through layout_serving, and an unpinned install already reads the ROCm 10 (next) layout. This PR builds on that routing: a two-component pin is one more stable pin that layout_serving places.

User-visible changes:

  • Two-component pins select a release line. --version 10.1 routes to the next layout like --version 10.1.0 and selects the newest stable release in the 10.1 line, with the same exact-arch requirement. --version 7.14 does the same on the canonical index. A major.minor.patch pin, including one with a prerelease suffix, stays an exact match. A prerelease such as 10.1.0a20260822 is never read as a line pin. A wheel line pin also matches a four-component publish such as 7.2.4.70204. On the nightly channel a line pin is refused with its own major (ROCm 11 and newer stable releases ... for 11.2). Line pins apply to every major on purpose: --version 7.14 would otherwise be an exact pin that no published release matches.

  • Tarball line pins pick by version. An unpinned tarball pick is still the newest upload. A line pin picks the newest stable release in the line, even when an older patch was uploaded later.

  • A failed ROCm 10 resolution suggests a release only when another release can help. The next-layout wheel and tarball failures no longer offer --channel nightly. On the release channel they list the stable ROCm 10 releases the next layout publishes, and suggest the newest, for:

    • a pin that no published release matches
    • a release with no torchaudio for this Python and platform
    • a uv pip compile that finds no installable package set for the release
    • a tarball catalog with no archive for the request

    The list leaves out the release the failed request attempted: the resolver records its own pick on the error, so the hint never re-derives it with a different ordering. It also leaves out anything a release-channel re-run would not route back to the next layout. Any other failure gets no version advice, because no release can clear it: an unknown device target, a uv that could not be fetched, an unreachable index, or an unpinned install where no build publishes an installable torchaudio (the unpinned pick already skips builds without a full stack, so every other release would fail the same way). A --channel nightly install or update that fails on the next layout gets no suggestion either: that layout publishes only prerelease builds for nightly, which the stable-release list cannot name, and a --channel release command would move the user off the channel they asked for.

    A uv pip compile failure is classified by the phrase "No solution found" in uv's output. therock-next-32 drives a real uv to that failure, so a uv that rewords it fails that scenario instead of silently dropping the advice.

  • The suggested command keeps the caller's shape. A pinned install is told which --version/--family to re-run with. rocm update, which has neither flag, is given a whole rocm install sdk --channel release --format <wheel|tarball> --version X --family <arch> command, with --devel when the runtime had the toolchain. A runtime that recorded no exact arch gets no version advice.

  • Help and docs. The --version help text, README.md, docs/testing.md and docs/manual-testing.md describe line pins and which failures get a suggestion.

Behavior change in an existing module: torchaudio_pin (therock/resolve/torchaudio.rs, already on main) now raises its "no torchaudio built for ROCm X" failure as the VersionUnsatisfied marker instead of a plain error, which is what makes that failure eligible for the release list above. The new module is therock/resolve/next_hint.rs, which holds the marker, the hint and the wheel failure wording, so wheel.rs stays under the module-size limit; it has a docs/architecture.md entry.

Coverage

All of these are loopback-fixture previews with no GPU tag, so the default e2e lane runs them. therock-next-26 and therock-next-27 are @requires-os:linux, because Windows refuses tarball installs.

  • @id:therock-next-24-two-component-line-pin: a wheel line pin previews 10.1.2, the newest stable build of the 10.1 line, not an older 10.1 build, a 10.1 prerelease or a 10.10 build.
  • @id:therock-next-25-unsatisfied-line-pin-skips-nightly: an unsatisfiable 10.1 pin fails without suggesting nightly or a 10.1 version, and re-running the suggested --version/--family succeeds.
  • @id:therock-next-26-tarball-line-pin-takes-newest-release: a tarball line pin takes 10.1.2 over a later-uploaded 10.1.0.
  • @id:therock-next-27-unsatisfied-tarball-line-pin-suggests-a-published-archive: an unpublished tarball line lists the archives the catalog publishes for the family and suggests the newest with the exact arch. The scenario re-runs that suggestion successfully.
  • @id:therock-next-28-update-with-no-exact-arch-gets-no-version-advice: an update of a runtime that recorded no exact arch fails without a --version or --family suggestion.
  • @id:therock-next-29-exact-pin-ignores-later-patches: --version 10.1.0 stays on 10.1.0 when 10.1.2 is published.
  • @id:therock-next-30-canonical-line-pin: --version 7.14 takes 7.14.1 from the canonical index, not 7.14.0, the 7.14.2rc1 prerelease or 7.15.0.
  • @id:therock-next-31-unpinned-install-with-no-installable-torchaudio-suggests-no-version: an unpinned install where no build has an installable torchaudio names the missing torchaudio and suggests no --version. It goes red with the guard removed.
  • @id:therock-next-32-failed-update-suggests-an-install-that-keeps-its-extras: an update whose newest release lists every package but has no installable set (a real uv pip compile reports no solution) suggests rocm install sdk --channel release --format wheel --version 10.0.0 --family gfx1200 --devel, leaving out the release that failed. The scenario previews exactly that command and checks it resolves 10.0.0 with the device extras and the toolchain. It goes red when the "No solution found" classification stops matching.

The nightly gate has no scenario, because no loopback fixture stands in for the nightly next layout's catalogs; a_nightly_next_layout_failure_suggests_no_release and the nightly case in tarball_failure_on_next_layout_suggests_a_published_release cover it, and both go red with the gate removed.

Unit tests:

  • two_component_version_pin_routes_and_matches_its_release_line covers routing, matching (including a same-minor, other-major release such as 11.1.0 against 10.1), the exact-arch and nightly refusals (including the major an 11.2 pin prints), the u32 overflow guard, four-component matching, 10.01 and post-releases.
  • next_layout_unsatisfied_hint_only_suggests_releases_that_were_not_tried and an_update_is_given_an_install_command_that_keeps_its_format_and_extras cover each state the hint can be in, both re-run wordings and the update command for both formats.
  • the_version_clearable_failures_are_marked_and_the_rest_are_not, a_next_layout_failure_no_version_can_clear_suggests_no_version and the_marker_survives_added_context_and_keeps_its_message cover which failures carry the suggestion, and a_nightly_next_layout_failure_suggests_no_release covers the nightly gate.
  • release_line_pin_prefers_the_newest_stable_tarball_in_the_line covers the tarball ordering, pinned and unpinned.
  • install_sdk_version_help_describes_line_and_exact_pins pins the help text.

Test plan

  • cargo fmt --all --check
  • cargo clippy --locked --workspace --all-targets -- -D warnings
  • cargo xtask check-module-size and cargo xtask check-architecture-doc
  • cargo test --locked --workspace --all-targets: all pass except tests that fail on this host only and that this change does not touch: service_control::tests::stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails (a real docker/podman install), and two build_script git_watch_paths_* tests (global commit signing in temporary repos).
  • cargo xtask e2e -- -n "therock-next": every loopback scenario passes, including therock-next-32 (23 of 23 in 10 to 32 on the final head). The live @requires-gpu scenarios therock-next-07 and therock-next-09 did not pass on this host: 07's install step passed but torch then saw no usable GPU, and 09's rocm engines install vllm step needs an interactive sudo. Neither reaches the code this PR changes.
  • cargo xtask smoke was not run locally: this host's rustc wrapper and network-namespace limits break its isolated build.
  • CI on this head.

@rominf
rominf requested a review from a team as a code owner October 30, 2026 16:29
@rominf
rominf requested a review from fredespi October 30, 2026 16:29

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · a754cac

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (apps/rocm/src/therock.rs, apps/rocm/src/main.rs, README.md, tests/e2e-cucumber/features/therock_next_generation.feature, tests/e2e-cucumber/tests/e2e/therock_steps.rs), base 3dec0eb → head a754cac.
No ticket named. The PR description accurately describes the change's scope and honestly flags its own test-plan gaps (clippy/full cargo test not run locally, GPU lanes not exercised) rather than overclaiming — no mismatch found.

Blocking

  • CI gate "Commit signatures + sign-off" is failing on the current head commit — this run
    cargo xtask verify-commits --base origin/main --require-verified reports: a754cac3 ... -> signature is not GitHub-"Verified". The commit carries a DCO Signed-off-by trailer but is not cryptographically signed / not GitHub-"Verified", which this repo's own AGENTS.md documents as a required, blocking gate ("a blocking CI gate (commit-signatures) re-checks every PR with GitHub 'Verified' status").
    Confidence 100 · mechanical · Fix: author signs the commit (SSH or GPG, with the key registered on GitHub) and re-pushes, per the job's own remediation text.

  • published_in_stable_line can't recognize a stable release published with more than three dot-separated components, contradicting its own sibling function's documented contract — apps/rocm/src/therock.rs:212-219 (published_in_stable_line), via stable_release_line_pin (:199-204) and the strict parse_version (:7012-7047). version_request_matches's doc comment (:180-186) promises a two-component pin "selects every stable release in that major.minor line," but parse_version("7.2.4.70204") returns None (its splitn(3,'.') leaves .70204 as an unparseable stage suffix), and stable_release_line_pin's own fallback also returns None on the same string (rest after major/minor is .4.70204, not empty). This exact 4-component shape is not hypothetical: this file's own long-standing doc comment on VersionOrderKey/parse_version_for_ordering (:6805-6876) states these strings "come from a Python simple index" and calls 7.2.4.70204 "the shape ROCm's own packages are named with." Every caller that reaches a line pin through matches_version (select_rocm_version, select_matching_pip_package_versions, select_tarball_candidate, highest_candidate_needs_next_python) inherits this gap: a release published in that shape would be silently invisible to a major.minor pin aimed at its line.
    Confidence 75 · logic · Fix: have published_in_stable_line compare against the lenient parse_version_for_ordering release vector (release[0]==major && release[1]==minor && stage==Stable) instead of (or in addition to) the strict parse_version, the way compare_version_strings already does for ordering.

  • next_layout_unsatisfied_hint's version_selector = None branch is new behavior introduced by this PR with no test exercising it — apps/rocm/src/therock.rs:7153-7172, the final _ => arm. It's reachable via rocm update --apply: install_sdk_for_update (:1219) always passes version_selector = None and layout = Some(SourceLayout::from_generation(...)) (:1235), and that Some(layout) bypasses select_source_layout entirely in both resolve_pip_runtime_with_timeout and resolve_tarball_artifact_with_timeout. On a runtime previously installed under the Next layout, a later failed update-apply lands on wheel_resolution_failure_context/tarball_resolution_failure_context with layout = Next, version_selector = None, hitting this branch. The only unit test of this function (unsatisfied_rocm_10_line_pin_does_not_suggest_nightly) covers Some(line pin), Some(exact pin), and Canonical + None, but never Next + None; the two new BDD scenarios only drive the Some(Version) fresh-install-preview path. This is new, untested, user-observable CLI output on a real failure path.
    Confidence 75 · logic · Fix: add a direct unit test for next_layout_unsatisfied_hint(None) (and/or the two failure-context functions called with SourceLayout::Next and None), asserting the wording and that it still avoids suggesting nightly.

Non-blocking

None

Decisions for the author

  • The "exact pin failed on Next layout" hint always appends "...and an exact GPU arch, for example --family gfx1200" — tradeoff/wording. By construction, select_source_layout only ever returns Ok(Next) after already confirming family_resolution.raw_arch is exact, so by the time this specific hint can fire for an exact version pin, the arch was already exact — the actual cause of failure is that the exact version itself isn't published. The advice isn't wrong, just not pointing at the real cause in that case. Confirm this one-size-fits-all phrasing (shared with the line-pin case, where it legitimately IS the needed fix) is the intended tradeoff rather than an oversight.
  • The two-component line-pin fix applies uniformly to every ROCm major, not just ROCm 10+ — tradeoff, confirmed deliberate and tested (the older_line/"7.14" case in two_component_version_pin_routes_and_matches_its_release_line). The PR's title/description frame the motivating bug around ROCm 10, but the fix also changes behavior for pre-10 two-component pins (e.g. --version 7.14), which previously never matched anything and now resolves. This is a desirable broadening, not a problem — flagging only so the author can confirm it's the intended scope, since the description doesn't call it out explicitly.

Positive signals

  • The select_tarball_candidate sort-order fix (by_version.then(by_mtime) vs by_mtime.then(by_version)) is genuinely exercised by release_line_pin_prefers_the_newest_stable_tarball_in_the_line: the fixture's 10.1.2 carries an older mtime than 10.1.0, so the test actually discriminates version-first from mtime-first ordering rather than passing either way.
  • The whole feature funnels through the single matches_version/version_request_matches choke point — every caller (select_rocm_version, select_matching_pip_package_versions, highest_candidate_needs_next_python, select_tarball_candidate, requested_source_layout) was checked and picks up the new line-pin semantics consistently, with no caller left on a stale exact-match assumption.
  • The two new BDD scenarios assert negative conditions as well as positive ones (e.g. scenario 16 explicitly asserts --channel nightly is not suggested and no install plan leaks out on failure), guarding against a vacuous pass.
  • README.md and the main.rs clap doc-comment were updated consistently in both places that describe this behavior, with no contradiction between them.

Deployment notes

None beyond the CI gate above — no schema/migration change, no new dependency, no breaking change to an existing working flow (two-component pins were previously non-functional, so this only adds capability).

What this covered

Full read of all 5 changed files at a754cac vs merge-base 3dec0eb: apps/rocm/src/therock.rs (13,012 lines) was read in full by one independent worker focused on the core version-matching/layout-selection logic; README.md, apps/rocm/src/main.rs's relevant doc comment, tests/e2e-cucumber/features/therock_next_generation.feature, and tests/e2e-cucumber/tests/e2e/therock_steps.rs were read in full by a second independent worker cross-checking docs/CLI-help/e2e tests against the implementation. I independently traced the blast radius across every caller of matches_version/RuntimeVersionSelector, the rocm update --apply code path, and confirmed the CI check-run failure directly from its job log.

Did not run: an actual cargo test/cargo build myself for the two logic findings above — both are hand-traced against the source, not executed (scored 75, not 100). History pass (git blame on the modified region) found only this PR's own commit touching it, nothing notable. Prior-changes pass: this PR has zero existing reviews, nothing to reconcile. Agent-instruction adherence checked against the root AGENTS.md only (no directory-scoped AGENTS.md/CLAUDE.md applies to the touched paths). Not reconciled against any PR discussion (none exists yet).

@volen-silo

Copy link
Copy Markdown
Collaborator

Reviewed at a754cac. I found no correctness defect in the code; the requests below are process items.

Blocking

  • "Commit signatures + sign-off" fails: the commit is verified:false, reason: unknown_key. The signing key isn't registered on the author's GitHub account (DCO sign-off is present). Register the key or re-sign, then push again.
  • Both Strix Halo Ubuntu E2E lanes (nightly and release) ran about 45 minutes and failed. I couldn't determine the cause from the logs, so I can't say whether they involve the new scenarios therock-next-15 / therock-next-16. Since this PR changes install resolution, please re-run or triage them before merge.

Non-blocking

  • docs/testing.md and docs/manual-testing.md aren't updated. AGENTS.md §5 asks for both when a flag's observable behavior changes. The README and --help are updated. I didn't check the docs against the PR head, so this may not apply if they don't list scenarios or --version semantics.
  • --version 7.14 now means "newest stable 7.14.x" rather than an exact match. This is described in the README and --help, but only a unit test covers the sub-10 case.
  • In published_in_stable_line, the stable_release_line_pin(published) fallback only matches an index entry that is literally "10.1". Harmless, but not covered by the unit test.
  • The PR body's test-plan checkboxes are all unticked, while the text says tests passed.

Checked by reading the diff

  • select_source_layout takes the major from the line parser, so 11.2 isn't misreported as ROCm 10. 10.1 doesn't prefix-match 10.10, and prereleases are excluded from line matching.
  • User-observable behavior is covered by therock-next-15 and therock-next-16. The failure step asserts the message together with the behavior it describes: the next index was searched, no nightly suggestion, and no install plan printed.

I didn't run the tests locally.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one blocking item; the rest is non-blocking.

Blocking

  • The commit's SSH signature shows as unverified on GitHub (reason unknown_key), so the "Commit signatures + sign-off" gate fails. The sign-off trailer is present. Please register the signing key on your account or re-sign the commit.

Non-blocking

  1. The recovery advice in next_layout_unsatisfied_hint can't clear the condition that printed it (inline comments below). The tests pin only the wording.
  2. The tarball line-pin ordering change has only a unit test and no scenario. The PR body doesn't say why.
  3. Two-component pins below ROCm 10 (e.g. --version 7.14) now also resolve to the newest stable release in that line. That's a reasonable behavior, but it deserves a line in the PR body.
  4. Please confirm whether docs/testing.md and docs/manual-testing.md need updates.
  5. The Test plan checkboxes in the PR body are all unchecked.

Two Strix Halo Ubuntu E2E lanes hit the 45m job timeout; that looks runner-related, so a rerun is worth trying.

Comment thread apps/rocm/src/therock.rs Outdated
Some(RuntimeVersionSelector::Version(version)) => {
if let Some((major, minor)) = stable_release_line_pin(version) {
format!(
"Re-run with a full version in that line, such as `--version {major}.{minor}.0`, and an exact GPU arch, for example `--family gfx1200`."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hint tells the user to re-run with --version {major}.{minor}.0, but when the line pin failed because the index publishes no stable 10.1.x, 10.1.0 fails the same way. Suggest naming the versions the next index does publish (or saying the line has no stable release) rather than a fixed .0.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d97ff58. The hint no longer suggests a fixed .0: it lists the stable ROCm 10 releases the next layout publishes, newest first, and leaves out the one the failed request resolved to (worked out with select_rocm_version, the function that picked it). It offers only versions next_layout_requested routes back to the next layout, and suggests the newest with the arch the request used. If nothing else is published it says so and names no command. therock-next-16 now re-runs the printed --version … --family … and asserts that it resolves.

Comment thread apps/rocm/src/therock.rs Outdated
)
} else {
format!(
"Re-run with that full version and an exact GPU arch, for example `--version {version} --family gfx1200`."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an exact pin that just failed, "Re-run with that full version" repeats the failed request, so it cannot clear the condition. AGENTS.md section 3 requires remediation commands to actually fix what printed them; scenario 16 and the unit test only pin the wording.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d97ff58. The release that just failed is never suggested again; an exact pin that is the only published release gets "publishes no other stable release" and no command. next_layout_unsatisfied_hint_only_suggests_releases_that_were_not_tried covers each state: a line pin whose newest release failed, an exact pin, the only release, prereleases only, no selector (the rocm update --apply path), no exact arch, an unreadable listing, and non-routable publishes.

@rominf
rominf force-pushed the fix/sdk-two-component-version-pin branch from a754cac to d97ff58 Compare November 3, 2026 10:42

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · d97ff58

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (README.md, apps/rocm/src/main.rs, apps/rocm/src/therock.rs, docs/manual-testing.md, docs/testing.md, tests/e2e-cucumber/features/therock_next_generation.feature, tests/e2e-cucumber/tests/e2e/therock_steps.rs), base 3dec0eb → head d97ff58. The first commit (c365dea) is tree-identical to the previously-reviewed a754cac; the second commit (d97ff58) substantially reworks next_layout_unsatisfied_hint and extends line-pin matching to four-component publishes, superseding the prior round's two logic findings — both were re-verified fresh here, not assumed fixed.
No ticket named. The PR description accurately describes the change's scope, including its own test-plan gaps (no full cargo build/cargo test/clippy locally) — no mismatch found.

Blocking

  • This PR head has unresolved merge conflicts against current origin/main — apps/rocm/src/therock.rs. GitHub reports mergeable_state: "dirty"; confirmed directly with git merge-tree $(git merge-base origin/main HEAD) origin/main HEAD, which produces real <<<<<<< .our / >>>>>>> .their conflict markers (4 separate hunks) inside therock.rs. Root cause: two already-merged main-branch PRs landed after this PR's merge-base — #824 ("split therock.rs into child modules, part 1 of 3: registry, version, io, process, fetch") and #828 ("...part 2 of 3: source, family, python, probe, tarball") — which relocated the exact functions this PR edits (RuntimeVersionSelector, SourceLayout, select_source_layout, the resolve_pip/tarball functions, etc.) out of the monolithic file and into child modules. This PR still adds ~805 lines to the old monolithic therock.rs. README.md and main.rs also show overlapping hunks but merge cleanly; only therock.rs has real conflicts.
    Confidence 100 · architectural · Fix: rebase onto current main and re-target the new/changed functions and tests into whichever child module(s) now hold this logic.

  • No CI has run at all on the current head commit (d97ff58645744bd5e89dbf71af05a21a48541105). A live check shows exactly one status (readthedocs, success) and zero check-runs for this sha; the Actions run list for this PR (CI/E2E-self-hosted/CodeQL) contains only runs against the previous head a754cac3609ccc34b8b540567e3a73673450cb07 (from 2026-10-30), where the CI job's own "Commit signatures + sign-off" gate had failed. The second commit was pushed 2026-11-03T09:10:07Z — over 5 hours before this review — yet triggered no workflow run, even though CI is actively running for unrelated PRs in this repo in the same time window, so this isn't a repo-wide outage. Per this repo's own AGENTS.md §12 ("A red check means 'not ready' until resolved"), an unrun check is not a passing one, and the signature gate that previously failed cannot be treated as resolved absent a fresh run.
    Confidence 100 · mechanical · Fix: re-trigger CI (push, or close/reopen, or a maintainer re-run) and confirm the commit-signing fix from the previous round actually registers as GitHub-"Verified" this time.

  • next_layout_unsatisfied_hint's two wording branches are not distinguished by any test — apps/rocm/src/therock.rs:7289-7295. The match version_selector { Some(_) => "Re-run with one of them...", None => "Install one of them... rocm install sdk --channel release --version ..." } arms share the same backtick-quoted --version X --family Y tail, and every existing test (unsatisfied_rocm_10_line_pin_does_not_suggest_nightly, next_layout_unsatisfied_hint_only_suggests_releases_that_were_not_tried) only asserts that common substring. Collapsing the Some(_) arm into the None arm's longer phrasing would leave every current assertion green.
    Confidence 70 · logic · Fix: add an assertion that the Some(version_selector) case specifically produces the shorter "Re-run with one of them" phrasing (and/or that it does not produce the longer rocm install sdk --channel release ... form).

  • The changed --version help text has no regression test pinning its content — apps/rocm/src/main.rs:787-792. This is new user-observable CLI output (the major.minor vs major.minor.patch help wording) and nothing asserts it, breaking from this same InstallTarget::Sdk struct's own established convention: sibling tests (install_sdk_help_describes_the_gate_as_replacing_the_active_default, install_sdk_help_separates_the_two_consents_yes_carries, main.rs:5621+) exist specifically because, per their own inline comment, "rocm install sdk --help is the most-read description... and it is the one surface a 'reword every site' pass can miss — this branch's own history has it being missed once and corrected in a follow-up." Only a generic ASCII-only sweep touches this text; nothing pins its content.
    Confidence 70 · mechanical · Fix: add an assertion (in the same Cli::command()...find_subcommand("sdk") style) that the rendered --version help text contains the new line-pin phrasing.

Non-blocking

  • apps/rocm/src/chat.rs's assistant system prompt still describes --version as taking "a requested exact version" — pre-existing text, not touched by this diff, but now mildly stale given --version also accepts a major.minor line pin. Connected by pattern relevance to this PR's own doc-sync work elsewhere (README/main.rs/docs were updated, this one wasn't touched). Worth a follow-up, not blocking.

Decisions for the author

None beyond what the author already resolved last round (the sub-ROCm-10 broadening and the Next-layout exact-pin hint wording were confirmed deliberate in the prior review round and are unchanged in this one).

Positive signals

  • therock-next-16's step rerun_with_suggested_version parses the actual suggested --version/--family out of the printed failure text and re-executes the CLI with it, asserting rc == 0 — a genuine "remediation actually clears the condition" test, not just a wording pin, directly satisfying this repo's AGENTS.md §3 rule that is the easiest one to fake.
  • release_line_pin_prefers_the_newest_stable_tarball_in_the_line isolates the specific prefer_newest_line_release sort branch: its fixture's 10.1.2 has an older mtime than 10.1.0, so the test would fail under either a whole-commit revert or a revert of just the sort-order change alone — not merely the former.
  • The four-component publish shape (7.2.4.70204) that blocked the previous review round is now handled, and the fix is traced consistently through every caller of matches_version/version_request_matches with no caller left on the old strict-parse assumption.
  • Doc-sync is thorough and consistent: README.md, the main.rs clap doc-comment, docs/testing.md, and docs/manual-testing.md all describe the same two-component-pin behavior with no contradiction, and cite real, existing @id: scenario tags.

Deployment notes

None beyond the blocking merge-conflict and CI-verification items above — no schema/migration change, no new dependency, no breaking change to an existing working flow.

What this covered

Full read of all 7 changed files at d97ff58 vs merge-base 3dec0eb, split across 7 independent fan-out workers (three covering apps/rocm/src/therock.rs by line range — 1-5000, 5000-9000, 9000-end — one each for apps/rocm/src/main.rs, the three doc files, the BDD feature+step-definition pair, and a dedicated AGENTS.md-adherence pass over the full diff), plus my own direct verification of the merge-conflict state (git merge-tree) and live CI/Actions state (GitHub API). Every file was confirmed read in full by its worker.

Did not run: cargo build/cargo test/clippy — no Rust toolchain available in this environment; all logic findings are hand-traced against the source (scored 70, not 100) except the two findings above confirmed by direct tool/API execution (git merge-tree; GitHub Actions run list), which score 100. History pass (git log/git blame) found the relevant region's only prior touches are unrelated refactor PRs #824/#828, which is itself the source of blocking finding #1. Prior-changes pass: the earlier review round's two logic findings (four-component matching; untested None branch) were re-verified fresh against the current code rather than assumed fixed, and both are now resolved; its CI finding is carried forward because it has never actually been re-checked. Agent-instruction adherence checked against the root AGENTS.md only. Not reconciled against the existing PR discussion (the juhovainio/rominf review threads) beyond what was independently re-derived from the code and live CI/merge state.


🤖 Generated with Claude Code

@rominf

rominf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Reply to the automated review on d97ff58:

  • Merge conflicts. Rebased onto main. The therock.rs split moved the code this PR edits, so the changes now live in these modules:

    • therock/version.rs: version_request_matches, stable_release_line_pin, published_in_stable_line
    • therock/source.rs: routing, selector_is_release_line
    • therock/resolve/wheel.rs: wheel_resolution_failure_context, next_layout_unsatisfied_hint
    • therock/resolve/tarball.rs: tarball_family_candidates, the line-pin ordering, tarball_resolution_failure_context
    • cli.rs: the --version help

    The scenarios are renumbered to therock-next-16, -17 and -18, because main now has a therock-next-15. The two commits are now ece751f and 61ea732. 61ea732 is the rebased d97ff58 that my earlier replies to the inline threads cite.

  • No CI on the head. 987d687 is pushed, and I'm watching CI on it.

  • The two re-run wordings of next_layout_unsatisfied_hint. d21ce90 asserts each arm's own phrasing, "Re-run with one of them" when the request is pinned and "Install one of them … rocm install sdk" when it is not, and asserts that the other arm's phrasing is absent. One correction to the finding: the existing assertion on "`--version 10.1.0 --family gfx1200`" already failed when I gave the pinned arm the unpinned wording. In that wording the backtick opens before rocm install sdk, not before --version. The new assertions make the difference explicit, so a reader doesn't depend on where a backtick falls.

  • The --version help text had no test. d21ce90 adds install_sdk_version_help_describes_line_and_exact_pins in cli.rs, where main now keeps the command tree. It reads the version argument's own help, so the --build-date doc can't satisfy it, and it fails with the old "Exact TheRock package version" text restored. It is a wording pin. Its comment names the tests that prove the behavior.

  • Non-blocking: the chat prompt's "requested exact version". No change. That sentence tells the assistant how to pass a version the user asked for exactly, which is still correct. A line pin goes through the same --version argv.

I also ran my own review pass, which led to two more commits:

  • d1a6e30 pins three branches that stayed green when broken alone: the major an 11.2 refusal prints, the u32 overflow guard on a pin component, and the unpinned tarball pick.
  • 987d687 corrects a docs/testing.md sentence that said a four-component release is never picked on the release channel. A wheel --version 7.2 does pick one.

Locally on 987d687:

  • fmt and workspace clippy are clean, and so is clippy on the e2e target.
  • The workspace tests pass except four that fail only on this host. One of them, in rocmd, is load-sensitive and passes on rerun.
  • cargo xtask e2e -- -n "therock-next-1[5-8]" passes: 4 scenarios.

@rominf
rominf requested a review from juhovainio November 9, 2026 14:14

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found no problems in the main change: line-pin routing, the matches_version call sites, tarball ordering and the docs all look right. Two issues in the new "re-run with a published release" hint are worth fixing before merge; details are inline.

  • For an unpinned tarball failure, the hint works out the attempted release by version order while the tarball pick is by upload time, so it can suggest the release that just failed.
  • The suggested command omits --format, so a failed tarball update is told to install a wheel.

Smaller notes: 10.01 parses as the 10.1 line, the tarball bail wording doesn't mention line pins, and post-release stage handling in the line matcher has no test.

The PR currently conflicts with main and needs a rebase. No code CI has run on this head, so none of this is CI-verified yet.

Comment thread apps/rocm/src/therock/resolve/wheel.rs Outdated
);
};
let scope = family_scope.map_or_else(String::new, |family| format!(" for {family}"));
let attempted = select_rocm_version(TheRockChannel::Release, published, version_selector);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attempted is computed with select_rocm_version, which picks the newest by version. An unpinned tarball pick (rocm update --apply) is newest by upload time, so for that caller the hint can exclude the wrong release and suggest the one that just failed. Could the caller pass in the version it actually attempted?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The hint no longer works out the attempted release itself. A failure another release can clear is raised as a VersionUnsatisfied that carries the release the resolver actually picked, and the hint leaves out exactly that one. A tarball failure means no archive was picked, so nothing is excluded. Now in therock/resolve/next_hint.rs, merged with current main in f65bb1e.

Comment thread apps/rocm/src/therock/resolve/wheel.rs Outdated
"Re-run with one of them, for example `--version {newest} --family {exact_arch}`."
),
None => format!(
"Install one of them, for example `rocm install sdk --channel release --version {newest} --family {exact_arch}`."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This suggested command has no --format, but next_layout_unsatisfied_hint is also reached from the tarball failure path. After an unpinned tarball update fails, the suggestion installs a wheel, not the tarball. Please add --format or make the hint aware of the caller's format.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The hint takes the caller's format: a failed tarball resolution suggests --format tarball, a wheel failure --format wheel, with --devel when the runtime had the toolchain. Unit tests cover both formats.

// "newest stable release in that line", even when an older patch was
// uploaded later. Exact pins match one archive, so the order does not
// change which file they select.
let prefer_newest_line_release = version_selector.is_some_and(selector_is_release_line);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Version-first ordering applies only to line pins here, while the failure hint's attempted (wheel.rs) uses version order for every selector. The two agree only in the pinned case, which is the root of the wheel.rs comments.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Nothing now depends on the two orderings agreeing: the hint doesn't recompute the attempted release, the resolver that made the pick records it. The tarball ordering is unchanged and pinned by release_line_pin_prefers_the_newest_stable_tarball_in_the_line (line pin: version first; unpinned: newest upload).

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 987d68772c4

Outcome: needs work. Full review of the whole change (10 files, base a3b4014→head 987d6877) via 5 independent fan-out passes plus direct verification of live PR/CI/merge state. No ticket named; the PR description accurately describes scope and honestly flags its own test-plan gaps.

Blocking

  • Real merge conflicts with current main, in apps/rocm/src/therock/resolve/wheel.rs. Confirmed directly (git merge-tree, real conflict markers), and GitHub reports mergeable_state: dirty. Cause: main's #954 landed after this PR's merge-base and touches the same region. Fix: rebase onto current main.
  • No CI has ever run on the current head (pushed ~hours before this review). The only CI runs ever triggered on this branch were on the very first commit (Oct 30), and that CI run's conclusion was failure — never re-verified since. Confirmed this isn't a repo-wide outage (other PRs' CI ran fine in the same window). Fix: push/re-trigger CI and confirm green on this head.
  • next_layout_unsatisfied_hint's unpinned-selector suggestion omits --format, so a failed rocm update --apply on a tarball runtime stuck on the Next layout (resolve_tarball_artifact_with_timeout is called with version_selector: None) gets told to run rocm install sdk --channel release --version X --family Y — which defaults to --format wheel, i.e. it suggests installing a wheel runtime to fix a broken tarball one. The only test for this path's None branch only covers SourceLayout::Canonical, not Next. Fix: thread a format/InstallFormat parameter through next_layout_unsatisfied_hint (the sibling family_resolution_hint already does this correctly) and include --format explicitly when it isn't the default.

Non-blocking

  • apps/rocm/src/chat/turn.rs's assistant system prompt still describes --version as "a requested exact version" — stale now that it also accepts a line pin; same doc-sync class this PR fixed in README/help/docs but missed here (pre-existing, not touched by this diff).
  • resolve_pip_runtime_with_timeout's index-validation failure message (wheel.rs:649) unconditionally says "canonical source" even when layout is Next — pre-existing from the recent module split, but this PR makes the Next layout reachable via more pins, so the mislabeling fires more often.

Decision for the author: the two-component line-pin behavior is broadened to every ROCm major, not just ROCm 10+ (e.g. --version 7.14 now resolves) — this looks deliberate and is tested, just flagging so it's confirmed as intended scope.

Positive signals: the tarball-ordering fix is exercised by a fixture that deliberately inverts version vs. upload-time order; the therock-next-17 e2e scenario re-runs the actual suggested recovery command rather than just pinning its wording; the four-component-publish matching gap from an earlier round is now handled consistently across every caller.

What this covered: all 10 changed files read in full across 5 independent workers (grouped by coupling) plus my own direct checks of live mergeability, CI/Actions history, and AGENTS.md adherence. Did not run cargo build/test/clippy (no Rust toolchain available here); logic findings are hand-traced against source. Not reconciled against existing PR review threads.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

rominf and others added 10 commits November 10, 2026 13:39
`--version 10.1` stayed on the canonical 7.x index because it did not
parse as major.minor.patch, then suggested `--channel nightly`, which
refuses ROCm 10. A numeric major.minor pin now selects the newest
stable publish in that line, and a failed ROCm 10 pin no longer points
at nightly.

Signed-off-by: Roman Inflianskas <roman.inflianskas@amd.com>
The next-layout failure hint told a failed `--version 10.1` to re-run
with `--version 10.1.0` and a failed exact pin to re-run with itself,
always with `--family gfx1200`. Neither could clear the failure.

The hint now lists the stable ROCm 10 releases the next layout
publishes, leaves out the release the failed request resolved to, keeps
only versions a re-run would route back to the next layout, and
suggests the newest with the arch the request used. With nothing else
published, or no exact arch, it names no version. A line pin also
matches four-component publishes such as 7.2.4.70204.

therock-next-17 re-runs the suggested command; therock-next-18 covers
the tarball line pin taking the newest release over the newest upload.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The `--version` help is the most-read description of what a
two-component pin does, and nothing asserted its wording. Read the
argument's own help, so the sibling `--build-date` doc cannot satisfy
it.

The re-run hint words a pinned request ("Re-run with one of them") and
the unpinned `rocm update --apply` path (a full `rocm install sdk`
command) differently. Assert each arm's own phrasing and the absence of
the other's, so a swap is named by the failure rather than caught by a
backtick position.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Three branches of the line-pin change stayed green when broken alone:
the major an `11.2` refusal prints (it fell back to the ROCm 10 floor),
the u32 overflow guard on a pin component, and the unpinned tarball
pick, which must keep "newest upload wins" while only a line pin orders
by version first. Each assertion goes red when its branch is reverted.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The comparison section said a four-component release such as
7.2.4.70204 is never picked on the release channel. That holds for an
unpinned pick and for the tarball catalog, but a wheel `--version 7.2`
selects it. Also move the `--version` paragraph below the dry-run
verification list it had split from its introduction.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A failed ROCm 10 resolution used to append the list of published
releases and a --version/--family re-run to every error, including
failures no other release can clear (an unknown device target, a uv
that could not be fetched, an unreachable index), and to rocm update,
which takes neither flag.

Failures a different --version can clear are now raised as a typed
VersionUnsatisfied: an unmatched pin, a release with no torchaudio for
this Python and platform, a uv pip compile with no solution, and a
tarball catalog with no archive for the request. Only those get the
release list, which leaves out the release just attempted. A pinned
install is told which --version and --family to re-run with. A failed
rocm update is given a whole rocm install sdk command that keeps the
runtime's format, exact arch and --devel toolchain, and a runtime that
recorded no exact arch gets no version advice at all.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The ROCm 10 recovery hint pushed resolve/wheel.rs past the 1,500-line
module-size limit. The torchaudio pin (the verified-release preference,
its newest-build fallback and the TorchaudioPin wording) is a
self-contained decision, so it moves to resolve/torchaudio.rs with its
tests. wheel.rs keeps the PEP 440 helpers and exposes them to the new
module as pub(super). No behavior changes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
New scenarios drive each outcome through the CLI against loopback
fixtures: an unsatisfied tarball pin lists the published archives and
its suggested re-run succeeds (therock-next-19), an update whose runtime
records no exact arch gets no version or --family advice (20), a failed
update suggests an install command that keeps the runtime's format and
toolchain and that command previews successfully (21), an exact pin
ignores later patches (22), and a canonical 7.14 line pin takes the
line's newest stable build (23). therock-next-16 now publishes several
10.1 builds, a prerelease and a 10.10 build, so it shows the line pin
takes the newest stable 10.1.x rather than any of those.

README, testing and manual-testing docs describe which failures get a
version suggestion and what rocm update suggests instead.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…-version-pin

Main made ROCm 10 the default and routes a version selector through
`layout_serving`. The two-component line pin now routes through it: a
`major.minor` pin is a stable release pin for that line, and the next-layout
hints list only releases a release-channel pin would route back to the next
layout.

An unpinned install takes the newest build with a full stack, so a torchaudio
failure there means no build has one; it no longer carries the marker that
triggers a version suggestion, since another version would fail the same way.
Scenario therock-next-29 (an update whose newest build lacks torchaudio) is
gone because that update now installs the older complete build; therock-next-31
covers the unpinned no-torchaudio failure. The line-pin scenarios are renumbered
24-30 after main's 16-23, and the wheel failure wording moves into next_hint.rs
to stay under the module-size limit.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…-version-pin

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/sdk-two-component-version-pin branch from 987d687 to f3a1d7a Compare November 13, 2026 11:48
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head f3a1d7ab54c7

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

DEGRADED REVIEW (see What this covered — the usual multi-agent fan-out could not run this round; this is a single-context pass).

Full review of the whole change. Implements the fix named in the title; no ticket identifier is named in the title or description, and none could be retrieved.

Blocking

  • The PR description's "Coverage" section names the wrong scenario IDs and omits most of what the change actually covers. It lists @id:therock-next-16-two-component-line-pin, -17-unsatisfied-line-pin-skips-nightly, and -18-tarball-line-pin-takes-newest-release as the three new scenarios. Those three IDs exist in tests/e2e-cucumber/features/therock_next_generation.feature but are pre-existing, unrelated scenarios (therock-next-16-earlier-release-pin-reads-the-canonical-stream, -17-unpinned-group-family-refused-and-the-named-pin-works, -18-unpinned-tarball-reads-the-newest-catalog). This change's actual new scenarios are therock-next-24 through -31 (eight scenarios, not three), correctly listed in this same diff's docs/testing.md update. The description's own text even says "the scenarios are renumbered after main's therock-next-15" but then goes on to cite the pre-renumbering IDs anyway. Four of the real scenarios aren't mentioned in the description at all: -28 (an update whose runtime records no exact arch gets no version/--family advice), -29 (an exact pin ignores a later patch in its line), -30 (a canonical 7.14 line pin), and -31 (an unpinned install with no installable torchaudio suggests no version). The unit-test names and docs/testing.md/docs/manual-testing.md entries in the Coverage section are accurate; only the @id: list is stale.
    Confidence 100 · mechanical · Fix: update the description's Coverage section to cite therock-next-24 through -31 with their real one-line summaries (the docs/testing.md entry added in this same diff already has correct wording to copy from).

Non-blocking

  • apps/rocm/src/chat/turn.rs's assistant system prompt still describes --version as being for "a requested exact version" (ROCM_CHAT_TOOL_SYSTEM_PROMPT, line 629) — stale now that --version also accepts a major.minor line pin. Pre-existing text, not touched by this diff, but connected by pattern relevance to this PR's own doc-sync work (README.md, --help, docs/testing.md, docs/manual-testing.md were all updated to describe the line-pin behavior; this prompt string was missed).

Decisions for the author

  • The two-component line-pin behavior is broadened to every ROCm major, not just ROCm 10+ — tradeoff. The title and most of the description frame the fix around ROCm 10 (rocm install sdk --version 10.1), but the same fix also makes a sub-10 pin like --version 7.14 resolve to the newest stable 7.14.x on the canonical index, which previously matched nothing. This is directly tested (older_line case in two_component_version_pin_routes_and_matches_its_release_line in therock/source.rs, plus the dedicated therock-next-30 scenario and its canonical-index fixture) and looks deliberate and well-covered, not an oversight. Flagging only so the author can confirm it's the intended scope, since the description frames the change narrowly around ROCm 10.

Positive signals

  • next_hint.rs's NextLayoutRecovery/RecoveryFormat correctly threads --format wheel/--format tarball through the suggested recovery command for both the pinned-install and the rocm update (no-pin) cases, including the tarball path's recovery.exact_arch: None case, which prints a <gfx> placeholder with an explanation instead of silently guessing or defaulting to wheel.
  • tarball.rs's ordering fix (by_version.then(by_mtime) for a line pin vs. by_mtime.then(by_version) otherwise) is exercised by a fixture that deliberately uploads the older 10.1.0 archive after the newer 10.1.2, so release_line_pin_prefers_the_newest_stable_tarball_in_the_line actually discriminates version-first from mtime-first ordering rather than passing under either rule.
  • The e2e rerun_with_suggested_version/rerun_tarball_with_suggested_version steps parse the actual suggested --version/--family out of the printed failure text and re-execute the CLI with it, asserting success — a genuine "the remediation clears the condition" test, not just a wording pin, and scenarios therock-next-25/-27 use it.
  • version.rs/source.rs carry unusually thorough doc comments explaining why each parsing/ordering choice is made (e.g. why a combined PEP 440 stage suffix is rejected rather than approximated, why local version metadata is dropped, why a major.minor pin reads published with the lenient ordering parser rather than the strict one) — the four-component-publish matching (7.2.4.70204) and the u32-overflow guard are both directly tested with comments explaining the specific risk each guards against.
  • The new next_layout_unsatisfied_hint deliberately strips the VersionUnsatisfied marker from a no-pin, empty-full-stack torchaudio failure (wheel.rs, the with_full_stack.is_empty() branch) specifically because "another --version fails the same way," preventing the hint from suggesting a retry that cannot possibly help.

Deployment notes

None. No schema/migration change, no new dependency. The behavior change (two-component pins now resolve instead of failing) is additive for users who previously got a failure.

What this covered

Degraded, single-context review: the usual independent multi-agent fan-out this method relies on could not be used this round — every attempt to spawn parallel review workers failed with a concurrent-subagent-limit error from shared infrastructure, and the limit did not clear after repeated checks, so per this method's own fallback rule I ran a single-context pass instead and am flagging it as degraded. This covers less of the surface than a fanned-out review would (measured elsewhere at roughly a third the findings), and re-running with fan-out available is recommended.

Within that constraint, I read in full: apps/rocm/src/therock/version.rs (567 lines), apps/rocm/src/therock/source.rs (1010 lines, including its large in-file test module), apps/rocm/src/therock/resolve/next_hint.rs (572 lines, a new file), and the complete diffs (with surrounding function bodies) of apps/rocm/src/therock/resolve/wheel.rs, apps/rocm/src/therock/resolve/tarball.rs, apps/rocm/src/therock/resolve/torchaudio.rs, apps/rocm/src/therock/resolve.rs, apps/rocm/src/therock.rs, apps/rocm/src/cli.rs, README.md, docs/architecture.md, docs/testing.md, docs/manual-testing.md, tests/e2e-cucumber/features/therock_next_generation.feature, and tests/e2e-cucumber/tests/e2e/therock_steps.rs, against merge-base e2e5c5bb3d34605888fcab61f4511ab24f215c3c → head f3a1d7ab54c7db28b7f55ff6e3750d2f228657ea. For the three largest files (wheel.rs at 3083 lines, tarball.rs at 683, therock_steps.rs at 3247, docs/testing.md at 5048) I read the full diff plus the functions it touches in context, but not every unchanged line of the file byte-for-byte.

I independently verified, directly against live state rather than assuming either was resolved: mergeability — git merge-tree of the PR head against current origin/main produces a clean merge with no conflict markers (only two unrelated commits landed on main since the merge-base, touching CI/doc files, not apps/rocm/src); GitHub's mergeable_state: "blocked" reflects required-review/status-check gating, not a conflict. CI — all 46 check-runs on the current head commit are success or skipped except one self-hosted E2E tests (MI300X, current) job still queued; no failures. Both of these were explicitly re-checked this round rather than carried over from an earlier one.

Did not run cargo build/cargo test/clippy myself (no Rust toolchain in this environment); the version-matching and hint-construction logic findings are hand-traced against the source and existing tests, not executed. History pass: not performed as a separate pass this round (folded into the direct reads above; nothing beyond this PR's own commits touches the reviewed regions). Prior-changes pass (review comments on earlier, unrelated changes to these files): not performed — out of scope for this round. Agent-instruction adherence was checked against the repository's root AGENTS.md (no directory-scoped AGENTS.md/CLAUDE.md applies to the touched paths), specifically its module-size limit (production code in both wheel.rs and cli.rs is confirmed under 1,500 lines), its docs/architecture.md-entry requirement (the new next_hint.rs has an accurate entry added in this diff), and its message/state co-assertion and markdown-link rules. This review did not read any prior review round's findings and was reached blind from the code; it is not reconciled against the existing PR review threads from juhovainio or earlier nowycondro rounds.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

…-version-pin

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Changes since the last review (head 987d6877). The branch now merges current main instead of rebasing onto it: main's #1033 rewrote the layout routing this PR edits, so the fix was ported onto layout_serving rather than replayed.

  • Routing. A major.minor pin is now one more stable pin that layout_serving places, so --version 10.1 reads the next layout and --version 7.14 the canonical index. The nightly refusal names the pin's own major.
  • The attempted release (your select_rocm_version vs upload-time point). The hint no longer recomputes it. A failure another release can clear is raised as a VersionUnsatisfied that carries the release the resolver itself picked, and the hint leaves out exactly that one.
  • --format. A failed rocm update is given a whole rocm install sdk --channel release --format <wheel|tarball> --version X --family <arch> command, with --devel when the runtime had the toolchain. A pinned install still gets the --version/--family re-run.
  • Hint scoping. The release list appears only for failures a different version can clear. New since main's full-stack filter: an unpinned install where no build has an installable torchaudio gets no suggestion, because every other release would fail the same way (therock-next-31).
  • Scenarios. Renumbered to therock-next-24 to -31 after main's 16 to 23. The old update-suggests-an-install scenario is gone: after feat(install): read the current stable and nightly TheRock hosts by default, keep earlier pins on the old ones #1033 that update takes the older complete build and succeeds, so there is no failure left to drive it. Its wording stays covered by unit tests; the PR description says where.

The PR description is rewritten from the final diff and lists the local gate results and the host-only gaps.

@rominf
rominf requested a review from juhovainio November 16, 2026 12:50

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on the description and scope; the logic findings are smaller and mostly about hint accuracy.

  1. The PR body says the torchaudio pin "moves from wheel.rs into torchaudio.rs ... the moves change no behavior". The diff doesn't do that: torchaudio.rs already exists on main, wheel.rs only gains imports, and torchaudio.rs changes its error to the new VersionUnsatisfied marker. That is a behavior-affecting change described as a no-op move, so please correct the body from the final diff.

  2. On a failed nightly resolution against the next layout, the stable-release hint still applies. published there is the nightly listing, so every prerelease name is filtered out and the user is told no stable release exists, or is handed a --channel release command for what was a nightly update. The channel only influences the canonical branch. Please gate this on the release channel (or give a nightly-appropriate message) and add a Nightly+Next test.

  3. Scope: this is two changes: the line pin that #808 asks for, and a recovery-hint rework (the new error type, the ~570-line next_hint.rs, the rocm update install-command suggestion). The title names only the pin. Splitting them would make each easier to review, or at least name the hint rework in the title.

  4. The uv failure is classified by the substring "No solution found" in stderr. A wording change in uv silently drops the version advice, and no scenario drives a real uv pip compile failure.

  5. The rocm update suggestion is only asserted as a string. Scenarios 25/27 run the pinned-install suggestion, but nothing shows that following the update command installs the same kind of runtime, and the hint can't know the suggested release solves or has a torchaudio.

  6. Minor: the doc links in torchaudio.rs don't resolve (see inline).

I haven't built or run anything locally; this is a read of the diff plus the CI state (green apart from the pending MI300X lane).

recovery: NextLayoutRecovery<'_>,
) -> String {
let (source_label, hint) = match layout {
SourceLayout::Next => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Next layout this applies the stable-release hint for the nightly channel too. published is the nightly listing here, so every nightly name is filtered out and the user is told no stable release exists, or is handed a --channel release command after a failed nightly update. Please gate on the release channel; no test covers Nightly+Next.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a020c6c. NextLayoutRecovery now carries the request's channel, and only a release-channel failure gets the release list. A nightly failure on the next layout gets the source line alone (wheel) or "No nightly archive for matched this request" (tarball), with no --version and no --channel release command. a_nightly_next_layout_failure_suggests_no_release and the nightly case added to tarball_failure_on_next_layout_suggests_a_published_release cover it, and both go red with the gate removed.

fn uv_compile_failure(rocm_version: &str, stderr: &str) -> anyhow::Error {
let detail = stderr.trim();
let message = format!("published ROCm X package metadata is not jointly installable: {detail}");
if detail.contains("No solution found") {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Classifying by the substring "No solution found" in uv's stderr is brittle: if uv rewords it, the failure silently loses its version advice. Nothing exercises a real uv pip compile failure.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept the substring, and now pinned it against real uv output: therock-next-32 (afcf0d4) builds an index whose newest release lists every package but whose torch requires an unpublished rocm, so a real uv pip compile fails with "No solution found". The scenario asserts the phrase and the release list, then runs the suggested command. With the match changed to a string uv does not print, it goes red. uv gives no machine-readable reason or distinct exit code for an unsatisfiable set, so a phrase match plus a scenario that drives real uv is the strongest check available; if uv rewords it, CI fails rather than the advice silently disappearing.

/// pinned build was also the newest one on the index.
///
/// Not `select_latest_stack_package` (in `wheel.rs`): `parse_version` reads only
/// Not [`select_latest_stack_package`]: [`parse_version`] reads only

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

select_latest_stack_package and parse_version aren't imported in this module, so these intra-doc links won't resolve; use plain backticks or a full path.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9750ee0: plain backticks, and the comment says select_latest_stack_package is in wheel.rs.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (apps/rocm/src/{cli.rs, therock.rs, therock/version.rs, therock/source.rs, therock/resolve.rs, therock/resolve/{wheel,tarball,torchaudio,next_hint}.rs}, README.md, docs/{architecture,manual-testing,testing}.md, tests/e2e-cucumber/features/therock_next_generation.feature, tests/e2e-cucumber/tests/e2e/therock_steps.rs), base 3e00084c41c982c5d2f395bcda4c6c8e371c1588 → head f65bb1e9e470972df78e1b67c6080f01bbcfe702.

Fixes #808 as claimed — the two-component line-pin routing bug it describes is genuinely fixed and well tested. The change also bundles in a substantial failure-recovery-hint system (a new ~570-line module and a new marked-error type) that #808 didn't ask for; see Decisions for the author.

Blocking

  • The next-layout failure-recovery hint hardcodes the release channel, ignoring the channel of the request that actually failed — apps/rocm/src/therock/resolve/next_hint.rs:121-216 (the alternatives filter at :150-156, the suggested command at :213), consumed from apps/rocm/src/therock/resolve/wheel.rs:806-852 and apps/rocm/src/therock/resolve/tarball.rs:246-275. NextLayoutRecovery/next_layout_unsatisfied_hint carry no channel field or parameter; both the candidate filter (layout_serving(TheRockChannel::Release, ...)) and the suggested command ("rocm install sdk --channel release ...") are hardcoded regardless of the caller's real channel. An unpinned nightly-channel request (or rocm update on a runtime recorded as nightly) routes to the Next layout exactly like a release request, and if it fails, this hint still fires: nightly builds publish as prerelease/build-dated strings, which fail the hardcoded stage == Stable filter, so the alternatives list comes out empty regardless of what the nightly catalog actually publishes — producing "no stable release ... so no --version pin can install from it yet," or, if anything slips through, a command that silently switches the user to --channel release. No test anywhere in this diff passes TheRockChannel::Nightly into either failure-context function. Confidence 82 · architectural · Fix: thread the real channel through NextLayoutRecovery, use it in the filter and the formatted --channel flag, and add a Nightly+Next test.

  • A new e2e step definition ships dead: it is never referenced by any scenario — tests/e2e-cucumber/tests/e2e/therock_steps.rs:544-553, pip_index_fixture_without_newest_torchaudio. A repo-wide search for its exact step text against every .feature file returns zero matches. cargo clippy doesn't catch this because the #[given(...)] macro registers the function in cucumber's step table, so rustc sees it as used. Confidence 95 · mechanical · Fix: delete it, or add the scenario it looks written for (a mirror of the existing ROCm-10.0.0 variant, for the newer build).

  • published_in_stable_line's major-equality check has no test that isolates it — a mutation removing it would pass the whole suite — apps/rocm/src/therock/version.rs:292-298. Every negative assertion this PR adds differs from a matching positive one only in the minor component with the same major; the one cross-major fixture case differs in both major and minor, so it can't isolate the major conjunct either. Deleting the major check leaves every test in this diff green. Confidence 80 · logic/test-coverage · Fix: add an assertion such as !line.matches_version("11.1.0") for a 10.1 line pin.

  • The PR's own description mischaracterizes a real behavior change in torchaudio.rs as a no-op file move — apps/rocm/src/therock/resolve/torchaudio.rs:122-138. torchaudio.rs already exists as its own module at the merge-base — this diff doesn't create or move it (only next_hint.rs is genuinely new). What it actually changes is torchaudio_pin's failure path: a plain bail! becomes a VersionUnsatisfied-marked error, making a torchaudio-shaped failure newly eligible for the "try another release" hint — a real, user-visible behavior change described as "Internal... the moves change no behavior." Confidence 90 · Step-9 scope-misstatement · Fix: correct that paragraph to describe the actual change.

  • The rocm update whole-install-command recovery suggestion is unit-tested only, never driven end-to-end to prove it clears the condition it's printed for — apps/rocm/src/therock/resolve/next_hint.rs:199-216 (unpinned branch), reached from apps/rocm/src/therock.rs:279. This repo's own AGENTS.md (§3) states a remediation command "must exist, accept those flags, and actually clear the condition," and that a unit test of the internal helper does not discharge this. The pinned-install case of this same hint is covered end to end (therock-next-25/-27); the update case is not. The PR description names the gap but only cites unit tests as coverage, which the project's own rule says isn't sufficient, and the existing therock-next-28 fixture pattern shows an e2e scenario here is practical. Confidence 75 · logic/test-coverage · Fix: add a scenario driving an update failure into this branch and re-running the suggested command.

Non-blocking

  • docs/architecture.md's entries for therock/source.rs and therock/version.rs (and version.rs's own module doc) weren't updated for the new line-pin grammar, unlike the sibling tarball.rs/wheel.rs entries that were — docs/architecture.md:152,155; apps/rocm/src/therock/version.rs:5-10. The architecture-doc check only verifies an entry exists, not that its prose is current. Confidence 75.
  • apps/rocm/src/chat/turn.rs's assistant system prompt still describes --version as being for "a requested exact version" — pre-existing, untouched by this diff, but stale now that a line pin is also accepted, and connected by pattern relevance to this PR's own doc-sync work elsewhere (README, --help, both docs files were updated; this string wasn't). Confidence 70.

Decisions for the author

  • The uv pip compile resolver-failure classification is a substring match ("No solution found") on uv's stderr — tradeoff, explained by its own doc comment, but untested against real uv output; a future uv wording change would silently stop marking these failures as recoverable, with no test to catch it. Worth confirming this is an accepted risk.
  • The two-component line-pin behavior is broadened to every ROCm major, not just ROCm 10+ (e.g. --version 7.14 now resolves) — tested and looks deliberate; flagging only to confirm it's the intended scope, since the title/issue frame the fix narrowly around ROCm 10.
  • Scope: this PR is two changes — the line-pin routing fix #808 asks for, and a separate, larger failure-recovery-hint rework (a new error type, a new ~570-line module, the rocm update install-command suggestion). The title names only the first. Consider splitting, or naming the hint rework in the title/description.

Positive signals

  • assert_selected_rocm_version (e2e) doesn't just assert the expected version appears — it asserts every other candidate in the fixture is absent, closing off each specific wrong-branch outcome in one call, reused across three scenarios.
  • The tarball ordering fix is exercised by a fixture that deliberately uploads the older release after the newer one, so the test discriminates version-first from upload-time-first ordering rather than passing under either rule.
  • Two scenarios (therock-next-25/-27) re-run the suggested recovery command and assert it succeeds, rather than only pinning its wording — a direct, well-executed answer to this project's own "assert the message together with the behavior" rule.
  • The --version help text change is pinned by a dedicated test that cross-references the exact unit test and scenario IDs it depends on.
  • The four-component-publish matching gap and the u32-overflow case each have dedicated regression tests for edge cases that are easy to miss.

Deployment notes

None. CI is green on this head (all required checks pass; one neutral CodeQL status and expected skipped advisory/platform-gated jobs). No merge conflicts against current origin/main — confirmed directly via git merge-tree (GitHub's mergeable_state: blocked reflects required-review gating, not a conflict). No schema/migration change, no new dependency; the behavior change is additive (two-component pins previously resolved nothing).

What this covered

Full read of all changed files at f65bb1e9e470972df78e1b67c6080f01bbcfe702 vs merge-base 3e00084c41c982c5d2f395bcda4c6c8e371c1588. Three groups (therock/version.rs+therock/source.rs+therock.rs; therock/resolve/{tarball,torchaudio,next_hint}.rs; the e2e feature file + step definitions) ran as independent fan-out passes, blind to each other, this report, and each other's results. The other three groups (therock/resolve/{wheel,resolve}.rs plus next_hint.rs/torchaudio.rs for context; cli.rs+README.md; the three docs/*.md files) could not be spawned as independent workers after repeated attempts hit a shared concurrency limit that further retries were not permitted to work around, so those three were read directly in this same consolidating context instead — flagged here as not independent passes, per this method's own disclosure rule for a degraded fan-out.

Agent-instruction adherence was checked against the root AGENTS.md, including its module-size limit (all touched files confirmed under the 1,500-production-line cap), its architecture-doc requirement, and its explicit rules on remediation-command testing and scope-accurate descriptions. A history pass (git log/git blame on the touched regions) was folded into the per-group reads rather than run separately. A "prior changes" pass (review comments on earlier, unrelated changes to these files) was not performed. Mergeability and live CI state were checked directly against the current head and current origin/main rather than assumed. Findings were reached blind from the code before any part of this report was reconciled against the PR's existing review discussion; two of the blocking findings above were also independently raised in that discussion, which is noted here only as corroboration, not as their source.


🤖 Automated by an AI agent

.filter(|version| attempted != Some(version.as_str()))
.filter(|version| {
layout_serving(
TheRockChannel::Release,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this hint hardcodes the release channel, ignoring the channel of the request that actually failed.

NextLayoutRecovery/next_layout_unsatisfied_hint carry no channel field or parameter. This filter always asks layout_serving(TheRockChannel::Release, ...), and recovery_command (line 213 below) always prints --channel release, regardless of whether the failing request was on the release or the nightly channel.

An unpinned request on the nightly channel (rocm install sdk --channel nightly, or rocm update on a runtime whose manifest records channel: nightly) routes to the Next layout the same way a release request does (layout_serving's no-selector branch always returns Next), and both wheel_resolution_failure_context and tarball_resolution_failure_context build this hint for that case too. Nightly builds publish as prerelease/build-dated strings, which fail this hardcoded stage == Stable filter, so alternatives comes out empty regardless of what the nightly catalog actually publishes — the user gets "It publishes no stable release ... so no --version pin can install from it yet," or, if anything slips through, a suggestion to run with --channel release, silently steering them off the channel they asked for.

No test in this diff passes TheRockChannel::Nightly into either failure-context function or into this function.

Confidence 82 · architectural · Fix: add a channel: TheRockChannel here (or to NextLayoutRecovery), use it in the filter and in the formatted --channel flag, and add a Nightly+Next test.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a020c6c (the earlier thread on line 267 of this file raised the same point). The filter and the printed --channel now use the request's channel instead of a hardcoded release. NextLayoutRecovery now carries the request's channel, and only a release-channel failure gets the release list. A nightly failure on the next layout gets the source line alone (wheel) or "No nightly archive for matched this request" (tarball), with no --version and no --channel release command. a_nightly_next_layout_failure_suggests_no_release and the nightly case added to tarball_failure_on_next_layout_suggests_a_published_release cover it, and both go red with the gate removed.

/// an installable one. The mirror of the fixture below.
#[given(
"a ROCm 10 pip index fixture whose ROCm 10.1.0 torchaudio is only a wheel this host cannot install"
)]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — dead code: this step definition is never referenced by any scenario.

pip_index_fixture_without_newest_torchaudio / "a ROCm 10 pip index fixture whose ROCm 10.1.0 torchaudio is only a wheel this host cannot install" — a repo-wide search for this exact step text against every .feature file returns zero matches. cargo clippy won't catch this: the #[given(...)] macro registers the function in cucumber's step table, so rustc sees it as used.

Confidence 95 · mechanical · Fix: either delete this function, or add the scenario it looks like it was written for (a mirror of the existing ROCm-10.0.0 variant a few lines below, for the newer build).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in afcf0d4: the unused step is deleted. The update scenario it belonged to is replaced by therock-next-32, which uses its own fixture step.

/// four-component release such as `7.2.4.70204`, and that is still a stable
/// release of the 7.2 line. The ordering key drops trailing zeros, so a
/// missing minor component reads as zero, as PEP 440 pads it.
fn published_in_stable_line(published: &str, major: u32, minor: u32) -> bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — the major-equality conjunct here has no test that isolates it; a mutation removing it would pass the whole suite.

Every negative matches_version(...) assertion this PR adds (in therock/source.rs's two_component_version_pin_routes_and_matches_its_release_line, plus the tarball.rs/wheel.rs tests built on it) differs from a matching positive one only in the minor component with the same major ("10.0.0"/"10.10.0" vs line 10.1; "7.15.0" vs line 7.14; "7.20.4.70204" vs line 7.2). The one cross-major case in the fixtures ("7.14.1" checked against line 10.1) differs in both major and minor, so it doesn't isolate the major check either.

Deleting && key.release.first().copied().unwrap_or(0) == u64::from(major) (or replacing it with a tautology) leaves every test in this diff green, because no test publishes a version whose major differs from the pin's while the minor coincides (e.g. 11.1.0 against line 10.1).

Confidence 80 · logic/test-coverage · Fix: add one assertion such as !line.matches_version("11.1.0") for line = RuntimeVersionSelector::version("10.1").

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in afcf0d4: two_component_version_pin_routes_and_matches_its_release_line now asserts 10.1 matches neither 11.1.0 nor 9.1.0. With the major conjunct replaced by true, that test fails.

);
// Another ROCm release can publish a torchaudio this host can install,
// so a different `--version` can clear this one.
return Err(VersionUnsatisfied::new(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — the PR description characterizes this file's change as a no-op "move," but it is neither a move nor behavior-preserving.

torchaudio.rs already exists as its own module at the merge-base — this diff doesn't create or move it (only next_hint.rs is genuinely new). What actually changes here is torchaudio_pin's failure path: a plain bail!(...) becomes return Err(VersionUnsatisfied::new(Some(rocm_version), ...).into()). That's a real, user-visible behavior change — a torchaudio-shaped wheel failure now becomes eligible for the new "try another release" recovery hint, where before it wasn't — described in the PR body as "Internal: the torchaudio pin moves from wheel.rs into therock/resolve/torchaudio.rs... The moves change no behavior."

Confidence 90 · Step-9 scope-misstatement · Fix: correct the description's "Internal" paragraph to describe the actual change (an error-type/behavior change in an existing module), not a move.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, thanks. The PR description no longer calls it a move: it now says torchaudio_pin in the existing torchaudio.rs raises its "no torchaudio built for ROCm X" failure as the VersionUnsatisfied marker, and that this is what makes the failure eligible for the release list. The only new module is next_hint.rs.

} else {
""
};
format!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this whole-install-command suggestion (the rocm update / unpinned path) is unit-tested only, never driven end-to-end to prove it actually clears the condition it's printed for.

This repo's own AGENTS.md (§3) says a remediation command "must exist, accept those flags, and actually clear the condition that printed it," and that "a unit test asserting the internal helper does NOT discharge this; it proves the function, not the behavior." The pinned-install case of this same hint is covered that way (therock-next-25/-27 re-run the suggested command and assert success), but no scenario in this diff drives an rocm update failure into this branch and re-runs the suggested command.

The PR description itself names the gap ("The update's install-command suggestion... has no scenario now... is covered by [two unit tests]"), but per the rule just quoted, unit coverage of the internal helper doesn't discharge this, and therock-next-28's existing loopback-fixture pattern for a registered update runtime shows an e2e scenario here is practical, not impractical.

Confidence 75 · logic/test-coverage · Fix: add a scenario that registers a Next-layout runtime whose update fails into this branch (e.g. a uv pip compile failure), asserts the suggested command, and re-runs it to confirm success.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in afcf0d4 as therock-next-32. It registers a ROCm 10 wheel runtime with the toolchain, serves an index whose newest release has no installable package set (real uv pip compile, no solution), and runs rocm update --apply --dry-run. The failure suggests rocm install sdk --channel release --format wheel --version 10.0.0 --family gfx1200 --devel and leaves out the release that failed; the scenario then previews exactly that command and checks it resolves 10.0.0 with the device extras and the toolchain.

Comment thread docs/architecture.md Outdated
| [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs) | Resolving the wheel stack (`rocm`, `torch`, `torchvision`, `torchaudio`) and device payload a request installs (`resolve_pip_runtime`, `preview_pip_runtime`); on the next layout it drops a `rocm` build that lacks a framework package (`rocm_versions_with_full_stack`) before pinning torchaudio. | domain |
| [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs) | Resolving the wheel stack (`rocm`, `torch`, `torchvision`, `torchaudio`) and device payload a request installs (`resolve_pip_runtime`, `preview_pip_runtime`); on the next layout it drops a `rocm` build that lacks a framework package (`rocm_versions_with_full_stack`) before pinning torchaudio; a `major.minor` pin matches the newest release in that line. | domain |
| [`therock/resolve/torchaudio.rs`](../apps/rocm/src/therock/resolve/torchaudio.rs) | Which torchaudio build a ROCm 10 or newer install pins, and why (`torchaudio_pin`, `TorchaudioPin`); the plan line and install-log note derive from the same pin. | domain |
| [`therock/source.rs`](../apps/rocm/src/therock/source.rs) | Where an install reads from: channel, hosting layout (the next layout's `stable.repo.amd.com`/`nightly.repo.amd.com`, or the earlier stream kept for pins that exist only there) and artifact bases (with the opt-in base override), the one routing rule `layout_serving` and its refusals (`select_source_layout`, `gate_install_layout`), plus the pinned version selector and the resolved family. | domain |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking — this entry (and therock/version.rs's own module doc comment) wasn't updated for the new line-pin grammar, unlike the sibling tarball.rs/wheel.rs entries that were.

source.rs and version.rs implement the core of this PR's grammar (version_request_matches, stable_release_line_pin, published_in_stable_line), but are the two entries that don't mention it. cargo xtask check-architecture-doc only checks that an entry exists, not that its prose is current (per AGENTS.md §6), so this won't be caught mechanically.

Confidence 75 · Fix: add a clause to this row and to version.rs's module-level //! doc naming the line-pin concept.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9750ee0: the source.rs and version.rs rows and version.rs's module doc now name the major.minor line pin. The same commit updates the assistant system prompt in chat/turn.rs, which described --version as an exact version only.

The next-layout recovery hint assumed the release channel: it filtered the
listing for stable releases and printed `--channel release`. A nightly
request also reads the next layout, which publishes only prerelease builds
for it, so a failed nightly install or update was told no stable release
exists, or handed a command that moves it to the release channel.

`NextLayoutRecovery` now carries the request's channel, the filter and the
printed `--channel` use it, and a nightly failure gets no release list:
the wheel context prints the source line alone and the tarball context
says no nightly archive matched.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
therock-next-32 registers a ROCm 10 runtime whose newest release lists every
package but has no installable set, so a real `uv pip compile` reports no
solution. The update fails with the install command for the other release,
and the scenario runs that command and checks it resolves with the same
extras. It goes red if the "No solution found" classification stops
matching uv's output.

Also isolate the line-pin major check (`10.1` must not match `11.1.0` or
`9.1.0`) and drop a step definition no scenario used.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The version.rs module doc and the architecture entries for source.rs and
version.rs now name the `major.minor` line pin, the assistant prompt no
longer says `--version` is only an exact version, and torchaudio.rs's doc
names functions it does not import in plain backticks.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…-version-pin

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

# Conflicts:
#	tests/e2e-cucumber/tests/e2e/therock_steps.rs
@rominf rominf changed the title fix(sdk): treat a two-component version pin as that release line fix(sdk): treat a two-component version pin as that release line, and suggest a ROCm 10 release only when one can help Nov 17, 2026
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Changes since the review of f65bb1e9:

  • Nightly on the next layout (a020c6c): the recovery hint now takes the request's channel, and a nightly failure gets no release list and no --channel release command.
  • The rocm update install suggestion end to end (afcf0d4): new therock-next-32 drives an update into a real uv pip compile "No solution found", asserts the suggested rocm install sdk ... --devel command, and runs it. The same scenario pins the uv phrase the classification depends on.
  • Tests and docs (afcf0d4, 9750ee0): the line-pin major check is isolated (11.1.0 and 9.1.0 against 10.1), the unused step is gone, the architecture entries, version.rs module doc and the assistant prompt describe line pins, and the torchaudio.rs doc links are plain backticks.
  • Description: rewritten from the final diff. It now says what changed in torchaudio.rs (an error-type change that makes that failure eligible for the release list, not a move) and names both parts of the change.
  • Merged current main (f9a6d0a); the only conflict was the e2e fixture builder, where main's unpaired-torch case and the new unsolvable-build fixture now both apply.

On the review-level points:

  • Scope. Agreed that it is two changes; the title and summary now name both. I would rather not split now: the hint rework is what replaces the --channel nightly advice rocm install sdk --version 10.1 searches the 7.x index and fails #808 reports, and it has been through four review rounds in this PR. If you still prefer two PRs, I can split out the hint work as a follow-up.
  • Line pins for every major. Intentional. Accepting 10.1 but treating 7.14 as an exact version no release matches would make the same grammar mean two things; therock-next-30 covers 7.14.
  • "No solution found". Kept, with a scenario that drives real uv to it (details on the thread). uv offers no structured reason or distinct exit code for an unsatisfiable set.

@rominf
rominf requested a review from juhovainio November 17, 2026 13:02

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-read the whole change at this head. The two-component pin routing and matching look right to me, and the scenarios that re-run the suggested command are a good way to prove the advice rather than just pin the wording. I found nothing that breaks the #808 fix itself; the change I'm asking for is about the recovery hint:

  1. The suggested release comes from the unfiltered rocm index listing, so the newest "alternative" may lack torchaudio or a solvable stack on the host and fail the same way (details inline). Either narrow the candidates to releases that can actually install, or soften the wording and the README / testing.md claim that it suggests the newest.
  2. chat/turn.rs now describes line pins in the assistant prompt. Please confirm skills/rocm-cli-assistant/SKILL.md and docs/llm-tool-use.md don't still say exact-version-only; they aren't in the diff, so I couldn't check.

Not blocking, but worth a thought: this PR carries both the #808 line-pin fix and the larger recovery-hint rework (marker type, update-command synthesis, torchaudio refactor). The body discloses both, but splitting them would make each easier to review and revert. I also did not check callers of matches_version / parse_version outside the diff, which now see line pins.

format!("No archive{scope} was found for {requested} on {layout}.")
}
};
let mut alternatives = published

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

published here is the unfiltered rocm index listing, so the newest "alternative" may lack torchaudio or a solvable stack for this Python/host and fail the same way (the e2e fixtures only cover cases where the suggestion happens to work). Consider narrowing to releases with a full stack, or wording the suggestion as untested.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

atus Review — needs work

Full review of the whole change (apps/rocm/src/{cli.rs, therock.rs, therock/version.rs, therock/source.rs, therock/resolve.rs, therock/resolve/{wheel,tarball,torchaudio,next_hint}.rs, chat/turn.rs}, README.md, docs/{architecture,manual-testing,testing}.md, tests/e2e-cucumber/features/therock_next_generation.feature, tests/e2e-cucumber/tests/e2e/therock_steps.rs), base 9e50eac5a03df41a2f23e73dac3a7ec470ea78a7 → head f9a6d0ad749c1e385020d08675aaf0618884ef35.

Fixes #808 as claimed: the two-component line-pin routing bug (--version 10.1 reading the wrong index and suggesting an impossible --channel nightly) is genuinely fixed and well tested. The change also bundles a substantial failure-recovery-hint rework (a new ~595-line module, a typed VersionUnsatisfied marker, update-command synthesis) that #808 didn't literally ask for, though #808's own "Possible solution" section anticipated it ("The canonical-stream hint should not be used for a ROCm 10 pin"). No other ticket is named.

Blocking

  • The next-layout recovery hint suggests a release without checking whether that release itself can actually install — it can recommend a release that fails the same way the attempted one did. apps/rocm/src/therock/resolve/next_hint.rs:163-177, fed by apps/rocm/src/therock/resolve/wheel.rs:826,862 (unfiltered rocm_versions passed straight through to the hint for every failure path) and the narrow existing guard at wheel.rs:1001,1032 (with_full_stack.is_empty(), which only suppresses all suggestions when literally zero releases work, and which can't even run for a pinned request since with_full_stack is always empty there by construction, wheel.rs:974). Full trace and a concrete counter-example are in the inline comment. This directly contradicts the PR's own stated and titled goal — "suggest a ROCm 10 release only when one can help" — and no test in this diff exercises the case where 2+ alternatives are published and the newest of them is also broken (every existing fixture has either zero working alternatives, correctly suppressed, or exactly one alternative that happens to work).
    Confidence 85 · architectural · Fix: filter the hint's alternatives through the same full-stack/solvability check rocm_versions_with_full_stack already applies to the unpinned auto-pick, and add a test with 2+ alternatives where the newest is also unable to install.

Non-blocking

None. (Several lower-confidence candidates were investigated and dropped after verification: a tarball four-component-version exclusion turned out to be explicit, deliberate, documented behavior, not a gap; an unreachable <gfx>-placeholder match arm for one enum variant is harmless defensive code; a double computation of tarball_family_candidates on the failure-only path is a pure style nit; none of these clear the bar for a reported finding.)

Decisions for the author

  • Scope: this PR is still two changes — the line-pin routing fix #808 asks for, and a separately-substantial failure-recovery-hint rework (new error marker, new module, rocm update install-command synthesis). This has been raised in multiple earlier review rounds and is disclosed in the PR body; still worth a final confirmation that shipping both together, rather than splitting, is the intended approach for this repo.

Positive signals

  • The channel-hardcoding gap from an earlier review round is now fixed correctly and thoroughly: NextLayoutRecovery carries the request's real channel, suggests_releases() gates the entire hint (including the install-command branch) on channel == Release, and both the wheel and tarball sides have dedicated nightly-specific regression tests.
  • A previously dead e2e step definition (pip_index_fixture_without_newest_torchaudio) has been removed, and the published_in_stable_line major-vs-minor check (10.1 must not match 11.1.0/9.1.0) now has its own isolating assertion — both confirmed fixed by direct diff inspection.
  • The --version help text now has a dedicated regression test reading the clap help directly (install_sdk_version_help_describes_line_and_exact_pins), closing a gap flagged in an earlier round.
  • Doc-sync is thorough and consistent across README.md, --help, docs/testing.md, docs/manual-testing.md, docs/architecture.md and the chat assistant's system prompt; skills/rocm-cli-assistant/SKILL.md and docs/llm-tool-use.md were checked directly and contain no --version-specific wording that needed updating.
  • Three e2e scenarios (therock-next-25/-27/-32) genuinely parse the suggested command out of real CLI output and re-execute it to prove it clears the condition, rather than only pinning its wording.
  • The tarball ordering fix is exercised by a fixture that deliberately uploads the older release after the newer one, so the test discriminates version-first from upload-time-first ordering rather than passing under either rule.

Deployment notes

No schema/migration change, no new dependency. CI on the current head is green on every job that reports a conclusion (46 check-runs; one self-hosted E2E tests (MI300X, current) lane was still queued at review time, but the consolidated E2E consolidated report job it feeds already completed successfully, so this reads as runner availability rather than a signal to wait on). No real merge conflicts against current main — confirmed directly via git merge-tree (zero conflict markers); GitHub's fluctuating mergeable_state (blocked/unknown) reflects required-review/status-check gating, not a dirty merge.

What this covered

Full diff 9e50eac5…f9a6d0ad read in full. Four groups ran as independent, blind fan-out workers (version.rs/source.rs/therock.rs/resolve.rs; wheel.rs/tarball.rs/torchaudio.rs/next_hint.rs; the e2e feature+step-definition pair) after an initial concurrency-limit failure was resolved by a single re-spawn, per this method's own retry rule — each was blind to the PR description, to each other, and to all prior review discussion. The fourth group (cli.rs, chat/turn.rs, README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md) could not be spawned as an independent worker after two attempts hit a shared concurrency limit; it was read directly in this consolidating context instead and is flagged here as not an independent pass, per this method's degraded-fan-out disclosure rule.

The blocking finding above was not raised by any of the three independent workers — it surfaced at consolidation, prompted by re-examining a still-open point from an earlier review round at this exact head commit, then independently re-verified from scratch against the current source (the quoted line numbers, call chain and counter-example are from direct tracing done here, not assumed from that round). Two other points from that same earlier round were also checked directly rather than carried forward: the assistant-prompt/skill-doc staleness question is resolved (checked clean, see Positive signals), and the "two changes in one PR" scope question is restated above as a Decision for the author since it remains unresolved.

Did not run cargo build/cargo test/clippy myself (no Rust toolchain in this environment); the blocking finding and all logic observations are hand-traced against the source, not executed. History and prior-changes passes were not run as separate passes. Agent-instruction adherence was checked against the root AGENTS.md (module-size cap, architecture-doc requirement, remediation-command and doc-sync rules). Not reconciled against the full existing PR review discussion beyond the two specific points named above.


🤖 Automated by an AI agent

Comment on lines +163 to +177
let mut alternatives = published
.iter()
.filter(|version| attempted != Some(version.as_str()))
.filter(|version| {
layout_serving(
recovery.channel,
Some(&RuntimeVersionSelector::Version((*version).clone())),
) == SourceLayout::Next
})
.map(String::as_str)
.collect::<Vec<_>>();
alternatives.sort_by(|left, right| compare_version_strings(right, left));
alternatives.dedup();
let other = if attempted.is_some() { " other" } else { "" };
let Some(newest) = alternatives.first() else {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the suggested "other release" is never checked for the same failure it's recovering from — it can suggest a release that fails identically.

alternatives here is filtered only by (1) not the version that was just attempted, and (2) layout_serving(...) == SourceLayout::Next (same-channel routing). Nothing here checks whether the candidate actually has torchaudio / a solvable uv pip compile set — the two failure modes this exact hint exists to recover from.

Traced the only source of published: wheel.rs:826 loads the raw, unfiltered rocm package index (load_simple_index_versions(..., "rocm", ...)), and wheel.rs:862 passes that same unfiltered list straight through as Some(&rocm_versions) to wheel_resolution_failure_context, for every failure path (pinned or unpinned, torchaudio-missing or uv-no-solution). The only existing safeguard, wheel.rs:1001/1032 (with_full_stack.is_empty()), only fires when version_selector.is_none() and literally zero published releases have a full stack anywhere — it suppresses the suggestion entirely in that one narrow case, but does nothing when some releases work and some don't. In particular, for a pinned request (--version 10.1.0 with no torchaudio), with_full_stack is always Vec::new() by construction (wheel.rs:974: if version_selector.is_none() { ... } else { Vec::new() }), so that guard can never even evaluate the real availability for a pinned failure.

Concretely: if the next layout publishes 10.1.0 (requested, no torchaudio), 10.3.0 (newest, also no torchaudio) and 10.2.0 (has torchaudio), this code picks 10.3.0 as "the newest other release" and tells the user to re-run with it — which fails the same way, when 10.2.0 would have worked. This directly contradicts the PR's own stated goal ("suggest a ROCm 10 release only when one can help") and its title. No test in this diff exercises the case where more than one other release is published and the newest of them is also unable to install — every existing fixture (therock-next-31/32, the unit tests here) has either zero working alternatives (suppressed correctly) or exactly one alternative that happens to work.

Fix: have next_layout_unsatisfied_hint (or its caller) filter alternatives against the same full-stack/solvability check rocm_versions_with_full_stack already performs for the unpinned auto-pick, rather than only the attempted-version and layout-routing exclusions, and add a test with 2+ alternatives where the newest one is also broken.

Confidence: 85 — directly traced against the source, not assumed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rocm install sdk --version 10.1 searches the 7.x index and fails

4 participants