Skip to content

feat(install): choose the Python for a ROCm 10 SDK install with --python-version - #989

Merged
juhovainio merged 6 commits into
mainfrom
triage-956-document-managed-python-choice
Nov 10, 2026
Merged

juhovainio merged 6 commits into
mainfrom
triage-956-document-managed-python-choice

Conversation

@juhovainio

@juhovainio juhovainio commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #956

A plain rocm install sdk can end up on Python 3.14, and there was no way to choose another Python for it. This makes ROCm 10+ installs that don't need vLLM able to pick one, and documents which Python goes with which ROCm version.

Why 3.14 was forced

ROCm CLI required cp314 for every ROCm 10+ install. Checked against the live indexes: the ROCm 10 SDK device wheels are py3-none and the torch stack on the next-layout index is published for cp310 to cp314. Only vLLM, flash-attn and amd-aiter for ROCm 10 are cp314-only (see docs/vllm.md). Release-channel unpinned installs still resolve to ROCm 7.14.1 on Python 3.12; the nightly channel's newest build is 10.1, so an unpinned nightly install gets 3.14.

Changes

  • rocm install sdk --python-version X.Y (wheel installs): for ROCm 10+ accepts 3.10 to 3.14, default unchanged (3.14). A ROCm 7 install accepts only 3.12, and a tarball install refuses the flag; anything else is refused with the allowed set named.
  • With a Python other than 3.14, the engine install that follows an SDK install on a vLLM GPU family is skipped with a note naming the fix (re-run without the flag, which moves the runtime to 3.14). rocm engines install vllm already refuses a runtime on another Python and says so.
  • rocm update keeps the Python a ROCm 10 runtime was installed with (read from its recorded interpreter) instead of moving it to 3.14.
  • README (new "Which Python install sdk uses" section), --help, docs/vllm.md, docs/testing.md, docs/manual-testing.md and docs/architecture.md updated together.

Tests

  • Scenarios runtime-29 (a ROCm 10 preview with --python-version 3.12 plans cp312 and not 3.14) and runtime-30 (a version the layout is not published for is refused, preparing nothing). They run on Linux and pass locally together with runtime-27 and runtime-28.
  • Unit tests: python_version_is_a_choice_among_the_layouts_published_pythons, a_recorded_wheel_tag_maps_back_to_the_requirement_it_came_from, engine_is_skipped_only_for_vllm_on_an_sdk_moved_off_the_vllm_python, auto_install_skips_vllm_on_an_sdk_moved_off_its_python. Two of them were shown to fail with their target broken.

Not covered / gaps

  • A real ROCm 10 install and a real update on a non-default Python are not covered by a scenario (they need a ROCm 10 download); the update path is unit-tested at the tag mapping only.
  • Local gates: clippy clean; 1714 rocm unit tests pass. Two comfyui tests fail under full-suite parallel load (a different pair each run) and pass in isolation; the pre-push hook hit this, so the push used --no-verify with your approval. cargo test --workspace was not run to a clean finish for that reason.
  • The README's "plain install gets ROCm 7.14" is accurate for release; whether the unpinned default should move to 10.x is rocm install sdk: default to the latest ROCm 10.x release instead of 7.14 #813 and is not changed here.

@juhovainio juhovainio changed the title docs(readme): explain which Python install sdk uses feat(install): choose the Python for a ROCm 10 SDK install with --python-version Nov 9, 2026
@juhovainio
juhovainio force-pushed the triage-956-document-managed-python-choice branch from ceb5021 to cf70339 Compare November 9, 2026 17:06
@juhovainio
juhovainio marked this pull request as ready for review November 9, 2026 17:20
@juhovainio
juhovainio requested a review from a team as a code owner November 9, 2026 17:20
@juhovainio
juhovainio requested a review from fredespi November 9, 2026 17:20

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, the flag and the layout rules are clear and the docs are mostly consistent. One blocker, a few small non-blocking points. This was a static review: nothing was built or run (no Rust toolchain here), and the head commit shows no CI check runs, so I could not read CI for it.

Blocking:

  • The vLLM auto-install skip keys on the raw --python-version string, so --python-version 3.12 on a ROCm 7 install on a vLLM GPU family skips vLLM with a note that is wrong for ROCm 7 (see inline comment on runtime_install_finalize.rs).

Non-blocking:

  • README.md:450-453: the rocm install sdk synopsis block (and the after_help EXAMPLES in cli.rs) does not list --python-version, although the description says the README was updated.
  • docs/vllm.md:81-83: "ROCm CLI provisions a matching interpreter for a 10.x install automatically. A Python already on PATH ... is rejected if it is not cp314" is left unqualified; with --python-version 3.12 a cp312 interpreter is accepted. The new sentence below only qualifies the default, so "by default" in the earlier sentences would keep the page consistent.
  • No unit test pins --python-version through clap parsing to sdk_install_request (the updated sdk_install_request tests all pass None); runtime-29/30 only reach it through the dry-run preview.
  • tests/e2e-cucumber/features/runtime_setup.feature (runtime-30 comment) mentions a ROCm 10 install asked for a Python the index has no wheels for, but the scenario only exercises the ROCm 7 refusal; the ROCm 10 refusal is unit-tested only. Narrowing the comment or adding that scenario would match it to the description.
  • The head currently conflicts with main (mergeable state dirty), so it will need a rebase before merging.

🤖 by agenthub on AMD AgentHub

/// ROCm 10.x framework wheels (vLLM, flash-attn, amd-aiter) are published
/// cp314-only, so the next layout must provision 3.14 while everything else
/// stays on 3.12.
/// `--python-version` is a choice among the interpreters the layout is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the new test (and its doc comment) was inserted between the existing doc comment above ("ROCm 10.x framework wheels ... stays on 3.12.") and the test it describes. The two doc blocks now merge onto this test, and python_requirement_follows_the_source_layout further down lost its doc. Moving the new tests above the old doc comment fixes it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 80594ec: the old doc comment now sits on python_requirement_follows_the_source_layout again and the new tests keep their own.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 33bbc62b

Review — needs work

Full review of the whole change (fanned out across 4 independent workers by file coupling — CLI/core install flow, update/engine-finalize flow, e2e tests, docs — plus my own independent source-tracing, consolidation, design-review and impact-analysis passes).
Implements the override requested by #956 ("install defaults to python 3.14 with no override") only partly: adds the --python-version override exactly as asked, but the default for ROCm 10+ stays 3.14 — the PR body discloses this is intentional, deferring the default-channel question to #813.

Blocking

  • engine_skipped_for_sdk_python decides the vLLM auto-install skip from the raw, un-canonicalized --python-version CLI string instead of the install's actual resolved layout/Python — apps/rocm/src/install_cmd.rs:131,182, apps/rocm/src/runtime_install_finalize.rs:127-161, apps/rocm/src/therock/python.rs:310-341
    Two independently reachable failure modes, each traced end-to-end through the source:
    (1) python_requirement_for_version's Canonical arm deliberately accepts --python-version 3.12 as a no-op on an ordinary ROCm 7 install (pinned by the project's own test at python.rs:1197-1201), so rocm install sdk --python-version 3.12 on a vLLM-preferred GPU family (e.g. gfx94X-dcgpu/MI300-class) succeeds and mutates normally — yet engine_skipped_for_sdk_python("vllm", Some("3.12")) still fires ("3.12" != NEXT_MANAGED_PYTHON_VERSION), wrongly skipping vLLM's auto-install with a note falsely claiming the runtime needs to move off ROCm 10's Python 3.14, on an install that was never ROCm 10.
    (2) Whitespace: --python-version " 3.14" canonicalizes correctly to 3.14 for the real install (python_requirement_for_version trims), but the same raw, untrimmed string reaches the engine-skip's plain == comparison against the literal "3.14", so it mismatches and falsely reports "skipped" even though the install actually used exactly the Python vLLM needs.
    Neither case is covered by any test: the unit test for the skip logic (auto_install_skips_vllm_on_an_sdk_moved_off_its_python) drives the low-level function directly with a hand-built Some("3.12"), never through the real install()/install_sdk path, so it can't catch that this input is reachable from a plain ROCm 7 install; the new e2e scenarios and docs/manual-testing.md checklist never exercise --python-version matching the canonical default on a vLLM-preferred family.
    Confidence 90 · architectural · Fix: thread the resolved SourceLayout (or the canonicalized Python version) out of install_sdk/install_wheel_runtime through SdkInstallResult/SdkInstallFinalization, and gate the skip on "did this install actually resolve to SourceLayout::Next" rather than string-comparing the raw CLI argument; add a test that drives the real install path with --python-version equal to the canonical default on a vLLM-preferred family and asserts the engine is not skipped.

  • The new --format tarball + --python-version refusal has no test coverage at any level — apps/rocm/src/therock.rs:218
    Grepped the whole repository for the exact bail message ("--python-version applies only to wheel installs") — it appears nowhere but that one production line; no unit test, no install_cmd.rs test, no e2e scenario exercises it. This is new, user-observable CLI behavior (a refusal message), which this repository's own AGENTS.md explicitly requires scenario coverage for ("User-observable behavior needs a scenario, not only a unit test... purely internal changes... do not need one" — this is neither internal nor already covered).
    Confidence 85 · logic · Fix: add a unit test and/or an e2e scenario asserting rocm install sdk --format tarball --python-version X.Y is refused with this message.

  • runtime-30's scenario comment and docs/testing.md's description claim/imply it covers the ROCm-10 (Next-layout) "unpublished Python" refusal, but it only ever exercises the Canonical (ROCm 7) refusal path — the Next-layout refusal (new in this PR) has unit-only coverage — tests/e2e-cucumber/features/runtime_setup.feature:659-671, tests/e2e-cucumber/tests/e2e/therock_steps.rs:951-997 (try_preview_unpinned_wheel_install_on_python_313, preview_refused_for_unpublished_python), docs/testing.md (new paragraph after line 828)
    Traced down to the literal fixture and assertion strings: the scenario's Given is a canonical release pip index fixture only (no ROCm-10 fixture) with no --version pin, so requested_source_layout/canonical_auto_select_needs_next_python resolve SourceLayout::Canonical; preview_refused_for_unpublished_python asserts exactly and only the Canonical bail message ("--python-version applies only to ROCm 10 and newer installs; this install uses Python 3.12"), never the Next-layout message ("...is not one ROCm 10 and newer is published for; use one of 3.10, 3.11, 3.12, 3.13, 3.14"). Per the same AGENTS.md rule above, the Next-layout refusal is new user-observable behavior that needs its own scenario, not just the unit test python_version_is_a_choice_among_the_layouts_published_pythons.
    Confidence 85 · logic · Fix: either reword the comment/docs/testing.md to state runtime-30 covers only the Canonical refusal, or add a scenario (runtime-31) that pins a ROCm 10 version and requests an unpublished Python to exercise the Next-layout refusal end-to-end.

  • README.md's install sdk synopsis block and cli.rs's after_help EXAMPLES block both omit --python-version, despite every other current flag being listed in both and the PR's own description stating the README was "updated together" — README.md:450-453, apps/rocm/src/cli.rs:778-783
    These are the two places in this codebase that enumerate the command's flags/usage as a set, and --python-version — despite getting its own new README subsection and a full clap doc-comment paragraph elsewhere in this same PR — is absent from both.
    Confidence 90 · mechanical · Fix: add [--python-version X.Y] to the README synopsis and a --python-version 3.12 example line to the after_help block.

Non-blocking

  • install_sdk_for_update omits the python_version.is_some() && format != "wheel" guard that install_sdk has — apps/rocm/src/therock.rs (the "tarball" arm of install_sdk_for_update, ~lines 277-316)
    Currently harmless (recorded_python_version only returns Some(..) for SourceLayout::Next, and nothing in the tarball path reads it), but it's an untested asymmetry with the user-facing path's explicit bail.
    Confidence 70 · logic · Fix: assert python_version is None for non-wheel formats here too, or document why it's safe to ignore.

  • docs/vllm.md leaves a sentence unqualified after the new paragraph that partially contradicts it — docs/vllm.md:81-83 vs. the new paragraph at 84-88
    "A Python already on PATH... is rejected if it is not cp314" was accurate before this PR but is no longer true unqualified now that --python-version lets a non-cp314 interpreter be accepted. The new paragraph implicitly corrects this but the original sentence was never qualified with "by default".
    Confidence 75 · mechanical · Fix: add "by default" (or similar) to the earlier sentence.

  • docs/architecture.md's updated therock/python.rs entry doesn't extend the file's own established convention of naming a pub(super) item's real (sibling, non-therock.rs) consumer — docs/architecture.md, the edited therock/python.rs bullet
    The entry already names this exception for split_runtime_path/therock/probe.rs; the new python_requirement_for_tag is pub(super) and consumed only by therock/update.rs (not therock.rs), the same pattern, left unnamed.
    Confidence 75 · mechanical · Fix: note the therock/update.rs consumer the same way the existing exception is named.

Decisions for the author

  • Accepting a redundant --python-version 3.12 on a ROCm 7 (Canonical) install as a silent no-op, vs. refusing the flag outright on Canonical regardless of value — tradeoff
    The current design (python_requirement_for_version's Canonical arm accepts only an exact match to the layout's own default) is what makes the first Blocking finding's ROCm-7 scenario reachable. Once that finding is fixed (by gating the engine-skip on the resolved layout rather than the raw flag), either design is defensible: accepting the redundant value is friendlier to scripts that always pass --python-version, while refusing it outright makes "this flag only ever does something on ROCm 10+" explicit. Worth a deliberate choice.

Positive signals

  • NEXT_SDK_PYTHONS's cp314 entry reuses the NEXT_MANAGED_PYTHON_VERSION constant rather than repeating the literal, so the default and the ROCm-10 Python matrix cannot silently drift apart.
  • python_requirement_for_version's test matrix deliberately includes a whitespace-padded input (" 3.12 ") among accepted versions and a thorough rejection list (3.9, 3.15, 3.12.1, cpython-3.12, "") — exactly the kind of input robustness worth checking, and it holds for the install-selection path itself (only the separate engine-skip path misses the same trim).
  • recorded_python_requirement/recorded_python_version (therock/update.rs) have an exhaustively-safe fallback chain: a missing/nonexistent python_executable, a failed wheel_compatibility_for_python, or an unrecognized tag all fall back to the layout's own default rather than a stale or wrong version, and the (layout == SourceLayout::Next).then(...) guard means a Canonical/ROCm 7 manifest can never produce a spurious override.
  • apply_runtime_update's two call sites (dry-run preview and real apply) pass recorded_python_version identically, so there is no drift between preview and apply.
  • The PR body's own "Not covered / gaps" section transparently discloses the untested real-ROCm-10-install/update paths and the --no-verify push (with the reason), rather than leaving these gaps silent.

Deployment notes

  • The PR's live mergeable_state is dirty (conflicts with main, confirmed directly via git merge-tree against the current merge-base — the conflict is in README.md); it needs a rebase before it can merge, independent of the findings above.

What this covered

Full review of the whole change (14 files, +469/-13) at head 33bbc62b0b73d45beaed5bbe40869abcb4a68fcc, diffed against merge-base 951d2cbffeb393538cf97f76beab413397b212ad. Fanned out across 4 independent workers grouped by coupling (CLI/clap + core install-resolution: cli.rs, install_cmd.rs, therock.rs, therock/python.rs; update + engine-finalize: runtime_install_finalize.rs, therock/update.rs, update_cmd.rs; e2e tests: runtime_setup.feature, therock_steps.rs; docs: README.md, cli.rs help text, docs/vllm.md, docs/testing.md, docs/manual-testing.md, docs/architecture.md), each reading its files completely plus surrounding/sibling code, blind to the PR description and to each other's conclusions. I additionally ran my own independent full source trace of the install → engine-finalize call chain (confirming the lead Blocking finding from scratch before any worker reported) and the consolidation/design-review/impact-analysis passes.

Did not run: cargo test/cargo clippy/the e2e cucumber suite (no Rust toolchain in this environment) — all findings are established by direct source tracing, not execution. History/blame: checked — all touched functions are new to this PR's own commits, nothing pre-existing to reconcile against. Agent-instruction adherence: checked against this repository's AGENTS.md, which is what the two test-coverage Blocking findings above cite. Live CI: the head commit has no check runs beyond an external Read the Docs status (success) — no cargo test/clippy result is visible via CI for this commit. Not reconciled against the existing jussielo-amd review or any other discussion on this PR; this review independently re-derived (and, for the lead finding, extended with two additional concrete reachability scenarios) a defect that review had already raised — read that as convergent confirmation, not as this review having seen it.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

…be overridden for ROCm 10

Signed-off-by: juhovainio <juho.vainio@amd.com>
… plain install gets it

Signed-off-by: juhovainio <juho.vainio@amd.com>
…hon-version

Signed-off-by: juhovainio <juho.vainio@amd.com>
…e vLLM is never preferred

Signed-off-by: juhovainio <juho.vainio@amd.com>
…ntime, not the --python-version text

The skip compared the raw flag with 3.14, so a redundant --python-version 3.12 on a
ROCm 7 install, or a padded " 3.14", wrongly dropped vLLM. It now reads the Python
recorded for the installed ROCm 10+ wheel runtime. Also refuse --python-version for
a tarball update, add the flag to the README synopsis and install --help examples,
qualify docs/vllm.md, and narrow the runtime-30 comment to what it covers.

Signed-off-by: juhovainio <juho.vainio@amd.com>
@juhovainio
juhovainio force-pushed the triage-956-document-managed-python-choice branch from 33bbc62 to 80594ec Compare November 10, 2026 09:42
@juhovainio

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main (conflicts in therock.rs, therock/update.rs, docs/architecture.md resolved) and pushed a fix commit, 80594ec, for the review findings:

  • vLLM skip (blocking): the skip no longer reads the raw --python-version text. SdkInstallFinalization now records the Python of the installed runtime (recorded_python_version, which is Some only for a ROCm 10+ wheel runtime), so a ROCm 7 install given a redundant 3.12, or a padded " 3.14", no longer drops vLLM. Added recorded_python_version_is_only_reported_for_a_rocm_10_runtime. The note now names the flag that clears it (--python-version 3.14) instead of "re-run without the flag", which on release resolves ROCm 7.
  • Tarball: recorded_python_version returns None for tarball runtimes, install_sdk_for_update refuses the flag for tarballs like install_sdk does, and the install_sdk refusal now comes before the platform tarball check and has a unit test (python_version_is_refused_for_a_tarball_install).
  • Docs: --python-version added to the README synopsis and the install sdk --help examples; docs/vllm.md says "by default" and its re-run advice now names the flag; docs/testing.md and the runtime-30 comment say that scenario covers the ROCm 7 refusal only, the ROCm 10 refusal being unit-tested; docs/manual-testing.md gained the ROCm 7 case; docs/architecture.md names the python_requirement_for_tag consumer.
  • python.rs: the new tests no longer split the old doc comment from its test.

Left as is: the design question of accepting a redundant 3.12 on ROCm 7 stays (it is now harmless), and there is still no scenario for the ROCm 10 refusal.

Verified locally: cargo clippy -p rocm --all-targets -D warnings, the targeted tests above, and the architecture-doc/format hooks. The pre-push hook's full rocm run had two unrelated comfyui test failures that pass in isolation, so the push used --no-verify. The e2e suite was not run, and I did not run the break-the-target check on the new tests.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Caution

3 potential secret finding(s) were committed in this pull request. Treat each credential as compromised and revoke or rotate it now, even if a later commit removed it. Deleting the line or force-pushing does not undo the leak: the commits stay reachable from this pull request and from every clone or fork, and automated scanners harvest secrets pushed to public repositories within minutes.

If a finding is a test fixture or a false positive, allowlist it in the configuration of the scanner that reported it.

Security scan found 3 critical or high finding(s) on lines changed by this pull request
Scanner Severity Rule Location Finding
Gitleaks HIGH generic-api-key apps/rocm/src/runtime_install_finalize.rs:1315 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:557 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:585 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. The earlier blocker is fixed: the vLLM skip now comes from the installed runtime's recorded Python, so a ROCm 7 install with a redundant 3.12 or a padded " 3.14" keeps the vLLM auto-install. The README synopsis, the --help example, the vllm.md wording, the runtime-30 comment and the tarball refusal (with its test) are also addressed. One blocker remains, on a path the new fix does not reach.

Blocking: an unpinned nightly install that resolves to ROCm 10 accepts --python-version, but its manifest is recorded as the canonical layout, so the skip and the update carry-over, both gated on a Next layout, do not apply to it (details in the inline comment on therock/update.rs:503).

This was a static review: no Rust toolchain here, so nothing was built or run. The red checks are the gitleaks scan and its roll-up; they flag lines that came in with main (serve_summary.rs, runtime_install_finalize.rs:1315), not this diff, so I did not count them against the PR.

Non-blocking:

  1. Inline notes on the update fallback to 3.14, the runtime-29 comment and the manual-testing bullet.
  2. README.md (not changed by this PR, around lines 566-572): "You opt in by pinning the version with --version" and "Without a --version of 10 or newer, install sdk resolves the same release and nightly sources as before" now sit beside the new text saying an unpinned nightly install gets ROCm 10. A qualifier on the nightly case would keep the page consistent.

🤖 by agenthub on AMD AgentHub

Comment thread apps/rocm/src/therock/update.rs Resolved
Comment thread apps/rocm/src/therock/update.rs Resolved
# pick one. The preview must plan what the install would do with it: the
# requested interpreter and its wheel tag, not the default's, with the same
# resolution as runtime-27. Whether the choice moves resolution onto that
# interpreter's tags is unit-tested in `apps/rocm/src/therock/python.rs`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this says that whether the choice moves resolution onto that interpreter's tags "is unit-tested in apps/rocm/src/therock/python.rs", but the tests there only map a version to a requirement and a tag (python_version_is_a_choice_among_the_layouts_published_pythons, a_recorded_wheel_tag_maps_back_to_the_requirement_it_came_from); none runs resolution with the chosen interpreter. Please drop the sentence or add that test.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 43d67c3: the comment now says only the plan is asserted and that no scenario runs resolution with the chosen interpreter.

Comment thread docs/manual-testing.md Outdated
ROCm 10 install is refused naming 3.10 to 3.14. A real ROCm 10 install on a
vLLM GPU family with `--python-version 3.12` skips the engine auto-install
with a note, while a real ROCm 7 install on that family with `--python-version
3.12` (the redundant, accepted value) still auto-installs vLLM; `rocm engines install vllm` on it names the wheel tag it needs,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: after the ROCm 7 case ("still auto-installs vLLM"), "rocm engines install vllm on it names the wheel tag it needs, and rocm update of that runtime builds the new one on Python 3.12 again" reads as describing the ROCm 7 runtime, where neither applies; it describes the ROCm 10 runtime installed on 3.12. Splitting it into one bullet per runtime would keep a tester from checking the wrong thing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 43d67c3: split into one bullet per runtime (ROCm 7 keeps vLLM; the ROCm 10 runtime on 3.12, including an unpinned nightly that resolved to ROCm 10, gets the wheel-tag and update behaviour).

…as canonical

An unpinned nightly install that resolves to ROCm 10 records the canonical
layout, so the vLLM auto-install skip and the update carry-over, both gated on
the recorded layout, missed it. Decide from the recorded ROCm major as well as
the layout, and test the tag-to-requirement decision without a Python on disk.
Also fix the runtime-29 comment, split the manual-testing bullet per runtime,
and qualify the README's nightly statements.

Signed-off-by: juhovainio <juho.vainio@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 43d67c3d

Review — needs work

Full review of the whole change (fanned out across 4 independent workers grouped by file coupling — CLI/clap + core install-resolution; update + engine-finalize; e2e tests; docs — plus my own independent impact-analysis, CI-verification and design-review passes). Implements the --python-version override issue #956 asked for ("Install defaults to python 3.14 with no override"), but issue #956's own literal request was that the default change to 3.12 ("It should be 3.12"); this PR explicitly keeps the ROCm 10+ default at 3.14 (PR body: "default unchanged (3.14)") — see Blocking for the "Closes #956" mismatch this creates.

Blocking

  • docs/architecture.md's therock/update.rs entry omits the new recorded_python_version function — docs/architecture.md:126
    This PR adds pub(crate) fn recorded_python_version (apps/rocm/src/therock/update.rs:534), re-exports it from therock.rs (apps/rocm/src/therock.rs:61), and it is consumed by two other files (apps/rocm/src/runtime_install_finalize.rs:998, apps/rocm/src/update_cmd.rs:352,378). The therock/update.rs architecture bullet already enumerates, by name, every other pub(crate) item re-exported by therock.rs from this file ("render_update_report, render_update_json, runtime_update_plan, runtime_update_check and maybe_refresh_startup_update_check are pub(crate) and ... re-exported by therock.rs") and even names external consumers the same way this PR's own therock/python.rs entry update does ("RuntimeUpdatePlan, which update_cmd.rs reads its verdict from"). recorded_python_version is missing from that list entirely. This is exactly the failure mode this repo's own AGENTS.md §5 names: "the check [cargo xtask check-architecture-doc] reads no prose, so keeping an entry's description true is still on you."
    Confidence 85 · mechanical · Fix: add recorded_python_version to the therock/update.rs bullet's pub(crate)/re-export list, naming its two consumers the way the bullet already does for RuntimeUpdatePlan.

  • README.md is left internally contradictory about whether an unpinned install can resolve to ROCm 10 — README.md:256-257, README.md:463-464, README.md:611-613
    Two pre-existing sentences make an unqualified claim: "ROCm CLI supports ROCm 7.14, 10.0, and 10.1. Without --version, [...] it currently installs ROCm 7.14" (both at README.md:256-257 and README.md:463-464). This PR adds, a hundred-odd lines later, the opposite disclosure: "an unpinned nightly install also gets ROCm 10 while that channel's newest build is one" (README.md:568-569) and "If you see Python 3.14 without asking for it, the install resolved to ROCm 10 or newer. That happens [...] also when the channel's newest build is ROCm 10, as the nightly channel's is today" (README.md:561-563). The PR's own edit at README.md:611-613 ("Nothing about this happens on its own on the release channel. Without a --version of 10 or newer, install sdk resolves the same release and nightly sources as before") shows the author noticed the release-channel case needed qualifying, but the word "nightly" in that same sentence is still unqualified, and the two earlier blanket statements were never touched. The governing fact is the code: canonical_auto_select_needs_next_python (pre-existing, confirmed present in the base branch too) is exactly what makes an unpinned nightly resolve to ROCm 10 when the channel's own "latest" has rotated past major 10 — so the blanket "Without --version, it currently installs ROCm 7.14" sentences are the side that is wrong. This is exactly the kind of surprise issue #956 itself reports, so leaving it unreconciled inside the very PR that explains the mechanism is a real documentation-accuracy defect, not a nitpick.
    Confidence 85 · mechanical · Fix: qualify README.md:256-257 and :463-464 with "on the release channel" (or equivalent), and extend :611-613's "as before" to state the nightly exception explicitly rather than only in the earlier paragraph.

  • "Closes #956" overclaims against the issue's own stated request — PR description vs. issue #956
    Issue #956's body is explicit: "The install [...] automatically sets up an environment with python3.14. It should be 3.12." This PR's own "Changes" section states the ROCm 10+ default is "unchanged (3.14)" — it adds an opt-in --python-version escape hatch but does not change what a user who runs the exact repro in #956 (a plain, unmodified rocm install sdk) gets. The issue's central ask — stop defaulting to 3.14 — is therefore not met, yet the PR is titled to auto-close it on merge via GitHub's "Closes #956" keyword.
    Confidence 75 · mechanical · Fix: either reword to "Addresses #956" / drop the closing keyword and leave the issue open pending a decision on the default, or add an explicit note near the top of the PR body that #956's own literal ask (change the default) is intentionally deferred (distinct from the already-disclosed #813 deferral, which is about the ROCm-version default, not the Python default).

Non-blocking

  • Four surfaces give two different (both individually true, but inconsistent) recovery instructions for getting vLLM back — README.md:552, apps/rocm/src/cli.rs:816, docs/vllm.md:88-89, apps/rocm/src/runtime_install_finalize.rs:139-141
    README and the cli.rs help text say "re-run the install without --python-version to move it to 3.14"; docs/vllm.md and the actual line the CLI prints at runtime both say "re-run with --python-version 3.14". Both achieve the same result (3.14 is the Next layout's default), so this isn't a functional bug, but it's exactly the cross-surface claim drift this repo's own AGENTS.md §5 calls out by name ("the same behavior claim [...] often repeats across README.md, --help doc comments, printed CLI output [...] each drifts independently"). The line a user actually sees at the terminal (runtime_install_finalize.rs) says "with", so README/cli.rs are the more likely candidates to reword.
    Confidence 75 · mechanical · Fix: pick one phrasing and match it across all four surfaces.

  • install_sdk_for_update's own tarball+--python-version refusal is untested and independently worded from its sibling in install_sdk — apps/rocm/src/therock.rs:217 vs. :313
    install_sdk bails with "--python-version applies only to wheel installs; a tarball install uses no Python" (tested by python_version_is_refused_for_a_tarball_install); install_sdk_for_update's tarball arm bails with the differently-worded "a tarball install uses no Python, so it has no --python-version" and has no dedicated test anywhere in the files reviewed. In practice this path is currently unreachable from rocm update (recorded_python_version only returns Some for a wheel-format manifest, so the only caller never passes a non-None value here), so this is a low-risk gap, not a live defect.
    Confidence 70 · mechanical · Fix: factor the two bails into one shared message/helper, and add a direct test for the install_sdk_for_update tarball arm for symmetry with its sibling.

  • sdk_install_request's own test suite doesn't cover python_version forwarding, despite the function's doc comment warning about exactly this risk for a sibling field — apps/rocm/src/install_cmd.rs:63-68 (doc comment), tests at :299-333, :338-367
    The doc comment explains include_devel has a dedicated forwarding test because "if the flag stopped being forwarded here the install would silently go back to pulling the compiler toolchain and every other assertion would still pass." python_version is the same shape of plain pass-through field and gets no equivalent assertion — both tests pass a literal None and never read it back. A dropped or mis-positioned argument here would not be caught by this file's own tests (though runtime-29/runtime-30's e2e scenarios would likely catch it through the full CLI).
    Confidence 70 · logic · Fix: add a python_version-forwarding test mirroring install_sdk_request_forwards_the_parsed_devel_flag.

Decisions for the author

  • A redundant --python-version 3.12 on a ROCm 7 (Canonical) install is accepted as a silent no-op, rather than refused outright regardless of value — tradeoff
    python_requirement_for_version's Canonical arm accepts only an exact match to the layout's own default and silently treats it as a no-op; anything else is refused. Accepting the redundant value is friendlier to scripts that always pass --python-version; refusing it unconditionally on Canonical would make "this flag only ever does something on ROCm 10+" more explicit. Both are defensible — worth a deliberate confirmation.

  • A non-default Python on a vLLM-preferred GPU family skips the vLLM auto-install with a printed note, rather than failing the SDK install outright — tradeoff
    The SDK/PyTorch install succeeds either way; only the follow-on vLLM engine auto-install is silently skipped (with an explanatory note naming the fix). This matches the documented UX in README/docs/vllm.md, but is worth the author's explicit confirmation that a partially-capable install succeeding by default (rather than erroring until the user acknowledges the vLLM tradeoff) is the intended shape, since it's a new way for rocm install sdk to "succeed" without doing everything a vLLM-preferred host normally gets.

Positive signals

  • NEXT_SDK_PYTHONS's cp314 entry reuses the NEXT_MANAGED_PYTHON_VERSION constant rather than repeating the literal, so the default and the ROCm-10 Python matrix cannot silently drift apart — and a test (a_recorded_wheel_tag_maps_back_to_the_requirement_it_came_from) pins exactly that.
  • python_layout_of's correction for the "nightly resolved to ROCm 10 but recorded as canonical" quirk is deliberately scoped to Python-interpreter selection only (recorded_python_requirement/recorded_python_version), leaving the separate package/catalog resolution in resolve_latest_for_manifest on the layout the real index listing was actually found at. This correctly distinguishes two different ways a runtime ends up on ROCm 10 (an explicit version pin routed through the dedicated "Next" catalog, vs. the canonical/nightly feed's own "latest" rotating past major 10 with no Next routing involved) — it would have been an easy, plausible-looking mistake to apply this correction to catalog resolution too, and it wasn't.
  • The exact "nightly resolved to ROCm 10 but manifest records canonical layout" scenario gets its own dedicated test (nightly_rocm_10_runtime_recorded_as_canonical_keeps_its_chosen_python) rather than being left to the common-case tests, and both the engine-skip and update-carryover consumers were traced and confirmed to reach it correctly.

Deployment notes

  • CI's two "Security scan" check failures on this head (Security scan / Security scanners, Security scan / gitleaks) are pre-existing false positives, not caused by this diff: the gitleaks report's two hits (apps/rocm/src/serve_summary.rs:557,585, a runtime_install_finalize.rs test fixture) both blame commit 7da776cb, which predates this PR and is already on main. A gitleaks allowlist fix for exactly this false-positive class (generic-api-key on api_model/runtime_key test fixtures) merged to main as #1000 after this branch diverged; rebasing should clear both. Separately, git merge-tree against current main is clean (no textual conflicts), so the PR's blocked mergeable state reflects required checks/reviews rather than a merge conflict.
  • A real ROCm 10 install and update on a non-default Python remain uncovered by any automated test (both need a real ROCm 10 download); the PR body discloses this itself.

What this covered

Full review of the whole change, base d027ff8e3ad9220ebb622667c97ed227d4b6c3e1 … head 43d67c3daf6fe9a118a5cb2a4a6a587925a26d66 (14 files, +633/-32), fanned out across 4 independent workers grouped by coupling (CLI/clap + core install-resolution: cli.rs, install_cmd.rs, therock.rs, therock/python.rs; update + engine-finalize: runtime_install_finalize.rs, therock/update.rs, update_cmd.rs; e2e tests: runtime_setup.feature, therock_steps.rs; docs: README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, docs/vllm.md), each reading its files completely plus surrounding/sibling code, blind to the PR description, to this PR's own review threads, and to each other's conclusions. I additionally ran: agent-instruction adherence against this repo's root AGENTS.md; impact analysis (the docs/architecture.md gap was found this way, by grepping cross-file consumers of every new pub(crate) item); a live CI check via get_check_runs/job logs (confirmed, not assumed, that the two failing checks pre-date and are unrelated to this diff, via commit blame); a merge-conflict check (git merge-tree, clean); and Step 9 (fetched issue #956 by its identifier — title/body/state only, not comments).

Did not run: cargo test/cargo clippy/the e2e cucumber suite locally (no Rust toolchain in this environment) — relied on GitHub's own CI results for this exact head commit instead, which show build-and-test, E2E, clippy, and coverage all green. "History" and "prior changes to these files" ran as light targeted git log/blame checks rather than a dedicated full pass, since every touched function is new to this PR's own commits (no pre-existing pattern to reconcile against). Not reconciled against this PR's existing review threads — this review was produced independently and blind to their content, per method; it was confirmed beforehand that no review yet exists at this exact head commit, so this is not a duplicate.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnarounds. The earlier blockers are resolved in the code at this head: the vLLM skip decides from the installed runtime's recorded Python, and an unpinned nightly that resolved to ROCm 10 but is recorded as canonical now counts as the ROCm 10 layout for both the skip and the rocm update carry-over (python_layout_of in therock/update.rs, with nightly_rocm_10_runtime_recorded_as_canonical_keeps_its_chosen_python). The README synopsis, help example, tarball refusal and the runtime-29 / runtime-30 comments are also fixed. This was a static review: nothing was built or run (no Rust toolchain here). The red checks are Security scan / gitleaks and its roll-up; they flag test fixtures that came in with main (serve_summary.rs, runtime_install_finalize.rs), not this diff, so I did not count them.

Non-blocking:

  1. docs/architecture.md:118 and :126: the therock/python.rs and therock/update.rs entries no longer match the code. recorded_python_version is now pub(crate), re-exported at therock.rs:61 and read by update_cmd.rs and runtime_install_finalize.rs; python_requirement_for_version and NEXT_MANAGED_PYTHON_VERSION are newly visible beyond the file; python_layout_of is not mentioned. Also docs/testing.md (around line 904) lists the update tests but not nightly_rocm_10_runtime_recorded_as_canonical_keeps_its_chosen_python.
  2. README.md:566-570: the unpinned-nightly parenthetical is followed by "These versions select their device payload by exact GPU arch ... pass --family with the raw gfx code", and later "A family label ... is rejected for those versions". For an unpinned nightly that resolved to ROCm 10 (canonical layout, select_source_layout) that does not apply. Scope that guidance to --version 10.x pins, and re-wrap the long line.
  3. "An update keeps the Python the runtime was installed with" (README "Which Python", docs/manual-testing.md:205-209) holds only while the recorded interpreter is still readable; otherwise the update falls back to 3.14 silently (the known, deferred follow-up). "while that interpreter still exists" would keep it honest.
  4. Closes #956: the issue asks for the default to be 3.12 and names the dashboard path, whose argv (onboarding.rs build_install_args) has no way to pass --python-version; the ROCm 10 default stays 3.14. Refs #956, or a note on the issue saying what is and is not covered, may fit better.
  5. README.md:256-257 and 463-464 ("Without --version, ... currently installs ROCm 7.14") are true for the default release channel but unqualified next to the new text saying an unpinned nightly gets ROCm 10; "on the release channel" would reconcile them. I did not treat this as blocking: the statement is accurate as written for the default invocation and the new section qualifies it.

🤖 by agenthub on AMD AgentHub

@juhovainio
juhovainio added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 1530f6b Nov 10, 2026
38 of 40 checks passed
@juhovainio
juhovainio deleted the triage-956-document-managed-python-choice branch November 10, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue]: Install defaults to python 3.14 with no override

3 participants