Skip to content

fix(therock): never delete an install folder to rebuild its venv (ROCMAI-467) - #568

Open
rominf wants to merge 17 commits into
mainfrom
fix/uv-venv-repair-ownership
Open

rominf wants to merge 17 commits into
mainfrom
fix/uv-venv-repair-ownership

Conversation

@rominf

@rominf rominf commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #533.

When the Python in an install sdk folder no longer ran, or no longer matched the required interpreter, the wheel install rebuilt the environment by deleting the whole folder. With --prefix, that folder is whatever the user named, so a home directory with a failing ~/bin/python was deleted, and only then did the rebuild fail. Even a folder ROCm CLI created holds more than the environment: ComfyUI keeps the user's models, workflows and outputs under apps/comfyui.

Now:

  • Reuse only a real venv. An existing environment is reused only when it has pyvenv.cfg, its Python runs, reports the required wheel tag, and reports the folder as its own sys.prefix. Without the last check, a venv whose pyvenv.cfg was lost leaves bin/python behaving as the base interpreter, and the package install would go into the base Python (for example a pyenv or conda base).

  • A missing or empty folder gets a new venv.

  • Any other non-empty folder goes through one check, whether or not its Python is still there. Without it, a partly cleared venv handed to a plain uv venv is deleted whole by uv 0.7, built over with stale packages by 0.9, and refused for good by 0.10.

  • That check refuses:

    • a protected system location, judged both as written and by where it resolves;
    • a folder with no record of a finished ROCm CLI install. Proof is either the in-tree .rocm-cli-runtime.json from a wheel install naming this folder, or a folder inside runtimes/wheel in ROCm CLI's data folder (which covers an install that failed before writing its manifest).

    The refusal names the folder and what was found, including which reuse condition failed (no pyvenv.cfg, a Python that does not run, a foreign sys.prefix, a wheel-tag mismatch). It changes nothing, and says to choose an empty folder or one where a ROCm CLI install finished.

  • For an owned folder, only the venv's own entries are removed, in this order: bin/Scripts, lib/lib64/Lib, include/Include, CACHEDIR.TAG, and pyvenv.cfg last. A clear that stops part-way therefore never leaves something that looks like a working venv. Links are unlinked, not followed. An entry that cannot be inspected (anything but "not found") stops the clear with an error rather than counting as removed. It then runs uv venv --allow-existing and prints what it kept, read from the folder after clearing, and why the old venv could not be used. The next run finishes a clear that stopped part-way.

  • uv venv always runs with UV_VENV_CLEAR=0, and never without --allow-existing over a non-empty folder.

  • --dry-run runs nothing from the install folder. install_sdk_dry_run is a model-callable tool that runs without approval and with a model-chosen --prefix, so a preview must not execute a bin/python found there. The preview decides the venv step from the folder's files:

    • where the files decide it, it prints one outcome: a new venv, a rebuild in place (--allow-existing, after removing the venv's entries, which the command line does not show), or a refusal (command: none and a refusal: line with the install's message);
    • for a folder holding both a Python and pyvenv.cfg, reuse depends on running that Python, so the preview says it cannot tell and prints command: for reuse plus command_if_rebuilt: or refusal_if_not_reused: for the other outcome.

    The interpreter search a preview runs also stops, without running it, when a Python it would check (ROCM_CLI_PYTHON, including a bare name resolved through PATH, the first on PATH, or the saved managed Python) is inside --prefix. The python -m venv support probe runs in a fresh folder of its own, so a venv.py in a working directory that is also the prefix (--prefix .) cannot stand in for the standard library's; an interpreter given as a relative path (ROCM_CLI_PYTHON=.venv/bin/python) is anchored to the current directory first, so it is still found from there. A previewed uv venv command is followed by an env: line naming the UV_VENV_CLEAR=0 the install sets, rather than a VAR=value prefix only POSIX shells accept.

    With ROCM_CLI_DISABLE_TORCH_ALIGNMENT set (the torch-alignment opt-out main added to install sdk), a reused venv keeps the torch it holds, and only that venv's Python can say which. The preview no longer asks it: its commands install the full stack, as the install does for a venv it creates or rebuilds, and where reuse cannot be decided from the files a torch_kept: line says it cannot tell which torch would be kept. A real install still decides this as before.

  • Text read from the folder is escaped. A marker's recorded path or format, a Python's sys.prefix and the names of kept entries are printed with terminal control, format and line-separator characters shown as \u{..} escapes, so a hostile folder cannot drive the terminal or the assistant's transcript through a refusal or rebuild line.

Tradeoff: a wheel install's marker counts as ownership even if the venv itself is gone, so in that folder anything the user later put under bin, lib or include is cleared. Requiring pyvenv.cfg instead would make a part-way clear unrecoverable.

Tests

  • Unit tests. The folder decision (empty, refused with the condition named, owned by the marker) reads only files and is tested on every platform, the Windows job included (the_venv_folder_decision_reads_files_on_every_platform). The rest run on Unix, because their interpreter and uv stand-ins are shell scripts. Each guard was mutation-checked (the test goes red with the guard removed). They cover:

    • refusal of an unowned folder, with the tree left byte-identical, and the refusal's advice getting past it;
    • each reuse condition named in the refusal when it is the one that failed, including a Python that does not report its sys.prefix or its wheel tag;
    • an owned rebuild keeping ComfyUI data, asserted together with the exact printed line;
    • every point a clear can stop at, converging on the next run (each stop asserting whether it converges by reuse or by a rebuild), and a clear that cannot inspect an entry stopping before uv runs;
    • a Python without pyvenv.cfg, or with a foreign sys.prefix, never being reused;
    • markers naming another folder, tarball markers, a manifest-less managed venv, a linked lib64, and protected paths by spelling and by resolution;
    • the order a clear removes entries in;
    • the dry-run lines for every folder state, with a planted Python in each that records every run and must record none (a_dry_run_reads_the_install_folder_and_runs_nothing_from_it), and the preview's interpreter search refusing a PATH, absolute ROCM_CLI_PYTHON or bare ROCM_CLI_PYTHON interpreter inside --prefix without running it;
    • the venv-support probe starting in its own folder (the_venv_probe_runs_in_a_folder_of_its_own), accepting a relative interpreter (the_venv_probe_accepts_an_interpreter_given_as_a_relative_path; the anchoring itself is a_relative_interpreter_path_is_anchored_to_the_current_directory, which runs on every platform), and marker text with control characters printed escaped;
    • which preview plans print the opt-out's torch_kept: line, one row per folder shape with and without the opt-out, and that every previewed command carries the full torch stack (a_preview_under_the_alignment_opt_out_cannot_tell_whether_torch_is_kept);
    • a literal 56-row table of folder shapes, with expected outcomes and user paths written out, including folders holding only a README, .DS_Store or .git.
  • An opt-in test (ROCM_CLI_TEST_UV_BINARIES, ROCM_CLI_TEST_PYTHON) runs every clear stop point against real uv 0.7.19, 0.9.30 and 0.10.0, and passes on all three.

  • Gherkin scenarios run the real CLI with no GPU or network:

    • @id:runtime-install-sdk-refuses-unowned-prefix (runtime-33): the dry run prints command: none and the refusal without running the folder's Python, the install refuses, and the advised empty folder works.
    • @id:runtime-install-sdk-rebuild-keeps-comfyui-data (runtime-34): the preview says it cannot tell reuse from a rebuild, runs nothing, and shows the rebuild command with its env: line; the install rebuilds and names what it kept and why.
    • @id:runtime-install-preview-runs-nothing-from-prefix (runtime-35): a preview of a folder holding someone else's venv says what it cannot tell, gives the refusal for the other outcome, and runs the planted Python zero times.
    • @id:runtime-install-preview-stops-at-prefix-python (runtime-36): with ROCM_CLI_PYTHON naming the folder's own Python, the preview stops, says why, and runs it zero times.
    • @id:runtime-install-preview-opt-out-runs-nothing-from-prefix (runtime-37): with ROCM_CLI_DISABLE_TORCH_ALIGNMENT set over someone else's venv, the preview says it cannot tell which torch would be kept, shows the full stack, and runs the planted Python zero times.
    • @id:runtime-install-preview-relative-python (runtime-38): a ROCM_CLI_PYTHON relative to the folder the CLI runs in is accepted.

    They use a loopback index, a ROCm 10 pin with an exact arch, and a fake uv that, like uv 0.10, refuses a non-empty folder without --allow-existing and wipes it on --clear. They are @requires-os:linux because the fakes are shell scripts, so Windows has no e2e coverage of this path; the guard code is platform-neutral and built by the Windows job.

Notes

  • Home itself is not on the protected-location list on main, so a home folder is refused by the ownership check. fix(uninstall): refuse system folders and homes, require ROCm CLI's marker elsewhere, and warn when a shared cache will go (ROCMAI-465) #569 extends that list.
  • comfyui install --reinstall's handling of apps/comfyui/source was fixed separately in fix(comfyui): reinstall replaces only ComfyUI's code, never the user's files #546 (merged).
  • With no --prefix, the install folder is ROCm CLI's own per-version folder under its data folder, and the preview's interpreter search does not check PATH against it (the folder is not known until the version is resolved, after the interpreter is chosen). The venv step never runs anything from it.
  • A preview that runs nothing from the folder cannot tell, for a folder holding both a Python and pyvenv.cfg, whether the venv would be reused; that includes a same-version reinstall into ROCm CLI's own managed folder. It says so rather than guess. For the same reason a --prefix that contains the first Python the search would check (for example --prefix ~ with a Python under ~/.local/bin first on PATH) stops the preview with that explanation.
  • The resolved preview's tail now prints python_env: (and an unresolved ROCm 10 preview prints it as not resolved), so the two still print the same keys.
  • command: none; the install will refuse this folder is now command: none, with the reason on the refusal: line.
  • README, the --prefix and --dry-run help, docs/testing.md, docs/manual-testing.md, docs/llm-tool-use.md, docs/vllm.md and docs/architecture.md are updated.

@rominf rominf changed the title fix(therock): never delete an install folder to rebuild its venv fix(therock): never delete an install folder to rebuild its venv (ROCMAI-467) Oct 26, 2026
@rominf
rominf force-pushed the fix/uv-venv-repair-ownership branch from b731d0b to afe2fa3 Compare November 3, 2026 10:31
@rominf
rominf force-pushed the fix/uv-venv-repair-ownership branch from afe2fa3 to 6890592 Compare November 5, 2026 14:31
@rominf
rominf marked this pull request as ready for review November 6, 2026 08:12
@rominf
rominf requested a review from a team as a code owner November 6, 2026 08:12
@rominf
rominf requested a review from jussielo-amd November 6, 2026 08:12

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the careful work here: refusing to delete a folder that isn't provably ours, and clearing only the venv's own entries in a crash-safe order, is the right fix. I reviewed the code only and did not run the tests.

Two blockers (inline):

  1. Windows test build is red: two test-only imports are unused on Windows under -D warnings.
  2. --dry-run now executes the prefix's bin/python, and install_sdk_dry_run is a model-callable, no-approval tool with a model-chosen prefix, so a preview can run an untrusted binary.

Non-blocking, in short (some inline):

  • The refusal text blames "does not run or does not match" even when the cause is a missing pyvenv.cfg or foreign sys.prefix.
  • The dry-run command: line omits the venv-file clearing and the UV_VENV_CLEAR=0 pin, so pasting it doesn't reproduce the install; the refusal text is also repeated after "the install will refuse this folder:".
  • docs/manual-testing.md still says --dry-run "never prompts or refuses" right above the new bullet saying it predicts a refusal; the README's "managed runtimes folder" is narrower in code (<data>/runtimes/wheel).
  • The branch also conflicts with main in apps/rocm/src/main.rs (the --prefix doc comment), so it needs a rebase.
  • The lychee failure looks like a download problem on the runner, not this PR.

🤖 by agenthub on AMD AgentHub

Comment thread apps/rocm/src/therock/python.rs Outdated
Comment on lines +959 to +1601

#[cfg(test)]
mod tests {
use super::super::managed_runtime_root;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker (red CI): these two imports (managed_runtime_root, PROGRESS_LINE_SINK) are only used by #[cfg(unix)] tests, so on Windows they are unused and -D warnings fails the test build ("Windows build and test (debug)" / windows-build-and-test). Gating the two use lines with #[cfg(unix)] (or importing them inside the unix tests) should fix it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 13338cf: both imports are now #[cfg(unix)], since only the Unix tests use them (the tests that run everywhere import the sink in their own bodies). I couldn't build for Windows locally, so the Windows CI job on the new head is the check.

Comment thread apps/rocm/src/therock/python.rs Outdated
let env_python = venv_python_path(install_root);
env_python.is_file()
&& install_root.join("pyvenv.cfg").is_file()
&& run_command(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker (security): --dry-run now executes <prefix>/bin/python (--version, the sys.prefix probe and the wheel-tag probe) to predict reuse. install_sdk_dry_run is a model-callable tool treated as read-only, so it runs without approval, and its --prefix is model-chosen (only system folders are rejected in internal_mcp_install_sdk_args). A folder containing a hostile bin/python (e.g. a downloaded/cloned tree) would therefore be run with the user's rights from a "preview". The real install is approval-gated, the preview isn't. Suggest keeping the dry run static (check that the Python, pyvenv.cfg and the marker exist and say "would reuse if it runs"), or skipping the exec on the model-callable path.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2940fe4 and fadfb2b. A dry run now runs nothing from the install folder:

  • The venv step is decided from the folder's files only (dry_run_venv_plan). Where the files settle it (empty folder, no Python, no pyvenv.cfg), the preview prints one outcome. For a folder with both a Python and pyvenv.cfg, reuse depends on running that Python, so the preview says it cannot tell. It then prints command: for reuse and command_if_rebuilt: or refusal_if_not_reused: for the other outcome.
  • The interpreter search a preview runs stops, without running it, when the Python it would check first is inside --prefix. That covers PATH, an absolute ROCM_CLI_PYTHON, and a bare one resolved through PATH.
  • The python -m venv support probe now starts in a fresh folder of its own. -m puts the working directory first on sys.path, so with --prefix . a venv.py there could otherwise have run.

Tests plant a Python that records every run and assert the record stays empty:

  • a_dry_run_reads_the_install_folder_and_runs_nothing_from_it (every folder state)
  • a_dry_run_runs_no_python_from_the_install_folder_while_searching
  • the_venv_probe_runs_in_a_folder_of_its_own
  • scenarios runtime-31 (someone else's venv) and runtime-32 (ROCM_CLI_PYTHON inside the folder)
  • the previews in runtime-29 and runtime-30

Each guard was removed once to confirm its test goes red. docs/llm-tool-use.md now states the rule for read-only tools.

Comment thread apps/rocm/src/therock/python.rs Outdated
let python = venv_python_path(install_root);
let mut found = vec![if python.is_file() {
format!(
"a Python at {} that does not run or does not match",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (UX): whenever bin/python exists the refusal says it "does not run or does not match". But a Python that runs fine and simply has no pyvenv.cfg, or reports a different sys.prefix (e.g. --prefix pointing at a conda env), is refused too, and this wording is then misleading. Naming the failed condition ("no pyvenv.cfg", "its sys.prefix is X", "version mismatch") would let the user act on it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2940fe4. The reuse check now returns the condition that failed: no pyvenv.cfg, a Python that does not run, whose sys.prefix is X, not this folder, or a wheel-tag mismatch. The refusal and the rebuild line both name it. a_refusal_names_the_condition_that_kept_the_venv_from_being_reused covers each case and asserts that "does not match" is gone.

Comment thread apps/rocm/src/therock/python.rs Outdated
/// Remove one venv entry without following a link out of the folder: `lib64`
/// is a symlink to `lib` in a uv venv, and a link must go, not its target.
fn remove_venv_entry(path: &Path) -> Result<()> {
let Ok(metadata) = fs::symlink_metadata(path) else {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: symlink_metadata failures of any kind (not just NotFound) return Ok(()) here, so a permission or I/O error is reported as a successful clear and uv venv --allow-existing then runs over a stale entry. Only swallowing ErrorKind::NotFound would keep the "never leaves a half-cleared venv" promise honest.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2940fe4. Only NotFound counts as already removed. Any other symlink_metadata error stops the clear with failed to inspect <path> before uv venv --allow-existing runs. a_clear_that_cannot_inspect_a_venv_entry_stops_before_uv covers it with a folder that can be listed but not searched; it skips itself where permissions are not enforced (root).

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 6890592

Outcome: needs work. The core approach (refuse unless provably owned; clear only the venv's own entries in a crash-safe order) is sound and mostly well-tested, but one new security-relevant regression, several confirmed correctness/test-quality defects, and a red Windows CI job should be fixed first. Implements the repro in #533 (a --prefix folder deleted by a failing interpreter); one thing the issue's own "expected behavior" asks for (the protected-location check always running) is only partially wired — see finding 2.

Blocking

  1. --dry-run now executes whatever binary sits at an attacker/model-chosen --prefix, with no approval gate. dry_run_venv_lines (apps/rocm/src/therock/python.rs:464) calls existing_venv_is_reusable, which runs <prefix>/bin/python --version and -c "import sys; print(sys.prefix)" (python.rs:154-172) purely to predict the preview text. Before this PR, the dry-run path never touched the prefix at all (confirmed against the base commit: it only built uv venv args). install_sdk_dry_run is registered as a no-approval, model-callable chat tool (apps/rocm/src/tool_policy.rs:191,993) whose --prefix the model supplies (therock.rs:545 and wheel_plan.rs:231 both already comment on the prefix being model-chosen, but only for the shell-quoting concern, not for this). Net effect: a "preview that changes nothing" now runs an arbitrary program at a path nothing has approved. Fix: don't execute the existing interpreter from the dry-run path, or gate dry-run on the same approval install_sdk itself requires.

  2. Protected-location refusal is skipped for a missing/empty --prefix folder — python.rs:290 (venv_folder_state) returns Empty and proceeds straight to uv venv whenever the folder has no entries, before refuse_protected_venv_rebuild (line 334) ever runs. A --prefix naming a not-yet-created directory under a protected root bypasses the protection entirely, contradicting both the README's "...or one in a protected system location, is refused" and #533's own ask to always run this check. Fix: check protection regardless of the emptiness branch.

  3. Refusal text blames the wrong cause — python.rs:391 (unowned_venv_rebuild_refusal) reports "a Python at … that does not run or does not match" whenever bin/python exists, even when the real cause is a missing pyvenv.cfg or a foreign sys.prefix (not run failures). Misdirects the user toward an interpreter problem. Fix: name the actual cause.

  4. A "clear stop" test case doesn't test what it claims — ClearStop::PartOfBinKeepingPython (python.rs:2109/2147, exercised by every_point_a_clear_can_stop_at_converges_on_the_next_run at :2241) only deletes bin/activate, which existing_venv_is_reusable never inspects, so it silently exercises plain reuse rather than partial-clear resumption and would stay green even if the clear/rebuild path were deleted. Fix: corrupt something the reuse check actually inspects, or drop the variant.

  5. New cross-platform venv-ownership logic has no Windows-reachable test — every new unit test is #[cfg(unix)]/cfg(target_os="linux"), and both new Gherkin scenarios are @requires-os:linux, yet the logic is platform-agnostic (VENV_OWNED_ENTRIES lists Scripts/Lib/Include; the protected-location helper has a distinct Windows branch). The PR text discloses the e2e half but gives no reason for the unit-test half. Worth a reachable Windows test, or an explicit reasoned deferral, given the feature's whole purpose is preventing data loss.

  6. New e2e fake-executable helper reintroduces a documented ETXTBSY race — tests/e2e-cucumber/tests/e2e/therock_steps.rs:1683 (write_executable) writes the fake uv/Python via plain fs::write + set_permissions instead of rocm_core::fake_executable::write_fake_executable, the helper that exists specifically to avoid this race and that every other *_steps.rs file already uses. Both new scenarios run on the no-GPU lane at 64x concurrency, where the race is reachable. Fix: use the shared helper.

  7. Windows CI is red from this change — "Windows build and test (debug)" and windows-build-and-test fail to compile: unused imports super::super::managed_runtime_root and ...process::PROGRESS_LINE_SINK at python.rs:959-960, under -D warnings. Confirmed directly from the job log.

  8. Branch conflicts with main — a merge check shows real conflicts in apps/rocm/src/main.rs and README.md against current main (matches the PR's dirty mergeable state). Needs a rebase.

Non-blocking

  • README.md / docs/manual-testing.md say the refused-folder dry-run prints exactly command: none; the code appends "; the install will refuse this folder".
  • The e2e fake-uv script's --clear/wipe branch is unreachable (production always passes UV_VENV_CLEAR=0 without --clear) — harmless dead code in the test fixture only.

Decisions for the author

  • Tradeoff already called out in the PR body: an in-tree marker counts as ownership even if the venv itself is gone, so anything a user later placed under bin/lib/include there gets cleared too; requiring pyvenv.cfg instead would make a part-way clear unrecoverable. Reasonable either way — just confirm it's deliberate.

What this covered: the full diff against the merge-base, all 8 changed files read completely including surrounding code, via three independent passes (core logic, the new unit-test module, the e2e Gherkin/steps) plus direct tracing of the dry-run/tool-policy call chain and CI logs. The lychee failure and the self-hosted GPU-lane E2E failures (an unrelated serving scenario, and a WSL download timeout) look like pre-existing infra/runner issues unrelated to this diff and are not re-reported as findings. Not reconciled against this PR's existing review thread.

Not a human review. This is an automated code review; please verify findings independently before acting on them.

When the Python in an install folder failed `--version` or no longer
matched the required interpreter, `ensure_uv_venv` ran `remove_dir_all`
on the whole folder. With `install sdk --prefix` that folder is whatever
the user named, so a home directory with a failing `~/bin/python` was
deleted, and only then did the rebuild fail. Even a runtime ROCm CLI
created holds data that is not the venv: ComfyUI keeps the user's
models, workflows and outputs under `<install_root>/apps/comfyui`.

The rebuild now:

- refuses a protected system location, by the path as given and by
  where it resolves;
- refuses any folder ROCm CLI cannot show it created: it needs
  `pyvenv.cfg` and either the in-tree `.rocm-cli-runtime.json` naming
  this folder or a place inside the managed runtimes folder. The
  refusal names the folder and what was found, changes nothing, and
  says to choose an empty folder or one ROCm CLI created;
- removes only the venv's own entries (`bin`/`Scripts`, `lib`/`Lib`,
  `lib64`, `include`, `pyvenv.cfg`, `CACHEDIR.TAG`), unlinking rather
  than following links, then runs `uv venv --allow-existing`, and prints
  what it kept, computed from the folder after clearing;
- pins `UV_VENV_CLEAR=0` for `uv venv`, so a value exported in the
  user's shell cannot make uv clear the folder instead.

Fixes #533

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…hip check

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
… venv step

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…d folder

A clear that stopped right after pyvenv.cfg left bin/python, a symlink to
the base interpreter on Linux. It still ran and reported the right wheel
tag, so the next run judged it reusable and installed ROCm and torch into
the base Python. Reuse now also needs pyvenv.cfg and the interpreter's
sys.prefix to be the install folder, and the clear removes bin first and
pyvenv.cfg last again.

The dry run printed a uv venv command even for a folder the install would
refuse. It now takes the install's own decisions (reuse, fresh, rebuild,
refuse) and prints command: none for a refusal.

The refusal now says no finished ROCm CLI install was recorded, which is
also true of a folder whose install failed before writing its manifest.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The interpreter goes first and pyvenv.cfg last, so a clear that stops
part-way never leaves a Python that still looks like a venv. The reuse
check now refuses such a Python on its own, so no convergence test fails
when the order is reversed; pin the order directly.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
`managed_runtime_root` and `PROGRESS_LINE_SINK` are imported at the top of
the test module, but only `#[cfg(unix)]` tests use them; the tests that run
on every platform import the sink in their own bodies. On Windows both
imports were unused, which `-D warnings` turns into a failed test build.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
`install sdk --dry-run` ran `<prefix>/bin/python` (`--version`, its
`sys.prefix` and its wheel tag) to predict whether the venv there would be
reused. `install_sdk_dry_run` is a model-callable tool that runs without
approval and with a model-chosen `--prefix`, so a folder holding a hostile
`bin/python` would be executed by a preview.

The preview now decides the venv step from the folder's files only. Where
those decide it (no Python, no `pyvenv.cfg`, an empty folder) it prints one
outcome as before. Where only running the Python could tell, it says it
cannot tell, prints `command:` for reuse and `command_if_rebuilt:` or
`refusal_if_not_reused:` for the other outcome. The interpreter search a
preview runs stops, without running it, when the Python it would check
first (on PATH, or ROCM_CLI_PYTHON) is inside `--prefix`. The previewed
`uv venv` carries the `UV_VENV_CLEAR=0` the install pins, and a refusal is
printed once, on its own `refusal:` line.

Two smaller review findings in the same code:

- The refusal and the rebuild line said "does not run or does not match"
  whatever failed. They now name the condition: no `pyvenv.cfg`, a Python
  that does not run, a foreign `sys.prefix`, or a wheel-tag mismatch.
- Clearing a venv entry treated every `symlink_metadata` error as "already
  gone". Only `NotFound` does now; any other error stops the clear before
  `uv venv --allow-existing` runs over a stale entry.

Scenario runtime-31 plants a recording Python in a folder holding someone
else's venv and checks that the preview runs it zero times; runtime-29 and
runtime-30 now check the same of their previews. README, `--help`,
docs/testing.md, docs/manual-testing.md and docs/llm-tool-use.md say what
the preview does and does not run.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Follow-ups from an independent review of the dry-run change:

- The venv-support probe runs `python -m venv`, and `-m` puts the working
  directory first on `sys.path`. With `--dry-run --prefix .` that
  directory is the install folder, so a `venv.py` there would run in place
  of the standard library's. The probe now starts in its own fresh folder.
- A refusal or rebuild line prints text read from the folder (a marker's
  recorded path and format, a Python's `sys.prefix`, the kept entries).
  Control, format and line-separator characters in it are printed as
  `\u{..}` escapes, so a hostile folder cannot drive the terminal or the
  assistant's transcript.
- The managed wheel-runtimes folder is named through `managed_format_root`
  in therock.rs instead of spelling `"runtimes"` in python.rs, which also
  writes the managed-Python registry; main's registry-writer scan flags a
  file that spells both and writes.
- Scenario runtime-32 covers the preview stopping, without running it,
  when ROCM_CLI_PYTHON names a Python inside the install folder.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…Windows

Follow-ups from a second automated review:

- The `PartOfBinKeepingPython` clear stop leaves a whole venv, so it is
  reused rather than rebuilt, and the convergence test passed whichever
  way it went. Every stop now asserts which way it converges: that one is
  reused, every other one is rebuilt through uv.
- The folder decision (empty, refused with the condition named, owned by
  the wheel marker) reads only files. A new test runs it on every
  platform with that platform's venv layout, so the Windows job covers
  it; the tests that run an interpreter stay Unix-only.
- The e2e steps write their stand-in scripts through
  `rocm_core::fake_executable::write_fake_executable`, which avoids the
  ETXTBSY race a plain write and chmod can hit under the suite's
  concurrency.
- README and docs/manual-testing.md quote the refused folder's
  `command:` line as it is printed.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The fake uv the venv tests run wrote the stand-in Python with a bare
`cat`, while its other commands use `/bin/...`. A test elsewhere in the
crate empties PATH for a moment, and a fake uv started in that window
found no `cat`, created the Python as an empty file and left the next
exec failing with "Exec format error". `/bin/cat` takes PATH out of it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/uv-venv-repair-ownership branch from 6890592 to d80c8c3 Compare November 10, 2026 11:26
@rominf

rominf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main and pushed. This addresses the review body from @jussielo-amd and the automated review of 6890592. The four inline threads each have their own reply.

Review body (non-blocking items)

  • The dry-run uv venv commands now start with UV_VENV_CLEAR=0, as the install runs them. The text states that a rebuild first removes the venv's own entries, which the command line does not show.
  • A refusal is printed once, on its own refusal: line.
  • docs/manual-testing.md no longer says --dry-run "never prompts or refuses". It now says the preview never prompts or asks for the replace consent, and still reports the install's refusals. docs/testing.md had the same claim and is fixed too.
  • The README now says the managed folder is runtimes/wheel in ROCm CLI's data folder.
  • The conflicts are resolved. main moved the clap tree to cli.rs and the dry-run tail to therock/wheel_plan.rs, and the venv step is wired into that tail. The scenarios are renumbered to runtime-29 to runtime-32, because main now uses 23 to 28.

Automated review of 6890592

  1. Dry run executing the prefix's Python: fixed. See the reply on the security thread.
  2. Protected-location check for a missing or empty prefix: not changed. The check guards deletion, and an empty folder has nothing to delete; a missing folder is created by uv venv. Refusing creation under /opt (or /usr/local on another distro's list) would block a deliberate system-wide install that main allows, which is wider than install sdk: a failing bin/python makes the wheel install delete the whole --prefix, rocm-owned or not #533. The assistant path already filters system prefixes before the CLI is reached.
  3. Refusal wording: fixed in 2940fe41. It now names the condition that failed.
  4. PartOfBinKeepingPython: kept, now with an assertion. That stop leaves a whole venv, so it is reused. Every clear stop now asserts which way it converges: that one is reused, every other one is rebuilt through uv. If either path is broken, the test fails. (2ae9ebdf)
  5. Windows coverage: the folder decision reads only files, so the_venv_folder_decision_reads_files_on_every_platform now runs it on every platform with that platform's venv layout: empty, refused with the condition named, and owned by the marker. The tests that run an interpreter stay Unix-only because their stand-ins are shell scripts. (2ae9ebdf)
  6. ETXTBSY: the e2e steps now write their stand-ins through rocm_core::fake_executable::write_fake_executable. (2ae9ebdf)
  7. Windows build: fixed in 13338cf0.
  8. Conflicts: rebased.
  • command: none wording: README and docs/manual-testing.md now quote the line as printed.
  • The fake uv's --clear branch is still there on purpose. It is what turns a regression that passes --clear, or drops the UV_VENV_CLEAR=0 pin, into a wiped folder that the tests see.

Since the review, also

  • Text the folder supplies is printed with control and bidi characters escaped. That text is a marker's recorded path or format, a Python's sys.prefix, and the kept entry names. (fadfb2b9)
  • main's registry-writer scan flagged python.rs, which spells both "runtimes" and "registry" and writes a file. The managed wheel folder is now named through managed_format_root in therock.rs.
  • The fake uv calls /bin/cat. A bare cat failed whenever another test had emptied PATH at that moment. (d80c8c31)

@rominf
rominf requested a review from jussielo-amd November 10, 2026 11:27
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one regression. This was a static review of d80c8c3; I did not run the tests. The deletion fix itself looks right, and the earlier blockers are resolved when I read the code.

Earlier items:

  • The Windows test build is fixed: the two imports are now #[cfg(unix)], and the Windows jobs are green.
  • --dry-run runs nothing from the install folder any more. The reuse probe is called only from ensure_uv_venv, the preview decides from the folder's files, and the search stops without running a Python inside --prefix.
  • The refusal now names the condition that failed, the printed line carries UV_VENV_CLEAR=0 and a refusal: line, the docs no longer say the preview never refuses, and the branch merges cleanly with main.
  • E2E tests (Strix Halo, Windows, current) was still in progress and the MI300X and W7900 lanes were queued; please check them before merging.

Blocking:

  • The python -m venv probe now runs with current_dir set to a fresh folder, which breaks a relative interpreter. ROCM_CLI_PYTHON=.venv/bin/python passes the wheel-tag check from the current directory, but the probe then looks it up under the new directory on Unix and fails, so the install and preview say "ROCM_CLI_PYTHON is not usable" for a Python that works. main has no current_dir here. Making the program path absolute before spawning, and adding a relative-path test, would fix it (inline).

Non-blocking:

  • The pasteable command: starts with POSIX-only UV_VENV_CLEAR=0, and some docs still describe the old form (inline).
  • Unit-test stand-in scripts still use fs::write plus set_permissions, against the fake-executable rule in docs/testing.md (inline).
  • The preview refuses even for the user's own --prefix when the first PATH Python is inside it, for example an activated venv at the prefix, while the real install proceeds. This is documented, but it may surprise users.
  • Small doc drift: the "not resolved" key lists in the README, manual-testing.md and testing.md omit the new python_env: line. manual-testing.md says a rebuild needs a marker and leaves out the runtimes/wheel route. The "first Python" wording does not mention that the saved managed Python is also checked.
  • The protected-location check is skipped for a missing or empty folder. I read that as acceptable, since nothing is deleted and the description explains why, so I did not count it as blocking.

🤖 by agenthub on AMD AgentHub

program,
args.iter()
.map(String::as_str)
.collect::<Vec<_>>()
.as_slice(),
Some(&probe_root),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: starting the probe in probe_root breaks an interpreter given as a relative path. ROCM_CLI_PYTHON is passed through as written (python_launcher_install_ready(Path::new(value), ..) in the resolver, and relative PATH entries come through dir.join). wheel_compatibility_for_python runs it from the current directory and succeeds, but here the child changes directory first, so on Unix a program like .venv/bin/python is looked up under the new probe_root, the launch fails, and the user gets "ROCM_CLI_PYTHON is not usable for ROCm setup" for a Python that works. On main the probe has no current_dir, so this is a regression, and no test uses a relative interpreter. Suggest making program absolute (std::path::absolute) before current_dir is applied, plus a test with a relative ROCM_CLI_PYTHON.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 302697e. verify_python_can_create_venv now passes the interpreter through anchored_to_current_dir before the probe starts in its own folder: a relative path with a folder in it (.venv/bin/python) is made absolute against the current directory, and a bare name (python3) is left to PATH.

Covered by:

  • the_venv_probe_accepts_an_interpreter_given_as_a_relative_path (Unix). Its relative path starts with src/, which the test's working directory has and the probe's fresh folder does not, so it fails without the fix whatever TMPDIR is.
  • a_relative_interpreter_path_is_anchored_to_the_current_directory, with no platform gate, so it also runs on the Windows job.
  • Scenario runtime-38, which runs the CLI from a folder with ROCM_CLI_PYTHON=fake-python/python3.14.

All three were checked to go red with the anchoring removed.

Comment thread apps/rocm/src/therock/python.rs Outdated
let install = format!("uv {}", display(install_args));
let with_venv = |rebuilding: bool| {
format!(
"UV_VENV_CLEAR=0 uv {} && uv {}",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the pasteable command: now starts with UV_VENV_CLEAR=0 uv ..., which is POSIX-only (it fails in cmd and PowerShell), and command: none; the install will refuse this folder reads as two shell commands if pasted. The docs still say the line shows uv venv --python 3.12 ... and is "not copy-pasteable verbatim" (docs/manual-testing.md, docs/testing.md), so they'd need the same update. Consider printing the pin as a separate env: line, or noting it is POSIX syntax.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in 302697e. The command: lines no longer start with a VAR=value prefix. A previewed uv venv command is now followed by its own line:

env: UV_VENV_CLEAR=0 for uv venv, as the install sets it so that a UV_VENV_CLEAR=1 in the environment cannot make uv clear the folder; set it before running the command

A refused folder prints command: none, with the reason on the refusal: line. With the prefix gone, the uv venv --python 3.12 ... wording in docs/testing.md and docs/manual-testing.md is accurate again. The README and manual-testing.md describe the env: line.

Comment thread apps/rocm/src/therock/python.rs Outdated
fs::create_dir_all(python.parent().unwrap()).unwrap();
let _ = fs::remove_file(&python);
fs::write(&python, "#!/bin/sh\nexit 1\n").unwrap();
fs::set_permissions(&python, fs::Permissions::from_mode(0o755)).unwrap();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the e2e helper now uses write_fake_executable, but these unit-test stand-ins (here and around lines 3240, 3582, 4125, 4145) still do fs::write + set_permissions(0o755) and then run the script. docs/testing.md ("Tests that run a fake executable") says that pattern gives intermittent Text file busy on Linux. Switching to rocm_core::fake_executable::write_fake_executable would match it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 302697e. These stand-ins now go through rocm_core::fake_executable::write_fake_executable:

  • plant_failing_venv_python
  • write_logging_fake_uv
  • plant_working_venv_python
  • plant_recording_venv_python
  • the real-uv wrapper
  • the wrong-tag rewrite

The merge with main moved these tests from python.rs into therock/venv.rs and therock/test_support.rs, to keep python.rs under the module-size limit.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head d80c8c3

Review — needs work

Full review of the whole change (merge-base 0d5a0447…d80c8c3). Fixes #533 (an install sdk --prefix folder being deleted whole to rebuild its venv); this change implements the full fix (refuse-unless-owned, selective crash-safe clear, no-execution --dry-run) plus docs updates — no narrower scope claimed.

Blocking

  1. A relative ROCM_CLI_PYTHON interpreter is misreported as unusable, for both the real install and the preview. verify_python_can_create_venv (apps/rocm/src/therock/python.rs:1571-1586) now spawns the candidate interpreter with cwd set to a freshly-created scratch directory (run_command_in(program, args, Some(&probe_root), ...)), added specifically to stop python -m venv from importing a hostile venv.py out of an attacker-controlled --prefix. But when program is a relative path with a directory component — e.g. ROCM_CLI_PYTHON=.venv/bin/python (find_python_launcher_in, line 1397, uses Path::new(value) verbatim, never canonicalized) — Unix program-path resolution for a path containing / happens against the process's cwd at exec time, which is now the unrelated probe directory, not the user's original working directory. wheel_compatibility_for_python/capture_python_stdout (process.rs) never set cwd, so the earlier wheel-tag check on the same relative path succeeds — only this probe fails, with an ENOENT-style error, producing ROCM_CLI_PYTHON is not usable for ROCm setup: .venv/bin/python for an interpreter that works fine. Confirmed this is a genuine regression: the base commit's equivalent function called plain run_command (no cwd argument at all). Reproduced the underlying fork/chdir/exec mechanism directly (a relative program path fails to resolve once a different cwd is set on the child, verified with an equivalent subprocess.run repro). No test exercises a relative ROCM_CLI_PYTHON/PATH entry.
    Confidence 90 · logic · Fix: resolve program to an absolute path (e.g. against std::env::current_dir() if relative) before passing it into run_command_in with an overridden cwd, and add a relative-path regression test.

  2. New unit-test helpers in this PR hand-roll a fake executable instead of using the project's own race-safe helper. docs/testing.md:258-261 states explicitly: "Plant a stand-in python, uv, ... with rocm_core::fake_executable::write_fake_executable, not std::fs::write followed by set_permissions(0o755)," and explains the exact ETXTBSY race this avoids under cargo test's multi-threaded model. apps/rocm/src/therock/python.rs's new test module uses the safe helper in several places (e.g. lines ~1611, 1644, 2050, 3049) but four new helpers added by this PR — plant_failing_venv_python (~3153), write_logging_fake_uv (~3198-3239), plant_recording_venv_python (~4112), plant_working_venv_python (~4133) — all use the exact hand-rolled fs::write(...).unwrap(); fs::set_permissions(..., 0o755).unwrap(); pattern the doc names as the thing not to do, in the same file that otherwise follows the rule.
    Confidence 85 · mechanical · Fix: route these four helpers through write_fake_executable.

  3. Branch currently has real, unresolved conflicts with main. Verified directly (not taken on anyone's word): git merge --no-commit --no-ff origin/main from this PR's head produces content conflicts in apps/rocm/src/therock/python.rs (two independent test-module additions landed in the same spot), docs/architecture.md (the therock/python.rs entry was edited on both sides), and tests/e2e-cucumber/features/runtime_setup.feature. This matches the PR's own mergeable_state: dirty.
    Confidence 95 · architectural · Fix: rebase onto current main and resolve.

Non-blocking

  • venv_folder_state (python.rs:379-397) returns Empty and skips refuse_protected_venv_rebuild whenever the --prefix folder has no entries (including a not-yet-created path). README.md:544 reads ("...or one in a protected system location, is refused") as applying regardless of emptiness, but a --prefix naming an empty/missing subpath under a protected root (e.g. an unused directory under /usr) gets a brand-new venv created there without refusal. No data is destroyed (nothing pre-existing to lose), so the practical risk is narrow, but it's a real gap against the stated contract.
  • Two existing_venv_reuse failure branches — "a Python at … that does not report its sys.prefix" and "...whose wheel tag could not be read" (python.rs:208, 222) — have no test that reaches them; grepping the entire test module for either exact string returns nothing, so deleting those specific .map_err wrappers wouldn't fail any test.
  • docs/testing.md:845 and docs/manual-testing.md:180 each list the NeedsUv/unresolved-preview fields as target, runtime_key, package_specs, command and manifest — but this PR's own new python_env: line is also printed as not resolved by this preview in that exact branch (wheel_plan.rs:252), and isn't in either list.
  • docs/manual-testing.md:201 says a venv is rebuilt "if ... its venv does not run" — but the very next sentence's own two examples contradict that: "no pyvenv.cfg beside it" never runs anything, and "whose sys.prefix is ..., not this folder" is reported after the Python ran successfully. README's parallel section doesn't make this mistake.
  • The new fake-uv test fixture's --clear/UV_VENV_CLEAR=1 handling in write_logging_fake_uv is unreachable in practice: production always runs with UV_VENV_CLEAR=0 and no --clear. Harmless, test-fixture-only dead code.
  • One new Gherkin scenario's "runs the planted Python zero times" assertion is checked via absence-of-a-side-effect/presence-of-a-file proxies rather than a direct invocation-count assertion against the fake's own log — weaker than it could be, not a demonstrated bug.

Decisions for the author

  • A .rocm-cli-runtime.json marker counts as ownership even if the venv itself is gone — tradeoff, already named in the PR body: "a wheel install's marker counts as ownership even if the venv itself is gone, so in that folder anything the user later put under bin, lib or include is cleared. Requiring pyvenv.cfg instead would make a part-way clear unrecoverable." Confirm that's still the intended tradeoff.
  • The clear/rebuild unit tests (the 56-row table, mutation-checked guards, etc.) are entirely #[cfg(unix)], even though VENV_OWNED_ENTRIES and refuse_protected_venv_rebuild are genuinely cross-platform (Windows has its own Scripts/Lib/Include entries and its own protected-path branch) — non-blocking-improvement. The PR text explains why ("their interpreter and uv stand-ins are shell scripts"), which satisfies this repo's own disclosure convention, so this isn't a rule violation. Given the feature's whole purpose is preventing data loss, a Windows-reachable version (e.g. a trivial compiled stub or a .cmd/PowerShell stand-in) would still be worth it.

Positive signals

  • The originally-reported dry-run security issue — --dry-run executing whatever binary sits at a model-chosen --prefix with no approval gate — is fixed cleanly here, and I traced it end-to-end rather than taking that on faith: dry_run_venv_plan and everything it calls (venv_shape_finding, venv_folder_state) read only filesystem metadata and spawn nothing; separately, LookupPurpose::refuse_preview_python_in_prefix (python.rs:1307) explicitly bails before running any PATH/ROCM_CLI_PYTHON candidate that resolves to inside the preview's --prefix.
  • shown_from_folder's control-character escaping is scoped correctly: it's applied exactly to text actually read from the (possibly hostile) folder's contents — a venv's reported sys.prefix, a marker's recorded path/format, kept-entry names — while leaving CLI-supplied path text alone, which is reasonable since that's independently validated earlier by the tool-argument layer (tool_policy.rs/freeform_planner.rs both refuse control characters in a tool call's arguments before this code ever runs).

Deployment notes

None beyond Blocking #3 (rebase required before this can merge).

What this covered

Full diff read, all 15 changed files completely (apps/rocm/src/therock/python.rs — both production code and its ~3000-line new test module, process.rs, wheel_plan.rs, consent.rs, cli.rs, therock.rs, therock/resolve/wheel.rs, therock/update.rs, tests/e2e-cucumber/features/runtime_setup.feature, tests/e2e-cucumber/tests/e2e/therock_steps.rs, and README.md/docs/architecture.md/docs/llm-tool-use.md/docs/manual-testing.md/docs/testing.md), via four independent fan-out passes (core production logic, the unit-test module, the e2e scenarios, and docs-vs-code consistency) plus my own direct tracing of the dry-run call chain and the cwd/relative-path regression (including reproducing the underlying OS path-resolution behavior and diffing against the pre-PR implementation), and a live git merge --no-commit against current origin/main to verify mergeability rather than trust the PR's reported state.

Did not run: the workspace's actual test suite (no Rust toolchain available in this environment) — the regression and test-quality findings above are established by direct code/doc tracing, not by execution. A rough manual estimate suggested python.rs's production-line count (excluding test-gated code) may be close to or over the repo's 1,500-line module-size cap, but the margin of error in a manual count is too wide to report with confidence, and that check is explicitly advisory-only in this repo today, so it's omitted rather than guessed at. CI: all 45 completed checks are green; one self-hosted hardware lane, "E2E tests (W7900, current)", has been stuck queued for over 24 hours — looks like a runner-availability gap, not something this diff caused. Not reconciled against this PR's existing review thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

Resolve conflicts with main and port this branch's behaviour onto it:

- wheel_plan.rs: the preview's venv step (dry_run_venv_plan) now also
  hands uv the kept-torch install specs main added; main's new
  kept-torch test passes the paths the tail now needs.
- runtime_setup.feature: main took runtime-29..32, so this branch's
  scenarios become runtime-33..36; docs/testing.md and the step-file
  comment follow.
- docs/architecture.md: main's per-crate tables win; the venv
  description moves into them.
- docs/manual-testing.md: keep this branch's dry-run wording (it can
  refuse) beside main's new source-provenance bullet.

python.rs went over the now-blocking 1,500 production-line limit, so
the venv-folder decision (ensure_uv_venv, venv_folder_state,
dry_run_venv_plan and helpers) moves, unchanged, into therock/venv.rs
with its tests; the shared test fakes move to therock/test_support.rs.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
… a relative Python

Answers the review of d80c8c3 and adapts this branch to what main added
since (the torch-alignment opt-out in the SDK install).

- With ROCM_CLI_DISABLE_TORCH_ALIGNMENT set, a dry run asked the install
  folder's own Python which torch it held, so a preview ran a bin/python
  from a model-chosen --prefix. A preview now runs nothing from the
  folder: its commands install the full stack, as the install does for a
  venv it creates or rebuilds, and where reuse cannot be decided from the
  files a torch_kept: line says it cannot tell which torch would be kept.
  The preview's commands and a real rebuild therefore name the same
  requirements.
- The python -m venv support probe starts in a folder of its own, so an
  interpreter given relative to the user's folder (ROCM_CLI_PYTHON=
  .venv/bin/python) was looked up from there and reported unusable. The
  path is anchored to the current directory first; a bare name is left
  to PATH.
- The preview printed UV_VENV_CLEAR=0 as a command prefix, which only
  POSIX shells accept. It is now an env: line beside the uv venv command,
  and a refused folder prints command: none.
- Unit-test stand-ins go through write_fake_executable, the two
  untested reuse failures (no sys.prefix, no wheel tag) get rows, and
  the Unix-only fixtures' imports are gated so the Windows build stays
  clean.
- New scenarios runtime-37 (opt-out preview runs nothing) and runtime-38
  (relative ROCM_CLI_PYTHON); docs updated to match.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed 302697e on top of a merge of current main; this was a plain push, not a force-push. Each inline thread has its own reply.

A new instance of the dry-run issue came in with main. #1031 added the torch-alignment opt-out to install sdk. Its preview asked the install folder's own Python which torch it held, so with ROCM_CLI_DISABLE_TORCH_ALIGNMENT set, a --dry-run ran a bin/python from --prefix again. The same commit fixes this:

  • A preview now runs nothing from the folder, and its commands install the full torch stack.
  • Where the files cannot decide reuse, a torch_kept: line says the preview cannot tell which torch a reused venv would keep.
  • Scenario runtime-37 plants a Python that records every run and checks that the record stays empty.

@jussielo-amd, your remaining non-blocking items:

  • The preview refuses when the first PATH Python is inside --prefix (for example, an activated venv at the prefix). Kept as is. Telling the user's own venv apart from a folder someone else put there needs running that Python, which the preview no longer does. The message says why it stopped and how to preview anyway: take that Python off PATH, or choose another --prefix.
  • Doc drift:
    • The "not resolved" key lists in the README, manual-testing.md and testing.md now include python_env: and env:.
    • manual-testing.md names both ownership routes (.rocm-cli-runtime.json, or a folder inside runtimes/wheel) and the reuse conditions.
    • The "first Python" wording now names ROCM_CLI_PYTHON, the first Python on PATH, and the saved managed Python.
  • Protected-location check for a missing or empty folder: unchanged, as you read it.

Automated review of d80c8c3:

  1. Relative interpreter: fixed; see its thread.
  2. Fake executables: fixed; see its thread.
  3. Conflicts: resolved by merging main.

Its non-blocking items and decisions:

  • Untested reuse branches ("does not report its sys.prefix", "whose wheel tag could not be read"): both are now rows in a_refusal_names_the_condition_that_kept_the_venv_from_being_reused, and each was checked to fail when its message is swapped.
  • The manual-testing.md line "its venv does not run": reworded to list each reuse condition.
  • The fake uv's --clear and UV_VENV_CLEAR=1 handling: kept on purpose. Production never takes that path. The fake behaves like uv so that if the pin or --allow-existing were ever dropped, the tests would hit uv's real destructive behaviour instead of a fake that does nothing.
  • runtime-35's "zero runs": that is the direct count. The planted Python appends a line every time it runs, and assert_no_prefix_python_ran asserts that its log does not exist.
  • A marker counts as ownership: this tradeoff is still intended, for the reason given in the description.
  • Unix-only clear tests: the folder decision runs on Windows (the_venv_folder_decision_reads_files_on_every_platform), as does the new path-anchoring test. Clearing with planted interpreters stays Unix-only, as the description says.

@rominf
rominf requested a review from jussielo-amd November 16, 2026 13:05
the_venv_folder_decision_reads_files_on_every_platform runs on Windows
too, but the import it needs was gated to Unix, so the Windows test build
failed to compile.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full review of the whole change (merge-base ee605065…head 302697e2, 19 files, +3462/-312). Fixes #533 (an install sdk --prefix folder being deleted whole to rebuild its venv); implements the full fix described in the PR body (refuse-unless-owned, selective crash-safe clear, no-execution --dry-run), not a narrower scope.

Blocking

  1. Windows CI is red: a cross-platform test fails to compile on Windows. the_venv_folder_decision_reads_files_on_every_platform (apps/rocm/src/therock/venv.rs:1711) carries no #[cfg(unix)] gate — its own doc comment says it's meant to run "on every platform, the Windows CI job included" — but it calls unix_time_millis() at line 1716, and the only import of that symbol (use rocm_core::unix_time_millis;, line 621) is #[cfg(unix)]-gated. Confirmed directly from the current head's live CI logs: both Windows build and test (debug) and windows-build-and-test report FAILURE, with error[E0425]: cannot find function \unix_time_millis` in this scope --> apps\rocm\src\therock\venv.rs:1716:13. Confidence 100 · logic · Fix: drop the #[cfg(unix)]on that oneuse` (or import it unconditionally), or use a platform-neutral timestamp source for this test's folder-name uniqueness. (inline comment on the line)

Non-blocking

  • wheel_plan.rs:260's unreachable!() for (PlanStack::Resolved(_), None) isn't ruled out by the type signature — a panic!-shaped landmine rather than a statically-impossible case. Dead under every current call site today; a future caller constructing a Resolved plan without its paths would panic instead of failing to compile. (inline comment)
  • venv.rs:290's refuse_protected_venv_rebuild only runs once the --prefix folder is known to have entries, so a not-yet-created or empty path under a protected root (e.g. an unused directory under /usr) gets a venv created there without the protected-location refusal — narrower than README's "...or one in a protected system location, is refused" reads for the general case. Low practical risk (nothing pre-existing is destroyed), and a related gap is already tracked by the PR's own notes for a follow-up. (inline comment)
  • tests/e2e-cucumber/tests/e2e/therock_steps.rs:2806's fake-uv --clear/UV_VENV_CLEAR=1 handling is unreachable from any of this PR's 6 new scenarios (production always pins UV_VENV_CLEAR=0); mirrors an identical pattern in an existing unit-test fixture, so it reads as deliberate fidelity rather than oversight. Harmless. (inline comment)

Decisions for the author

  • anchored_to_current_dir's relative-interpreter/cwd fix is applied unconditionally on all platforms, including Windows — non-blocking-improvement. Its own doc comment says it's a no-op on Windows (Win32 resolves a relative program path against the parent's cwd regardless of a child's current_dir() override), but that assumption isn't exercised by any Windows-run test that actually spawns a process: the two cwd-sensitive regression tests are #[cfg(unix)], and only the pure anchored_to_current_dir unit test runs on Windows CI (checking its output, not real spawn behavior). Worth either a Windows-reachable spawn-level test or an explicit note that the Win32 assumption is taken on documentation rather than verified.
  • The previewed env: line for a uv venv command documents only UV_VENV_CLEAR=0, not UV_HTTP_TIMEOUT/UV_CACHE_DIR (also set by uv_command_env for every uv invocation) — tradeoff. A user pasting the previewed command verbatim won't get the managed cache/timeout behavior. Explicitly scoped by the doc comment to "what the install sets for uv venv," and the gap existed (more silently) before this PR, so this reads as a deliberate, documented narrowing. Confirm that's acceptable.
  • A .rocm-cli-runtime.json marker counts as ownership even if the venv itself is gone — tradeoff, already named in the PR body: anything a user later placed under bin/lib/include in that folder gets cleared too; requiring pyvenv.cfg instead would make a part-way clear unrecoverable. Surfacing per the PR's own description; no new information this round.

Positive signals

  • The dry-run security property — a preview must never execute a binary at a model/caller-chosen --prefix — was traced end-to-end across independent passes over venv.rs, python.rs, and consent.rs/therock.rs via different call chains and confirmed to hold, including the kept_torch step that previously ran a prefix Python unconditionally and is now explicitly skipped under dry_run (therock.rs:504-510).
  • The relative-ROCM_CLI_PYTHON/cwd regression this kind of probe is prone to (a working-directory override breaking relative-path resolution) is fixed correctly and consistently between the wheel-compatibility check and the venv-creation probe, backed by a genuine regression test using an actually-spawned fake interpreter.
  • The crash-safe clear order (interpreter first, pyvenv.cfg last) is both correctly implemented and directly tested, and every clear-stop point is proven to converge.
  • New e2e and unit test code consistently uses the shared rocm_core::fake_executable::write_fake_executable helper (with its own doc comment citing the race it avoids) rather than hand-rolling fs::write + set_permissions.
  • One write_wheel_plan writer serves both the preview and the real install, so a line the plan gains can't drift between the two paths.

Deployment notes

None beyond the blocking Windows CI failure above — a required check (clippy, which also enforces compile success / -D warnings repo-wide) gates mergeability regardless of review outcome.

What this covered

Full diff read via five passes: venv.rs + test_support.rs (read directly); python.rs + process.rs; wheel_plan.rs + resolve/wheel.rs + kept_torch.rs; consent.rs + therock.rs + cli.rs + update.rs; the e2e Gherkin feature + steps. Docs (README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, docs/llm-tool-use.md, docs/vllm.md) were spot-checked directly against the diff hunks and the code they describe: dry-run per-state output, the managed-runtimes-folder path, the NOT_RESOLVED field list, rebuild-reason wording, and docs/architecture.md's venv.rs entry all matched the code.

CI was read live for the current head, not inferred: two required jobs red (Windows build and test (debug), windows-build-and-test, root-caused above), one self-hosted lane (E2E tests (MI300X, current)) stuck queued (looks like runner availability, not this diff), everything else green. Mergeability was verified directly with a local git merge --no-commit --no-ff against current origin/main: clean, no conflicts.

Not reconciled against this PR's prior review rounds from other reviewers — this is an independent read of the current head. It happens to confirm several previously-flagged issue classes (a relative-interpreter/cwd regression, a hand-rolled fake-executable race, stale doc wording, an earlier Windows-import compile break) are fixed as of this commit, and that this is a new, different compile break on the same CI lane.


🤖 Automated by an AI agent

let root = std::env::temp_dir().join(format!(
"rocm-cli-venv-decision-{}-{}-{}",
std::process::id(),
unix_time_millis(),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — Windows build/test is red because of this line. This test has no #[cfg(unix)] gate, and its own doc comment (a few lines up) says it's meant to run "on every platform, the Windows CI job included." But unix_time_millis() here is only in scope via use rocm_core::unix_time_millis; at line 621, which is #[cfg(unix)]-gated. Confirmed directly from the current head's CI logs (windows-build-and-test / Windows build and test (debug), both FAILURE):

error[E0425]: cannot find function `unix_time_millis` in this scope
  --> apps\rocm\src\therock\venv.rs:1716:13

Fix: make the unix_time_millis import available unconditionally (drop the #[cfg(unix)] on that one use), or use a platform-neutral source for this test's folder-name uniqueness.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b808b3b, pushed before this review posted: the unix_time_millis import is now unconditional. On b808b3b windows-build-and-test and Windows build and test (debug) are both green.

Comment thread apps/rocm/src/therock/wheel_plan.rs Outdated
}
PlanStack::NeedsUv { .. } => {
(PlanStack::Resolved(_), None) => {
unreachable!("a resolved plan's preview is written with the paths its venv step reads")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: write_dry_run_tail's signature (plan: &WheelPlan, paths: Option<&AppPaths>) lets plan.stack and paths vary independently, so this (Resolved, None) arm isn't ruled out by the type system — it's a panic!-shaped landmine rather than a statically-impossible case. Every current call site avoids it (confirmed by enumerating all three: therock.rs always passes Some with Resolved, unresolved_wheel_preview always passes None with NeedsUv), so it's dead today, but a future caller constructing a Resolved plan without its paths would panic instead of failing to compile.

Suggested fix: make the invalid combination unrepresentable, e.g. require paths: &AppPaths unconditionally and ignore it in the NeedsUv arm, or fold it into the Resolved stack itself.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 06c2464. paths now lives on ResolvedStack, which is the only case whose venv step needs it, and write_dry_run_tail no longer takes it. A resolved plan can't be built without its paths, so the unreachable! arm is gone.

install_root: &Path,
python_finding: &str,
) -> Result<VenvFolder> {
if !folder_has_entries(install_root)? {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: refuse_protected_venv_rebuild (line 293) only runs once the folder is known to have entries — a --prefix naming a not-yet-created or empty path under a protected root (e.g. an unused directory under /usr) gets a brand-new venv created there without ever hitting the protected-location check, which is narrower than README's "...or one in a protected system location, is refused" reads for the general case. Low practical risk since nothing pre-existing is destroyed in this branch, and the PR's own notes already track a related gap (the protected-location list itself) for a follow-up. Worth confirming this scope is intentional.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That scope is intentional and unchanged. The check guards deletion: a missing or empty folder has nothing to delete, and uv venv creates a missing one. Refusing to create a venv under a protected root would block a deliberate system-wide install, for example under /opt, which main already allows. jussielo-amd accepted this reading in the review of d80c8c3.

[ "$arg" = --allow-existing ] && allow=1
[ "$arg" = --clear ] && clear=1
done
[ "$allow" = 0 ] && [ "${{UV_VENV_CLEAR-}}" = 1 ] && clear=1

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this --clear/UV_VENV_CLEAR=1 handling in the fake uv stand-in is unreachable from any of this PR's 6 new scenarios — production code always passes UV_VENV_CLEAR=0, and the target folder here is always either empty or passed with --allow-existing. Mirrors an identical pattern already in an existing venv.rs unit-test fixture, so it reads as deliberate fidelity to real uv 0.10 behavior rather than an oversight — harmless, test-fixture-only dead code. No action needed unless it was meant to be exercised by a scenario.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is deliberate. Production never takes that path. The fake behaves like uv 0.10 so that if the UV_VENV_CLEAR=0 pin or --allow-existing were ever dropped, the scenarios would hit uv's real destructive behaviour, rather than pass against a fake that ignores the flag. No change.

write_dry_run_tail took the paths as an Option beside the plan, so a
resolved plan without them was a runtime unreachable!() rather than a
type error. The paths now live on ResolvedStack, the one case whose venv
step reads them, and the tail takes the plan alone.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Answering the automated review of 302697e:

  • Blocking (Windows compile): fixed in b808b3b, which was pushed before this review posted. Windows CI is green on that commit; see the thread.
  • The unreachable! in write_dry_run_tail: made unrepresentable in 06c2464; see the thread.
  • Empty or missing prefix under a protected root, and the fake uv's --clear handling: both intentional; see the threads.
  • Anchoring a relative interpreter on Windows: the function no longer depends on how Windows resolves a relative path. It makes the path absolute before the probe starts, and an absolute path names the same file on every platform. The doc comment says exactly that, and no longer claims that it's a no-op on Windows. The pure function is tested on Windows (a_relative_interpreter_path_is_anchored_to_the_current_directory). The spawn-level test stays Unix-only because its stand-in interpreter is a shell script.
  • env: names only UV_VENV_CLEAR=0: intentional. That variable is the one whose absence can delete the folder: a UV_VENV_CLEAR=1 inherited from the shell makes uv clear it. UV_HTTP_TIMEOUT and UV_CACHE_DIR change only how long uv waits and where it caches, so a pasted command without them still produces the same venv.
  • A marker counts as ownership: this tradeoff is unchanged, as you noted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

install sdk: a failing bin/python makes the wheel install delete the whole --prefix, rocm-owned or not

4 participants