Repository navigation
ROCMAI-184: test ROCm 7 and ROCm 10 runtimes side by side, and clear a removed rollback target from the active marker - #1106
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
Uninstalling the runtime that is the rollback target cleared config.previous_runtime_key but left data/runtimes/active.json naming it, so storage retention kept holding a key for a runtime that no longer exists. Uninstall now clears it from the marker too. The marker writer moves next to ActiveRuntimeMarker so activation and uninstall share the one atomic write; the marker keeps naming the runtime in use, and a rollback target that names another runtime is left alone. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Runtimes installed before the ROCm 10 layout and runtimes installed from it coexist on one machine, and the CLI has to keep reading the older record as it was written while it lists, switches between and removes the newer ones. Nothing exercised that. Mock-lane scenarios runtime-lifecycle-19/20/21 plant a ROCm 7 runtime installed before the ROCm 10 layout plus ROCm 10 release and nightly runtimes for one GPU family. They list, activate each in turn, uninstall one runtime and then the active one while the others' folders and records stay untouched, and refuse a runtime id shared by two runtimes with both keys named. The Windows release lane also runs them, so the Windows paths are covered on a required lane. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The comments called write_active_runtime_marker the marker's one writer, but a failed activation's restore writes or removes the marker directly, and uninstalling the runtime in use and `rocm config set-default-runtime`/`clear-default-runtime` remove it. The registry module doc also said uninstall only reads and clears the marker, when it now rewrites the rollback target too. Name each of them. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
98ef627 to
481ddca
Compare
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed the diff for the side-by-side ROCm 7 / ROCm 10 runtime scenarios and the rollback-marker fix. I found nothing blocking, and CI was green when I looked (the CodeQL scan was still pending). This is a read of the diff only; I did not build or run the tests myself.
Three non-blocking notes, left inline:
- A damaged
active.jsonnow makesrocm runtimes uninstallof an unrelated runtime fail after its folder and registry entry are already removed. Treating an unparsable marker as "no rollback target" (or warning), plus a test for it, would close that gap. - The storage-retention effect is asserted by reading
active.jsonrather than throughrocm storage. The PR body explains the choice, so I'm fine with it. - A fixture comment cites a bare commit hash; describing the old layout in words will age better.
The scenario coverage, docs, and architecture entry all line up with the PR description.
| } else { | ||
| // The marker keeps naming the runtime in use; only a rollback target | ||
| // that names the removed runtime is dropped, as the config already did. | ||
| forget_active_runtime_rollback_target(paths, &manifest.runtime_key)?; |
There was a problem hiding this comment.
A malformed active.json makes this ? fail the uninstall of an unrelated runtime after its folder and registry entry are already gone. Consider treating an unparsable marker as no rollback target (or warning), and add a test for it.
There was a problem hiding this comment.
Thanks — the gap is real, but it predates this PR rather than coming from this ?. On main, apply_runtime_uninstall already calls active_runtime_marker_matches(paths, &manifest.runtime_key)? after the folder and registry entry are removed, and that function reads and parses active.json, returning an error for an unreadable or invalid-JSON marker. So a malformed marker already fails the uninstall of any runtime, unrelated ones included, at that earlier ?. The new forget_active_runtime_rollback_target call in the else branch only runs once that parse has succeeded (or there is no file), so it adds no new way to reach this state.
To keep this PR to its scope, I've recorded it as a follow-up on the ticket: treat an unparsable marker as no marker / no rollback target (with a warning), or read the marker before anything is removed, plus a test with an invalid-JSON marker.
| let marker_path = active_runtime_marker_path(paths); | ||
| // As in `active_runtime_marker_matches`: anything but a file there is no | ||
| // marker, so there is no rollback target to drop. | ||
| if !marker_path.is_file() { |
There was a problem hiding this comment.
Only a non-file marker is tolerated here; an unreadable or invalid-JSON marker returns Err. Worth confirming this matches how active_runtime_marker_matches treats the same file.
There was a problem hiding this comment.
Confirmed, the two readers agree. Both active_runtime_marker_matches and forget_active_runtime_rollback_target treat anything other than a regular file at the path as "no marker" (!marker_path.is_file() → Ok(false)), and both return an error with the same failed to read … / failed to parse … context for an unreadable or invalid-JSON file. Tolerating a malformed marker is the follow-up noted in the other thread, and it would apply to both readers together.
| #[derive(Debug, Clone, Copy, PartialEq, Eq)] | ||
| enum Role { | ||
| /// ROCm 7.14.0 from the canonical release stream, installed by the CLI as | ||
| /// it stood just before the ROCm 10 layout was added (commit `a6fa76db`'s |
There was a problem hiding this comment.
Prefer describing the pre-ROCm-10 record layout in words over a bare commit hash; the hash can't be verified from the repo text and goes stale after history rewrites.
There was a problem hiding this comment.
Agreed that a bare hash is a weak reference. a6fa76db is in main's history (it is #329, which added the ROCm 10 "next" install layout), and the comment also describes the record in words: it has a wheel composition and a fingerprinted key, but no source_layout_generation or devel. I've left the hash so this approved head isn't disturbed, and recorded a follow-up on the ticket to cite #329 or drop the reference.
Conflicts with #1106 (side-by-side ROCm 7/ROCm 10 runtimes): - runtimes_cmd.rs: #1106 moved write_active_runtime_marker into runtime_registry.rs; import it from there alongside RuntimeProvenance. The listing is unchanged by #1106, so the row -> install_root: -> provenance: order stands. - runtime_lifecycle.feature: #1106 added runtime-lifecycle-19/20/21, so the provenance listing scenario is renumbered to runtime-lifecycle-22 (its @id is unchanged), with the references in docs/testing.md, examine.feature and runtime_registry.rs updated to match. - docs/architecture.md: the runtime_registry.rs entry carries both the marker writer and RuntimeProvenance; the runtime_lifecycle_steps.rs entry keeps the provenance listing beside #1106's new runtime_side_by_side_steps.rs entry. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Problem
rocm runtimes uninstallon the runtime that is the rollback target clearedprevious_runtime_keyin the config, but the active-runtime marker (<data>/runtimes/active.json) kept naming it asprevious_runtime_key.rocm storagereads the marker's keys as runtimes to hold, so a stale key could hold a later reinstall of the same runtime key in storage retention, even though nothing pointed at it as a rollback target any more.This turned up while adding coverage for ROCm 7 and ROCm 10 runtimes installed side by side for one GPU family (ROCMAI-184). This PR covers that ticket's side-by-side and cleanup coverage criterion plus the fix. It does not close the ticket.
Change
previous_runtime_keyandprevious_runtime_id), keeping every other field. Keys match without regard to case, as elsewhere in the registry. A marker naming another runtime, or with no rollback target, is left byte for byte as it was. Nothing at the marker path, or a folder there, counts as no marker, as the existing active-runtime check already treats it, so uninstalling an unrelated runtime does not fail.write_active_runtime_markermoved fromruntimes_cmd.rsintoruntime_registry.rsunchanged, beside the newforget_active_runtime_rollback_target.Coverage
Mock-lane scenarios in
runtime_lifecycle.feature. Each plants three managed wheel runtimes for one GPU family: a ROCm 7 runtime of the shape the CLI wrote just before the ROCm 10 layout (it has awheel_compositionbut no top-levelsource_layout_generationordevel), a ROCm 10 release runtime (next-v1layout) and a ROCm 10 nightly (the last nightly published in the canonical layout).@id:runtime-lifecycle-side-by-side-across-packaging-change(runtime-lifecycle-19): all three are listed as ready under their own keys, each can be activated in turn, and no install record is rewritten.@id:runtime-lifecycle-side-by-side-uninstall-keeps-the-others(runtime-lifecycle-20): uninstalling the ROCm 10 release runtime (the rollback target) leaves the other two with their folders and unchanged records, and leaves the nightly in use with no rollback target in either the config or the marker. Then uninstalling the runtime in use leaves nothing in use, and the ROCm 7 runtime can still be activated. This is the scenario that fails without the fix.@id:runtime-lifecycle-side-by-side-shared-name-refused(runtime-lifecycle-21): activating by the runtime id the two release runtimes share is refused, naming both keys, and nothing becomes active.Scenarios 19 and 21 pin existing behaviour.
Unit tests:
runtime_uninstall.rs:uninstalling_the_rollback_target_clears_it_from_the_active_marker,uninstall_clears_a_rollback_target_only_the_marker_still_names,uninstall_treats_a_folder_at_the_marker_path_as_no_marker.runtime_registry.rs:forgetting_a_rollback_target_leaves_a_marker_naming_another_runtime_untouched,forgetting_a_rollback_target_leaves_a_marker_without_one_untouched,forgetting_a_rollback_target_matches_its_key_without_regard_to_case,forgetting_a_rollback_target_without_a_marker_changes_nothing.therock.rs:wheel_runtime_key_matches_the_side_by_side_e2e_fixtureties the fixture's literal keys towheel_runtime_key. The planting step asserts the same literals, so a change on either side fails one of the two.Each new test was shown to go red under a targeted break of the code it covers.
Lanes
E2E_ALSO_RUN_IDSinci.yml. The xtask contract test (renamed toci_windows_lifecycle_lane_also_selects_exactly_the_hermetic_non_lifecycle_scenarios) now pins that list exactly and holds the PowerShell copy indocs/testing.mdequal to it.Docs
docs/testing.md,docs/architecture.mdandtests/e2e-cucumber/README.mdare updated. README,--helpanddocs/manual-testing.mdare unchanged. The only user-visible effect is narrow: a stale rollback key no longer holds a later reinstall of the same key in storage retention.Verification
Passed locally:
cargo fmt,cargo clippy --workspace --all-targets -- -D warnings, the targeted unit tests, the xtask contract tests,cargo xtask check-architecture-doc,check-module-sizeandcheck-crate-edges, and theruntime_lifecyclee2e feature (20 of 20 non-GPU scenarios).Did not pass on this host, for environmental reasons outside the diff:
cargo xtask smokestops at therocmdsandbox step, because bwrap is not permitted on this host. Not re-run on main.TMPDIR, one becausedockeris a podman shim. Not re-run on main.@requires-gpu) fails here, and fails the same way on main.Not covered
The scenarios test record shape and coexistence, not package contents or update paths.
The oldest legacy shape, from before
wheel_compositionexisted, is not planted.Serving from a legacy record needs a GPU, so there is no mock-lane scenario for it.
The stub library names are unversioned.
The e2e rollback check reads the marker file directly, because only
rocm storageconsumes that field.Known siblings, pre-existing and unchanged here:
config.previous_runtime_key;preferred_runtime_idandlast_installed_runtime_id, the ComfyUIcurrent.jsonand service records are not reconciled on uninstall;The ticket's exact-package-lock criterion is out of scope here and waits on an answer from the ticket owner.
If this PR fixes a bug, searched
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. No rows reference ROCMAI-184.If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per
docs/architecture.md. Not applicable: no subcommand or subsystem added.If this PR adds, renames or removes a production source file (or a
.rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry.runtime_side_by_side_steps.rsis added with an entry, and the entries forruntime_registry.rs,runtime_uninstall.rsandruntimes_cmd.rsare updated for the moved writer.Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state. No message about what the CLI did changed; the only new text is the read and parse error contexts naming the marker path.