Skip to content

ROCMAI-184: test ROCm 7 and ROCm 10 runtimes side by side, and clear a removed rollback target from the active marker - #1106

Merged
rominf merged 3 commits into
mainfrom
ROCMAI-184-test-side-by-side-runtimes
Nov 16, 2026
Merged

rominf merged 3 commits into
mainfrom
ROCMAI-184-test-side-by-side-runtimes

Conversation

@rominf

@rominf rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Problem

rocm runtimes uninstall on the runtime that is the rollback target cleared previous_runtime_key in the config, but the active-runtime marker (<data>/runtimes/active.json) kept naming it as previous_runtime_key. rocm storage reads the marker's keys as runtimes to hold, so a stale key could hold a later reinstall of the same runtime key in storage retention, even though nothing pointed at it as a rollback target any more.

This turned up while adding coverage for ROCm 7 and ROCm 10 runtimes installed side by side for one GPU family (ROCMAI-184). This PR covers that ticket's side-by-side and cleanup coverage criterion plus the fix. It does not close the ticket.

Change

  • Uninstall now also drops the removed runtime from the marker's rollback target (previous_runtime_key and previous_runtime_id), keeping every other field. Keys match without regard to case, as elsewhere in the registry. A marker naming another runtime, or with no rollback target, is left byte for byte as it was. Nothing at the marker path, or a folder there, counts as no marker, as the existing active-runtime check already treats it, so uninstalling an unrelated runtime does not fail.
  • Activation and uninstall now share one marker writer: write_active_runtime_marker moved from runtimes_cmd.rs into runtime_registry.rs unchanged, beside the new forget_active_runtime_rollback_target.

Coverage

Mock-lane scenarios in runtime_lifecycle.feature. Each plants three managed wheel runtimes for one GPU family: a ROCm 7 runtime of the shape the CLI wrote just before the ROCm 10 layout (it has a wheel_composition but no top-level source_layout_generation or devel), a ROCm 10 release runtime (next-v1 layout) and a ROCm 10 nightly (the last nightly published in the canonical layout).

  • @id:runtime-lifecycle-side-by-side-across-packaging-change (runtime-lifecycle-19): all three are listed as ready under their own keys, each can be activated in turn, and no install record is rewritten.
  • @id:runtime-lifecycle-side-by-side-uninstall-keeps-the-others (runtime-lifecycle-20): uninstalling the ROCm 10 release runtime (the rollback target) leaves the other two with their folders and unchanged records, and leaves the nightly in use with no rollback target in either the config or the marker. Then uninstalling the runtime in use leaves nothing in use, and the ROCm 7 runtime can still be activated. This is the scenario that fails without the fix.
  • @id:runtime-lifecycle-side-by-side-shared-name-refused (runtime-lifecycle-21): activating by the runtime id the two release runtimes share is refused, naming both keys, and nothing becomes active.

Scenarios 19 and 21 pin existing behaviour.

Unit tests:

  • runtime_uninstall.rs: uninstalling_the_rollback_target_clears_it_from_the_active_marker, uninstall_clears_a_rollback_target_only_the_marker_still_names, uninstall_treats_a_folder_at_the_marker_path_as_no_marker.
  • runtime_registry.rs: forgetting_a_rollback_target_leaves_a_marker_naming_another_runtime_untouched, forgetting_a_rollback_target_leaves_a_marker_without_one_untouched, forgetting_a_rollback_target_matches_its_key_without_regard_to_case, forgetting_a_rollback_target_without_a_marker_changes_nothing.
  • therock.rs: wheel_runtime_key_matches_the_side_by_side_e2e_fixture ties the fixture's literal keys to wheel_runtime_key. The planting step asserts the same literals, so a change on either side fails one of the two.

Each new test was shown to go red under a targeted break of the code it covers.

Lanes

  • Scenarios 19 to 21 need no GPU or network, so they run on the required Linux e2e lane.
  • They are added to the Windows release lane's E2E_ALSO_RUN_IDS in ci.yml. The xtask contract test (renamed to ci_windows_lifecycle_lane_also_selects_exactly_the_hermetic_non_lifecycle_scenarios) now pins that list exactly and holds the PowerShell copy in docs/testing.md equal to it.
  • The Windows lane was not run locally.

Docs

docs/testing.md, docs/architecture.md and tests/e2e-cucumber/README.md are updated. README, --help and docs/manual-testing.md are unchanged. The only user-visible effect is narrow: a stale rollback key no longer holds a later reinstall of the same key in storage retention.

Verification

Passed locally: cargo fmt, cargo clippy --workspace --all-targets -- -D warnings, the targeted unit tests, the xtask contract tests, cargo xtask check-architecture-doc, check-module-size and check-crate-edges, and the runtime_lifecycle e2e feature (20 of 20 non-GPU scenarios).

Did not pass on this host, for environmental reasons outside the diff:

  • cargo xtask smoke stops at the rocmd sandbox step, because bwrap is not permitted on this host. Not re-run on main.
  • Two unit tests in the full workspace run fail: one on a socket path that is too long under a long TMPDIR, one because docker is a podman shim. Not re-run on main.
  • runtime-lifecycle-12 (@requires-gpu) fails here, and fails the same way on main.

Not covered

  • The scenarios test record shape and coexistence, not package contents or update paths.

  • The oldest legacy shape, from before wheel_composition existed, is not planted.

  • Serving from a legacy record needs a GPU, so there is no mock-lane scenario for it.

  • The stub library names are unversioned.

  • The e2e rollback check reads the marker file directly, because only rocm storage consumes that field.

  • Known siblings, pre-existing and unchanged here:

    • uninstalling the active runtime deletes the marker but keeps config.previous_runtime_key;
    • engines' preferred_runtime_id and last_installed_runtime_id, the ComfyUI current.json and service records are not reconciled on uninstall;
    • if the marker step fails, uninstall returns a plain error after the folder, registry entry and config have already been removed, as main's existing active-marker path does;
    • storage batch removal aborts on that error;
    • the marker read-modify-write is not locked.
  • The ticket's exact-package-lock criterion is out of scope here and waits on an answer from the ticket owner.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. No rows reference ROCMAI-184.

  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. Not applicable: no subcommand or subsystem added.

  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry. runtime_side_by_side_steps.rs is added with an entry, and the entries for runtime_registry.rs, runtime_uninstall.rs and runtimes_cmd.rs are updated for the moved writer.

  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state. No message about what the CLI did changed; the only new text is the read and parse error contexts naming the marker path.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

Uninstalling the runtime that is the rollback target cleared
config.previous_runtime_key but left data/runtimes/active.json naming it,
so storage retention kept holding a key for a runtime that no longer
exists. Uninstall now clears it from the marker too.

The marker writer moves next to ActiveRuntimeMarker so activation and
uninstall share the one atomic write; the marker keeps naming the runtime
in use, and a rollback target that names another runtime is left alone.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Runtimes installed before the ROCm 10 layout and runtimes installed from
it coexist on one machine, and the CLI has to keep reading the older
record as it was written while it lists, switches between and removes the
newer ones. Nothing exercised that.

Mock-lane scenarios runtime-lifecycle-19/20/21 plant a ROCm 7 runtime
installed before the ROCm 10 layout plus ROCm 10 release and nightly
runtimes for one GPU family. They list, activate each in turn, uninstall
one runtime and then the active one while the others' folders and records
stay untouched, and refuse a runtime id shared by two runtimes with both
keys named. The Windows release lane also runs them, so the Windows paths
are covered on a required lane.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The comments called write_active_runtime_marker the marker's one writer,
but a failed activation's restore writes or removes the marker directly,
and uninstalling the runtime in use and `rocm config
set-default-runtime`/`clear-default-runtime` remove it. The registry module
doc also said uninstall only reads and clears the marker, when it now
rewrites the rollback target too. Name each of them.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-184-test-side-by-side-runtimes branch from 98ef627 to 481ddca Compare November 16, 2026 18:48
@rominf
rominf marked this pull request as ready for review November 16, 2026 19:07
@rominf
rominf requested a review from a team as a code owner November 16, 2026 19:07
@rominf
rominf requested a review from fredespi November 16, 2026 19:07

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff for the side-by-side ROCm 7 / ROCm 10 runtime scenarios and the rollback-marker fix. I found nothing blocking, and CI was green when I looked (the CodeQL scan was still pending). This is a read of the diff only; I did not build or run the tests myself.

Three non-blocking notes, left inline:

  • A damaged active.json now makes rocm runtimes uninstall of an unrelated runtime fail after its folder and registry entry are already removed. Treating an unparsable marker as "no rollback target" (or warning), plus a test for it, would close that gap.
  • The storage-retention effect is asserted by reading active.json rather than through rocm storage. The PR body explains the choice, so I'm fine with it.
  • A fixture comment cites a bare commit hash; describing the old layout in words will age better.

The scenario coverage, docs, and architecture entry all line up with the PR description.

} else {
// The marker keeps naming the runtime in use; only a rollback target
// that names the removed runtime is dropped, as the config already did.
forget_active_runtime_rollback_target(paths, &manifest.runtime_key)?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A malformed active.json makes this ? fail the uninstall of an unrelated runtime after its folder and registry entry are already gone. Consider treating an unparsable marker as no rollback target (or warning), and add a test for it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the gap is real, but it predates this PR rather than coming from this ?. On main, apply_runtime_uninstall already calls active_runtime_marker_matches(paths, &manifest.runtime_key)? after the folder and registry entry are removed, and that function reads and parses active.json, returning an error for an unreadable or invalid-JSON marker. So a malformed marker already fails the uninstall of any runtime, unrelated ones included, at that earlier ?. The new forget_active_runtime_rollback_target call in the else branch only runs once that parse has succeeded (or there is no file), so it adds no new way to reach this state.

To keep this PR to its scope, I've recorded it as a follow-up on the ticket: treat an unparsable marker as no marker / no rollback target (with a warning), or read the marker before anything is removed, plus a test with an invalid-JSON marker.

let marker_path = active_runtime_marker_path(paths);
// As in `active_runtime_marker_matches`: anything but a file there is no
// marker, so there is no rollback target to drop.
if !marker_path.is_file() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a non-file marker is tolerated here; an unreadable or invalid-JSON marker returns Err. Worth confirming this matches how active_runtime_marker_matches treats the same file.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, the two readers agree. Both active_runtime_marker_matches and forget_active_runtime_rollback_target treat anything other than a regular file at the path as "no marker" (!marker_path.is_file() → Ok(false)), and both return an error with the same failed to read … / failed to parse … context for an unreadable or invalid-JSON file. Tolerating a malformed marker is the follow-up noted in the other thread, and it would apply to both readers together.

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Role {
/// ROCm 7.14.0 from the canonical release stream, installed by the CLI as
/// it stood just before the ROCm 10 layout was added (commit `a6fa76db`'s

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer describing the pre-ROCm-10 record layout in words over a bare commit hash; the hash can't be verified from the repo text and goes stale after history rewrites.

@rominf rominf Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that a bare hash is a weak reference. a6fa76db is in main's history (it is #329, which added the ROCm 10 "next" install layout), and the comment also describes the record in words: it has a wheel composition and a fingerprinted key, but no source_layout_generation or devel. I've left the hash so this approved head isn't disturbed, and recorded a follow-up on the ticket to cite #329 or drop the reference.

@rominf
rominf added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 54df940 Nov 16, 2026
52 checks passed
@rominf
rominf deleted the ROCMAI-184-test-side-by-side-runtimes branch November 16, 2026 19:53
rominf added a commit that referenced this pull request Nov 17, 2026
Conflicts with #1106 (side-by-side ROCm 7/ROCm 10 runtimes):

- runtimes_cmd.rs: #1106 moved write_active_runtime_marker into
  runtime_registry.rs; import it from there alongside RuntimeProvenance.
  The listing is unchanged by #1106, so the row -> install_root: ->
  provenance: order stands.
- runtime_lifecycle.feature: #1106 added runtime-lifecycle-19/20/21, so
  the provenance listing scenario is renumbered to runtime-lifecycle-22
  (its @id is unchanged), with the references in docs/testing.md,
  examine.feature and runtime_registry.rs updated to match.
- docs/architecture.md: the runtime_registry.rs entry carries both the
  marker writer and RuntimeProvenance; the runtime_lifecycle_steps.rs
  entry keeps the provenance listing beside #1106's new
  runtime_side_by_side_steps.rs entry.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants