Skip to content

feat(lemonade): announce installs, show their progress, and report them in storage - #853

Merged
volen-silo merged 2 commits into
mainfrom
rocmai-825-lemonade-per-runtime
Nov 9, 2026
Merged

volen-silo merged 2 commits into
mainfrom
rocmai-825-lemonade-per-runtime

Conversation

@volen-silo

@volen-silo volen-silo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

On native Windows, after rocm engines install lemonade and then rocm install sdk, the next rocm serve silently spent about 10 minutes installing a second full Lemonade (~24 GiB) under the new runtime's folder. Nothing said it was happening, no free space was checked, and the first copy was left referenced by nothing. (ROCMAI-825.)

Root cause

  • env_root_for_self_managed_engine (apps/rocm/src/engines_cmd.rs) places Lemonade under the active runtime's install_root/engines, so each new runtime gets its own tree. That placement is unchanged here; a shared tree is tracked separately as ROCMAI-831.
  • run_lemonade_backend_install (engines/lemonade/src/process.rs) started the backend install with CREATE_NO_WINDOW. A probe in a real console window showed that with that flag cmd /c echo, a Python child and lemonade.exe --version all print nothing; without it all three print. That is why the install was silent.

Change

Commit 1: announce installs and show progress

  • Before writing a new Lemonade tree, serve, engines install lemonade and --reinstall print the target folder, an upper bound on the download ("up to about 4.4 GiB": the smaller backend may need less) and the disk use ("up to about 25.0 GiB"), the free space on that drive, and any existing copy that is not being reused (new engines/lemonade/src/footprint.rs). A copy counts only if the install record points at a folder that still holds an install; a dangling record is not announced as an existing copy. The notice and both space warnings print the same two figures (all derived from the same constants; the safety margin is used only for the comparison). The rocm storage pointer is given only for a copy storage lists (the data folder's tree and trees inside registered ROCm installs). When the install replaces an existing tree (--reinstall, a new Lemonade version, a missing server binary: the runtime folder is wiped first), that tree's size is counted as free for the comparison and the notice says so.
  • Free space has two tiers. It refuses only below 16 MiB, the smallest install that can work (the archive plus what extracts outside bin, measured on a real tree). Below the ROCm backend's download or the finished tree it warns and continues, because the backend is chosen after this preflight and backend_alignment.rs falls back from rocm to vulkan, which needs less. Install-time serve announces only; there is no existing consent step for installs during serve.
  • CREATE_NO_WINDOW is now applied only when neither stdout nor stderr is a console, decided inside backend_install_command (engines/lemonade/src/process.rs), which takes the console state and the hide step, so one test builds the real command for all four console states and asserts both whether the window was hidden and the command's arguments. Lemonade's own progress lines show again. No spinner: it would overwrite those lines.

Commit 2: storage and runtime uninstall

  • rocm storage report gets an "Engine installs" section listing each Lemonade copy with its size. Lemonade's install record is read as one of three states (no record; an unreadable record; a record naming a folder that does or does not still hold the server binary), which gives these statuses:
    • in use: the record names this copy.
    • not used right now; Lemonade runs from another install: the record names a different copy that really holds an install. Only the data-folder copy is called reclaimable, and its note says it is only used while no ROCm install is active. A copy inside a registered runtime is never offered for deletion: it says it is reused if that ROCm install becomes the active one.
    • not known to be used; the install record names a folder with no install, and not known to be used; the install record could not be read: neither is marked reclaimable, so the report never advises deleting the only tree that works.
    • not recorded as installed: there is no record.
      Also in --json.
  • The "removed with its ROCm install" note on a per-runtime row follows the same rule rocm runtimes uninstall applies (should_remove_runtime_install_root): for a read-only or imported runtime, uninstall only unregisters it, so the row says ROCm CLI does not delete that install's folder.
  • rocm runtimes uninstall warns that the runtime's Lemonade goes with it, and afterwards clears Lemonade's install record so the next serve does not trip on a deleted folder. The lemonade: line says what happened: cleared; deletion failed (it names the record to delete); or unreadable record (it names the plain rocm engines install lemonade, which rewrites the record and keeps a current tree, so --reinstall is not needed). The unreadable-record line prints only when the removed runtime's folder held a Lemonade tree. storage remove-old-installs gets the same through uninstall_runtime.

Docs: README (Disk space, Inference engines), docs/testing.md, docs/manual-testing.md, docs/architecture.md.

PR template checklist

  • Stale xfail rows. tests/e2e-cucumber/expectations.toml has no row naming ROCMAI-825 or lemonade-installs; the only Lemonade rows are the EAI-7423 Linux serve ones, which this change does not touch. Nothing was stale.
  • New subsystem in its own file. The announcement, free-space check and install-record view live in engines/lemonade/src/footprint.rs; nothing grew in lib.rs beyond a small seam (install_and_record) and one call site.
  • Architecture doc. docs/architecture.md has the footprint.rs entry, and the prose of the storage.rs and runtime_uninstall.rs entries is updated; cargo xtask check-architecture-doc passes.
  • Messages read against the code path after them, tests assert state.
Message Test asserting the state, not only the wording
"Installing Lemonade ... into " and the "up to about" download and disk line unit a_new_location_is_announced...; scenarios 01 and 02 check nothing was downloaded, extracted or recorded
"Lemonade is already installed at X; that copy is not reused ... See rocm storage" the same unit test (derived from a record whose folder holds an install); a_record_naming_a_folder_with_no_install_is_not_announced_as_a_copy shows it is absent otherwise
Refusal "not enough free disk space ... Nothing was downloaded." scenarios 01 and 02 against the disk; the unit verdict test
Space warnings "The install will continue" the verdict test asserts a warning, not a refusal. The call site matching on the verdict is trivial and has no test, because proving it continues would start the download
Storage statuses and notes (in use, not used right now, not known to be used, unreadable, removed with it / not removed with it) storage unit tests, including two registered runtimes and a read-only runtime; scenarios 03, 05, 06
Uninstall plan line "inside this runtime folder and would be removed with it" scenario 04 (plan, result and disk state)
Uninstall results: cleared, deletion failed (names the record), unreadable (names the plain command) the_lemonade_record_line_matches_what_happened (wording); runtime_uninstall_clears..., ...reports_a_lemonade_record_it_could_not_read and the unrelated-runtime test (state); a_plain_install_replaces_a_corrupt_record_and_keeps_the_current_tree shows the advised command clears the condition

Tests

  • Unit tests: in footprint.rs (announcement, the two free-space tiers, the install-record API and its three states, the plain install replacing a corrupt record), process.rs (the hide-console decision on the real command), storage.rs (the statuses and the read-only note) and runtime_uninstall.rs (record cleanup and its report line).
  • Scenarios, new feature tests/e2e-cucumber/features/lemonade_installs.feature, mock lane (the free-space and install-state hooks exist only in the e2e-test-hooks build): lemonade-installs-01 (engines install with no usable space is refused before anything is downloaded), -02 (the same through rocm serve --engine lemonade --managed, so the serve-side announcement is covered; to reach the install on a no-GPU lane the hooks-only no-GPU waiver in serve_cmd.rs also honours the free-space variable), -03 (storage lists both copies and flags the unused one), -04 (uninstalling the runtime that held Lemonade clears the record), -05 (a record naming a folder with no install does not make the survivor reclaimable), -06 (a read-only runtime's row does not say removed with its ROCm install). They passed locally against a hooks build, 6 of 6; scenarios 01 and 02 now report 1 MiB free, because the warning range cannot be a scenario (it would start the download) and is unit-tested only.
  • cargo nextest run -p rocm-engine-lemonade -p rocm (1748 passed), clippy for those crates and e2e-cucumber (including --test e2e), cargo fmt --check, cargo xtask check-crate-edges, cargo xtask check-architecture-doc, lychee on the changed docs: pass.

The new tests fail when their target is broken (each break applied alone against the final tree, then reverted):

Test Break applied Red?
a_new_location_is_announced_with_the_install_it_does_not_reuse drop the "already installed" line; print "this downloads about" instead of "up to about" yes (each)
a_current_tree_is_not_announced, a_tree_without_a_backend_is_announced always announce; ignore a missing backend yes (each)
a_record_naming_a_folder_with_no_install_is_not_announced_as_a_copy use a record whether or not it holds an install; holds_install always true yes (each)
only_space_below_the_smallest_install_refuses_and_less_than_the_rocm_backend_warns refuse at the ROCm download size; remove the download-warning tier; flip the refuse comparison yes (each)
the_notice_and_the_warnings_print_the_same_download_and_disk_figures download warning prints the margined value; disk warning prints the margined value yes (each)
a_reinstall_counts_the_space_of_the_install_it_replaces replaced size always 0; verdict ignores it; drop the "replaced first" line; count the folder even when the tree is kept yes (each)
the_storage_pointer_is_only_given_for_a_copy_storage_lists pointer for every copy; for none; storage rule ignores registered runtimes yes (each)
the_record_is_forgotten_only_when_its_tree_was_removed clear whatever the record points at yes
a_record_that_cannot_be_deleted_is_reported_not_swallowed swallow the deletion error yes
an_unreadable_record_is_distinct_from_no_record unparseable record becomes "no record" yes
a_record_reports_whether_its_folder_still_holds_an_install holds_install always true yes
a_plain_install_replaces_a_corrupt_record_and_keeps_the_current_tree skip writing the record; force a reinstall yes (each)
the_backend_install_hides_its_window_only_when_no_output_is_a_console never hide; hide always; ignore stderr yes (each)
same run_lemonade_backend_install passes a no-op hide function no (see Gaps)
storage tests (in use / not used right now / dangling / unreadable / read-only / two registered runtimes) record elsewhere reported as in use; revert the status wording; mark any unused copy reclaimable; remove the no-install arm; unreadable treated as no record; removed_with_runtime always true yes (each)
runtime_uninstall tests skip the record cleanup; unreadable outcome prints nothing; advice names --reinstall; always report an unreadable record; failed-deletion line claims "no longer recorded" yes (each)
scenarios 01, 02 suppress the printed notice; print the old wording yes (each)
scenario 03 record elsewhere reported as in use; revert the status wording yes (each)
scenarios 04, 05, 06 skip the record cleanup; remove the no-install arm; removed_with_runtime always true yes (each)

Gaps: the call site in run_lemonade_backend_install passes hide_child_console_window and the real console probe, and no test would notice a no-op swapped in there (the creation flag cannot be read back off a built command), so the Windows console claim rests on the manual check in docs/manual-testing.md. Scenarios 01 and 02 were not broken at the refusal threshold itself, because that would start the download; the unit test covers it. The serve-side no-GPU waiver was not mutation-tested, because the development host has a GPU. There is no black-box scenario for the plain-command advice, because that would download.

CI

On the previous head the one unexpected failure was dash-43 on E2E tests (ubuntu-26.04): a 30 s wait for the dashboard warnings badge, in code this PR does not touch, while all six lemonade-installs scenarios passed in that job. It is intermittent: that job passed 174 of 177 runs since the scenario was added, and the plain E2E tests job failed twice on dash-gen-tps-expiry-boundary, which shares the same 30 s wait (dash_steps.rs). I did not change dash code here. On this head all 29 checks that ran pass, including E2E tests (ubuntu-26.04) and the required E2E tests job, where lemonade-installs-01 to 06 ran. The self-hosted GPU lanes are not required and gave no result for the previous head: E2E tests (MI300X) and (W7900) got no runner for 24 hours, and the Ubuntu lanes failed after about 33 minutes (the lane problem tracked outside this PR).

Not in this PR

  • Other lemond launch paths in process.rs use the same flag and may hide output the same way.
  • A failed backend alignment can force up to three multi-GB backend reinstalls per install; remembering failed alignments belongs with the shared-tree follow-up (ROCMAI-831).

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The announce-before-install, free-space check and per-runtime storage listing all look right, and the scenarios assert both the message and the resulting state. I'm requesting changes on two storage-report claims that can be wrong; both are inline.

Requested changes

  1. A dangling install record makes every surviving Lemonade copy read "not used / reclaimable" (inline on storage.rs).
  2. The "removed with it" note is printed for every per-runtime row, including read-only/imported runtimes where it is false (inline on storage.rs).

Smaller notes (non-blocking)

  • The Windows console-progress fix is covered only by a pure-helper unit test. The user-visible claim rests on the manual check you disclosed, which is fine, but a swapped stdout/stderr argument or a dropped call would not turn that test red.
  • If clearing the install record fails after the folder is deleted, the only signal is a stderr warning. Consider surfacing it in the uninstall result so the record isn't left dangling silently. This ties into item 1, since an unreadable record is treated as "not recorded".
  • The serve-time announcement has no scenario of its own (only the engines install path is covered). Low risk since both go through the same install path, but the PR text says serve announces.
  • Please state in the PR text that the new tests were shown to fail with their target broken (AGENTS section 3).
  • Nits: footprint.rs is out of alphabetical order in docs/architecture.md, and a few comment/doc lines in runtime_uninstall.rs and docs/testing.md are very long.

Comment thread apps/rocm/src/storage.rs Outdated
complete && measured.complete,
)
});
let status = match recorded.as_deref() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This marks the copy Unused whenever any record exists elsewhere. If the recorded runtime_dir no longer has an install (e.g. a runtime uninstalled before this change), nothing is actually in use, yet every surviving tree is reported "not used" and "reclaimable", and a later serve reuses the data-folder tree. Following the delete advice would force a ~25 GiB reinstall. Suggest treating a record whose runtime has no install (tree_has_install) as NotRecorded, or giving it a distinct "record is dangling" state, and adding a test for it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and fixed in 11ce27f. Lemonade's install record is now read as three states (no record; an unreadable record; a record naming a folder that does or does not still hold the server binary), and storage report has five statuses. Only not used is called reclaimable, and only when the record names a different copy that really holds an install. A record naming a folder with no install reads "not known to be used; the install record names a folder with no install", and an unreadable one reads "... could not be read"; neither is marked reclaimable, so the report can no longer advise deleting the only working tree. Tests: a_record_naming_a_folder_with_no_install_does_not_make_the_survivor_reclaimable, an_unreadable_record_leaves_every_copy_unmarked, engine_install_status_follows_the_record, and scenario lemonade-installs-05; each goes red when its target is broken (table in the description).

Comment thread apps/rocm/src/storage.rs
);
}
if usage.runtime_key.is_some() {
let _ = writeln!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note says the tree is "removed with" the ROCm install unconditionally. For a read-only/imported runtime, uninstall only unregisters it, so the claim is false; the uninstall plan already guards on will_remove_install_root(). Please derive the note from the same decision (or soften it) and cover a read-only runtime row with a test.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, that claim was false for a read-only or imported runtime. The note now follows the same rule rocm runtimes uninstall applies (should_remove_runtime_install_root): for a read-only runtime the row says ROCm CLI does not delete that install's folder, so uninstalling it leaves the copy. A unit test covers a read-only row and also runs the real uninstall to check the copy survives, and scenario lemonade-installs-06 covers it end to end.

@volen-silo
volen-silo force-pushed the rocmai-825-lemonade-per-runtime branch from 75fbd1f to 11ce27f Compare November 6, 2026 15:15
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Pushed 11ce27f (two commits, rebased onto main). Both requested changes are done (threads), and the smaller notes:

  • Console-progress fix: the hide decision now lives in the real command builder (backend_install_command), and one test builds the command for all four console states and asserts both whether the window was hidden and the arguments. One gap remains and is stated in the description: a no-op swapped in at the call site run_lemonade_backend_install would not turn a test red (the creation flag cannot be read back off a built command), so that claim still rests on the manual check.
  • Record cleanup failure: the lemonade: line of rocm runtimes uninstall now says whether the record was cleared, could not be deleted (and which one to delete), or could not be read, instead of only a stderr warning; an unreadable record is distinct from "not recorded".
  • Serve announcement: new scenario lemonade-installs-02 runs rocm serve --engine lemonade --managed against a full drive. To reach the install on a no-GPU lane, the hooks-only no-GPU waiver in serve_cmd.rs also honours the free-space variable.
  • Red evidence: the description now has a table of every new test with the break applied and whether it went red. The only "no" is the call-site gap above.
  • Nits: footprint.rs is in alphabetical order in docs/architecture.md, and the long lines I added in runtime_uninstall.rs and docs/testing.md are wrapped.

Scenarios are now lemonade-installs-01..06. They passed locally against a hooks build (6 of 6); this head's CI run is the first for 02 and 04 to 06. Required checks are running on this head.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 6, 2026 15:26

Both requested changes are addressed in 11ce27f (a dangling or unreadable install record no longer marks any copy reclaimable, and the "removed with it" note follows the uninstall plan's decision), along with the smaller notes; see the replies, the summary comment and the updated description. A re-review is welcome.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The announcement before install, the free-space check and the per-runtime storage listing all read well, and the scenarios assert both the message and the resulting state. The five storage statuses, the refuse/warn thresholds and the record clearing match the description. This was a static review; I did not build or run the tests.

Blocking: CI is red on this head. In the CI run for 11ce27f, E2E tests (ubuntu-26.04) failed on dash-43 (the 30 s wait for the dashboard warnings badge timed out). Nothing in this diff touches the dashboard, the plain E2E tests job on the same commit is green, and all six lemonade-installs scenarios passed in the failing job, so I expect a flake. A red check still stops the merge, though. Please rerun it; the PR also conflicts with main now, so the rebase will start a fresh run anyway. If it fails the same way again it is worth a look.

Non-blocking, inline: a dangling install record produces a false "already installed" line; the hard refusal assumes the ROCm backend download; the unreadable-record advice names the heavier --reinstall; the "reclaimable" note ignores which runtime is active. Also: the PR title and commit subjects do not follow Conventional Commits (CONTRIBUTING.md), and the description does not carry the three checklist items from the PR template.


🤖 by agenthub on AMD AgentHub

Comment thread engines/lemonade/src/footprint.rs Outdated
if !extraction && !backend_missing {
return None;
}
let other_install = recorded_runtime_dir(paths)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: recorded_runtime_dir returns the recorded folder even when it no longer holds an install (holds_install: false), so other_install is set for a dangling record and the notice at line 124 says "Lemonade is already installed at ; that copy is not reused ... rocm storage lists both". In that state nothing is installed there and rocm storage lists nothing for it, which is the state the storage report handles separately. Records left behind by runtime uninstalls from before this PR would trigger it. Suggest building other_install from install_record() and only for Recorded { holds_install: true, .. }.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 551d9cb. other_install is now built only from a record whose folder still holds an install, so a dangling record (including one left by a runtime uninstalled before this PR) no longer produces the "already installed" line; a_record_naming_a_folder_with_no_install_is_not_announced_as_a_copy pins it and goes red if the check is dropped. I also removed "rocm storage lists both" from the notice, since storage cannot list a copy outside the data folder or a registered runtime; it now says "See rocm storage."

Comment thread engines/lemonade/src/footprint.rs Outdated
let Some(available) = self.available else {
return SpaceVerdict::Fits;
};
let download = rocm_core::with_margin(FRESH_INSTALL_DOWNLOAD_BYTES);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the hard refusal uses the ROCm backend download (4.7 GB), but the backend is only chosen after this preflight, and backend_alignment.rs falls back to vulkan on hosts where Lemonade's ROCm build is unsupported. A host like that with roughly 1-4.9 GiB free would be refused for a download it does not need, and sees a notice naming the ROCm backend. The doc comment on the constant already says the figure varies with the GPU family. Consider warning instead of refusing here, or wording the notice as an upper bound.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: the backend is chosen after the preflight and the list is ["rocm", "vulkan"] (llamacpp_backend.rs), so a Vulkan-fallback host could be refused for a download it does not need. The notice now says "up to about 4.4 GiB ... up to about 25 GiB" and no longer names the ROCm backend as certain. Refusal happens only below 16 MiB, the smallest install that can work (the 4.8 MB archive plus what extracts outside bin, measured on a real tree); below the ROCm download or the finished tree it warns and continues, and below the download the warning says a smaller backend may still fit. Scenarios 01 and 02 now report 1 MiB free; the warning range is unit-tested only, because a scenario would have to start the download.

Comment thread apps/rocm/src/runtime_uninstall.rs Outdated
RecordCleanup::Unreadable(error) => Some(format!(
"Lemonade's install record could not be read ({error}), so it was not \
checked against this runtime; if Lemonade stops working, run \
`rocm engines install lemonade --reinstall`"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: for an unreadable record the advice is rocm engines install lemonade --reinstall, which wipes and re-extracts the ~25 GiB tree. A plain rocm engines install lemonade already rewrites the record (prepare_embeddable returns the manifest and lib.rs calls write_manifest) and keeps a current tree, so the plain command should clear the condition for a fraction of the cost. Also, this line prints on every runtime uninstall while the record is unreadable, even when the removed runtime held no Lemonade. Suggest naming the plain command, and a test that it really replaces a corrupt record (AGENTS.md section 3 asks that advice naming a command be shown to clear the condition).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified in the code: the install path writes the record and never reads it, so a plain rocm engines install lemonade replaces a corrupt record and keeps the current tree. The advice now names the plain command, and a_plain_install_replaces_a_corrupt_record_and_keeps_the_current_tree runs the real sequence over a corrupt record and asserts a readable record naming an install that is there with the tree untouched (red if the record is not written, or if a reinstall is forced). The line also now prints only when the removed runtime's folder held a Lemonade tree, with a test for an unrelated runtime.

Comment thread apps/rocm/src/storage.rs Outdated
if usage.status == EngineInstallStatus::Unused {
let _ = writeln!(
output,
" note: reclaimable; nothing removes it automatically. Delete its runtime, downloads and extract folders to free the space"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Unused is decided only from the install record, not from which runtime is active. With two runtimes that each hold a Lemonade copy, the one the record does not point at is called "not used ... reclaimable", but activating that runtime and serving would reuse it, so following the advice can force the ~25 GiB reinstall this report is meant to avoid. Suggest wording it as "not used right now" (and that it is reused if its runtime becomes active), or not marking a copy inside a registered runtime as reclaimable.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, a copy inside a registered runtime would be reused if that runtime became active. The status now reads "not used right now; Lemonade runs from another install". Only the data-folder copy is called reclaimable, and its note says it is only used while no ROCm install is active; a copy inside a registered runtime is never offered for deletion and says it is reused if that ROCm install becomes the active one. a_copy_inside_another_registered_runtime_is_not_offered_for_deletion covers two registered runtimes plus a record, and scenario 03 asserts the new wording.

@volen-silo volen-silo changed the title ROCMAI-825: announce Lemonade installs, show their progress, and report them in storage feat(lemonade): announce installs, show their progress, and report them in storage Nov 9, 2026
@volen-silo
volen-silo force-pushed the rocmai-825-lemonade-per-runtime branch from 11ce27f to 551d9cb Compare November 9, 2026 12:21
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Pushed 551d9cb (two commits, rebased onto main). On the review:

  • CI (blocking): dash-43 is an intermittent 30 s wait in code this PR does not touch. It failed once in 177 runs of that job since the scenario was added (the other job failure there is a different test), and the plain E2E tests job failed twice on dash-gen-tps-expiry-boundary, which shares the same wait in dash_steps.rs; the evidence is in the description. I did not change dash code here, and the rebase restarted the run; the required checks are running on this head.
  • The four non-blocking points are fixed (threads): dangling record, refusal threshold, unreadable-record advice with a test that the plain command clears it, and the "not used right now" wording.
  • Conventions: the commits and the PR title now use Conventional Commits (feat(lemonade): ..., feat(storage): ...), and the description carries the four template checklist items with answers.
  • The description also has a table of every new or changed test with the break applied and whether it went red.

@volen-silo
volen-silo force-pushed the rocmai-825-lemonade-per-runtime branch from 551d9cb to 7620d9b Compare November 9, 2026 12:48
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Pushed 7620d9b (rebased once more onto main: #912 changed the install-manifest path helper in install.rs, and the conflict was in that one function). All 29 checks that ran on this head pass, including E2E tests (ubuntu-26.04), where dash-43 passed this time, and the required E2E tests job, where lemonade-installs-01 to 06 ran. The blocking CI item is therefore resolved; the non-blocking points are addressed as described in the previous comment.

@volen-silo
volen-silo dismissed jussielo-amd’s stale review November 9, 2026 13:02

All the points are addressed in 551d9cb and rebased as 7620d9b: the blocking CI item (an intermittent dash-43 timeout, with evidence in the description) passes on this head, and the four non-blocking points, the Conventional Commits titles and the PR template items are done. See the replies and the summary comments. A re-review is welcome.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the whole diff. Nothing here looks like a correctness break, but there are a few things I'd like fixed before merge, mostly around the free-space warnings saying something different from the install notice.

  1. The notice printed first says the ROCm backend is "up to about 4.4 GiB" and "25.0 GiB of disk space", but the two Tight warnings print the margin-inflated values (~4.8 GiB / ~26.x GiB). The same output shows two different numbers, and the unit test pins the inflated string. Print the unmargined constants and keep the margin for the comparison only. (inline comments)
  2. tree_has_install is exported from the engine crate but nothing in this PR calls it (storage uses InstallRecord::holds_install). Remove it or use it. (inline)
  3. --reinstall is space-checked as if the existing tree were not about to be wiped, so it can warn spuriously. Low impact since it only warns, but worth a sentence in the warning text or netting out the existing tree.
  4. The notice says "See rocm storage" for the copy that isn't reused, but storage only lists trees under the data folder and registered runtimes. A recorded install elsewhere would be announced but not listed.

For awareness: the Windows call site in run_lemonade_backend_install that passes the hide function has no test (the PR says so), and the progress fix rests on a manual probe. The E2E tests lanes that run the new lemonade-installs scenarios were still pending when I looked, so those are not yet evidence. I did not run the tests locally, so the break-and-confirm-red table in the PR body is unverified by me.

The refuse-below-16-MiB / warn-above tiering reads as a reasonable deliberate tradeoff.

Comment thread engines/lemonade/src/footprint.rs Outdated
let download = rocm_core::with_margin(FRESH_INSTALL_DOWNLOAD_BYTES);
if available < download {
return SpaceVerdict::Tight(format!(
"Warning: only {} is free on the drive holding {}, and Lemonade's llama.cpp ROCm backend alone is a download of up to about {}. The install will continue: if Lemonade picks a smaller backend for this GPU it may fit, otherwise it will fail partway through.",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The install notice says the ROCm backend is "up to about 4.4 GiB", but this warning prints the margin-inflated figure (with_margin adds a floor/proportional margin), so the same output shows two different numbers. Print FRESH_INSTALL_DOWNLOAD_BYTES here and keep the margin for the comparison only.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4c8edc8. The notice and both warnings now print the unmargined figures (4.4 GiB / 25.0 GiB); the margin is used only for the comparison. All three strings come from the same helpers, and the_notice_and_the_warnings_print_the_same_download_and_disk_figures fails if any of them shows a different number.

Comment thread engines/lemonade/src/footprint.rs Outdated
let disk = rocm_core::with_margin(FRESH_INSTALL_DISK_BYTES);
if available < disk {
return SpaceVerdict::Tight(format!(
"Warning: free disk space may be insufficient to install Lemonade's llama.cpp ROCm backend: up to about {} needed, {} available on the drive holding {}. The install will continue, but may fail partway through.",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: the notice says 25.0 GiB of disk space, this says "up to about 26.x GiB", and the unit test pins "up to about 26". Print FRESH_INSTALL_DISK_BYTES unmargined so the two lines agree, and assert that value in the test.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4c8edc8, same change: the disk warning prints the unmargined 25.0 GiB, and the test asserts the margined values are absent.

Comment thread engines/lemonade/src/footprint.rs Outdated
}

/// Whether `tree_root` holds an extracted Lemonade runtime.
pub fn tree_has_install(tree_root: &Path) -> bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tree_has_install is exported from the crate but nothing in this PR calls it (storage uses InstallRecord::holds_install). Remove it or use it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d655afb: tree_has_install and its re-export are removed. Nothing called it; storage uses InstallRecord::holds_install.

… its progress on Windows

ROCMAI-825. A fresh Lemonade install with its llama.cpp ROCm backend is
a download of up to ~4.7 GB and a tree of up to ~25 GiB, and
`rocm serve` starts one on its own whenever the install target has no
current tree, for example the active runtime's own folder after
`rocm install sdk`. It used to print one "Preparing..." line and then
nothing for ten minutes.

Before writing a new tree, the engine now prints where it installs,
the ROCm backend's download and disk use as upper bounds, the free
space on that drive, and any existing Lemonade install it is not
reusing (never a record whose folder no longer holds an install). The
backend is only chosen after this point, and Lemonade falls back to a
smaller one where its ROCm build is unsupported, so the figures are
"up to" and only warn. The one thing that refuses, before anything is
downloaded, is a drive that cannot hold Lemonade itself (its archive
and extracted files, about 16 MiB), the one figure that holds for
every backend.

The same figures are printed in the notice and in both warnings (the
margin is used for the comparison only), and an install that replaces
the existing tree (`--reinstall`, a new Lemonade version) deletes it
before downloading, so its size is counted as free and the notice says
so.

`lemonade backends install` inherited the console but was started with
CREATE_NO_WINDOW. A child started that way writes nothing to a
console it inherited: from a real console, `cmd /c echo`, a Python
child and `lemonade.exe --version` all printed without the flag and
printed nothing with it. So the download progress vanished from an
interactive console while still reaching a redirected log. The flag
is now applied only when neither stdout nor stderr is a console.

The window decision lives in the command builder
(`backend_install_command`), which takes the console state and the hide
step, so a unit test drives the real command for each console state and
fails if the hide is dropped, unconditional, or ignores a stream.

Scenarios lemonade-installs-01 (`rocm engines install lemonade`) and -02
(`rocm serve`) run on the mock lane with the e2e-test-hooks build. They
report 1 MiB free through a test-only override, which also waives
serve's no-GPU refusal as serve-18's failure hook does, and check
against the disk that nothing was downloaded. The warning range is
covered by unit tests, since a scenario would have to start the download.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
…tall record

ROCMAI-825. Lemonade installs into the active runtime's folder when one
is active and into the data folder otherwise, so installing Lemonade
before the first SDK and serving after it leaves two full copies (24.3
GiB each on the release-test host). `rocm storage report` counted both
but named neither: one hid inside a runtime's size, the other inside
the data folder total, and once the record moved to the runtime's copy
the data folder's copy was used by nothing and never cleaned up.

The report now has an `Engine installs` section listing each Lemonade
tree with its size (the install parts only, not the record, logs and
state that share the data folder's tree). Reporting only; nothing is
deleted. A copy is `in use` when the install record points into it. A
copy is `not used right now` when the record points at another copy
that really holds an install, which is not a promise it stays unused:
Lemonade installs into the active runtime's folder, so a copy inside a
registered runtime is reused as soon as that runtime becomes active
and is never marked reclaimable. Only the data folder's copy, which no
runtime owns, is. A record that names a folder with no install, or that
cannot be read, leaves it unknown which copy works, so no copy is
called unused: following the delete advice there would have removed
the only working tree and forced a ~25 GiB reinstall. A copy inside an
external or imported runtime is not said to be removed with it, using
the same rule uninstall applies.

`rocm runtimes uninstall` (and `storage remove-old-installs`, which
delegates to it) deletes the runtime folder, Lemonade's copy included,
but left Lemonade's install record naming the deleted files. The plan
now says Lemonade goes with the runtime, and once the folder is gone
the record is cleared and the result says so. A record that could not
be deleted is reported with the file to delete. A record that could
not be read is reported only when the removed runtime held a Lemonade
tree, and names a plain `rocm engines install lemonade`, which writes
the record without reading it and keeps a current tree (shown by a test
that runs the real record-writing sequence over a corrupt record), not
the wiping `--reinstall`.

The install notice points at `rocm storage` for a copy that is not
reused only when storage lists it (the data folder's tree or a tree
inside a registered ROCm install), so a recorded install anywhere else
is announced without a pointer that would be false for it.

Scenarios lemonade-installs-03 (both copies listed, unused one flagged),
-04 (uninstall clears the record), -05 (dangling record keeps the
survivor off the reclaimable list) and -06 (a read-only runtime's copy
survives uninstall and is not said to be removed), all on the mock lane.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I found nothing blocking: the notice, the console-progress fix, the free-space check and the per-runtime listing in rocm storage match what the ticket asked for, the new scenarios assert state as well as wording, docs are in sync, and CI is green on the current head.

A few non-blocking points, three of them inline:

  • registered_install_roots re-parses the registry on its own instead of using the shared manifest loader, so a layout change would quietly drop the "See rocm storage" pointer.
  • directory_bytes walks the existing runtime tree synchronously before the first line prints.
  • The Windows console fix is covered only manually at the real call site (the PR text says so).
  • Cosmetic: "Installing Lemonade" is immediately followed by the older "Preparing Lemonade embeddable" line, and two helpers in test_support.rs sit after mod tests.

This is based on reading the diff and the CI results; I did not run it locally.

/// The install roots of the ROCm installs registered with ROCm CLI, read from
/// the runtime registry the same way storage finds its runtimes. Best effort: an
/// unreadable entry is skipped.
fn registered_install_roots(paths: &AppPaths) -> Vec<PathBuf> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This re-reads the runtime registry as untyped JSON with a hard-coded runtimes/registry path, while storage and uninstall go through load_runtime_manifests. If the registry layout or location changes, storage_lists silently returns false and the "See rocm storage" pointer disappears with no test failing. Consider having the CLI pass the install roots in, or sharing one path helper from rocm-core.

/// Total size of the files under `dir`, best effort: symlinks are not followed
/// and anything unreadable counts as zero, because this only sizes the space an
/// install is about to free.
fn directory_bytes(dir: &Path) -> u64 {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

directory_bytes is a synchronous full walk of the existing runtime tree (up to ~25 GiB of files) before any notice is printed, on exactly the path meant to stop being silent. It only feeds the "replaced first, counted as free" figure. Consider printing the notice first or bounding the walk.

.with_context(|| format!("failed to run {}", manifest.lemonade.display()))?;
if !status.success() {
bail!("Lemonade backend install failed with status {status}");
if consoles.child_may_hide_its_console() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call site (OutputConsoles::detect() plus hide_child_console_window) is the part of the Windows fix no automated test exercises; a no-op hide step here would stay green. The PR text says so; flagging so it stays on the manual-test list for each release.

@volen-silo
volen-silo added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit 5e3b313 Nov 9, 2026
30 checks passed
@volen-silo
volen-silo deleted the rocmai-825-lemonade-per-runtime branch November 9, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants