Repository navigation
feat(lemonade): announce installs, show their progress, and report them in storage - #853
Conversation
0d66d97 to
9108da7
Compare
9108da7 to
45152ec
Compare
45152ec to
75fbd1f
Compare
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for this. The announce-before-install, free-space check and per-runtime storage listing all look right, and the scenarios assert both the message and the resulting state. I'm requesting changes on two storage-report claims that can be wrong; both are inline.
Requested changes
- A dangling install record makes every surviving Lemonade copy read "not used / reclaimable" (inline on
storage.rs). - The "removed with it" note is printed for every per-runtime row, including read-only/imported runtimes where it is false (inline on
storage.rs).
Smaller notes (non-blocking)
- The Windows console-progress fix is covered only by a pure-helper unit test. The user-visible claim rests on the manual check you disclosed, which is fine, but a swapped stdout/stderr argument or a dropped call would not turn that test red.
- If clearing the install record fails after the folder is deleted, the only signal is a stderr warning. Consider surfacing it in the uninstall result so the record isn't left dangling silently. This ties into item 1, since an unreadable record is treated as "not recorded".
- The serve-time announcement has no scenario of its own (only the
engines installpath is covered). Low risk since both go through the same install path, but the PR text says serve announces. - Please state in the PR text that the new tests were shown to fail with their target broken (AGENTS section 3).
- Nits:
footprint.rsis out of alphabetical order indocs/architecture.md, and a few comment/doc lines inruntime_uninstall.rsanddocs/testing.mdare very long.
| complete && measured.complete, | ||
| ) | ||
| }); | ||
| let status = match recorded.as_deref() { |
There was a problem hiding this comment.
This marks the copy Unused whenever any record exists elsewhere. If the recorded runtime_dir no longer has an install (e.g. a runtime uninstalled before this change), nothing is actually in use, yet every surviving tree is reported "not used" and "reclaimable", and a later serve reuses the data-folder tree. Following the delete advice would force a ~25 GiB reinstall. Suggest treating a record whose runtime has no install (tree_has_install) as NotRecorded, or giving it a distinct "record is dangling" state, and adding a test for it.
There was a problem hiding this comment.
Agreed, and fixed in 11ce27f. Lemonade's install record is now read as three states (no record; an unreadable record; a record naming a folder that does or does not still hold the server binary), and storage report has five statuses. Only not used is called reclaimable, and only when the record names a different copy that really holds an install. A record naming a folder with no install reads "not known to be used; the install record names a folder with no install", and an unreadable one reads "... could not be read"; neither is marked reclaimable, so the report can no longer advise deleting the only working tree. Tests: a_record_naming_a_folder_with_no_install_does_not_make_the_survivor_reclaimable, an_unreadable_record_leaves_every_copy_unmarked, engine_install_status_follows_the_record, and scenario lemonade-installs-05; each goes red when its target is broken (table in the description).
| ); | ||
| } | ||
| if usage.runtime_key.is_some() { | ||
| let _ = writeln!( |
There was a problem hiding this comment.
This note says the tree is "removed with" the ROCm install unconditionally. For a read-only/imported runtime, uninstall only unregisters it, so the claim is false; the uninstall plan already guards on will_remove_install_root(). Please derive the note from the same decision (or soften it) and cover a read-only runtime row with a test.
There was a problem hiding this comment.
Right, that claim was false for a read-only or imported runtime. The note now follows the same rule rocm runtimes uninstall applies (should_remove_runtime_install_root): for a read-only runtime the row says ROCm CLI does not delete that install's folder, so uninstalling it leaves the copy. A unit test covers a read-only row and also runs the real uninstall to check the copy survives, and scenario lemonade-installs-06 covers it end to end.
75fbd1f to
11ce27f
Compare
|
Pushed 11ce27f (two commits, rebased onto main). Both requested changes are done (threads), and the smaller notes:
Scenarios are now |
Both requested changes are addressed in 11ce27f (a dangling or unreadable install record no longer marks any copy reclaimable, and the "removed with it" note follows the uninstall plan's decision), along with the smaller notes; see the replies, the summary comment and the updated description. A re-review is welcome.
jussielo-amd
left a comment
There was a problem hiding this comment.
Thanks for this. The announcement before install, the free-space check and the per-runtime storage listing all read well, and the scenarios assert both the message and the resulting state. The five storage statuses, the refuse/warn thresholds and the record clearing match the description. This was a static review; I did not build or run the tests.
Blocking: CI is red on this head. In the CI run for 11ce27f, E2E tests (ubuntu-26.04) failed on dash-43 (the 30 s wait for the dashboard warnings badge timed out). Nothing in this diff touches the dashboard, the plain E2E tests job on the same commit is green, and all six lemonade-installs scenarios passed in the failing job, so I expect a flake. A red check still stops the merge, though. Please rerun it; the PR also conflicts with main now, so the rebase will start a fresh run anyway. If it fails the same way again it is worth a look.
Non-blocking, inline: a dangling install record produces a false "already installed" line; the hard refusal assumes the ROCm backend download; the unreadable-record advice names the heavier --reinstall; the "reclaimable" note ignores which runtime is active. Also: the PR title and commit subjects do not follow Conventional Commits (CONTRIBUTING.md), and the description does not carry the three checklist items from the PR template.
🤖 by agenthub on AMD AgentHub
| if !extraction && !backend_missing { | ||
| return None; | ||
| } | ||
| let other_install = recorded_runtime_dir(paths) |
There was a problem hiding this comment.
Non-blocking: recorded_runtime_dir returns the recorded folder even when it no longer holds an install (holds_install: false), so other_install is set for a dangling record and the notice at line 124 says "Lemonade is already installed at ; that copy is not reused ... rocm storage lists both". In that state nothing is installed there and rocm storage lists nothing for it, which is the state the storage report handles separately. Records left behind by runtime uninstalls from before this PR would trigger it. Suggest building other_install from install_record() and only for Recorded { holds_install: true, .. }.
There was a problem hiding this comment.
Agreed, fixed in 551d9cb. other_install is now built only from a record whose folder still holds an install, so a dangling record (including one left by a runtime uninstalled before this PR) no longer produces the "already installed" line; a_record_naming_a_folder_with_no_install_is_not_announced_as_a_copy pins it and goes red if the check is dropped. I also removed "rocm storage lists both" from the notice, since storage cannot list a copy outside the data folder or a registered runtime; it now says "See rocm storage."
| let Some(available) = self.available else { | ||
| return SpaceVerdict::Fits; | ||
| }; | ||
| let download = rocm_core::with_margin(FRESH_INSTALL_DOWNLOAD_BYTES); |
There was a problem hiding this comment.
Non-blocking: the hard refusal uses the ROCm backend download (4.7 GB), but the backend is only chosen after this preflight, and backend_alignment.rs falls back to vulkan on hosts where Lemonade's ROCm build is unsupported. A host like that with roughly 1-4.9 GiB free would be refused for a download it does not need, and sees a notice naming the ROCm backend. The doc comment on the constant already says the figure varies with the GPU family. Consider warning instead of refusing here, or wording the notice as an upper bound.
There was a problem hiding this comment.
Verified: the backend is chosen after the preflight and the list is ["rocm", "vulkan"] (llamacpp_backend.rs), so a Vulkan-fallback host could be refused for a download it does not need. The notice now says "up to about 4.4 GiB ... up to about 25 GiB" and no longer names the ROCm backend as certain. Refusal happens only below 16 MiB, the smallest install that can work (the 4.8 MB archive plus what extracts outside bin, measured on a real tree); below the ROCm download or the finished tree it warns and continues, and below the download the warning says a smaller backend may still fit. Scenarios 01 and 02 now report 1 MiB free; the warning range is unit-tested only, because a scenario would have to start the download.
| RecordCleanup::Unreadable(error) => Some(format!( | ||
| "Lemonade's install record could not be read ({error}), so it was not \ | ||
| checked against this runtime; if Lemonade stops working, run \ | ||
| `rocm engines install lemonade --reinstall`" |
There was a problem hiding this comment.
Non-blocking: for an unreadable record the advice is rocm engines install lemonade --reinstall, which wipes and re-extracts the ~25 GiB tree. A plain rocm engines install lemonade already rewrites the record (prepare_embeddable returns the manifest and lib.rs calls write_manifest) and keeps a current tree, so the plain command should clear the condition for a fraction of the cost. Also, this line prints on every runtime uninstall while the record is unreadable, even when the removed runtime held no Lemonade. Suggest naming the plain command, and a test that it really replaces a corrupt record (AGENTS.md section 3 asks that advice naming a command be shown to clear the condition).
There was a problem hiding this comment.
Verified in the code: the install path writes the record and never reads it, so a plain rocm engines install lemonade replaces a corrupt record and keeps the current tree. The advice now names the plain command, and a_plain_install_replaces_a_corrupt_record_and_keeps_the_current_tree runs the real sequence over a corrupt record and asserts a readable record naming an install that is there with the tree untouched (red if the record is not written, or if a reinstall is forced). The line also now prints only when the removed runtime's folder held a Lemonade tree, with a test for an unrelated runtime.
| if usage.status == EngineInstallStatus::Unused { | ||
| let _ = writeln!( | ||
| output, | ||
| " note: reclaimable; nothing removes it automatically. Delete its runtime, downloads and extract folders to free the space" |
There was a problem hiding this comment.
Non-blocking: Unused is decided only from the install record, not from which runtime is active. With two runtimes that each hold a Lemonade copy, the one the record does not point at is called "not used ... reclaimable", but activating that runtime and serving would reuse it, so following the advice can force the ~25 GiB reinstall this report is meant to avoid. Suggest wording it as "not used right now" (and that it is reused if its runtime becomes active), or not marking a copy inside a registered runtime as reclaimable.
There was a problem hiding this comment.
Right, a copy inside a registered runtime would be reused if that runtime became active. The status now reads "not used right now; Lemonade runs from another install". Only the data-folder copy is called reclaimable, and its note says it is only used while no ROCm install is active; a copy inside a registered runtime is never offered for deletion and says it is reused if that ROCm install becomes the active one. a_copy_inside_another_registered_runtime_is_not_offered_for_deletion covers two registered runtimes plus a record, and scenario 03 asserts the new wording.
11ce27f to
551d9cb
Compare
|
Pushed 551d9cb (two commits, rebased onto main). On the review:
|
551d9cb to
7620d9b
Compare
|
Pushed 7620d9b (rebased once more onto main: #912 changed the install-manifest path helper in |
All the points are addressed in 551d9cb and rebased as 7620d9b: the blocking CI item (an intermittent dash-43 timeout, with evidence in the description) passes on this head, and the four non-blocking points, the Conventional Commits titles and the PR template items are done. See the replies and the summary comments. A re-review is welcome.
juhovainio
left a comment
There was a problem hiding this comment.
I went through the whole diff. Nothing here looks like a correctness break, but there are a few things I'd like fixed before merge, mostly around the free-space warnings saying something different from the install notice.
- The notice printed first says the ROCm backend is "up to about 4.4 GiB" and "25.0 GiB of disk space", but the two Tight warnings print the margin-inflated values (~4.8 GiB / ~26.x GiB). The same output shows two different numbers, and the unit test pins the inflated string. Print the unmargined constants and keep the margin for the comparison only. (inline comments)
tree_has_installis exported from the engine crate but nothing in this PR calls it (storage usesInstallRecord::holds_install). Remove it or use it. (inline)--reinstallis space-checked as if the existing tree were not about to be wiped, so it can warn spuriously. Low impact since it only warns, but worth a sentence in the warning text or netting out the existing tree.- The notice says "See
rocm storage" for the copy that isn't reused, but storage only lists trees under the data folder and registered runtimes. A recorded install elsewhere would be announced but not listed.
For awareness: the Windows call site in run_lemonade_backend_install that passes the hide function has no test (the PR says so), and the progress fix rests on a manual probe. The E2E tests lanes that run the new lemonade-installs scenarios were still pending when I looked, so those are not yet evidence. I did not run the tests locally, so the break-and-confirm-red table in the PR body is unverified by me.
The refuse-below-16-MiB / warn-above tiering reads as a reasonable deliberate tradeoff.
| let download = rocm_core::with_margin(FRESH_INSTALL_DOWNLOAD_BYTES); | ||
| if available < download { | ||
| return SpaceVerdict::Tight(format!( | ||
| "Warning: only {} is free on the drive holding {}, and Lemonade's llama.cpp ROCm backend alone is a download of up to about {}. The install will continue: if Lemonade picks a smaller backend for this GPU it may fit, otherwise it will fail partway through.", |
There was a problem hiding this comment.
The install notice says the ROCm backend is "up to about 4.4 GiB", but this warning prints the margin-inflated figure (with_margin adds a floor/proportional margin), so the same output shows two different numbers. Print FRESH_INSTALL_DOWNLOAD_BYTES here and keep the margin for the comparison only.
There was a problem hiding this comment.
Fixed in 4c8edc8. The notice and both warnings now print the unmargined figures (4.4 GiB / 25.0 GiB); the margin is used only for the comparison. All three strings come from the same helpers, and the_notice_and_the_warnings_print_the_same_download_and_disk_figures fails if any of them shows a different number.
| let disk = rocm_core::with_margin(FRESH_INSTALL_DISK_BYTES); | ||
| if available < disk { | ||
| return SpaceVerdict::Tight(format!( | ||
| "Warning: free disk space may be insufficient to install Lemonade's llama.cpp ROCm backend: up to about {} needed, {} available on the drive holding {}. The install will continue, but may fail partway through.", |
There was a problem hiding this comment.
Same as above: the notice says 25.0 GiB of disk space, this says "up to about 26.x GiB", and the unit test pins "up to about 26". Print FRESH_INSTALL_DISK_BYTES unmargined so the two lines agree, and assert that value in the test.
There was a problem hiding this comment.
Fixed in 4c8edc8, same change: the disk warning prints the unmargined 25.0 GiB, and the test asserts the margined values are absent.
| } | ||
|
|
||
| /// Whether `tree_root` holds an extracted Lemonade runtime. | ||
| pub fn tree_has_install(tree_root: &Path) -> bool { |
There was a problem hiding this comment.
tree_has_install is exported from the crate but nothing in this PR calls it (storage uses InstallRecord::holds_install). Remove it or use it.
There was a problem hiding this comment.
Fixed in d655afb: tree_has_install and its re-export are removed. Nothing called it; storage uses InstallRecord::holds_install.
… its progress on Windows ROCMAI-825. A fresh Lemonade install with its llama.cpp ROCm backend is a download of up to ~4.7 GB and a tree of up to ~25 GiB, and `rocm serve` starts one on its own whenever the install target has no current tree, for example the active runtime's own folder after `rocm install sdk`. It used to print one "Preparing..." line and then nothing for ten minutes. Before writing a new tree, the engine now prints where it installs, the ROCm backend's download and disk use as upper bounds, the free space on that drive, and any existing Lemonade install it is not reusing (never a record whose folder no longer holds an install). The backend is only chosen after this point, and Lemonade falls back to a smaller one where its ROCm build is unsupported, so the figures are "up to" and only warn. The one thing that refuses, before anything is downloaded, is a drive that cannot hold Lemonade itself (its archive and extracted files, about 16 MiB), the one figure that holds for every backend. The same figures are printed in the notice and in both warnings (the margin is used for the comparison only), and an install that replaces the existing tree (`--reinstall`, a new Lemonade version) deletes it before downloading, so its size is counted as free and the notice says so. `lemonade backends install` inherited the console but was started with CREATE_NO_WINDOW. A child started that way writes nothing to a console it inherited: from a real console, `cmd /c echo`, a Python child and `lemonade.exe --version` all printed without the flag and printed nothing with it. So the download progress vanished from an interactive console while still reaching a redirected log. The flag is now applied only when neither stdout nor stderr is a console. The window decision lives in the command builder (`backend_install_command`), which takes the console state and the hide step, so a unit test drives the real command for each console state and fails if the hide is dropped, unconditional, or ignores a stream. Scenarios lemonade-installs-01 (`rocm engines install lemonade`) and -02 (`rocm serve`) run on the mock lane with the e2e-test-hooks build. They report 1 MiB free through a test-only override, which also waives serve's no-GPU refusal as serve-18's failure hook does, and check against the disk that nothing was downloaded. The warning range is covered by unit tests, since a scenario would have to start the download. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
…tall record ROCMAI-825. Lemonade installs into the active runtime's folder when one is active and into the data folder otherwise, so installing Lemonade before the first SDK and serving after it leaves two full copies (24.3 GiB each on the release-test host). `rocm storage report` counted both but named neither: one hid inside a runtime's size, the other inside the data folder total, and once the record moved to the runtime's copy the data folder's copy was used by nothing and never cleaned up. The report now has an `Engine installs` section listing each Lemonade tree with its size (the install parts only, not the record, logs and state that share the data folder's tree). Reporting only; nothing is deleted. A copy is `in use` when the install record points into it. A copy is `not used right now` when the record points at another copy that really holds an install, which is not a promise it stays unused: Lemonade installs into the active runtime's folder, so a copy inside a registered runtime is reused as soon as that runtime becomes active and is never marked reclaimable. Only the data folder's copy, which no runtime owns, is. A record that names a folder with no install, or that cannot be read, leaves it unknown which copy works, so no copy is called unused: following the delete advice there would have removed the only working tree and forced a ~25 GiB reinstall. A copy inside an external or imported runtime is not said to be removed with it, using the same rule uninstall applies. `rocm runtimes uninstall` (and `storage remove-old-installs`, which delegates to it) deletes the runtime folder, Lemonade's copy included, but left Lemonade's install record naming the deleted files. The plan now says Lemonade goes with the runtime, and once the folder is gone the record is cleared and the result says so. A record that could not be deleted is reported with the file to delete. A record that could not be read is reported only when the removed runtime held a Lemonade tree, and names a plain `rocm engines install lemonade`, which writes the record without reading it and keeps a current tree (shown by a test that runs the real record-writing sequence over a corrupt record), not the wiping `--reinstall`. The install notice points at `rocm storage` for a copy that is not reused only when storage lists it (the data folder's tree or a tree inside a registered ROCm install), so a recorded install anywhere else is announced without a pointer that would be false for it. Scenarios lemonade-installs-03 (both copies listed, unused one flagged), -04 (uninstall clears the record), -05 (dangling record keeps the survivor off the reclaimable list) and -06 (a read-only runtime's copy survives uninstall and is not said to be removed), all on the mock lane. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
7620d9b to
d655afb
Compare
juhovainio
left a comment
There was a problem hiding this comment.
Looks good to me. I found nothing blocking: the notice, the console-progress fix, the free-space check and the per-runtime listing in rocm storage match what the ticket asked for, the new scenarios assert state as well as wording, docs are in sync, and CI is green on the current head.
A few non-blocking points, three of them inline:
registered_install_rootsre-parses the registry on its own instead of using the shared manifest loader, so a layout change would quietly drop the "Seerocm storage" pointer.directory_byteswalks the existing runtime tree synchronously before the first line prints.- The Windows console fix is covered only manually at the real call site (the PR text says so).
- Cosmetic: "Installing Lemonade" is immediately followed by the older "Preparing Lemonade embeddable" line, and two helpers in
test_support.rssit aftermod tests.
This is based on reading the diff and the CI results; I did not run it locally.
| /// The install roots of the ROCm installs registered with ROCm CLI, read from | ||
| /// the runtime registry the same way storage finds its runtimes. Best effort: an | ||
| /// unreadable entry is skipped. | ||
| fn registered_install_roots(paths: &AppPaths) -> Vec<PathBuf> { |
There was a problem hiding this comment.
This re-reads the runtime registry as untyped JSON with a hard-coded runtimes/registry path, while storage and uninstall go through load_runtime_manifests. If the registry layout or location changes, storage_lists silently returns false and the "See rocm storage" pointer disappears with no test failing. Consider having the CLI pass the install roots in, or sharing one path helper from rocm-core.
| /// Total size of the files under `dir`, best effort: symlinks are not followed | ||
| /// and anything unreadable counts as zero, because this only sizes the space an | ||
| /// install is about to free. | ||
| fn directory_bytes(dir: &Path) -> u64 { |
There was a problem hiding this comment.
directory_bytes is a synchronous full walk of the existing runtime tree (up to ~25 GiB of files) before any notice is printed, on exactly the path meant to stop being silent. It only feeds the "replaced first, counted as free" figure. Consider printing the notice first or bounding the walk.
| .with_context(|| format!("failed to run {}", manifest.lemonade.display()))?; | ||
| if !status.success() { | ||
| bail!("Lemonade backend install failed with status {status}"); | ||
| if consoles.child_may_hide_its_console() { |
There was a problem hiding this comment.
This call site (OutputConsoles::detect() plus hide_child_console_window) is the part of the Windows fix no automated test exercises; a no-op hide step here would stay green. The PR text says so; flagging so it stays on the manual-test list for each release.
Problem
On native Windows, after
rocm engines install lemonadeand thenrocm install sdk, the nextrocm servesilently spent about 10 minutes installing a second full Lemonade (~24 GiB) under the new runtime's folder. Nothing said it was happening, no free space was checked, and the first copy was left referenced by nothing. (ROCMAI-825.)Root cause
env_root_for_self_managed_engine(apps/rocm/src/engines_cmd.rs) places Lemonade under the active runtime'sinstall_root/engines, so each new runtime gets its own tree. That placement is unchanged here; a shared tree is tracked separately as ROCMAI-831.run_lemonade_backend_install(engines/lemonade/src/process.rs) started the backend install withCREATE_NO_WINDOW. A probe in a real console window showed that with that flagcmd /c echo, a Python child andlemonade.exe --versionall print nothing; without it all three print. That is why the install was silent.Change
Commit 1: announce installs and show progress
serve,engines install lemonadeand--reinstallprint the target folder, an upper bound on the download ("up to about 4.4 GiB": the smaller backend may need less) and the disk use ("up to about 25.0 GiB"), the free space on that drive, and any existing copy that is not being reused (newengines/lemonade/src/footprint.rs). A copy counts only if the install record points at a folder that still holds an install; a dangling record is not announced as an existing copy. The notice and both space warnings print the same two figures (all derived from the same constants; the safety margin is used only for the comparison). Therocm storagepointer is given only for a copy storage lists (the data folder's tree and trees inside registered ROCm installs). When the install replaces an existing tree (--reinstall, a new Lemonade version, a missing server binary: the runtime folder is wiped first), that tree's size is counted as free for the comparison and the notice says so.bin, measured on a real tree). Below the ROCm backend's download or the finished tree it warns and continues, because the backend is chosen after this preflight andbackend_alignment.rsfalls back fromrocmtovulkan, which needs less. Install-timeserveannounces only; there is no existing consent step for installs during serve.CREATE_NO_WINDOWis now applied only when neither stdout nor stderr is a console, decided insidebackend_install_command(engines/lemonade/src/process.rs), which takes the console state and the hide step, so one test builds the real command for all four console states and asserts both whether the window was hidden and the command's arguments. Lemonade's own progress lines show again. No spinner: it would overwrite those lines.Commit 2: storage and runtime uninstall
rocm storage reportgets an "Engine installs" section listing each Lemonade copy with its size. Lemonade's install record is read as one of three states (no record; an unreadable record; a record naming a folder that does or does not still hold the server binary), which gives these statuses:in use: the record names this copy.not used right now; Lemonade runs from another install: the record names a different copy that really holds an install. Only the data-folder copy is called reclaimable, and its note says it is only used while no ROCm install is active. A copy inside a registered runtime is never offered for deletion: it says it is reused if that ROCm install becomes the active one.not known to be used; the install record names a folder with no install, andnot known to be used; the install record could not be read: neither is marked reclaimable, so the report never advises deleting the only tree that works.not recorded as installed: there is no record.Also in
--json.rocm runtimes uninstallapplies (should_remove_runtime_install_root): for a read-only or imported runtime, uninstall only unregisters it, so the row says ROCm CLI does not delete that install's folder.rocm runtimes uninstallwarns that the runtime's Lemonade goes with it, and afterwards clears Lemonade's install record so the next serve does not trip on a deleted folder. Thelemonade:line says what happened: cleared; deletion failed (it names the record to delete); or unreadable record (it names the plainrocm engines install lemonade, which rewrites the record and keeps a current tree, so--reinstallis not needed). The unreadable-record line prints only when the removed runtime's folder held a Lemonade tree.storage remove-old-installsgets the same throughuninstall_runtime.Docs: README (Disk space, Inference engines),
docs/testing.md,docs/manual-testing.md,docs/architecture.md.PR template checklist
tests/e2e-cucumber/expectations.tomlhas no row naming ROCMAI-825 orlemonade-installs; the only Lemonade rows are the EAI-7423 Linux serve ones, which this change does not touch. Nothing was stale.engines/lemonade/src/footprint.rs; nothing grew inlib.rsbeyond a small seam (install_and_record) and one call site.docs/architecture.mdhas thefootprint.rsentry, and the prose of thestorage.rsandruntime_uninstall.rsentries is updated;cargo xtask check-architecture-docpasses.a_new_location_is_announced...; scenarios 01 and 02 check nothing was downloaded, extracted or recordedrocm storage"a_record_naming_a_folder_with_no_install_is_not_announced_as_a_copyshows it is absent otherwisethe_lemonade_record_line_matches_what_happened(wording);runtime_uninstall_clears...,...reports_a_lemonade_record_it_could_not_readand the unrelated-runtime test (state);a_plain_install_replaces_a_corrupt_record_and_keeps_the_current_treeshows the advised command clears the conditionTests
footprint.rs(announcement, the two free-space tiers, the install-record API and its three states, the plain install replacing a corrupt record),process.rs(the hide-console decision on the real command),storage.rs(the statuses and the read-only note) andruntime_uninstall.rs(record cleanup and its report line).tests/e2e-cucumber/features/lemonade_installs.feature, mock lane (the free-space and install-state hooks exist only in thee2e-test-hooksbuild):lemonade-installs-01(engines installwith no usable space is refused before anything is downloaded),-02(the same throughrocm serve --engine lemonade --managed, so the serve-side announcement is covered; to reach the install on a no-GPU lane the hooks-only no-GPU waiver inserve_cmd.rsalso honours the free-space variable),-03(storage lists both copies and flags the unused one),-04(uninstalling the runtime that held Lemonade clears the record),-05(a record naming a folder with no install does not make the survivor reclaimable),-06(a read-only runtime's row does not say removed with its ROCm install). They passed locally against a hooks build, 6 of 6; scenarios 01 and 02 now report 1 MiB free, because the warning range cannot be a scenario (it would start the download) and is unit-tested only.cargo nextest run -p rocm-engine-lemonade -p rocm(1748 passed), clippy for those crates ande2e-cucumber(including--test e2e),cargo fmt --check,cargo xtask check-crate-edges,cargo xtask check-architecture-doc, lychee on the changed docs: pass.The new tests fail when their target is broken (each break applied alone against the final tree, then reverted):
a_new_location_is_announced_with_the_install_it_does_not_reusea_current_tree_is_not_announced,a_tree_without_a_backend_is_announceda_record_naming_a_folder_with_no_install_is_not_announced_as_a_copyholds_installalways trueonly_space_below_the_smallest_install_refuses_and_less_than_the_rocm_backend_warnsthe_notice_and_the_warnings_print_the_same_download_and_disk_figuresa_reinstall_counts_the_space_of_the_install_it_replacesthe_storage_pointer_is_only_given_for_a_copy_storage_liststhe_record_is_forgotten_only_when_its_tree_was_removeda_record_that_cannot_be_deleted_is_reported_not_swallowedan_unreadable_record_is_distinct_from_no_recorda_record_reports_whether_its_folder_still_holds_an_installholds_installalways truea_plain_install_replaces_a_corrupt_record_and_keeps_the_current_treethe_backend_install_hides_its_window_only_when_no_output_is_a_consolerun_lemonade_backend_installpasses a no-op hide functionremoved_with_runtimealways trueruntime_uninstalltests--reinstall; always report an unreadable record; failed-deletion line claims "no longer recorded"removed_with_runtimealways trueGaps: the call site in
run_lemonade_backend_installpasseshide_child_console_windowand the real console probe, and no test would notice a no-op swapped in there (the creation flag cannot be read back off a built command), so the Windows console claim rests on the manual check indocs/manual-testing.md. Scenarios 01 and 02 were not broken at the refusal threshold itself, because that would start the download; the unit test covers it. The serve-side no-GPU waiver was not mutation-tested, because the development host has a GPU. There is no black-box scenario for the plain-command advice, because that would download.CI
On the previous head the one unexpected failure was
dash-43onE2E tests (ubuntu-26.04): a 30 s wait for the dashboard warnings badge, in code this PR does not touch, while all six lemonade-installs scenarios passed in that job. It is intermittent: that job passed 174 of 177 runs since the scenario was added, and the plainE2E testsjob failed twice ondash-gen-tps-expiry-boundary, which shares the same 30 s wait (dash_steps.rs). I did not change dash code here. On this head all 29 checks that ran pass, includingE2E tests (ubuntu-26.04)and the requiredE2E testsjob, where lemonade-installs-01 to 06 ran. The self-hosted GPU lanes are not required and gave no result for the previous head:E2E tests (MI300X)and(W7900)got no runner for 24 hours, and the Ubuntu lanes failed after about 33 minutes (the lane problem tracked outside this PR).Not in this PR
lemondlaunch paths inprocess.rsuse the same flag and may hide output the same way.