Skip to content
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1088,9 +1088,15 @@ with the ROCm SDK in use, and rewrites the two pins it owns
(`therock.version` and `llamacpp.rocm-stable` in `resources/backend_versions.json`)
to do so. If installing an already-pinned backend fails, it restores those two
pins from the cached Lemonade archive (it does not download), prints a warning,
and installs once more; see
and installs once more. When the backend cannot be aligned to the active ROCm
version, the install keeps the backend Lemonade pins. For 24 hours after that,
an install with the same ROCm version, pin and Lemonade version reuses that
backend without trying the alignment again; if preparing it fails,
the error says the alignment was skipped and why.
`rocm engines install lemonade --reinstall` re-extracts Lemonade without that
record, so it tries the alignment again sooner. See
[Lemonade backend alignment](https://github.com/ROCm/rocm-cli/blob/main/docs/engine-plugins.md#lemonade-backend-alignment-on-engine-install) for the details and for
`ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT`, which turns both off.
`ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT`, which turns all of this off.

Before Lemonade writes a new install, whether from `rocm engines install
lemonade` or the first `rocm serve` that needs it, it prints the folder it is
Expand Down
5 changes: 3 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -658,9 +658,10 @@ The Lemonade engine adapter (`rocm-engine-lemonade`): installs the embeddable Le
| --- | --- | --- |
| [`lib.rs`](../engines/lemonade/src/lib.rs) | Crate root: CLI and envelope dispatch, the service-lifecycle orchestration that calls across the modules, and the crate-root re-exports `apps/rocm` uses. | flat |
| [`main.rs`](../engines/lemonade/src/main.rs) | The `rocm-engine-lemonade` binary's entry point; it only calls `rocm_engine_lemonade::run_cli()`. | flat |
| [`alignment_record.rs`](../engines/lemonade/src/alignment_record.rs) | The record, inside the runtime directory, that aligning the llama.cpp backend to a ROCm version failed: reading, writing and clearing it, whether it still skips the same alignment (same target, pin and Lemonade version, under 24 hours), the line printed when it does, and the error context added when the install it then runs fails. | domain |
| [`archive_cache.rs`](../engines/lemonade/src/archive_cache.rs) | The cached embeddable-archive layer: SHA-256 verification, verified publication into the cache, and the per-archive lock `ensure_cached_archive` returns as an `ArchiveCacheGuard` to hold while using the bytes. | domain |
| [`archive_extract.rs`](../engines/lemonade/src/archive_extract.rs) | The hardened in-process tar.gz and zip extractor for the embeddable archive, and the containment-checked `copy_tree` into the runtime directory. | domain |
| [`backend_alignment.rs`](../engines/lemonade/src/backend_alignment.rs) | Aligning the llama.cpp backend with the ROCm SDK (Tier 1, Tier 2, revert), including the `backend_versions.json` pin and setting ROCm backend directories aside around a forced reinstall. | domain |
| [`backend_alignment.rs`](../engines/lemonade/src/backend_alignment.rs) | Aligning the llama.cpp backend with the ROCm SDK (Tier 1, Tier 2, revert), including the `backend_versions.json` pin, setting ROCm backend directories aside around a forced reinstall, and skipping an alignment [`alignment_record.rs`](../engines/lemonade/src/alignment_record.rs) recorded as failed. | domain |
| [`device_policy.rs`](../engines/lemonade/src/device_policy.rs) | Device policy and GPU selection: the GPU-required check, `--device`/`--gpu` parsing, the GPU ordinals to pin before a launch, the device availability `detect` reports, and the notice that names a non-ROCm GPU backend (Vulkan) as the one serving. | domain |
| [`direct_llama.rs`](../engines/lemonade/src/direct_llama.rs) | Hugging Face checkpoint and GGUF resolution, including a registered model name resolved through the installed Lemonade catalog, which route a name serves by, and the direct serve path that runs a packaged `llama-server` on a GGUF (`serve_direct_llama_server`). | domain |
| [`footprint.rs`](../engines/lemonade/src/footprint.rs) | What a Lemonade install costs on disk: the pre-install notice and free-space check, and the read-only view of Lemonade's install trees and record. | domain |
Expand Down Expand Up @@ -860,7 +861,7 @@ Gherkin `.feature` files under `features/` backed by Rust step functions that dr
| [`tests/e2e/engines_steps.rs`](../tests/e2e-cucumber/tests/e2e/engines_steps.rs) | Steps for [`engine_shell.feature`](../tests/e2e-cucumber/features/engine_shell.feature): `rocm engines shell` under a pseudo-terminal, since the defect shows only in what the terminal renders. | flat |
| [`tests/e2e/examine_steps.rs`](../tests/e2e-cucumber/tests/e2e/examine_steps.rs) | Steps for [`examine.feature`](../tests/e2e-cucumber/features/examine.feature): `rocm examine`, the active runtime's provenance, version and help output, engine listing and the driver plan preview. | flat |
| [`tests/e2e/host_guard.rs`](../tests/e2e-cucumber/tests/e2e/host_guard.rs) | The host check the Windows driver-install steps run before any CLI call, kept free of the cucumber world so a separate test can include it. | flat |
| [`tests/e2e/lemonade_install_steps.rs`](../tests/e2e-cucumber/tests/e2e/lemonade_install_steps.rs) | Steps for [`lemonade_installs.feature`](../tests/e2e-cucumber/features/lemonade_installs.feature): what the user is told about the Lemonade copies on disk, from planted trees and install records. | flat |
| [`tests/e2e/lemonade_install_steps.rs`](../tests/e2e-cucumber/tests/e2e/lemonade_install_steps.rs) | Steps for [`lemonade_installs.feature`](../tests/e2e-cucumber/features/lemonade_installs.feature): what the user is told about the Lemonade copies on disk, from planted trees and install records; also the steps of runtime-33 in [`runtime_setup.feature`](../tests/e2e-cucumber/features/runtime_setup.feature), where a Lemonade backend that could not be aligned is not tried again by the next install. | flat |
| [`tests/e2e/lifecycle_steps.rs`](../tests/e2e-cucumber/tests/e2e/lifecycle_steps.rs) | Steps for [`install_lifecycle.feature`](../tests/e2e-cucumber/features/install_lifecycle.feature): `xtask package`, the root installer and the installed binaries, from install through uninstall. | flat |
| [`tests/e2e/logs_steps.rs`](../tests/e2e-cucumber/tests/e2e/logs_steps.rs) | Steps for [`logs.feature`](../tests/e2e-cucumber/features/logs.feature): `rocm logs` searches over command-log files planted inside the tail window it reads. | flat |
| [`tests/e2e/model_catalog_steps.rs`](../tests/e2e-cucumber/tests/e2e/model_catalog_steps.rs) | Steps for [`model_catalog.feature`](../tests/e2e-cucumber/features/model_catalog.feature): the verbose catalog's AIM entry, its Docker-unavailable flag and its staleness warning. | flat |
Expand Down
37 changes: 37 additions & 0 deletions docs/engine-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,43 @@ was pinned before the attempt — the packaged pin on a fresh install or after
`--reinstall`, but whatever alignment last wrote otherwise — rather than
being kept.

A failed alignment is remembered so that it is not repeated on every install.
The revert puts back a pin that still differs from the active ROCm version, so
without that every later `rocm engines install lemonade`, and every `rocm serve`
that installs Lemonade, would download both builds and force-reinstall the
pinned backend again. When the revert's install succeeds, the attempt (the
active ROCm version, the pins it started from, the llama.cpp builds tried and
the Lemonade version) is recorded in
`.rocm-cli-alignment-failed.json` in the Lemonade runtime tree. For the next 24
hours an install for the same active ROCm version, pin and Lemonade version
does not attempt the alignment: it installs the backend already
pinned without forcing a reinstall, so a backend already installed is reused,
and prints a line saying the alignment could not be done, when it failed, and
how many hours from now an install tries it again (nothing retries on its own).
If preparing that pinned backend fails (which can happen before anything is
installed, for example when `lemond` does not start), the error says so and that
the alignment was skipped because of the earlier failure, and says that `rocm
engines install lemonade --reinstall` re-extracts the runtime tree without the
record, so it tries the alignment again (downloading the backend again); the
line it adds says only what that command does, not that it fixes the error
beneath. Run at any time, the same command tries the alignment again at once.
A different active ROCm version, a new
Lemonade version (which also re-extracts the tree), or a record 24 hours old or
older (or dated in the future) runs the alignment as before, and an
alignment that succeeds removes the record. The record is consulted only when an
install could align: not with the opt-out set, on an unsupported platform, with
no active ROCm version or an unreadable pin, or when the pin already equals the
active ROCm version, all of which leave it untouched and say nothing about it.
Past those checks, and before the backend probe starts `lemond`, a record that
applies skips the alignment, and one that cannot be read is removed with a
warning; if it cannot be removed either, the warning says so and names `rocm
engines install lemonade --reinstall`, which removes it. A record that cannot be
written leaves the next install to try again, also with a warning. Setting
`ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT` neither reads nor changes the
record. The record does not note the rocm-cli version, so upgrading rocm-cli
does not clear it: it still expires after 24 hours, and `--reinstall` clears it
sooner.

A pin an earlier alignment kept can go stale: the llama.cpp or TheRock build it
names may later be withdrawn upstream. When the pin already equals the active
ROCm version the install does not attempt another alignment, so if installing
Expand Down
26 changes: 26 additions & 0 deletions docs/manual-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -487,6 +487,32 @@ ROCm backend directory under `bin/llamacpp/`, and run
- `therock.version` and `llamacpp.rocm-stable` in `backend_versions.json` hold the
packaged values again, and the unrelated key is still there.

To check that a failed backend alignment is not repeated (Windows or native
Linux with a GPU Lemonade runs its ROCm backend on, and an active managed ROCm
SDK): build with `cargo build -p rocm --features rocm/e2e-test-hooks`, set
`ROCM_E2E_LEMONADE_ALIGNMENT_TARGET_VERSION=7.99.0` (a ROCm version Lemonade
publishes no backend for), and run `rocm engines install lemonade --reinstall`,
then `rocm engines install lemonade`. Expected result:

- The first install warns `could not align Lemonade's ROCm backend to 7.99.0;
reverting to the default pinned version`, exits 0, and leaves
`.rocm-cli-alignment-failed.json` in the Lemonade runtime tree.
- The second install prints no alignment warning and does not download or
reinstall the backend (`Using installed Lemonade llamacpp:rocm backend.`). It
prints `Lemonade's ROCm backend could not be aligned to ROCm 7.99.0 less than
an hour ago; ...`, naming `rocm engines install lemonade --reinstall` and
that an install run 24 hours from now or later tries it again, and exits 0.
- Running the first command again attempts the alignment again (the warning
is back), because `--reinstall` removed the record with the runtime tree.
- Not a manual step: when preparing the pinned backend fails on the second
install, its error says so, that the alignment was skipped because of the
earlier failure, and that `rocm engines install lemonade --reinstall`
re-extracts the runtime tree without that record; the record is kept and the
skip line above is not printed. There is no way to make that install fail on
demand, so this is covered by the unit test
`a_skipped_alignment_whose_install_fails_reports_the_error_and_claims_nothing`
in [backend_alignment.rs](../engines/lemonade/src/backend_alignment.rs).

To check what a Lemonade install says about its size (no download needed for
the refusal): with a debug build, set
`ROCM_E2E_LEMONADE_FREE_SPACE_BYTES=1048576` (1 MiB free), then in a fresh data
Expand Down
53 changes: 51 additions & 2 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -992,8 +992,10 @@ pushed (`PRE_COMMIT_FROM_REF..PRE_COMMIT_TO_REF`, the commits the remote does no
The directory-based selection cannot see a crate's tests reading a file from outside the crate,
so the hook widens it: any Rust file also selects `xtask` (its contract tests scan every source
file), `README.md`, `docs/` and `skills/` also select `rocm` and `xtask` (the advised-commands
test reads them and `rocm` compiles in the assistant skill), and the e2e PowerShell fixture also
selects `rocm-core` (its tests `include_str!` it). The table is `CROSS_CRATE_READS` in
test reads them and `rocm` compiles in the assistant skill), the e2e PowerShell fixture also
selects `rocm-core` (its tests `include_str!` it), and the e2e Lemonade install steps also select
`rocm-engine-lemonade` (its `alignment_record` tests `include_str!` them to check the record's file
name). The table is `CROSS_CRATE_READS` in
[prepush.rs](../xtask/src/prepush.rs), and a unit test scans the workspace for `include_str!`,
`include_bytes!` and manifest-relative reads that leave their crate and fails when one is not
covered, so a new cross-crate read cannot be added without a decision. The scan has a limit: a read
Expand Down Expand Up @@ -2663,6 +2665,53 @@ lemonade` against the shared tree) reaches the restore only on a runner whose
shared tree still holds a stale pin; on a healthy runner the install succeeds and
the restore never runs, so a green lane does not show that it works.

Lemonade failed-alignment record focused tests (ROCMAI-437; a failed
alignment is not repeated by the next install for 24 hours; see
[Lemonade backend alignment on engine install](engine-plugins.md#lemonade-backend-alignment-on-engine-install)):

```bash
cargo test -p rocm-engine-lemonade record

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: cargo test -p rocm-engine-lemonade record is advertised here as one of the "focused tests" for the alignment record, but cargo's test filter is an unanchored substring match, so this command also runs several unrelated tests whose names contain "record" elsewhere in the crate (e.g. install-footprint-record tests in footprint.rs, server-record tests in direct_llama.rs, record_runtime_version/prepare_embeddable_records_... in install.rs, and recording tests in lib.rs/router_serve.rs). It still runs the intended tests, just not only them, which undercuts the "focused" framing. A narrower filter (e.g. alignment_record:: or the specific test-name prefixes already used on the next two lines) would make the claim accurate.

Confidence 80 · documentation-accuracy

cargo test -p rocm-engine-lemonade a_failed_alignment_is_not_repeated
cargo test -p rocm-engine-lemonade a_skipped_alignment_whose_install_fails
```

The tests are in
[alignment_record.rs](../engines/lemonade/src/alignment_record.rs) and
[backend_alignment.rs](../engines/lemonade/src/backend_alignment.rs), one per
state of the record described in the section linked above, and their names say
which: no record (`with_no_record_the_alignment_runs_as_before`), one that
applies (the regression test,
`a_failed_alignment_is_not_repeated_by_the_next_install_within_a_day`: the
second install calls no alignment tier and installs without forcing, and its
notice is asserted with the pin file), one that applies checked before the
backend probe starts `lemond`
(`a_record_that_applies_skips_before_the_backend_probe_runs`), one for another
ROCm version, one a day old, one an early exit never consults (one table test,
a row per exit, with an unreadable record on disk), an unreadable one past
those exits (removed with a warning on every path: probe error, Vulkan and
ROCm, in `an_unreadable_record_is_removed_with_one_warning_on_every_path`; and,
on the ROCm-probe path only, the probe asserting from inside that the removal
and warning came first, in
`an_unreadable_record_is_removed_and_reported_before_the_backend_probe_runs`), one that cannot be removed either, one that cannot be written, one
a later verified alignment removes, one that cannot be removed after that
alignment, a revert whose install fails writing none (asserted in
`align_revert_restores_the_backend_dir_when_the_fallback_install_fails`), the
opt-out leaving one alone, and a skipped alignment whose install fails
(`a_skipped_alignment_whose_install_fails_reports_the_error_and_claims_nothing`:
its error chain asserted in three rows, `lemond` not starting, a transient
failure and the backend retry's own advice, each with the kept record and no
notice). Removal by `--reinstall` is
`reinstall_removes_the_record_of_a_failed_alignment` in
[install.rs](../engines/lemonade/src/install.rs), and
`the_record_file_name_matches_the_one_the_e2e_steps_hard_code` reads the e2e
steps' copy of the record's file name, so the two cannot drift. The scenario
`runtime-lemonade-backend-alignment-failure-remembered` (runtime-33 in
[runtime_setup.feature](../tests/e2e-cucumber/features/runtime_setup.feature))
runs the two installs for real on the `@nightly @requires-gpu` Lemonade lanes
(not under WSL2); it targets ROCm 7.99.0 through
`ROCM_E2E_LEMONADE_ALIGNMENT_TARGET_VERSION`, compiled only into the
`rocm/e2e-test-hooks` build.

Lemonade backend selection and alignment focused tests (reading the
`llamacpp` rows out of `lemonade backends` when other recipes have indented
rows of their own, and the second alignment attempt choosing the newest
Expand Down
Loading
Loading