Skip to content

fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm backend - #855

Merged
rominf merged 17 commits into
mainfrom
fix/lemonade-2026.44
Nov 17, 2026
Merged

rominf merged 17 commits into
mainfrom
fix/lemonade-2026.44

Conversation

@rominf

@rominf rominf commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

On a ROCm 10.1 SDK, rocm engines install lemonade could not give Lemonade a llama.cpp ROCm backend paired with that SDK. Lemonade 11.5.1 pins TheRock 7.13.0 and llama.cpp b9752, and downloads TheRock from a location that does not serve ROCm 10.x. This pins Lemonade 2026.44.1, which pins TheRock 10.1.0 and llama.cpp b11580. That llama.cpp build ships ROCm 10.1, 10.0 and 7.14 archives for Linux and Windows. Lemonade 2026.44.1 installs TheRock from AMD's whl-next wheel index first and falls back to a tarball from stable.repo.amd.com/rocm/core/tarball/.

Why not 2026.45.3, the newest stable release. Its Linux lemond needs glibc 2.43 and OpenSSL 3.2.0 (objdump -T: GLIBC_2.43, OPENSSL_3.2.0), reported upstream as lemonade-sdk/lemonade#4123. On the Ubuntu 24.04 Strix Halo lanes (Ubuntu release, Ubuntu nightly and WSL2), which have glibc 2.39 and OpenSSL 3.0, it exited with version `GLIBC_2.43' not found, and rocm engines install lemonade failed with "Lemonade server did not become ready". 2026.44.1's lemond needs glibc 2.38 and OpenSSL 3.0.0, the same as every stable release from 11.5.1 through 2026.44.1. The only 24.04 build of 2026.45.3 is the lemonade-server package in Lemonade's Launchpad PPA (ppa:lemonade-team/stable). That is a system package rather than the embeddable archive rocm-cli extracts, and a PPA serves only its newest published version, so it cannot be pinned the way a GitHub release asset can.

Merge order: this lands before #941. #941 removes runtime-deps.toml and moves the Lemonade pin and its digests into runtimes/mise.toml and runtimes/mise.lock. After this merges, #941 redoes the pin at 2026.44.1 there and carries LINUX_LEMOND_NEEDS along with the digests. #820 (ROCm 10.1 as the default SDK) depends on this PR: its MI300X Lemonade scenarios fail on 11.5.1.

What changes

Linux and Windows:

  • rocm engines install lemonade installs Lemonade 2026.44.1 (runtime_id: lemonade-embeddable-2026.44.1). The version is pinned in runtime-deps.toml. The new archive digests (EMBEDDABLE_LINUX_SHA256 and EMBEDDABLE_WINDOWS_SHA256) are recorded and checked as before.

  • Backend selection now reads lemonade backends by row layout instead of matching recipe names. Before, a recipe rocm-cli did not know, and the indented rows after it, were read as llama.cpp backends. Recipes new since 11.5.1 include llamacpp-hrx, ds4 and thenoise. An update_required backend is now reinstalled and an update_available one is reused; both used to be dropped. At the v2026.44.1 tag, Lemonade's own Action for an update_required row is lemonade backends install llamacpp:<backend>, the command rocm-cli runs, and BackendManager::install_backend has no early return for an installed backend while online; for rocm-stable it also removes and re-fetches TheRock (src/cpp/server/system_info.cpp, src/cpp/server/backend_manager.cpp). The state is read at the Status column the header names: the first token that starts at or after it, so a field wider than its column still works. A line whose Status slot holds no state, such as a warning printed mid-table, is skipped without ending the recipe above it, even when its text contains a state word, so the indented backends after it still count. With no header, a first-column line names a recipe only if its first word is shaped like a recipe name.

  • When alignment's first attempt fails, the second attempt now picks the newest lemonade-sdk/llama.cpp release that ships the archive for the active ROCm version on this host's platform. It used to take the newest release. Releases do not always ship every ROCm archive:

    • b12013 through b12086 shipped only qualification reports;
    • b11997 shipped no ROCm 7.14 archive;
    • b11610 shipped no Windows ROCm 10.x archive.

    The second attempt makes one unauthenticated GitHub API call listing the 30 newest releases. An asset counts only once GitHub reports it uploaded. The build it installs is a moving target, not a pin, and Lemonade does not checksum it: the checksums table in Lemonade's backend_versions.json has no lemonade-sdk/llama.cpp entry.

  • docs/engine-plugins.md and MANIFEST.md describe the new listing and the downloads Lemonade makes itself. docs/engine-plugins.md also says what a Lemonade bump must record: the new digests and, on Linux, what the new lemond needs from the host.

Linux and WSL only:

  • rocm engines install lemonade now refuses a host whose glibc or OpenSSL falls short of what the pinned lemond needs, before downloading anything. The engine also refuses to start that lemond on such a host. For each library that falls short, the message names the version needed and what the host has. On an Ubuntu 20.04 host (glibc 2.31, OpenSSL 1.1 only) it reads:

    this host's libraries are too old for the Lemonade 2026.44.1 server: it needs glibc 2.38 or newer (this host's libc.so.6 provides up to GLIBC_2.31) and OpenSSL 3.0.0 or newer (this host's loader cache has no libssl.so.3). Nothing was downloaded.
    

    (The line goes on: "If this host provides these libraries outside the loader cache (LD_LIBRARY_PATH, a Nix or Guix profile), set ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK=1 to skip this check.")

    On Ubuntu 22.04 (glibc 2.35, OpenSSL 3.0.2) only the glibc part appears. Before, such a host downloaded and extracted the archive and then failed with "Lemonade server did not become ready". The needs are recorded in LINUX_LEMOND_NEEDS, beside the archive digests in engines/lemonade/src/install.rs. They are keyed by the Linux archive digest, so a bump that does not re-read them fails a unit test.

    What the message reports for the host is the newest version node the library defines, which is what the loader compares, not necessarily its release: Ubuntu 24.04's glibc 2.39 defines up to GLIBC_2.39, but its OpenSSL 3.0.13 defines only up to OPENSSL_3.0.3. So a version shortfall reads "(this host's libc.so.6 provides up to GLIBC_2.31)".

    The host's libraries are found through the ldconfig cache. Each one reads as one of three things:

    • a version: the newest GLIBC_* or OPENSSL_* version node the listed libc.so.6 or libssl.so.3 defines, which is what the dynamic loader checks. The file is read with the object crate's ELF reader; object 0.37 is already linked into rocm through backtrace, so it adds no crate there, and it is one new crate for the standalone rocm-engine-lemonade binary;
    • missing: the cache lists libc.so.6 but not libssl.so.3, which is how a host with only OpenSSL 1.1 reads;
    • unknown: the cache cannot be read, lists not even libc.so.6, or the listed library does not parse. Unknown never refuses.

    With the 2026.44.1 pin, OpenSSL can only fall short as missing, because every libssl.so.3 defines OPENSSL_3.0.0. An OpenSSL 3 older than a pin's need is still reported by version, which mattered for 2026.45.3's 3.2.0. The probe runs once per process.

    The sentence after the shortfall follows the install's branch. A fresh install says "Nothing was downloaded." Where a tree of the pinned version is already in place, a re-run had nothing to download, so it says instead "Nothing was changed: the Lemonade 2026.44.1 already installed in <folder> stays as it is, but its server cannot start on this host either." The start-time refusal says "Lemonade was not started."

    Because only the ldconfig cache is read, a host that provides OpenSSL 3 or glibc outside it (LD_LIBRARY_PATH, a Nix or Guix profile, a user prefix, a stale cache) would be refused although lemond could load there. Setting ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK (any value) turns both refusals into a warning: that names the shortfall and the variable, and Lemonade goes ahead. The refusal message names the variable. The README's minimum-platform paragraph now states OpenSSL 3.0.0 (libssl.so.3), says which releases need what, and describes the refusal. It also says the check reads glibc and OpenSSL only; GLIBCXX_3.4.32 (libstdc++) is not checked.

    docs/wsl.md, the fix-wsl-5-distro-too-old rationale rocm fix prints and the matching note rocm diagnose prints said the glibc 2.38 / GLIBCXX_3.4.32 floor is what "every published Lemonade embeddable" needs. 2026.45.3 makes that false, so they now name the pinned build and add that newer Lemonade releases need more. The printed text changes; the check and its outcome do not, so no scenario changes.

Windows has no glibc or OpenSSL requirement to check, so this refusal does not exist there.

Verification

  • Each recorded digest equals both GitHub's published sha256 and the SHA-256 of the downloaded archive, for lemonade-embeddable-2026.44.1-ubuntu-x64.tar.gz and -windows-x64.zip. Both archives package the same backend_versions.json apart from line endings. It pins TheRock 10.1.0 and llama.cpp rocm-stable b11580, and b11580 ships ubuntu-rocm-10.1, -10.0 and -7.14 archives plus the matching win-rocm-* zips.

  • Host requirements of each stable release's Linux lemond, from objdump -T (every ELF file in the 2026.44.1 and 2026.45.3 archives was checked):

    Release glibc OpenSSL
    2026.45.3 2.43 3.2.0
    2026.44.1, 2026.43.3, 2026.42.1, 2026.41.1, 2026.40.0, 2026.39.1, 11.9.0, 11.8.1, 11.5.1 2.38 3.0.0
  • Unit tests cover:

    • the 2026.44 lemonade backends table;
    • the update states;
    • the release-listing choice, including the asset upload state;
    • the host-platform choice in the second attempt, through an injected listing fetch, so the Windows job checks the Windows archive name;
    • the digest format;
    • the host-library check: one row per state the probe can produce (both libraries new enough, exactly at the need, glibc too old, OpenSSL 3 missing alone and with glibc too old, an OpenSSL 3 older than a 3.2.0 need, versions unknown), and the message for each;
    • the probe over a planted ldconfig -p: a cache listing libc.so.6 and only libssl.so.1.1 reads OpenSSL as missing and is refused, while an unreadable cache, a cache with no libc.so.6 and an unparseable library read as unknown. The ELF reader is also checked against this host's real libc.so.6 and libssl.so.3;
    • starting lemond: a stand-in lemond is not started on a host that falls short, which the test asserts together with the message, and is started on a host that meets the needs, for a tree of another version, or with ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK set;
    • the opt-out turning a refusal into the warning, and the refusal naming it;
    • the closing sentence for each install state: no tree, a tree of another version (both "Nothing was downloaded."), and a tree of the pinned version ("Nothing was changed ...");
    • a warning line between the llamacpp row and its indented rows.

    Each test was seen to fail with its target broken.

  • @id:lemonade-installs-refused-on-too-old-host-libraries (lemonade-installs-07, new) runs on the Linux mock lane through the rocm/e2e-test-hooks override of the host's libraries. It reports an Ubuntu 20.04-like host (ROCM_E2E_LEMONADE_HOST_GLIBC=2.31, ROCM_E2E_LEMONADE_HOST_OPENSSL=missing), a state the real probe produces, and asserts the message and that nothing was downloaded, extracted or recorded. It passes locally. It goes red when the install-time refusal drops its "Nothing was downloaded." outcome, and when the hook stops reporting OpenSSL as missing.

  • @id:lemonade-installs-host-library-check-opt-out (lemonade-installs-08, new, Linux mock lane) sets the same old-host hooks plus ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK=1 and reports the drive nearly full. It asserts the warning and that the install went past the host check to the space refusal, with nothing downloaded.

  • @id:lemonade-installs-refused-on-too-old-host-libraries-when-installed (lemonade-installs-09, new, Linux mock lane) plants a tree of the pinned version, re-runs the install on the old-host hooks, and asserts the "Nothing was changed" sentence, the absence of "Nothing was downloaded.", and that every file of the planted tree is byte-for-byte unchanged.

  • No scenario covers the refusal to start lemond (it sits in spawn_lemond_with_parent, which every spawn, including the router serve, goes through); it is unit-tested only. No lane can reach it: rocm serve runs the install check first, a Linux serve with a packaged llama-server never starts lemond, and the mock lane has no GPU for serve to resolve. It guards a host whose libraries changed after install.

  • Gated scenarios. The PR's headline outcome, a ROCm 10.1 SDK getting a paired ROCm backend, is exercised only by runtime-16 below, which is @nightly on the Windows Strix Halo lane. No pre-merge check fails if that outcome regresses; the unit tests cover the alignment state machine and the backend-version parsing it relies on.

    • @id:runtime-lemonade-backend-alignment-reported (runtime-16, Windows Strix Halo nightly GPU lane) accepts both routes: aligned on an SDK older than the pin, untouched on ROCm 10.1. Where the SDK is the packaged pin, it also compares the installed llamacpp.rocm-stable tag with the packaged one and requires that no alignment was reported or logged. Where the pin moved off the packaged one, it requires the install to report the alignment, so a CLI that aligns silently fails on the 7.14 lanes. expectations.toml carries a deterministic Windows xfail row for it (ROCMAI-899), and one for serve-06 (ROCMAI-900), both written while 11.5.1 was pinned. Their reasons now say so instead of calling 11.5.1 the pin. Both scenarios are @nightly, which pull-request lanes do not run, so this branch's CI cannot show whether they now XPASS. I will check the first Windows nightly run after this merges and delete each row it reports as a stale XPASS, in a follow-up PR.
    • @id:runtime-lemonade-backend-alignment-opt-out (runtime-17, nightly GPU lanes) compares the pin, and the llamacpp.rocm-stable tag paired with it, with the ones in the verified archive.

    Neither runs in default CI. On a ROCm 10.1 lane the ROCm version alone cannot tell an untouched pin from one rewritten to the same value, which is why both also compare the llama.cpp tag; the 7.14 legs are what exercise alignment.

  • The Strix Halo Ubuntu release, Ubuntu nightly and WSL2 lanes, which failed on 2026.45.3, passed on 2026.44.1 (run 42990647881 at 1138a073).

  • Local gate on the merged tree: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p rocm-engine-lemonade --all-targets --features e2e-test-hooks -- -D warnings, cargo xtask check-module-size, check-architecture-doc and check-crate-edges pass. cargo xtask e2e -- -n "lemonade-installs-" passes all 9 scenarios. lemonade-installs-09 was seen to fail with the closing sentence fixed to "Nothing was downloaded.", and the opt-out and closing-sentence unit tests were each seen to fail with their target broken. cargo test --workspace --all-targets passes except four tests. Three fail on this host alone: service_control::tests::stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails and two build_metadata git_watch_paths_* tests. The fourth, rocm-core's endpoint_readiness::tests::a_merge_spares_another_writers_staged_file, which this PR does not touch, failed once under the full parallel run and passed when re-run alone three times. cargo xtask smoke passes up to its rocmd sandbox examine step, where bubblewrap cannot create a network namespace on this host (Failed RTM_NEWADDR: Operation not permitted).

Not verified locally:

  • a real lemond 2026.44.1 serving a model;
  • running either lemond in an Ubuntu 24.04 container (the host requirements above come from the binaries' symbol versions);
  • anything on Windows, including cargo clippy for the Windows target;
  • whether stable.repo.amd.com serves a 7.14.x TheRock tarball or wheels.

Those run only on CI: the Windows job and the self-hosted GPU lanes.

@juhovainio

Copy link
Copy Markdown
Collaborator

Linux test results from a separate effort to bump Lemonade. We will not open a competing bump; this PR owns the pin, and the notes below are meant to save duplicate work.

What was tested: Lemonade 2026.45.3 (the newest stable release, 2026-11-04), not 2026.44.1, on Linux (Strix Halo gfx1151 under WSL2), with each SDK installed into its own isolated root. I only checked 2026.44.1's binary and backend_versions.json (TheRock 10.1.0, llama.cpp b11580, same two ROCm 10 hosts); I did not run it. Nothing was run on Windows.

Verified archive digests for 2026.45.3, in case the pin moves later:

  • ubuntu-x64.tar.gz: fa780b91fd76cf1ef8fb2fd9a6d77506470b45dae162b6a1e46882fe2c056783 (downloaded and hashed locally)
  • windows-x64.zip: 27bedfb6bbc521e17910b76e4067a54622843bd68e5b2995bdb2e3954cf8b192 (GitHub's published asset digest; not downloaded)

Results with 2026.45.3

SDK rocm engines install lemonade rocm serve ...:Q4_K_M --engine lemonade runtime-16 / runtime-17 / serve-08
ROCm 10.0.0 Aligned Lemonade's ROCm backend to 10.0.0 using its pinned llama.cpp build ready 3 of 3 pass
ROCm 10.1.0 no alignment attempted (pin equals SDK); backend installs n/a runtime-16 and runtime-17 fail, serve-08 passes

The 10.1.0 failures match what the PR description says: runtime-16 expects an alignment line that cannot appear when the pin already equals the SDK, and runtime-17's left != right assertion compares 10.1.0 with 10.1.0. They fail because the lane cannot distinguish the cases, not because of a product defect. A 10.0.0 leg (or any SDK that differs from the pin) is the one that exercises alignment on a ROCm 10 stream.

serve-07 fails on Linux independently of this PR. It fails the same way on current main with Lemonade 11.5.1 (main-branch binary f38f08e7): Lemonade did not download 'Qwen3-0.6B-GGUF' for direct ROCm serving. The model does load in Lemonade, but direct_llama_model_path in engines/lemonade/src/direct_llama.rs only resolves a local file, an owner/repo:variant name, or the default model, so the bare recipe name bails once the packaged llama-server is found on Linux. I did not establish what changed since serve-07 last passed on this host, so this is a note, not a diagnosis.

Out of scope here: aligning to a 7.x SDK. Lemonade's TheRock hosts are compiled in and only serve ROCm 10.

@rominf
rominf force-pushed the fix/lemonade-2026.44 branch from 806c9f0 to 75bb2f7 Compare November 10, 2026 11:26
@rominf rominf changed the title fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm backend fix(lemonade): pin Lemonade 2026.45.3 so a ROCm 10.1 SDK gets a ROCm backend Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@rominf
rominf force-pushed the fix/lemonade-2026.44 branch from 75bb2f7 to 1138a07 Compare November 11, 2026 08:53
@rominf rominf changed the title fix(lemonade): pin Lemonade 2026.45.3 so a ROCm 10.1 SDK gets a ROCm backend fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm backend Nov 11, 2026
@rominf
rominf marked this pull request as ready for review November 11, 2026 13:39
@rominf
rominf requested a review from a team as a code owner November 11, 2026 13:39

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the whole diff. I found nothing that breaks the pin itself, and I checked that both recorded archive digests match GitHub's published sha256 for v2026.44.1. The parser reading the row layout instead of a recipe list is a good change, LINUX_LEMOND_NEEDS being tied to the digest by a test is a good guard, and the injected Tier-2 fetch lets the Windows job check the Windows asset name. Requesting changes for the items below, mainly the first one, plus the rebase.

Needs a fix

  • The OpenSSL half of the host check cannot fire for this pin. The probe reads only libssl.so.3, and every libssl.so.3 defines OPENSSL_3.0.0, which is all the pin needs. A host that really has OpenSSL 1.1.1 has no libssl.so.3 in the ldconfig cache, so the probe returns unknown and never refuses; the user still gets "did not become ready". The README line and the PR-body example ("this host has OpenSSL 1.1.1") describe a state the real probe can't produce, and the e2e scenario reaches it only through the hook override. Either refuse when ldconfig worked but libssl.so.3 is absent, or narrow the claim. See the inline comments.
  • The branch conflicts with main and needs a rebase.

Smaller points

  • runtime-16 now passes on ROCm 10.1 without any alignment happening, which the PR body acknowledges. The ROCMAI-899 Windows xfail row in expectations.toml is untouched and may turn into a stale XPASS, so please confirm who removes it after the first nightly.
  • probe_host_libraries runs ldconfig -p and reads libc/libssl on every spawn_lemond. The cost is small, but a OnceLock would avoid repeating it.
  • The README's "every published build of it needs those versions" now contradicts the docs' own note that 2026.45.3 needs glibc 2.43 / OpenSSL 3.2.0; worth tightening here.
  • The Tier-2 llama.cpp build is unpinned and unchecksummed, and is now fetched on Windows too. It is documented, but it is a conscious tradeoff for the author and maintainers to accept.

Comment thread engines/lemonade/src/host_libraries.rs Outdated
};
HostLibraries {
glibc: probe("libc.so.6", "GLIBC_"),
openssl: probe("libssl.so.3", "OPENSSL_"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The OpenSSL probe reads only libssl.so.3, and the pin needs 3.0.0, which every libssl.so.3 defines, so this can never report a shortfall for the current pin. A host with OpenSSL 1.1.1 has no libssl.so.3 in the cache, so this returns None and never refuses. Consider refusing when ldconfig worked but libssl.so.3 is absent, or narrowing the README/docs/PR claim.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a6adb69. Each library now reads as a version, missing, or unknown. When the ldconfig cache was read and lists libc.so.6 but not libssl.so.3, OpenSSL is missing and the host is refused with "OpenSSL 3.0.0 or newer (this host's loader cache has no libssl.so.3)". That is the OpenSSL 1.1-only case. An unreadable cache, a cache that lists not even libc.so.6, and a library that does not parse still read as unknown and never refuse. The cache reading is now a pure function (host_libraries_from_cache), and a_read_cache_without_libssl_3_reports_openssl_missing_and_refuses runs it over planted ldconfig -p output for each of those cases. lemonade-installs-07 now reports a state the real probe produces: glibc 2.31 and no libssl.so.3. The probe is also cached in a OnceLock, as suggested in the review body.

Comment thread README.md
downloading anything and names the glibc or OpenSSL version Lemonade needs and
the one the host has.

> [!IMPORTANT]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The claim that the message names the OpenSSL version "the host has" is not reachable on a pre-3.x host with the real probe (see the comment in host_libraries.rs); the e2e scenario only gets there through the hook override. Either fix the probe or narrow this wording.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a6adb69. The README now says what the probe can report: the glibc version needed and the one the host has, or that the host has no libssl.so.3, the library of OpenSSL 3. The PR body example is now an Ubuntu 20.04 host (glibc 2.31, OpenSSL 1.1 only), which is the message the real probe produces there.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 1138a073b8

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (24 files, +1552/-256), split into six independent file-coupled passes (host-library refusal core: install.rs/host_libraries.rs/supporting crate exports; process.rs/lib.rs wiring; backend_alignment.rs; llamacpp_backend.rs/direct_llama.rs; the e2e cucumber feature/step files; docs/config), each reading its files completely at the PR head, plus this consolidation and a short design-review pass.
No single tracked ticket is claimed closed by this PR. It references ROCMAI-899 (a Windows xfail row context) and ROCMAI-825 (prior disk-footprint work) as background, cites the upstream lemonade-sdk/lemonade#4123 issue for why 2026.45.3 isn't pinned instead, and states this PR must land before companion PR #941 (which later moves this same pin into runtimes/mise.toml) and is a dependency of #820.

Blocking

  • The new "refuse to start lemond on a too-old host" behavior has no end-to-end/Gherkin coverage, only a unit test — engines/lemonade/src/process.rs (spawn_lemond_on/refuse_lemond_on_too_old_host, the single choke point for every "start lemond" call site: a fresh install's backend probe, backend-alignment re-probes, and a later rocm serve against an already-installed tree), tests/e2e-cucumber/features/lemonade_installs.feature (scenario lemonade-installs-07 only exercises the install-time refusal in lib.rs's install_response, via rocm engines install lemonade).
    This repo's own AGENTS.md (§3) requires a Gherkin scenario for user-observable CLI behavior changes and says explicitly: "a unit test asserting the internal helper does NOT discharge this; it proves the function, not the behavior." The repo already has the matching precedent one scenario pair away: lemonade-installs-01/-02 cover the analogous disk-space refusal for both the install path and the serve path. This PR adds only the install-side scenario for the library-floor refusal; the serve-time path (a different outcome string, "Lemonade was not started." vs. "Nothing was downloaded.", and reachable even when Lemonade is already installed) is covered only by process.rs's unit test. The PR description lists the unit test but does not state or justify the e2e gap, so it isn't retired by the stated intent. The existing ROCM_E2E_LEMONADE_HOST_GLIBC/_OPENSSL test hooks already used by -07 make a paired -08-style scenario straightforward to add.
    Confidence 80 · architectural · Fix: add a Gherkin scenario (and step) driving rocm serve against an already-installed, pinned-version Lemonade tree with the host-library test hooks set too old, asserting the "...Lemonade was not started." message and that no server process starts — mirroring -02's relationship to -01 — or state in the PR text why this path is intentionally left to unit coverage only.

Non-blocking

None

Decisions for the author

  • The host-library refusal applies only to a tree matching the exact currently-pinned LEMONADE_VERSION, not to a leftover tree from an older pin — tradeoff
    process.rs's refuse_lemond_on_too_old_host short-circuits to Ok(()) when manifest.version != rocm_deps::LEMONADE_VERSION, and this is deliberate and commented ("a tree left by an older pin is replaced by the next install, which runs the same check before downloading"). But the PR's own verification table shows 2026.44.1, 2026.43.3, 2026.42.1, 2026.41.1, 2026.40.0, 2026.39.1, 11.9.0, 11.8.1 and 11.5.1 all need the identical glibc 2.38/OpenSSL 3.0.0 floor — only 2026.45.3 differs. So a host too old for the current pin is also too old for any of those earlier pins, and a leftover tree from one of them hits the exact "did not become ready" failure this PR exists to eliminate, with no refusal firing (the next install's re-check only runs if an install is actually triggered). Confirm this is acceptable scope (the check only acts on independently-objdump-verified needs for one digest, which is a reasonable conservatism) rather than an oversight.

Positive signals

  • LibVersion compares glibc/OpenSSL versions numerically (not lexicographically), with a dedicated regression test feeding GLIBC_2.39 before GLIBC_2.4 to catch exactly the "2.9 vs 2.10"-style string-comparison trap.
  • host_shortfall's test matrix covers every combination (both ok, exactly at need, each library too old alone, both too old, one/both unknown) independently, and the "a library version that can't be read is unknown and never refuses" safety invariant holds under tracing through every fallible step in host_libraries.rs.
  • The Tier-2 llama.cpp release/asset matching in backend_alignment.rs scans the entire candidate list via max_by (not first-match) and matches assets by exact generated filename rather than substring or position, avoiding both the "rocm-10.1 vs rocm-10.10" false-positive class and the "pair two lists by index" anti-pattern AGENTS.md warns against; an HTTP/lookup failure degrades to a logged warning and a fallback install rather than a hard failure.
  • The row-layout rework of parse_llamacpp_backend_statuses in llamacpp_backend.rs is backed by regression tests that are traced to actually fail against the pre-PR parser (an unknown recipe's indented rows misread as llamacpp backends; update_required/update_available silently dropped), not just happy-path fixtures.
  • LINUX_LEMOND_NEEDS is held equal to the pinned archive digest by a unit test, so a future version bump that forgets to re-derive the host-library needs fails the build loudly instead of silently shipping stale needs.
  • The install-time e2e scenario (lemonade-installs-07) that does exist correctly asserts both the refusal message and the on-disk state (nothing downloaded/extracted/recorded), matching AGENTS.md's rule that a behavioral claim and its resulting state belong in the same test.

Deployment notes

  • This changes observable behavior for operators on an older Linux host: a host below glibc 2.38 / OpenSSL 3.0.0 will now be refused up front with a specific message, where it previously appeared to attempt an install/serve and failed later with an opaque "Lemonade server did not become ready."
  • The Lemonade pin moves from 11.5.1 to 2026.44.1 rather than the newer 2026.45.3, specifically because 2026.45.3's Linux server needs glibc 2.43/OpenSSL 3.2.0, which the project's supported Ubuntu 24.04 lanes do not have; anyone tracking Lemonade's release notes independently should be aware this is an intentional, documented pin rather than lag.

What this covered

Every one of the 24 changed files read completely at the PR head (f7c5fef7a7…1138a073b8, merge-base with main), across six independently-run, file-coupled passes plus this consolidation and a short design-review pass; all CI checks visible via gh pr view --json statusCheckRollup were green except two self-hosted GPU e2e lanes (MI300X, W7900) still queued at review time (not a failure, just incomplete at the moment of this review). Production-line counts for every touched engines/lemonade/src/*.rs file were independently confirmed well under the repo's 1,500-line module-size limit by locating each file's #[cfg(test)] mod tests boundary, consistent with the green clippy CI job that enforces it.

Did not run: history/prior-changes passes beyond a light git log skim of the touched files (nothing notable surfaced); no cargo test/cargo clippy was executed locally (relied on the green CI result for compilation and test-suite confirmation, per the skill's "read what CI already reports" guidance); the two queued GPU lanes above were not waited on. The PR's own comment thread was read incidentally while gathering context (a non-blocking community comment reporting independent test results on 2026.45.3 the author had already summarized in the PR description, plus an automated security-scan comment); this was not used to shape or suppress any of the blind fan-out findings above, all of which were reached from the code and the repository's own AGENTS.md/test-convention text. Not reconciled against any other review or discussion thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of 1138a07: I read the changed files and diff but did not build or run tests; I rely on CI for that. No blockers. The pin, both archive digests, the glibc 2.38 / OpenSSL 3.0.0 needs keyed to the Linux digest, the row-layout parse of lemonade backends, the refusal ordering (before the footprint check and before any download) and the second-attempt release choice all look right.

Non-blocking:

  • host_libraries.rs: the OpenSSL half of the refusal can't fire on a real host (inline). The glibc half covers every realistic case, so I'm not blocking on it, but the wording in the PR text, README and docs promises more than the probe reports.
  • README.md: "every published build of it needs those versions" contradicts the note about 2026.45.3 (inline).
  • tests/e2e-cucumber/expectations.toml (not in the diff): the runtime-16 row and the serve-large-model row still say "Lemonade 11.5.1, the pin in runtime-deps.toml", which is no longer true. The description covers the runtime-16 stale-XPASS case; the other row's reason also needs rewording or removal.
  • The lemond start-time refusal (process.rs) has only a unit test, while the install-time one has lemonade-installs-07. A paired rocm serve scenario like the existing disk-space pair would close that, but it is a gap in coverage, not a defect.
  • docs/manual-testing.md section 3 doesn't mention the new host-too-old refusal.

Before merge: the branch conflicts with main and needs a rebase, and the W7900 and MI300X lanes were still queued when I looked.


🤖 by agenthub on AMD AgentHub

Comment thread engines/lemonade/src/host_libraries.rs Outdated
};
HostLibraries {
glibc: probe("libc.so.6", "GLIBC_"),
openssl: probe("libssl.so.3", "OPENSSL_"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the OpenSSL half of the check cannot fire for this pin on a real host. Every libssl.so.3 defines OPENSSL_3.0.0 or newer, which is all the pin needs, and a host with only OpenSSL 1.1.1 has no libssl.so.3 in the cache, so this returns None (unknown, never refuses). The "(this host has OpenSSL 1.1.1)" wording in the PR text, the README and the unit-test rows is reachable only through the e2e hook.

I'm not blocking on it because the glibc half already catches every realistic old host (anything with OpenSSL 1.1.x ships a glibc below 2.38), so the user-facing outcome is as described. Either refuse when the cache was read, libc.so.6 is listed and libssl.so.3 is not ("OpenSSL 3 (libssl.so.3) not found"), or narrow the OpenSSL wording to what the probe can actually report.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a6adb69, the first way you suggest: when the cache was read and lists libc.so.6 but not libssl.so.3, the host is refused with "this host's loader cache has no libssl.so.3". A version shortfall for OpenSSL 3 is still reported by version; no pin needs that today, but 2026.45.3's 3.2.0 would have, and a unit test covers it. The unit-test rows, README, docs and PR body no longer use "this host has OpenSSL 1.1.1"; they describe the missing-libssl.so.3 case instead.

Comment thread README.md Outdated
back to. Ubuntu 22.04 ships glibc 2.35 and cannot run it; Ubuntu 24.04 provides
glibc 2.39 and `GLIBCXX_3.4.33`.
`GLIBCXX_3.4.32`, and OpenSSL 3.0.0 or newer: that is what the Lemonade engine
is linked against, and every published build of it needs those versions, so

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: "every published build of it needs those versions, so there is no older release to fall back to" now disagrees with docs/engine-plugins.md and the comment in install.rs, which say 2026.45.3 needs glibc 2.43 and OpenSSL 3.2.0. Something like "every build up to 2026.44.1 needs glibc 2.38 and OpenSSL 3.0.0 (the newest, 2026.45.3, needs more)" keeps both true.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a6adb69. The README now says every stable release from 11.5.1 through 2026.44.1, the pin, needs glibc 2.38 and OpenSSL 3.0.0, and that the newest, 2026.45.3, needs glibc 2.43 and OpenSSL 3.2.0, which is why it is not pinned. I dropped "there is no older release to fall back to", since I only checked releases from 11.5.1 on.

…backend

Lemonade 11.5.1 pins TheRock 7.13.0 and llama.cpp b9752, and downloads
TheRock from a location that does not serve ROCm 10.x, so on a ROCm 10.1
SDK its llamacpp:rocm backend could not be paired with the SDK in use.
Lemonade 2026.44.1 pins TheRock 10.1.0 and llama.cpp b11580, whose release
ships ROCm 10.1 (and 10.0 and 7.14) archives for Linux and Windows, and
installs TheRock from AMD's whl-next wheel index or core tarball location.
The embeddable layout, binary names and the lemond/lemonade flags rocm-cli
uses are unchanged; the new archive digests are recorded.

Not 2026.45.3, the newest stable release: its Linux lemond needs glibc
2.43 and OpenSSL 3.2.0 and exits with a loader error on Ubuntu 24.04,
which has glibc 2.39 and OpenSSL 3.0.13. 2026.44.1's needs glibc 2.38
and OpenSSL 3.0.0, the same as every release back to 11.5.1.

Two adaptations the new release line needs:

- `lemonade backends` lists recipes rocm-cli did not know (llamacpp-hrx,
  ds4, thenoise) and adds Tier and Format columns. The parser matched
  recipe rows against a fixed list, so an unknown recipe's rows, and every
  indented row after them, were read as llamacpp backends. It now reads
  the recipe from the row layout, and reads Lemonade's update_required and
  update_available states as installable and installed instead of
  dropping the row.
- The second alignment attempt took lemonade-sdk/llama.cpp's newest
  release, but a release can ship without some or all of its ROCm
  archives: b12013 through b12086 carried only qualification reports, and
  others skip one ROCm version or one platform. It now lists the 30 newest
  releases and picks the most recently published one that ships the exact
  archive for the active ROCm version on this platform.

The runtime-16 and runtime-17 scenarios assumed the active SDK never equals
the packaged pin, which is no longer true on a ROCm 10.1 lane: runtime-16
now checks that the installed pin equals the active SDK by whichever route
the host takes, and runtime-17 compares the pin with the one inside the
verified archive.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The runtime-17 step that finds the cached Lemonade archive compared the
`.zip` extension with `ends_with`, which clippy's pedantic
`case_sensitive_file_extension_comparisons` rejects. The e2e target is
excluded from `--all-targets`, so only the Windows job's
`cargo clippy -p e2e-cucumber --test e2e` compiles it and caught it.
Derive the archive's stem once, comparing either extension ignoring ASCII
case, and reuse it for the member path.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A Linux lemond built against a newer glibc or OpenSSL than the host has
exits at once with a dynamic-loader error ("version `GLIBC_2.43' not
found"). The user saw only "Lemonade server did not become ready", after
the archive had been downloaded and extracted; this is how 2026.45.3
failed on the Ubuntu 24.04 GPU lanes.

Record what the pinned lemond needs (glibc 2.38 and OpenSSL 3.0.0 for
2026.44.1, read with objdump -T) beside the archive digests, keyed by the
Linux digest so a bump that does not re-read them fails a unit test.
Read the host's versions as the newest GLIBC_/OPENSSL_ version node its
libc.so.6 and libssl.so.3 define, the same thing the loader compares,
locating them through the ldconfig cache. On Linux,
`rocm engines install lemonade` now refuses such a host before
downloading anything, and starting lemond is refused too. Both name the
version needed and the version the host has for each short library. A
host version that cannot be read is treated as unknown and never refuses.

lemonade-installs-07 drives the install refusal through an e2e-test-hooks
override of the host versions, and checks the message and that nothing
was downloaded.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The OpenSSL half of the host check could not fire. The probe read only
libssl.so.3, and every libssl.so.3 defines OPENSSL_3.0.0, which is all
the pinned lemond needs. A host with only OpenSSL 1.1 has no libssl.so.3
at all, so the probe reported OpenSSL as unknown, never refused, and the
user still got "Lemonade server did not become ready".

Each library is now a version, missing, or unknown. It is missing when
the ldconfig cache was read and lists libc.so.6 but not that library,
and the refusal then says the host's loader cache has no libssl.so.3.
A cache that could not be read, one that lists not even libc.so.6, and
a library that does not parse still read as unknown and never refuse.
The cache reading is now a pure function, unit-tested over planted
`ldconfig -p` output, and the probe runs once per process.

lemonade-installs-07 now reports a host the probe can actually produce,
Ubuntu 20.04-like: glibc 2.31 and no libssl.so.3. The README, the docs
and docs/manual-testing.md section 3 describe the same two outcomes, and
the README no longer says every build needs glibc 2.38, since 2026.45.3
needs more. The start-time refusal has no scenario, because no lane can
reach it: `rocm serve` runs the install check first, a Linux serve with
a packaged llama-server never starts lemond, and the mock lane has no
GPU for serve to resolve.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The runtime-16 (ROCMAI-899) and serve-06 (ROCMAI-900) Windows rows were
written while Lemonade 11.5.1 was pinned, and their reasons still said
so. Say they were observed with 11.5.1 and have not been re-checked with
2026.44.1, which fetches TheRock elsewhere and whose llama.cpp build
ships a ROCm 7.14 archive. Both scenarios are @nightly, so pull-request
lanes have not run them yet; a stale XPASS on the first Windows nightly
after this pin lands is the signal to delete the row.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/lemonade-2026.44 branch from 1138a07 to 202490c Compare November 12, 2026 11:07
@rominf

rominf commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks both. Since the reviews on 1138a073, the branch is rebased onto the current main, and these two commits are new:

  • a6adb697 makes the OpenSSL half of the host check reachable and updates the docs to match.
  • 202490ca rewords the two Windows rows in expectations.toml.

The first three commits are unchanged apart from the rebase. The only conflict was docs/architecture.md, which #1016 restructured into tables. host_libraries.rs now has a row there, and the install.rs and process.rs rows are extended.

@juhovainio

  • OpenSSL can't fire: fixed in a6adb697. A cache that was read and lists libc.so.6 but no libssl.so.3 now reads OpenSSL as missing and refuses. Details are in the inline reply.
  • Rebase: done.
  • ROCMAI-899 row: runtime-16 and serve-06 are both @nightly, and pull-request lanes don't run those, so this branch's CI can't show whether either row now XPASSes. I've reworded both reasons so they no longer call 11.5.1 the pin. I'll check the first Windows nightly run after this merges and remove any row it reports as a stale XPASS, in a follow-up PR.
  • OnceLock: done. The probe runs once per process.
  • README: fixed (inline reply).
  • Unpinned Tier-2 llama.cpp build: agreed, it is a tradeoff. It is documented in docs/engine-plugins.md and MANIFEST.md. It only runs when the pinned build can't be paired with the active SDK, and the alternative there is a backend that doesn't match the SDK at all.

@jussielo-amd

  • OpenSSL and README: fixed in a6adb697 (inline replies).

  • expectations.toml: both rows are reworded, the runtime-16 one (ROCMAI-899) and the serve-large-model one (ROCMAI-900). Each reason now says it was observed with 11.5.1 and has not been re-checked with 2026.44.1, and when to delete it.

  • docs/manual-testing.md section 3: now covers the host-too-old refusal, both on a real Ubuntu 20.04 or 22.04 host and with the debug hooks.

  • Scenario for the start-time refusal: I didn't add one, because no lane can reach it:

    • rocm serve runs the install check first;
    • a Linux serve with a packaged llama-server never starts lemond;
    • the mock lane has no GPU for serve to resolve.

    It guards a host whose libraries changed after the install. The unit tests in process.rs cover it, and the PR body and docs/testing.md now say so.

  • Lanes: the Strix Halo Ubuntu release, Ubuntu nightly and WSL2 lanes passed on 2026.44.1 at 1138a073. W7900 and MI300X were still queued then; I'm watching them on 202490ca.

The automated review's two points

  • Start-time scenario: same answer as above.
  • Only the pinned version is checked: this is deliberate. The recorded needs were read from one archive. A tree from an older pin is replaced by the next rocm engines install lemonade, which runs the same check before it downloads anything.

One test-only change: the process.rs test that execs a stand-in lemond failed once in about twenty runs. It now retries only on ETXTBSY. I believe the cause is another test thread forking while the freshly written script was still open, but I couldn't reproduce it to confirm.

@rominf
rominf requested a review from juhovainio November 12, 2026 11:07

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't find a defect that breaks the change. The digests match GitHub's published sha256 values, and the refusal message is asserted together with the disk state. I'm still requesting changes, mainly on test coverage of the headline outcome.

  • 10.1 outcome coverage. "A ROCm 10.1 SDK gets a ROCm backend" is only exercised by a Windows @nightly @requires-gpu scenario, and on a 10.1 lane its "paired" check compares the packaged 10.1.0 with itself, so it holds whatever the CLI did. Nothing on a default lane asserts the outcome (AGENTS.md section 3). See the inline comment on runtime-16.
  • runtime-17 sensitivity. Replacing assert_ne!(pinned, active) means a rewritten pin is indistinguishable from an untouched one when the active SDK is 10.1. The PR body discloses this, but the opt-out check is weaker on exactly the lanes this PR targets.
  • Docs. docs/manual-testing.md says "debug build" for the library override, but the hook needs the e2e-test-hooks feature.
  • Question. update_required is mapped to "installable". Is it confirmed that lemonade backends install actually reinstalls a build in that state? The fixture is synthetic text, so nothing in the diff shows it.
  • Scope. The PR bundles the pin bump, a new Linux glibc/OpenSSL refusal (new object dependency and e2e hook), the parser rewrite and Tier-2 selection. The refusal in particular looks separable. If you keep them together, please say why in the PR text.

Comment thread docs/manual-testing.md Outdated
(`libssl.so.3`). On a real older host, such as Ubuntu 22.04 (glibc 2.35) or
Ubuntu 20.04 (glibc 2.31, OpenSSL 1.1 only), run
`rocm engines install lemonade --yes` in a fresh data folder. On a newer host,
use a debug build and set `ROCM_E2E_LEMONADE_HOST_GLIBC=2.31` and

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The override is gated on the e2e-test-hooks feature (host_libraries.rs), not on debug assertions, and docs/testing.md calls it the "rocm/e2e-test-hooks build". A plain debug build ignores these variables and runs a real install. Say "a build with the rocm/e2e-test-hooks feature".

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0b2add5. docs/manual-testing.md now says "a build with the rocm/e2e-test-hooks feature", matching the gate in host_libraries.rs and docs/testing.md.

return;
}
assert_eq!(
pinned_version, active_version,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the active SDK is 10.1 this compares the archive's packaged 10.1.0 with 10.1.0, so it holds whatever the CLI did. Consider asserting something only a correctly paired backend produces (e.g. the installed llama.cpp backend's ROCm archive version) so the 10.1 route is not tautological.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0b2add5. When the installed ROCm pin equals the archive-packaged pin (the ROCm 10.1 case), the step now also asserts that the installed llamacpp.rocm-stable tag equals the packaged one. An alignment that rewrote the pin would also have moved that tag, so the check no longer holds whatever the CLI did. I could not run it on a ROCm 10.1 GPU lane from here; it compiles and the lane itself is the check. If a packaged file has no such key, both sides read as absent and compare equal rather than panicking.

// Lemonade 2026.44.1) this cannot tell an untouched pin from one rewritten to
// the same value; the absent alignment and revert lines above are then the
// whole check.
assert_eq!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replacing assert_ne!(pinned, active) loses the ability to detect a rewritten pin on a ROCm 10.1 lane (active == packaged). Acknowledged in the PR body; consider an additional signal for that lane.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0b2add5, the same way: the packaged pin survives the install also compares the installed llamacpp.rocm-stable tag with the packaged one, so a pin rewritten to the same ROCm version on a 10.1 lane is still caught through its paired llama.cpp tag. The comment above the assertion says what each of the two checks can and cannot tell.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 202490ca

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (26 files, +1771/-261), split into three independent file-coupled passes (host-library refusal core: install.rs/host_libraries.rs/process.rs/lib.rs/Cargo.toml/rocm-deps/rocm-core's ldconfig_cache export/the lemonade-installs e2e pair; backend alignment and llama.cpp selection: backend_alignment.rs/llamacpp_backend.rs/direct_llama.rs/the runtime-16/17 scenario and its step assertions/expectations.toml; docs and e2e-coverage consistency: README/MANIFEST/docs/engine-plugins.md/docs/manual-testing.md/docs/testing.md/docs/architecture.md), each reading its files completely at the PR head, plus this consolidation and a short design pass.

No tracked ticket is claimed closed by this PR. It names ROCMAI-899 and ROCMAI-900 in expectations.toml row context (not retrievable from here — no tracker tool available to this review) and states this PR lands before a companion PR (#941) and is a dependency of another (#820); background only, not chased further.

Blocking

  • docs/manual-testing.md's new host-library repro instructions silently no-op without an undocumented Cargo feature flag — docs/manual-testing.md:394-408 (new paragraph added by this PR), specifically line 399: "On a newer host, use a debug build and set ROCM_E2E_LEMONADE_HOST_GLIBC=2.31 and ROCM_E2E_LEMONADE_HOST_OPENSSL=missing first." The env-var override these instructions depend on is compiled in only under #[cfg(all(feature = "e2e-test-hooks", target_os = "linux"))] in engines/lemonade/src/host_libraries.rs (HOST_GLIBC_TEST_ENV/HOST_OPENSSL_TEST_ENV, read inside probe_host_libraries), and e2e-test-hooks is an opt-in Cargo feature with no default (engines/lemonade/Cargo.toml), not something a plain cargo build ("a debug build") enables. The same doc file's own convention elsewhere spells this out correctly — line 661 says "a --features rocm/e2e-test-hooks debug build" — so the omission here is an inconsistency within the document itself, not just an outside claim. A tester on a host whose real glibc/OpenSSL are new enough (the stated precondition) who follows these instructions literally gets a binary where the #[cfg] branch is stripped out entirely: probe_host_libraries() falls through to the real ldconfig cache, finds the host genuinely adequate, and rocm engines install lemonade proceeds normally instead of producing the documented "Expected result" — with nothing telling the tester the override silently did not take effect.
    Confidence 88 · mechanical · Fix: add --features rocm/e2e-test-hooks (matching the phrasing already used at line 661) to the build instruction in this paragraph.

  • Branch has a merge conflict with main and cannot be merged as-is — PR metadata reports mergeable_state: "dirty" against base 8f086ad9c0fe85e7bab2bd8272dc88638938e4fe. Not a code defect, but an objective merge blocker independent of review outcome.
    Confidence 100 · mechanical · Fix: rebase onto current main and resolve the conflict.

Non-blocking

None

Decisions for the author

  • The host-library floor check covers glibc and OpenSSL but not libstdc++ (GLIBCXX_*), which the README still lists as part of the requirement — tradeoff
    README.md:46-54 states the non-Ubuntu-24.04 equivalent requirement as "glibc 2.38 with GLIBCXX_3.4.32, and OpenSSL 3.0.0 or newer", framed as "what the Lemonade engine is linked against." The actual refusal (host_libraries.rs's LemondHostNeeds/host_shortfall, and docs/engine-plugins.md) only ever probes the newest GLIBC_* and OPENSSL_* version nodes from libc.so.6 and libssl.so.3 — libstdc++.so.6/GLIBCXX_* is never read or compared anywhere in the new code. A host with adequate glibc and OpenSSL but an older/mismatched libstdc++ would pass the refusal and could still hit the loader-error-reads-as-"did not become ready" failure this PR exists to eliminate. This may be a deliberate scope limit (checking a third library adds real complexity, and glibc/libstdc++ vintages correlate closely enough in practice on mainstream distros that it may never diverge in the field) rather than an oversight — worth confirming that's the intended scope rather than an unnoticed gap, and if so, narrowing the README sentence to match what is actually checked.

Positive signals

  • host_libraries.rs's probe cleanly distinguishes Missing (cache read, lists libc.so.6, but not the library in question — refuses) from Unknown (cache unreadable, or lists nothing at all — never refuses), with direct unit-test coverage of the distinction including a real-host integration test against this machine's actual libc.so.6/libssl.so.3.
    engines/lemonade/src/host_libraries.rs
  • The host-library refusal is wired through a single choke point (process::spawn_lemond) for both the install path (install_response in lib.rs, before any download) and every later server-start path (including the backend-alignment probe), with no bypass found by tracing the actual call graph rather than trusting the comments describing it.
    engines/lemonade/src/process.rs, engines/lemonade/src/lib.rs
  • LINUX_LEMOND_NEEDS is tied to the pinned Linux archive digest by a unit test (linux_lemond_needs_belong_to_the_pinned_linux_archive), so a future version bump that forgets to re-derive the host-library needs fails the build instead of silently shipping stale needs.
    engines/lemonade/src/install.rs
  • The rewritten parse_llamacpp_backend_statuses reads the recipe from row layout rather than matching a fixed recipe list, verified against both the pre-2026.44 table format and the new Tier/Format column layout, with regression tests that are traced to actually fail against the old parser's known misreadings (an unknown recipe's indented rows, and the update_required/update_available states being silently dropped).
    engines/lemonade/src/llamacpp_backend.rs
  • The Tier 2 llama.cpp release matching correctly requires state == "uploaded" on the candidate asset, matches by exact generated filename (avoiding a rocm-10.1 vs rocm-10.10 false positive), and scopes correctly by platform.
    engines/lemonade/src/backend_alignment.rs

Deployment notes

  • This changes observable behavior for operators on an older Linux host: a host below glibc 2.38 / OpenSSL 3.0.0 is now refused up front with a specific message, where it previously appeared to attempt an install or serve and failed later with an opaque "Lemonade server did not become ready."
  • The Lemonade pin moves from 11.5.1 to 2026.44.1 rather than the newer 2026.45.3, specifically because 2026.45.3's Linux server needs glibc 2.43/OpenSSL 3.2.0, which the project's supported Ubuntu 24.04 lanes do not have.
  • At review time, the self-hosted MI300X and W7900 GPU E2E lanes show cancelled rather than a completed result (queued ~80 minutes before cancellation); every lane that actually completed, including the required Mock lane, is green. Per this repo's own gating policy (only the Mock lane is a required/blocking check; other platform lanes are informational), this does not block merge, but it means those two lanes' results are currently unverified for this commit.

What this covered

Every one of the 26 changed files read completely at the PR head (8f086ad9…202490ca), across three independently-run, file-coupled fan-out passes plus this consolidation and a short design-review pass. All CI checks visible via pull_request_read(method="get_check_runs") were green except the two self-hosted GPU lanes noted above (cancelled, not failed, and non-blocking per the project's own documented gating policy). mergeable_state was read directly from the PR object ("dirty").

Did not run: history/prior-changes passes (no git log/blame or search of prior reviews on these files was performed); no cargo test/cargo clippy was executed locally — relied on the green CI result for compilation and test-suite confirmation. Ticket lookups for ROCMAI-899/ROCMAI-900 could not be made (no tracker tool available to this review); their content was not guessed. While confirming no prior review from this automation's account already existed at this exact head commit, this review's author incidentally saw the text of two existing human review rounds (one from an older commit in this PR's history, one at the current head) and one from another automated reviewer, all surfaced by the same account/commit-matching check; none of that text was used to originate, shape, or suppress any finding above — every finding here was re-derived independently from the code during the blind fan-out passes, though full independence from that incidental exposure cannot be guaranteed the way it would be from a review that never touched the discussion at all. Not otherwise reconciled against any review thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

… pin

On a ROCm 10.1 lane the active SDK equals the pin Lemonade 2026.44.1 ships,
so comparing the ROCm version cannot show a rewritten pin. Also compare the
paired llamacpp.rocm-stable tag, and name the e2e-test-hooks feature in the
manual-testing note.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

# Conflicts:
#	tests/e2e-cucumber/expectations.toml

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 3ab5ec04b5

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (26 files, +1796/-262 against merge-base 691834a0), split into four independent file-coupled passes (host-library refusal core: install.rs/host_libraries.rs/process.rs/lib.rs/engines/lemonade/Cargo.toml/rocm-core's ldconfig_cache export/rocm-deps/Cargo.lock/runtime-deps.toml; backend alignment and llama.cpp selection: backend_alignment.rs/llamacpp_backend.rs/direct_llama.rs/the runtime-setup feature and steps/expectations.toml; the lemonade_installs e2e feature and steps plus the e2e-report markdown fixture; docs and manifest: README/MANIFEST/docs/architecture.md/docs/engine-plugins.md/docs/manual-testing.md/docs/testing.md), plus a dedicated AGENTS.md-adherence pass over the whole diff, a dedicated git-history/prior-convention pass, and this consolidation plus a short design-review pass. Two of the six fan-out workers hit a transient concurrent-subagent-limit error on first launch and were re-spawned once each, successfully, per this method's retry rule.

No ticket is claimed closed by this PR. It names ROCMAI-899 and ROCMAI-900 as context for pre-existing nightly xfail rows in expectations.toml (not retrievable — no tracker tool available to this review) and states this PR must land before a companion PR that moves this same pin elsewhere, and is a dependency of another PR that changes the default SDK version; background only, not chased further.

Blocking

  • docs/wsl.md's WSL2 prerequisite list restates the pre-PR host-library requirement and was not updated alongside the near-identical claim this PR edited in README.md, so it now omits the OpenSSL floor and the new refusal behavior — docs/wsl.md:16-20 (confirmed unedited by this PR via diff), vs. the edited README.md:46-56. docs/wsl.md still reads "Ubuntu 22.04 is not supported: it ships glibc 2.35, below the minimum that every published Lemonade embeddable requires (glibc 2.38 and GLIBCXX_3.4.32)... Ubuntu 24.04 provides glibc 2.39 and GLIBCXX_3.4.33" — no mention of OpenSSL 3.0.0/libssl.so.3, and no mention that rocm engines install lemonade now refuses such a host up front with a specific message, both of which README.md, docs/engine-plugins.md and docs/manual-testing.md were correctly updated to state in this same PR. AGENTS.md §5 directly names this failure mode: "the same behavior claim... often repeats across README.md, ... docs/.md... grep for the claim's wording across all of them, not just the surface you're editing."*
    Confidence 85 · mechanical · Fix: update docs/wsl.md's prerequisite bullet to add the OpenSSL requirement and mention the new install-time refusal, matching the wording already updated elsewhere in this PR.

  • README.md's restated host-floor requirement lists GLIBCXX_3.4.32 as part of "what the Lemonade engine is linked against," but the automated check this PR adds never verifies libstdc++/GLIBCXX at all — and the PR's own docs/engine-plugins.md, edited in the same diff, correctly scopes the check to glibc and OpenSSL only — README.md:46-48: "the equivalent requirement is glibc 2.38 with GLIBCXX_3.4.32, and OpenSSL 3.0.0 or newer... that is what the Lemonade engine is linked against", immediately followed by a description of the new automated refusal. docs/engine-plugins.md:150-152 instead correctly says only "rocm engines install lemonaderefuses a host whose glibc or OpenSSL is older than that."* I verified directly inengines/lemonade/src/host_libraries.rsthat onlyGLIBC_andOPENSSL_*version nodes are ever read (a unit test at line 491 explicitly assertsnewest_version_node(names, "GLIBCXX_")isNonein its own test setup) —GLIBCXX/libstdc++ is not part of LemondHostNeeds` or the refusal logic anywhere. A reader of README alone would reasonably believe a host meeting glibc+OpenSSL but running an older libstdc++ is covered by the new refusal; it is not, and would still hit the pre-PR "did not become ready" failure this PR exists to eliminate.
    Confidence 85 · mechanical · Fix: narrow README's wording to match what's actually checked (glibc + OpenSSL only), or state explicitly that GLIBCXX is a linking requirement not covered by the automated refusal.

Non-blocking

  • engines/lemonade/Cargo.toml's comment justifying the new object dependency ("it adds no crate to the shipped binary") is true only for the rocm binary, not for rocm-engine-lemonade itself, which also ships — engines/lemonade/Cargo.toml:43-44. I confirmed via Cargo.lock that object is pulled in by backtrace (via color-eyre, reached only through apps/rocm/the dash crates) and, separately, now directly by rocm-engine-lemonade — but rocm-engine-lemonade's own dependency list in Cargo.lock does not include backtrace, so for that binary (listed in scripts/build_single_exe_release.py's shipped-artifact list as rocm-engine-lemonade.exe/rocm-engine-lemonade) this is a genuinely new linked crate, not a free ride off an existing one.
    Confidence 85 · mechanical · Fix: narrow the comment to say which binary the "no new crate" claim applies to, since impact is minor (the dependency itself is small) but the comment's factual claim is false for one of the two binaries it could describe.

Decisions for the author

  • The host-library refusal checks glibc and OpenSSL but deliberately does not check libstdc++ (GLIBCXX_*), which the engine is also linked against — tradeoff
    This is a design-scope question distinct from the README/engine-plugins.md documentation mismatch above: even if the docs are tightened to say so accurately, the refusal itself still leaves a host with adequate glibc/OpenSSL but a mismatched libstdc++ unguarded against the same class of opaque failure this PR exists to eliminate. Checking a third library adds real complexity, and glibc/libstdc++ vintages correlate closely enough on mainstream distros that this gap may be low-probability in practice — worth the author confirming this is accepted scope rather than an oversight.
  • The llama.cpp Tier-2 release lookup is an unauthenticated GitHub API call made on every rocm engines install lemonade invocation that reaches it (the expected steady state until Lemonade's packaged pin catches up to the active ROCm version) — tradeoff
    On a shared-IP CI fleet (self-hosted runners, nightly lanes) this could intermittently hit GitHub's 60/hour anonymous rate limit and degrade (loudly, not silently) to the revert path. Worth confirming this is an accepted risk given the safe, non-silent degradation.
  • This PR bundles the version pin bump with a new host-library refusal feature, a backend-status parser rewrite, and a new Tier-2 release-selection algorithm — tradeoff
    All four are plausibly coupled to the pin itself (the new release needs new host-library floors recorded, has new backend recipes the old parser misread, and has a sparser release/archive matrix than before), so this reads as one coherent change rather than a bundling anti-pattern, but it's large enough (+1796/-262) that the author may want to confirm the coupling is intentional rather than incidental.

Positive signals

  • host_libraries.rs's probe cleanly distinguishes a version, "missing" (cache lists libc.so.6 but not the library in question), and "unknown" (cache unreadable or unparseable), with direct unit coverage of the distinction plus a real-host integration test against this machine's actual libc.so.6/libssl.so.3; "unknown never refuses" is a documented, deliberate fail-open choice.
  • LINUX_LEMOND_NEEDS is tied to the pinned Linux archive digest by a unit test (linux_lemond_needs_belong_to_the_pinned_linux_archive), so a future version bump that forgets to re-derive the host-library floor fails the build instead of silently shipping stale needs.
  • The rewritten parse_llamacpp_backend_statuses (row-layout instead of a fixed recipe list) is backed by regression tests traced against both the pre-2026.44 table format and the new Tier/Format column layout, covering exactly the misreadings (unknown recipes' indented rows, dropped update_required/update_available states) the old parser had.
  • The Tier-2 llama.cpp release/asset matching scans the full candidate list via max_by rather than first-match, matches assets by exact generated filename (avoiding a rocm-10.1 vs rocm-10.10 false positive), and degrades to a loud warning plus the revert path on any lookup failure rather than failing silently.
  • backend_alignment.rs and install.rs were refactored to make previously nightly/GPU-only-covered branches (opt-out, unsupported-platform gate, probe-error fallback) unit-testable via injected dependencies, directly addressing AGENTS.md's platform-gated-code coverage rule.
  • The one e2e scenario that does exist for the install-time refusal (lemonade-installs-07) asserts the refusal message and the resulting on-disk state (nothing downloaded/extracted/recorded) in the same test, and is wired to the real production code path (not a parallel test-only path) via a cfg(feature = "e2e-test-hooks")-gated environment override.
  • docs/architecture.md does have a correct entry for the new host_libraries.rs file, satisfying cargo xtask check-architecture-doc.
  • expectations.toml's pre-existing ROCMAI-899/900 xfail rows were updated to flag themselves as likely-stale once the new pin is observed on a nightly run, rather than silently lingering with outdated reasoning.

Deployment notes

  • This changes observable behavior for operators on an older Linux/WSL host: a host below glibc 2.38 / OpenSSL 3.0.0 is now refused up front with a specific message, where it previously appeared to attempt an install or serve and failed later with an opaque "Lemonade server did not become ready."
  • The Lemonade pin moves from 11.5.1 to 2026.44.1 rather than the newer 2026.45.3, because 2026.45.3's Linux server needs glibc 2.43/OpenSSL 3.2.0, which the project's supported Ubuntu 24.04 lanes do not have — documented, not lag.
  • At review time, all CI checks for this commit are green except the self-hosted MI300X GPU e2e lane, still queued; the Strix Halo Windows/WSL2 GPU lanes completed successfully. mergeable_state reads blocked (not dirty): the merge-conflict issue earlier review rounds flagged against prior commits in this PR's history appears resolved, since the current head is itself a merge of origin/main.

What this covered

Every one of the 26 changed files read completely at the PR head (691834a0…3ab5ec04b5), across four independently-run, file-coupled fan-out passes, a dedicated AGENTS.md-adherence pass over the whole diff, a dedicated git-history/prior-convention pass, plus this consolidation and a short design-review pass done directly rather than via further sub-workers. All findings above were independently verified against the checked-out source after the fan-out (Cargo.lock dependency graph, docs/wsl.md's diff-absence, host_libraries.rs's own test asserting GLIBCXX_ is never read) rather than taken on a worker's word alone.

Did not run: cargo test/cargo clippy locally — relied on the green CI result for compilation and test-suite confirmation. Ticket lookups for ROCMAI-899/ROCMAI-900 could not be made (no tracker tool available to this review); their content was not guessed. As part of confirming no prior review from this account already existed at this exact head commit, this review's author necessarily read the text of prior human and automated review rounds on this PR (at earlier commits) before any fan-out worker was launched; none of that text was given to any fan-out worker, and none of it was used to originate, shape, or suppress any finding above — every finding was re-derived independently from the code during the blind fan-out passes — though full independence from that incidental exposure cannot be guaranteed the way it would be from a review that never touched the discussion at all. Not otherwise reconciled against any review thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

# Conflicts:
#	docs/architecture.md
#	engines/lemonade/src/direct_llama.rs
#	engines/lemonade/src/process.rs
#	tests/e2e-cucumber/features/runtime_setup.feature
@rominf
rominf requested a review from juhovainio November 16, 2026 10:42

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough work here: the per-state tests, the stand-in lemond that is never started, and lemonade-installs-07 asserting both the message and the on-disk state are what I'd want to see. One thing blocks merge; the rest are non-blocking.

Blocking

  • The prek cargo-fmt check is red at this head. rustfmt wants the two spawn_lemond_unchecked(...) calls in engines/lemonade/src/process.rs broken one argument per line (inline comments below). The PR body says cargo fmt --check passed locally, so it may be a later edit. cargo fmt --all should fix it.

Non-blocking

  • The install refusal runs before any "already installed" decision, and its text is a fixed "Nothing was downloaded." On a host whose libraries changed after a working install, a plain re-run is refused even though nothing would have been downloaded. Wording that is true on both paths, or deriving the line from the branch taken, would avoid the mismatch.
  • OpenSSL is reported "Missing" purely from ldconfig -p not listing an x86-64 libssl.so.3. Hosts that provide OpenSSL 3 outside the ld cache (LD_LIBRARY_PATH, Nix/Guix-style, user prefix) would be refused although lemond could load, and there is no escape hatch. A doc note or an override would help.
  • host_libraries.rs is compiled on Windows only under cfg(test), so refuse_if_host_too_old, host_libraries_from_cache, newest_version_node and x86_64_library_path are reachable only from tests there. Windows clippy with -D warnings is still pending and wasn't run locally; please confirm there is no dead_code warning.
  • tests/e2e-cucumber/src/expectation.rs still says "the two rows for Lemonade 11.5.1's failed backend alignment". It is outside the diff, but this change makes the claim stale.
  • runtime-16 passes trivially on a ROCm 10.1 lane (pin == active == packaged), and the XPASS of the two Windows xfail rows is unobserved because those scenarios are @nightly. Fine as long as the promised follow-up is tracked.

I couldn't independently verify the archive digests or the objdump glibc/OpenSSL numbers from the diff alone.

Comment thread engines/lemonade/src/process.rs Outdated
log_mode,
);
#[cfg(not(target_os = "linux"))]
spawn_lemond_unchecked(manifest, host, port, log_path, process_env, parent, log_mode)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustfmt wants this call broken one argument per line, which is what the red prek cargo-fmt check is flagging. cargo fmt --all fixes it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 96956bf (cargo fmt --all; the prek check is green on it). It was a later edit, made while resolving the merge with main.

Comment thread engines/lemonade/src/process.rs Outdated
log_mode: LogMode,
) -> Result<LemondChild> {
refuse_lemond_on_too_old_host(manifest, host_libraries)?;
spawn_lemond_unchecked(manifest, host, port, log_path, process_env, parent, log_mode)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same rustfmt failure as above for this spawn_lemond_unchecked call.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 96956bf (cargo fmt --all; the prek check is green on it). It was a later edit, made while resolving the merge with main.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…omment

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf requested a review from juhovainio November 16, 2026 15:34

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found no blocking problems in the code at c03c7b0, but I'm requesting changes for a few open items, most of them carried over from my earlier round. Left as inline comments:

  • The glibc/OpenSSL refusal has no escape hatch. A host with OpenSSL 3 outside the ldconfig cache is refused at install and at every lemond start.
  • The install refusal always says "Nothing was downloaded.", even where a plain re-run on an already-installed host is what gets refused.
  • update_required is mapped to "installable" with nothing showing that the install actually reinstalls.
  • runtime-16 is weaker than before on a 7.14 lane: a CLI that aligns but stops reporting it now passes.
  • The backends parser can drop indented rows after a stray warning line, and one unit test needs ldconfig.

Still open from before: the scope bundling (pin bump together with the refusal and the object dependency) is explained in the body, but a split is still worth considering.

CI is not finished at this head. clippy, the Linux and Windows build-and-test jobs, coverage and CodeQL are pending. Windows clippy in particular needs to land green, since the new host_libraries.rs also compiles in Windows test builds. The OpenSSL probe fix, README wording, OnceLock, manual-testing wording, the llama.cpp tag check and the fmt fix are all addressed.

This is from reading the diff only; I haven't built or run it.

let cache_lists_libc = x86_64_library_path(cache, GLIBC_SONAME).is_some();
let probe = |soname: &str, prefix: &str| match x86_64_library_path(cache, soname) {
Some(path) => read_version(path, prefix).map_or(HostLibrary::Unknown, HostLibrary::Version),
None if cache_lists_libc => HostLibrary::Missing,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing is inferred only from ldconfig -p not listing an x86-64 libssl.so.3. Hosts with OpenSSL 3 outside the cache (LD_LIBRARY_PATH, Nix/Guix, user prefix, stale cache) are refused at install and at every lemond spawn, with no override. Please add a doc note or an opt-out env var.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9764b29 with an opt-out. Setting ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK (any value, read like ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT) turns both refusals, at install and at every lemond start, into a warning: that still names the shortfall and the variable, and Lemonade goes ahead. The refusal message now ends by naming it: "If this host provides these libraries outside the loader cache (LD_LIBRARY_PATH, a Nix or Guix profile), set ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK=1 to skip this check."

  • Unit tests: a_disabled_check_warns_instead_of_refusing (host_libraries.rs) and a_disabled_check_starts_the_pinned_lemond_on_a_host_it_would_refuse (process.rs, the stand-in lemond actually runs).
  • New scenario lemonade-installs-08 (Linux mock lane): old-host hooks plus the variable. It asserts the warning and that the install went past the host check to the space refusal, with nothing downloaded.
  • README, docs/engine-plugins.md and docs/manual-testing.md describe the variable and the cases it is for.

Comment thread engines/lemonade/src/lib.rs Outdated
rocm_deps::LEMONADE_VERSION,
&crate::install::LINUX_LEMOND_NEEDS,
&crate::host_libraries::probe_host_libraries(),
"Nothing was downloaded.",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The refusal runs before any already-installed decision and always ends "Nothing was downloaded." On a host whose libraries changed after a working install, this refuses a plain re-run. Consider wording that is true on both paths, or deriving the outcome from the branch taken.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9764b29 by deriving the sentence from the branch. footprint::pinned_tree_in_place uses the same needs_extraction test the install uses. With no tree, or a tree of another version, the refusal still ends "Nothing was downloaded." With a tree of the pinned version already in place, it ends "Nothing was changed: the Lemonade 2026.44.1 already installed in <folder> stays as it is, but its server cannot start on this host either."

  • Unit test: the_host_refusal_outcome_follows_whether_the_pinned_tree_is_in_place covers all three states.
  • New scenario lemonade-installs-09 (Linux mock lane) plants a pinned tree and re-runs the install on the old-host hooks. It asserts that sentence, that "Nothing was downloaded." is absent, and that every file of the planted tree is byte-for-byte unchanged.

}
let reported_aligned = output
.contains("Aligned Lemonade's ROCm llama.cpp backend to match the installed ROCm SDK");
if !reported_aligned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the pin differs from the packaged one (alignment happened) and the "Aligned ..." line is missing, this branch only checks two stderr strings and passes. Require reported_aligned when pinned != packaged, so a CLI that aligns silently is caught on the 7.14 lanes.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9764b29. The step now splits on whether the installed pin equals the packaged one. If it differs (the 7.14 lanes), the install must report "Aligned Lemonade's ROCm llama.cpp backend ...", so a CLI that aligns silently fails. If it equals (ROCm 10.1), the llama.cpp tag must be the packaged one, and no alignment may be reported in stdout or logged in stderr. The feature comment for runtime-16 says so. It runs only on the Windows Strix Halo nightly lane, so this branch's CI does not exercise it.

}
match token {
"installed" => Some("installed"),
"installable" | "update_required" => Some("installable"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it confirmed that lemonade backends install llamacpp:rocm reinstalls a build that is in update_required state? The fixture is synthetic text; a sentence in the PR or a link to Lemonade's source would settle it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed from Lemonade's source at the v2026.44.1 tag, which I read; I have not run it.

  • In src/cpp/server/system_info.cpp, a version mismatch sets state = "update_required" with action = get_install_command(recipe, backend), i.e. lemonade backends install llamacpp:rocm. That is the command rocm-cli runs, with no extra flags.
  • In src/cpp/server/backend_manager.cpp, BackendManager::install_backend returns early for an installed backend only when offline, when fetching executables is disabled, or when GitHub is unreachable. Otherwise it reinstalls. For rocm-stable it checks backend_update_required(...), removes the existing TheRock folder and fetches it again.

The doc comment on parse_llamacpp_backend_statuses now cites both (9764b29), and the PR body says so.

let (backend, rest) = if continues_recipe {
(first, &tokens[1..])
} else {
first.clone_into(&mut current_recipe);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any column-0 line with 2+ tokens now becomes the current recipe, so a warning line between a llamacpp row and its indented rows drops the continuation backends. Consider only switching recipe on lines that carry a state token.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9764b29. A first-column line now names a recipe only if it carries a state token. Any other first-column line (a warning Lemonade printed mid-table) is skipped and leaves the current recipe as it was, so the indented rows after it still count. a_warning_line_inside_the_table_does_not_end_the_llamacpp_rows puts a warning between the llamacpp row and its indented rocm (update_required) and vulkan rows, and checks that all three are read. It fails with the new check removed.

/// OpenSSL 3 `libssl.so.3` defines `OPENSSL_3.0.0` at least.
#[cfg(all(target_os = "linux", target_arch = "x86_64", target_env = "gnu"))]
#[test]
fn the_running_hosts_glibc_defines_a_readable_version() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.expect("ldconfig -p") fails on hosts without ldconfig (minimal containers, NixOS). Skip the test when it returns None.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9764b29. The test now returns early, printing a skip line, when ldconfig -p cannot be run. A missing cache is the "unknown" case, which the planted-cache tests already cover.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

# Conflicts:
#	tests/e2e-cucumber/expectations.toml
…t a refusal left undone

The check reads only the ldconfig cache, so a host that provides OpenSSL 3
or glibc some other way (LD_LIBRARY_PATH, a Nix or Guix profile) was refused
with no way past it. ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK turns both
refusals into a warning, and the refusal names it (lemonade-installs-08).

A refused re-run where the pinned Lemonade is already installed said
"Nothing was downloaded." about a download it would never have made; it
now says the installed copy stays as it is (lemonade-installs-09).

Also: the backends parser no longer ends a recipe at a first-column line
with no state; runtime-16 requires the install to report an alignment
whenever the pin moved; and the ELF reader test skips on a host without
ldconfig.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

# Conflicts:
#	docs/architecture.md
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

On the split suggestion in your review: I'd keep the pin bump and the host-library refusal in this PR. The refusal guards the pin this PR sets. Its needs (LINUX_LEMOND_NEEDS) are recorded beside the new archive digests and tied to them by a unit test, so a later bump cannot keep stale needs. The object dependency exists only for that check. The problem itself is not new: 11.5.1 also needed glibc 2.38, and an older host got "Lemonade server did not become ready" with it as well. It surfaced while choosing this pin (2026.45.3's glibc 2.43). Splitting now would also restart both reviews. If you still want them split, say so and I'll do it.

This round's fixes are in 9764b29, with replies on each thread.

@rominf
rominf requested a review from juhovainio November 17, 2026 11:16
… outside tests

Their one caller, the host-library refusal, is Linux only, so a Windows
build rejected them as dead code.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (27 files, +1827/-263 against merge-base ee605065). No ticket is claimed closed by this PR; it names ROCMAI-899/900 as background context for pre-existing nightly xfail rows, references the upstream lemonade-sdk/lemonade#4123 issue for why 2026.45.3 isn't pinned, and states it must land before companion PR #941 and is a dependency of #820.

Blocking

  • This PR's own fix shows "every published Lemonade embeddable needs glibc 2.38" is false, but the identical claim is left stale in five other places, two of which are strings printed directly to users — docs/wsl.md:18-20, crates/rocm-core/src/fix/catalog.rs (fix-wsl-5-distro-too-old's rationale, printed by rocm fix), crates/rocm-core/src/diagnose/wsl.rs (the same fix's notes, printed by rocm diagnose), crates/rocm-core/src/examine.rs (WSL_MIN_UBUNTU's doc comment), skills/rocm-doctor/reference.md:188. README.md (this diff) correctly retires "every published build of it needs those versions" because 2026.45.3 needs glibc 2.43/OpenSSL 3.2.0 — but the five sibling surfaces repeat the old blanket wording verbatim, confirmed untouched by this diff. AGENTS.md §5 names exactly this failure mode: the same behavior claim repeating across docs and printed CLI output, each drifting independently.
    Confidence 90 · mechanical · Fix: update the wording in all five places to the same "2026.44.1 needs this; the newest release needs more" nuance README now carries.
  • Branch has unresolved merge conflicts with current main — docs/architecture.md and tests/e2e-cucumber/expectations.toml, both independently edited on main since this branch diverged. Verified directly: git merge-tree origin/main pr-855 reports CONFLICT (content) in both files, matching GitHub's reported mergeable_state: dirty.
    Confidence 100 · mechanical · Fix: rebase onto current main and resolve both conflicts.

Non-blocking

  • Cargo.toml's "adds no crate to the shipped binary" comment is true only for the rocm binary, not for the separately-shipped rocm-engine-lemonade binary — engines/lemonade/Cargo.toml:43-44. rocm already links object 0.37.3 transitively via rocm-dash-daemon/rocm-dash-tui → color-eyre → backtrace (confirmed via Cargo.lock's single deduped object entry), but rocm-engine-lemonade (its own [[bin]], bundled per scripts/build_single_exe_release.py) has neither backtrace nor color-eyre in its dependency graph, so object is a genuinely new linked crate there.
    Confidence 85 · mechanical · Fix: scope the comment to rocm specifically, or acknowledge the addition to the standalone adapter binary.
  • LINUX_LEMOND_NEEDS.archive_sha256 hand-retypes the digest already bound to EMBEDDABLE_LINUX_SHA256 — engines/lemonade/src/install.rs:56, instead of referencing the constant three lines above. The accompanying doc comment flags the duplication itself (a unit test holds the two equal); a shared constant removes the drift risk rather than just detecting it.
    Confidence 72 · mechanical · Fix: archive_sha256: EMBEDDABLE_LINUX_SHA256.

Decisions for the author

  • The lemond-start refusal (process.rs) has no e2e scenario, only unit coverage — answered-by-intent. PR body: "No scenario covers the refusal to start lemond... No lane can reach it: rocm serve runs the install check first, a Linux serve with a packaged llama-server never starts lemond, and the mock lane has no GPU for serve to resolve." Would otherwise have been blocking against AGENTS.md §3 (install path has lemonade-installs-07, start path does not); the PR directly names and justifies the gap, so this is something to confirm rather than a defect.
  • GLIBCXX/libstdc++ is not part of the automated refusal (only GLIBC_*/OPENSSL_* symbols are read in host_libraries.rs) — tradeoff. README's lead-in sentence groups GLIBCXX_3.4.32 with glibc/OpenSSL as "what the Lemonade engine is linked against" before describing the refusal; the refusal-describing sentence itself is correctly scoped to glibc/libssl.so.3 only, but a reader skimming only the first sentence could infer broader coverage than exists. Confirm the scope is intentional.
  • The refusal only applies to a tree matching the exact pinned LEMONADE_VERSION — tradeoff, commented as intentional (a leftover tree from an older pin is replaced by the next install's own check), though the PR's own verification table shows every release from 11.5.1 through 2026.44.1 needs the identical floor.
  • The OpenSSL check only consults the ldconfig cache — tradeoff, documented scope. A host providing OpenSSL 3 outside it (LD_LIBRARY_PATH, Nix/Guix-style, a user prefix) is refused even though lemond could load; no escape hatch exists.
  • The Tier-2 llama.cpp GitHub release lookup is unauthenticated and uncached, made on every alignment attempt that reaches it — degrades safely (logged warning + revert path) on rate-limiting; worth confirming as accepted risk on shared-IP CI fleets.
  • update_required/update_available state mapping is unit-tested against a hand-written fixture only, with no e2e evidence that lemonade backends install actually reinstalls a build in that state — tradeoff, given how impractical reproducing a genuine stale backend is in CI.

Positive signals

  • host_libraries.rs cleanly distinguishes a version, "missing" (cache lists libc.so.6 but not the library), and "unknown" (cache unreadable/unparseable); "unknown never refuses" holds under tracing every fallible step, plus a real-host integration test against this machine's actual libc.so.6/libssl.so.3.
  • spawn_lemond_with_parent is a genuine single choke point: all three production call sites (router serve, the helper pull lemond, backend install/detection) run the refusal-gated spawn before invoking the lemonade binary.
  • parse_llamacpp_backend_statuses's row-layout rewrite is verified against a fixture modeled on Lemonade 2026.44.1's real table (including the new llamacpp-hrx/ds4/thenoise recipes and the inserted Tier/Format columns), with a dedicated test proving a neighboring recipe's indented rows are never misattributed.
  • lemonade-installs-07 asserts both the refusal message and the resulting on-disk state (nothing downloaded/extracted/recorded) in the same test, and the env-var test hooks are applied per-child-process with no cross-scenario pollution risk.
  • Tier-2 release/asset matching scans the full candidate list via max_by rather than first-match and matches by exact generated filename, avoiding a rocm-10.1 vs rocm-10.10 false positive.
  • LINUX_LEMOND_NEEDS is tied to the pinned Linux archive digest by a unit test, so a future bump that forgets to re-derive the host-library floor fails the build instead of shipping stale needs.

Deployment notes

  • This changes observable behavior on an older Linux/WSL host: below glibc 2.38 / OpenSSL 3.0.0, both rocm engines install lemonade and starting lemond now refuse up front with a specific message, rather than silently proceeding to an opaque "Lemonade server did not become ready" later.
  • The Lemonade pin moves from 11.5.1 to 2026.44.1, not the newer 2026.45.3, because 2026.45.3's Linux lemond needs glibc 2.43/OpenSSL 3.2.0, which the project's supported Ubuntu 24.04 lanes do not have — documented, not lag.
  • Required CI (the ci.yml workflow: clippy, prek/fmt, build-and-test, Windows build/test, CodeQL, etc.) is fully green at this head. The separate, non-required "E2E self-hosted" workflow shows one real failure (the MI300X lane): a pre-warm step fails installing vLLM because the installed torch build reports 0 GPU devices for the active SDK — unrelated to this PR's Lemonade changes, and .github/workflows/e2e-selfhosted.yml's own header comment documents this workflow as non-blocking ("none of their check names are in branch protection's required-status-check list").

What this covered

Every one of the 27 changed files read completely at the PR head (ee605065…c03c7b0b), across four independently-run, file-coupled fan-out passes — host-library refusal core (install.rs/host_libraries.rs/process.rs/lib.rs/Cargo.toml/rocm-deps/rocm-core's host_gpu exports/runtime-deps.toml); backend alignment and llama.cpp selection (backend_alignment.rs/llamacpp_backend.rs/direct_llama.rs/runtime_setup.feature/runtime_steps.rs/expectations.toml); the lemonade_installs e2e feature/steps/expectation.rs/crates/e2e-report's markdown; docs and manifest (README/MANIFEST/docs/*.md, plus docs/wsl.md and the CLI-printed strings checked against the diff though outside it) — plus this consolidation done directly: a merge-conflict check via git merge-tree, the Cargo.lock dependency-graph verification behind the two Cargo.toml/install.rs findings, the module-size check (all touched files' production-line counts confirmed under the 1,500 limit, excluding #[cfg(test)] bodies), and CI check-run inspection including job logs for the one failing lane.

Did not run: a dedicated git-history pass beyond a light git log check on touched files (nothing notable surfaced); no cargo test/cargo clippy executed locally (relied on the green required CI for compilation/test confirmation); no ticket retrieval was needed (none named). Not reconciled against any prior review or discussion thread — this review's findings were reached blind, before reading any existing comments on this PR.


🤖 Automated by an AI agent

Comment thread README.md
@@ -46,10 +46,20 @@ serving is Linux or WSL2 only (see [docs/vllm.md](docs/vllm.md)).

The minimum supported Linux release, native or under WSL2, is Ubuntu 24.04. On

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph correctly retires the old "every published build of it needs those versions" framing (2026.45.3 needs more), and the refusal-describing sentence later on is correctly scoped to glibc/libssl.so.3 only. But the identical old framing — "every published Lemonade embeddable requires/is linked against glibc 2.38 (and GLIBCXX_3.4.32)" — is left unedited in several other places this PR doesn't touch, two of which are strings printed directly to users:

  • docs/wsl.md:18-20
  • crates/rocm-core/src/fix/catalog.rs's fix-wsl-5-distro-too-old rationale (printed by rocm fix)
  • crates/rocm-core/src/diagnose/wsl.rs's matching notes (printed by rocm diagnose)
  • crates/rocm-core/src/examine.rs's WSL_MIN_UBUNTU doc comment (says "See docs/wsl.md")
  • skills/rocm-doctor/reference.md:188

AGENTS.md §5 names exactly this failure mode: a behavior claim repeating across docs and printed CLI output, each drifting independently. Since this PR is what invalidates the "every published build" framing (by introducing 2026.45.3 as the counter-example), the sibling surfaces should get the same "2026.44.1 needs this; the newest release needs more" nuance.

Confidence 90 · mechanical

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3779afb. Four of the five places now name the pinned build instead of "every published" one, and add that newer Lemonade releases need more:

  • docs/wsl.md
  • the fix-wsl-5-distro-too-old rationale rocm fix prints (fix/catalog.rs)
  • the matching note rocm diagnose prints (diagnose/wsl.rs)
  • WSL_MIN_UBUNTU's doc comment (examine.rs)

The printed strings do not name a version, so they stay true across bumps. No test pins that text, and cargo xtask catalog --check passes (the generated catalog does not carry it).

I left skills/rocm-doctor/reference.md alone. Its row says "the 2.38 / GLIBCXX_3.4.32 the engines need", which is true of the pinned build and does not make the "every published" claim. It is also a parsed e2e fixture.

On GLIBCXX (from the review body): the README now says that the check reads glibc and OpenSSL only, and that GLIBCXX_3.4.32 (libstdc++) is not checked.

Comment thread engines/lemonade/Cargo.toml Outdated
# Reads the version nodes the host's libc.so.6 and libssl.so.3 define, to
# refuse a host older than the pinned `lemond` needs (`src/host_libraries.rs`).
# 0.37 with only the ELF reader: `backtrace` already links this version into
# `rocm`, so it adds no crate to the shipped binary.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"adds no crate to the shipped binary" is true for the rocm binary (which already links object 0.37.3 transitively via rocm-dash-daemon/rocm-dash-tui → color-eyre → backtrace; Cargo.lock dedupes to a single object 0.37.3 entry, so this crate's new direct dependency resolves to the same one). It's false for the separately-shipped rocm-engine-lemonade binary ([[bin]] below, bundled per scripts/build_single_exe_release.py): that crate's own Cargo.lock dependency list has neither backtrace nor color-eyre, so object is a genuinely new linked crate there.

Confidence 85 · mechanical · Fix: scope the comment to rocm specifically, or acknowledge the small addition to the standalone adapter binary.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3779afb. The comment now says object adds no crate to rocm, where backtrace already links 0.37, and that it is one new crate for the standalone rocm-engine-lemonade binary. The PR body says the same.

#[cfg(any(target_os = "linux", test))]
pub(crate) const LINUX_LEMOND_NEEDS: crate::host_libraries::LemondHostNeeds =
crate::host_libraries::LemondHostNeeds {
archive_sha256: "16f156a7f44a32b20936e76019b24dbccc7b82e60a3acea12e8614da8bca8e90",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hand-retypes the same hex string already bound to EMBEDDABLE_LINUX_SHA256 three lines above, rather than archive_sha256: EMBEDDABLE_LINUX_SHA256. The doc comment above even calls out the duplication ("which a unit test holds equal to EMBEDDABLE_LINUX_SHA256") — a shared constant would remove the drift risk that test exists to guard against, rather than just detecting it after the fact.

Confidence 72 · mechanical · Fix: archive_sha256: EMBEDDABLE_LINUX_SHA256.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm keeping the literal, on purpose. archive_sha256 records which archive the glibc and OpenSSL needs were read from, by objdump -T on that archive's lemond. The unit test holding it equal to EMBEDDABLE_LINUX_SHA256 is the point. A bump changes EMBEDDABLE_LINUX_SHA256, the test fails, and it stays red until someone re-reads the new lemond's needs and updates both glibc/openssl and this digest. With archive_sha256: EMBEDDABLE_LINUX_SHA256 the test could never fail, and a bump would silently keep the old needs. That is the drift the record exists to catch, and docs/engine-plugins.md describes this step of a bump.

2026.45.3 needs glibc 2.43, so the floor belongs to the pinned build.
Reword it in docs/wsl.md, the fix-wsl-5 rationale `rocm fix` prints,
the matching note `rocm diagnose` prints, and WSL_MIN_UBUNTU's doc
comment. Also say that the host-library check does not read GLIBCXX,
and that `object` is one new crate for the standalone Lemonade adapter
binary.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

On the points in the review body that are not inline threads (the review was written against c03c7b0b):

  • Merge conflicts: resolved by merging current main into the branch (ff1673df) rather than rebasing, so the existing approval stands. GitHub reports no conflict at 3779afba.
  • No escape hatch for OpenSSL outside the ldconfig cache: added in 9764b29c. ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK turns both refusals into a warning, and the refusal names it (lemonade-installs-08).
  • update_required reinstall: confirmed against Lemonade's v2026.44.1 source; details are on that thread.
  • The lemond-start refusal has no scenario: confirmed intentional, for the reason in the PR body. Its opt-out path is unit-tested too.
  • Refusal only for a tree of the pinned version: intentional. The recorded needs belong to the pinned archive; a tree from another pin is replaced by the next install, which runs the check first.
  • Unauthenticated Tier-2 release lookup: accepted. It is one call per alignment that reaches Tier 2, and a failure, including a rate limit, logs a warning and takes the revert path, as the PR body says.
  • GLIBCXX not checked: intentional; the README now says so (3779afba).

@rominf
rominf requested a review from juhovainio November 17, 2026 12:05

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through the whole diff at 3779afb. I found nothing that blocks the merge, and the stale "every published build needs glibc 2.38" docs from my earlier round are fixed. Two smaller items are worth addressing before it goes in, both as inline comments below.

  1. The llama.cpp backend table parser can still lose rows. The first-column check treats a line as a recipe row if any later token is a state word, so a warning that happens to contain one ("installed", "update_required") starts a bogus recipe and the indented llamacpp rows after it are dropped. The new warning test only covers a warning with no state word, so it doesn't catch this.
  2. The OpenSSL version the host check reports is the newest version node the library defines, not its release. A real libssl.so.3 would print "3.0.0", and the "3.0.13" row in the unit test is a value the probe can't produce. This can't be hit with the 2026.44.1 pin, but the fixture and the message should be honest about it.

One coverage gap to state in the PR text: "a ROCm 10.1 SDK gets a ROCm backend" is only exercised by the gated nightly Windows scenario (runtime-16), so nothing blocking pre-merge fails if that outcome regresses.

Judgement calls I'm leaving to you: the Tier-2 llama.cpp build is an unchecksummed moving target, the opt-out env var is presence-only (so =0 also disables the check), and I'm fine with keeping the digest literal in LINUX_LEMOND_NEEDS given the test that ties it to the pin.

CI shows nothing failing. The Windows jobs were still pending when I looked, so the Windows compile of the cfg(any(linux, test)) helpers is unconfirmed.

// first-column line (a warning Lemonade printed mid-table) is
// skipped without ending the recipe above it, so the indented
// rows that follow still count for that recipe.
if !rest.iter().any(|token| is_state(token)) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A column-0 line with a state word anywhere after the first token (e.g. Warning: llamacpp:rocm is installed but outdated) becomes the current recipe here, and the indented llamacpp rows after it are dropped. The new warning test has no state word in its text, so it doesn't cover this. Consider requiring a recipe-shaped first token (no trailing :/.) or the state at the Status column, and add a test with a state word in the warning.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 391bea4. The parser now reads the state at the Status column the header names: the first token that starts at or after that column, so a field wider than its column, which pushes the status right, still works. A line whose Status slot holds no state is skipped and leaves the current recipe as it was, even when its text contains installed or update_required. With no header, a first-column line names a recipe only if its first word is shaped like a recipe name (lower case, digits, -, _), so Warning: never does.

Tests:

  • a_warning_line_inside_the_table_does_not_end_the_llamacpp_rows now also has your example, Warning: llamacpp:rocm is installed but outdated, and a line whose first word is recipe-shaped, lemonade is installed but update_required for llamacpp:rocm.
  • a_warning_with_a_state_word_does_not_end_the_llamacpp_rows_without_a_header covers the headerless table.
  • a_field_wider_than_its_column_still_finds_the_state covers the overflow.

Each fails with its target broken: the header's column ignored, any first word accepted as a recipe, and >= changed to ==.

let needed = LibVersion::parse(needed)
.unwrap_or_else(|| panic!("pinned {name} need {needed:?} is not a version"));
match have {
HostLibrary::Version(have) if have < needed => short.push(format!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version printed here is the newest version node the library defines, so a real libssl.so.3 (3.0.13) would report 3.0.0. "this host has OpenSSL 3.0.13" and the unit-test row using it are values the probe can't produce. Unreachable for the current pin, but either word it as "defines OpenSSL x" or note the limit in the doc comment.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, and on a real host it is not even 3.0.0: this Ubuntu 24.04 host's libssl.so.3 (OpenSSL 3.0.13) defines up to OPENSSL_3.0.3, while its libc.so.6 (glibc 2.39) defines up to GLIBC_2.39. Fixed in 391bea4 by naming the node for both libraries, since the probe reads nodes for glibc too: "glibc 2.38 or newer (this host's libc.so.6 provides up to GLIBC_2.31)". The doc comment on host_shortfall and a comment at the format string explain node versus release.

The unit-test rows that stood for Ubuntu 24.04 now use 3.0.3, a value the probe produces there. The process.rs test, the lemonade-installs-07/08/09 steps, docs/manual-testing.md, the README sentence and the PR body example use the new wording.

…sion node a host library provides

A first-column warning with a state word in its text ("... is installed
but outdated") started a bogus recipe and dropped the llama.cpp rows after
it. The state is now read at the Status column the header names; with no
header, only a recipe-shaped first word starts a recipe.

The host-library check reports the newest version node a library defines,
not its release: Ubuntu 24.04's OpenSSL 3.0.13 defines up to OPENSSL_3.0.3.
The message now says "this host's libc.so.6 provides up to GLIBC_2.31",
and test rows use values the probe can produce.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

On the rest of your review at 3779afba:

  • Coverage gap: the PR body now says that the headline outcome, a ROCm 10.1 SDK getting a paired ROCm backend, is exercised only by runtime-16, which is @nightly on the Windows Strix Halo lane, so no pre-merge check fails if it regresses.
  • Windows compile of the cfg(any(linux, test)) helpers: confirmed. windows-build-and-test and the other Windows jobs passed at 3779afba (the previous head had failed on exactly that dead-code error, fixed in 9605dbb6).
  • Opt-out is presence-only: kept, so it reads the same as ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT and ROCM_CLI_DISABLE_TORCH_ALIGNMENT, where any value, 0 included, disables. Its doc comment says so.
  • Unchecksummed Tier-2 build: accepted as stated in the PR body. Lemonade publishes no checksum for those builds.

@rominf
rominf requested a review from juhovainio November 17, 2026 12:33

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through the diff for this PR and didn't find anything blocking. A few small, non-blocking notes are left inline. One more, with low confidence: the host-library check reads only libssl.so.3, and lemond may also need libcrypto.so.3. It's worth confirming that one libssl check is enough.

Two gaps are already disclosed in the PR and look acceptable to me: the start-time lemond refusal is covered by unit tests only, and the headline outcome (a ROCm 10.1 SDK getting a paired ROCm backend) is exercised only on the Windows nightly lane.

This is a read of the diff only. I haven't built it or run the tests locally.

Comment thread docs/testing.md
`update_required`/`update_available` backend states):

```bash
cargo test -p rocm-engine-lemonade llamacpp_backend::tests::parse

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The focused-test filters here don't match the new a_warning_line_inside_the_table_does_not_end_the_llamacpp_rows, a_warning_with_a_state_word_does_not_end_the_llamacpp_rows_without_a_header or a_field_wider_than_its_column_still_finds_the_state tests. Consider adding llamacpp_backend::tests::a_warning and llamacpp_backend::tests::a_field_wider.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 9ee7338: llamacpp_backend::tests::a_warning (matches both warning-line tests) and llamacpp_backend::tests::a_field_wider. The paragraph above the block now mentions the warning-line case too. I checked that each filter runs its tests.

/// Whether the user has turned the host-library check off.
#[cfg(target_os = "linux")]
pub(crate) fn host_library_check_disabled() -> bool {
std::env::var_os(LEMONADE_HOST_LIBRARY_CHECK_DISABLED_ENV).is_some()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presence-only check: ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK=0 (or empty) also disables the check, while the refusal text says to set =1. It's documented as "any value" and consistent with the alignment opt-out, so fine if intended.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended: it reads the same as ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT and ROCM_CLI_DISABLE_TORCH_ALIGNMENT, where presence is the signal and the doc comment says "any value, empty included". The refusal suggests =1 because that is the obvious spelling to type, and it works. No change.

let timeout = Duration::from_secs(15);
/// How many of [`LLAMACPP_ROCM_STABLE_REPO`]'s newest releases Tier 2 searches
/// for one that ships the archive it needs. The repository publishes a build
/// about twice a day, so this covers roughly the last two weeks; one listing page

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"about twice a day ... roughly the last two weeks ... about 2 MB" describes an external service and can't be derived from this repo; the constant's value doesn't depend on it. Consider softening or dropping the figures.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 9ee7338. The doc comment now says only what the repo controls: 30 is one page of the release listing, so one request, and how far back that reaches depends on how often the repository publishes.

# lands means the cause is gone, and the row is deleted then.
[["runtime-lemonade-backend-alignment-reported"]]
when = { effective_engine = "lemonade", os = "windows" }
bug = "ROCMAI-899"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ROCMAI-899 is already Done while this row stays until a nightly shows an XPASS. Fine as a deliberate hold; worth noting who deletes the row.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted in 9ee7338. The comment above the two rows now says the author of the 2026.44.1 pin (me) deletes them in a follow-up once the first Windows nightly shows the XPASS, and that ROCMAI-899 is already closed while the row waits for that evidence.

…gures, say who deletes the xfail rows

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 6803163 Nov 17, 2026
41 checks passed
@rominf
rominf deleted the fix/lemonade-2026.44 branch November 17, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants