Skip to content

fix(lemonade): stop retrying a failed backend alignment on every install (ROCMAI-437) - #1087

Open
rominf wants to merge 7 commits into
mainfrom
ROCMAI-437-fix-lemonade-alignment-retry
Open

rominf wants to merge 7 commits into
mainfrom
ROCMAI-437-fix-lemonade-alignment-retry

Conversation

@rominf

@rominf rominf commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Why

When Lemonade's llama.cpp ROCm backend cannot be aligned to the active ROCm version, the install reverts the pin and force-reinstalls the backend Lemonade ships. The revert leaves the pin different from the active ROCm version, so the next process finds the same mismatch and starts over: nothing recorded that the alignment had already failed.

On a self-hosted E2E lane this turned one upstream outage into a time sink. An upstream Lemonade llama.cpp ROCm asset returned 404 for the pinned tag and a ROCm tarball returned 403 for the latest tag, so every rocm engines install lemonade and every Lemonade rocm serve in the job redid both alignment tiers, downloaded the tier-2 archive again and force-reinstalled the ~3.3 GB packaged backend. In the failing runs that happened five times per job and pushed the 45-minute lanes to their cap. It was not a retry loop inside one command: Lemonade's own downloader gives up quickly, and it was rocm-cli that re-ran the whole alignment in every process. A user whose active ROCm version has no published backend pays the same cost on every serve.

What changes

  • After a failed alignment whose revert install succeeds, the attempt is recorded in .rocm-cli-alignment-failed.json inside the Lemonade runtime tree: the target ROCm version, the pins it started from, the llama.cpp builds tried, the Lemonade version and the time.
  • For 24 hours, an install with the same target ROCm version, pinned version and Lemonade version skips the alignment, deciding so before the backend probe that starts lemond. It installs the pinned backend without forcing a reinstall, so an installed backend is reused, and once that install succeeds prints to stderr: Lemonade's ROCm backend could not be aligned to ROCm <target> <less than an hour | N hours> ago; using the backend pinned for ROCm <pin> instead. Run `rocm engines install lemonade --reinstall` to try again now; otherwise an install run N hours from now or later tries it again. The hours left come from the same 24-hour constant the skip decision uses and round up from the full remaining time, so the line never names a time before the record expires. Nothing retries on its own.
  • If preparing that pinned backend fails after the skip (which can happen before anything is installed, for example when lemond does not start), the error is wrapped with a line built from the record: preparing the backend pinned for ROCm failed, the alignment to ROCm was skipped because it failed ago, and rocm engines install lemonade --reinstall re-extracts the runtime tree without that record, so it tries the alignment again (downloading the backend again). The line is on every error that path can return, claims neither an install nor a backend kind, and says what the command does rather than that it fixes the error beneath, which may be transient. The record is kept and the skip notice is not printed. The backend retry's own --reinstall advice becomes a shared constant (BACKEND_INSTALL_RETRY_FAILED) so the test can pin that whole error chain.
  • The command the notice names does what it says: --reinstall deletes and re-extracts the runtime tree, which removes the record, so the next alignment runs. A new Lemonade version re-extracts the tree too.
  • The record is not keyed on the rocm-cli version. The engine crate sees only the workspace package version, which is not bumped per release, so a key on it would never change; an upgrade therefore does not clear a record, which still expires after 24 hours and is cleared sooner by --reinstall.
  • A different target ROCm version, a record exactly 24 hours old or older, or a failure time in the future runs the alignment as before. A successful alignment removes the record; if that removal fails, a warning says so.
  • The record is consulted only when an install could align. The opt-out (ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT), an unsupported platform, no active ROCm version, an unreadable pin, and a pin that already equals the active ROCm version all return before it and leave it untouched and unmentioned.
  • Past those checks, a record that cannot be read is removed, with a warning that says only that (whether the alignment then runs is still up to the backend probe). If it cannot be removed, the warning gives the removal error and names rocm engines install lemonade --reinstall, which removes it. A record that cannot be written produces a warning and the next install tries again.
  • New test-only variable ROCM_E2E_LEMONADE_ALIGNMENT_TARGET_VERSION overrides the ROCm version the alignment targets. It is compiled in only with the rocm/e2e-test-hooks feature, not in release builds.
  • README.md, docs/engine-plugins.md, docs/manual-testing.md, docs/testing.md and docs/architecture.md describe the record, its 24-hour window, when it is consulted, the error when the install after a skip fails, and the retry command. The gate list is stated once, in docs/engine-plugins.md; docs/testing.md links to it and lists which test covers which state.
  • The pre-push hook's CROSS_CRATE_READS table (xtask/src/prepush.rs) gains the e2e Lemonade install steps file, so a push touching only that file also runs rocm-engine-lemonade's tests, whose file-name test include_str!s it. docs/testing.md lists the entry with the others.

Verification

  • Unit tests cover each state of the record: none; one that skips the alignment (the regression test: the second install calls no alignment tier, installs without forcing, and its notice is asserted together with the pin file), and does so without running the backend probe; another target ROCm version; a record 24 hours old; one that cannot be written; one removed by a later successful alignment, and one whose removal after that alignment fails (warning, alignment kept); one removed by --reinstall (exercised through the real install path); one the opt-out leaves alone; a skipped alignment whose install fails (its whole error chain asserted in three rows: lemond not starting, a transient failure, and the backend retry's own advice, each with the record kept and no notice); and a failed alignment whose revert install also fails, which writes no record.
  • An unreadable record has a test for each place it can be met: every early return listed above leaves it on disk with no output; past them it is removed with exactly one warning whether the probe selects ROCm, fails, or selects Vulkan; the record is already gone and its warning already printed by the time the backend probe runs; and when removal fails, the warning carries the removal error and the --reinstall command. Each asserts the warning together with the on-disk state.
  • The record's own tests cover the window boundary (24 h exactly runs the alignment), mismatched pin and Lemonade version, a future failure time, and the notice's hour wording including sub-second rounding at the end of the window. The file-name test reads the e2e steps' copy of the name from source, so renaming it on either side fails the engine's unit tests, which run in PR CI, rather than only the nightly scenario.
  • Scenario @id:runtime-lemonade-backend-alignment-failure-remembered (runtime-33 in runtime_setup.feature) runs two real installs: a --reinstall that fails to align to 7.99.0, then a plain install that must reuse the pinned backend and print the notice. It is tagged @requires-gpu @requires-bare-metal @requires-engine:lemonade @nightly, so it runs on the GPU nightly Lemonade lanes, not in PR CI. It was not run locally (no GPU on the development host).
  • The unreadable-record warning has no scenario: producing a corrupt record needs writing into the shared runtime tree mid-scenario on a GPU runner. The unit tests above assert that warning with the resulting file state on every probe outcome instead.
  • Passed locally: cargo clippy (workspace, all targets, including the e2e-test-hooks feature and the e2e test target), cargo fmt, cargo xtask check-module-size, check-architecture-doc, lychee, cargo test -p rocm-engine-lemonade, cargo test -p xtask prepush (including the guard every_cross_crate_read_is_covered_by_the_widening_rule, which fails with the new table entry removed), and the e2e-cucumber feature-naming test.
  • cargo test --workspace --all-targets passed apart from failures in crates this change does not touch, caused by the host: a noexec /tmp, a podman-backed docker shim, and temp-file probes timing out on a slow host.
  • Not verified: cargo xtask smoke stopped at its sandbox step because bubblewrap is broken on the development host; cargo xtask coverage and the license-header hook were not run (tools not installed; headers checked by hand). The original failure was not reproduced live, since that needs an upstream backend asset to be missing; the scenario stands in for it by targeting a ROCm version with no published backend.

Known gaps

  • A record can outlive a successful alignment if its removal fails. The user is warned, the record expires after 24 hours, and --reinstall clears it.
  • runtime-33 runs only on the nightly GPU Lemonade lanes, and it assumes Lemonade selects its ROCm backend there. On a bare-metal host where Lemonade selects Vulkan instead, the alignment is never attempted and the scenario's Given fails rather than skips. (WSL2, where Lemonade selects Vulkan, is excluded by @requires-bare-metal.) The e2e harness decides skips only before a scenario runs, from its tags and the host capability probe, so skipping on Lemonade's own backend choice would need a new probe that starts lemond.
  • runtime-33 carries no OS tag (unlike runtime-16, which is Windows-only), so it also runs on the Linux nightly GPU Lemonade lanes. There, a --reinstall whose revert install fails (the Lemonade backend download is known to be flaky on those lanes) exits non-zero and leaves no record, so the Given fails.

Questions for the reviewer

Two tradeoffs in the behaviour above are deliberate but open; please say whether they are the right ones.

  1. Transient failures are remembered for the full 24 hours. The record is written whenever the alignment fails (tier 1 fails, and tier 2 fails or cannot be tried) and the revert install succeeds, whatever made it fail. A transient failure (a network error, a rate limit, or a failed lookup of the latest llama.cpp release) is recorded the same way as "no build exists for this ROCm version", so one transient failure suppresses the alignment for 24 hours. The alternative is to record only failures that are not network-class, or to give those a shorter window. Is a single window for every failure acceptable?
  2. The only way to retry sooner is rocm engines install lemonade --reinstall. It deletes and re-extracts the whole Lemonade runtime tree, so every installed backend goes with it: the ~3.3 GB pinned ROCm backend that the skip exists to reuse is downloaded again (by the retry itself if the alignment fails again), and any other backend is downloaded again when next needed. The alternative is a lighter, record-only retry that clears the record and runs the alignment without touching the tree, which would be a new user-facing option. Is --reinstall an acceptable retry path, or is that option worth adding?

The second review pass judged both reasonable, documented tradeoffs; the first reviewer's answer is pending. ROCMAI-1155 tracks the alternatives (recording the cause and skipping only permanent failures, a lighter retry), together with keying the record on the rocm-cli release version.

Scope

This is the product-side fix. The rest of the ticket is in follow-ups: ROCMAI-1153 (a step-level timeout below the job cap pinned by a contract test, and the failing URL and status plus a "cancelled at cap" line in the job summary), ROCMAI-1154 (deadlines and entry/exit lines for the rocm commands the test harness spawns), and ROCMAI-1155 (recording the failure's cause, a lighter retry, and keying the record on the rocm-cli release version). No overlap with #1009, which retries the queue lane's network fetches and does not touch the alignment.


  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No rows for ROCMAI-437.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation).
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

…all (ROCMAI-437)

When neither alignment tier could match Lemonade's ROCm backend to the
active ROCm version, the revert put back a pin that still differed from
that version and nothing recorded the failure. Every later `rocm engines
install lemonade`, and every `rocm serve` that installs Lemonade, then
ran both tiers again, re-downloaded the tier-2 archive and
force-reinstalled the packaged backend; a nightly GPU lane showed this
five times in a single job.

A failed alignment is now recorded in the Lemonade runtime directory.
For 24 hours an install for the same active ROCm version, pin and
Lemonade version skips the alignment, reuses the backend already
installed, and prints when it failed and how to retry:
`rocm engines install lemonade --reinstall`, which deletes the runtime
tree and the record with it. A successful alignment removes the record;
an unreadable or unwritable record warns and falls back to trying again.

Scenario runtime-33 runs the two installs on the GPU nightly lanes,
targeting an unpublished ROCm version through the e2e-test-hooks
override ROCM_E2E_LEMONADE_ALIGNMENT_TARGET_VERSION.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…-437)

An unreadable record of a failed alignment is now removed, with a warning
that claims only that: whether the alignment then runs is decided later by
the backend probe. When it cannot be removed either, the warning carries the
removal's own error and names the command that removes it.

The skip notice takes its window from the retry constant the decision uses,
rounds up from the full time left so it never names a moment before the
record expires, and no longer implies that anything retries on its own.

Tests now cover every install that returns before the record is consulted,
the removal failures after an unreadable record and after a verified
alignment, sub-second rounding, and the record file name the e2e steps
hard-code. The docs and the runtime-33 assertions are made exact: the
scenario now checks the record and the reverted pin left by the failed
attempt and that the next install keeps both.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-437-fix-lemonade-alignment-retry branch from 6d64a8a to fd5954f Compare November 16, 2026 16:48
…CMAI-437)

Pin the behaviour the review asked about rather than only the code:

- a revert whose install fails writes no record, so the next install
  never reuses a backend that failed to install;
- a record that applies and an unreadable record are both handled before
  the backend probe, which starts lemond, so a skipped install never pays
  for the probe and the unreadable record's warning comes first;
- the engine test now reads the e2e steps' copy of the record file name,
  so a rename on either side fails on every lane instead of only on the
  nightly scenario.

Also document the two llama.cpp tag fields as informational: the skip
decision does not read them.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ROCMAI-437)

The three assertions that pin the `rocm engines install lemonade
--reinstall` advice now name the tests that prove it:
`reinstall_removes_the_record_of_a_failed_alignment` shows that the
reinstall removes the failure record, and
`with_no_record_the_alignment_runs_as_before` shows that the alignment
then runs again. This follows the repository's rule that pinned
remediation text points at the test proving the behavior.

Also re-wrap a docs paragraph without changing its wording.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…AI-437)

The alignment-record test in rocm-engine-lemonade include_str!s the e2e
Lemonade install steps to check that the record file name they hard-code
still matches the engine's. The pre-push hook selects crates by
directory, so a push touching only that steps file never ran the
engine's tests. Add the file to CROSS_CRATE_READS so it also selects
rocm-engine-lemonade, and document the entry beside the others in
docs/testing.md. This also satisfies the guard test
every_cross_crate_read_is_covered_by_the_widening_rule, which fails
on an uncovered cross-crate read.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf marked this pull request as ready for review November 16, 2026 19:12
@rominf
rominf requested a review from a team as a code owner November 16, 2026 19:12
@rominf
rominf requested a review from juhovainio November 16, 2026 19:12

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this and found nothing that is wrong in correctness terms. The fix does what the ticket's root-cause item asks, and the regression test, the runtime-33 scenario and the doc updates are all in place. A few non-blocking things I'd like addressed or answered before merge, so I'm marking it as changes requested:

  • Failure cause isn't recorded. A transient failure (network error, 403) and a permanent one (404) are treated the same, and both suppress alignment for 24h. The only way to retry sooner is --reinstall, which re-extracts the whole runtime tree. Details in the inline comment.
  • Record key ignores the rocm-cli version. Low impact, but a new release with a fix for the failure would still be skipped until the 24h passes.
  • Size and duplication. About 120 production lines against about 1100 test lines, and the same state list is repeated in three docs, which is a drift risk.
  • Nits: the target_unpublished_rocm_version doc comment misdescribes when the env var is re-armed, and runtime-33 fails rather than skips if Lemonade picks Vulkan on a lane.
  • Ticket scope: ROCMAI-437 items 2-5 (step timeout below the job cap, failing URL and status in the job summary, cap-cancel distinguishable from a test failure, Windows teardown bound) are not in this PR. Please confirm a follow-up PR/ticket exists.
  • Not verified: whether rocm/e2e-test-hooks forwards to the lemonade engine, since the Cargo.toml files aren't in the diff. runtime-33 also hasn't run on hardware because the nightly GPU lanes didn't run.

lemonade_version: manifest.version.clone(),
failed_at_unix_secs: unix_secs(now()),
};
if let Err(error) = write_alignment_failure(&manifest.runtime_dir, &failure) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The record is written for any failed alignment, whether the cause was a permanent 404 or a transient network error, and either one suppresses alignment for 24h. Since align only returns a bool the cause is not available here, but it would be worth either recording it and skipping only on the permanent kind, or offering a retry cheaper than --reinstall (which re-extracts the whole runtime tree). The PR body asks this as an open question, so this is just my vote for handling it before or soon after merge.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept as it is, deliberately: one 24 h window for every failure, with --reinstall as the only faster retry. As you note, align returns only a bool, so skipping only permanent failures means carrying the failure's cause out of both tiers first, and a record-only retry would be a new user-facing option; the 24 h bound caps what a transient failure can cost in the meantime. The second review pass reached the same view, that both are reasonable, documented tradeoffs. Recording the cause and skipping only the permanent kind (or a lighter retry than --reinstall) is tracked as ROCMAI-1155. No code change for this one.

// Not forced: the backend the revert installed is the one wanted.
// The line is printed only once that install has succeeded, since
// it says the pinned backend is in use.
fallback_install(manifest, false)?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skip path: fallback_install(manifest, false) runs before the notice, and an install failure returns the error with no notice and the record kept. That is covered by a_skipped_alignment_whose_install_fails_reports_the_error_and_claims_nothing. If that install keeps failing, the record also keeps skipping realignment, with no way short of --reinstall to recover. This is probably fine, but the error message could mention the retry command.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 008ba67. On the skip path the install is now wrapped with a context built from the record (AlignmentFailure::skipped_install_failure in alignment_record.rs), so the error says preparing the pinned backend failed, that the alignment was skipped because it failed N ago, and that rocm engines install lemonade --reinstall re-extracts the runtime tree without the record, so the alignment is tried again and the backend downloaded again. It claims neither an install nor a backend kind, because the failure can come before anything is installed (for example lemond not starting), and it says what the command does rather than promising it fixes the error beneath.

a_skipped_alignment_whose_install_fails_reports_the_error_and_claims_nothing now has three rows (lemond does not start, a transient failure, the backend retry's own advice), each asserting the whole error chain together with the kept record and no notice.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this independently: the full diff, the complete content of alignment_record.rs and backend_alignment.rs, the e2e scenario and step definitions, and the five updated docs.

No blocking findings. The core mechanism checks out: the 24h window arithmetic (including the "hours left" rounding), the five early-return gates that leave the record untouched before it's consulted, and the match logic are all correctly implemented and covered by dedicated tests. CI, CodeQL, and the security scan all pass on this commit. The one red check (E2E tests (MI300X, current)) fails inside the shared cargo xtask e2e-prewarm step on a vLLM torch/device mismatch before any Lemonade scenario runs — unrelated to this diff, which only touches the Lemonade engine.

This largely confirms what @juhovainio already raised, so rather than re-litigate I'll add specifics and one correction:

  • The "same state list repeated in three docs" is actually two docs — docs/engine-plugins.md:86-93 and docs/testing.md:2417-2424 (inline comments on both). The only other place with a condensed version of the same list is a Gherkin comment (runtime_setup.feature:725-727), not a doc.
  • On the skip-path error-handling gap flagged at backend_alignment.rs:233: confirmed — the propagated error only carries --reinstall advice when it comes from the one retry path that already prints it; an earlier-chain failure surfaces no record-aware remedy at all.
  • The record-key point extends to the rocm-cli version too (alignment_record.rs:125): a rocm-cli release fixing the underlying bug would still be suppressed by a pre-existing record for up to 24h.

One more, independent of the above: the target_unpublished_rocm_version doc comment (lemonade_install_steps.rs:417-419) still misdescribes which step re-arms the override (detail inline).

On the two open questions in the PR description: I don't see a strong case against either the single 24h window covering both transient and permanent failures, or --reinstall being the only faster retry path — both are reasonable, bounded, and documented tradeoffs. Worth confirming deliberately rather than treating as a gap.

Good test coverage for a ~120-line production change — every record state has a dedicated case, including both removal-failure paths and sub-second rounding at the window boundary.

Given the standing changes-requested review on this exact commit covers much of the same ground and hasn't been addressed by a new commit yet, I'm leaving this as a comment rather than approving or re-requesting changes myself.

🤖 Automated by an AI agent

// Not forced: the backend the revert installed is the one wanted.
// The line is printed only once that install has succeeded, since
// it says the pinned backend is in use.
fallback_install(manifest, false)?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, echoing the existing comment here: when fallback_install fails after a skip decision, the error returned via ? doesn't always carry the --reinstall remedy — only when the failure originates from the one retry path that already prints it. An earlier-chain failure surfaces no mention of the record or how to clear it, even though the record is left untouched. Non-blocking — this is explicitly pinned by a_skipped_alignment_whose_install_fails_reports_the_error_and_claims_nothing — but worth folding the retry-command advice into every error this branch can surface, not just the one path through it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 008ba67 for every error this branch can return, not only the retry path: the record-aware context is attached to fallback_install(manifest, false) itself (.with_context(|| failure.skipped_install_failure(now))), so it is on the chain whether the failure comes from starting lemond, the backend listing, or the backend retry, whose own --reinstall advice stays beneath it. The test pins the full chain for those three shapes, with the record kept and no notice; details in the reply to the first comment on this line.

lemonade_version: &str,
now: SystemTime,
) -> bool {
self.target_rocm_version == target_rocm_version

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

matches keys the record on target/pinned ROCm version and Lemonade version, but not on the rocm-cli version itself. A rocm-cli release that fixes the underlying alignment bug would still be skipped by a pre-existing record for up to 24h, since nothing here invalidates the record on a version bump. Low impact given the 24h bound and the --reinstall escape hatch, but worth a one-line mention in the doc comment if that's intentional.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed, deliberately, and 008ba67 now says so on skips_alignment. The Lemonade engine crate sees only the workspace package version, which is 0.1.0 and not bumped per release (apps/rocm/build.rs explains this); the per-release ROCM_CLI_VERSION_NUMBER exists only in apps/rocm. Keying the record on the version the engine can see would never change between releases, so it would look like a fix without being one. The doc comment (and docs/engine-plugins.md) state that the record is not keyed on the rocm-cli version, so an upgrade does not clear it: it expires after 24 h and --reinstall clears it sooner. Keying on the real release version is added to ROCMAI-1155.

Comment thread docs/engine-plugins.md Outdated
different active ROCm version, a new Lemonade version (which also re-extracts
the tree), or a record 24 hours old or older (or dated in the future) runs the
alignment as before, and an alignment that succeeds removes the record. The
record is consulted only when an install could align: not with the opt-out set,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph (the full gate-condition list plus the unreadable/unwritable-record warning behavior) is repeated near-verbatim in docs/testing.md:2417-2424. Both are currently accurate against backend_alignment.rs's actual gate order, but nothing links them, so a future change to the gating logic has to be mirrored by hand in two places. Consider stating the list once here and having testing.md link to this section instead of re-enumerating (testing.md already links to section headings elsewhere for similar content).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 008ba67: the gate conditions and the unreadable/unwritable-record behaviour are now stated once, in this section ("Lemonade backend alignment on engine install"). docs/testing.md links here and keeps only which test covers which state, naming each test.

Comment thread docs/testing.md Outdated
(`reinstall_removes_the_record_of_a_failed_alignment` in
[install.rs](../engines/lemonade/src/install.rs)), one the opt-out leaves alone,
and a skipped alignment whose install fails (the error is reported and no notice
is printed). The unreadable record has a row for each place it can be met: every

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same list as docs/engine-plugins.md:86-93 — see that comment. Note the two lists group the conditions slightly differently (this one lists "no active ROCm version" and "an unreadable pin" as two separate items; engine-plugins.md bundles them into one clause). Both are correct — they are genuinely two distinct branches in the source — but the difference itself is a symptom of there being no single source of truth for this list.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 008ba67 together with the engine-plugins.md comment: this section no longer re-enumerates the gates. It links to "Lemonade backend alignment on engine install" in docs/engine-plugins.md, the single statement of them, and lists only the test for each state (the early exits are one table test with a row per exit).

/// into is shared with every other scenario on the runner.
const UNPUBLISHED_ROCM_VERSION: &str = "7.99.0";

/// The override applies to one command only (`run_rocm_with_scenario_env`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc comment still says "each step that runs rocm sets it again first," but tracing the three steps that use the override, neither step that actually runs rocm re-arms it before its own invocation — the preceding Given step re-arms it only after its own run, for the step that follows. Priming always happens in the immediately-preceding step, not in the step that runs rocm. Suggested reword: "the step that is about to run rocm either sets this itself or relies on an immediately-preceding step having set it; a step that runs rocm and is followed by another such step must re-arm it after its own run."

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 008ba67 with essentially your wording: "the step that is about to run rocm either sets it itself or relies on an immediately preceding step having set it, and a step that runs rocm and is followed by another such step re-arms it after its own run", plus the concrete case (the reinstall Given re-arms it for the shared When the user installs the Lemonade engine).

…de-alignment-retry

Main changed Tier 2 of the backend alignment to look up the newest
llama.cpp build that ships an archive for the active ROCm version, so the
lookup now takes the target version. The failed-alignment record keeps
storing that lookup's result, and the tests this branch added pass the
target-taking lookup.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…nstall fails (ROCMAI-437)

After a recorded failure skips the backend alignment, a failing install of
the pinned backend used to surface only the error beneath, leaving no hint
that the alignment had been skipped or how to try it again. The error now
says the alignment was skipped and why, and what
`rocm engines install lemonade --reinstall` does: it re-extracts the
runtime tree without the record, so it tries the alignment again and
downloads the backend again. The line says only what that command does,
not that it fixes the error beneath, which may be transient.

The record is deliberately not keyed on the rocm-cli version: the engine
cannot see the per-release version, so it expires after 24 hours and
`--reinstall` clears it sooner.

The gate list is stated once in docs/engine-plugins.md and docs/testing.md
links to it; the e2e override comment describes which step re-arms it; the
backend retry advice is a shared constant so the test can pin the full
error chain against it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for both passes. Since they were on f247212, the branch has two new commits: b7e01d6 merges main (conflict resolution only: Tier 2 now uses main's newest_tag_with_asset lookup, which takes the target ROCm version, and the record stores that lookup's result), and 008ba67 carries the review fixes. Each inline point is answered in its own thread; the rest of the two review bodies:

  • Failure cause / cheaper retry and the rocm-cli version key: both kept, with the reasons in the threads. Both alternatives are tracked in ROCMAI-1155.
  • Size and duplication: the duplicated docs are fixed; the gates are stated once in docs/engine-plugins.md and docs/testing.md links there. The test volume is the per-state coverage AGENTS.md asks for (a test per state of the record, asserting the message together with the resulting state), so I have kept it.
  • runtime-33 failing rather than skipping when Lemonade picks Vulkan: the e2e harness decides skips only before a scenario runs, from its tags and the host capability probe (tests/e2e-cucumber/src/expectation.rs); there is no way to skip from inside a step. Skipping on Lemonade's own backend choice would need a new capability probe that starts lemond, so this stays a documented gap (listed under Known gaps in the description).
  • Ticket items 2-5: in follow-ups. ROCMAI-1153: a step timeout below the job cap, and the failing URL/status plus a cap-cancel line in the job summary. ROCMAI-1154: deadlines and entry/exit lines for the rocm commands the harness spawns; the 17.6-minute silence on Windows was most likely inside a rocm serve the harness runs with no deadline, rather than in teardown.
  • rocm/e2e-test-hooks forwarding: it does. apps/rocm/Cargo.toml lists rocm-engine-lemonade/e2e-test-hooks in its e2e-test-hooks feature.
  • Hardware: runtime-33 has still not run on hardware; it is tagged for the nightly GPU Lemonade lanes only.

@rominf
rominf requested a review from juhovainio November 17, 2026 17:37

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (merge-base bfff5216…head 008ba67b, 14 files, +1721/-15).
Implements the ROCMAI-437 product-side fix: records a failed Lemonade llama.cpp ROCm-backend alignment and skips re-attempting it for 24 hours, per the PR's own detailed description. The description's "Scope" section explicitly names three follow-up tickets (ROCMAI-1153/1154/1155) for what is deliberately not in this PR, and its "Questions for the reviewer" section discloses two open tradeoffs (a single 24h window for both transient and permanent failures; --reinstall as the only faster retry path) rather than presenting them as resolved — matches the diff.

Blocking

None

Non-blocking

  • Missing integration-level test coverage for mismatched pin / mismatched Lemonade-version skip-decision inputs — engines/lemonade/src/backend_alignment.rs:232 (inline comment)
    Confidence 72 · test-coverage

Decisions for the author

  • Transient and permanent alignment failures share one 24h suppression window — tradeoff. The PR body raises this itself as an open question; a prior review round found no strong case against it either. ROCMAI-1155 (named in the PR's own "Scope" section) tracks recording the failure's cause and a narrower window as a follow-up. Reasonable to leave as is for this PR.
  • --reinstall is the only faster retry path, and it re-downloads the whole ~3.3GB backend tree — tradeoff. Same disclosure; a lighter, record-only retry is named as a follow-up candidate in ROCMAI-1155. Reasonable to leave as is for this PR.
  • The record is not keyed on the rocm-cli release version — tradeoff, explicitly disclosed and reasoned in the PR body (the workspace package version isn't bumped per release, so keying on it would be inert) and tracked under the same follow-up ticket.

Positive signals

  • The fix-23-style duplication risk this class of change usually carries is avoided: the retry advice shown on a skip-path install failure is a single shared constant (BACKEND_INSTALL_RETRY_FAILED) reused by both the original backend-retry path and this PR's new skip-path error chain, so the two can't drift apart silently.
  • The skip decision is made and the record is read/removed before the backend probe that spawns lemond, and every one of the five early-return gates (opt-out, unsupported platform, no active SDK version, unreadable/absent pin, pin-already-matches-target) leaves the record file untouched — verified directly against the code, not just the doc comment's claim.
  • The remediation command the skip notice names (rocm engines install lemonade --reinstall) is verified end-to-end to actually clear the condition it's printed for: a dedicated test plants a record, confirms a plain install keeps it, then confirms --reinstall removes it and re-extracts the tree.
  • A prior review round's specific gap — "the propagated error only carries --reinstall advice when it comes from the one retry path that already prints it; an earlier-chain failure surfaces no record-aware remedy at all" — is fixed in this PR's final commit (008ba67b), independently re-verified by tracing the error-wrapping now applied uniformly across all three described failure sub-variants (lemond not starting, a transient failure, and the backend retry's own advice), each with a dedicated test.
  • New markdown links are all relative and cite by symbol, not line number; the gate list is stated once (in docs/engine-plugins.md) with docs/testing.md linking to it rather than repeating it, closing a documentation-duplication point a prior review round raised.
  • The include_str! cross-crate check (the e2e steps' hard-coded record filename is read from source and compared) runs as part of cargo test on every PR via the new prepush.rs CROSS_CRATE_READS entry, not only on the nightly GPU lane — so a filename rename on either side is caught long before the one e2e scenario that exercises the real file would run.

Deployment notes

None — purely additive engine-internal behavior (a new on-disk record file and a skip path), no API or CLI-surface change, no migration.

What this covered

Full diff (merge-base bfff5216b50ba997e380a90daf3d356dfb44e3f0…head 008ba67bac4c5c4a497b5ee4d9c4e6722b62f84b), all 14 changed files read completely, across five independent fan-out passes grouped by coupling: backend_alignment.rs core logic (full 3490-line file); the integration points (install.rs, lib.rs, llamacpp_backend.rs); the e2e scenario/step-definition/prepush wiring; the five touched docs, cross-checked against the actual runtime behavior; and agent-instruction adherence (root AGENTS.md) plus version history on the touched files. All five passes' candidates were verified against the source before inclusion; four of the five returned no findings at all, and the one surviving finding is a test-coverage gap at a function whose underlying boolean logic is already unit-tested elsewhere, not a live defect.

CI at the current head: all 39 checks completed, every one success or an expected skipped (the Skill checks (skillscope) job), no failures.

Reconciled against the existing review thread: a prior CHANGES_REQUESTED round (at an earlier head, f247212a) raised several points — failure-cause not recorded, record key ignoring rocm-cli version, doc duplication across three files, a stale doc comment, unconfirmed ticket scope, and two unverified items. A second round on the same commit judged the two open tradeoffs reasonable without re-requesting changes itself. Checked each independently against the current head rather than taking the PR body's account on faith: the doc-duplication point is fixed (the gate list is now stated once); the stale target_unpublished_rocm_version doc-comment nit was not reproduced at this head; the ticket-scope question is now answered directly in the PR's own "Scope" section (three named follow-up tickets); the skip-path error-handling gap a later comment flagged is fixed in the final commit as described above. The remaining points (failure cause, record key, retry weight) are now explicitly disclosed, reasoned tradeoffs with a named tracking ticket rather than silent gaps, which this review treats as resolved via disclosure rather than as defects. Not reconciled against anything beyond the submitted review bodies and the PR's own description.


🤖 Automated by an AI agent

match read_alignment_failure(&manifest.runtime_dir) {
StoredAlignmentFailure::Recorded(failure) => {
let now = now();
if failure.skips_alignment(&target_version, &pinned_version, &manifest.version, now) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: missing integration-level test coverage for mismatched pin / mismatched Lemonade-version skip-decision inputs. Every test exercising a record in this file uses a fixture where the on-disk pin equals the record's pinned_rocm_version and manifest.version always equals the record's lemonade_version — only the target_rocm_version dimension is varied here. A regression that stopped comparing pinned_version or manifest.version correctly in this wiring (as opposed to the pure skips_alignment function, which is unit-tested for this in alignment_record.rs) would not be caught by any test in this file.

Confidence 72 · test-coverage · Fix: add 1-2 integration tests varying the on-disk pin (keeping target/lemonade-version fixed) and varying manifest.version (keeping target/pin fixed), asserting align_calls == 1 in each case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants