Repository navigation
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
…all (ROCMAI-437) When neither alignment tier could match Lemonade's ROCm backend to the active ROCm version, the revert put back a pin that still differed from that version and nothing recorded the failure. Every later `rocm engines install lemonade`, and every `rocm serve` that installs Lemonade, then ran both tiers again, re-downloaded the tier-2 archive and force-reinstalled the packaged backend; a nightly GPU lane showed this five times in a single job. A failed alignment is now recorded in the Lemonade runtime directory. For 24 hours an install for the same active ROCm version, pin and Lemonade version skips the alignment, reuses the backend already installed, and prints when it failed and how to retry: `rocm engines install lemonade --reinstall`, which deletes the runtime tree and the record with it. A successful alignment removes the record; an unreadable or unwritable record warns and falls back to trying again. Scenario runtime-33 runs the two installs on the GPU nightly lanes, targeting an unpublished ROCm version through the e2e-test-hooks override ROCM_E2E_LEMONADE_ALIGNMENT_TARGET_VERSION. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…-437) An unreadable record of a failed alignment is now removed, with a warning that claims only that: whether the alignment then runs is decided later by the backend probe. When it cannot be removed either, the warning carries the removal's own error and names the command that removes it. The skip notice takes its window from the retry constant the decision uses, rounds up from the full time left so it never names a moment before the record expires, and no longer implies that anything retries on its own. Tests now cover every install that returns before the record is consulted, the removal failures after an unreadable record and after a verified alignment, sub-second rounding, and the record file name the e2e steps hard-code. The docs and the runtime-33 assertions are made exact: the scenario now checks the record and the reverted pin left by the failed attempt and that the next install keeps both. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
6d64a8a to
fd5954f
Compare
…CMAI-437) Pin the behaviour the review asked about rather than only the code: - a revert whose install fails writes no record, so the next install never reuses a backend that failed to install; - a record that applies and an unreadable record are both handled before the backend probe, which starts lemond, so a skipped install never pays for the probe and the unreadable record's warning comes first; - the engine test now reads the e2e steps' copy of the record file name, so a rename on either side fails on every lane instead of only on the nightly scenario. Also document the two llama.cpp tag fields as informational: the skip decision does not read them. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ROCMAI-437) The three assertions that pin the `rocm engines install lemonade --reinstall` advice now name the tests that prove it: `reinstall_removes_the_record_of_a_failed_alignment` shows that the reinstall removes the failure record, and `with_no_record_the_alignment_runs_as_before` shows that the alignment then runs again. This follows the repository's rule that pinned remediation text points at the test proving the behavior. Also re-wrap a docs paragraph without changing its wording. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…AI-437) The alignment-record test in rocm-engine-lemonade include_str!s the e2e Lemonade install steps to check that the record file name they hard-code still matches the engine's. The pre-push hook selects crates by directory, so a push touching only that steps file never ran the engine's tests. Add the file to CROSS_CRATE_READS so it also selects rocm-engine-lemonade, and document the entry beside the others in docs/testing.md. This also satisfies the guard test every_cross_crate_read_is_covered_by_the_widening_rule, which fails on an uncovered cross-crate read. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
I went through this and found nothing that is wrong in correctness terms. The fix does what the ticket's root-cause item asks, and the regression test, the runtime-33 scenario and the doc updates are all in place. A few non-blocking things I'd like addressed or answered before merge, so I'm marking it as changes requested:
- Failure cause isn't recorded. A transient failure (network error, 403) and a permanent one (404) are treated the same, and both suppress alignment for 24h. The only way to retry sooner is
--reinstall, which re-extracts the whole runtime tree. Details in the inline comment. - Record key ignores the rocm-cli version. Low impact, but a new release with a fix for the failure would still be skipped until the 24h passes.
- Size and duplication. About 120 production lines against about 1100 test lines, and the same state list is repeated in three docs, which is a drift risk.
- Nits: the
target_unpublished_rocm_versiondoc comment misdescribes when the env var is re-armed, and runtime-33 fails rather than skips if Lemonade picks Vulkan on a lane. - Ticket scope: ROCMAI-437 items 2-5 (step timeout below the job cap, failing URL and status in the job summary, cap-cancel distinguishable from a test failure, Windows teardown bound) are not in this PR. Please confirm a follow-up PR/ticket exists.
- Not verified: whether
rocm/e2e-test-hooksforwards to the lemonade engine, since the Cargo.toml files aren't in the diff. runtime-33 also hasn't run on hardware because the nightly GPU lanes didn't run.
| lemonade_version: manifest.version.clone(), | ||
| failed_at_unix_secs: unix_secs(now()), | ||
| }; | ||
| if let Err(error) = write_alignment_failure(&manifest.runtime_dir, &failure) { |
There was a problem hiding this comment.
The record is written for any failed alignment, whether the cause was a permanent 404 or a transient network error, and either one suppresses alignment for 24h. Since align only returns a bool the cause is not available here, but it would be worth either recording it and skipping only on the permanent kind, or offering a retry cheaper than --reinstall (which re-extracts the whole runtime tree). The PR body asks this as an open question, so this is just my vote for handling it before or soon after merge.
There was a problem hiding this comment.
Kept as it is, deliberately: one 24 h window for every failure, with --reinstall as the only faster retry. As you note, align returns only a bool, so skipping only permanent failures means carrying the failure's cause out of both tiers first, and a record-only retry would be a new user-facing option; the 24 h bound caps what a transient failure can cost in the meantime. The second review pass reached the same view, that both are reasonable, documented tradeoffs. Recording the cause and skipping only the permanent kind (or a lighter retry than --reinstall) is tracked as ROCMAI-1155. No code change for this one.
| // Not forced: the backend the revert installed is the one wanted. | ||
| // The line is printed only once that install has succeeded, since | ||
| // it says the pinned backend is in use. | ||
| fallback_install(manifest, false)?; |
There was a problem hiding this comment.
Skip path: fallback_install(manifest, false) runs before the notice, and an install failure returns the error with no notice and the record kept. That is covered by a_skipped_alignment_whose_install_fails_reports_the_error_and_claims_nothing. If that install keeps failing, the record also keeps skipping realignment, with no way short of --reinstall to recover. This is probably fine, but the error message could mention the retry command.
There was a problem hiding this comment.
Fixed in 008ba67. On the skip path the install is now wrapped with a context built from the record (AlignmentFailure::skipped_install_failure in alignment_record.rs), so the error says preparing the pinned backend failed, that the alignment was skipped because it failed N ago, and that rocm engines install lemonade --reinstall re-extracts the runtime tree without the record, so the alignment is tried again and the backend downloaded again. It claims neither an install nor a backend kind, because the failure can come before anything is installed (for example lemond not starting), and it says what the command does rather than promising it fixes the error beneath.
a_skipped_alignment_whose_install_fails_reports_the_error_and_claims_nothing now has three rows (lemond does not start, a transient failure, the backend retry's own advice), each asserting the whole error chain together with the kept record and no notice.
juhovainio
left a comment
There was a problem hiding this comment.
I reviewed this independently: the full diff, the complete content of alignment_record.rs and backend_alignment.rs, the e2e scenario and step definitions, and the five updated docs.
No blocking findings. The core mechanism checks out: the 24h window arithmetic (including the "hours left" rounding), the five early-return gates that leave the record untouched before it's consulted, and the match logic are all correctly implemented and covered by dedicated tests. CI, CodeQL, and the security scan all pass on this commit. The one red check (E2E tests (MI300X, current)) fails inside the shared cargo xtask e2e-prewarm step on a vLLM torch/device mismatch before any Lemonade scenario runs — unrelated to this diff, which only touches the Lemonade engine.
This largely confirms what @juhovainio already raised, so rather than re-litigate I'll add specifics and one correction:
- The "same state list repeated in three docs" is actually two docs —
docs/engine-plugins.md:86-93anddocs/testing.md:2417-2424(inline comments on both). The only other place with a condensed version of the same list is a Gherkin comment (runtime_setup.feature:725-727), not a doc. - On the skip-path error-handling gap flagged at
backend_alignment.rs:233: confirmed — the propagated error only carries--reinstalladvice when it comes from the one retry path that already prints it; an earlier-chain failure surfaces no record-aware remedy at all. - The record-key point extends to the rocm-cli version too (
alignment_record.rs:125): a rocm-cli release fixing the underlying bug would still be suppressed by a pre-existing record for up to 24h.
One more, independent of the above: the target_unpublished_rocm_version doc comment (lemonade_install_steps.rs:417-419) still misdescribes which step re-arms the override (detail inline).
On the two open questions in the PR description: I don't see a strong case against either the single 24h window covering both transient and permanent failures, or --reinstall being the only faster retry path — both are reasonable, bounded, and documented tradeoffs. Worth confirming deliberately rather than treating as a gap.
Good test coverage for a ~120-line production change — every record state has a dedicated case, including both removal-failure paths and sub-second rounding at the window boundary.
Given the standing changes-requested review on this exact commit covers much of the same ground and hasn't been addressed by a new commit yet, I'm leaving this as a comment rather than approving or re-requesting changes myself.
🤖 Automated by an AI agent
| // Not forced: the backend the revert installed is the one wanted. | ||
| // The line is printed only once that install has succeeded, since | ||
| // it says the pinned backend is in use. | ||
| fallback_install(manifest, false)?; |
There was a problem hiding this comment.
Confirmed, echoing the existing comment here: when fallback_install fails after a skip decision, the error returned via ? doesn't always carry the --reinstall remedy — only when the failure originates from the one retry path that already prints it. An earlier-chain failure surfaces no mention of the record or how to clear it, even though the record is left untouched. Non-blocking — this is explicitly pinned by a_skipped_alignment_whose_install_fails_reports_the_error_and_claims_nothing — but worth folding the retry-command advice into every error this branch can surface, not just the one path through it.
There was a problem hiding this comment.
Fixed in 008ba67 for every error this branch can return, not only the retry path: the record-aware context is attached to fallback_install(manifest, false) itself (.with_context(|| failure.skipped_install_failure(now))), so it is on the chain whether the failure comes from starting lemond, the backend listing, or the backend retry, whose own --reinstall advice stays beneath it. The test pins the full chain for those three shapes, with the record kept and no notice; details in the reply to the first comment on this line.
| lemonade_version: &str, | ||
| now: SystemTime, | ||
| ) -> bool { | ||
| self.target_rocm_version == target_rocm_version |
There was a problem hiding this comment.
matches keys the record on target/pinned ROCm version and Lemonade version, but not on the rocm-cli version itself. A rocm-cli release that fixes the underlying alignment bug would still be skipped by a pre-existing record for up to 24h, since nothing here invalidates the record on a version bump. Low impact given the 24h bound and the --reinstall escape hatch, but worth a one-line mention in the doc comment if that's intentional.
There was a problem hiding this comment.
Not changed, deliberately, and 008ba67 now says so on skips_alignment. The Lemonade engine crate sees only the workspace package version, which is 0.1.0 and not bumped per release (apps/rocm/build.rs explains this); the per-release ROCM_CLI_VERSION_NUMBER exists only in apps/rocm. Keying the record on the version the engine can see would never change between releases, so it would look like a fix without being one. The doc comment (and docs/engine-plugins.md) state that the record is not keyed on the rocm-cli version, so an upgrade does not clear it: it expires after 24 h and --reinstall clears it sooner. Keying on the real release version is added to ROCMAI-1155.
| different active ROCm version, a new Lemonade version (which also re-extracts | ||
| the tree), or a record 24 hours old or older (or dated in the future) runs the | ||
| alignment as before, and an alignment that succeeds removes the record. The | ||
| record is consulted only when an install could align: not with the opt-out set, |
There was a problem hiding this comment.
This paragraph (the full gate-condition list plus the unreadable/unwritable-record warning behavior) is repeated near-verbatim in docs/testing.md:2417-2424. Both are currently accurate against backend_alignment.rs's actual gate order, but nothing links them, so a future change to the gating logic has to be mirrored by hand in two places. Consider stating the list once here and having testing.md link to this section instead of re-enumerating (testing.md already links to section headings elsewhere for similar content).
There was a problem hiding this comment.
Fixed in 008ba67: the gate conditions and the unreadable/unwritable-record behaviour are now stated once, in this section ("Lemonade backend alignment on engine install"). docs/testing.md links here and keeps only which test covers which state, naming each test.
| (`reinstall_removes_the_record_of_a_failed_alignment` in | ||
| [install.rs](../engines/lemonade/src/install.rs)), one the opt-out leaves alone, | ||
| and a skipped alignment whose install fails (the error is reported and no notice | ||
| is printed). The unreadable record has a row for each place it can be met: every |
There was a problem hiding this comment.
Same list as docs/engine-plugins.md:86-93 — see that comment. Note the two lists group the conditions slightly differently (this one lists "no active ROCm version" and "an unreadable pin" as two separate items; engine-plugins.md bundles them into one clause). Both are correct — they are genuinely two distinct branches in the source — but the difference itself is a symptom of there being no single source of truth for this list.
There was a problem hiding this comment.
Fixed in 008ba67 together with the engine-plugins.md comment: this section no longer re-enumerates the gates. It links to "Lemonade backend alignment on engine install" in docs/engine-plugins.md, the single statement of them, and lists only the test for each state (the early exits are one table test with a row per exit).
| /// into is shared with every other scenario on the runner. | ||
| const UNPUBLISHED_ROCM_VERSION: &str = "7.99.0"; | ||
|
|
||
| /// The override applies to one command only (`run_rocm_with_scenario_env` |
There was a problem hiding this comment.
This doc comment still says "each step that runs rocm sets it again first," but tracing the three steps that use the override, neither step that actually runs rocm re-arms it before its own invocation — the preceding Given step re-arms it only after its own run, for the step that follows. Priming always happens in the immediately-preceding step, not in the step that runs rocm. Suggested reword: "the step that is about to run rocm either sets this itself or relies on an immediately-preceding step having set it; a step that runs rocm and is followed by another such step must re-arm it after its own run."
There was a problem hiding this comment.
Fixed in 008ba67 with essentially your wording: "the step that is about to run rocm either sets it itself or relies on an immediately preceding step having set it, and a step that runs rocm and is followed by another such step re-arms it after its own run", plus the concrete case (the reinstall Given re-arms it for the shared When the user installs the Lemonade engine).
…de-alignment-retry Main changed Tier 2 of the backend alignment to look up the newest llama.cpp build that ships an archive for the active ROCm version, so the lookup now takes the target version. The failed-alignment record keeps storing that lookup's result, and the tests this branch added pass the target-taking lookup. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…nstall fails (ROCMAI-437) After a recorded failure skips the backend alignment, a failing install of the pinned backend used to surface only the error beneath, leaving no hint that the alignment had been skipped or how to try it again. The error now says the alignment was skipped and why, and what `rocm engines install lemonade --reinstall` does: it re-extracts the runtime tree without the record, so it tries the alignment again and downloads the backend again. The line says only what that command does, not that it fixes the error beneath, which may be transient. The record is deliberately not keyed on the rocm-cli version: the engine cannot see the per-release version, so it expires after 24 hours and `--reinstall` clears it sooner. The gate list is stated once in docs/engine-plugins.md and docs/testing.md links to it; the e2e override comment describes which step re-arms it; the backend retry advice is a shared constant so the test can pin the full error chain against it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
|
Thanks for both passes. Since they were on f247212, the branch has two new commits: b7e01d6 merges main (conflict resolution only: Tier 2 now uses main's
|
juhovainio
left a comment
There was a problem hiding this comment.
Review — no blocking findings
Full review of the whole change (merge-base bfff5216…head 008ba67b, 14 files, +1721/-15).
Implements the ROCMAI-437 product-side fix: records a failed Lemonade llama.cpp ROCm-backend alignment and skips re-attempting it for 24 hours, per the PR's own detailed description. The description's "Scope" section explicitly names three follow-up tickets (ROCMAI-1153/1154/1155) for what is deliberately not in this PR, and its "Questions for the reviewer" section discloses two open tradeoffs (a single 24h window for both transient and permanent failures; --reinstall as the only faster retry path) rather than presenting them as resolved — matches the diff.
Blocking
None
Non-blocking
- Missing integration-level test coverage for mismatched pin / mismatched Lemonade-version skip-decision inputs —
engines/lemonade/src/backend_alignment.rs:232(inline comment)
Confidence 72 · test-coverage
Decisions for the author
- Transient and permanent alignment failures share one 24h suppression window — tradeoff. The PR body raises this itself as an open question; a prior review round found no strong case against it either. ROCMAI-1155 (named in the PR's own "Scope" section) tracks recording the failure's cause and a narrower window as a follow-up. Reasonable to leave as is for this PR.
--reinstallis the only faster retry path, and it re-downloads the whole ~3.3GB backend tree — tradeoff. Same disclosure; a lighter, record-only retry is named as a follow-up candidate in ROCMAI-1155. Reasonable to leave as is for this PR.- The record is not keyed on the rocm-cli release version — tradeoff, explicitly disclosed and reasoned in the PR body (the workspace package version isn't bumped per release, so keying on it would be inert) and tracked under the same follow-up ticket.
Positive signals
- The
fix-23-style duplication risk this class of change usually carries is avoided: the retry advice shown on a skip-path install failure is a single shared constant (BACKEND_INSTALL_RETRY_FAILED) reused by both the original backend-retry path and this PR's new skip-path error chain, so the two can't drift apart silently. - The skip decision is made and the record is read/removed before the backend probe that spawns
lemond, and every one of the five early-return gates (opt-out, unsupported platform, no active SDK version, unreadable/absent pin, pin-already-matches-target) leaves the record file untouched — verified directly against the code, not just the doc comment's claim. - The remediation command the skip notice names (
rocm engines install lemonade --reinstall) is verified end-to-end to actually clear the condition it's printed for: a dedicated test plants a record, confirms a plain install keeps it, then confirms--reinstallremoves it and re-extracts the tree. - A prior review round's specific gap — "the propagated error only carries
--reinstalladvice when it comes from the one retry path that already prints it; an earlier-chain failure surfaces no record-aware remedy at all" — is fixed in this PR's final commit (008ba67b), independently re-verified by tracing the error-wrapping now applied uniformly across all three described failure sub-variants (lemond not starting, a transient failure, and the backend retry's own advice), each with a dedicated test. - New markdown links are all relative and cite by symbol, not line number; the gate list is stated once (in
docs/engine-plugins.md) withdocs/testing.mdlinking to it rather than repeating it, closing a documentation-duplication point a prior review round raised. - The
include_str!cross-crate check (the e2e steps' hard-coded record filename is read from source and compared) runs as part ofcargo teston every PR via the newprepush.rsCROSS_CRATE_READSentry, not only on the nightly GPU lane — so a filename rename on either side is caught long before the one e2e scenario that exercises the real file would run.
Deployment notes
None — purely additive engine-internal behavior (a new on-disk record file and a skip path), no API or CLI-surface change, no migration.
What this covered
Full diff (merge-base bfff5216b50ba997e380a90daf3d356dfb44e3f0…head 008ba67bac4c5c4a497b5ee4d9c4e6722b62f84b), all 14 changed files read completely, across five independent fan-out passes grouped by coupling: backend_alignment.rs core logic (full 3490-line file); the integration points (install.rs, lib.rs, llamacpp_backend.rs); the e2e scenario/step-definition/prepush wiring; the five touched docs, cross-checked against the actual runtime behavior; and agent-instruction adherence (root AGENTS.md) plus version history on the touched files. All five passes' candidates were verified against the source before inclusion; four of the five returned no findings at all, and the one surviving finding is a test-coverage gap at a function whose underlying boolean logic is already unit-tested elsewhere, not a live defect.
CI at the current head: all 39 checks completed, every one success or an expected skipped (the Skill checks (skillscope) job), no failures.
Reconciled against the existing review thread: a prior CHANGES_REQUESTED round (at an earlier head, f247212a) raised several points — failure-cause not recorded, record key ignoring rocm-cli version, doc duplication across three files, a stale doc comment, unconfirmed ticket scope, and two unverified items. A second round on the same commit judged the two open tradeoffs reasonable without re-requesting changes itself. Checked each independently against the current head rather than taking the PR body's account on faith: the doc-duplication point is fixed (the gate list is now stated once); the stale target_unpublished_rocm_version doc-comment nit was not reproduced at this head; the ticket-scope question is now answered directly in the PR's own "Scope" section (three named follow-up tickets); the skip-path error-handling gap a later comment flagged is fixed in the final commit as described above. The remaining points (failure cause, record key, retry weight) are now explicitly disclosed, reasoned tradeoffs with a named tracking ticket rather than silent gaps, which this review treats as resolved via disclosure rather than as defects. Not reconciled against anything beyond the submitted review bodies and the PR's own description.
🤖 Automated by an AI agent
| match read_alignment_failure(&manifest.runtime_dir) { | ||
| StoredAlignmentFailure::Recorded(failure) => { | ||
| let now = now(); | ||
| if failure.skips_alignment(&target_version, &pinned_version, &manifest.version, now) { |
There was a problem hiding this comment.
Non-blocking: missing integration-level test coverage for mismatched pin / mismatched Lemonade-version skip-decision inputs. Every test exercising a record in this file uses a fixture where the on-disk pin equals the record's pinned_rocm_version and manifest.version always equals the record's lemonade_version — only the target_rocm_version dimension is varied here. A regression that stopped comparing pinned_version or manifest.version correctly in this wiring (as opposed to the pure skips_alignment function, which is unit-tested for this in alignment_record.rs) would not be caught by any test in this file.
Confidence 72 · test-coverage · Fix: add 1-2 integration tests varying the on-disk pin (keeping target/lemonade-version fixed) and varying manifest.version (keeping target/pin fixed), asserting align_calls == 1 in each case.
Why
When Lemonade's llama.cpp ROCm backend cannot be aligned to the active ROCm version, the install reverts the pin and force-reinstalls the backend Lemonade ships. The revert leaves the pin different from the active ROCm version, so the next process finds the same mismatch and starts over: nothing recorded that the alignment had already failed.
On a self-hosted E2E lane this turned one upstream outage into a time sink. An upstream Lemonade llama.cpp ROCm asset returned 404 for the pinned tag and a ROCm tarball returned 403 for the latest tag, so every
rocm engines install lemonadeand every Lemonaderocm servein the job redid both alignment tiers, downloaded the tier-2 archive again and force-reinstalled the ~3.3 GB packaged backend. In the failing runs that happened five times per job and pushed the 45-minute lanes to their cap. It was not a retry loop inside one command: Lemonade's own downloader gives up quickly, and it was rocm-cli that re-ran the whole alignment in every process. A user whose active ROCm version has no published backend pays the same cost on every serve.What changes
.rocm-cli-alignment-failed.jsoninside the Lemonade runtime tree: the target ROCm version, the pins it started from, the llama.cpp builds tried, the Lemonade version and the time.lemond. It installs the pinned backend without forcing a reinstall, so an installed backend is reused, and once that install succeeds prints to stderr:Lemonade's ROCm backend could not be aligned to ROCm <target> <less than an hour | N hours> ago; using the backend pinned for ROCm <pin> instead. Run `rocm engines install lemonade --reinstall` to try again now; otherwise an install run N hours from now or later tries it again.The hours left come from the same 24-hour constant the skip decision uses and round up from the full remaining time, so the line never names a time before the record expires. Nothing retries on its own.lemonddoes not start), the error is wrapped with a line built from the record: preparing the backend pinned for ROCm failed, the alignment to ROCm was skipped because it failed ago, androcm engines install lemonade --reinstallre-extracts the runtime tree without that record, so it tries the alignment again (downloading the backend again). The line is on every error that path can return, claims neither an install nor a backend kind, and says what the command does rather than that it fixes the error beneath, which may be transient. The record is kept and the skip notice is not printed. The backend retry's own--reinstalladvice becomes a shared constant (BACKEND_INSTALL_RETRY_FAILED) so the test can pin that whole error chain.--reinstalldeletes and re-extracts the runtime tree, which removes the record, so the next alignment runs. A new Lemonade version re-extracts the tree too.--reinstall.ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT), an unsupported platform, no active ROCm version, an unreadable pin, and a pin that already equals the active ROCm version all return before it and leave it untouched and unmentioned.rocm engines install lemonade --reinstall, which removes it. A record that cannot be written produces a warning and the next install tries again.ROCM_E2E_LEMONADE_ALIGNMENT_TARGET_VERSIONoverrides the ROCm version the alignment targets. It is compiled in only with therocm/e2e-test-hooksfeature, not in release builds.CROSS_CRATE_READStable (xtask/src/prepush.rs) gains the e2e Lemonade install steps file, so a push touching only that file also runsrocm-engine-lemonade's tests, whose file-name testinclude_str!s it. docs/testing.md lists the entry with the others.Verification
--reinstall(exercised through the real install path); one the opt-out leaves alone; a skipped alignment whose install fails (its whole error chain asserted in three rows:lemondnot starting, a transient failure, and the backend retry's own advice, each with the record kept and no notice); and a failed alignment whose revert install also fails, which writes no record.--reinstallcommand. Each asserts the warning together with the on-disk state.@id:runtime-lemonade-backend-alignment-failure-remembered(runtime-33 inruntime_setup.feature) runs two real installs: a--reinstallthat fails to align to 7.99.0, then a plain install that must reuse the pinned backend and print the notice. It is tagged@requires-gpu @requires-bare-metal @requires-engine:lemonade @nightly, so it runs on the GPU nightly Lemonade lanes, not in PR CI. It was not run locally (no GPU on the development host).cargo clippy(workspace, all targets, including thee2e-test-hooksfeature and the e2e test target),cargo fmt,cargo xtask check-module-size,check-architecture-doc, lychee,cargo test -p rocm-engine-lemonade,cargo test -p xtask prepush(including the guardevery_cross_crate_read_is_covered_by_the_widening_rule, which fails with the new table entry removed), and the e2e-cucumber feature-naming test.cargo test --workspace --all-targetspassed apart from failures in crates this change does not touch, caused by the host: a noexec/tmp, a podman-backeddockershim, and temp-file probes timing out on a slow host.cargo xtask smokestopped at its sandbox step because bubblewrap is broken on the development host;cargo xtask coverageand the license-header hook were not run (tools not installed; headers checked by hand). The original failure was not reproduced live, since that needs an upstream backend asset to be missing; the scenario stands in for it by targeting a ROCm version with no published backend.Known gaps
--reinstallclears it.@requires-bare-metal.) The e2e harness decides skips only before a scenario runs, from its tags and the host capability probe, so skipping on Lemonade's own backend choice would need a new probe that startslemond.--reinstallwhose revert install fails (the Lemonade backend download is known to be flaky on those lanes) exits non-zero and leaves no record, so the Given fails.Questions for the reviewer
Two tradeoffs in the behaviour above are deliberate but open; please say whether they are the right ones.
rocm engines install lemonade --reinstall. It deletes and re-extracts the whole Lemonade runtime tree, so every installed backend goes with it: the ~3.3 GB pinned ROCm backend that the skip exists to reuse is downloaded again (by the retry itself if the alignment fails again), and any other backend is downloaded again when next needed. The alternative is a lighter, record-only retry that clears the record and runs the alignment without touching the tree, which would be a new user-facing option. Is--reinstallan acceptable retry path, or is that option worth adding?The second review pass judged both reasonable, documented tradeoffs; the first reviewer's answer is pending. ROCMAI-1155 tracks the alternatives (recording the cause and skipping only permanent failures, a lighter retry), together with keying the record on the rocm-cli release version.
Scope
This is the product-side fix. The rest of the ticket is in follow-ups: ROCMAI-1153 (a step-level timeout below the job cap pinned by a contract test, and the failing URL and status plus a "cancelled at cap" line in the job summary), ROCMAI-1154 (deadlines and entry/exit lines for the
rocmcommands the test harness spawns), and ROCMAI-1155 (recording the failure's cause, a lighter retry, and keying the record on the rocm-cli release version). No overlap with #1009, which retries the queue lane's network fetches and does not touch the alignment.tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No rows for ROCMAI-437.)docs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation)..rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry (cargo xtask check-architecture-docfails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.