Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 38 additions & 0 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -815,6 +815,44 @@ Docker exercises AIM serve. The `e2e-gpu-nightly-strix` job logs whether Docker
is reachable ("Report whether Docker is reachable"); check that line before
relying on this coverage.

### Lemonade managed-serve failure evidence

On a failed serve, the shared serve steps preserve the full service log under
`service-logs/` in the E2E results artifact before stopping the service. The engine
is taken from `--engine X` or `--engine=X`, or else from the printed serve plan; if
neither names one, the report says `engine unknown: backend evidence skipped`.
Lemonade failures also include the installed runtime manifest's version/backend, configured
`backend_versions.json`, candidate GPU `llama-server` paths, SHA-256 fingerprints
and `version.txt`,
and each candidate's `--version` and `--help` output. At most eight candidates
are hashed and probed, with two read-only probes per candidate. Each probe retains
up to 32 KiB of stdout and 32 KiB of stderr; only streams that discard bytes are
marked as truncated. Across sixteen probes this is up to 1 MiB of retained raw
output, plus metadata and formatting (invalid UTF-8 bytes may expand when rendered).
Each probe has a five-second timeout. Hashing and probes share a 20-second budget
after inventory discovery; hashing runs off the async executor, rejects files
over 32 MiB, and checks its deadline between 64 KiB reads. Filesystem operations
and process cleanup are not preemptible, so this is not a hard wall-clock limit
on the entire report.
A candidate or configured build pin is **not proof** of which binary the router
launched; a failed probe is retained as evidence, not treated as an identity.
Probes inherit the harness environment rather than reconstructing the engine's
managed library paths, so a probe loader error alone does not establish why the
serve failed.

Use the evidence to distinguish backend startup, download, and router readiness
failures; [issue #897](https://github.com/ROCm/rocm-cli/issues/897) tracks the
Strix Halo Ubuntu investigation. A backend's ready line does not establish that
the router serves the requested model, and a readiness timeout alone does not
establish a shutdown. Diagnostics do not change the CLI's asynchronous
managed-launch contract or imply a backend fix.

Exercise the diagnostics and expectation scoping without a GPU:

```bash
cargo test -p e2e-cucumber --lib lemonade
```

## Runtime Selection And Activation

List registered managed or read-only runtimes and their exact side-by-side keys:
Expand Down
3 changes: 2 additions & 1 deletion tests/e2e-cucumber/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,9 @@ rand.workspace = true
reqwest = { version = "0.13", features = ["json"] }
serde.workspace = true
serde_json.workspace = true
sha2.workspace = true
tempfile = "3"
tokio.workspace = true
tokio = { workspace = true, features = ["process", "io-util"] }
toml = "1.1"
# `ServeDir` backs the install scenario's loopback download server (see
# `src/loopback_http.rs`), so the E2E suite does not hand-roll HTTP framing,
Expand Down
104 changes: 42 additions & 62 deletions tests/e2e-cucumber/expectations.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,60 +43,44 @@ when = {}
bug = "EAI-7219"
reason = "Engines do not report a consistent resolved model name for a short name."

# EAI-7423 was INTERMITTENT on Strix Halo Ubuntu as of 2026-08-05, not fixed:
# which of its scenarios pass varied run to run (run 30995473578: 2 XPASS; run
# 30993213188: 3 XPASS; run 30939967076: all 6). A deterministic-XPASS row fails
# that lane on unrelated PRs, so the EAI-7423 lemonade-linux rows below carry
# `flaky = true` — either outcome is tolerated while the bug stays open. The flag
# is per-row and is NOT derived from `bug`, so a new EAI-7423 row needs it added
# explicitly. Revisit under EAI-7853 before deleting any of them: which lemonade
# build a given run used cannot currently be read back from this lane, so an
# XPASS streak here is not by itself evidence that the bug is fixed.
# ROCm/rocm-cli#897 tracks multiple Strix Halo Ubuntu Lemonade serve signatures:
# the child rejects --no-mmap (41984921053), backend readiness on :8001 without
# the model appearing on :11435 (41984503055), and backend download failure
# (41986772728). A backend ready line does not prove a later shutdown.
# EAI-7423 described only the earlier ready-then-unavailable symptom.
# Retain flaky=true because historical runs also passed intermittently; do not
# infer a fix from a single XPASS. Failed serves now capture candidate executable
# --version/--help and configured pins separately, before stopping the service.
#
# On a lemonade-native LINUX host (Strix Halo Ubuntu) the lemonade managed serve
# loads the model and reaches ready on :8001, then is shut down ~0.08s later, so
# the endpoint never becomes reachable (EAI-7423, root-caused on the box
# 2026-07-15 — NOT the earlier "backend install hides the plan line" theory, which
# was disproven: the `engine:` line IS emitted). os=linux: on native Windows this
# scenario PASSES (run 29338475133 Strix-Windows), so it must NOT be xfail'd there.
# os=linux: native Windows passed (29338475133 Strix-Windows).
[["serve-default-engine-working-endpoint"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the endpoint never becomes reachable."
bug = "#897"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# On a lemonade-native LINUX host the served model's endpoint drops before
# inference: the lemonade managed serve reaches ready then is shut down ~0.08s
# later (EAI-7423, root-caused on the box 2026-07-15 — not a Vulkan hang).
# os=linux: passes on native Windows (run 29338475133).
# The same serve precondition blocks inference; native Windows passed.
[["serve-default-engine-inference"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so inference never succeeds."
bug = "#897"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# On a lemonade LINUX host this scenario does a real lemonade serve, which reaches
# ready then shuts down immediately (EAI-7423, see serve-lemonade-inference).
# os=linux: on native Windows lemonade serve works, so this PASSES there (verified
# run 29338475133 Strix-Windows XPASS) and must not be xfail'd.
# The readiness contract depends on the same real Lemonade serve.
[["serve-readiness-contract"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the ready-to-serve contract fails."
bug = "#897"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# --- EAI-7423: on a lemonade LINUX host the managed serve reaches ready then is
# shut down immediately, so inference never happens (root-caused on the box
# 2026-07-15; previously mis-attributed to the EAI-7052 Vulkan hang). os=linux:
# native Windows lemonade serve works (run 29338475133 Strix-Windows XPASS'd
# this), so scope the xfail to Linux. ---
# Explicit Lemonade serving shares the same failure on the Ubuntu lane.
[["serve-lemonade-inference"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so inference never succeeds."
bug = "#897"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

Expand All @@ -107,16 +91,11 @@ bug = "EAI-7223"
reason = "Chat requests carrying tool definitions are not accepted by the served vLLM model."
serve_timeout_secs = 90

# On a lemonade LINUX host this needs a live lemonade serve, which reaches ready
# then shuts down immediately (EAI-7423). os=linux: passes on native Windows
# lemonade (run 29354305288 XPASS'd it), consistent with the other lemonade
# serve scenarios. therock_family=gfx* scopes this to a REAL GPU host: the
# scenario now also runs on the no-GPU mock lane (backed by MockServer, not a
# real lemonade serve), where EAI-7423 does not apply and it must expect-pass.
# Chat's mock lane must remain expect-pass: only a real GPU serve is affected.
[["chat-tool-definitions-accepted"]]
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the chat request never reaches a live model."
bug = "#897"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

Expand All @@ -125,15 +104,12 @@ flaky = true
# scenario exercises the CLI `rocm chat --prompt` path (which correctly discovers the
# model via /v1/models). Verified expect-pass on MI300X vLLM (XPASS on run 29322186691,
# then 3/3 green re-runs), so there is NO vLLM xfail for this scenario — the CLI chat
# path works. Only the lemonade LINUX path stays xfail: it needs a live lemonade
# serve, which reaches ready then shuts down immediately (EAI-7423); native Windows
# lemonade serve works (run 29338475133 XPASS). therock_family=gfx* scopes this to
# a REAL GPU host: the scenario also runs on the no-GPU mock lane (MockServer, not
# a real lemonade serve), where EAI-7423 does not apply and it must expect-pass.
# path works. Only the real Lemonade Linux serve stays xfail under #897;
# the no-GPU MockServer path and native Windows remain expect-pass.
[["chat-end-to-end-local-model"]]
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the chat request never reaches a live model."
bug = "#897"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

Expand All @@ -144,23 +120,27 @@ when = {}
bug = "EAI-7384"
reason = "Managed runtime path nests recursively (runtimes/wheel/...) on re-provisioning."

# The GPU-backed bench scenario benchmarks a really served model, so on a
# lemonade LINUX host it inherits the same managed-serve bug as the sibling
# serve/chat rows below `serve-default-engine-working-endpoint`: the serve
# reaches ready then shuts down immediately, so this scenario's `a model is
# being served on GPU` precondition never yields a live endpoint. Scoped
# identically — os=linux only, because native Windows lemonade serve works, and
# therock_family=gfx* so it applies only on a real GPU host. `flaky = true`
# matches those rows too: which of them pass varies run to run on that lane, and
# a deterministic-XPASS row would fail the lane on unrelated PRs.
# Bench's real-serve precondition is blocked by #897 on the same Linux hosts.
#
# This does NOT weaken the coverage the bench fix relies on: the five
# MockServer-backed bench scenarios (bench-01 to bench-05) run unxfail'd on
# every lane, including this one, and they are what pin the request path, the
# failure reporting, the CSV columns, and the rollup-split warning.
[["bench-load-real-serve"]]
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the benchmark never reaches a live model."
bug = "#897"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# serve-40 has the same serve precondition; this is not an automation-helper bug.
# xfail + flaky tolerates either outcome, so on this lane the "no background
# helper" assertion carries no signal until #897 is closed: a pass here does not
# show the scenario's own behaviour is healthy, and a failure is not attributed
# to it. Remove the row once #897 is fixed.
[["serve-stop-leaves-no-background-helper"]]
Comment thread
jussielo-amd marked this conversation as resolved.
Resolved
when = { effective_engine = "lemonade", os = "linux" }
bug = "#897"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true
2 changes: 1 addition & 1 deletion tests/e2e-cucumber/features/model_serving.feature
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Feature: Model serving
# (`serve_hf_checkpoint`), which bails explicitly if no backend binary is found
# under bin/llamacpp/<backend>/ — unlike the short-recipe-name path above, whose
# managed-lemonade fallback would mask the identical failure behind the
# unrelated EAI-7423 xfail. Reuses the same small Qwen3-0.6B-GGUF checkpoint as
# #897 xfail. Reuses the same small Qwen3-0.6B-GGUF checkpoint as
# `serve-lemonade-inference` (cache-shared, no extra download) so this stays a
# fast per-PR canary rather than needing the @nightly large-checkpoint path.
@id:serve-hf-checkpoint-inference @requires-gpu @requires-engine:lemonade
Expand Down
33 changes: 33 additions & 0 deletions tests/e2e-cucumber/src/expectation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -826,6 +826,39 @@ mod tests {
}
}

#[test]
fn lemonade_serve_failures_track_897_without_xfailing_mock_or_windows() {
let matrix = Expectations::parse(include_str!("../expectations.toml")).unwrap();
for id in [
"bench-load-real-serve",
"chat-tool-definitions-accepted",
"chat-end-to-end-local-model",
"serve-lemonade-inference",
"serve-stop-leaves-no-background-helper",
] {
let scenario = decl(&[&format!("id:{id}")]);
assert!(
matches!(
resolve(&scenario, &cap("strix-ubuntu"), &matrix, Included::default()),
Expectation::ExpectXfail { bug, flaky: true, .. } if bug == "#897"
),
"{id}"
);
assert_eq!(
matrix.serve_timeout_for(id, &cap("strix-ubuntu"), "lemonade"),
Some(90),
"{id}",
);
for platform in ["mock", "strix-windows"] {
assert_eq!(
resolve(&scenario, &cap(platform), &matrix, Included::default()),
Expectation::ExpectPass,
"{id} on {platform}",
);
}
}
}

fn decl(tags: &[&str]) -> ScenarioDecl {
ScenarioDecl::from_tags(tags)
}
Expand Down
Loading
Loading