Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 33 additions & 21 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -720,8 +720,8 @@ place fails it), so a change of route is caught even when the endpoint still ans
inference and stops the scenario-owned service. It needs a usable AMD GPU on a
Lemonade Linux lane (`@requires-gpu`, `@requires-os:linux`,
`@requires-engine:lemonade`). It has no flaky expectations row, so a route
regression fails the lane. Of the other direct-route serves, `serve-07` keeps its
flaky Linux row; `serve-08` has none, so it fails the lane too.
regression fails the lane. The other direct-route serves, `serve-07` and
`serve-08`, have no row either, so a regression there fails the lane too.

The daemon's Lemonade contracts in
[telemetry_contract.rs](../crates/rocm-dash-daemon/tests/telemetry_contract.rs)
Expand Down Expand Up @@ -754,18 +754,16 @@ The dashboard scenarios in
claims. One remaining model shows its throughput, an empty or unreachable
server shows no models, and a reload restores both.
- `dash-lemonade-managed-inference-journey`: a GPU-gated real Lemonade serve and
inference followed by dashboard observation. On Linux Lemonade hosts it is
expected to be flaky and its serve is bounded by `serve_timeout_secs` (readiness polling; the CLI process is killed only 300 s after that) in
[expectations.toml](../tests/e2e-cucumber/expectations.toml). The model must
inference followed by dashboard observation. It carries no row in
[expectations.toml](../tests/e2e-cucumber/expectations.toml), so it is
expected to pass on every Linux Lemonade host that runs it; its flaky
ROCMAI-898 row was removed with the others (see
[Lemonade managed-serve failure evidence](#lemonade-managed-serve-failure-evidence)). The model must
appear as a full row in Observe, and its instance detail must show it READY
on the port it was served on. This is real engine acceptance;
the protocol fixtures do not prove GPU inference or engine installation.
The flaky row matches every lane that can run this scenario, so the whole
journey is advisory: its dashboard, instance detail, and post-quit inference
checks cannot block a merge. Only the serve step is intermittent, but
expectations apply per scenario. It also does not gate the
"Lemonade — managed runtime installed" wording.
Unit tests gate that wording instead:
It runs only on GPU lanes, so the
"Lemonade — managed runtime installed" wording is also gated by unit tests:
`managed_installed_verdict_follows_detect_result` in
[dash.rs](../apps/rocm/src/dash.rs), and the engine pane glyph and verdict
tests in [pane.rs](../crates/rocm-dash-tui/src/ui/tabs/pane.rs). No other
Expand Down Expand Up @@ -1768,9 +1766,20 @@ is the `kill/reap failed` branch, which needs an operating system that refuses t
kill a child it just started.

Use the evidence to distinguish backend startup, download, and router readiness
failures; ROCMAI-898 tracks the Strix Halo Ubuntu investigation (first tracked
as [issue #897](https://github.com/ROCm/rocm-cli/issues/897), which is closed). A
backend's ready line does not establish that the router serves the requested
failures. The diagnostics were added for the Strix Halo Ubuntu Lemonade serve
failures of ROCMAI-898 (first tracked as
[issue #897](https://github.com/ROCm/rocm-cli/issues/897)), which [#963](https://github.com/ROCm/rocm-cli/pull/963) fixed.
Twelve of the scenarios that carried ROCMAI-898 expected-failure rows in
[expectations.toml](../tests/e2e-cucumber/expectations.toml) no longer carry a
row on Linux lemonade hosts, and
`lemonade_linux_serve_scenarios_have_no_xfail_after_rocmai_898` in
[expectation.rs](../tests/e2e-cucumber/src/expectation.rs) guards them against a
row that matches one of the hosts it models.
The other two, `serve-50` and `serve-52`, keep flaky rows under ROCMAI-1023 for
a different symptom: the serve is ready, but `rocm services list` shows no
`anthropic endpoint:` row.

A backend's ready line does not establish that the router serves the requested
model, and a readiness timeout alone does not establish a shutdown. Diagnostics do not change the CLI's asynchronous
managed-launch contract or imply a backend fix.

Expand Down Expand Up @@ -3223,12 +3232,8 @@ bad config from failing a `serve` after its launch. The user-visible side is `au
(mock lane: enable starts a live helper, disabling the last watcher makes it
exit) and `serve-40` in
[model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature)
(GPU lane: serving with no watcher enabled starts no helper; on a Linux host
that runs it with lemonade it is an expected failure in
[expectations.toml](../tests/e2e-cucumber/expectations.toml) while lemonade
serving on the Strix Halo Ubuntu lanes is broken (ROCMAI-898), so the helper check is
not reached on those lanes. The Strix Halo Windows lane runs it and passes, and
the row also matches other Linux lemonade hosts, where a pass is tolerated).
(GPU lane: serving with no watcher enabled starts no helper; it is pinned to
lemonade and expected to pass on every GPU lane that runs it, Linux and Windows).

Anthropic-compatible `/v1/messages` endpoint (ROCMAI-597). The translation and
its HTTP front are tested in
Expand All @@ -3254,7 +3259,14 @@ checks the reply is an event stream that reassembles into text, with usage in
`message_delta`) in
[model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature).
They run on the GPU lanes (`@requires-gpu`), with whichever engine the host
serves with. The proxy's unit tests cover the stream event sequence, a stream
serves with. On a Linux host serving with lemonade, `serve-50` and `serve-52`
carry flaky rows in
[expectations.toml](../tests/e2e-cucumber/expectations.toml) under ROCMAI-1023
(`services list` intermittently shows no `anthropic endpoint:` row after the
service is ready), and `serve-53` has an expected-failure row under EAI-7223
on vLLM, which is started without a tool-call parser.

The proxy's unit tests cover the stream event sequence, a stream
that ends early, an engine refusal before the first byte, and the refusal of a
streaming request that offers `tools` (no scenario covers that refusal). The
tool-use translation (`tools`, `tool_choice`, `tool_use` and `tool_result`
Expand Down
183 changes: 33 additions & 150 deletions tests/e2e-cucumber/expectations.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@
# so an XPASS is tolerated (reported, not stale)
#
# An empty `when = {}` means "always xfail" (platform-independent bug).
#
# The scenarios in `LEMONADE_LINUX_SERVE_IDS` in src/expectation.rs, whose
# ROCMAI-898 rows were removed when Linux Lemonade serving was fixed, are
# guarded by `lemonade_linux_serve_scenarios_have_no_xfail_after_rocmai_898`
# against a row that matches a host it models. If that test fails, its failure
# message says what to do.

# --- EAI-7219: short-name expansion (vLLM does not show an expanded name) ---
# What the lane logs show, so the scope below is not read as more than it is:
Expand Down Expand Up @@ -73,176 +79,53 @@ when = {}
bug = "EAI-7219"
reason = "A serve of a short name does not show the full model name the engine will serve: vLLM either stops before printing a serve plan or prints the alias unexpanded, and the lemonade half is only reached once that stops happening."

# ROCMAI-898 tracks multiple Strix Halo Ubuntu Lemonade serve signatures (first
# tracked as GitHub issue ROCm/rocm-cli#897, since closed; ROCMAI-898 is the open
# ticket and what the rows below name):
# the child rejects --no-mmap (41984921053), backend readiness on :8001 without
# the model appearing on :11435 (41984503055), and backend download failure
# (41986772728). A backend ready line does not prove a later shutdown.
# EAI-7423 described only the earlier ready-then-unavailable symptom.
# Retain flaky=true because historical runs also passed intermittently; do not
# infer a fix from a single XPASS. The flag is per-row and is NOT derived from
# `bug`, so a new ROCMAI-898 row needs `flaky = true` added explicitly.
# Failed serves now capture configured pins, candidate executable paths and
# fingerprints before stopping the service, and each candidate's --version/--help
# after it.
# Revisit under EAI-7853 before deleting any of these rows: which lemonade build
# a given run used could not be read back from this lane, and until a failed run
# there has carried that evidence, an XPASS streak is not by itself evidence that
# the bug is fixed.
# --- ROCMAI-1023: `services list` does not show the Anthropic endpoint row ---
# On the Strix Halo Ubuntu lanes the lemonade serve itself succeeds and the model
# answers on /v1/models, but `rocm services list` then shows no
# `anthropic endpoint:` row within the 30s poll of `listed_anthropic_endpoint`
# (tests/e2e/serving_steps.rs). Seen intermittently on main after the Lemonade
# serve fix (#963): serve-50 in jobs 129254733795 and 129309536682, serve-52 in
# jobs 129309527901, 129318300169, 129700937355 and 130496332636.
#
# os=linux: native Windows passed (29338475133 Strix-Windows).
[["serve-default-engine-working-endpoint"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# The same serve precondition blocks inference; native Windows passed.
[["serve-default-engine-inference"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# The readiness contract depends on the same real Lemonade serve.
[["serve-readiness-contract"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# The Anthropic-endpoint scenarios start from `a model is being served on GPU`
# like the rows above, so they hit the same Linux lemonade serve failure.
# serve-51, serve-53 and serve-54 read the endpoint through the same poll (their
# When steps look it up before sending a request) yet carry no row. In the six
# jobs above, serve-51 passed in all six, and serve-53 and serve-54 (added later)
# ran only in 130496332636 and passed there. Those jobs ran while the three
# still had ROCMAI-898 rows, so each had one serve attempt, not two.
# A row for them would rest on no observed failure; add one if they start
# failing the same way.
#
# `flaky = true` because both scenarios pass in most runs. No
# `serve_timeout_secs`: the serve becomes ready, so a shorter serve wait would
# only cut short a slow but healthy serve. A matching row still tolerates any
# failure of the scenario on these hosts, not only this symptom: a serve that
# never becomes ready is also reported as the expected failure, and the harness
# gives an expected-failure scenario one serve attempt instead of two.
[["serve-anthropic-endpoint-advertised"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model, so no endpoint is left to advertise."
serve_timeout_secs = 90
bug = "ROCMAI-1023"
reason = "After the lemonade service is ready, `rocm services list` intermittently shows no `anthropic endpoint:` row, so the Anthropic endpoint is not advertised."
flaky = true

[["serve-anthropic-messages-inference"]]
[["serve-anthropic-endpoint-readiness"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model, so the Anthropic request never gets an answer."
serve_timeout_secs = 90
flaky = true

[["serve-anthropic-messages-tool-use"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model, so the tool-use request never gets an answer."
serve_timeout_secs = 90
bug = "ROCMAI-1023"
reason = "After the lemonade service is ready, `rocm services list` intermittently shows no `anthropic endpoint:` row, so readiness cannot be seen to cover the Anthropic endpoint."
flaky = true

# --- EAI-7223: requests with tool definitions rejected (vLLM path) ---
[["serve-anthropic-messages-tool-use"]]
when = { effective_engine = "vllm" }
bug = "EAI-7223"
reason = "The served vLLM model is started without a tool-call parser, so it rejects a request that names a tool; the endpoint relays that refusal."
serve_timeout_secs = 90

[["serve-anthropic-endpoint-readiness"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model, so readiness never covers the Anthropic endpoint."
serve_timeout_secs = 90
flaky = true

[["serve-anthropic-messages-streaming"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model, so the streaming Anthropic request never gets an answer."
serve_timeout_secs = 90
flaky = true

# Explicit Lemonade serving shares the same failure on the Ubuntu lane.
[["serve-lemonade-inference"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# serve-40 serves the same small GGUF model through the same lemonade managed serve
# as the rows above, so on a lemonade LINUX host its When step never sees the model
# on /v1/models and panics before the Then assertion runs. Scoped like those rows
# (os=linux). Known from run 41986772728: serve-40 passes on the Strix Halo Windows
# lane, is skipped on the Strix Halo WSL2 lane (no AMD GPU detected), and fails on
# both Strix Halo Ubuntu lanes. The row also matches any other Linux host that runs
# the scenario with lemonade (the MI300X and W7900 lanes had not reported in that
# run); whether lemonade serves there is not verified, and `flaky = true` tolerates
# a pass on such a host. The cause on the Strix Halo Ubuntu lanes is not settled
# and varies between runs (ROCMAI-898), so this row names that ticket rather than
# EAI-7423
# and carries `flaky = true` like its siblings. While it is in place the
# scenario's own check (serve starts no background helper when no watcher is
# enabled) is not exercised on the hosts it matches, and a genuine failure of that
# check there would also read as a failure that was expected. `flaky` also means a
# fixed ROCMAI-898 is not flagged as a stale row, so delete the row by hand when
# ROCMAI-898 is
# closed.
[["serve-stop-leaves-no-background-helper"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "Lemonade serving fails on the Strix Halo Ubuntu lanes, so the model never appears on /v1/models and the helper check is never reached."
serve_timeout_secs = 90
flaky = true

# The dashboard journey does the same real lemonade serve as serve-lemonade-inference,
# so on a Linux lemonade host it is just as intermittent. The bounded serve timeout
# makes a stuck serve fail this one scenario instead of stalling the whole lane.
[["dash-lemonade-managed-inference-journey"]]
when = { effective_engine = "lemonade", os = "linux" }
bug = "ROCMAI-898"
reason = "The managed Lemonade serve on Linux is intermittent: it can reach ready and then shut down, or never become ready, so the journey cannot be relied on to complete."
serve_timeout_secs = 90
flaky = true

# --- EAI-7223: chat requests with tool definitions rejected (vLLM path) ---
[["chat-tool-definitions-accepted"]]
when = { effective_engine = "vllm" }
bug = "EAI-7223"
reason = "Chat requests carrying tool definitions are not accepted by the served vLLM model."
serve_timeout_secs = 90

# Chat's mock lane must remain expect-pass: only a real GPU serve is affected.
[["chat-tool-definitions-accepted"]]
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# EAI-7221 (TUI chat sends hardcoded 'local-model' instead of querying /v1/models)
# was previously mapped here as a vLLM-path xfail, but that bug is TUI-only and this
# scenario exercises the CLI `rocm chat --prompt` path (which correctly discovers the
# model via /v1/models). Verified expect-pass on MI300X vLLM (XPASS on run 29322186691,
# then 3/3 green re-runs), so there is NO vLLM xfail for this scenario — the CLI chat
# path works. Only the real Lemonade Linux serve stays xfail under ROCMAI-898;
# the no-GPU MockServer path and native Windows remain expect-pass.
[["chat-end-to-end-local-model"]]
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# Bench's real-serve precondition is blocked by ROCMAI-898 on the same Linux hosts.
#
# This does NOT weaken the coverage the bench fix relies on: the five
# MockServer-backed bench scenarios (bench-01 to bench-05) run unxfail'd on
# every lane, including this one, and they are what pin the request path, the
# failure reporting, the CSV columns, and the rollup-split warning.
[["bench-load-real-serve"]]
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true }
bug = "ROCMAI-898"
reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable."
serve_timeout_secs = 90
flaky = true

# --- Lemonade 11.5.1 cannot align its llama.cpp backend to a ROCm 7.14.x SDK ---
# `rocm engines install lemonade` tries to align Lemonade's `llamacpp:rocm`
# backend with the active ROCm SDK. Lemonade 11.5.1, the pin in
Expand Down
5 changes: 2 additions & 3 deletions tests/e2e-cucumber/features/model_serving.feature
Original file line number Diff line number Diff line change
Expand Up @@ -910,9 +910,8 @@ Feature: Model serving
# Linux only: registered names take the router on Windows both with and
# without the direct llama-server route, so a Windows run cannot tell the
# two behaviours apart.
# No expectations row is added for this scenario. The flaky Linux rows cover
# the direct llama-server route, and this scenario takes the router route, so
# those rows' failure mode does not describe it. It is nightly-only.
# No expectations row: it is expected to pass on every Linux lemonade host
# that runs it. It is nightly-only.
@id:serve-lemonade-complex-registered-inference @requires-gpu @requires-os:linux @requires-engine:lemonade @nightly
Scenario: serve-57 - A registered vision GGUF retains the Lemonade router and managed lifecycle
Given a managed runtime is active
Expand Down
Loading
Loading