Repository navigation
Description
Current behavior
On the Strix Halo Ubuntu E2E lanes, rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed exits 0, but http://127.0.0.1:11435/v1/models never lists the model, so every scenario that needs a served lemonade model fails. Five scenarios fail this way on both the release and the nightly leg of every run I read since #838 added the fifth: bench-06, chat-06, chat-07, serve-07 and serve-40 (the first four fail the same way in earlier runs). Each one panics with endpoint http://127.0.0.1:11435/v1/models did not serve model Qwen3-0.6B after 1 attempt(s) of 90s each (or the equivalent wording from serve_and_wait, which waits 300s in serve-40).
The lemonade log captured with the failure is not the same from run to run. I found three shapes:
- The backend's
llama-serveris rejected for a command-line argument and the router gives up. This is the shape in run 41984921053 (both legs, on ci: skip the E2E scenarios when clippy is not green #894), run 41982405007, run 41857662389 and run 41869138017:
[Info] (LlamaCpp) Using LlamaCpp Backend: rocm-stable
[Info] (LlamaCpp) Starting llama-server...
[Error] (Process) error: invalid argument: --no-mmap
[Error] (Router) Backend load failed: llama-server failed to start
[Error] (Router) Retry also failed: llama-server failed to start
[Error] (Server) Failed to load model: llama-server failed to start
-
Lemonade logs
(WrappedServer) llama-server is ready!on port 8001 and starts its watchdog, and the harness still never sees the model on 11435 within 90s. No[Error]line is logged. This is the shape in run 41984503055, which is the same symptom the existing xfail rows describe. -
In run 41986772728 the log has
Failed to download llama-server from: https://github.com/lemonade-sdk/llama.cpp/releases/download/b9752/llama-b9752-bin-ubuntu-rocm-7.14-x64.tar.gz - Download failed after 6 attempts.twice, and aterminate called after throwing an instance of 'std::system_error'with allama_serverstack in the same job.
Versions where the log names them: Lemonade embeddable 11.5.1, llama.cpp backend b9752 (rocm-stable).
Expected behavior
rocm serve ... --engine lemonade --managed on a Strix Halo Ubuntu host either serves the model or fails with a message that names the cause, and the five scenarios pass or carry xfail rows that name the real bug or bugs.
What is and is not established
- The string
--no-mmapdoes not occur in this repository (git greponmainfinds nothing), so the argument is passed by Lemonade to its ownllama-server. I have not determined whether Lemonade 11.5.1 passes a flag theb9752build does not accept, or whether thellama-serveron the runner is not the build Lemonade expects. - I have not determined why the shape differs between runs. The runs above come from different runners in the same pool, so a per-runner difference (the llama.cpp backend already present on disk, or a failed backend download falling back to another build) is possible but unchecked.
- The five scenarios were failing before shape 1 appears in the logs. They fail in the 2026-11-04 08:41 UTC
mainrun (run 41801508506), where the job log has neither--no-mmapnorFailed to load model. I have not looked at what that run's lemonade output shows instead. - The existing xfail rows for these scenarios cite EAI-7423, "Lemonade managed serve reaches ready then shuts down immediately", and are
flaky = true. That matches shape 2 but not shape 1, so either the rows now cover more than one bug or the attribution is stale. - The same logs show
device state: NOT drained after 120s (319 MiB free of 512 MiB, floor 450 MiB)before serve-07 and serve-40. That is the drain-wait check test(e2e): judge the serve drain-wait by the pool the engine allocates from #867 reworks for APUs, and it may or may not be related to a failed start.
Possible solution
- Capture the lemonade log and the
llama-serverbuild identity on every failure of these five scenarios, so a run says which shape it hit and which backend build served it. - Run the
b9752llama-serverwith--helpon a runner that shows shape 1 and see whether it lists--no-mmap; if not, find out which component adds the flag. - fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm backend #855 (draft) moves the Lemonade pin from 11.5.1 to 2026.44.1. Check whether the lane's behavior changes there before spending time on 11.5.1.
- Once the cause or causes are known, re-point the xfail rows at this issue or delete them.
Your environment
- rocm-cli channel: release and nightly (both legs fail)
- Platform: Linux
- OS / Distro: Ubuntu, self-hosted Strix Halo runner
- GPU/APU: Strix Halo (gfx1151)
Additional context
Related, different symptoms: #378 (stale router model passing the readiness check), #247, #260, #553.
Activity
test(e2e): report unknown serve engine and trim probe noise
jussielo-amd commentedon Nov 5, 2026
Reminder for whoever closes this issue: #900 adds an expected-failure row for serve-40 (
serve-stop-leaves-no-background-helper) intests/e2e-cucumber/expectations.toml, scoped to lemonade on Linux withbug = "#897". The row isflaky, so a fixed #897 will not make it fail as a stale row; it has to be deleted by hand when this issue is closed. The advisory stale-xfail hint only names the row on a pull request that mentions #897.test(e2e): keep the per-row flaky rule in the expectations header
test(e2e): cover Linux hosts without a usable GPU in the #897 expecta…
test(e2e): expect serve-40 to fail on lemonade Linux hosts while ROCm…
Merge origin/main into the Lemonade serve failure evidence branch
test(e2e): keep the revisit-before-deleting caveat in the expectation…
test(e2e): point the Lemonade serve xfail rows at the open ticket
Capture backend identity for Lemonade managed-serve failures (ROCm#912)