Skip to content

E2E: lemonade serving fails on the Strix Halo Ubuntu lanes, with a different log signature depending on the run #897

Closed

Description

@jussielo-amd

Current behavior

On the Strix Halo Ubuntu E2E lanes, rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed exits 0, but http://127.0.0.1:11435/v1/models never lists the model, so every scenario that needs a served lemonade model fails. Five scenarios fail this way on both the release and the nightly leg of every run I read since #838 added the fifth: bench-06, chat-06, chat-07, serve-07 and serve-40 (the first four fail the same way in earlier runs). Each one panics with endpoint http://127.0.0.1:11435/v1/models did not serve model Qwen3-0.6B after 1 attempt(s) of 90s each (or the equivalent wording from serve_and_wait, which waits 300s in serve-40).

The lemonade log captured with the failure is not the same from run to run. I found three shapes:

  1. The backend's llama-server is rejected for a command-line argument and the router gives up. This is the shape in run 41984921053 (both legs, on ci: skip the E2E scenarios when clippy is not green #894), run 41982405007, run 41857662389 and run 41869138017:
[Info] (LlamaCpp) Using LlamaCpp Backend: rocm-stable
[Info] (LlamaCpp) Starting llama-server...
[Error] (Process) error: invalid argument: --no-mmap
[Error] (Router) Backend load failed: llama-server failed to start
[Error] (Router) Retry also failed: llama-server failed to start
[Error] (Server) Failed to load model: llama-server failed to start
  1. Lemonade logs (WrappedServer) llama-server is ready! on port 8001 and starts its watchdog, and the harness still never sees the model on 11435 within 90s. No [Error] line is logged. This is the shape in run 41984503055, which is the same symptom the existing xfail rows describe.

  2. In run 41986772728 the log has Failed to download llama-server from: https://github.com/lemonade-sdk/llama.cpp/releases/download/b9752/llama-b9752-bin-ubuntu-rocm-7.14-x64.tar.gz - Download failed after 6 attempts. twice, and a terminate called after throwing an instance of 'std::system_error' with a llama_server stack in the same job.

Versions where the log names them: Lemonade embeddable 11.5.1, llama.cpp backend b9752 (rocm-stable).

Expected behavior

rocm serve ... --engine lemonade --managed on a Strix Halo Ubuntu host either serves the model or fails with a message that names the cause, and the five scenarios pass or carry xfail rows that name the real bug or bugs.

What is and is not established

  • The string --no-mmap does not occur in this repository (git grep on main finds nothing), so the argument is passed by Lemonade to its own llama-server. I have not determined whether Lemonade 11.5.1 passes a flag the b9752 build does not accept, or whether the llama-server on the runner is not the build Lemonade expects.
  • I have not determined why the shape differs between runs. The runs above come from different runners in the same pool, so a per-runner difference (the llama.cpp backend already present on disk, or a failed backend download falling back to another build) is possible but unchecked.
  • The five scenarios were failing before shape 1 appears in the logs. They fail in the 2026-11-04 08:41 UTC main run (run 41801508506), where the job log has neither --no-mmap nor Failed to load model. I have not looked at what that run's lemonade output shows instead.
  • The existing xfail rows for these scenarios cite EAI-7423, "Lemonade managed serve reaches ready then shuts down immediately", and are flaky = true. That matches shape 2 but not shape 1, so either the rows now cover more than one bug or the attribution is stale.
  • The same logs show device state: NOT drained after 120s (319 MiB free of 512 MiB, floor 450 MiB) before serve-07 and serve-40. That is the drain-wait check test(e2e): judge the serve drain-wait by the pool the engine allocates from #867 reworks for APUs, and it may or may not be related to a failed start.

Possible solution

  • Capture the lemonade log and the llama-server build identity on every failure of these five scenarios, so a run says which shape it hit and which backend build served it.
  • Run the b9752 llama-server with --help on a runner that shows shape 1 and see whether it lists --no-mmap; if not, find out which component adds the flag.
  • fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm backend #855 (draft) moves the Lemonade pin from 11.5.1 to 2026.44.1. Check whether the lane's behavior changes there before spending time on 11.5.1.
  • Once the cause or causes are known, re-point the xfail rows at this issue or delete them.

Your environment

  • rocm-cli channel: release and nightly (both legs fail)
  • Platform: Linux
  • OS / Distro: Ubuntu, self-hosted Strix Halo runner
  • GPU/APU: Strix Halo (gfx1151)

Additional context

Related, different symptoms: #378 (stale router model passing the readiness check), #247, #260, #553.

Activity

  1. added
    bugSomething isn't working
    enginesInference engines (vLLM, Lemonade etc.) related issues
    on Nov 5, 2026
  2. jussielo-amd commented on Nov 5, 2026

    @jussielo-amd
    CollaboratorAuthor

    Reminder for whoever closes this issue: #900 adds an expected-failure row for serve-40 (serve-stop-leaves-no-background-helper) in tests/e2e-cucumber/expectations.toml, scoped to lemonade on Linux with bug = "#897". The row is flaky, so a fixed #897 will not make it fail as a stale row; it has to be deleted by hand when this issue is closed. The advisory stale-xfail hint only names the row on a pull request that mentions #897.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't workingenginesInference engines (vLLM, Lemonade etc.) related issues

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions