Skip to content

E2E: serve-40 has no xfail row for lemonade on Linux, so both Strix Halo Ubuntu lanes report an unexpected failure on every run #898

Closed

Description

@jussielo-amd

Current behavior

Both E2E tests (Strix Halo, Ubuntu, release, current) and E2E tests (Strix Halo, Ubuntu, nightly, current) have failed on every main push run since #838 merged on 2026-11-04 at 08:55 UTC. That is all 23 completed main push runs from run 41803309126 (commit db7e540, the merge of #838) to run 41986772728, and the same two jobs on pull requests, for example run 41984921053 on #894.

The last green run on those lanes is run 41801508506 (commit 8479e6a, the commit before #838), which ends with Reconciliation: 7 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 0 unexpected failure(s). Every run since ends with Reconciliation: 7 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 1 unexpected failure(s).

The one unexpected failure is serve-40 (serve-stop-leaves-no-background-helper), the scenario #838 added in tests/e2e-cucumber/features/model_serving.feature. It is tagged @requires-gpu @requires-engine:lemonade. Its When step serves Qwen3-0.6B-GGUF on lemonade and waits for /v1/models to list the model. On these lanes lemonade does not serve the model (see #897), so the step panics before the Then assertion, "serving started no background helper", ever runs:

Step panicked. Captured output: `rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed` exited 0, but http://127.0.0.1:11435/v1/models never served `Qwen3-0.6B` within 300s

The scenarios that make the same serve call (serve-07, chat-06, chat-07 and bench-06) already have xfail rows in tests/e2e-cucumber/expectations.toml. serve-40 has none, so its failure is not reconciled. The other two expected failures on these lanes, serve-01/serve-02 and runtime-02, are unrelated and already known; #876 and #873 are open for them.

Expected behavior

The lane reports 0 unexpected failures while the lemonade serving problem in #897 is open, as it did before #838, so that a real regression on the lane shows up as a red job instead of being hidden behind one that is already red.

Possible solution

Add an xfail row for serve-stop-leaves-no-background-helper scoped to lemonade on Linux, as the sibling rows are, naming #897 as the bug, and delete it when #897 is closed (the reconciliation reports a stale row as an XPASS). The serve-lemonade-inference row (serve-07) sets serve_timeout_secs = 90, which is why serve-07 gives up after 90s while serve-40 waits the full 300s on every run; the new row should set it too.

While the row is in place the scenario's actual check, that serve starts no background helper when no watcher is enabled, is not exercised on these lanes. I have not checked which lanes run serve-40 green, so I do not know whether that guard is covered anywhere else.

Your environment

  • rocm-cli channel: release and nightly
  • Platform: Linux
  • OS / Distro: Ubuntu, self-hosted Strix Halo runner
  • GPU/APU: Strix Halo (gfx1151)

Additional context

Neither lane is in the required-check list, so nothing is blocked from merging; the cost is a red job on every run. Per its description, #838 saw the same two lanes run out of time at 45 minutes on the pull request itself and fail on other open pull requests before it merged; that was before serve-40 existed on main.

Activity

  1. self-assigned this
    on Nov 5, 2026
  2. jussielo-amd commented on Nov 6, 2026

    @jussielo-amd
    CollaboratorAuthor

    Resolved by #900, which added the missing serve-40 xfail row (scoped to lemonade on Linux, citing #897). Since that merge, both Strix Halo Ubuntu lanes (release and nightly) have been green on all 14 main push runs of E2E self-hosted, and the release leg of the latest one (run 42113546816) ends with Reconciliation: 7 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 0 unexpected failure(s).

    The lemonade serving problem itself is not fixed: serve-40 still fails as expected on those lanes. The row is flaky, so it has to be deleted by hand once serving works, and until then serve-40's own check (serve starts no background helper) is not exercised there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions