Skip to content

Nightly Windows E2E lanes crash with a stack overflow in serve-06's failure path #1003

Closed

Description

@juhovainio

Symptom

Both Windows nightly E2E lanes (release and nightly, Strix Halo) of run 42670581923 end with the test process dying instead of reporting results:

Scenario: serve-06 - A large platform-specific model serves and responds to inference
 ✔  Given a managed runtime is active
thread 'main' has overflowed its stack
process didn't exit successfully: ...e2e-....exe (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)

The crash comes about 41 minutes after the scenario starts, which is the scenario's 2400 s wait for the model to be served. It happens before the step's failure message is printed. No reconciliation summary is produced, and no report artifact is uploaded for the lane.

Run: https://github.com/ROCm/rocm-cli/actions/runs/42670581923 (jobs "E2E tests (Strix Halo, Windows, incl. nightly-only) (release)" and "(nightly)"), commit 227968dc.

What is known

  • serve-06 is already expected to fail on a 7.14.x SDK on Windows (the expectation added in ROCMAI-899, ROCMAI-900: expect the two Windows Lemonade scenarios to fail on a 7.14.x SDK #964), so the stall itself is not the new problem. The process should record an expected failure and continue.
  • The previous scheduled nightly, on commit 3dc4a31a, ran the same scenario to the same stall and handled the failure normally, printing a full summary.
  • Commits between those two include Capture backend identity for Lemonade managed-serve failures #912 (capture backend identity for Lemonade managed-serve failures), which added the evidence gathering that now runs when a managed serve stalls (serve_failure_evidence and lemonade_diagnostics). It runs in exactly the window where the crash happens, so it is my first suspect. I read its directory search and it is depth-bounded, so I did not find the cause by reading. This is a lead, not a confirmed cause.

What is not known

  • Whether the overflow is in that evidence capture or elsewhere in the stall report path (log archiving, report formatting).
  • Whether the Linux Lemonade lanes hit the same crash. They had no runners in this run.

Impact

Until this is fixed, every nightly Windows lane loses its whole report whenever serve-06 stalls, so it also hides the result of every other scenario on that lane.

Suggested next step

Run only serve-06 on a Windows runner (or a debug build) with a larger main-thread stack, or with a backtrace of the overflow, to see which function recurses or holds the large frame. Bisecting the commits between 3dc4a31a and 227968dc on the Windows lane would also confirm or rule out #912.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions