Skip to content

test(e2e): run the cucumber suite on a large-stack thread - #1005

Merged
juhovainio merged 1 commit into
mainfrom
issue-1003-run-e2e-suite-big-stack
Nov 10, 2026
Merged

juhovainio merged 1 commit into
mainfrom
issue-1003-run-e2e-suite-big-stack

Conversation

@juhovainio

@juhovainio juhovainio commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1003

The cucumber run is one future polled by block_on on the calling thread, so every step future (and the evidence futures a failing serve step awaits) is held on that thread's stack. Windows gives a main thread 1 MiB; the nightly serve-06 stall overflowed it (STATUS_STACK_OVERFLOW) instead of reporting an expected failure. main now spawns a 64 MiB thread, builds the tokio runtime there and runs the suite; panics are re-raised on join.

Verification

  • cargo clippy -p e2e-cucumber --all-targets -D warnings and the pre-push hooks pass; PR CI is green (the Windows PR-lane job included), apart from GPU lanes still queued at the time of writing.
  • Test drive on the Strix Halo Windows runner: a manual e2e-selfhosted.yml dispatch on this branch with platform=strix-windows, include_nightly=true, name_filter=serve-06 (run 42796765397). serve-06 stalled for its full 2400 s wait, printed the failure with its evidence, archived the service log and printed the cucumber summary (1 scenario (1 failed)) and the Report: line, with no STATUS_STACK_OVERFLOW. That is the point where the nightly run of 42670581923 died. report.json, report.html and junit.xml were uploaded.
  • Not observed: the Reconciliation: line and the exit code. The dispatched job hit the Windows lane's fixed 45-minute timeout-minutes (e2e-selfhosted.yml, e2e-gpu-strix-windows; the annotation reads "The job has exceeded the maximum execution time of 45m0s"), about 1 s after the Report: line, while the harness was still finishing. Unlike the Linux Strix lane, this lane's cap does not rise with include_nightly, so a 2400 s serve-06 wait plus setup cannot fit. The scheduled Windows nightly has a 90-minute cap. Scenario expectations were therefore not seen reconciled on this run.
  • The cause is inferred (large nested async state on the 1 MiB main stack, likely grown by the Capture backend identity for Lemonade managed-serve failures #912 evidence path), not confirmed by a backtrace. The run above shows the failure path now completes, not which frame was recursing or large.
  • No regression test: the harness = false e2e binary has no unit-test hook, and the overflow is Windows-specific.

No user-observable CLI behavior changes; harness-only, so no scenario.

The suite is one future polled on the calling thread, so its step and
failure-evidence futures live on that thread's stack. Windows main
threads get 1 MiB, and a stalled serve-06 overflowed it instead of
reporting the failure. Drive the suite from a 64 MiB thread.

Signed-off-by: juhovainio <juho.vainio@amd.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@juhovainio
juhovainio marked this pull request as ready for review November 10, 2026 10:46
@juhovainio
juhovainio requested a review from a team as a code owner November 10, 2026 10:46
@juhovainio
juhovainio requested review from fredespi and jussielo-amd and removed request for fredespi November 10, 2026 10:46

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 69e49922f

Review — no blocking findings

Full review of the whole change (single file, +29/-2).
Implements the workaround suggested in #1003 ("run serve-06 ... with a larger main-thread stack"); does not perform the issue's other suggested step (bisecting #912 to confirm the root cause), and the PR description itself says so plainly ("Not verified... inferred, not confirmed by a backtrace").

Review mode: single-pass, not fanned out. The diff touches one function in one file (tests/e2e-cucumber/tests/e2e.rs, the main/run_suite split at lines 1265-1293), with no second coupled file to assign to a second worker. I have a spawn tool and could fan out, but Step 3's "scale the review to the subject" explicitly allows a single pass where one worker can read the whole scope completely, and splitting a single-concern, single-file diff into parallel groups (Step 4a) or into several Step-4b source-reading passes run by different workers would not have increased coverage here — one reader held the diff, the full ~360-line run_suite body it now wraps, the crate's Cargo.toml/lint config, and the precedent in apps/rocm/build.rs throughout. This is a deliberate single-pass choice, not a degraded fan-out.

Blocking

None

Non-blocking

None

Decisions for the author

  • Thread-spawn vs. linker /STACK size for the Windows overflow — tradeoff
    The codebase already has a precedent for a small Windows main-thread stack causing overflows: apps/rocm/build.rs sets /STACK:8388608 via cargo:rustc-link-arg-bin for the production rocm binary, with a comment noting "test-harness worker threads already receive Rust's independently configured stack size." This PR instead gives the e2e test binary's effective main thread a larger stack by spawning a dedicated 64 MiB thread and running the suite there, rather than adding an MSVC-only linker arg for this binary. Both are valid: the linker-arg approach keeps the real main thread as the stack owner but is Windows/MSVC-specific and would need its own rustc-link-arg-bin entry for the e2e binary; the spawned-thread approach used here is portable across platforms and keeps the choice in Rust code at the cost of a slightly less obvious "main" (the thread named main no longer runs the suite). No change requested — just confirming the choice between the two established patterns was deliberate.
  • 64 MiB is a guess at the needed margin, not a confirmed bound — tradeoff, already disclosed by the author
    The PR description is explicit that the root cause (where the large stack usage actually comes from, plausibly the #912 evidence-gathering path) was not traced, so there's no evidence 64 MiB is enough margin rather than merely large enough to not have reproduced in testing so far. If the real cause is unbounded growth (e.g., a recursive or looping construction) rather than a large-but-fixed frame, a bigger stack delays the crash without fixing it. The PR already says the only regression signal available is the next nightly Windows run, and that no regression test is possible (harness = false, Windows-specific). Nothing further is asked here beyond confirming this is understood and accepted.

Positive signals

  • Propagating the suite's panic across the thread boundary with std::panic::resume_unwind(payload) rather than swallowing it or just logging — this preserves the exact process-level panic/exit-code behavior (and the existing "cucumber silences the panic hook during .run()" trade-off noted elsewhere in the file) instead of silently losing a suite-ending panic now that it happens on a non-main thread.
  • SUITE_STACK_BYTES's doc comment gives the actual Windows-vs-Linux stack sizes and cites #1003 inline, so the magic number is traceable without digging through the PR history.
  • Naming the spawned thread "e2e-suite" means a panic message now identifies which thread panicked instead of the less informative main.

Deployment notes

None — test-harness-only change; std::process::exit(1) calls inside run_suite still terminate the whole process regardless of which thread invokes them, so CI's pass/fail gating on exit code is unaffected.

What this covered

Read the full diff against d027ff8e3ad9220ebb622667c97ed227d4b6c3e1...69e49922f3ae811525d49a4b5a709e4695d301cf (one file, tests/e2e-cucumber/tests/e2e.rs), the entire run_suite body it now wraps (through its two std::process::exit calls), the crate's Cargo.toml (confirmed tokio's rt-multi-thread feature was already present via the workspace default, unchanged by this PR), the workspace root Cargo.toml's clippy lint configuration (confirms clippy::future_not_send is part of the enabled pedantic group, so the new #[allow(...)] is consistent with existing policy rather than a new exemption), apps/rocm/build.rs (the existing Windows-stack-size precedent referenced above), AGENTS.md (no rule bears on this change beyond the production-file size/architecture-doc checks, which this test file is exempt from), and git log for this file (no related prior history). Also checked the PR's CI status: all completed checks pass, including the Windows E2E job (E2E tests (Strix Halo, Windows, current)); two jobs (MI300X, W7900) were still queued and not yet counted either way. Confirmed via get_reviews immediately before publishing that no review existed yet on this PR at this commit.

Did not run: history pass beyond git log (no notable prior commits touching this region); prior-changes pass (no earlier review rounds exist on this PR). Not reconciled against any discussion here — the PR's single existing comment is an automated security-scan bot notice, not a human review thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The main split is small and correct: a named 64 MiB thread builds the runtime and runs block_on(run_suite()), resume_unwind keeps panics and the exit code, and the process::exit(1) calls in run_suite still end the whole process from the spawned thread. This was a static review: nothing was built or run (no Rust toolchain here). CI on this head is green apart from the GPU lanes that were still queued.

Non-blocking:

  • tests/e2e-cucumber/tests/e2e.rs:1265-1272: only the thread that drives block_on gets the 64 MiB stack. Tokio worker threads and spawn_blocking tasks (for example tests/e2e-cucumber/src/chat_request.rs:43, tests/e2e-cucumber/tests/e2e/serving_steps.rs:364) keep tokio's default stack, so a deep frame there would still overflow. Either set .thread_stack_size(..) on the runtime builder, or say in the doc comment that only the driving thread is enlarged.
  • PR description: Closes #1003 will close the issue while the cause is inferred and the Reconciliation: line and exit code were not observed on Windows. Refs #1003, or closing once a Windows run reaches reconciliation, may be more accurate. Your call.
  • AGENTS.md asks that a missing regression test be covered at another level, or the gap stated. The description states the gap; it could also name the lane that will exercise this (the Windows nightly serve-06).

🤖 by agenthub on AMD AgentHub

@juhovainio
juhovainio added this pull request to the merge queue Nov 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 10, 2026
@juhovainio
juhovainio added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 6862dd3 Nov 10, 2026
63 of 64 checks passed
@juhovainio
juhovainio deleted the issue-1003-run-e2e-suite-big-stack branch November 10, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nightly Windows E2E lanes crash with a stack overflow in serve-06's failure path

3 participants