Repository navigation
test(e2e): run the cucumber suite on a large-stack thread - #1005
Conversation
The suite is one future polled on the calling thread, so its step and failure-evidence futures live on that thread's stack. Windows main threads get 1 MiB, and a stalled serve-06 overflowed it instead of reporting the failure. Drive the suite from a 64 MiB thread. Signed-off-by: juhovainio <juho.vainio@amd.com>
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 69e49922f
Review — no blocking findings
Full review of the whole change (single file, +29/-2).
Implements the workaround suggested in #1003 ("run serve-06 ... with a larger main-thread stack"); does not perform the issue's other suggested step (bisecting #912 to confirm the root cause), and the PR description itself says so plainly ("Not verified... inferred, not confirmed by a backtrace").
Review mode: single-pass, not fanned out. The diff touches one function in one file (tests/e2e-cucumber/tests/e2e.rs, the main/run_suite split at lines 1265-1293), with no second coupled file to assign to a second worker. I have a spawn tool and could fan out, but Step 3's "scale the review to the subject" explicitly allows a single pass where one worker can read the whole scope completely, and splitting a single-concern, single-file diff into parallel groups (Step 4a) or into several Step-4b source-reading passes run by different workers would not have increased coverage here — one reader held the diff, the full ~360-line run_suite body it now wraps, the crate's Cargo.toml/lint config, and the precedent in apps/rocm/build.rs throughout. This is a deliberate single-pass choice, not a degraded fan-out.
Blocking
None
Non-blocking
None
Decisions for the author
- Thread-spawn vs. linker
/STACKsize for the Windows overflow — tradeoff
The codebase already has a precedent for a small Windows main-thread stack causing overflows:apps/rocm/build.rssets/STACK:8388608viacargo:rustc-link-arg-binfor the productionrocmbinary, with a comment noting "test-harness worker threads already receive Rust's independently configured stack size." This PR instead gives thee2etest binary's effective main thread a larger stack by spawning a dedicated 64 MiB thread and running the suite there, rather than adding an MSVC-only linker arg for this binary. Both are valid: the linker-arg approach keeps the real main thread as the stack owner but is Windows/MSVC-specific and would need its ownrustc-link-arg-binentry for thee2ebinary; the spawned-thread approach used here is portable across platforms and keeps the choice in Rust code at the cost of a slightly less obvious "main" (the thread namedmainno longer runs the suite). No change requested — just confirming the choice between the two established patterns was deliberate. - 64 MiB is a guess at the needed margin, not a confirmed bound — tradeoff, already disclosed by the author
The PR description is explicit that the root cause (where the large stack usage actually comes from, plausibly the #912 evidence-gathering path) was not traced, so there's no evidence 64 MiB is enough margin rather than merely large enough to not have reproduced in testing so far. If the real cause is unbounded growth (e.g., a recursive or looping construction) rather than a large-but-fixed frame, a bigger stack delays the crash without fixing it. The PR already says the only regression signal available is the next nightly Windows run, and that no regression test is possible (harness = false, Windows-specific). Nothing further is asked here beyond confirming this is understood and accepted.
Positive signals
- Propagating the suite's panic across the thread boundary with
std::panic::resume_unwind(payload)rather than swallowing it or just logging — this preserves the exact process-level panic/exit-code behavior (and the existing "cucumber silences the panic hook during.run()" trade-off noted elsewhere in the file) instead of silently losing a suite-ending panic now that it happens on a non-main thread. SUITE_STACK_BYTES's doc comment gives the actual Windows-vs-Linux stack sizes and cites#1003inline, so the magic number is traceable without digging through the PR history.- Naming the spawned thread
"e2e-suite"means a panic message now identifies which thread panicked instead of the less informativemain.
Deployment notes
None — test-harness-only change; std::process::exit(1) calls inside run_suite still terminate the whole process regardless of which thread invokes them, so CI's pass/fail gating on exit code is unaffected.
What this covered
Read the full diff against d027ff8e3ad9220ebb622667c97ed227d4b6c3e1...69e49922f3ae811525d49a4b5a709e4695d301cf (one file, tests/e2e-cucumber/tests/e2e.rs), the entire run_suite body it now wraps (through its two std::process::exit calls), the crate's Cargo.toml (confirmed tokio's rt-multi-thread feature was already present via the workspace default, unchanged by this PR), the workspace root Cargo.toml's clippy lint configuration (confirms clippy::future_not_send is part of the enabled pedantic group, so the new #[allow(...)] is consistent with existing policy rather than a new exemption), apps/rocm/build.rs (the existing Windows-stack-size precedent referenced above), AGENTS.md (no rule bears on this change beyond the production-file size/architecture-doc checks, which this test file is exempt from), and git log for this file (no related prior history). Also checked the PR's CI status: all completed checks pass, including the Windows E2E job (E2E tests (Strix Halo, Windows, current)); two jobs (MI300X, W7900) were still queued and not yet counted either way. Confirmed via get_reviews immediately before publishing that no review existed yet on this PR at this commit.
Did not run: history pass beyond git log (no notable prior commits touching this region); prior-changes pass (no earlier review rounds exist on this PR). Not reconciled against any discussion here — the PR's single existing comment is an automated security-scan bot notice, not a human review thread.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
jussielo-amd
left a comment
There was a problem hiding this comment.
Looks good. The main split is small and correct: a named 64 MiB thread builds the runtime and runs block_on(run_suite()), resume_unwind keeps panics and the exit code, and the process::exit(1) calls in run_suite still end the whole process from the spawned thread. This was a static review: nothing was built or run (no Rust toolchain here). CI on this head is green apart from the GPU lanes that were still queued.
Non-blocking:
tests/e2e-cucumber/tests/e2e.rs:1265-1272: only the thread that drivesblock_ongets the 64 MiB stack. Tokio worker threads andspawn_blockingtasks (for exampletests/e2e-cucumber/src/chat_request.rs:43,tests/e2e-cucumber/tests/e2e/serving_steps.rs:364) keep tokio's default stack, so a deep frame there would still overflow. Either set.thread_stack_size(..)on the runtime builder, or say in the doc comment that only the driving thread is enlarged.- PR description:
Closes #1003will close the issue while the cause is inferred and theReconciliation:line and exit code were not observed on Windows.Refs #1003, or closing once a Windows run reaches reconciliation, may be more accurate. Your call. - AGENTS.md asks that a missing regression test be covered at another level, or the gap stated. The description states the gap; it could also name the lane that will exercise this (the Windows nightly serve-06).
🤖 by agenthub on AMD AgentHub
Closes #1003
The cucumber run is one future polled by
block_onon the calling thread, so every step future (and the evidence futures a failing serve step awaits) is held on that thread's stack. Windows gives a main thread 1 MiB; the nightly serve-06 stall overflowed it (STATUS_STACK_OVERFLOW) instead of reporting an expected failure.mainnow spawns a 64 MiB thread, builds the tokio runtime there and runs the suite; panics are re-raised on join.Verification
cargo clippy -p e2e-cucumber --all-targets -D warningsand the pre-push hooks pass; PR CI is green (the Windows PR-lane job included), apart from GPU lanes still queued at the time of writing.e2e-selfhosted.ymldispatch on this branch withplatform=strix-windows,include_nightly=true,name_filter=serve-06(run 42796765397). serve-06 stalled for its full 2400 s wait, printed the failure with its evidence, archived the service log and printed the cucumber summary (1 scenario (1 failed)) and theReport:line, with noSTATUS_STACK_OVERFLOW. That is the point where the nightly run of 42670581923 died.report.json,report.htmlandjunit.xmlwere uploaded.Reconciliation:line and the exit code. The dispatched job hit the Windows lane's fixed 45-minutetimeout-minutes(e2e-selfhosted.yml,e2e-gpu-strix-windows; the annotation reads "The job has exceeded the maximum execution time of 45m0s"), about 1 s after theReport:line, while the harness was still finishing. Unlike the Linux Strix lane, this lane's cap does not rise withinclude_nightly, so a 2400 s serve-06 wait plus setup cannot fit. The scheduled Windows nightly has a 90-minute cap. Scenario expectations were therefore not seen reconciled on this run.harness = falsee2e binary has no unit-test hook, and the overflow is Windows-specific.No user-observable CLI behavior changes; harness-only, so no scenario.