Skip to content

test(e2e): let an xfail row's serve_timeout_secs beat the scenario's serve-timeout tag - #1081

Open
juhovainio wants to merge 2 commits into
mainfrom
worktree-serve-06-windows-timeout
Open

juhovainio wants to merge 2 commits into
mainfrom
worktree-serve-06-windows-timeout

Conversation

@juhovainio

Copy link
Copy Markdown
Collaborator

Problem

serve-06 (serve-large-model-inference) carries @serve-timeout:2400 for the hosts where it passes. On native Windows with Lemonade it is a known failure (ROCMAI-900): the serve never becomes ready. The harness resolved the wait as tag first, then xfail row, so the tag won and every Windows lane waited the full 2400s before failing. In the nightly runs of 2026-10-27 and 2026-11-10 that was about 41 minutes per Windows job, which made the Windows lanes the longest in the run (about 65 minutes against 19 to 35 for the Linux lanes).

The docs already describe the intent (the tag lengthens an expected-pass serve, the row shortens a known-bug serve); the precedence in Expectations::serve_timeout_override did not follow it, and the expectations.toml comment recorded the gap ("still holds the Windows lane for the full 2400s wait").

Change

  • Expectations::serve_timeout_override: the shortest matching xfail serve_timeout_secs now wins; the @serve-timeout tag applies where no row matches the host.
  • expectations.toml: the Windows serve-large-model-inference row gets serve_timeout_secs = 600 (the step default), and its comment is updated.
  • Doc comments in e2e.rs and serving_steps.rs updated to the new precedence.

Only serve-06 is affected: no other row sets serve_timeout_secs on a scenario that also has a @serve-timeout tag. Hosts where no row matches (Linux, WSL2, MI300X, W7900) keep the 2400s window.

Verification

  • cargo test -p e2e-cucumber --lib: 343 passed. cargo clippy -p e2e-cucumber --all-targets -- -D warnings and cargo fmt --check are clean.
  • The tests that pin the precedence fail when it is reverted (serve_timeout_override_prefers_the_row_and_needs_an_id, lemonade_alignment_rows_apply_only_to_native_windows_with_lemonade) and pass with this change. The first replaces the test that asserted a tag lengthens a row's timeout, which is the behaviour being changed. The second now asserts the resolved wait for serve-06 on Windows (600) and on a host with no row (2400).

Scope and gaps

  • Test-harness only: no change to CLI behaviour, output, exit codes or files, so no new Gherkin scenario. The covered scenario is the existing @id:serve-large-model-inference (serve-06), which stays an expected failure on native Windows with Lemonade.
  • Not run on a Windows GPU lane: there is no recent nightly run to compare against. The first Windows run after merge should show serve-06 failing in about 10 minutes instead of about 41.
  • 600s is a judgement call. If the underlying bug is fixed, delete the row and the tag's 2400s applies again.
  • A separate stack overflow that ended the 2026-11-10 Windows jobs after this wait is already fixed by test(e2e): run the cucumber suite on a large-stack thread #1005, so it is not part of this change.

@juhovainio
juhovainio requested a review from a team as a code owner November 13, 2026 13:03
@juhovainio
juhovainio requested a review from tomastola November 13, 2026 13:03
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@juhovainio
juhovainio requested a review from rominf November 13, 2026 13:38

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 62ae35f3

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change. Implements the precedence fix described in the PR body (xfail row's serve_timeout_secs now beats the scenario's @serve-timeout tag); no GitHub issue is named, and the bug references (ROCMAI-899/ROCMAI-900) are external tracker IDs not retrievable from here.

Blocking

None

Non-blocking

  • The comment block mixes two different descriptions of serve-06's Windows failure mode without reconciling them — tests/e2e-cucumber/expectations.toml:253-254 (unchanged by this PR) and the bug/reason text at :274 (also unchanged) describe the known bug as the server becoming ready and then crashing "in the SDK's Tensile library on the first inference request." The paragraph this PR adds at :260-264 instead justifies the new serve_timeout_secs = 600 by saying "the failing serve never became ready within that window," i.e. a readiness-poll timeout rather than a post-ready crash — which is also how the PR description frames it ("the serve never becomes ready"). If the server actually becomes ready quickly and only fails on the subsequent inference POST (as the unchanged prose says), the harness's existing model_is_ready readiness poll would return well under the old 2400s, and shortening serve_timeout_secs would have no effect on the ~41-minute lane time the PR is trying to fix — so only one of these two descriptions can be the real mechanism. Both read as confident, factual claims about the same bug in the same comment block, and nothing here says which one is stale. Worth reconciling (or just dropping the now-superseded "crashes in the SDK's Tensile library" framing at :253-254/:274 if the newer readiness-timeout evidence is what's current) so a future reader — and whoever eventually verifies ROCMAI-900 is fixed — isn't working from two different theories of the failure.
    Confidence 70 · mechanical · Fix: update or remove whichever of the two failure-mode descriptions the 2026-10-27/2026-11-10 run evidence has superseded, so the row's reason and the surrounding comment agree on what actually happens on the Windows lane.

Decisions for the author

None

Positive signals

  • The precedence flip is logically sound and narrowly scoped: since an xfail row only matches a host where the scenario is already expected to fail there, letting the row win over the @serve-timeout tag can only shorten the wait on a host that was never going to pass anyway — it cannot truncate a legitimately slow passing serve on another host, because the tag only applies where no row matches (src/expectation.rs's serve_timeout_override).
  • Good test coverage of both directions of the new precedence — serve_timeout_override_prefers_the_row_and_needs_an_id now asserts the row wins on the known-bad host and that the tag still lengthens the wait on a host with no matching row, and lemonade_alignment_rows_apply_only_to_native_windows_with_lemonade was extended to check both the Windows (Some(600)) and Ubuntu (Some(2400)) resolution for serve-06 instead of only asserting the old tag-always-wins behavior.
  • The expectations.toml row comment cites concrete evidence (specific nightly run dates and observed lane duration) for the chosen 600s value rather than an unexplained magic number.

Deployment notes

None — test-harness-only change (expectations matrix, resolver, and their doc comments/tests); no CLI behavior, output, or production code is touched.

What this covered

Full diff (e2e5c5bb3d…62ae35f3f9) across all four changed files, plus: the complete ScenarioDecl/Expectations/XfailEntry definitions and every caller of serve_timeout_for/serve_timeout_override traced via repo-wide grep (tests/e2e-cucumber/tests/e2e.rs's before hook and inference_timeout_for, tests/e2e-cucumber/tests/e2e/serving_steps.rs's serve_and_wait/serve_gpu_model/setup_large_gpu_model, and runtime_lifecycle_steps.rs's use of model_is_ready); the full surrounding expectations.toml comment block for both xfail rows; the model_serving.feature scenario tags for serve-06; and tests/e2e-cucumber/README.md's tag-reference table for staleness. CI on the reviewed commit is all green (including the E2E tests and affected-crate test jobs) except one E2E tests (MI300X, current) job still queued at review time — not treated as a finding since it had not failed.

Did not run: a local build/test (no cargo available in this environment) — relied on the green CI run for the reviewed commit instead. History/prior-changes passes were folded into the single pass above given the PR's narrow scope; agent-instruction adherence checked against this repo's AGENTS.md (no violations found — no new/renamed source files requiring docs/architecture.md entries, docs are already in sync).
Not reconciled against any discussion on the PR — this review was reached blind from the code and the PR's own description only.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 2e0bf0b0

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. An earlier automated round from this account exists at the prior head 62ae35f3 (now stale); this is a fresh, independent pass against the current head and does not reuse its findings.

Review — needs work

Full review of the whole change. Implements the precedence fix described in the PR body (an xfail row's serve_timeout_secs now beats the scenario's @serve-timeout tag); ROCMAI-899/ROCMAI-900 are external tracker IDs not retrievable from here, and no GitHub issue is named as closed by this change (the #1005 mention is unrelated context, verified accurate: that PR is merged and fixes the stack-overflow it describes).

Blocking

  • A required check is failing on the reviewed commit — E2E consolidated report (self-hosted) (run 43598171626). Its "Check every lane that ran uploaded its report" step fails because the E2E tests (MI300X, current) job was cancelled after sitting for a full 24 hours with no steps recorded, which reads as the self-hosted MI300X runner never picking the job up (GitHub's own job-duration cap, not a test failure) rather than anything this diff touches — the diff is test-harness timeout logic unrelated to MI300X dispatch. All other checks on this commit are green, including Test (affected crates), clippy, E2E tests (Strix Halo, Windows/WSL2, current), and Commit signatures + sign-off.
    Confidence 85 · architectural · Fix: re-run the self-hosted E2E workflow (or wait for an MI300X runner to become available) rather than changing code; flag the runner-availability gap to whoever owns that pool if it recurs.

Non-blocking

None

Decisions for the author

None

Positive signals

  • The precedence flip is logically sound and provably narrow: is_xfail and serve_timeout_for share the same Condition::matches gate, so a row can only shorten the wait on a host where the scenario is already expected to fail there — verified by grepping every @serve-timeout-tagged scenario (serve-large-model-inference, serve-aim-inference, serve-aim-lifecycle-parity) against expectations.toml and confirming only serve-06 carries a matching row, exactly as the PR description claims.
  • Downstream consumers of serve_timeout_override traced and confirmed unaffected: e2e.rs's isolate_cmd (raises ROCM_CLI_VLLM_READY_TIMEOUT_SECS/ROCM_CLI_AIM_READY_TIMEOUT_SECS) and inference_timeout_for both read the same value, but serve-06 resolves to the lemonade engine (unaffected by those two vLLM/AIM-only env vars) and fails at the serve step itself before any inference request — so lowering its resolved window from 2400s to 600s changes nothing for either consumer.
  • Good test coverage of both directions: serve_timeout_override_prefers_the_row_and_needs_an_id now asserts the row wins where it matches and that the tag still lengthens the wait where no row matches (traced by hand: reverting the .or() swap in Expectations::serve_timeout_override flips the first assertion from Some(90) to Some(2400), so the test does exercise the change). lemonade_alignment_rows_apply_only_to_native_windows_with_lemonade was extended to assert serve-06's real resolution on both Windows (Some(600)) and Ubuntu (Some(2400)) using the actual expectations.toml and feature-file tags, not just synthetic fixtures.
  • The second commit reconciles a real inconsistency: it rewrites the expectations.toml comment and reason so both describe serve-06's Windows failure as "launch exits 0, server never becomes ready" rather than leaving one paragraph say that and another say "crashes on first inference request" — the two previously disagreed about the actual failure mode.

Deployment notes

None — test-harness-only change (expectations matrix, resolver, and their doc comments/tests); no CLI behavior, output, or production code is touched.

What this covered

Full diff (e2e5c5bb3d…2e0bf0b029, verified via git merge-base) across all four changed files. Read tests/e2e-cucumber/src/expectation.rs in full (2485 lines: ScenarioDecl, Condition, XfailEntry, Expectations, and every test in the module); read tests/e2e-cucumber/expectations.toml in full; read the relevant sections of tests/e2e-cucumber/tests/e2e.rs (the serve_timeout_override field doc, the before hook, the env-var-raising site in isolate_cmd, and inference_timeout_for) and tests/e2e-cucumber/tests/e2e/serving_steps.rs; cross-checked every @serve-timeout-tagged scenario in the feature files against expectations.toml rows via git grep to independently verify the PR's "only serve-06 is affected" claim; checked tests/e2e-cucumber/README.md for staleness (none — its existing prose already matched the now-fixed precedence, so it needed no change); checked commit structure and messages (2 clean, signed-off, non-noise commits); read CI results via statusCheckRollup and traced the one failing/cancelled pair to its job logs.

Did not run: no local cargo/rustc toolchain in this sandbox — relied on tracing the logic by hand (including a manual mutation-trace of the .or() swap) plus the green Test (affected crates) / E2E tests CI jobs on this exact head commit as corroboration that the changed unit tests pass. History and prior-changes passes were not run separately given the PR's narrow, self-contained scope (2 commits, 4 files); both commits were read directly instead. Agent-instruction adherence: no AGENTS.md/CLAUDE.md found in the touched directories to check against.
Not reconciled against any discussion on the PR — this review was reached blind from the code, CI results, and the PR's own description only.


🤖 Automated by an AI agent

@juhovainio
juhovainio force-pushed the worktree-serve-06-windows-timeout branch from 2e0bf0b to 6668814 Compare November 16, 2026 14:32
@juhovainio

Copy link
Copy Markdown
Collaborator Author

Update on what has changed since the first push.

Review finding (non-blocking): two descriptions of serve-06's Windows failure
The row's comment and reason said the server crashes "on the first inference request", while the new serve_timeout_secs paragraph said the serve never becomes ready. The lane logs support the second: the 2026-10-27 Windows log shows rocm serve ... exited 0, but http://127.0.0.1:11435/v1/models never served ... within 2400s, and the failing step is the serve step, not the chat request. 66688147 rewrites both places to say that, keeping the access violation in the Tensile library from the service log as the suspected cause (still not reproduced off Windows). Nothing else in the repo pins the old wording.

Rebased onto current main
The branch was 12 commits behind, including #1080 (bounds torch and torchvision on the ROCm 10 install) and #1069. The rebase was clean, and the diff is unchanged apart from rewrapping one overlong comment line. Both commits are signed and signed off. cargo test -p e2e-cucumber --lib (367 passed), clippy with -D warnings, and cargo fmt --check pass on the rebased head.

MI300X lane

Current state
All other checks on 66688147 have passed. The MI300X job is the only one outstanding.

@juhovainio
juhovainio force-pushed the worktree-serve-06-windows-timeout branch from 6668814 to f5203f2 Compare November 17, 2026 10:26
…serve-timeout tag

serve-06 carries @serve-timeout:2400 for the hosts where it passes. On
native Windows with Lemonade it is a known failure (ROCMAI-900) whose
serve never becomes ready, and the tag won over the xfail row, so each
Windows lane waited the full 2400s (about 41 minutes) before failing.

A row means the serve is a known bug on this host, so its wait now takes
precedence; the tag still lengthens the wait wherever no row matches.
The Windows serve-06 row gets serve_timeout_secs = 600 (the step
default). Other hosts keep the 2400s window.

Signed-off-by: juhovainio <juho.vainio@amd.com>
…show it

The row's comment and reason said the server crashes on the first
inference request, while the new serve_timeout_secs paragraph said the
serve never becomes ready. The Windows lane logs show the second: the
launch exits 0, /v1/models never lists the model within the wait, and
the failing step is the serve step. Say that in both places, keeping
the Tensile access violation from the service log as the suspected cause.

Signed-off-by: juhovainio <juho.vainio@amd.com>
@juhovainio
juhovainio force-pushed the worktree-serve-06-windows-timeout branch from f5203f2 to 772a260 Compare November 17, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants