Repository navigation
test(e2e): let an xfail row's serve_timeout_secs beat the scenario's serve-timeout tag - #1081
juhovainio wants to merge 2 commits into
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head 62ae35f3
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change. Implements the precedence fix described in the PR body (xfail row's serve_timeout_secs now beats the scenario's @serve-timeout tag); no GitHub issue is named, and the bug references (ROCMAI-899/ROCMAI-900) are external tracker IDs not retrievable from here.
Blocking
None
Non-blocking
- The comment block mixes two different descriptions of serve-06's Windows failure mode without reconciling them —
tests/e2e-cucumber/expectations.toml:253-254(unchanged by this PR) and thebug/reasontext at:274(also unchanged) describe the known bug as the server becoming ready and then crashing "in the SDK's Tensile library on the first inference request." The paragraph this PR adds at:260-264instead justifies the newserve_timeout_secs = 600by saying "the failing serve never became ready within that window," i.e. a readiness-poll timeout rather than a post-ready crash — which is also how the PR description frames it ("the serve never becomes ready"). If the server actually becomes ready quickly and only fails on the subsequent inference POST (as the unchanged prose says), the harness's existingmodel_is_readyreadiness poll would return well under the old 2400s, and shorteningserve_timeout_secswould have no effect on the ~41-minute lane time the PR is trying to fix — so only one of these two descriptions can be the real mechanism. Both read as confident, factual claims about the same bug in the same comment block, and nothing here says which one is stale. Worth reconciling (or just dropping the now-superseded "crashes in the SDK's Tensile library" framing at:253-254/:274if the newer readiness-timeout evidence is what's current) so a future reader — and whoever eventually verifies ROCMAI-900 is fixed — isn't working from two different theories of the failure.
Confidence 70 · mechanical · Fix: update or remove whichever of the two failure-mode descriptions the2026-10-27/2026-11-10run evidence has superseded, so the row'sreasonand the surrounding comment agree on what actually happens on the Windows lane.
Decisions for the author
None
Positive signals
- The precedence flip is logically sound and narrowly scoped: since an xfail row only matches a host where the scenario is already expected to fail there, letting the row win over the
@serve-timeouttag can only shorten the wait on a host that was never going to pass anyway — it cannot truncate a legitimately slow passing serve on another host, because the tag only applies where no row matches (src/expectation.rs'sserve_timeout_override). - Good test coverage of both directions of the new precedence —
serve_timeout_override_prefers_the_row_and_needs_an_idnow asserts the row wins on the known-bad host and that the tag still lengthens the wait on a host with no matching row, andlemonade_alignment_rows_apply_only_to_native_windows_with_lemonadewas extended to check both the Windows (Some(600)) and Ubuntu (Some(2400)) resolution for serve-06 instead of only asserting the old tag-always-wins behavior. - The
expectations.tomlrow comment cites concrete evidence (specific nightly run dates and observed lane duration) for the chosen600s value rather than an unexplained magic number.
Deployment notes
None — test-harness-only change (expectations matrix, resolver, and their doc comments/tests); no CLI behavior, output, or production code is touched.
What this covered
Full diff (e2e5c5bb3d…62ae35f3f9) across all four changed files, plus: the complete ScenarioDecl/Expectations/XfailEntry definitions and every caller of serve_timeout_for/serve_timeout_override traced via repo-wide grep (tests/e2e-cucumber/tests/e2e.rs's before hook and inference_timeout_for, tests/e2e-cucumber/tests/e2e/serving_steps.rs's serve_and_wait/serve_gpu_model/setup_large_gpu_model, and runtime_lifecycle_steps.rs's use of model_is_ready); the full surrounding expectations.toml comment block for both xfail rows; the model_serving.feature scenario tags for serve-06; and tests/e2e-cucumber/README.md's tag-reference table for staleness. CI on the reviewed commit is all green (including the E2E tests and affected-crate test jobs) except one E2E tests (MI300X, current) job still queued at review time — not treated as a finding since it had not failed.
Did not run: a local build/test (no cargo available in this environment) — relied on the green CI run for the reviewed commit instead. History/prior-changes passes were folded into the single pass above given the PR's narrow scope; agent-instruction adherence checked against this repo's AGENTS.md (no violations found — no new/renamed source files requiring docs/architecture.md entries, docs are already in sync).
Not reconciled against any discussion on the PR — this review was reached blind from the code and the PR's own description only.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head 2e0bf0b0
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. An earlier automated round from this account exists at the prior head 62ae35f3 (now stale); this is a fresh, independent pass against the current head and does not reuse its findings.
Review — needs work
Full review of the whole change. Implements the precedence fix described in the PR body (an xfail row's serve_timeout_secs now beats the scenario's @serve-timeout tag); ROCMAI-899/ROCMAI-900 are external tracker IDs not retrievable from here, and no GitHub issue is named as closed by this change (the #1005 mention is unrelated context, verified accurate: that PR is merged and fixes the stack-overflow it describes).
Blocking
- A required check is failing on the reviewed commit —
E2E consolidated report (self-hosted)(run 43598171626). Its "Check every lane that ran uploaded its report" step fails because theE2E tests (MI300X, current)job wascancelledafter sitting for a full 24 hours with no steps recorded, which reads as the self-hosted MI300X runner never picking the job up (GitHub's own job-duration cap, not a test failure) rather than anything this diff touches — the diff is test-harness timeout logic unrelated to MI300X dispatch. All other checks on this commit are green, includingTest (affected crates),clippy,E2E tests (Strix Halo, Windows/WSL2, current), andCommit signatures + sign-off.
Confidence 85 · architectural · Fix: re-run the self-hosted E2E workflow (or wait for an MI300X runner to become available) rather than changing code; flag the runner-availability gap to whoever owns that pool if it recurs.
Non-blocking
None
Decisions for the author
None
Positive signals
- The precedence flip is logically sound and provably narrow:
is_xfailandserve_timeout_forshare the sameCondition::matchesgate, so a row can only shorten the wait on a host where the scenario is already expected to fail there — verified by grepping every@serve-timeout-tagged scenario (serve-large-model-inference,serve-aim-inference,serve-aim-lifecycle-parity) againstexpectations.tomland confirming only serve-06 carries a matching row, exactly as the PR description claims. - Downstream consumers of
serve_timeout_overridetraced and confirmed unaffected:e2e.rs'sisolate_cmd(raisesROCM_CLI_VLLM_READY_TIMEOUT_SECS/ROCM_CLI_AIM_READY_TIMEOUT_SECS) andinference_timeout_forboth read the same value, but serve-06 resolves to thelemonadeengine (unaffected by those two vLLM/AIM-only env vars) and fails at the serve step itself before any inference request — so lowering its resolved window from 2400s to 600s changes nothing for either consumer. - Good test coverage of both directions:
serve_timeout_override_prefers_the_row_and_needs_an_idnow asserts the row wins where it matches and that the tag still lengthens the wait where no row matches (traced by hand: reverting the.or()swap inExpectations::serve_timeout_overrideflips the first assertion fromSome(90)toSome(2400), so the test does exercise the change).lemonade_alignment_rows_apply_only_to_native_windows_with_lemonadewas extended to assert serve-06's real resolution on both Windows (Some(600)) and Ubuntu (Some(2400)) using the actualexpectations.tomland feature-file tags, not just synthetic fixtures. - The second commit reconciles a real inconsistency: it rewrites the
expectations.tomlcomment andreasonso both describe serve-06's Windows failure as "launch exits 0, server never becomes ready" rather than leaving one paragraph say that and another say "crashes on first inference request" — the two previously disagreed about the actual failure mode.
Deployment notes
None — test-harness-only change (expectations matrix, resolver, and their doc comments/tests); no CLI behavior, output, or production code is touched.
What this covered
Full diff (e2e5c5bb3d…2e0bf0b029, verified via git merge-base) across all four changed files. Read tests/e2e-cucumber/src/expectation.rs in full (2485 lines: ScenarioDecl, Condition, XfailEntry, Expectations, and every test in the module); read tests/e2e-cucumber/expectations.toml in full; read the relevant sections of tests/e2e-cucumber/tests/e2e.rs (the serve_timeout_override field doc, the before hook, the env-var-raising site in isolate_cmd, and inference_timeout_for) and tests/e2e-cucumber/tests/e2e/serving_steps.rs; cross-checked every @serve-timeout-tagged scenario in the feature files against expectations.toml rows via git grep to independently verify the PR's "only serve-06 is affected" claim; checked tests/e2e-cucumber/README.md for staleness (none — its existing prose already matched the now-fixed precedence, so it needed no change); checked commit structure and messages (2 clean, signed-off, non-noise commits); read CI results via statusCheckRollup and traced the one failing/cancelled pair to its job logs.
Did not run: no local cargo/rustc toolchain in this sandbox — relied on tracing the logic by hand (including a manual mutation-trace of the .or() swap) plus the green Test (affected crates) / E2E tests CI jobs on this exact head commit as corroboration that the changed unit tests pass. History and prior-changes passes were not run separately given the PR's narrow, self-contained scope (2 commits, 4 files); both commits were read directly instead. Agent-instruction adherence: no AGENTS.md/CLAUDE.md found in the touched directories to check against.
Not reconciled against any discussion on the PR — this review was reached blind from the code, CI results, and the PR's own description only.
🤖 Automated by an AI agent
2e0bf0b to
6668814
Compare
|
Update on what has changed since the first push. Review finding (non-blocking): two descriptions of serve-06's Windows failure Rebased onto current MI300X lane
Current state |
6668814 to
f5203f2
Compare
…serve-timeout tag serve-06 carries @serve-timeout:2400 for the hosts where it passes. On native Windows with Lemonade it is a known failure (ROCMAI-900) whose serve never becomes ready, and the tag won over the xfail row, so each Windows lane waited the full 2400s (about 41 minutes) before failing. A row means the serve is a known bug on this host, so its wait now takes precedence; the tag still lengthens the wait wherever no row matches. The Windows serve-06 row gets serve_timeout_secs = 600 (the step default). Other hosts keep the 2400s window. Signed-off-by: juhovainio <juho.vainio@amd.com>
…show it The row's comment and reason said the server crashes on the first inference request, while the new serve_timeout_secs paragraph said the serve never becomes ready. The Windows lane logs show the second: the launch exits 0, /v1/models never lists the model within the wait, and the failing step is the serve step. Say that in both places, keeping the Tensile access violation from the service log as the suspected cause. Signed-off-by: juhovainio <juho.vainio@amd.com>
f5203f2 to
772a260
Compare
Problem
serve-06(serve-large-model-inference) carries@serve-timeout:2400for the hosts where it passes. On native Windows with Lemonade it is a known failure (ROCMAI-900): the serve never becomes ready. The harness resolved the wait as tag first, then xfail row, so the tag won and every Windows lane waited the full 2400s before failing. In the nightly runs of 2026-10-27 and 2026-11-10 that was about 41 minutes per Windows job, which made the Windows lanes the longest in the run (about 65 minutes against 19 to 35 for the Linux lanes).The docs already describe the intent (the tag lengthens an expected-pass serve, the row shortens a known-bug serve); the precedence in
Expectations::serve_timeout_overridedid not follow it, and theexpectations.tomlcomment recorded the gap ("still holds the Windows lane for the full 2400s wait").Change
Expectations::serve_timeout_override: the shortest matching xfailserve_timeout_secsnow wins; the@serve-timeouttag applies where no row matches the host.expectations.toml: the Windowsserve-large-model-inferencerow getsserve_timeout_secs = 600(the step default), and its comment is updated.e2e.rsandserving_steps.rsupdated to the new precedence.Only serve-06 is affected: no other row sets
serve_timeout_secson a scenario that also has a@serve-timeouttag. Hosts where no row matches (Linux, WSL2, MI300X, W7900) keep the 2400s window.Verification
cargo test -p e2e-cucumber --lib: 343 passed.cargo clippy -p e2e-cucumber --all-targets -- -D warningsandcargo fmt --checkare clean.serve_timeout_override_prefers_the_row_and_needs_an_id,lemonade_alignment_rows_apply_only_to_native_windows_with_lemonade) and pass with this change. The first replaces the test that asserted a tag lengthens a row's timeout, which is the behaviour being changed. The second now asserts the resolved wait for serve-06 on Windows (600) and on a host with no row (2400).Scope and gaps
@id:serve-large-model-inference(serve-06), which stays an expected failure on native Windows with Lemonade.