Skip to content

test: ignore every test whose result depends on a wall-clock budget or deadline (#1067) - #1180

Merged
zzylol merged 1 commit into
mainfrom
test/ignore-wall-clock-budget-tests
Nov 2, 2026
Merged

zzylol merged 1 commit into
mainfrom
test/ignore-wall-clock-budget-tests

Conversation

@zzylol

@zzylol zzylol commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Why

Tests whose result depends on a wall-clock budget fail on a slow or loaded machine. For example, structural_seed_timeout_preserves_validated_incumbent sleeps 1.1 s against a 1 s search budget; it passed 3 of 3 runs alone and failed when two other builds shared the CPU. #1166 ignored the nine such tests that had flaked on CI. Stage 3 is being replaced (#1095), and #1067 moves the search to optimal-only stopping, so until then every test of this kind is ignored, not only those seen flaking.

What

36 tests get #[ignore = "flaky: depends on wall-clock search budgets (#1067)"] or #[ignore = "flaky: depends on a wall-clock deadline (#1067)"]. They run with cargo test -- --ignored.

  • Search budgets, a real search under wall_time / time_limit_ms whose result can change when the budget is hit:
    • plan-selection refinement/planner.rs: 6 tests;
    • refinement/search.rs: the 2 tests that sleep past their budget;
    • integration-tests: finer_aggregate_reuse, identical_exact_queries (2), independent_plan_seed (3), joint_search_equivalence (7), memo_enumeration (1), production_rollup_workloads (4), sharing_issue_regressions (1);
    • planner runtime_install.rs: 2 tests.
  • Deadlines, Instant::now() plus a few ms or s, expecting expiry or completion:
    • executor physical_planner (2);
    • logical-asap-optimization shared_configurations.rs (1);
    • types flat.rs (1), tests/pane_chain.rs (2), tests/physical_export.rs (1).

Not ignored:

  • tests whose deadline is already expired (Some(Instant::now()), Duration::ZERO, a zero budget), which are deterministic;
  • search.rs / materialization.rs / objectives.rs unit tests whose fake evaluators or tiny graphs return far inside their 5–30 s budgets;
  • the budget_exhausted flag test.

How

The tests were selected by a scan of every #[test] / #[tokio::test] whose body, or a helper in the same file that it calls (transitively), sets a real budget (wall_time: Duration::from_*(n>0), time_limit_ms: Some(..), thread::sleep, Instant::now() + Duration). The list was then triaged by hand into the groups above.

Key code interfaces added / modified

None (test attributes only).

Major algorithm changes

None.

Before this PR

These tests could fail on a loaded CI runner or developer machine, on branches unrelated to them.

After this PR

They are skipped by default and still run with --ignored.

Verification

  • cargo fmt --all --check: clean.
  • cargo check --tests for asap-types, asap-executor, asap-logical-asap-optimization, asap-plan-selection, asap-planner, asap-integration-tests: clean.
  • cargo test -p asap-plan-selection --lib -- --ignored --list lists the 8 newly ignored plan-selection tests.

…r deadline (#1067)

Follows #1166. A search budget (wall_time, time_limit_ms) or a short real
deadline (Instant::now() + a few ms or s) decides these tests' outcome on a
slow or loaded machine; e.g. structural_seed_timeout_preserves_validated_incumbent
sleeps 1.1 s against a 1 s budget and fails under load. Stage 3 is being
replaced (#1095), and #1067 moves to optimal-only stopping; until then they
are ignored. Tests whose deadline is already expired (Instant::now(),
Duration::ZERO) or whose fake evaluator never reaches its budget stay.
@zzylol zzylol added the testing Corpus, regression, differential, or benchmark tests label Nov 2, 2026
@zzylol
zzylol merged commit 59e8825 into main Nov 2, 2026
3 checks passed
milindsrivastava1997 added a commit that referenced this pull request Nov 2, 2026
…1175)

#1166 and #1180 ignored the tests seen failing on deadlines; every other
test that reaches the time-limited search (plan_installable, e2e_plan,
optimize, run_planning, or the planner binary) can fail the same way
under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or
rewriting each once Stage 3's design (#1095, #1067) is final. Tests
that only use select_among_plans or installable_plan_of (no deadline)
still run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Nov 2, 2026
Reruns the part 2 rename over the files main changed (#1174, #1178, #1180,
#1182) and reapplies the review's wording fixes there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Nov 2, 2026
…ding (#1163, part 2) (#1176)

* refactor: rename every remaining pane name to the tumbling-window wording (#1163, part 2)

Helper types, functions, modules, files, test names, comments and docs: Pane*
becomes TumblingWindow*, pane_* becomes tumbling_window_*, and the word
"pane" becomes "tumbling window" (e.g. PaneWidthDomain ->
TumblingWindowWidthDomain, pane_chain.rs -> tumbling_window_chain.rs,
expand_pane_chains -> expand_tumbling_window_chains). The planning report's
pane_width_domain and pane_widths_pending become tumbling_window_* too, and
the DAG viewer upgrades them in older documents.

The deployed wire contract keeps its names: TumblingWindowLayout still reads
and writes pane_width_ms and pane_origin_ms, WindowEdgeCoverage pane_aligned,
TumblingWindowCoverageError ZeroPaneWidth / UnknownPaneOrigin / pane_phase_ms,
and WindowEdgeCompatibility RequiresAlignedPanePhaseOrExactBoundaryResidual.
No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: apply review of #1163 part 2

- viewer: expanded tumbling-window nodes keep one CSS class (tumblingWindow)
  that app.js styles, with a test; JS data fields are camelCase;
- historical docs, artifact paths and plan ids keep their recorded names;
- code names inside comments and labels keep identifier form;
- wording: doubled words, hyphenated compound modifiers, contrasted terms;
- snapshot labels updated to the new display and cost text;
- test constants name what they hold (TUMBLING_WINDOW_COUNT, ...);
- the coverage errors' wire names are tested; cargo fmt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: snapshot keys in sorted order after the tumbling_window_retention_ms rename

Part 1 renamed pane_retention_ms in the snapshot JSON in place; the snapshots
store keys sorted, so the renamed key now sorts after proven_optimal. The
regenerated snapshots equal the checked-in ones as JSON (22 of 22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: rename pane names that main added since part 2 started (#1163)

Reruns the part 2 rename over the files main changed (#1174, #1178, #1180,
#1182) and reapplies the review's wording fixes there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
milindsrivastava1997 added a commit that referenced this pull request Nov 2, 2026
…1175)

#1166 and #1180 ignored the tests seen failing on deadlines; every other
test that reaches the time-limited search (plan_installable, e2e_plan,
optimize, run_planning, or the planner binary) can fail the same way
under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or
rewriting each once Stage 3's design (#1095, #1067) is final. Tests
that only use select_among_plans or installable_plan_of (no deadline)
still run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
milindsrivastava1997 added a commit that referenced this pull request Nov 2, 2026
…1175) (#1181)

#1166 and #1180 ignored the tests seen failing on deadlines; every other
test that reaches the time-limited search (plan_installable, e2e_plan,
optimize, run_planning, or the planner binary) can fail the same way
under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or
rewriting each once Stage 3's design (#1095, #1067) is final. Tests
that only use select_among_plans or installable_plan_of (no deadline)
still run.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol deleted the test/ignore-wall-clock-budget-tests branch November 3, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant