Repository navigation
Planner tests depend on wall-clock search budgets and flake under load #1067
Description
Why
Some planner tests assert which plan the search selects, or how its cost compares to another plan's, while the search stops on wall-clock budgets. Under CPU contention (a full cargo test --workspace run, or a busy CI runner) the search can stop earlier and pick a different plan, so the test fails.
Observed while working on #1063: memo_covers_whole_workload_pass2::topk_reuse failed once in a full run (memo … costs more than whole-workload …), then passed 3/3 when run alone. Its margin is about 0.02% (0.7755 vs 0.7756 per second).
Root cause (likely; to confirm)
The file header says the memo search "is bounded by MEMO_EVALUATIONS, with a time limit long enough not to bind". But inside the search, each phase has its own wall-clock budget, capped regardless of the overall limit (refinement/search.rs:317-328):
- baseline:
min(wall_time / 5, 2 s) - direct:
min(wall_time / 3, 2 s) - neighborhood:
min(wall_time / 5, 1 s)
So a large overall milp_time_limit_ms plus max_evaluations doesn't make the search deterministic. How many proposals a phase gets through still depends on machine speed and load.
Affected tests
Assert a selected plan or a cost relation under a time-bounded search:
integration-tests/tests/memo_covers_whole_workload_pass2.rs: all 7 (observed flake:topk_reuse)integration-tests/tests/sharing_issue_regressions.rs(30 s limit)integration-tests/tests/finer_aggregate_reuse.rs,identical_exact_queries.rs,independent_plan_seed.rs,production_rollup_workloads.rs,rollup_controlled.rs(30 s limit)integration-tests/tests/joint_search_equivalence.rs(10 s budget, bounded vs exhaustive)plan-selection/src/search/route.rsunit tests with 10 s limits- Any test using the default search (
milp_time_limit_ms: None→ 30 s) that asserts a specific plan. Not enumerated yet.
Race a sleep against a wall budget:
plan-selection/src/search/refinement/search.rs:1218, :1398(sleep 1.1 s / 2.1 s vs 1 s / 2 s budgets)
Assert a timeout does fire (low risk; would break only on a much faster machine):
types/tests/pane_chain.rs:459(1 ms deadline over 1M panes)route.rstests withmilp_time_limit_ms: Some(0)/Some(1)
Proposed fix
- Give the search a deterministic stopping mode for tests: phase limits in evaluations or attempts, with wall-clock only as a backstop that tests set high enough never to bind. The phase caps above are the main change.
- Switch the affected tests to that mode. Check that each still passes, and that none relies on a timeout to finish quickly.
- For tests that test timeout behavior itself, keep wall-clock but assert only the outcome a timeout guarantees (e.g. an error or a valid plan), not timing-dependent details.
Relation to #1066
#1066 trims slow tests. Several of its items touch these same tests: N1 (lower milp_time_limit_ms), N2 (memo_covers_whole_workload_pass2 to nightly), S1 (shrink the sleep/budget pairs). Lowering limits or shrinking budgets makes these failures more likely, and moving a test to nightly hides a flake without fixing it. Ideally this issue lands first, or together with those items.
Activity
milindsrivastava1997 commentedon Nov 2, 2026
Timing-dependent failures seen in CI on 2026-11-02 that this issue does not list yet:
devtools/tests/corpus_snapshots.rscorpus_plans_match_snapshots: under load a round stops at its own wall-clock budget and selects a different plan, reported as a snapshot diff.hit_time_limitonly sees the whole budget being used, not a round's own deadline.devtools/tests/stage_pipeline.rs: one test per run failed withexecution-scope expansion exceeded the deadline(default search).integration-tests/tests/planner_layering_example2.rsproduction_memo_selects_a_validated_example2_plan:Cost { .. Overflow }from the reference pricer, depending on which round was selected.#1164 fixes the last two causes, plus the
joint_search_equivalencefailure listed above: a preparation timeout ended planning instead of keeping earlier rounds' plans, and the test-only reference pricer failed on an overflowing candidate the search rejects.Tried and dropped for the corpus test: reporting a per-round
deadline_reachedand failing on it like the whole-budget limit. A release run on an idle machine showedo11y_benchq02, q06, q19, q22, q23 and q25 reaching an intermediate round's 5 s cap (INTERMEDIATE_ROUND_MAX) on every run while their plans match the snapshots, so the flag cannot tell load-induced drift from a stable plan. The intermediate-round cap and seed-round cap (SEED_ROUND_MAX) are wall-clock round budgets on top of the phase caps above; the deterministic stopping mode should cover them too.milindsrivastava1997 commentedon Nov 2, 2026
Decision (2026-11-02): park the fix for now. Mark the tests observed to flake on wall-clock budgets
#[ignore = "flaky: depends on wall-clock search budgets (#1067)"], and come back to this issue later.Direction for when we return: no time-based or count-based stopping anywhere, production included. The search stops only when it proves the plan optimal (
CandidateSearchStatus::Optimal). That means removing the phase caps, the round caps (SEED_ROUND_MAX,INTERMEDIATE_ROUND_MAX, running theCompletepane-width domain directly),MAX_EVALUATIONS,max_refinements, the 2 s joint-seed cap inrefinement/planner.rs, the HiGHStime_limit, and thetime_limit_mssetting. HiGHS already runs withmip_rel_gap = mip_abs_gap = 0. Corpus snapshot changes from this are acceptable. Main risk: planning time becomes unbounded, so measure the corpus first.Ignored now (each has failed on several unrelated branches,
mainincluded):planner_layering_example2::production_memo_selects_a_validated_example2_planjoint_search_equivalence::{custom_accuracy_policy_is_respected, different_windows_and_raw_retention_policies_produce_valid_plans, seeded_small_mixed_workloads_return_validated_plans}sharing_issue_regressions::{grouped_filtered_quantile_plans, nested_exact_sum_reuses_newest_panes, slower_repeat_reuses_faster_sum, sql_sum_shares_nested_windows_and_slower_readers}(exact_coarse_sum_reuses_fine_sumwas already ignored for Stage 3 keeps a MatAtQuery build whose NotMaterialized sibling avoids a second scan (#923 workload) #1142)rollup_controlled::production_distinct_rollup_finds_shared_kmvNot ignored, because the failures look like real bugs rather than timing:
stage_pipeline::workload_examples_preserve_data_and_valid_materialized_views:direct-cost objective … disagrees with evaluator …memo_enumeration::choices_whose_candidates_read_each_other_are_left_out: seen only on refactor branchescorpus_plans_match_snapshotsis already ignored in PR CI and runs nightly.test: ignore tests that flake on wall-clock search budgets (#1067) (#…
test: ignore every test whose result depends on a wall-clock budget o…
test: ignore the remaining tests that plan under a wall-clock budget (#…
test: ignore the remaining tests that plan under a wall-clock budget (#…
test: ignore the remaining tests that plan under a wall-clock budget (#…