Repository navigation
test: ignore every test whose result depends on a wall-clock budget or deadline (#1067) - #1180
Merged
Merged
Conversation
…r deadline (#1067) Follows #1166. A search budget (wall_time, time_limit_ms) or a short real deadline (Instant::now() + a few ms or s) decides these tests' outcome on a slow or loaded machine; e.g. structural_seed_timeout_preserves_validated_incumbent sleeps 1.1 s against a 1 s budget and fails under load. Stage 3 is being replaced (#1095), and #1067 moves to optimal-only stopping; until then they are ignored. Tests whose deadline is already expired (Instant::now(), Duration::ZERO) or whose fake evaluator never reaches its budget stay.
milindsrivastava1997
added a commit
that referenced
this pull request
Nov 2, 2026
…1175) #1166 and #1180 ignored the tests seen failing on deadlines; every other test that reaches the time-limited search (plan_installable, e2e_plan, optimize, run_planning, or the planner binary) can fail the same way under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or rewriting each once Stage 3's design (#1095, #1067) is final. Tests that only use select_among_plans or installable_plan_of (no deadline) still run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Nov 2, 2026
zzylol
added a commit
that referenced
this pull request
Nov 2, 2026
…ding (#1163, part 2) (#1176) * refactor: rename every remaining pane name to the tumbling-window wording (#1163, part 2) Helper types, functions, modules, files, test names, comments and docs: Pane* becomes TumblingWindow*, pane_* becomes tumbling_window_*, and the word "pane" becomes "tumbling window" (e.g. PaneWidthDomain -> TumblingWindowWidthDomain, pane_chain.rs -> tumbling_window_chain.rs, expand_pane_chains -> expand_tumbling_window_chains). The planning report's pane_width_domain and pane_widths_pending become tumbling_window_* too, and the DAG viewer upgrades them in older documents. The deployed wire contract keeps its names: TumblingWindowLayout still reads and writes pane_width_ms and pane_origin_ms, WindowEdgeCoverage pane_aligned, TumblingWindowCoverageError ZeroPaneWidth / UnknownPaneOrigin / pane_phase_ms, and WindowEdgeCompatibility RequiresAlignedPanePhaseOrExactBoundaryResidual. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: apply review of #1163 part 2 - viewer: expanded tumbling-window nodes keep one CSS class (tumblingWindow) that app.js styles, with a test; JS data fields are camelCase; - historical docs, artifact paths and plan ids keep their recorded names; - code names inside comments and labels keep identifier form; - wording: doubled words, hyphenated compound modifiers, contrasted terms; - snapshot labels updated to the new display and cost text; - test constants name what they hold (TUMBLING_WINDOW_COUNT, ...); - the coverage errors' wire names are tested; cargo fmt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: snapshot keys in sorted order after the tumbling_window_retention_ms rename Part 1 renamed pane_retention_ms in the snapshot JSON in place; the snapshots store keys sorted, so the renamed key now sorts after proven_optimal. The regenerated snapshots equal the checked-in ones as JSON (22 of 22). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: rename pane names that main added since part 2 started (#1163) Reruns the part 2 rename over the files main changed (#1174, #1178, #1180, #1182) and reapplies the review's wording fixes there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
milindsrivastava1997
added a commit
that referenced
this pull request
Nov 2, 2026
…1175) #1166 and #1180 ignored the tests seen failing on deadlines; every other test that reaches the time-limited search (plan_installable, e2e_plan, optimize, run_planning, or the planner binary) can fail the same way under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or rewriting each once Stage 3's design (#1095, #1067) is final. Tests that only use select_among_plans or installable_plan_of (no deadline) still run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
milindsrivastava1997
added a commit
that referenced
this pull request
Nov 2, 2026
…1175) (#1181) #1166 and #1180 ignored the tests seen failing on deadlines; every other test that reaches the time-limited search (plan_installable, e2e_plan, optimize, run_planning, or the planner binary) can fail the same way under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or rewriting each once Stage 3's design (#1095, #1067) is final. Tests that only use select_among_plans or installable_plan_of (no deadline) still run. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Tests whose result depends on a wall-clock budget fail on a slow or loaded machine. For example,
structural_seed_timeout_preserves_validated_incumbentsleeps 1.1 s against a 1 s search budget; it passed 3 of 3 runs alone and failed when two other builds shared the CPU. #1166 ignored the nine such tests that had flaked on CI. Stage 3 is being replaced (#1095), and #1067 moves the search to optimal-only stopping, so until then every test of this kind is ignored, not only those seen flaking.What
36 tests get
#[ignore = "flaky: depends on wall-clock search budgets (#1067)"]or#[ignore = "flaky: depends on a wall-clock deadline (#1067)"]. They run withcargo test -- --ignored.wall_time/time_limit_mswhose result can change when the budget is hit:plan-selectionrefinement/planner.rs: 6 tests;refinement/search.rs: the 2 tests that sleep past their budget;integration-tests:finer_aggregate_reuse,identical_exact_queries(2),independent_plan_seed(3),joint_search_equivalence(7),memo_enumeration(1),production_rollup_workloads(4),sharing_issue_regressions(1);plannerruntime_install.rs: 2 tests.Instant::now()plus a few ms or s, expecting expiry or completion:executorphysical_planner(2);logical-asap-optimizationshared_configurations.rs(1);typesflat.rs(1),tests/pane_chain.rs(2),tests/physical_export.rs(1).Not ignored:
Some(Instant::now()),Duration::ZERO, a zero budget), which are deterministic;search.rs/materialization.rs/objectives.rsunit tests whose fake evaluators or tiny graphs return far inside their 5–30 s budgets;budget_exhaustedflag test.How
The tests were selected by a scan of every
#[test]/#[tokio::test]whose body, or a helper in the same file that it calls (transitively), sets a real budget (wall_time: Duration::from_*(n>0),time_limit_ms: Some(..),thread::sleep,Instant::now() + Duration). The list was then triaged by hand into the groups above.Key code interfaces added / modified
None (test attributes only).
Major algorithm changes
None.
Before this PR
These tests could fail on a loaded CI runner or developer machine, on branches unrelated to them.
After this PR
They are skipped by default and still run with
--ignored.Verification
cargo fmt --all --check: clean.cargo check --testsforasap-types,asap-executor,asap-logical-asap-optimization,asap-plan-selection,asap-planner,asap-integration-tests: clean.cargo test -p asap-plan-selection --lib -- --ignored --listlists the 8 newly ignored plan-selection tests.