Repository navigation
Description
Problem, by example
The same exact query twice, one copy every 10 s and one every 15 s, over a counter m with 6 series sampled every second, on a deployment that keeps no raw data (raw_data_retained: false):
q0: sum by (job) (rate(m[1m])) fixed_interval 10000
q1: sum by (job) (rate(m[1m])) fixed_interval 15000
This issue uses the names from #1163 (SummaryTumblingWindowChain for today's PaneChain, tumbling_window_* for today's pane_*). Outputs quoted from today's planner keep the old names.
plan_installable on origin/main (c83aa19) selects one SummaryTumblingWindowChain of 4 × 15000 ms, built at ingestion time, and both queries read it. The plan's materialized views:
VIEW node=2 refresh=AtIngestion windows=[(15000, (-15000, 0))] retention_ms=60000
VIEW node=5 refresh=AtIngestion windows=[(15000, (-30000, -15000))] retention_ms=60000
VIEW node=8 refresh=AtIngestion windows=[(15000, (-45000, -30000))] retention_ms=60000
VIEW node=11 refresh=AtIngestion windows=[(15000, (-60000, -45000))] retention_ms=60000
The backend builds each tumbling window once, on the 15 s grid anchored at the epoch. q0 evaluates at multiples of 10 s, so at +10 s, +20 s, +40 s and +50 s its newest tumbling window ends up to 10 s before the evaluation time, and q0 reads the wrong 60 s. Run through the stand-in runtime in crates/planner/tests/runtime_install.rs (IngestionBuilds::Tumbling, which builds tumbling windows as the backend does) and compared with the query-time reference:
q0 (every 10 s) at +0 s: matches reference
q1 (every 15 s) at +0 s: matches reference
q0 (every 10 s) at +10 s: MISMATCH plan=[a: 1593.94, b: 3257.93] reference=[a: 1665.15, b: 3343.12]
q1 (every 15 s) at +15 s: matches reference
q0 (every 10 s) at +20 s: MISMATCH plan=[a: 1867.29, b: 3833.82] reference=[a: 1906.83, b: 3889.88]
q0 (every 10 s) at +30 s: matches reference
q1 (every 15 s) at +30 s: matches reference
q0 (every 10 s) at +40 s: MISMATCH plan=[a: 2043.98, b: 4161.77] reference=[a: 2206.98, b: 4495.12]
q1 (every 15 s) at +45 s: matches reference
q0 (every 10 s) at +50 s: MISMATCH plan=[a: 2325.59, b: 4745.93] reference=[a: 2377.98, b: 4818.66]
q0 (every 10 s) at +60 s: matches reference
q1 (every 15 s) at +60 s: matches reference
sum by (job) (sum_over_time(m[1m])) gives the same pattern: q0 is wrong at every evaluation that is not a multiple of 15 s. The query is exact, so any difference from the reference is a wrong answer.
stage_pipeline --text shows the same selection for rate(x[1m]) every 10 s and every 15 s:
node2: C21 PaneChain pane_count=4 pane_width_ms=15000 <- node0
node_of_output_reference: [node1, node1]
reader_constraints: AllowedOutputs { candidate: PhysicalCandidateId(21), outputs: [0, 1] }
What should happen instead: an ingestion-time chain read by both queries must have a width that divides both intervals: 5000 ms (12 tumbling windows), or each query must read its own chain. A 15000 ms chain may serve q0 only when it is built at query time, over the window ending at q0's evaluation time.
Why it happens
- Stage 1 offers widths that fit only one reader. The two copies are hash-consed into one memo group read by both queries.
local_pane_candidates_delta(crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs:173) builds the width domain from all readers' grids {10000, 15000}. It then callspane_candidates(&roots, &[Some(grid)])once per offered width, as if that width were the only reader's grid. So the group gets 10000 ms and 15000 ms chains next to the 5000 ms one, and nothing records which readers each chain can serve. - Stage 2's ingestion check skips readers with no declared phase.
ingestion_eligibility(crates/physical-asap-optimization/src/lib.rs:816) checks that every reader of a chain evaluates on the chain's tumbling-window boundaries. Butplan_pane_phasereturns no origin forRepeatedDemand::FixedInterval, and the loopcontinues for such a reader ("A reader of unknown phase is taken to run on them"). AFixedIntervalreader is taken to evaluate at multiples of its interval, phase 0, as the whole-window branch below assumes. Nothing checks that the interval is a multiple of the width. The whole-window branch does check this (interval_ms.is_multiple_of(window_ms),WindowOffEpochGrid); the chain branch does not. - What already works nearby:
- Readers that declare a phase are checked (
PanesOffPhase). - Range evaluations whose step the width does not divide are pruned (
PaneWidthOffRangeStep). queries_at_two_phases_read_panes_on_both_schedulescovers two copies at the same interval and different phases.- No test covers two copies at different intervals.
- Readers that declare a phase are checked (
Proposal
- Stage 2 (the correctness fix). In
ingestion_eligibility's chain branch, a repeated reader with no declared phase must haveinterval_ms % tumbling_window_width_ms == 0; otherwise the chain is not eligible at ingestion time (PanesOffPhase, or a new reason such asPanesOffInterval). A 15000 ms chain then stays a query-time candidate only, and Stage 3 has to pick the 5000 ms chain (or separate builds) to keep the tumbling windows at ingestion time. - Stage 1 (fewer wrong candidates). Give a chain read by several readers of one group a width that divides every reader's grid, i.e. the gcd. A width that fits only some readers is useful only if those readers can read a different candidate than the others. One memo group serves all its readers with one choice, so that does not work today. Pass 2: offer only gcd tumbling-window widths; finer divisors are dominated #1171 narrows the width domain to gcd widths and should take the readers of one group into account the same way.
Fix 1 alone makes the plans correct. Fix 2 removes candidates that fix 1 would only rule out for ingestion time.
Scope and acceptance
In scope: Stage 2's ingestion eligibility for SummaryTumblingWindowChain readers with a fixed interval and no declared phase, and the regression test.
Out of scope: the general width-domain pruning (#1171).
Acceptance:
- Regression test. In
crates/planner/tests/runtime_install.rs, plansum by (job) (rate(m[1m]))andsum by (job) (sum_over_time(m[1m]))each every 10 s and every 15 s on a deployment withraw_data_retained: false. Run every evaluation over one minute withIngestionBuilds::Tumbling. Every answer equals the query-time reference. Today this test fails at +10, +20, +40 and +50 s for the 10 s copy. - Unit test.
ingestion_eligibilityrejects a 15000 ms chain read by aFixedInterval(10000)reader and accepts a 5000 ms chain read byFixedInterval(10000)andFixedInterval(15000). - No other plan changes.
corpus_snapshotsandsketch_plan_snapshotschange only where a chain's width does not divide a reader's interval, and the PR lists each change.
Status and related work
- Verified (2026-11-02): reproduced on origin/main c83aa19 through
plan_installableand the stand-in runtime ofruntime_install.rs, for bothrateandsum_over_time. The test is a scratch test, not committed. - Pass 2: offer only gcd tumbling-window widths; finer divisors are dominated #1171: gcd widths for Pass 2's tumbling-window candidates. Found while checking its "mixed grids" acceptance case.
- feat(devtools): stage_pipeline --text prints every planner stage's output as text #1159:
stage_pipeline --text, which showed the selection.
Activity
zzylol commentedon Nov 2, 2026
Fixed by #1178 (merged as 2c99451): every reader with a fixed interval must have an interval the chain's width divides before the chain can be kept at ingestion time.
Follow-up found in review, not fixed by #1178 (it predates it): an
EstimatedRatereader has no fixed interval and no phase, so the chain branch ofingestion_eligibilitystillcontinues for it. An ingestion-time chain therefore stays eligible for a reader whose evaluation times are unknown. The whole-window branch rejects the same reader (WindowNotFixed). Example: a predictableEstimatedRatereader of a 15 s chain withraw_data_retained: falseevaluates at arbitrary times and reads windows that end up to 15 s early. RejectingEstimatedRatereaders in the chain branch, as the whole-window branch does, would close it.fix(stage2): keep a tumbling-window chain at ingestion time only on i…