Skip to content

Stage 2: an ingestion-time tumbling-window chain serves a reader whose interval its width does not divide (wrong answers) #1173

Closed

Description

@zzylol

Problem, by example

The same exact query twice, one copy every 10 s and one every 15 s, over a counter m with 6 series sampled every second, on a deployment that keeps no raw data (raw_data_retained: false):

q0: sum by (job) (rate(m[1m]))     fixed_interval 10000
q1: sum by (job) (rate(m[1m]))     fixed_interval 15000

This issue uses the names from #1163 (SummaryTumblingWindowChain for today's PaneChain, tumbling_window_* for today's pane_*). Outputs quoted from today's planner keep the old names.

plan_installable on origin/main (c83aa19) selects one SummaryTumblingWindowChain of 4 × 15000 ms, built at ingestion time, and both queries read it. The plan's materialized views:

VIEW node=2  refresh=AtIngestion windows=[(15000, (-15000, 0))]      retention_ms=60000
VIEW node=5  refresh=AtIngestion windows=[(15000, (-30000, -15000))] retention_ms=60000
VIEW node=8  refresh=AtIngestion windows=[(15000, (-45000, -30000))] retention_ms=60000
VIEW node=11 refresh=AtIngestion windows=[(15000, (-60000, -45000))] retention_ms=60000

The backend builds each tumbling window once, on the 15 s grid anchored at the epoch. q0 evaluates at multiples of 10 s, so at +10 s, +20 s, +40 s and +50 s its newest tumbling window ends up to 10 s before the evaluation time, and q0 reads the wrong 60 s. Run through the stand-in runtime in crates/planner/tests/runtime_install.rs (IngestionBuilds::Tumbling, which builds tumbling windows as the backend does) and compared with the query-time reference:

q0 (every 10 s) at +0 s:  matches reference
q1 (every 15 s) at +0 s:  matches reference
q0 (every 10 s) at +10 s: MISMATCH plan=[a: 1593.94, b: 3257.93] reference=[a: 1665.15, b: 3343.12]
q1 (every 15 s) at +15 s: matches reference
q0 (every 10 s) at +20 s: MISMATCH plan=[a: 1867.29, b: 3833.82] reference=[a: 1906.83, b: 3889.88]
q0 (every 10 s) at +30 s: matches reference
q1 (every 15 s) at +30 s: matches reference
q0 (every 10 s) at +40 s: MISMATCH plan=[a: 2043.98, b: 4161.77] reference=[a: 2206.98, b: 4495.12]
q1 (every 15 s) at +45 s: matches reference
q0 (every 10 s) at +50 s: MISMATCH plan=[a: 2325.59, b: 4745.93] reference=[a: 2377.98, b: 4818.66]
q0 (every 10 s) at +60 s: matches reference
q1 (every 15 s) at +60 s: matches reference

sum by (job) (sum_over_time(m[1m])) gives the same pattern: q0 is wrong at every evaluation that is not a multiple of 15 s. The query is exact, so any difference from the reference is a wrong answer.

stage_pipeline --text shows the same selection for rate(x[1m]) every 10 s and every 15 s:

node2:  C21  PaneChain pane_count=4 pane_width_ms=15000  <- node0
node_of_output_reference: [node1, node1]
reader_constraints: AllowedOutputs { candidate: PhysicalCandidateId(21), outputs: [0, 1] }

What should happen instead: an ingestion-time chain read by both queries must have a width that divides both intervals: 5000 ms (12 tumbling windows), or each query must read its own chain. A 15000 ms chain may serve q0 only when it is built at query time, over the window ending at q0's evaluation time.

Why it happens

  • Stage 1 offers widths that fit only one reader. The two copies are hash-consed into one memo group read by both queries. local_pane_candidates_delta (crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs:173) builds the width domain from all readers' grids {10000, 15000}. It then calls pane_candidates(&roots, &[Some(grid)]) once per offered width, as if that width were the only reader's grid. So the group gets 10000 ms and 15000 ms chains next to the 5000 ms one, and nothing records which readers each chain can serve.
  • Stage 2's ingestion check skips readers with no declared phase. ingestion_eligibility (crates/physical-asap-optimization/src/lib.rs:816) checks that every reader of a chain evaluates on the chain's tumbling-window boundaries. But plan_pane_phase returns no origin for RepeatedDemand::FixedInterval, and the loop continues for such a reader ("A reader of unknown phase is taken to run on them"). A FixedInterval reader is taken to evaluate at multiples of its interval, phase 0, as the whole-window branch below assumes. Nothing checks that the interval is a multiple of the width. The whole-window branch does check this (interval_ms.is_multiple_of(window_ms), WindowOffEpochGrid); the chain branch does not.
  • What already works nearby:
    • Readers that declare a phase are checked (PanesOffPhase).
    • Range evaluations whose step the width does not divide are pruned (PaneWidthOffRangeStep).
    • queries_at_two_phases_read_panes_on_both_schedules covers two copies at the same interval and different phases.
    • No test covers two copies at different intervals.

Proposal

  1. Stage 2 (the correctness fix). In ingestion_eligibility's chain branch, a repeated reader with no declared phase must have interval_ms % tumbling_window_width_ms == 0; otherwise the chain is not eligible at ingestion time (PanesOffPhase, or a new reason such as PanesOffInterval). A 15000 ms chain then stays a query-time candidate only, and Stage 3 has to pick the 5000 ms chain (or separate builds) to keep the tumbling windows at ingestion time.
  2. Stage 1 (fewer wrong candidates). Give a chain read by several readers of one group a width that divides every reader's grid, i.e. the gcd. A width that fits only some readers is useful only if those readers can read a different candidate than the others. One memo group serves all its readers with one choice, so that does not work today. Pass 2: offer only gcd tumbling-window widths; finer divisors are dominated #1171 narrows the width domain to gcd widths and should take the readers of one group into account the same way.

Fix 1 alone makes the plans correct. Fix 2 removes candidates that fix 1 would only rule out for ingestion time.

Scope and acceptance

In scope: Stage 2's ingestion eligibility for SummaryTumblingWindowChain readers with a fixed interval and no declared phase, and the regression test.

Out of scope: the general width-domain pruning (#1171).

Acceptance:

  • Regression test. In crates/planner/tests/runtime_install.rs, plan sum by (job) (rate(m[1m])) and sum by (job) (sum_over_time(m[1m])) each every 10 s and every 15 s on a deployment with raw_data_retained: false. Run every evaluation over one minute with IngestionBuilds::Tumbling. Every answer equals the query-time reference. Today this test fails at +10, +20, +40 and +50 s for the 10 s copy.
  • Unit test. ingestion_eligibility rejects a 15000 ms chain read by a FixedInterval(10000) reader and accepts a 5000 ms chain read by FixedInterval(10000) and FixedInterval(15000).
  • No other plan changes. corpus_snapshots and sketch_plan_snapshots change only where a chain's width does not divide a reader's interval, and the PR lists each change.

Status and related work

Activity

  1. added
    bugSomething isn't working
    optimizerWorkload optimization and plan selection
    windowWindow functions, frames, panes, or temporal semantics
    on Nov 2, 2026
  2. zzylol commented on Nov 2, 2026

    @zzylol
    ContributorAuthor

    Fixed by #1178 (merged as 2c99451): every reader with a fixed interval must have an interval the chain's width divides before the chain can be kept at ingestion time.

    Follow-up found in review, not fixed by #1178 (it predates it): an EstimatedRate reader has no fixed interval and no phase, so the chain branch of ingestion_eligibility still continues for it. An ingestion-time chain therefore stays eligible for a reader whose evaluation times are unknown. The whole-window branch rejects the same reader (WindowNotFixed). Example: a predictable EstimatedRate reader of a 15 s chain with raw_data_retained: false evaluates at arbitrary times and reads windows that end up to 15 s early. Rejecting EstimatedRate readers in the chain branch, as the whole-window branch does, would close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingoptimizerWorkload optimization and plan selectionwindowWindow functions, frames, panes, or temporal semantics

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions