The planner does not construct candidate group-by labels that are finer than what a query asks for #914

Activity

1. milindsrivastava1997 commented on Oct 26, 2026

   

   milindsrivastava1997

   on Oct 26, 2026

   ContributorAuthor

   More actions

   Plan

   The planner lives in ProjectASAP/sketch-bench, crate rqe-optimizer/, not in this repo. A worktree for this work already exists in the sketch-bench checkout at .agent/worktrees/planner/union-groupings (branch planner/union-groupings, from main, no changes yet). Functions are cited by name because line numbers drift.

   Background: how candidates are built today

   • rqe-optimizer/src/candidates.rs, the function that calls candidate_deployments: it groups RAQEs by (capability, metric, spatial_filter, grouping_labels). Each query grouping G gets two kinds of candidates:

     • exact-grouping families, which serve only RAQEs grouped by G;
     • roll-up families (family_properties(..).mergeable_across_groups), which serve every RAQE grouped by a subset of G, using subset_group.

     Hydra gets one grid over the metric's schema. These are the only groupings ever generated.

   • Time already has the analogue: subset_gcds in candidates.rs generates the gcd of every subset of slides in one pass. Labels have nothing like it.

   • Cost needs card(G) for the deployment's grouping: group_count in src/analytical_cost_model.rs indexes facts[metric].cardinality[grouping], and it panics if the entry is missing. Callers fill MetricFacts.cardinality (src/lib.rs) only for query groupings and the full label set.

   • Accuracy for a roll-up (src/saturation.rs) reads data_shape at the RAQE's grouping and the fan-out card(G_d) / card(G_r). A unioned grouping therefore needs a cardinality but no data_shape.

   Design decisions (agreed)

   1. Where card(union) comes from: the caller supplies it. A union is a candidate only if facts.cardinality has an entry for it, otherwise it is skipped. The planner doesn't estimate it.
   2. Which unions: every subset union of the query groupings for each (capability, metric, filter), generated in one pass the same way subset_gcds generates slides. There are at most 2^|labels| per metric.
   3. Per-series grouping: the metric's full label set (MetricFacts.labels, card = series count) is always a candidate, even when no query groups by it.
   4. What to re-run: the case-study sweeps (sketch-bench PR test: E2E throughput/latency tests + sliding window bug fix #229) and the autosketch-vs-asap results.

   Proposed, not yet approved

   • Planner (src/candidates.rs)
     • Add subset_unions(groupings).
     • The candidate groupings become every query grouping, every subset union of them, and the full label set, minus any union without a cardinality.
     • A grouping that no query uses exactly gets only the roll-up families (mergeable_across_groups && !answers_any_subgrouping). An exact-grouping family at a grouping nobody asks for serves nothing. Hydra is unchanged.
   • Validation (validate_facts in src/lib.rs): today the check that dropping labels never adds groups covers only RAQE groupings plus the full label set. Extend it to every label set in cardinality, since unions can now be read.
   • Tests, written first, in candidates.rs:
     • the six anomaly groupings yield the (region, service, endpoint, status) union candidate;
     • a union without a cardinality is skipped;
     • the full label set is a candidate;
     • a unioned grouping gets no exact-only family.
   • Runner (rqe-optimizer/examples/autosketch_vs_asap.rs, where it builds MetricFacts): for schema metrics, fill cardinality for every subset of the schema's labels.
     • Use the generator's grouping_cardinality rule. Labels are nested, e.g. endpoint is per service, so (service, endpoint) has 625 groups and (region, endpoint) has 4 × 625. A plain product of label cardinalities is wrong.
     • Cap each subset's cardinality at the stream's series count, as the runner already does for the full schema.
     • Without this, the synthetic tables supply no union cardinalities and nothing changes.
   • Results
     • Stack the branch on eval/case-study-sweeps (PR test: E2E throughput/latency tests + sliding window bug fix #229) and re-run its 38 tables. The README's Reproduce section has the command.
     • Re-run the autosketch-vs-asap results.
     • Update rqe-optimizer/results/case-studies/README.md. Finding 4 should change: the base anomaly workload should drop to 1 MV, and the drill-down query should no longer make the plan cheaper. Report the endpoints-per-service value where the plan splits off a (service, region) MV. Today it splits between 50 and 100, with the drill-down present; the paper's §6.3.1 text quotes it.

   Expected outcome to check

   For the base anomaly workload (4 regions, 25 services, 25 endpoints per service, 4 statuses, i.e. 10,000 fine groups; 2e6 samples/s), we expect 1 MV at (region, service, endpoint, status), cheaper than the 0.052 vCPU the drill-down variant gets today. If not, find out why before re-running everything else.

   Risks

   • The per-series candidate enlarges the MILP: 2e6 series for anomaly, 1e4+ for perf. It should lose on ingest but costs planning time. If planning time jumps noticeably, check with the author before capping.
   • Other callers (e2e harness, examples/workload_scenarios.rs) gain only the per-series candidate until they supply union cardinalities.
2. milindsrivastava1997

   changed the title [-]Planner never considers a grouping that no query asks for[/-] [+]The planner does not construct candidate group-by labels that are finer than what a query asks for[/+] on Oct 26, 2026