Repository navigation
Conversation
…ries identity (#1207 PR 3)
…ique key (#1207 PR 3) One rule for a top-k's ranked rows, exact or a heap's readout: one row per (partition, item) when the item identity is complete (ranked_rows_unique_keys, item_identity_is_complete). The exact top-k's group derived no key over per-series rows, so a heap readout candidate did not output the group's schema.
…e replaced operators' groups (#1207 PR 3)
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 6, 2026 23:55
6686f66 to
d4bde07
Compare
zzylol
changed the base branch from
refactor/1207-memo
to
refactor/1207-lineage
November 7, 2026 00:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#1207's Order, row "3. Pass 1": Pass 1 adds each group's initial candidates (exact, exact accumulator, summary with one candidate per θ, fused) to the memo, with every intermediate result its own group (#1203: one operator per candidate, no private nodes). It also re-creates the Pass 1 test coverage that PR 0b deleted, with tests that build their input
Memothrough theMemomethods, so no stage crate depends on another, even in tests.PR 0 left Pass 1 stopping at a private stub
logical_node_trees(memo): target discovery walked node trees, and candidates came from a whole-workload node-tree composition (compose_logical_candidate) sized per budget share. Neither has a place in the memo model (concepts doc §3: "Everything that was computed on operator-node trees is computed on groups").What
computation_targets(memo, supported_summaries)readsmemo.groups(): each group'sexpressionandgroup_properties. It finds oneComputationTargetper measure of eachAggregategroup with a realization besides plain exact execution, once however many queries read it. Consumers are the group'sparents. Thelogical_node_treesstub is deleted.Memo::find_or_add_groupandMemo::add_candidate(add_pass1_candidates):SummaryAgg[ExactAggregate(kind)](one candidate, no θ) and aFinalizeExactAccumulatorover it.avgfinalizes itssumandcountstates and divides them: a PromQLBinaryOp, or for SQL aJoingroup and aProject;SummaryAgg[family]per supported family, with one build candidate per θ inparameter_list(family), all with the group's own expression (θ only inLogicalProperties::parameters). A separate readout groupSummaryEstimate(state)is read by the result group;Sumbuild over the rows below the inner sum, added as a candidate of the outer sum's own accumulator state group, Logical memo candidate as (one operator over groups, logical properties θ) with a requirement per input; no private nodes #1203 "fused builds"). Their builds list the absorbed groups infused_groups;Aggregategroup per measure, each with its own candidates, joined back on the group keys by one candidate of the aggregate's group;HAVING: a key-only consumerFiltergroup gets the candidates that push its predicate below the aggregate (aFiltergroup over the input rows, which the states read). A valueHAVINGstays aFilterabove each readout.add_candidaterejects it otherwise), so a readout whose column names differ from the group's (quantileforquantile_0_9) is its own group, read by a renamingProjectcandidate. A heap top-k's Float64 estimate is cast to the exact top-k's column type (a SQL count ranks Int64).LogicalRequirementsof each consumer input (input_requirements): a top-k readout of a heap state needsheap_size ≥ k(the family's weakest listed width and depth withheap_size = k). Every other input has no floor. See "Decisions" for validity caps and the metric rule.Unsupported, e.g. a SQLcount(*)with no non-null column): the group keeps its other candidates. A candidate that breaks the node rules or the group's output is a bug and fails Pass 1 (PerComputationCandidateError::Candidate).pass1_memo(memo, supported_summaries)no longer takes the queries' accuracy targets, the accuracy model, evidence or data. They served onlywithout_summaries_no_query_can_use, which composed whole-workload node trees to prune families no query could use; that pruning is query-sensitive (§4: "query-sensitive pruning follows logical generation and reachability analysis") and moves to PR 4's pruning caps.ranked_rows_unique_keys,item_identity_is_complete,exact_topk_unique_keysinir/schema/topk.rs). Before, the exact top-k's group derived no unique key over per-series rows while the heap readout'sProjectcarried (job, series identity) fromtopk_schema_of_build, sotopk by (job) (10, sum_over_time(m[1m]))'s heap readout failedadd_candidate's exact check. This PR is the first whose strict check exercises it; PRs 2/2b are not rewritten.$promql_series_identitycolumn the planner realizes before Stage 0 (with_promql_series_identity), so its renamingProjectoutputs the group's exact (closed) schema. A group whose schema is open (identity not realized) gets no renamed readout: aProjectlists its columns and would drop the unlisted labels.compose_logical_candidate,compose,compose_roots,rewrite,substitute,substitute_measure,with_names_of,whole_expression,nested_sum,Sizing,ChosenComputation,Chosen,PerComputationCandidates(withcombinations,absorbed_choice_varies,without_unsupported_summaries,without_summaries_no_query_can_use),output_can_meet_target,sources_read,absorbed_targets,generate_per_computation_candidates,logical_node_trees,ComputationTarget::{choice_count, computation_for},PerComputationCandidateError::InvalidChoice, the Pass 1 featuretest-supportand itstest_supportmodule;asap_accuracy::test_support::{estimate_has_bound, estimate_bound_over_raw_data_samples, local_guarantee}(their only callers were Pass 1's tests, which now live inasap-accuracy); the executor test helpercomputation_candidates(no caller) and the executor's dev-dependencies on Stage 0 and Pass 1.Decisions and pending items
heap_size ≥ kof a top-k readout.fused_groupsfused_groupslists the groups of the operators it replaces. The whole-expression heap's builds list the absorbed sums' groups; the nested-sum merge's build lists the inner sum's group. Of the doc's rules, top-k and sum∘sum exist; min∘min and max∘max are not added here (doc TODO: revisit each rule's inputs and outputs).memo.groups()[id]: expression andgroup_properties.nested_sum_input,whole_expression_input,reads_only_time_series,measure_input,realize_keyed_additive_summary_input,summed_per_series_samples,ranked_valuetake(&Memo, MemoGroupId).MemoGroupAsInputstays types-internal. A privateraw_data_samples_of_metric(memo, group)walks the groups to the metric's scan (accuracy's node version and node-treebind_guaranteesare deleted in PR 4).SummaryCoverage::derive_from_inputsis alreadypub(crate)in PR 2b: no change.add_candidate's output check==), as PR 2 defines it. Qualifiers no longer exist (PR 2b);closedstays compared (see What, "Series identity").pass1_memo(memo, supported_summaries)accuracy_targets,accuracy,evidenceanddata: they served only the whole-workload pruning that PR 4's caps replace.PerComputationCandidateError::{Candidate, LogicalProperties};HavingEntry/ComputationTargetover groupsHAVINGpush-downFilter's predicate goes below in theFiltergroup's own candidates, whatever the aggregate's other consumers read (old Pass 1 pushed it only when every consumer applied it, #1089, because a candidate replaced the aggregate for all of them).planning::explicit_local_candidates,planning::explicit_summary_candidate,avg_as_sum_and_count)todo!("#1207 PR 6: ..."): realizing a plan from the memo is PR 6's realizer. Interim inside the merge unit.New abstractions not in #1207 or the doc (pending approval)
pub(crate)item_identity_is_complete,ranked_rows_unique_keys,exact_topk_unique_keys; privateRankedSource,ranked_source(the walkranked_columnalready did)types/src/ir/schema/topk.rsany_measure_filtered<C>(was over node predicates)types/src/ir/operator/non_asap.rsnon_asap(memo, group),schema(memo, group),raw_data_samples_of_metric(memo, group),ranked_column(memo, group)realization.rsranked_columnis types' rule over the operators a Stage 0 group holds (Project renaming, Filter, Aggregate).computation_targets,add_pass1_candidates,AggregateWays,SingleMeasure,add_aggregate_candidates,add_fused_candidate,measure_output,accumulator_state,finalized_with,accumulator_build,exact_accumulator_build,add_output_candidate,renaming_project,add_group,add_state_group,add_candidate,input_requirements,heap_floor,unless_unsupported,expression,without_unsupported_summaries(free fn)per_computation_candidates.rsmetric_schema,scan_operator,group,scan_group,aggregate_group(_with),query(test_fixtures.rs);memo_printout,label,candidates,groups_where,state_group,range,instant,table_group,compare,filter,divide,quantile,sum,count,topk,pushed_below,candidates_for,algorithms(Pass 1 tests);schema_of(typestopk.rstest);pass1,pass1_over,having_pushed_below(per_computation_candidates_sql.rs);pass1,root_group,aggregate_groups,candidates_of_aggregates,sketch_builds,avg_candidate_groups,pass1_sql(stage1_logical_asap_optimization.rs)How
computation_targetswalksmemo.groups(). For eachAggregateexpression it records the ownhavingand each parentFiltergroup whose input is the aggregate (key-only →BelowAggregate). It lists the ways per measure (computation_candidates_for_intent, unchanged), then detects fused computations over groups (add_whole_expression_targets,add_nested_sum_targets,close_absorptions), prunes a PromQL top-k's own heaps, and drops unsupported families.add_pass1_candidatesadds every Stage 0 group's own expression as a candidate, then, per aggregate group (and per key-onlyFiltergroup above it),add_aggregate_candidatesbuilds each way bottom-up:add_state_group(θ list or one exact build),add_groupfor each intermediate (the operator is also its group's candidate), andadd_output_candidatefor the top operator: added to the result group, or, when its column names differ, added as its own group and read throughrenaming_project.Project; an open group gets none.add_candidate(this crate) →LogicalProperties::new(θ in the list, none on exact operators) →input_requirements→Memo::add_candidate.Key code interfaces added / modified
stage1-pass1-per-computation-candidates/src/logical_memo_candidates.rsstage1-pass1-per-computation-candidates/src/per_computation_candidates.rstypes/src/stages/candidates.rstypes/src/ir/schema/topk.rs(schema derivation fix; crate-internal)types/src/stages/per_computation_logical_memo_candidates.rstypes/src/ir/operator/non_asap.rs,accuracy/src/test_support.rsUsage (from
running_example_gives_one_build_per_theta_and_separate_readouts, run at this head):Major algorithm changes
Pass 1 candidate generation.
todo!(). Before PR 0, it built one whole-workload node tree per choice and sized each sketch for an equal budget share; the result was copied into private node chains.compose_logical_candidate(choice)→ node tree per choice → sizing peraccuracy_budget_parts→ private chains. After: targets over groups → per way, groups bottom-up throughfind_or_add_group, one build candidate per listed θ, one operator per candidate.add_candidate); the ported frontend tests (stage1_logical_asap_optimization::{sql,promql}_candidates_output_the_query_columns) run every SQL and PromQL query of the old composition test through Pass 1.|parameter_list(family)|(4–70) build candidates to one shared state group, instead of one node tree per whole-workload combination.Before / After
The doc's running example (concepts doc §1): query A = p95 of
latency, query B = p99 / p50 of it (running_example_gives_one_build_per_theta_and_separate_readouts). Before (PR 2):pass1_memopanics attodo!("#1207 PR 3: discover targets over the memo groups, not node trees"). After, the memo (printed by the test;_=PerSubpopulationInstance):Mapping to the doc's table: G1 here is the doc's G1 (
latency, viaScanG0 andTimeRangeG1); G6 = doc G2 (KLL state, one build per θ of KLL's list; the doc'sk = 200/345are illustrative); G8 = doc G2′ (DDSketch); G2/G3/G4 = doc G3/G4/G5, each with its exact aggregate and a renamingProjectover a readout group per family (G7/G9, G10/G11, G12/G13); G5 = doc G6. The doc shows G4/G5 reading only G2′: that is the metric rule, which drops the KLL → ratio edges in PR 4 (decision 2).Verification
cargo check,cargo clippy --workspace --all-targets --all-features -- -D warningsandcargo fmt --all --checkare clean.Pass 1's tests (
stage1-pass1-per-computation-candidates/src/per_computation_candidates_tests.rs, 28 tests; plus 5 inrealization.rs,keyed_heap_input_refuses_a_filtered_countnow over a memo). Each builds its memo throughMemo::find_or_add_groupandMemo::add_query(test_fixtures.rs); no other stage crate is a dependency, even in tests. PR 0b's list (#1229), by category:running_example_gives_one_build_per_theta_and_separate_readoutsper_series_readouts_keep_the_series_identityquantile_over_timereadout'sProjectoutputs the per-series aggregate's exact closed schema, mapping the estimate's$promql_series_identitycolumn onto the group'san_open_per_series_aggregate_offers_no_renamed_readoutmulti_measure_aggregate_has_one_target_per_measureaggregate_without_realization_is_not_a_targetan_aggregate_read_by_several_roots_is_one_targetasap_input_and_invalid_accuracy_are_rejectedunsupported_summaries_are_not_offeredevery_group_offers_its_own_expressionexact_accumulator_is_a_state_group_and_a_finalizemulti_measure_candidate_joins_single_measure_groupsonly_a_topk_readout_has_a_parameter_floorheap_size ≥ kkey_only_having_is_pushed_below_into_the_filter_groupFiltergets candidates over filtered input rows; the aggregate keeps its ownvalue_having_stays_aboveglobal_aggregate_having_stays_aboveHAVINGstays aboveown_value_having_filters_each_readouthavingis aFilterover each readoutwhole_expression_topk_heaps_read_the_raw_samplesjob, withRawDataSamplesOfMetric, one build per θ,fused_groups= the absorbed sumscount_ranked_topk_counts_with_unit_weightranked_item_is_the_keys_outside_the_partitionnested_sum_merges_into_the_outer_state_groupfused_groups= the inner sumnested_sum_merge_needs_a_sum_of_sumsmaxgets no mergeupdate_weights_follow_the_measurecardinality_keeps_…,frequency_statistics_keep_universal_choices,topk_keeps_both_specialized_heap_choices,summary_candidates_are_unsized,approximate_count_lists_its_accumulator_and_summaries,a_count_offers_no_count_sketchSchema derivation fix (
types/src/ir/schema/topk.rs):exact_topk_and_heap_readout_derive_the_same_unique_keys— fortopk by (job) (10, sum_over_time(m[1m]))over realized series identity, the exact top-k's group and a heap's ranked readout derive the same non-empty unique key.Guarantee composition is no longer Pass 1's: candidates carry θ, and each build's ε follows from θ (PR 1's
parameter_list); composition to the query root iserror_propagation(PR 4) and constraint (K) (PR 8). The two Count-Min bound tests that lived in Pass 1 but testasap-accuracy(count_min_estimate_has_no_bound_over_an_open_requirement,count_min_estimate_over_raw_data_samples_is_gated_on_the_metric) moved toaccuracy/src/propagation.rs.Ported to the memo (
integration-tests):per_computation_candidates_sql.rs(the fourHAVINGtests, the three heap-update tests, over Pass 1's memo of SQL queries) andstage1_logical_asap_optimization.rs(itspromql()helper realizes series identity, as the planner does):sql_candidates_output_the_query_columnsandpromql_candidates_output_the_query_columns(were*_candidates_compose: every query of the old composition test goes through Pass 1 with every candidate's output checked byadd_candidate),topk_offers_both_heap_families(wastopk_estimates_carry_no_frequency_bound; the guarantee half moves to PR 8),topk_over_an_aggregate_ranks_its_labels,subquery_in_a_measure_filter_is_an_input_group_of_the_state(was…_is_substituted),filtered_sql_avg_is_null_without_matching_rows,promql_avg_divides_sum_and_count_vectors,sql_avg_keeps_its_explicit_output_name,sql_count_star_without_a_non_null_column_is_only_exact(was…_is_unsupported),sql_count_of_a_nullable_column_counts_its_values.Deleted:
devtools/src/candidate_labels.rsconnected_labels_describe_selected_statesandconnected_labels_name_actual_shared_state(they labelled composed node trees; tools are PR 9).Test status
cargo test -p <crate> --all-features --no-fail-fast, per crate, passed / failed / ignored. PR 2b = #1233 (measured at b5daf86; f2370de adds an assert only); PR 3 = this PR at d4bde07. No test fails inside Pass 1; planning stops at Pass 2's firsttodo!()(PR 4: "rule 1, tumbling layouts as candidates of the state group").asap-accuracywith its test)count_min_estimate_over_raw_data_samples_is_gated_on_the_metricstops at PR 4 (local_guaranteeof θ), as it did in Pass 1exact_topk_and_heap_readout_derive_the_same_unique_keys)asap_plannersubprocesses (PR 4 / PR 9); −2 tests deleted (composed-node labels)materialized_views)crate_graph(inasap-integration-tests) passes: no stage crate depends on another, in[dependencies]or[dev-dependencies](the executor's dev-dependencies on Stage 0 and Pass 1 are gone).Done means
reader,kept, MILProw,σ/Schedule,EvaluationSchedule,logical_node,child_groups,accuracy_budget_parts,operators_kept_in_definition,group_as_input,BuiltOrInput,qualifier): no hit in the code; in the body only the names of deleted items (logical_node_trees,accuracy_budget_parts,qualifierin PR 2b's summary) and SQL/PromQL data rows (row/rows).refactor/1207-lineagef2370de).