[zzylol]

Merge unit. This is PR 2 of #1207's merge unit (PRs 0–9). It is stacked on PR 1 (#1227, refactor/1207-properties), which is stacked on PR 0b (#1229) and PR 0 (#1226). The PRs merge into main together, after PR 9. This PR fills in the Memo methods and Stage 0 that PR 0 left as todo!(), and adds the tests of the group properties and summary coverage that PR 0 derives from input groups. Planning still panics at the first todo!() of Pass 1 (PR 3); see "Test status".

Why

The concepts doc (section 3) says that rules add groups only through Memo::find_or_add_group and candidates only through Memo::add_candidate. Rules match group expressions, never add the same expression/property pair twice, and never copy consumers to make a new producer visible. Pass 1 (PR 3) and Pass 2 (PR 4) are built on these two methods. PR 0 left them, Memo::add_query and Stage 0's memo_of_queries as todo!(). PR 0 also made GroupProperties (schema, result kind, summary coverage, sharing) derived from the input groups by the existing rules (derive_group_properties, OperatorInput, MemoGroupAsInput); this PR stores them.

Before PR 0, Stage 0 had its own node-level hash-consing (cse.rs). Its equality compared schema, result kind, guarantee and children, so a group lookup by expression would have been a second equality, and the two could disagree. #1207 asks for one equality, and (decided 2026-11-06) a group is its expression.

What

• Memo::find_or_add_group(expression) -> Result<MemoGroupId, SchemaDerivationError> hash-conses on the expression: the operator, its arguments and its input group ids.
  • It first derives the expression's GroupProperties with derive_group_properties (PR 0): the existing node rules (schema derivation, input validation, merge check) over the input groups' group_properties and expression, never a node tree. An expression that breaks them returns the error, and the memo is unchanged.
  • If an equal expression already has a group, it returns that group.
  • Otherwise it adds MemoGroup { expression, group_properties, .. } and lists the new group as a parent of each input group.
  • θ is not an argument of any operator, so builds of one family over one input that differ only in θ are one group.
• Memo::add_candidate(group, candidate) -> Result<CandidateForMemoGroupId, CandidateError> derives the candidate's properties the same way (CandidateError::Schema). It then requires the derived schema and result kind to equal the group's (CandidateError::OutputDiffersFromGroup). Both checks run before the dedup and before any change, so a rejected candidate leaves the memo unchanged.
  • A candidate with the same expression and the same logical properties as one already in the group returns that candidate's id and is dropped.
  • A new candidate is appended to MemoGroup::candidates. group becomes a parent of each group it reads, and the group's queries are added to those groups and every group below them.
• Memo::add_query(root) -> usize appends the root to query_roots and lists the query on every group the root reaches.
• Summary coverage tests. types/tests/memo_summary_coverage.rs checks PR 0's group-based coverage through find_or_add_group, on the worked example of asap-primitive-schema.md §4.2.2 (see "Test status").
• Design doc. docs/design_docs/proposals/asap-primitive-schema.md §4.2.2 describes the derivation over memo groups (SummaryCoverage::derive_from_inputs, stored in GroupProperties::summary_coverage), with the worked example G1–G10 and its three cases, and unlifted_operators: Vec<Operator<Option<MemoGroupId>>> (a scalar subquery names its group, so such definitions compare); §4.2.3 ("When it runs") and §6.6 follow. The node-tree notation coverage() in §5's examples and §6.1 is not rewritten here.
• Memo::candidate(id) is unchanged (PR 1).
• Stage 0 (asap-stage0-memo): memo_of_queries(roots) -> Result<Memo, SchemaDerivationError> visits each node of the queries' DAGs once, children first. It calls find_or_add_group on the node's operator over its children's groups, then add_query per root.
  • It returns the first node-rule error. The facade propagates it: run_planning with ?, devtools' run_stages_until as PlanningStagesError::Validation, and sketch_coverage counts that root as rejected (PR 0).
• No output-contract check exists in the stack. same_output_contract and output_contract_satisfied exist only on the closed Logical memo candidate as (one operator over groups, logical properties θ) with a requirement per input; no private nodes #1203 branch. add_candidate's output check replaces them, so PR 4 (Pass 2 rules) and PR 6 (realizer) must not port them.
• Two code comments that pointed at Stage 0's deleted cse module now point at Memo::find_or_add_group (schema_support.rs, canonicalize.rs).

Decisions and pending items

Item	Decision	Status
A group is its expression	MemoGroup { expression, group_properties, parents, queries, candidates }. Hash-consing compares and hashes expression directly: equal operator and arguments (typed PartialEq, confirmed by equal JSON, so signed zeros stay apart and NaN never matches) over the same input groups at the same positions.	approved 2026-11-06
Both methods return Result; one set of node rules for groups and candidates	derive_group_properties (PR 0): the existing rules over MemoGroupAsInputs. A candidate's schema and result kind must equal the group's (==).	approved 2026-11-06
GroupProperties, SummaryStateDefinition { input_group, unlifted_operators }, OperatorInput, MemoGroupAsInput, derive_group_properties, coverage only in the memo	Defined in PR 0, used here.	approved 2026-11-06
InputOrTemporaryBuild / TemporaryBuild (a build a rule constructs only while it derives, never stored in the memo); unlifted_operators: Vec<Operator<Option<MemoGroupId>>> (None = an operator's main input, Some(g) = a group a scalar subquery reads), so states whose definitions read subqueries merge; no node-tree merged_window_build wrapper (accuracy's merged-windows guarantee is todo!() until PR 4)	Decided in PR 0 (see #1226); the subquery test is here.	approved 2026-11-06
input_group: Option<MemoGroupId>, OperatorInput pub + Clone, derive_from_inputs over MemoGroupAsInput only	PR 0's pending items (see #1226).	pending
An expression whose copies may return different rows (unordered LIMIT, Dedup on some columns, row-position window functions; GroupProperties::shareable_when_equal)	It always gets a new group and is never returned for another expression: Stage 0's legality rule, kept.	pending
A shareable parent over such children	Two groups, since its input groups differ (DISTINCT over two unordered LIMIT 10s). Stage 0's old node equality compared children by value and made them one node. Both are correct; the new rule shares less.	pending
A duplicate candidate	Same expression and same logical properties: θ compared as an argument, window layout by ==. The duplicate's input_requirements, fused_groups and origin are not compared, and are dropped with it.	pending
What parents holds	The groups that read the group through their expression (find_or_add_group) or through one of their candidates (add_candidate), each listed once. fused_groups are not reads.	pending
What queries holds	Ascending. add_query lists the query on its root's groups and everything below them; add_candidate extends the candidate's group's queries to the groups the candidate reads and everything below them. "Below" follows expressions and candidates' inputs. fused_groups get no queries.	pending
A candidate must produce its group's exact schema and result kind	So Pass 1 (PR 3) must compose a readout candidate for a frontend aggregate group whose output equals the aggregate's (names included), or add the readout as its own group.	pending (consequence for PR 3)
Stage 0's error	memo_of_queries returns the first node-rule error (Result<Memo, SchemaDerivationError>). The facade validates the roots first, so it signals a frontend bug.	pending

New abstractions not in #1207 or the doc

Item	Where	Why it is needed	Status
private field Memo::groups_by_expression_hash: HashMap<u64, Vec<MemoGroupId>>	types/src/stages/candidates.rs	Lookup in O(1) expected per call. Groups are private and added only through find_or_add_group, so the index covers every group.	approved 2026-11-06
private Memo::groups with Memo::groups() -> &[MemoGroup] (PR 0)	same	Groups are added only through find_or_add_group.	approved 2026-11-06
CandidateError::{Schema(SchemaDerivationError), OutputDiffersFromGroup { group }} (PR 0)	same	The two ways add_candidate rejects a candidate; both are bugs in the rule that built it.	approved 2026-11-06
Memo::add_query(&mut self, root) -> usize (PR 0)	same	With groups private, Stage 0 cannot set queries; this adds a root and lists the query on what it reaches.	approved 2026-11-06
private module stages::hash_consing: same_expression, expression_hash, same_value	types/src/stages/hash_consing.rs	The one equality; groups and candidates both use it. It compares expressions over group ids, not nodes. Sharing legality is the stored GroupProperties::shareable_when_equal.	pending
private add_parent, same_expression_and_logical_properties, Memo::add_queries_below	types/src/stages/candidates.rs	Keep parents and queries up to date; find a duplicate candidate.	pending
private GroupOfNode, group_of	stage0-memo/src/lib.rs	Visit each node of the queries' DAGs once, children first.	pending
test helpers scan, kll_state, quantile_readout, unordered_limit, project_first_column, candidate, kll_k (memo_tests); node, scan, quantile_agg(_over), compare, memo_of, root, one_group, scalar_of_vector, summary_agg, unordered_limit, literal (Stage 0 tests); scan, job_is, a_plus_b_above_10, filter, range, kll, merge, kll_of_last_minute_where, coverage, kll_unlifted, jobs, last_minute (memo_summary_coverage.rs)	test modules	Fixtures. candidate builds LogicalProperties directly, because the doc example's k = 200 and k = 345 are not in today's KLL list.	pending

Deleted (earlier revisions): MemoGroup::logical_node, MemoGroup::child_groups, MemoGroup::expression(), Memo::indexed_group_count, and the previous revision of this PR's Memo::logical_node_trees, logical_node_tree and node_over_groups (rebuilt node trees; the rules now read groups through MemoGroupAsInput). The input groups are expression.children().

How

• Node rules. derive_group_properties (PR 0) maps each input group id to a MemoGroupAsInput { memo, group_id } and runs the generic rules on the expression: output_schema, validate_inputs, output_kind, is_shareable_when_equal, and SummaryCoverage::derive_from_inputs (an error rejects a SummaryMerge; any other operator without coverage gets None). An input is read through its stored group_properties and its expression; nothing is rebuilt.
• Lookup. find_or_add_group derives the properties, then hashes the operator's arguments (as JSON, input groups replaced by ()) together with the input group ids, and compares the groups in that bucket with same_expression. Only shareable expressions are looked up and indexed. On a miss it pushes the group with its group_properties, and adds it to its inputs' parents.
• Candidates. add_candidate runs the node rules on the candidate and compares the output with the group's before touching the memo. It then scans the group's own candidates for a duplicate; a group has few candidates. On a miss it appends the candidate, calls add_parent per input group, and walks downward from the inputs with a worklist, inserting the group's queries in sorted position. The walk does not descend from a group that already had every query, because the groups below it have them too.
• Stage 0. group_of maps each original Rc<OperatorNode> by address to its group, so a diamond is visited once. It interns the children first, then calls find_or_add_group on the node's operator over their groups. add_query then adds each root in workload order, so every group's query list comes out ascending.

Major algorithm changes

Stage 0 hash-consing.

• Problem. There were two equalities: Stage 0's node equality (cse::same_node: schema, result kind, guarantee, own fields, children by pointer and then by value) and the group lookup the rules need. A Pass 1 rule adding SummaryAgg[KLL](G1) could fail to find Stage 0's equal group whenever the two disagreed, for example on the guarantee.
• Before.
  1. share_common_sub_dags interns nodes bottom-up into an Rc table (SipHash of the node's JSON, then same_node).
  2. number assigns group ids.
  3. pair maps the original nodes to the groups.
• After.
  1. group_of calls Memo::find_or_add_group(operator over child groups) bottom-up.
  2. add_query is called per root.
• Correctness. Two expressions are one group only if their arguments are equal and their input groups are the same groups, so equal groups compute equal values by induction. The legality gate is unchanged. A group's properties come from the same rules that check frontend nodes, and a candidate must reproduce its group's schema and result kind. Tests: the 13 Stage 0 tests, 10 Memo tests and 7 coverage tests below.
• Complexity. Hashing and comparing an expression is O(its arguments), since its children are u32 ids. The rules read only the input groups' stored properties and, where a rule looks below an input (a LIMIT over a Sort, a merge of a chain, the coverage walk through Filter/TimeRange/TimeShift/Project), their expressions: O(arguments + the few groups such a rule walks) per call, O(N) for Stage 0 over N nodes. Coverage is derived once per group and a merge reads its inputs' stored coverage.
• Limits. A shareable parent over unshareable children is no longer shared (see the decisions table).

Key code interfaces

Bodies are elided as ;. The signatures are PR 0's; this PR fills in their bodies.

crates/types/src/stages/candidates.rs

/// The memo, Stage 0's and Stage 1's one type (stays).
#[derive(Debug, Clone, Default)]
pub struct Memo {
    /// The groups, by id; added only through `find_or_add_group`. // private (PR 0)
    groups: Vec<MemoGroup>,
    /// Each query's root over groups, by index in the workload. // unchanged
    pub query_roots: Vec<QueryRoot<MemoGroupId>>,
    /// The shareable groups by the hash of their expression. // new (private)
    groups_by_expression_hash: HashMap<u64, Vec<MemoGroupId>>,
}

impl Memo {
    /// The groups, by id. // PR 0
    pub fn groups(&self) -> &[MemoGroup];
    /// The group of `expression` (operator, arguments, input groups), added if new. // body filled in
    pub fn find_or_add_group(&mut self, expression: Operator<MemoGroupId>)
        -> Result<MemoGroupId, SchemaDerivationError>;
    /// Adds a query with root `root`; returns its index. // body filled in
    pub fn add_query(&mut self, root: QueryRoot<MemoGroupId>) -> usize;
    /// Adds `candidate` to `group`, or returns the duplicate's id; checks it by the node rules. // body filled in
    pub fn add_candidate(&mut self, group: MemoGroupId, candidate: CandidateForMemoGroup)
        -> Result<CandidateForMemoGroupId, CandidateError>;
    /// The candidate `id`. // unchanged (PR 1)
    pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup;
}

/// A candidate `add_candidate` rejects; the memo is unchanged (PR 0).
pub enum CandidateError {
    /// Its expression fails the node rules: a bug in the rule that built it. // PR 0
    Schema(SchemaDerivationError),
    /// Its output schema or kind differs from the group's: a bug in the rule that built it. // PR 0
    OutputDiffersFromGroup { group: MemoGroupId },
}

crates/types/src/stages/memo.rs

/// One memo group (PR 0; only field docs changed here).
pub struct MemoGroup {
    /// What it computes: one operator, with its arguments; each id in it is an input group. // unchanged (PR 0)
    pub expression: Operator<MemoGroupId>,
    /// Derived once when the group is created, from its input groups. // unchanged (PR 0), now stored
    pub group_properties: GroupProperties,
    /// The groups that read it through their expression or a candidate's. // unchanged (now maintained)
    pub parents: Vec<MemoGroupId>,
    /// The queries whose roots reach it, ascending. // unchanged (now maintained)
    pub queries: Vec<usize>,
    /// Its candidates: empty after Stage 0. // unchanged
    pub candidates: Vec<CandidateForMemoGroup>,
}

crates/stage0-memo/src/lib.rs

/// Stage 0: the memo groups of the queries' roots, without candidates. // body filled in (signature PR 0)
pub fn memo_of_queries(roots: &[QueryRoot]) -> Result<Memo, SchemaDerivationError>;

Usage

Concepts doc, section 1, G1 and G2. Planning past Stage 0 still stops at PR 3's todo!(), so this flow is the one memo_tests::kll_builds_with_k_200_and_k_345_share_one_group, add_candidate_maintains_parents_and_queries and Stage 0's tests run. kll_build and readout_candidate stand for the tests' candidate(operator, θ) helper.

// Stage 0 over the frontends' DAG of query A (p95 of latency[1m]) gives G1 = Scan latency[1m] and A's root.
let mut memo: Memo = memo_of_queries(&roots)?; // every group without candidates; G1 lists query 0
// Pass 1 (PR 3) adds the KLL state group over G1: θ is not in the expression.
let g2 = memo.find_or_add_group(kll_state(g1))?; // SummaryAgg[KLL](G1) → G2, schema derived; G2 becomes a parent of G1
// One build candidate per listed θ, here the doc's k = 200 and k = 345.
let k200 = memo.add_candidate(g2, kll_build(g1, /* k */ 200))?; // { group: G2, index_in_group: 0 }
let same = memo.find_or_add_group(kll_state(g1))?; // == g2: one group for both sizes
let k345 = memo.add_candidate(same, kll_build(g1, /* k */ 345))?; // { group: G2, index_in_group: 1 }
memo.add_candidate(g2, kll_build(g1, 200))?; // == k200: duplicate, not added
// A readout offered to the state group does not compute G2's result: rejected, memo unchanged.
assert!(matches!(memo.add_candidate(g2, readout_candidate(g2, 0.5)), Err(CandidateError::OutputDiffersFromGroup { .. })));
// The readout group G3 = SummaryEstimate[q=0.95](G2) is query A's result in the tests.
let g3 = memo.find_or_add_group(quantile_readout(g2, 0.95))?; // G3 becomes a parent of G2
memo.add_query(QueryRoot::Operator(g3)); // G3, G2 and G1 list query 0

Before / After

KLL example (doc section 1): G1 = Scan latency[1m], G2 = SummaryAgg[KLL](G1) with k = 200 and k = 345.

	Before (PR 1)	After (this PR)
Stage 0	memo_of_queries panics (todo!)	groups G1, G3, … built through find_or_add_group; queries listed by add_query
Adding G2	find_or_add_group panics	G2 is added once with its group_properties (state schema, coverage); the second call returns G2
k = 200 and k = 345	add_candidate panics	two candidates of G2: index_in_group 0 and 1
k = 200 added twice	panics	the first id is returned; G2 keeps 2 candidates
A readout offered to G2	panics	Err(OutputDiffersFromGroup { group: G2 }), memo unchanged
G2's parents/queries after A's readout reads it	not maintained	parents [G3], queries [A] (and G1 queries [A])
Equality	Stage 0's node-level cse::same_node (deleted in PR 0)	find_or_add_group's expression equality, the only one

Test status

New tests, each stating the behavior it checks in its doc line:

• asap-types, stages::candidates::memo_tests (10):
  • find_or_add_group_returns_the_group_of_an_equal_expression
  • find_or_add_group_separates_equal_operators_over_different_inputs
  • find_or_add_group_never_returns_an_unshareable_group
  • find_or_add_group_lists_the_new_group_among_its_inputs_parents
  • find_or_add_group_returns_an_error_for_an_expression_with_no_schema (and adds nothing)
  • kll_builds_with_k_200_and_k_345_share_one_group
  • add_candidate_returns_the_existing_candidate_for_a_duplicate
  • add_candidate_maintains_parents_and_queries (also add_query)
  • add_candidate_rejects_a_candidate_whose_output_differs_from_its_group (memo unchanged)
  • add_candidate_rejects_a_candidate_whose_inputs_fail_validation: a readout of raw rows, whose schema derives and which only validate_inputs rejects; memo unchanged.
• asap-stage0-memo (13):
  • two_queries_sharing_a_sub_expression_get_one_group (also checks parents and queries)
  • a_query_s_repeated_sub_expression_is_one_group
  • an_already_shared_node_is_one_group
  • aggregates_that_differ_in_an_argument_are_different_groups
  • equal_keyless_nodes_that_return_the_same_rows_are_one_group
  • differing_accuracy_targets_are_two_groups_over_one_scan_group
  • a_scalar_referenced_vector_is_one_group_across_queries
  • readouts_share_their_state_group_but_not_each_other
  • dedup_is_one_group_only_on_every_column
  • unordered_limits_and_their_parents_are_different_groups
  • limits_over_a_sort_with_their_partitions_are_one_group
  • signed_zero_and_non_finite_literals_are_different_groups
  • memo_of_queries_returns_the_error_of_a_node_that_breaks_the_node_rules
• asap-types, types/tests/memo_summary_coverage.rs (8), on asap-primitive-schema.md §4.2.2's memo (G1 Scan latency; G4/G7/G10 KLL over the last minute of job = "api", job = "web", job = "api" ∧ a + b > 10):
  • group_properties_come_from_input_groups: scan schema, state kind and column, readout value column, shareable_when_equal (false for an unordered LIMIT), coverage only on the state.
  • a_summary_agg_over_a_job_filter_lifts_it_into_the_selection: G4 has input_group: Some(G1), unlifted_operators: [SummaryAgg[KLL]], selection job ∈ {api}, time (−60 s, 0].
  • states_with_equal_definitions_and_disjoint_selections_merge: G4 and G7 have one definition and definition_id; SummaryMerge(G4, G7) is a group covering job ∈ {api, web}.
  • states_with_overlapping_selections_do_not_merge: SummaryMerge(G4, G4) is InputsOverlap { Disjoint }; no group is added.
  • a_filter_that_is_no_selection_stays_unlifted: G10 keeps Filter(a + b > 10) in unlifted_operators; SummaryMerge(G4, G10) is DefinitionMismatch.
  • states_whose_definitions_read_the_same_subquery_merge: two states over Filter(job = … ∧ a > (SELECT threshold)) keep Filter(a > subquery(Some(threshold)), None) in unlifted_operators; their definitions are equal, their selections (job ∈ {api}, job ∈ {web}) disjoint, and SummaryMerge of them covers job ∈ {api, web}.
  • scan_predicates_lift_and_their_states_merge: PromQL matchers in the Scan (latency{job="api"}, latency{job="web"}) are lifted (input_group: None, the bare Scan last); their states merge, as before Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207.
  • a_tumbling_window_chain_covers_its_whole_window: a 5 × 1 min chain over G1 has the coverage of KLL(TimeRange[5m](G1)).
• PR 1's delivers_on_follows_the_alignment_table builds its memo through find_or_add_group and add_candidate since groups became private. It fails on PR 1 and passes here.

None of these tests depends on another stage crate; asap-stage0-memo has no dev-dependencies.

cargo test -p <crate> --all-features --no-fail-fast at the head (e710906). Every failure panics at a later PR's todo!() (or is the stale fixture, PR 9). Planning now passes Stage 0 and stops at Pass 1's target discovery (PR 3).

Crate	Passed	Failed	Ignored	Fails at
asap-types	192	0	0	—
asap-accuracy	75	0	0	—
asap-cost	31	1	0	check θ, which the exported node carries (PR 6): 1
asap-frontend-common	49	0	0	—
asap-sql-function-catalog	10	0	0	—
metricsql_common	3	0	0	—
asap-frontend-promql	218	0	0	—
asap-frontend-metricsql	15	0	0	—
asap-frontend-sql	174	0	0	—
asap-stage0-memo	13	0	0	—
asap-stage1-pass1-per-computation-candidates	12	1	0	the guarantee of θ from the candidate's LogicalProperties::parameters (PR 4): 1
asap-stage1-pass2-reuse-candidates	0	0	0	—
asap-stage2-physical-optimization	0	0	0	—
asap-stage3-plan-selection	3	0	0	—
asap-executor	210	63	1	check θ, which the exported node carries (PR 6): 32; the Hydra grid's sizes are θ, which the exported node carries (PR 6): 13; the view's coverage from its memo group's GroupProperties::summary_coverage (PR 6): 8; the weighted-frequency θ, which the exported node carries (PR 6): 7; check the weighted-frequency θ, which the exported node carries (PR 6): 2; discover targets over the memo groups, not node trees (PR 3): 1
asap-planner	9	19	1	discover targets over the memo groups, not node trees (PR 3): 19
asap-devtools	38	58	3	discover targets over the memo groups, not node trees (PR 3): 50; a group's node from its expression (PR 9): 5; an asap_planner subprocess that stops at a todo!(): 2; label a group by its expression (PR 9): 1
asap-integration-tests	84	87	4	discover targets over the memo groups, not node trees (PR 3): 83; the view's coverage from its memo group's GroupProperties::summary_coverage (PR 6): 3; the stale fixture (old wire format; PR 9): 1
Total	1136	229	9

Checks: cargo check --workspace --all-targets, cargo clippy --workspace --all-targets -D warnings and cargo fmt --all --check are clean.