[zzylol]

Part of #250. Golden fixtures: ProjectASAP/sketchlib-golden-bytes#8 (merge first; the submodule pointer then moves to the merged commit).

Why

CountSketchWithL2Norm keeps one L2 accumulator per row, and the UnivMon family's layers are this sketch. merge already recomputes each accumulator as the row's sum of squared counters. Still, the ASAPv1 payloads carried it (§3.19, §3.15, §3.16 and the Hydra UnivMon cells). The wire spec said it could not be derived for two reasons:

• update_weighted_with_hash_without_l2 moved counters without touching the accumulator. Only a property test called it.
• A saturated accumulator stepped down from i64::MAX on a decrement, so after saturation it no longer matched the counters.

With both gone, l2 is a cache of the counters. Per Q-DERIVED, a value the decoder can recompute does not ride the wire.

What

• The in-memory invariant. Each row keeps an exact i128 sum of its counters' squares, with each square capped at 2^63. Every write removes the old counter's term and adds the new one's, exactly and in O(1), including past saturation. The value is clamped to i64::MAX wherever it is read. A single capped term already reaches i64::MAX, so the clamped value is min(sum(counts[row]^2), i64::MAX). Each row's accumulator grows from 8 to 16 bytes, and memory_bytes / estimated_memory_bytes count that.
• One estimate change. Before this PR, a row whose sum of squares saturated and then decreased reported an understated l2, because the old accumulator stepped down from i64::MAX. It now reports the true value. Only in that case do l2, l2_squared, UnivMon's estimate_l2 and its terminal-mode heavy-hitter threshold change.
• Removed public API. Two methods could move counters without the accumulator:

Item	Readers before this PR
CountSketchWithL2Norm::update_weighted_with_hash_without_l2	one property test (tests/pbt/topk_wrappers.rs), now on update_weighted_with_hash
CountSketchWithL2Norm::as_storage_mut	none

• ASAPv1, per Q-VER. Each changed payload gets a new kind id, the next free variant in its family. The retired ids still decode: l2 is read, dropped and recomputed. They are never encoded.

Retired id (decode only)	New id	New payload
0x19 0x00 CountSketchWithL2Norm [counts, l2]	0x19 0x01	[counts]
0x10 0x00 UnivMon (8 fields, l2 at position 1)	0x10 0x01	[counts, heap_lens, keys, heap_counts, candidate_complete, bucket_size, update_mode]
0x11 0x00 two-tier UnivMon	0x11 0x01	the same 7 fields
0x07 0x04 Hydra, UnivMon counter	0x07 0x05	[cells], each cell the 7-field pyramid

Hydra's variants 0x07 0x00-0x07 0x04 are all taken, and the variant byte is Hydra's only version marker. So the new UnivMon-counter payload takes the next free variant, 0x07 0x05, and the reserved range for further base sketches becomes 0x07 0x06-0x07 0x7f.

• Nesting. EhBucketSketch, and through it ExponentialHistogram buckets, accepts a nested 0x19 0x00 or 0x10 0x00 and decodes it into the same variant. On re-encode the bucket gets 0x19 0x01 or 0x10 0x01. This mirrors what the DDSketch PR does for 0x05 0x00. No existing EH or EhBucketSketch golden fixture nests these kinds, so no nested fixture changes.
• Encoder check. Per Q-DERIVED, the encoder rejects a cached accumulator that differs from the capped-square sum the decoder recomputes. The error names the first row that disagrees, with both values.
• Retired l2 arrays. They are read element by element as i64 and not kept (RetiredL2), so retired payloads allocate nothing for l2, yet an l2 that is not an integer array is still rejected.
• Golden fixtures. The submodule moves to sketchlib-golden-bytes branch csl2-derived-l2 (baffc4b), which adds six *_no_l2 fixtures. Each one is its retired twin decoded and re-encoded. The branch also scopes the README's CountL2HH integer-width note to the retired fixture.
• Docs. docs/asapv1_wire_format.md: the registry, the Hydra mapping note, the §3 "derivable" rule, §3.9, §3.15, §3.16, the §3.18 nested-id table and rule 2, §3.19, the matrix-row bound paragraph, the integer-array list, Q-DERIVED, Q-HYDRA-IDS and Q-UNIVMON-SCOPE. Also the API pages (Count Sketch, CountSketchWithHeap, EhBucketSketch, Hydra, UnivMon, two-tier UnivMon), docs/message_pack_format.md, docs/tests.md, tests/PBT_COVERAGE.md and the CHANGELOG.
• Unchanged code. UnivMon's estimators, estimate_l1, Octo UnivMon and UnivMon-Q. Their results change only through the saturated-then-decreased case above.

How

• src/sketches/countsketch_l2_norm.rs:
  • row_l2_squared is now a Vector1D<i128>.
  • New private helpers: capped_square(c) = min(c^2, 2^63), row_capped_square_sum(row), and clamp_to_i64, which reads an accumulator clamped to [0, i64::MAX].
  • merge recomputes with row_capped_square_sum.
  • write_counter, the single write path for update_weighted_with_hash and apply_delta, does acc += capped_square(new) - capped_square(old).
  • l2_squared reads clamp_to_i64.
  • The two methods above are deleted.
• src/sketches/countsketch_l2_norm/wire.rs:
  • layer_state returns counts only and checks each row's accumulator against row_capped_square_sum.
  • rebuild_layer(rows, cols, seed_index, counts) recomputes the accumulators.
  • from_asapv1 picks the payload struct by kind id. CountSketchWithL2NormPayloadCarryingL2 reads the retired layout, with l2: RetiredL2.
  • RetiredL2 is a crate-private newtype. Its Deserialize visits the array as i64s and keeps none of them. Its Serialize writes held values; only tests use it, to craft retired bytes.
• src/sketch_framework/univmon/wire.rs:
  • PyramidPayload and the internal PyramidState / DecodedPyramid lose l2.
  • PyramidPayloadCarryingL2<K> (with l2: RetiredL2), its From into PyramidPayload<K>, and decode_pyramid_payload(payload, carries_l2) read the retired layout.
  • decode_pyramid gains a carries_l2 argument.
  • The decode-door check rows * layers == len(l2) is gone. Allocation stays bounded: check_layers caps layers at UNIVMON_MAX_LAYERS before the geometry vector is built, and rebuild_layers measures the total cell count against len(counts) before any matrix is built. The spec's §3.15 rule 5 and §3.16 rule 4 say so.
• src/sketch_framework/univmon_two_tier/wire.rs: the same switch on kind id. accumulator_count, whose only caller was the removed l2 length check, is deleted.
• src/sketch_framework/hydra/wire.rs:
  • decode_univmon_carrying_l2 reads 0x07 0x04.
  • unpack_univmon_cells(.., carries_l2) reads HydraUnivMonPayloadCarryingL2<K> and rewrites each cell in the current shape before handing it to UnivMon::from_asapv1.
• src/sketch_framework/eh_bucket_sketch/wire.rs: variant_name and rebuild_variant accept the two retired nested ids.
• src/asapv1/kind.rs:
  • COUNT_SKETCH_WITH_L2_NORM, UNIVMON, UNIVMON_TWO_TIER and HYDRA_UNIVMON now name the new ids.
  • The new *_CARRYING_L2 constants name the retired ones.
• New fixtures. A throwaway integration test decoded each retired fixture with this branch and wrote the re-encoding. The file is not committed. retired_l2_fixtures_decode_to_their_no_l2_twins checks every pair.

Key code interfaces added / modified

src/sketches/countsketch_l2_norm.rs

pub struct CountSketchWithL2Norm<H> {
    counts: Vector2D<i64>,          // unchanged
    row_l2_squared: Vector1D<i128>, // was Vector1D<i64>: exact sum of squares, each capped at 2^63
    ..
}
fn capped_square(counter: i64) -> i128;            // new, private: min(counter^2, 2^63)
fn row_capped_square_sum(row: &[i64]) -> i128;     // new, private: what a row's accumulator holds
fn clamp_to_i64(accumulator: i128) -> i64;         // new, private: the accumulator as read

impl<H: SketchHasher> CountSketchWithL2Norm<H> {
    pub fn update_weighted_with_hash(&mut self, hashed_val: u128, weight: i64); // unchanged signature; O(1) per row in every state
    pub fn apply_delta(&mut self, delta: CountSketchCellDelta);                 // same
    pub fn l2_squared(&self) -> f64;  // reads each row clamped at i64::MAX; true after a saturated row decreases
    pub fn memory_bytes(&self) -> usize;                                // counts 16 bytes per row accumulator (was 8)
    pub fn estimated_memory_bytes(rows: usize, cols: usize) -> usize;   // same
    pub fn merge(&mut self, other: &Self);                                      // unchanged behaviour
    // pub fn update_weighted_with_hash_without_l2(&mut self, hashed_val: u128, weight: i64); // removed
    // pub fn as_storage_mut(&mut self) -> &mut Vector2D<i64>;                                // removed
}

src/sketches/countsketch_l2_norm/wire.rs

impl<H: SketchHasher + HashProfile> CountSketchWithL2Norm<H> {
    /// Encodes kind 0x19 0x01, payload [counts]. Fails on a stale l2 cache.
    pub fn to_asapv1(&self) -> Result<Vec<u8>, RmpEncodeError>; // was 0x19 0x00, [counts, l2]
    /// Decodes 0x19 0x01, or the retired 0x19 0x00 with l2 dropped; l2 recomputed.
    pub fn from_asapv1(bytes: &[u8]) -> Result<Self, RmpDecodeError>;
}
pub(crate) fn layer_state<H>(sketch: &CountSketchWithL2Norm<H>) -> Result<&[i64], RmpEncodeError>; // was (&[i64], &[i64])
pub(crate) fn rebuild_layer<H>(rows: usize, cols: usize, seed_index: usize, counts: &[i64])
    -> Result<CountSketchWithL2Norm<H>, RmpDecodeError>; // lost its `l2: &[i64]` argument

src/sketch_framework/univmon/wire.rs (crate-private payload types)

pub(crate) struct PyramidPayload<K> { counts, heap_lens, keys, heap_counts, candidate_complete, bucket_size, update_mode } // lost `l2`
pub(crate) struct PyramidPayloadCarryingL2<K> { counts, l2: RetiredL2, heap_lens, keys, .. } // new: the retired layout
pub(crate) struct RetiredL2(pub(crate) Vec<i64>); // new (countsketch_l2_norm::wire): decodes an i64 array, keeps nothing
impl<K> From<PyramidPayloadCarryingL2<K>> for PyramidPayload<K>;                     // new: drops l2
pub(crate) fn decode_pyramid(key_type: &str, payload: &[u8], carries_l2: bool) -> Result<DecodedPyramid, RmpDecodeError>; // gained carries_l2

UnivMon, UnivMonTwoTier, Hydra and EhBucketSketch keep their public to_asapv1 / from_asapv1 signatures. Only the ids they write and accept change, as in the table above.

src/asapv1/kind.rs (crate-private)

pub(crate) const HYDRA_UNIVMON: &[u8] = &[0x07, 0x05];                         // was [0x07, 0x04]
pub(crate) const HYDRA_UNIVMON_CARRYING_L2: &[u8] = &[0x07, 0x04];             // new, retired
pub(crate) const UNIVMON: &[u8] = &[0x10, 0x01];                               // was [0x10, 0x00]
pub(crate) const UNIVMON_CARRYING_L2: &[u8] = &[0x10, 0x00];                   // new, retired
pub(crate) const UNIVMON_TWO_TIER: &[u8] = &[0x11, 0x01];                      // was [0x11, 0x00]
pub(crate) const UNIVMON_TWO_TIER_CARRYING_L2: &[u8] = &[0x11, 0x00];          // new, retired
pub(crate) const COUNT_SKETCH_WITH_L2_NORM: &[u8] = &[0x19, 0x01];             // was [0x19, 0x00]
pub(crate) const COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2: &[u8] = &[0x19, 0x00]; // new, retired

Usage

A producer ships a CountSketchWithL2Norm and a UnivMon to a receiver, and the receiver reports the same L2. Compiled and run as an example at eae2400:

use asap_sketchlib::{CountSketchWithL2Norm, DataInput, UnivMon};

// A producer's CountSketchWithL2Norm: the per-row L2 accumulators are kept in memory.
let mut sketch: CountSketchWithL2Norm = CountSketchWithL2Norm::with_dimensions(3, 64);
for (key, weight) in [("alpha", 9_i64), ("beta", 5), ("gamma", -4)] {
    sketch.update_weighted(&DataInput::Str(key), weight);
}

// The ASAPv1 payload is [counts] under kind 0x19 0x01; no l2 rides the wire.
let bytes = sketch.to_asapv1().expect("encode");
assert_eq!(&bytes[8..10], &[0x19, 0x01]);

// The receiver recomputes each row's sum of squares and reports the same L2.
let received: CountSketchWithL2Norm = CountSketchWithL2Norm::from_asapv1(&bytes).expect("decode");
assert_eq!(received.l2(), sketch.l2());

// UnivMon pyramids encode as 0x10 0x01 and decode to the same estimates.
let mut um = UnivMon::new(4, 2, 16, 3);
for key in 0..100_u64 {
    um.update_weighted(&DataInput::U64(key % 7), 1);
}
let um_bytes = um.to_asapv1().expect("encode");
assert_eq!(&um_bytes[8..10], &[0x10, 0x01]);
let um_back = UnivMon::from_asapv1(&um_bytes).expect("decode");
assert_eq!(um_back.estimate_l2(), um.estimate_l2());
// Printed: l2 sent 11.045361017187261 received 11.045361017187261
//          UnivMon l2 estimate 40.66939881532551 after the round trip

Bytes written before this PR (0x19 0x00, 0x10 0x00, 0x11 0x00, 0x07 0x04) still decode with the same calls. retired_l2_fixtures_decode_to_their_no_l2_twins checks this on the golden fixtures.

Before this PR

• Every CountSketchWithL2Norm, UnivMon, two-tier UnivMon and Hydra-over-UnivMon payload carried one i64 per row per layer for l2.
• A saturated row's accumulator could fall below its true sum after a decrement.
• update_weighted_with_hash_without_l2 could leave the accumulator stale.

After this PR

• These payloads carry the counters alone, and the receiver recomputes l2. The fixtures show the saving: the 3-layer 2x4 UnivMon fixture shrinks by 23 bytes, the two-tier fixture by 24, and the 2x4 CountL2HH fixture by 15.
• The accumulator always equals the counters' sum of squares, so a decoded sketch equals the producer's state field for field. Writes cost O(1) per row in every state.
• Estimates are unchanged, except after a row's sum of squares saturated and then decreased: there l2 is now the true value instead of an understatement. Round-trip tests compare l2, estimate_l2, entropy, cardinality and L1.

Verification

Only the tests covering the changed code, plus format and lint, at eae2400 (one commit on origin/main 21b6357):

• cargo test --lib -- countsketch_l2_norm univmon hydra eh_bucket_sketch eh:: octo eh_univmon sketch_catalog: 267 passed.
• cargo test --lib --features experimental -- countsketch_l2 univmon hydra eh_bucket_sketch eh::: 263 passed.
• cargo test --test asapv1_golden: 71 passed (default), 72 passed (experimental).
• cargo test --test e2e -- wire:: hydra_univmon: 17 passed (default), 18 passed (experimental).
• cargo test --release --test e2e -- octo:: univmon: 104 passed.
• cargo test --test pbt -- topk_wrappers univmon hydra: 43 passed. conformance_kit -- univmon l2 hydra: 2 passed. heap_memory_bytes: 1 passed.
• cargo test --test capability: 3 passed (default and experimental).
• cargo test --features octo-runtime --lib -- octo univmon countsketch_l2: 211 passed. cargo test --release --features octo-runtime --test e2e -- octo:: univmon: 127 passed.
• cargo fmt --check: clean. cargo clippy --all-targets, --features experimental and --all-features: no warnings.
• New tests:
  • count_sketch_with_l2_norm_payload_carries_no_l2 and univmon_payload_carries_no_l2 check the payload field counts.
  • *_retired_kind_decodes_and_recomputes_l2, for CountSketchWithL2Norm, UnivMon, two-tier UnivMon and Hydra: a retired envelope with a true, wrong-length, negative, stale, all-zero or empty l2 decodes and re-encodes to the new bytes.
  • count_sketch_with_l2_norm_retired_kind_rejects_a_non_integer_l2: an l2 of strings is rejected.
  • count_sketch_with_l2_norm_rejects_a_stale_l2_cache: the encoder check, and its error names row 0.
  • count_sketch_with_l2_norm_l2_stays_derivable_past_saturation: a row goes to 2^32, then back to 2^31, and the accumulator is 2^62 + 1.
  • Property test the_l2_accumulator_tracks_rows_into_saturation_and_back. A first weight of 2^33 saturates every row. Up to 47 more weights in ±2^33 follow, then all of them negated in reverse, back to zero. After every step, l2_squared equals the median of min(sum(c^2), i64::MAX) computed from the counters, and equals the merge and ASAPv1-decode recomputations.
  • eh_bucket_sketch_decodes_retired_l2_carrying_kind_ids checks the nested ids.
  • retired_l2_fixtures_decode_to_their_no_l2_twins checks every golden pair.
• Changed tests:
  • The property test count_sketch_with_l2_norm_writes_the_cells_the_shared_map_names now inserts through update_weighted_with_hash. It also asserts that the running l2 equals the one merge recomputes and the one the decoder recomputes.
  • known_state_matches_golden builds the recomputed accumulators, which read as [5368610562, i64::MAX], and also decodes the retired fixture.
  • The kind-byte assertions in the unit tests and tests/e2e/wire.rs expect the new ids.
• The tests were written together with the change, not run as failing tests first.
• Not run: the full pbt / e2e suites and benchmarks. CI is not run on this draft.

Limitations and follow-ups

• sketchlib-golden-bytes: branch csl2-derived-l2 (baffc4b) must merge first. The submodule pointer then moves to the merged commit.
• sketchlib-go has no codec for these kinds, only registry entries in wire/asapv1/kind.go (KindCountL2HH, KindUnivMon, KindUnivMonOptimized, KindHydraUnivMon, all at the old ids). Adding 0x19 0x01, 0x10 0x01, 0x11 0x01 and 0x07 0x05, marking the old ids retired, and updating TestRegistry's counts is a small follow-up. It must land once Go's golden submodule is bumped to the new fixtures. No Go change is in this PR.
• A retired payload's l2 is checked only for being an integer array. Its values are dropped, so a stale or crafted l2 in old bytes is silently replaced by the true value.
• The DDSketch PR for Sketch state carries fields no estimate needs (DDSketch sum/min/max, Bloom inserted, Space-Saving total, UnivMon-Q min/max) #250 also edits src/asapv1/kind.rs, eh_bucket_sketch/wire.rs, tests/asapv1_golden.rs, the wire spec and the CHANGELOG. Whichever merges second rebases; the edits touch different lines or table rows.
• Downstream pin bumps: ASAPPlanner and ASAPCollector, if either calls update_weighted_with_hash_without_l2 or as_storage_mut on a CountSketchWithL2Norm.

🤖 Generated with Claude Code