countsketch_l2_norm: recompute l2 on decode, drop it from the wire (new ASAPv1 kinds 0x19 0x01, 0x10 0x01, 0x11 0x01, 0x07 0x05)

CHANGELOG.md

@@ -85,6 +85,25 @@ signals a backwards-compatible change.
   drops the summary's `total`. The retired `0x18 0x00` / `0x1e 0x00` /
   `0x1f 0x00` still decode, with `total` dropped, and are never encoded. The
   `sketchlib-golden-bytes` submodule gains the `*_nototal_*` fixtures.
+- **Breaking: `CountSketchWithL2Norm`'s L2 accumulators always equal each
+  row's sum of squared counters** (saturated at `i64::MAX`). Each row keeps
+  an exact `i128` sum of its squares, each square capped at `2^63`, updated
+  in O(1) per write and clamped to `i64::MAX` when read. A row that
+  saturated and then decreased now reports its true `l2` instead of an
+  understatement; `l2`, `l2_squared`, UnivMon's `estimate_l2` and its
+  terminal-mode heavy-hitter threshold change only in that case. Each row's
+  accumulator is 8 bytes wider (`memory_bytes`, `estimated_memory_bytes`).
+  `update_weighted_with_hash_without_l2` and `as_storage_mut`, the two ways
+  to move counters without the accumulator, are removed; use
+  `update_weighted_with_hash`.
+- **ASAPv1: the L2 accumulators leave the wire.** CountSketchWithL2Norm
+  encodes as `0x19 0x01` (`[counts]`), UnivMon as `0x10 0x01`, two-tier
+  UnivMon as `0x11 0x01` and Hydra over UnivMon as `0x07 0x05`, each payload
+  without `l2`; the decoder recomputes it. The retired `0x19 0x00`,
+  `0x10 0x00`, `0x11 0x00` and `0x07 0x04` still decode, `l2` read and
+  dropped, and are never encoded; `EhBucketSketch` and `ExponentialHistogram`
+  buckets nested under the retired ids decode too. The
+  `sketchlib-golden-bytes` submodule gains the `*_no_l2` fixtures.
 - **Count Sketch estimates no longer allocate.** The per-row estimates a
   median is taken over are collected on the stack instead of in a fresh
   `Vec`: up to 20 rows for `Vector2D` (the fast and regular paths),

docs/api/api_count_sketch.md

@@ -47,7 +47,6 @@ fn try_update_weighted(&mut self, value: &DataInput, weight: f64) -> Result<(),
 // CountSketchWithL2Norm
 fn update_weighted(&mut self, val: &DataInput, weight: i64)
 fn update_weighted_with_hash(&mut self, hashed_val: u128, weight: i64)
-fn update_weighted_with_hash_without_l2(&mut self, hashed_val: u128, c: i64)
 ```
 
 `clear` zeroes every counter and returns the Nitro sampler to its start, keeping
@@ -101,16 +100,17 @@ payload is just `[counts]`, packed row-major with signed cells. The wire covers
 `cols <= u32::MAX`; `to_asapv1` and `from_asapv1` refuse a wider matrix in
 either mode.
 
-`CountSketchWithL2Norm` has its own ASAPv1 kind (`0x19 0x00`) and the same two methods,
+`CountSketchWithL2Norm` has its own ASAPv1 kind (`0x19 0x01`) and the same two methods,
 available for any `H: HashProfile`. Its counters are always `i64` and its
 column derivation is fixed by the algorithm, so the metadata carries no
 `counter_type` and no `mode` — only `seed_index`, `rows` and `cols`. It
 carries the same `1 <= rows <= 20` wire bound. The
-payload is `[counts, l2]`: the matrix packed row-major, then one L2
-accumulator per row. `l2` is carried rather than recomputed, because
-`update_weighted_with_hash_without_l2` moves counters without it and the
-accumulator saturates one way. The same `(counts, l2)` pair is what a `UnivMon`
-layer inlines into its own payload.
+payload is `[counts]`, the matrix packed row-major. Each row's L2 accumulator
+always equals that row's sum of squared counters (saturated at `i64::MAX`), so
+it stays in memory as a cache and the decoder recomputes it. The retired kind
+`0x19 0x00` also carried it as `[counts, l2]`; `from_asapv1` still reads it,
+dropping `l2`, and `to_asapv1` never writes it. The same `counts` run is what a
+`UnivMon` layer inlines into its own payload.
 
 ## Deltas
 

docs/api/api_count_sketch_with_heap.md

@@ -113,7 +113,7 @@ Entries are emitted in descending count (`f64::total_cmp` for `f64` counts),
 ties broken by a total order over the key, so a decoded sketch re-serializes
 byte-identically. The decoder rejects entries in any other order.
 
-`CountSketchWithL2Norm` is a different algorithm with its own ASAPv1 kind (`0x19 0x00`);
+`CountSketchWithL2Norm` is a different algorithm with its own ASAPv1 kind (`0x19 0x01`);
 see [the Count Sketch API doc](./api_count_sketch.md).
 
 ## Deltas

docs/api/api_eh_bucket_sketch.md

@@ -65,21 +65,22 @@ stripped, so each variant's own validation applies unchanged. An
 `ExponentialHistogram` bucket inlines the same triple, so there is one
 EhBucketSketch encoding.
 
-The ten encoded nested kind_ids, plus the retired `0x05 0x00`, which decodes
-into `DdSketch`:
+The ten encoded nested kind_ids. The retired `0x05 0x00`, `0x10 0x00` and
+`0x19 0x00` also decode, into `DdSketch`, `UnivMon` and
+`CountSketchWithL2Norm`:
 
 | Variant | Nested kind_id | Registry name | Feature |
 | ------- | -------------- | ------------- | ------- |
 | `CountMinSketch` | `0x02 0x00` | Count-Min | default |
 | `CocoSketch` | `0x0c 0x00` | Coco | default |
-| `CountSketchWithL2Norm` | `0x19 0x00` | CountSketchWithL2Norm | default |
+| `CountSketchWithL2Norm` | `0x19 0x01` (retired `0x19 0x00` decodes) | CountSketchWithL2Norm | default |
 | `CountSketch` | `0x04 0x00` | Count Sketch | default |
 | `DdSketch` | `0x05 0x01` (the retired `0x05 0x00` still decodes) | DDSketch | default |
 | `ElasticSketch` | `0x0b 0x00` | Elastic | default |
 | `Hll` | `0x01 0x02` | HLL Ertl-MLE | default |
 | `Kll` | `0x06 0x00` | KLL compact | default |
 | `Uniform` | `0x0d 0x00` | UniformSampling | `experimental` |
-| `UnivMon` | `0x10 0x00` | UnivMon | default |
+| `UnivMon` | `0x10 0x01` (retired `0x10 0x00` decodes) | UnivMon | default |
 
 Each id is pinned to one algorithm: `Hll` is Ertl-MLE, so `0x01 0x01` (Classic)
 and `0x01 0x03` (HIP) are rejected, and `Kll` is compact, so `0x06 0x01`

docs/api/api_hydra.md

@@ -78,8 +78,10 @@ fn from_asapv1(bytes: &[u8]) -> Result<Self, rmp_serde::decode::Error>
 
 ASAPv1 MessagePack, one kind_id per counter variant: `0x07 0x00` KLL,
 `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog,
-`0x07 0x04` UnivMon. `from_asapv1` routes on the kind_id and rejects
-any other one.
+`0x07 0x05` UnivMon. `from_asapv1` routes on the kind_id and rejects
+any other one, except the retired UnivMon counter `0x07 0x04`, whose cells also
+carried each layer's `l2`: it still decodes, `l2` dropped and recomputed, and
+is never written.
 
 The metadata carries the hash spec (Hydra hashes its subkeys, so it is present
 even for the KLL counter), the grid `rows` / `cols`, the key-column `schema`,

docs/api/api_univmon.md

@@ -92,16 +92,18 @@ fn to_asapv1(&self) -> Result<Vec<u8>, RmpEncodeError>
 fn from_asapv1(bytes: &[u8]) -> Result<Self, RmpDecodeError>
 ```
 
-These produce/consume the **ASAPv1** wire envelope (kind `0x10 0x00`) — see the
+These produce/consume the **ASAPv1** wire envelope (kind `0x10 0x01`) — see the
 [ASAPv1 wire format spec](../asapv1_wire_format.md). UnivMon hashes through the
 crate's default hasher, so its metadata carries that profile and no seed index:
 the bottom-layer finder's seed is fixed by the algorithm, and layer `i`'s
 counter hashes at seed index `i`.
 
 The pyramid shape (`layers`, `rows`, `cols`, `heap_size`) and
 the heaps' `key_type` live in the metadata; the payload is
-`[counts, l2, heap_lens, keys, heap_counts, candidate_complete, bucket_size,
-update_mode]`. Each layer's `CountSketchWithL2Norm` state is inlined rather than nested in
+`[counts, heap_lens, keys, heap_counts, candidate_complete, bucket_size,
+update_mode]`. Each layer's L2 accumulators are recomputed from its counters on
+decode; the retired kind `0x10 0x00` also carried them at position 1 and still
+decodes, `l2` dropped. Each layer's `CountSketchWithL2Norm` state is inlined rather than nested in
 an envelope of its own, and the heap indexes are rebuilt on load, so no index
 reaches the wire. Heap entries are emitted layer by layer, each layer in
 descending count with ties broken by a total order over the key, so a decoded

docs/api/api_univmon_two_tier.md

@@ -70,7 +70,7 @@ fn to_asapv1(&self) -> Result<Vec<u8>, RmpEncodeError>
 fn from_asapv1(bytes: &[u8]) -> Result<Self, RmpDecodeError>
 ```
 
-These produce/consume the **ASAPv1** wire envelope (kind `0x11 0x00`) — see the
+These produce/consume the **ASAPv1** wire envelope (kind `0x11 0x01`; the retired `0x11 0x00`, which also carried the layers' `l2`, still decodes) — see the
 [ASAPv1 wire format spec](../asapv1_wire_format.md). `UnivMonTwoTier` shares
 [`UnivMon`](./api_univmon.md)'s payload and differs only in its metadata: the
 two-tier layout (`layers`, `elephant_layers`, `elephant_rows`,