diff --git a/CHANGELOG.md b/CHANGELOG.md index 003fdf54..d34f8cfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -85,6 +85,25 @@ signals a backwards-compatible change. drops the summary's `total`. The retired `0x18 0x00` / `0x1e 0x00` / `0x1f 0x00` still decode, with `total` dropped, and are never encoded. The `sketchlib-golden-bytes` submodule gains the `*_nototal_*` fixtures. +- **Breaking: `CountSketchWithL2Norm`'s L2 accumulators always equal each + row's sum of squared counters** (saturated at `i64::MAX`). Each row keeps + an exact `i128` sum of its squares, each square capped at `2^63`, updated + in O(1) per write and clamped to `i64::MAX` when read. A row that + saturated and then decreased now reports its true `l2` instead of an + understatement; `l2`, `l2_squared`, UnivMon's `estimate_l2` and its + terminal-mode heavy-hitter threshold change only in that case. Each row's + accumulator is 8 bytes wider (`memory_bytes`, `estimated_memory_bytes`). + `update_weighted_with_hash_without_l2` and `as_storage_mut`, the two ways + to move counters without the accumulator, are removed; use + `update_weighted_with_hash`. +- **ASAPv1: the L2 accumulators leave the wire.** CountSketchWithL2Norm + encodes as `0x19 0x01` (`[counts]`), UnivMon as `0x10 0x01`, two-tier + UnivMon as `0x11 0x01` and Hydra over UnivMon as `0x07 0x05`, each payload + without `l2`; the decoder recomputes it. The retired `0x19 0x00`, + `0x10 0x00`, `0x11 0x00` and `0x07 0x04` still decode, `l2` read and + dropped, and are never encoded; `EhBucketSketch` and `ExponentialHistogram` + buckets nested under the retired ids decode too. The + `sketchlib-golden-bytes` submodule gains the `*_no_l2` fixtures. - **Count Sketch estimates no longer allocate.** The per-row estimates a median is taken over are collected on the stack instead of in a fresh `Vec`: up to 20 rows for `Vector2D` (the fast and regular paths), diff --git a/asapv1_golden b/asapv1_golden index 89b0c479..26b0f3d6 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 89b0c4798ab5e370bac8fab4b9250e16a67b4578 +Subproject commit 26b0f3d67aeb3c99512131857d6dd839c47473ae diff --git a/docs/api/api_count_sketch.md b/docs/api/api_count_sketch.md index 967c0956..dae26a0b 100644 --- a/docs/api/api_count_sketch.md +++ b/docs/api/api_count_sketch.md @@ -47,7 +47,6 @@ fn try_update_weighted(&mut self, value: &DataInput, weight: f64) -> Result<(), // CountSketchWithL2Norm fn update_weighted(&mut self, val: &DataInput, weight: i64) fn update_weighted_with_hash(&mut self, hashed_val: u128, weight: i64) -fn update_weighted_with_hash_without_l2(&mut self, hashed_val: u128, c: i64) ``` `clear` zeroes every counter and returns the Nitro sampler to its start, keeping @@ -101,16 +100,17 @@ payload is just `[counts]`, packed row-major with signed cells. The wire covers `cols <= u32::MAX`; `to_asapv1` and `from_asapv1` refuse a wider matrix in either mode. -`CountSketchWithL2Norm` has its own ASAPv1 kind (`0x19 0x00`) and the same two methods, +`CountSketchWithL2Norm` has its own ASAPv1 kind (`0x19 0x01`) and the same two methods, available for any `H: HashProfile`. Its counters are always `i64` and its column derivation is fixed by the algorithm, so the metadata carries no `counter_type` and no `mode` — only `seed_index`, `rows` and `cols`. It carries the same `1 <= rows <= 20` wire bound. The -payload is `[counts, l2]`: the matrix packed row-major, then one L2 -accumulator per row. `l2` is carried rather than recomputed, because -`update_weighted_with_hash_without_l2` moves counters without it and the -accumulator saturates one way. The same `(counts, l2)` pair is what a `UnivMon` -layer inlines into its own payload. +payload is `[counts]`, the matrix packed row-major. Each row's L2 accumulator +always equals that row's sum of squared counters (saturated at `i64::MAX`), so +it stays in memory as a cache and the decoder recomputes it. The retired kind +`0x19 0x00` also carried it as `[counts, l2]`; `from_asapv1` still reads it, +dropping `l2`, and `to_asapv1` never writes it. The same `counts` run is what a +`UnivMon` layer inlines into its own payload. ## Deltas diff --git a/docs/api/api_count_sketch_with_heap.md b/docs/api/api_count_sketch_with_heap.md index 45b8c258..42022ed6 100644 --- a/docs/api/api_count_sketch_with_heap.md +++ b/docs/api/api_count_sketch_with_heap.md @@ -113,7 +113,7 @@ Entries are emitted in descending count (`f64::total_cmp` for `f64` counts), ties broken by a total order over the key, so a decoded sketch re-serializes byte-identically. The decoder rejects entries in any other order. -`CountSketchWithL2Norm` is a different algorithm with its own ASAPv1 kind (`0x19 0x00`); +`CountSketchWithL2Norm` is a different algorithm with its own ASAPv1 kind (`0x19 0x01`); see [the Count Sketch API doc](./api_count_sketch.md). ## Deltas diff --git a/docs/api/api_eh_bucket_sketch.md b/docs/api/api_eh_bucket_sketch.md index c8ac6007..42acb0f3 100644 --- a/docs/api/api_eh_bucket_sketch.md +++ b/docs/api/api_eh_bucket_sketch.md @@ -65,21 +65,22 @@ stripped, so each variant's own validation applies unchanged. An `ExponentialHistogram` bucket inlines the same triple, so there is one EhBucketSketch encoding. -The ten encoded nested kind_ids, plus the retired `0x05 0x00`, which decodes -into `DdSketch`: +The ten encoded nested kind_ids. The retired `0x05 0x00`, `0x10 0x00` and +`0x19 0x00` also decode, into `DdSketch`, `UnivMon` and +`CountSketchWithL2Norm`: | Variant | Nested kind_id | Registry name | Feature | | ------- | -------------- | ------------- | ------- | | `CountMinSketch` | `0x02 0x00` | Count-Min | default | | `CocoSketch` | `0x0c 0x00` | Coco | default | -| `CountSketchWithL2Norm` | `0x19 0x00` | CountSketchWithL2Norm | default | +| `CountSketchWithL2Norm` | `0x19 0x01` (retired `0x19 0x00` decodes) | CountSketchWithL2Norm | default | | `CountSketch` | `0x04 0x00` | Count Sketch | default | | `DdSketch` | `0x05 0x01` (the retired `0x05 0x00` still decodes) | DDSketch | default | | `ElasticSketch` | `0x0b 0x00` | Elastic | default | | `Hll` | `0x01 0x02` | HLL Ertl-MLE | default | | `Kll` | `0x06 0x00` | KLL compact | default | | `Uniform` | `0x0d 0x00` | UniformSampling | `experimental` | -| `UnivMon` | `0x10 0x00` | UnivMon | default | +| `UnivMon` | `0x10 0x01` (retired `0x10 0x00` decodes) | UnivMon | default | Each id is pinned to one algorithm: `Hll` is Ertl-MLE, so `0x01 0x01` (Classic) and `0x01 0x03` (HIP) are rejected, and `Kll` is compact, so `0x06 0x01` diff --git a/docs/api/api_hydra.md b/docs/api/api_hydra.md index 7ac4a7c3..f4ceaf7a 100644 --- a/docs/api/api_hydra.md +++ b/docs/api/api_hydra.md @@ -78,8 +78,10 @@ fn from_asapv1(bytes: &[u8]) -> Result ASAPv1 MessagePack, one kind_id per counter variant: `0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, -`0x07 0x04` UnivMon. `from_asapv1` routes on the kind_id and rejects -any other one. +`0x07 0x05` UnivMon. `from_asapv1` routes on the kind_id and rejects +any other one, except the retired UnivMon counter `0x07 0x04`, whose cells also +carried each layer's `l2`: it still decodes, `l2` dropped and recomputed, and +is never written. The metadata carries the hash spec (Hydra hashes its subkeys, so it is present even for the KLL counter), the grid `rows` / `cols`, the key-column `schema`, diff --git a/docs/api/api_univmon.md b/docs/api/api_univmon.md index 72956687..7c63a4e1 100644 --- a/docs/api/api_univmon.md +++ b/docs/api/api_univmon.md @@ -92,7 +92,7 @@ fn to_asapv1(&self) -> Result, RmpEncodeError> fn from_asapv1(bytes: &[u8]) -> Result ``` -These produce/consume the **ASAPv1** wire envelope (kind `0x10 0x00`) — see the +These produce/consume the **ASAPv1** wire envelope (kind `0x10 0x01`) — see the [ASAPv1 wire format spec](../asapv1_wire_format.md). UnivMon hashes through the crate's default hasher, so its metadata carries that profile and no seed index: the bottom-layer finder's seed is fixed by the algorithm, and layer `i`'s @@ -100,8 +100,10 @@ counter hashes at seed index `i`. The pyramid shape (`layers`, `rows`, `cols`, `heap_size`) and the heaps' `key_type` live in the metadata; the payload is -`[counts, l2, heap_lens, keys, heap_counts, candidate_complete, bucket_size, -update_mode]`. Each layer's `CountSketchWithL2Norm` state is inlined rather than nested in +`[counts, heap_lens, keys, heap_counts, candidate_complete, bucket_size, +update_mode]`. Each layer's L2 accumulators are recomputed from its counters on +decode; the retired kind `0x10 0x00` also carried them at position 1 and still +decodes, `l2` dropped. Each layer's `CountSketchWithL2Norm` state is inlined rather than nested in an envelope of its own, and the heap indexes are rebuilt on load, so no index reaches the wire. Heap entries are emitted layer by layer, each layer in descending count with ties broken by a total order over the key, so a decoded diff --git a/docs/api/api_univmon_two_tier.md b/docs/api/api_univmon_two_tier.md index e2e2db93..d6793beb 100644 --- a/docs/api/api_univmon_two_tier.md +++ b/docs/api/api_univmon_two_tier.md @@ -70,7 +70,7 @@ fn to_asapv1(&self) -> Result, RmpEncodeError> fn from_asapv1(bytes: &[u8]) -> Result ``` -These produce/consume the **ASAPv1** wire envelope (kind `0x11 0x00`) — see the +These produce/consume the **ASAPv1** wire envelope (kind `0x11 0x01`; the retired `0x11 0x00`, which also carried the layers' `l2`, still decodes) — see the [ASAPv1 wire format spec](../asapv1_wire_format.md). `UnivMonTwoTier` shares [`UnivMon`](./api_univmon.md)'s payload and differs only in its metadata: the two-tier layout (`layers`, `elephant_layers`, `elephant_rows`, diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index ca389b8f..a957e54c 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -204,7 +204,8 @@ This registry is the master list of algorithms still to design payloads for, and | `0x07 0x01` | Hydra | Count-Min counter | Section 3.9 | implemented | | `0x07 0x02` | Hydra | Count Sketch counter | Section 3.9 | implemented | | `0x07 0x03` | Hydra | HyperLogLog counter | Section 3.9 | implemented | -| `0x07 0x04` | Hydra | UnivMon counter | Section 3.9 | implemented | +| `0x07 0x04` | Hydra | UnivMon counter, cells carrying `l2` | Section 3.9 | retired: decoded, `l2` dropped; never encoded | +| `0x07 0x05` | Hydra | UnivMon counter | Section 3.9 | implemented | | `0x08 0x00` | SetAggregator | Exact string set | Section 3.21 | implemented | | `0x08 0x80` | SetAggregator | SetAggregator delta | Section 3.22 | implemented | | `0x09 0x00` | - | - | TBD | retired / permanently reserved, never reassign | @@ -215,8 +216,10 @@ This registry is the master list of algorithms still to design payloads for, and | `0x0d 0x00` | UniformSampling (`Unstable`) | Priority sampling | Section 3.13 | implemented | | `0x0e 0x00` | KMV (`Unstable`) | k-minimum-values | Section 3.14 | implemented | | `0x0f 0x00` | HashSketchEnsemble | - | TBD | reserved / not designed | -| `0x10 0x00` | UnivMon | Pyramid | Section 3.15 | implemented | -| `0x11 0x00` | Two-tier UnivMon (`UnivMonTwoTier`) | Two-tier pyramid | Section 3.16 | implemented | +| `0x10 0x00` | UnivMon | Pyramid carrying `l2` | Section 3.15 | retired: decoded, `l2` dropped; never encoded | +| `0x10 0x01` | UnivMon | Pyramid | Section 3.15 | implemented | +| `0x11 0x00` | Two-tier UnivMon (`UnivMonTwoTier`) | Two-tier pyramid carrying `l2` | Section 3.16 | retired: decoded, `l2` dropped; never encoded | +| `0x11 0x01` | Two-tier UnivMon (`UnivMonTwoTier`) | Two-tier pyramid | Section 3.16 | implemented | | `0x12 0x00` | NitroBatch | - | TBD | reserved / not designed | | `0x13 0x00` | ExponentialHistogram | Sliding window over sketch buckets | Section 3.17 | implemented | | `0x14 0x00` | EhBucketSketch | Nested-sketch union | Section 3.18 | implemented | @@ -226,7 +229,8 @@ This registry is the master list of algorithms still to design payloads for, and | `0x17 0x01` | Bloom | Partitioned | Section 3.4 | implemented | | `0x18 0x00` | Space-Saving | Stream-Summary with `total` | Section 3.5 | retired: decoded (`total` dropped), never encoded | | `0x18 0x01` | Space-Saving | Stream-Summary | Section 3.5 | implemented | -| `0x19 0x00` | Count-Sketch-with-L2-norm (`CountSketchWithL2Norm`) | Count Sketch with L2 accumulators | Section 3.19 | implemented | +| `0x19 0x00` | Count-Sketch-with-L2-norm (`CountSketchWithL2Norm`) | Count Sketch carrying its L2 accumulators | Section 3.19 | retired: decoded, `l2` dropped; never encoded | +| `0x19 0x01` | Count-Sketch-with-L2-norm (`CountSketchWithL2Norm`) | Count Sketch counters (L2 recomputed) | Section 3.19 | implemented | | `0x1a 0x00` | UnivMon-Q (`UnivMonQ`) | Quantile pyramid | Section 3.20 | implemented | | `0x1b 0x00` | CountMinSketchFold | - | TBD | reserved / not designed | | `0x1c 0x00` | CountSketchFold | - | TBD | reserved / not designed | @@ -239,7 +243,7 @@ This registry is the master list of algorithms still to design payloads for, and **Mapping notes** (mismatches between `apis.md` and Go's `magic_ids.go`): - **CountMinSketchWithHeap vs CountSketchWithHeap.** Go's `MagicCountMinSketchWithHeap` (`0x03`) is the Count-*Min*-with-heap sketch (`apis.md` to CountMinSketchWithHeap). The Count-*Sketch*-with-heap sketch (`apis.md` to CountSketchWithHeap) is a distinct family and gets a fresh byte (`0x0a`), separate from `0x03`. -- **Hydra.** `apis.md` lists the "Hydra" framework; Go's only Hydra id is `MagicHydraKLLSketch` (`0x07`), so Hydra maps here to the `0x07` family. Each base sketch under Hydra has its own variant: `0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, `0x07 0x04` UnivMon. `0x07 0x05`-`0x07 0x7f` are **reserved for Hydra over further base sketches**; a concrete variant is allocated when that combination's payload is designed. +- **Hydra.** `apis.md` lists the "Hydra" framework; Go's only Hydra id is `MagicHydraKLLSketch` (`0x07`), so Hydra maps here to the `0x07` family. Each base sketch under Hydra has its own variant: `0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, `0x07 0x05` UnivMon. `0x07 0x04` was the UnivMon counter while its cells carried each layer's `l2`; it is retired (Q-VER), and since the variant byte is the only version Hydra has, the new cell payload took the next free variant, `0x07 0x05`. `0x07 0x06`-`0x07 0x7f` are **reserved for Hydra over further base sketches**; a concrete variant is allocated when that combination's payload is designed. - **SetAggregator / SetAggregatorDelta** (`0x08 0x00` / `0x08 0x80`) are exact string sets, not sketches, so `apis.md` does not list them. - **`0x19`-`0x1c`, `0x1e`-`0x1f`.** CountSketchWithL2Norm, UnivMon-Q, CountMinSketchFold, CountSketchFold, CountMinSketchWithSpaceSaving and CountSketchWithSpaceSaving have no counterpart in Go's `magic_ids.go`; their family bytes are allocated here first, and Go mirrors them from this registry. - **`Unstable`** rows mirror the `Unstable` status those sketches carry in `apis.md`. The kind_id is permanent either way; the payload and the sketch API may still change, and a change to the payload takes a new id (Q-VER). @@ -434,7 +438,7 @@ This is safe on both ends because serialization **fails closed**: Per sketch. **Raw state only**, a **positional msgpack array** in the order its kind_id implies. Rules: - No field that `kind_id` or the metadata already determines (no variant tag, no precision, no counter type, no mode). -- No field derivable from another (no HLL `precision`; no CMS `l1`/`l2`, which are `sum(count)` / `sum(count^2)` and are recomputed on decode). The rule is "derivable", not "named `l2`": CountSketchWithL2Norm and the UnivMon family carry a payload field spelled exactly `l2` (§3.19, §3.15, §3.16) and are right to, because that accumulator is **not** `sum(counts[row]^2)` — an insert path exists that moves counters without touching it, and it clamps. Nothing derivable rides the wire; nothing that only looks derivable is dropped. +- No field derivable from another (no HLL `precision`; no CMS `l1`/`l2`, which are `sum(count)` / `sum(count^2)` and are recomputed on decode; no CountSketchWithL2Norm or UnivMon-layer `l2`, which is each row's `sum(counts[row]^2)` saturated at `i64::MAX` and is recomputed on decode, §3.19). Nothing derivable rides the wire. - msgpack array (positional), never a keyed map. The exact msgpack types are in "Wire encoding rules". ### 3.1: HLL payload (`0x01 0x01` / `0x01 0x02` / `0x01 0x03`) @@ -493,7 +497,7 @@ On the **fast** path one hash serves the whole matrix wherever it can: the layou In the two packed layouts the rows are **bit fields of one hash**, not independent draws. `rows <= 20` is declared and enforced on **both** doors whichever layout a geometry selects, so one geometry rule covers the format. `with_dimensions` is free to build a wider matrix in memory; such a sketch is rejected on **both** sides, so the format never emits bytes it would refuse to read back. -The same `1 <= rows <= 20` bound is declared by **every matrix-shaped payload**, under whichever name that payload's rows carry: Bloom's slices (`0x17 0x01`, §3.4, as `BLOOM_MAX_SLICES`), Count Sketch (`0x04 0x00`, §3.6), CountMinSketchWithHeap (`0x03 0x00`), CountSketchWithHeap (`0x0a 0x00`), CountMinSketchWithSpaceSaving (`0x1e 0x01`) and CountSketchWithSpaceSaving (`0x1f 0x01`) with their base matrix, CountSketchWithL2Norm (`0x19 0x00`), Coco's table (`0x0c 0x00`), Elastic's light layer (`0x0b 0x00`), Hydra's grid and its matrix counters (`0x07 0x01` / `0x07 0x02`), and every UnivMon (`0x10 0x00`) and Two-tier UnivMon (`0x11 0x00`) layer. +The same `1 <= rows <= 20` bound is declared by **every matrix-shaped payload**, under whichever name that payload's rows carry: Bloom's slices (`0x17 0x01`, §3.4, as `BLOOM_MAX_SLICES`), Count Sketch (`0x04 0x00`, §3.6), CountMinSketchWithHeap (`0x03 0x00`), CountSketchWithHeap (`0x0a 0x00`), CountMinSketchWithSpaceSaving (`0x1e 0x01`) and CountSketchWithSpaceSaving (`0x1f 0x01`) with their base matrix, CountSketchWithL2Norm (`0x19 0x01`), Coco's table (`0x0c 0x00`), Elastic's light layer (`0x0b 0x00`), Hydra's grid and its matrix counters (`0x07 0x01` / `0x07 0x02`), and every UnivMon (`0x10 0x01`) and Two-tier UnivMon (`0x11 0x01`) layer. **CountSketchWithL2Norm never draws per row**, and neither do the UnivMon layers built from it. One `hash128_seeded(seed_index, key)` supplies every row: the column bits at `r * ceil_log2(cols)` and the sign at bit `127 - r`. It still declares `rows <= 20`, and it carries the second bound the packed layout makes binding — `(ceil_log2(cols) + 1) * rows <= 128`, the bits one 128-bit hash has to give out (§3.19). @@ -772,9 +776,9 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. ` Rules 3, 4, 5 and 6 are enforced on the **encode** side too, so the format never emits bytes it would refuse to read back. -### 3.9: Hydra payload (`0x07 0x00`-`0x07 0x04`) +### 3.9: Hydra payload (`0x07 0x00`-`0x07 0x05`) -A Hydra is a `rows x cols` grid of counters over a fixed set of named key columns; each record fans out into its `2^D - 1` subpopulations, and each subkey is hashed to one column per row. **The kind_id names the counter variant**, so the five ids are one algorithm over five cell types: `0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, `0x07 0x04` UnivMon. The payload carries no variant tag, per cell or otherwise, and a grid mixing variants has no encoding. +A Hydra is a `rows x cols` grid of counters over a fixed set of named key columns; each record fans out into its `2^D - 1` subpopulations, and each subkey is hashed to one column per row. **The kind_id names the counter variant**, so the five current ids are one algorithm over five cell types: `0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, `0x07 0x05` UnivMon. The retired `0x07 0x04` is the UnivMon counter whose cells were in the retired §3.15 shape that also carries `l2`; it still decodes (each cell's `l2` read and dropped, then recomputed) and is never encoded. The payload carries no variant tag, per cell or otherwise, and a grid mixing variants has no encoding. **Counters are inlined, not nested.** A cell's raw state goes straight into Hydra's positional array in the shape that counter's own section fixes; no cell carries an envelope, a magic or a metadata map of its own. @@ -798,11 +802,13 @@ The two ids share one payload type: one `Vec` through one encoder, read bac | ----- | ------- | ------ | ------- | | 0 | `cells` | array | one element per grid cell, **row-major**; each element is that cell's §3.3 array `[levels, items, coin]`, with `items` typed by `cell_item_type` | -**`0x07 0x04` UnivMon counter:** +**`0x07 0x05` UnivMon counter:** | Pos | Field | Type | Notes | | ----- | ------- | ------ | ------- | -| 0 | `cells` | array | one element per grid cell, **row-major**; each element is that cell's §3.15 pyramid array `[counts, l2, heap_lens, keys, heap_counts, candidate_complete, bucket_size, update_mode]`, with `keys` typed by `cell_key_type` | +| 0 | `cells` | array | one element per grid cell, **row-major**; each element is that cell's §3.15 pyramid array `[counts, heap_lens, keys, heap_counts, candidate_complete, bucket_size, update_mode]`, with `keys` typed by `cell_key_type` | + +The retired `0x07 0x04` has the same metadata and the same `cells`, each cell in the retired §3.15 shape `[counts, l2, heap_lens, keys, heap_counts, candidate_complete, bucket_size, update_mode]`. A fixed-size counter tiles one array whose length the metadata fixes exactly (`rows*cols*per_cell`, overflow-checked before anything is sized from it). A variable-size counter has no such stride, so its cells are carried one element each, in the counter's own payload shape. @@ -813,7 +819,7 @@ A fixed-size counter tiles one array whose length the metadata fixes exactly (`r | `0x07 0x00` KLL | yes, no seed index | `rows`, `cols`, `schema`, `cell_k`, `cell_m`, `cell_item_type` | | `0x07 0x01` / `0x07 0x02` | yes, `matrix_seed_index` | `rows`, `cols`, `schema`, `cell_rows`, `cell_cols`, `cell_counter_type`, `cell_mode` | | `0x07 0x03` HLL | yes, `canonical_seed_index` | `rows`, `cols`, `schema`, `cell_precision` | -| `0x07 0x04` UnivMon | yes, no seed index | `rows`, `cols`, `schema`, `cell_layers`, `cell_rows`, `cell_cols`, `cell_heap_size`, `cell_key_type` | +| `0x07 0x05` UnivMon (and the retired `0x07 0x04`) | yes, no seed index | `rows`, `cols`, `schema`, `cell_layers`, `cell_rows`, `cell_cols`, `cell_heap_size`, `cell_key_type` | `schema` is carried as the label list alone. `KeySchema` holds `labels` plus a pre-escaped cache; the cache is a pure function of the labels (the `\`, `:` and `;` escape), so only `labels` reaches the wire and `TryFrom>` rebuilds and re-validates the rest. `schema` must round-trip exactly: changing it would invalidate every subkey already hashed into the grid. @@ -827,12 +833,12 @@ The KLL counter's optional `compaction_seed` is not carried, so the bounded cost **Decode rules.** Fail **closed** on each, with an error and never a panic: -1. `kind_id` is one of the five; the decoder routes on it and each variant's decoder owns exactly one id. -2. The hash-spec group matches the target's `HashProfile` (Hydra hashes through the crate default, `DefaultXxHasher`). `cell_counter_type` (`"i32"`), `cell_mode` (`"fast"`), `cell_item_type` (`"f64"`) and `cell_precision` are pinned, since the counter variant fixes them; `rows` / `cols` / `schema` and the counter *dimensions* — `cell_rows` / `cell_cols`, `cell_k` / `cell_m`, `cell_layers`, `cell_heap_size` — are properties of the *stored* grid, so they are echoed back and then validated on their own. `0x07 0x04`'s `cell_key_type` belongs to neither list: it is not pinned (the cells' keys are whatever the caller inserted, as §3.5 describes) and it is not a dimension (nothing is sized from it). It is echoed back like the dimensions and validated by rule 5. -3. `rows` and `cols` are non-zero, `rows <= 20` (`MATRIX_MAX_ROWS`, §3.2 — the grid places a subkey the way a Count-Min matrix does, and declares the same bound) and `rows * cols` is overflow-checked. For the tiled variants the counter dimensions are non-zero, `cell_rows <= 20` for the two matrix counters and `rows * cols * per_cell` is overflow-checked too, and for `0x07 0x04` every one of `cell_layers` / `cell_rows` / `cell_cols` / `cell_heap_size` is non-zero. All **before** anything is sized from them. +1. `kind_id` is one of the five, or the retired `0x07 0x04`; the decoder routes on it, each variant's decoder owns exactly one current id, and the UnivMon decoder also reads the retired `0x07 0x04`. +2. The hash-spec group matches the target's `HashProfile` (Hydra hashes through the crate default, `DefaultXxHasher`). `cell_counter_type` (`"i32"`), `cell_mode` (`"fast"`), `cell_item_type` (`"f64"`) and `cell_precision` are pinned, since the counter variant fixes them; `rows` / `cols` / `schema` and the counter *dimensions* — `cell_rows` / `cell_cols`, `cell_k` / `cell_m`, `cell_layers`, `cell_heap_size` — are properties of the *stored* grid, so they are echoed back and then validated on their own. `0x07 0x05`'s `cell_key_type` belongs to neither list: it is not pinned (the cells' keys are whatever the caller inserted, as §3.5 describes) and it is not a dimension (nothing is sized from it). It is echoed back like the dimensions and validated by rule 5. +3. `rows` and `cols` are non-zero, `rows <= 20` (`MATRIX_MAX_ROWS`, §3.2 — the grid places a subkey the way a Count-Min matrix does, and declares the same bound) and `rows * cols` is overflow-checked. For the tiled variants the counter dimensions are non-zero, `cell_rows <= 20` for the two matrix counters and `rows * cols * per_cell` is overflow-checked too, and for `0x07 0x05` every one of `cell_layers` / `cell_rows` / `cell_cols` / `cell_heap_size` is non-zero. All **before** anything is sized from them. 4. The payload's length is measured against the declared geometry — `len(counts) == rows*cols*cell_rows*cell_cols`, `len(registers) == rows*cols*2^cell_precision`, `len(cells) == rows*cols` — before any grid or matrix is allocated. A declared grid larger than the payload carries costs nothing. -5. `0x07 0x04` reads `cells` as `cell_key_type`, which must be one of §3.5's fourteen names, with §3.5 rule 3's relabel limits. A grid whose cells hold no key requires `cell_key_type` `"u64"`, and a cell holding no key is handed to the UnivMon decoder under `"u64"`. -6. The three variable-size counters rebuild each cell **through that counter's own decoder** — the cell's blocks are wrapped back into the counter's envelope and handed to it — so every rule that decoder enforces holds for a Hydra cell too: KLL's level layout and `k` / `m` bounds (`0x07 0x00`), HLL's register count (`0x07 0x03`), UnivMon's per-layer geometry and heap runs (`0x07 0x04`). The two tiled matrix variants (`0x07 0x01` / `0x07 0x02`) do not: their cells are cut out of the flat `counts` run and built directly from storage, never through the counter's own `from_asapv1`, so what guards them is rule 3's geometry bounds and rule 4's length check and nothing further. There is nothing left for a counter decoder to check — a fixed-size matrix has no state beyond its cells — but a Go implementer must not expect a nested decode to run there. +5. `0x07 0x05` (and the retired `0x07 0x04`) reads `cells` as `cell_key_type`, which must be one of §3.5's fourteen names, with §3.5 rule 3's relabel limits. A grid whose cells hold no key requires `cell_key_type` `"u64"`, and a cell holding no key is handed to the UnivMon decoder under `"u64"`. +6. The three variable-size counters rebuild each cell **through that counter's own decoder** — the cell's blocks are wrapped back into the counter's envelope and handed to it — so every rule that decoder enforces holds for a Hydra cell too: KLL's level layout and `k` / `m` bounds (`0x07 0x00`), HLL's register count (`0x07 0x03`), UnivMon's per-layer geometry and heap runs (`0x07 0x05`; a retired `0x07 0x04` cell is first rewritten in the current §3.15 shape, its `l2` dropped). The two tiled matrix variants (`0x07 0x01` / `0x07 0x02`) do not: their cells are cut out of the flat `counts` run and built directly from storage, never through the counter's own `from_asapv1`, so what guards them is rule 3's geometry bounds and rule 4's length check and nothing further. There is nothing left for a counter decoder to check — a fixed-size matrix has no state beyond its cells — but a Go implementer must not expect a nested decode to run there. 7. `schema` is re-validated through `KeySchema::try_from`: non-empty, at most `MAX_KEY_COLUMNS` labels, no duplicates. 8. `prototype` is rebuilt as a fresh counter of the declared geometry; nothing about it is read from the payload. @@ -1021,22 +1027,23 @@ KMV carries the **hash-spec group** — it hashes, and a consumer must reproduce The encode side enforces rules 3 and 4 as well, and rejects a sketch holding the same digest twice. It also rejects a sketch whose `k` disagrees with the retained heap's own bound: `k` and the heap are both public fields, so an out-of-step pair is constructible, and it would put a retention bound on the wire that the sketch it came from was not keeping — the treatment Elastic gives `heavy_buckets` (§3.11). So the format never emits bytes it would refuse to read back. -### 3.15: UnivMon payload (`0x10 0x00`) +### 3.15: UnivMon payload (`0x10 0x01`; retired `0x10 0x00`) `UnivMon` is a pyramid of `layers` layers, each a `CountSketchWithL2Norm` (§3.19) of `rows x cols` plus an `TopKHeap` of capacity `heap_size`. Every layer has the same dimensions and the same heap capacity, and layer `i` hashes at seed index `i`, so all of that is metadata or derived and the payload is the layers' raw state: | Pos | Field | Type | Notes | | ----- | ------- | ------ | ------- | | 0 | `counts` | array | the layers' `CountSketchWithL2Norm` counters concatenated in layer order, each layer row-major; length `layers * rows * cols` | -| 1 | `l2` | array | the layers' L2 accumulators concatenated in layer order; length `layers * rows` | -| 2 | `heap_lens` | array | u32 entries held by each layer's heap, one per layer | -| 3 | `keys` | array | every layer's heap keys concatenated, cut by `heap_lens`; element type = `key_type`; homogeneous | -| 4 | `heap_counts` | array | i64 heap count, parallel to `keys` | -| 5 | `candidate_complete` | array | one bool per layer | -| 6 | `bucket_size` | u64 | total weight recorded | -| 7 | `update_mode` | u8 | `0` unset, `1` standard, `2` terminal-only | +| 1 | `heap_lens` | array | u32 entries held by each layer's heap, one per layer | +| 2 | `keys` | array | every layer's heap keys concatenated, cut by `heap_lens`; element type = `key_type`; homogeneous | +| 3 | `heap_counts` | array | i64 heap count, parallel to `keys` | +| 4 | `candidate_complete` | array | one bool per layer | +| 5 | `bucket_size` | u64 | total weight recorded | +| 6 | `update_mode` | u8 | `0` unset, `1` standard, `2` terminal-only | + +The layers' L2 accumulators are not carried: each is its row's sum of squared counters (§3.19), recomputed on decode. The retired `0x10 0x00` has the same metadata and an eight-element payload with `l2` (the layers' accumulators concatenated in layer order, `i64`) at position 1 and the fields above shifted by one; a decoder reads it, drops `l2`, and recomputes. It is never encoded. -**A layer's `CountSketchWithL2Norm` is inlined, not nested.** The layer's `counts` and `l2` are appended to the outer sketch's own positional arrays, and its `rows` / `cols` / `seed_index` come from the outer metadata plus the layer's position. No nested envelope, magic, version or metadata map is emitted per layer. +**A layer's `CountSketchWithL2Norm` is inlined, not nested.** The layer's `counts` are appended to the outer sketch's own positional arrays, and its `rows` / `cols` / `seed_index` come from the outer metadata plus the layer's position. No nested envelope, magic, version or metadata map is emitted per layer. **Metadata vs payload.** `layers`, `rows`, `cols` and `heap_size` are `new`'s four arguments — configuration that shapes the payload, so the descriptor carries them and every derived length is checked rather than re-stored. `key_type` is a structural param in the §3.5 sense: it fixes the element type of `keys`, and the payload cannot be read without it. One `key_type` covers every layer; a pyramid whose layers mix `OwnedDataInput` variants has no single one and fails to serialize, as a Space-Saving summary does. Structural-param order is `layers, rows, cols, heap_size, key_type`. @@ -1047,7 +1054,7 @@ UnivMon carries the hash-spec group — it hashes, and a consumer must reproduce - `update_mode` is acquired from the first update rather than configured, and it selects the query recurrence (`calc_terminal_g_sum` versus the standard layer recurrence). A decoder that guessed it would answer entropy and cardinality from the wrong reconstruction. - `candidate_complete` is not derivable. `len < heap_size` implies complete, but a full heap is ambiguous — exactly-filled-never-evicted and evicted-since look identical — and `mark_candidates_incomplete`, the OctoSketch delta path, lowers the flag with no observable trace. A decoder that re-derived it from heap fullness would call an evicted-from layer complete: `top_layer_complete()` would report it, and terminal-mode g-sums with `count_every_key = false` (entropy) would drop their `l2 / sqrt(heap_size)` threshold. -`bucket_size` is the exact L1 and is not recoverable from the counters, so it is carried. Everything else UnivMon reports — `estimate_l2`, `estimate_entropy`, `estimate_cardinality` — is recomputed from the layers and appears nowhere. +`bucket_size` is the exact L1 and is not recoverable from the counters, so it is carried. Everything else UnivMon reports — the layers' `l2`, `estimate_l2`, `estimate_entropy`, `estimate_cardinality` — is recomputed from the layers and appears nowhere. **Every layer is a CountSketchWithL2Norm.** `kind_id` fixes the per-layer counter, so the wire spells out each layer's CountSketchWithL2Norm state directly with no tag — the same rule that keeps a variant tag out of every other payload. Another per-layer counter needs either a new metadata structural param naming it, as Count-Min does with `counter_type`, or a new `kind_id`. @@ -1055,34 +1062,35 @@ UnivMon carries the hash-spec group — it hashes, and a consumer must reproduce **Decode rules.** Fail **closed** on each, with an error and never a panic: -1. `kind_id` is `0x10 0x00`; any other id is rejected. +1. `kind_id` is `0x10 0x01`, or the retired `0x10 0x00`, which selects the eight-element layout; any other id is rejected. 2. The hash-spec group matches `DefaultXxHasher`'s `HashProfile`. `layers`, `rows`, `cols`, `heap_size` and `key_type` are properties of the stored sketch, so they are echoed back rather than pinned. 3. `layers`, `rows`, `cols` and `heap_size` are **all non-zero**, and `layers <= 19` (`UNIVMON_MAX_LAYERS`). All of it is checked **before** the per-layer geometry vector is built, so it is the first thing standing between a declared shape and an allocation. The ceiling is a seed bound: layer `i` hashes at seed index `i`, and the layer finder hashes at `UNIVMON_BOTTOM_LAYER_SEED_INDEX` (19), so a 20th layer would share the finder's seed. The ceiling is checked on **both** doors; the four non-zero checks are the decode door's, since only a decoder is handed a shape it did not build. 4. `key_type` is one of §3.5's fourteen names; the `keys` array is read **as** that type, with §3.5 rule 3's relabel limits; and `keys` and `heap_counts` have equal length. A pyramid whose heaps hold no key requires `key_type` `"u64"`. -5. `rows * layers == len(l2)` exactly, with checked arithmetic, and checked **before** the layer geometry is built. Together with rule 3's ceiling this is what makes a declared `layers` far larger than the payload carries cost nothing: `rows >= 1`, so the equality caps `layers` at `len(l2)`, and the ceiling caps it at 64 regardless. -6. `len(heap_lens) == len(candidate_complete) == layers`; every layer's `rows` and `cols` are non-zero and `rows <= 20` (`MATRIX_MAX_ROWS`, §3.2 — a layer is a CountSketchWithL2Norm matrix; `cols.ilog2()` panics on zero) and its geometry fits the 128-bit key hash, `(ceil_log2(cols) + 1) * rows <= 128` (§3.19); the layers' total cell count and total accumulator count are computed with checked arithmetic and must equal `len(counts)` and `len(l2)`; and `len(keys) == len(heap_counts) == sum(heap_lens)`. +5. A declared `layers` far larger than the payload carries costs nothing: rule 3's ceiling caps the per-layer geometry vector before it is built, and rule 6 measures its total cell count against `len(counts)` before any matrix is. (A retired `0x10 0x00` payload's `l2` is not measured; it is dropped.) +6. `len(heap_lens) == len(candidate_complete) == layers`; every layer's `rows` and `cols` are non-zero and `rows <= 20` (`MATRIX_MAX_ROWS`, §3.2 — a layer is a CountSketchWithL2Norm matrix; `cols.ilog2()` panics on zero) and its geometry fits the 128-bit key hash, `(ceil_log2(cols) + 1) * rows <= 128` (§3.19); the layers' total cell count is computed with checked arithmetic and must equal `len(counts)`; and `len(keys) == len(heap_counts) == sum(heap_lens)`. 7. Each layer's entry count is `<= heap_size`, and no key appears twice within a layer — judged on the key's wire representation before any entry is seated, exactly as §3.7's rule 7 describes, since a layer's heap is rebuilt through the same code. `heap_size` **never sizes an allocation**: each heap reserves `min(entries this layer carries, heap_size, 1024)` and grows from there, so a crafted `heap_size` costs nothing and an honest one reserves no more than 1024 slots up front. It still fixes the decoded heap's capacity. -8. Every layer's `l2[i] >= 0`, per layer and per row — the same check §3.19's rule 7 makes on a stand-alone CountSketchWithL2Norm, applied on this door too, since a UnivMon layer *is* one. +8. Every layer's L2 accumulators are recomputed from its counters, as §3.19's rule 7 does for a stand-alone CountSketchWithL2Norm; the encoder rejects a layer whose cached accumulator disagrees. 9. `bucket_size` fits this target's `usize`. 10. `update_mode` is `0`, `1` or `2`; any other value is rejected. 11. Each layer's heap entries are in the emitted order above; any other order is rejected. The encode side enforces the same shape rules — a `layers` past 64, a layer that is not the declared size, a layer hashing at another layer's seed index, mixed or 128-bit keys, a layer count that disagrees with the declared `layers` — and rule 7, so the format never emits bytes it would refuse to read back: a layer heap holding the same `NaN` twice (§3.7's producer-side constraint) fails to serialize. -### 3.16: Two-tier UnivMon payload (`0x11 0x00`) +### 3.16: Two-tier UnivMon payload (`0x11 0x01`; retired `0x11 0x00`) `UnivMonTwoTier` is UnivMon with two tiers: layers `0 .. elephant_layers` use `elephant_rows x elephant_cols` counters and the rest use `mouse_rows x mouse_cols`. Nothing else about it differs, so it **shares §3.15's payload byte for byte** and differs only in its metadata — the way HLL Classic and Ertl-MLE share one payload and differ only by `kind_id`: | Pos | Field | Type | Notes | | ----- | ------- | ------ | ------- | | 0 | `counts` | array | the layers' `CountSketchWithL2Norm` counters concatenated in layer order, each layer row-major; length is the layers' total cell count under the two-tier layout | -| 1 | `l2` | array | the layers' L2 accumulators concatenated in layer order; length `min(elephant_layers, layers) * elephant_rows + max(0, layers - elephant_layers) * mouse_rows` | -| 2 | `heap_lens` | array | u32 entries held by each layer's heap, one per layer | -| 3 | `keys` | array | every layer's heap keys concatenated, cut by `heap_lens`; element type = `key_type`; homogeneous | -| 4 | `heap_counts` | array | i64 heap count, parallel to `keys` | -| 5 | `candidate_complete` | array | one bool per layer | -| 6 | `bucket_size` | u64 | total weight recorded | -| 7 | `update_mode` | u8 | `0` unset, `1` standard, `2` terminal-only | +| 1 | `heap_lens` | array | u32 entries held by each layer's heap, one per layer | +| 2 | `keys` | array | every layer's heap keys concatenated, cut by `heap_lens`; element type = `key_type`; homogeneous | +| 3 | `heap_counts` | array | i64 heap count, parallel to `keys` | +| 4 | `candidate_complete` | array | one bool per layer | +| 5 | `bucket_size` | u64 | total weight recorded | +| 6 | `update_mode` | u8 | `0` unset, `1` standard, `2` terminal-only | + +The retired `0x11 0x00` is to this section what the retired `0x10 0x00` is to §3.15: the same metadata, with the layers' `l2` accumulators at payload position 1, read and dropped on decode and never encoded. **Metadata vs payload.** `UnivMonTwoTier::new`'s arguments are the configuration, so the descriptor carries `layers`, `elephant_layers`, `elephant_rows`, `elephant_cols`, `mouse_rows`, `mouse_cols` and `heap_size`, in that order, followed by the heaps' `key_type`. Layer `i`'s dimensions are the elephant pair while `i < elephant_layers` and the mouse pair after — **derived**, so no per-layer geometry is stored, and a pyramid whose every layer is an elephant falls out of the same rule with no special case. The hash-spec discussion, the absent seed-index key, the `L2HH` no-variant-tag rule and the treatment of `update_mode` / `candidate_complete` / `bucket_size` are all exactly §3.15's. @@ -1092,15 +1100,15 @@ The encode side enforces the same shape rules — a `layers` past 64, a layer th **Decode rules.** §3.15's rules, with the two-tier layout in place of the single geometry: -1. `kind_id` is `0x11 0x00`; any other id is rejected. +1. `kind_id` is `0x11 0x01`, or the retired `0x11 0x00`, which selects the eight-element layout; any other id is rejected. 2. The hash-spec group matches `DefaultXxHasher`'s `HashProfile`; the seven layout fields and `key_type` are echoed back rather than pinned. 3. `layers`, `heap_size` and **all four** tier dimensions — `elephant_rows`, `elephant_cols`, `mouse_rows`, `mouse_cols` — are non-zero, and `layers <= 19` (`UNIVMON_MAX_LAYERS`, §3.15's rule 3: the same seed ceiling, split across the doors the same way — the ceiling on both, the non-zero checks on the decode door). All of it is checked before the per-layer geometry is built. **A tier no layer materializes must still declare non-zero dimensions.** An all-elephant pyramid carries a `mouse_rows` / `mouse_cols` pair on the wire all the same (see "One state, more than one encoding"), and a pyramid decoded with a zero pair panics the first time it is merged against a normally built one, whose constructor asserts all four are positive. `elephant_layers` carries no bound of its own; it only ever reaches the layout through `min(elephant_layers, layers)`. -4. The declared layout's accumulator count — `min(elephant_layers, layers) * elephant_rows + max(0, layers - elephant_layers) * mouse_rows` — equals `len(l2)` exactly, computed with checked arithmetic and checked **before** the per-layer geometry is built. With rule 3's ceiling above it, that is what makes a crafted `layers` or tier width cost nothing. -5. §3.15's rules 4, 6, 7, 8, 9, 10 and 11 apply verbatim over the derived per-layer geometry — including the per-layer `l2[i] >= 0` check and the per-layer 128-bit hash-budget check (§3.19). +4. Rule 3's ceiling caps the derived per-layer geometry before it is built, and §3.15's rule 6 measures its total cell count against `len(counts)` before any matrix is, so a crafted `layers` or tier width costs nothing. +5. §3.15's rules 4, 6, 7, 8, 9, 10 and 11 apply verbatim over the derived per-layer geometry — including the recomputed per-layer L2 accumulators and the per-layer 128-bit hash-budget check (§3.19). The encode side enforces §3.15's shape rules and rule 7 over the two-tier layout, the `layers <= 19` ceiling included. -**`UnivMonPool` is not a wire kind.** It is a `Vec` free list of pre-allocated scratch `UnivMon`s — `0x10 0x00`'s type, not this section's — beside a `total_allocated` counter: an allocator, not a sketch, with nothing an observer could query. `0x11 0x00` is `UnivMonTwoTier`'s alone. +**`UnivMonPool` is not a wire kind.** It is a `Vec` free list of pre-allocated scratch `UnivMon`s — `0x10 0x01`'s type, not this section's — beside a `total_allocated` counter: an allocator, not a sketch, with nothing an observer could query. `0x11 0x01` is `UnivMonTwoTier`'s alone. ### 3.17: ExponentialHistogram payload (`0x13 0x00`) @@ -1179,23 +1187,25 @@ The `kind_id` is carried rather than a name, because Section 1's registry is alr | `0x0b 0x00` | Elastic | `ElasticSketch(ElasticSketch)` | always | | `0x0c 0x00` | Coco | `CocoSketch(CocoSketch)` | always | | `0x0d 0x00` | UniformSampling | `Uniform(UniformSampling)` | `experimental` | -| `0x10 0x00` | UnivMon | `UnivMon(UnivMon)` | always | -| `0x19 0x00` | CountSketchWithL2Norm | `CountSketchWithL2Norm(CountSketchWithL2Norm)` | always | +| `0x10 0x01` | UnivMon | `UnivMon(UnivMon)` | always | +| `0x19 0x01` | CountSketchWithL2Norm | `CountSketchWithL2Norm(CountSketchWithL2Norm)` | always | + +The retired `0x10 0x00` and `0x19 0x00` (§3.15, §3.19), which also carried `l2`, still decode into the `UnivMon` and `CountSketchWithL2Norm` arms through those variants' own decoders, `l2` dropped and recomputed; the encoder writes only `0x10 0x01` and `0x19 0x01`. They are the same arms, not further entries among the ten. **Each id names one algorithm, and its siblings are not it.** `0x01 0x01`, `0x01 0x03` and `0x06 0x01` are rejected as unknown here: the arms hold `HyperLogLog` and the compact `Kll` and nothing else. `CountMinSketch` and `CountSketch` both hold `Vector2D` on the `FastPath`, so their descriptors carry `counter_type = "i32"` and `mode = "fast"`; `i32` is never widened, since a nested sketch must decode back into the arm it was stored as (Q-CS). **Feature-gating contract (cross-language; Go must honour it).** -1. The nested-id namespace is **fixed and identical in every build**. All eleven ids (the ten encoded ones plus the retired `0x05 0x00`, which decodes into the DDSketch arm) dispatch whether or not the `experimental` feature is on. An id is registry bytes, never an enum ordinal and never a discriminant, so no id's meaning can shift because a variant is compiled out. +1. The nested-id namespace is **fixed and identical in every build**. All thirteen ids (the ten encoded ones plus the retired `0x05 0x00`, `0x10 0x00` and `0x19 0x00`, which decode into the DDSketch, UnivMon and CountSketchWithL2Norm arms) dispatch whether or not the `experimental` feature is on. An id is registry bytes, never an enum ordinal and never a discriminant, so no id's meaning can shift because a variant is compiled out. 2. A decoder built **without** `experimental` that meets `0x0d 0x00` **fails closed**, before the blocks are assembled into anything, with an error naming the variant (`UniformSampling`) **and** the feature. It never misparses, skips, substitutes another variant, or falls through the unknown-id path. 3. An encoder built **without** `experimental` can never emit that id: the enum arm does not exist. -4. An id outside the eleven is rejected as unknown. +4. An id outside the thirteen is rejected as unknown. 5. The id-to-name lookup exists for **error messages only**. Encoding and decoding both dispatch on the bytes; the name influences neither. **Decode rules.** Fail **closed** on each, with an error and never a panic: 1. `kind_id` is `0x14 0x00`; any other id is rejected. The metadata is `metadata_version = 1` and nothing else; an unknown key or a missing key is rejected. -2. The nested `kind_id` is one of the eleven in the table (ten encoded, plus the retired `0x05 0x00`); anything else, **including a sibling algorithm's id within the same family**, is rejected as unknown — and rejected **before** the triple is assembled into an envelope. Nothing is ever built from an id the table does not carry, so an id too long for the envelope's one-byte `kind_id_len` field never reaches the framing that would have to hold it: it draws the same unknown-id error as any other stranger. +2. The nested `kind_id` is one of the ten encoded ids or the three retired ones (`0x05 0x00` in the table, and `0x10 0x00` / `0x19 0x00` above); anything else, **including a sibling algorithm's id within the same family**, is rejected as unknown — and rejected **before** the triple is assembled into an envelope. Nothing is ever built from an id the table does not carry, so an id too long for the envelope's one-byte `kind_id_len` field never reaches the framing that would have to hold it: it draws the same unknown-id error as any other stranger. 3. A nested id this build does not carry is rejected before `descriptor` and `state` are assembled into anything, with an error naming the variant and the feature. The availability check runs **first**, ahead of rule 2's: a feature-gated id is a known id, so `0x0d 0x00` in a build without `experimental` always draws the feature error and never the unknown-id one. 4. `descriptor` and `state` are handed to that variant's own decoder, wrapped in the variant's own `kind_id`. Every check that decoder makes applies here: geometry bounds, length agreement, allocation guards, and the hash-profile pin. 5. A `descriptor` / `state` pair that does not belong to the declared id fails inside that variant's decoder; no partial state is produced. @@ -1203,37 +1213,38 @@ The `kind_id` is carried rather than a name, because Section 1's registry is alr The encode side re-splits the bytes the variant produced and rejects an id that is not the arm's own, so the tag and the blocks can never disagree. -### 3.19: Count-Sketch-with-L2-norm payload (`0x19 0x00`) +### 3.19: Count-Sketch-with-L2-norm payload (`0x19 0x01`; retired `0x19 0x00`) -`CountSketchWithL2Norm` is a Count Sketch whose rows also carry a running L2 accumulator. Its counters are always `i64` and its column derivation is fixed by the algorithm, so — unlike Count-Min (§3.2) and Count Sketch (§3.6) — its metadata carries neither a `counter_type` nor a `mode`: `kind_id` already determines both. What is left is the geometry and the seed-list index it hashes with: +`CountSketchWithL2Norm` is a Count Sketch whose rows also keep a running L2 accumulator in memory. Its counters are always `i64` and its column derivation is fixed by the algorithm, so — unlike Count-Min (§3.2) and Count Sketch (§3.6) — its metadata carries neither a `counter_type` nor a `mode`: `kind_id` already determines both. What is left is the geometry and the seed-list index it hashes with: | Pos | Field | Type | Notes | | ----- | ------- | ------ | ------- | | 0 | `counts` | array | packed **row-major**, `rows*cols` cells; `i64`, **signed** | -| 1 | `l2` | array | one `i64` accumulator per row, length `rows`; every entry `>= 0` | + +The retired `0x19 0x00` has the same metadata and the payload `[counts, l2]`, `l2` one `i64` accumulator per row. A decoder reads it, drops `l2` unchecked, and recomputes; it is never encoded. **Metadata vs payload.** `rows`, `cols` and `seed_index` are chosen at construction (`CountSketchWithL2Norm::with_dimensions_and_seed_index`), so per the config-to-metadata rule they live in the descriptor and the payload omits them. Structural-param order is `seed_index, rows, cols`, mirroring Count-Min's `matrix_seed_index, rows, cols, ...`. `seed_index` is **not** a profile constant. Count-Min's `matrix_seed_index` is `H::MATRIX_SEED_INDEX`, read off the hasher; a CountSketchWithL2Norm's seed offset is per-instance state a caller chose, and UnivMon gives layer `i` the value `i`. So it is a **structural param echoed back on decode**, not a hash-spec field pinned against the target — the same treatment `rows` / `cols` get. -`l2` is carried rather than recomputed. It is *not* `sum(counts[row]^2)`: `update_weighted_with_hash_without_l2` moves counters without touching it, and the accumulator clamps to `[0, i64::MAX]` one way, so a decoder that recomputed it would report a different `l2` than the sketch it read. +**`l2` is recomputed, not carried.** In memory each row keeps an exact `i128` sum of its counters' squares, each square capped at `2^63`; every write removes the old counter's term and adds the new one's in O(1), and `merge` recomputes the sum from the counters. Read clamped at `i64::MAX` it is `min(sum(counts[row]^2), i64::MAX)`: a capped term alone already reaches `i64::MAX`, so the cap changes nothing once clamped, and a decrement after saturation lands on the true value. No path moves counters without the accumulator. So the decoder recomputes each row's accumulator from `counts`, and the encoder rejects a cached value that disagrees (Q-DERIVED). -**The shared sub-payload.** Positions 0 and 1 are the CountSketchWithL2Norm sub-payload. When a CountSketchWithL2Norm is a UnivMon layer it is **inlined**, not wrapped: the layer's `counts` and `l2` are appended to the outer sketch's own positional arrays, and its `rows` / `cols` / `seed_index` are derived from the outer metadata plus the layer's position. +**The shared sub-payload.** Position 0 is the CountSketchWithL2Norm sub-payload. When a CountSketchWithL2Norm is a UnivMon layer it is **inlined**, not wrapped: the layer's `counts` are appended to the outer sketch's own positional arrays, and its `rows` / `cols` / `seed_index` are derived from the outer metadata plus the layer's position. -**Emitted order (cross-language contract).** `counts` is row-major, row 0 first; `l2` is in row order. There is one encoding per state, so a decoded sketch re-serializes byte-identically, and an empty sketch has exactly one encoding. +**Emitted order (cross-language contract).** `counts` is row-major, row 0 first. There is one encoding per state, so a decoded sketch re-serializes byte-identically, and an empty sketch has exactly one encoding. **Decode rules.** Fail **closed** on each, with an error and never a panic: -1. `kind_id` is `0x19 0x00`; any other id is rejected. +1. `kind_id` is `0x19 0x01`, or the retired `0x19 0x00`, which selects the `[counts, l2]` layout; any other id is rejected. 2. The hash-spec group matches the **target hasher's** `HashProfile`. `seed_index`, `rows` and `cols` are properties of the *stored* sketch, so they are echoed back into the expected metadata rather than pinned. 3. `rows` and `cols` are both non-zero and `rows <= 20` (`MATRIX_MAX_ROWS`, §3.2). Checked before the matrix is built: the column mask is derived from `cols.ilog2()`, which panics on `cols == 0`. 4. The geometry fits one key hash: `(ceil_log2(cols) + 1) * rows <= 128`. CountSketchWithL2Norm never draws per row — one `hash128_seeded(seed_index, key)` supplies every row, `ceil_log2(cols)` column bits at offset `r * ceil_log2(cols)` and one sign bit at `127 - r` — so a geometry needing more bits than the hash carries has no valid cell mapping at all (§3.2). 5. `len(counts) == rows * cols` exactly, checked **before** the allocation, so crafted dimensions cannot drive a huge reserve. -6. `len(l2) == rows`. -7. Every `l2[i] >= 0` — a negative accumulator is not a state a sum of squares reaches. This is the whole of what `l2` is checked for, and it is the **strongest sound predicate** there is: `l2` is deliberately *not* cross-checked against `counts`, in either direction, because no sound relation holds. `update_weighted_with_hash_without_l2` moves counters without touching the accumulator, so a run of those on a fresh sketch leaves `l2` **below** `sum(counts[row]^2)`; the same call used to decrement after ordinary inserts leaves it **above**; and the accumulator clamps to `[0, i64::MAX]`, which pushes it below the true sum when a run would overflow and above it when a run would go negative. So neither `l2[i] <= sum(counts[i]^2)` nor its converse describes a reachable-state boundary, and a decoder that enforced either would reject sketches the algorithm produces. Non-negativity is what survives, and it is what both doors check. +6. (Retired `0x19 0x00` only.) `l2` is read as an array of `i64` and dropped; its length and values are not checked, since nothing is built from it. +7. Each row's L2 accumulator is recomputed as `min(sum(counts[row]^2), i64::MAX)`. On the encode side, a cached accumulator that differs from this value is rejected. 8. `seed_index` is **less than the declared `seed_list`'s length**. This is the stand-alone sketch's rule, on both doors. It does not reach an **inlined** UnivMon layer, whose index is its own position (§3.15) and is checked against that instead. -Rules 3 to 8 are enforced on the **encode** side too — a matrix whose cell count disagrees with its own dimensions fails to serialize, as Count Sketch's does, and so do a geometry that outruns the 128-bit hash budget and a seed index past the hasher's seed list — so the format never emits bytes it would refuse to read back. +Rules 3, 4, 5, 7 and 8 are enforced on the **encode** side too — a matrix whose cell count disagrees with its own dimensions fails to serialize, as Count Sketch's does, and so do a geometry that outruns the 128-bit hash budget and a seed index past the hasher's seed list — so the format never emits bytes it would refuse to read back. ### 3.20: UnivMon-Q payload (`0x1a 0x00`) @@ -1535,7 +1546,7 @@ A `"f32"` key is float32 (`0xca`); a decoder rejects any other type there (§3.5 - `light_counts` (§3.11) is Elastic's inlined light Count-Min, `i32` integers, fixed by `light_counter_type`. - `deltas` / `negative_deltas` (§3.23) take the element type of the cells they add to. -*Other arrays, by name.* `words` (§3.4) is Bloom's bit grid as unsigned 64-bit integers, one per packed word. `hashes` (§3.14) is KMV's retained digests as unsigned integers — a digest above `2^63` is still `uint`, never the `int` family. `l2` (§3.19, §3.15, §3.16) is one `i64` accumulator per row. `indices` (§3.23) are unsigned integers, except DDSketch's `indices` / `negative_indices`, which are signed bucket indices. `heap_counts` is `i64`, or float64 in §3.7, §3.10 and §3.23 when `counter_type` reads `"f64"`. Space-Saving's `counts` and `errors` (§3.5) and the `summary_counts` and `errors` of §3.24 and §3.25 are `u64`. `levels` and `coin` (§3.3) are integer arrays; `heap_lens` and `candidate_lens` are `u32` counts; `sizes` / `min_times` / `max_times` (§3.17), `priorities` (§3.13), `values` (§3.12), `candidate_keys` / `candidate_scores` / `occurrence_keys` / `occurrence_priority_high` / `occurrence_priority_low` (§3.20) are all unsigned integers; `vote_pos` / `vote_neg` (§3.11) are `i32`. +*Other arrays, by name.* `words` (§3.4) is Bloom's bit grid as unsigned 64-bit integers, one per packed word. `hashes` (§3.14) is KMV's retained digests as unsigned integers — a digest above `2^63` is still `uint`, never the `int` family. `l2` in the retired `0x19 0x00`, `0x10 0x00`, `0x11 0x00` and `0x07 0x04` payloads (§3.19, §3.15, §3.16, §3.9) is one `i64` accumulator per row. `indices` (§3.23) are unsigned integers, except DDSketch's `indices` / `negative_indices`, which are signed bucket indices. `heap_counts` is `i64`, or float64 in §3.7, §3.10 and §3.23 when `counter_type` reads `"f64"`. Space-Saving's `counts` and `errors` (§3.5) and the `summary_counts` and `errors` of §3.24 and §3.25 are `u64`. `levels` and `coin` (§3.3) are integer arrays; `heap_lens` and `candidate_lens` are `u32` counts; `sizes` / `min_times` / `max_times` (§3.17), `priorities` (§3.13), `values` (§3.12), `candidate_keys` / `candidate_scores` / `occurrence_keys` / `occurrence_priority_high` / `occurrence_priority_low` (§3.20) are all unsigned integers; `vote_pos` / `vote_neg` (§3.11) are `i32`. *Key and item arrays.* `keys` and `items` take the element type the metadata names — `key_type` for the `OwnedDataInput` keys of §3.5, §3.7, §3.10, §3.15, §3.16, §3.23, §3.24, §3.25 and Hydra's `cell_key_type` (§3.9), per §3.5's table; `item_type` for KLL's `items` (§3.3) and UniformSampling's `values` (§3.13). Four payloads name no such key and are fixed by their kind_id instead: Coco's `keys` (§3.12) and Elastic's `flow_ids` (§3.11) are nil-or-`str`, SetAggregator's `values` (§3.21) and SetAggregatorDelta's `added` / `removed` (§3.22) are `str`, and Coco's `values` are unsigned integers. @@ -1618,14 +1629,14 @@ Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketc - **Q-NEST**: a nested sketch's state is **inlined**, never wrapped in an envelope of its own. By the time a decoder reaches the payload it has read `kind_id` and the metadata, and together those fix the shape completely, so a nested envelope would repeat a magic sentinel, a version, an id, two length prefixes and a metadata map the outer one already carries — and would create a second place for the same fact to live, which a decoder would then have to cross-check. This governs CountMinSketchWithHeap's and CountSketchWithHeap's base matrix (§3.7, §3.10), CountMinSketchWithSpaceSaving's and CountSketchWithSpaceSaving's matrix and summary (§3.24, §3.25), Elastic's light Count-Min (§3.11), Hydra's counter cells (§3.9) and UnivMon's `CountSketchWithL2Norm` layers (§3.15). - **Q-NEST-HET**: a **heterogeneous** nested sketch — one whose algorithm is data rather than a type — carries the variant's own registry `kind_id` plus its metadata and payload blocks with the framing stripped (§3.18). No name string, no second identity namespace beside the Section 1 registry, and no per-variant encoding to specify: the blocks are byte for byte what that variant's own section fixes, and the variant's own decoder runs on them unchanged, so its geometry checks, allocation guards and hash-profile pin all apply inside the wrapper. `EhBucketSketch` (`0x14 0x00`) is the triple, and `ExponentialHistogram` (`0x13 0x00`) inlines one per bucket. - **Q-NEST-GATE**: the nested-id namespace is **fixed and identical in every build**. An id is registry bytes, never an enum ordinal, so no id's meaning shifts because a variant is compiled out. A decoder that meets an id its build does not carry fails **closed**, before the blocks are assembled, with an error naming the variant and the feature; it never misparses, skips, substitutes, or falls through the unknown-id path. -- **Q-HYDRA-IDS**: Hydra allocates **one `kind_id` per counter base sketch** (`0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, `0x07 0x04` UnivMon), so the payload carries no variant tag and a grid mixing counter variants has no encoding. `0x07 0x05`-`0x07 0x7f` stay reserved for Hydra over further base sketches. The five ids use four metadata schemas; Count-Min and Count Sketch share one, as the two KLL ids share one. +- **Q-HYDRA-IDS**: Hydra allocates **one `kind_id` per counter base sketch** (`0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, `0x07 0x05` UnivMon), so the payload carries no variant tag and a grid mixing counter variants has no encoding. A counter whose cell payload changes takes the next free variant (Q-VER): `0x07 0x04`, the UnivMon counter whose cells carried `l2`, is retired and decode-only. `0x07 0x06`-`0x07 0x7f` stay reserved for Hydra over further base sketches. The five ids use four metadata schemas; Count-Min and Count Sketch share one, as the two KLL ids share one. - **Q-NOHASH**: a sketch that never hashes its inputs omits the **hash-spec group entirely** — the group answers "how were keys hashed", and there is no truthful answer. This is Q-KLL generalized: KLL (§3.3), DDSketch (§3.8) and UniformSampling (§3.13) never hash, `ExponentialHistogram` (§3.17) and `EhBucketSketch` (§3.18) do not hash either, since the sketch inside them does, and `SetAggregator` (§3.21) and `SetAggregatorDelta` (§3.22) carry their members verbatim. Their metadata is structural-only and is not `HashProfile`-derived. A sketch that *does* hash carries the group, derived live from the hasher's profile, even when its counters do not hash (Hydra's KLL and UnivMon variants, §3.9). - **Q-SEEDIDX**: a **seed-index key is carried only for a hash whose index the sketch reads off the profile.** `HashProfile` declares two such constants, `CANONICAL_SEED_INDEX` and `MATRIX_SEED_INDEX`, and the metadata carries `canonical_seed_index` or `matrix_seed_index` accordingly. An index the algorithm fixes gets no key: Space-Saving's `0` (§3.5, and the summaries of §3.24 and §3.25), Coco's per-array `i` (§3.12), Elastic's heavy table (§3.11), Hydra's subkey fan-out at `HYDRA_SEED_INDEX` (§3.9) and the UnivMon family's `UNIVMON_BOTTOM_LAYER_SEED_INDEX` (§3.15). A per-instance seed a caller chose is a **structural param**, not a hash-spec field: `seed_index` on CountSketchWithL2Norm (§3.19) and UnivMon-Q (§3.20) is echoed back on decode rather than pinned against the target. - **Q-ORDER**: a payload whose in-memory container order does not survive a rebuild **pins an emitted order**, as a cross-language contract. Heaps, arenas and hash maps all have this property, so the rule reaches Space-Saving's triples (§3.5, §3.24, §3.25), the top-k heap entries (§3.7), KMV's digests (§3.14), UniformSampling's entries (§3.13), UnivMon-Q's candidates and occurrences (§3.20), and every layer- and bucket-indexed array. Two sketches holding the same state then emit the same bytes whatever order they were built in, and a decoded sketch re-serializes byte-identically. The heap key comparator is the same everywhere: descending count, ties by heap key order (§3.5). - **Q-CAP**: **a declared capacity never sizes an allocation.** Every structure is built from the run the payload actually carries, and the declared bound is checked against that run first; a payload naming a capacity of `2^32 - 1` with two entries costs two entries. This holds for every `capacity` (§3.5, §3.24, §3.25), every `heap_size`, KMV's `k`, UniformSampling's `total_seen`, UnivMon-Q's `ordered_samples`, and Hydra's grid and counter dimensions. - **Q-KEYTYPE**: `key_type` names the **exact `OwnedDataInput` variant** and is never widened, wherever heap keys reach the wire — Space-Saving (§3.5), CountMinSketchWithHeap and CountSketchWithHeap (§3.7, §3.10), CountMinSketchWithSpaceSaving and CountSketchWithSpaceSaving (§3.24, §3.25), the UnivMon family (§3.15, §3.16) and Hydra's UnivMon counter (§3.9). A container whose keys mix variants has no single name and **fails to serialize**; `OwnedDataInput::I128` / `U128` have no msgpack integer form and are not wire types. An empty container emits `"u64"`, and a decoder rejects an empty container under any other name, so its `key_type` has one value rather than one per producer. That canonicalizes the key naming and nothing else: a Two-tier UnivMon pyramid still has more than one encoding per state, since layouts differing only in an `elephant_layers` at or above `layers` build identical layers, and an all-elephant pyramid carries its unused mouse dimensions on the wire regardless (§3.16). What holds everywhere is the round trip: a decoded container re-serializes byte-identically. -- **Q-DERIVED**: state a decoder can recompute exactly is **not carried**, and state it cannot is. Recomputed: DDSketch's `count` from its stores and `zero_count` (§3.8), `ExponentialHistogram`'s `l2_mass` and `merge_norm` (§3.17), Elastic's `heavy_buckets` from the declared bucket count (§3.11), every heap's digest index and array order, and every summary a sketch answers from its own state. Carried: Elastic's `stale_copies`, UnivMon's `update_mode`, `candidate_complete` and `bucket_size`, UnivMon-Q's `ever_evicted`, `source_id` and `next_sequence`, UniformSampling's `priorities`, `total_seen` and `rng_state`, and CountSketchWithL2Norm's `l2`. Where a cached derived field exists in memory, the encoder rejects a value that disagrees with what the decoder would recompute. State the sketch's own readouts do not need is not carried either: DDSketch's `sum` / `min` / `max` were (`0x05 0x00`) and are now a separate exact aggregate, since nothing a DDSketch answers reads them (§3.8), and so was Bloom's `inserted` counter (`0x17 0x00`), now a separate exact count, since no membership answer reads it (§3.4). Space-Saving's `total` was carried too (`0x18 0x00`, and inside `0x1e 0x00` / `0x1f 0x00`); no Space-Saving readout reads it, so it is no longer sketch state, and a caller that needs it keeps a separate exact count (§3.5). +- **Q-DERIVED**: state a decoder can recompute exactly is **not carried**, and state it cannot is. Recomputed: DDSketch's `count` from its stores and `zero_count` (§3.8), CountSketchWithL2Norm's and every UnivMon layer's `l2` from its counters (§3.19; the retired `0x19 0x00`, `0x10 0x00`, `0x11 0x00` and `0x07 0x04` carried it), `ExponentialHistogram`'s `l2_mass` and `merge_norm` (§3.17), Elastic's `heavy_buckets` from the declared bucket count (§3.11), every heap's digest index and array order, and every summary a sketch answers from its own state. Carried: Elastic's `stale_copies`, UnivMon's `update_mode`, `candidate_complete` and `bucket_size`, UnivMon-Q's `ever_evicted`, `source_id` and `next_sequence`, and UniformSampling's `priorities`, `total_seen` and `rng_state`. Where a cached derived field exists in memory, the encoder rejects a value that disagrees with what the decoder would recompute. State the sketch's own readouts do not need is not carried either: DDSketch's `sum` / `min` / `max` were (`0x05 0x00`) and are now a separate exact aggregate, since nothing a DDSketch answers reads them (§3.8), and so was Bloom's `inserted` counter (`0x17 0x00`), now a separate exact count, since no membership answer reads it (§3.4). Space-Saving's `total` was carried too (`0x18 0x00`, and inside `0x1e 0x00` / `0x1f 0x00`); no Space-Saving readout reads it, so it is no longer sketch state, and a caller that needs it keeps a separate exact count (§3.5). - **Q-NIL**: an absent slot is msgpack **nil** (`0xc0`), never a sentinel value. An empty string is a reachable key in Coco (§3.12) and a reachable flow id in Elastic (§3.11), so `""` cannot double as "free"; UnivMon-Q's absent extrema are nil for the same reason (§3.20). Go must model these as nullable, never as a zero value standing in for absent. - **Q-RNG**: RNG state is carried **resumable** when the generator is cheap to reproduce in another language, and opaque otherwise. UniformSampling's `rng_state` is one SplitMix64 word with published constants, so a decoded sampler continues the same draw sequence (§3.13); KLL's `coin` carries the compaction RNG's position and its `compaction_seed` is opaque state Go preserves without interpreting (Q-KLL-SEED). -- **Q-UNIVMON-SCOPE**: every layer of `0x10 0x00` and `0x11 0x00` is a CountSketchWithL2Norm with no tag, and another per-layer counter needs a new structural param or a new id. `UnivMonPool` is a `Vec` free list of scratch `UnivMon`s — not of pyramids — with nothing an observer could query, so it is not a wire kind and holds no id. +- **Q-UNIVMON-SCOPE**: every layer of `0x10 0x01` and `0x11 0x01` (and of the retired `0x10 0x00` / `0x11 0x00`) is a CountSketchWithL2Norm with no tag, and another per-layer counter needs a new structural param or a new id. `UnivMonPool` is a `Vec` free list of scratch `UnivMon`s — not of pyramids — with nothing an observer could query, so it is not a wire kind and holds no id. - **Encoding**: metadata + payload are both msgpack; payload is a positional array. Byte-level rules in Section 4. diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 143df1fb..ecdda52f 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -76,8 +76,9 @@ beside `wire.rs`, declared `pub(crate)` so the other user can import it by path: - `src/sketches/countminsketch_heap/heap_wire.rs` — the top-k heap metadata schema, payload, `key_type` mapping, emitted order and heap rebuild that CountMinSketchWithHeap (`0x03 0x00`) and CountSketchWithHeap (`0x0a 0x00`) share. -- `src/sketches/countsketch_l2_norm/wire.rs` — CountSketchWithL2Norm (`0x19 0x00`), plus - the `[counts, l2]` layer sub-payload the whole UnivMon family inlines. +- `src/sketches/countsketch_l2_norm/wire.rs` — CountSketchWithL2Norm (`0x19 0x01`), plus + the `[counts]` layer sub-payload the whole UnivMon family inlines; the + retired `0x19 0x00` (`[counts, l2]`) still decodes, its `l2` dropped. Converted: @@ -128,13 +129,15 @@ Converted: hash-spec group. - **KMV** (`src/sketches/kmv/wire.rs`, `0x0e 0x00`) — the retained digests, strictly ascending. -- **CountSketchWithL2Norm** (`src/sketches/countsketch_l2_norm/wire.rs`, `0x19 0x00`) — - `[counts, l2]`. -- **Hydra** (`src/sketch_framework/hydra/wire.rs`, `0x07 0x00`-`0x07 0x04`) — one +- **CountSketchWithL2Norm** (`src/sketches/countsketch_l2_norm/wire.rs`, `0x19 0x01`) — + `[counts]`; each row's L2 accumulator is recomputed on decode. +- **Hydra** (`src/sketch_framework/hydra/wire.rs`, `0x07 0x00`-`0x07 0x03`, `0x07 0x05`) — one `kind_id` per counter variant; the cells' state tiles or nests one array. -- **UnivMon** (`src/sketch_framework/univmon/wire.rs`, `0x10 0x00`) and +- **UnivMon** (`src/sketch_framework/univmon/wire.rs`, `0x10 0x01`) and **Two-tier UnivMon** (`src/sketch_framework/univmon_two_tier/wire.rs`, - `0x11 0x00`) — one payload shape, two metadata schemas. + `0x11 0x01`) — one payload shape, two metadata schemas. The retired + `0x10 0x00`, `0x11 0x00` and `0x07 0x04` also carried the layers' `l2` + and still decode, `l2` dropped and recomputed. - **UnivMon-Q** (`src/sketch_framework/univmon_q/wire.rs`, `0x1a 0x00`) — the levels' counters, candidates, extrema and coordinated occurrence sample. - **ExponentialHistogram** (`src/sketch_framework/eh/wire.rs`, `0x13 0x00`) and diff --git a/docs/tests.md b/docs/tests.md index ba410558..66258e1b 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -394,18 +394,23 @@ Wire tests: [`src/sketches/countsketch_l2_norm/wire.rs`](../src/sketches/countsk | `count_sketch_with_l2_norm_estimates_and_l2_are_consistent` | CountSketchWithL2Norm updates keep estimate and L2 consistent. | For `CountSketchWithL2Norm(3x32)`, applies `+5` then `-2` to one key, verifies estimates `5.0` then `3.0`, and asserts non-trivial L2 (`>= 3.0`). | | `count_sketch_with_l2_norm_merge_combines_frequency_vectors` | CountSketchWithL2Norm merge combines per-key frequencies. | Merges two `CountSketchWithL2Norm(3x32)` sketches with same key counts `4` and `9`, then verifies merged estimate `== 13.0`. | | `count_sketch_with_l2_norm_round_trip_preserves_estimate_and_l2` | CountSketchWithL2Norm serialization round trip preserves estimate and L2. | Serializes/deserializes `CountSketchWithL2Norm::with_dimensions_and_seed_index(3,32,7)` after updates, verifying rows/cols and that both estimate and L2 remain unchanged (within `f64::EPSILON`). | -| `count_sketch_with_l2_norm_round_trip_serialization` | The envelope frames a sketch under kind_id `0x19 0x00`, and the state survives a round trip. | For a `3x32` sketch seeded at `7` over three signed weighted inserts, verifies the bytes open with the ASAPv1 magic, a `kind_id_len` of `2`, and `0x19 0x00`, that the metadata carries `seed_index` `7` and `3x32`, and that the decode matches on dimensions, seed index, the counter array, the `l2` accumulators, and `l2`. | +| `count_sketch_with_l2_norm_round_trip_serialization` | The envelope frames a sketch under kind_id `0x19 0x01`, and the state survives a round trip. | For a `3x32` sketch seeded at `7` over three signed weighted inserts, verifies the bytes open with the ASAPv1 magic, a `kind_id_len` of `2`, and `0x19 0x01`, that the metadata carries `seed_index` `7` and `3x32`, and that the decode matches on dimensions, seed index, the counter array, the recomputed `l2` accumulators, and `l2`. | | `count_sketch_with_l2_norm_decoded_re_serializes_identically` | A decoded sketch re-serializes byte for byte. | Verifies the populated `3x32` sketch's decode re-encodes to the bytes it came from. | | `count_sketch_with_l2_norm_negative_counters_round_trip` | Signed cells reach the wire and come back unchanged. | Round-trips a `2x8` sketch fed two negative weights and verifies the decoded slice equals the source, still holds a negative cell, and re-serializes identically. | | `count_sketch_with_l2_norm_seed_index_travels_with_the_sketch` | The seed index is state, not a profile constant. | Verifies `2x8` sketches seeded at `0` and at `9` do not share bytes, and that the decoded seed-`9` sketch reports a `seed_index` of `9`. | | `count_sketch_with_l2_norm_custom_hasher_profile_round_trips_and_is_self_describing` | The metadata describes the hasher that built the sketch rather than a hardcoded profile. | For a `3x32` sketch over a hasher declaring its own `HashProfile`, verifies the counter array round-trips, the bytes differ from the standard-profile sketch's over the same insert, and a standard-profile decode rejects them. | | `count_sketch_with_l2_norm_rejects_foreign_kind_ids` | The neighbouring universal-sketch kind_ids are refused. | Verifies `CountSketch`, `UnivMon`, `UnivMonTwoTier`, and `UnivMonQ` envelopes each fail to decode as a `CountSketchWithL2Norm`. | -| `count_sketch_with_l2_norm_rejects_crafted_geometry` | Every geometry rule fires before an allocation is sized from it. | Verifies crafted envelopes carrying a zero `cols`, a `MATRIX_MAX_ROWS x 2^24` declaration against three counters, a five-entry `l2` array against two rows, and a negative `l2` accumulator each fail to decode. | +| `count_sketch_with_l2_norm_rejects_crafted_geometry` | Every geometry rule fires before an allocation is sized from it. | Verifies crafted envelopes carrying a zero `cols` and a `MATRIX_MAX_ROWS x 2^24` declaration against three counters each fail to decode. | | `count_sketch_with_l2_norm_rejects_serializing_an_unfilled_matrix` | The encode side refuses a matrix its own decoder would reject. | Verifies a `2x4` sketch whose counts are replaced by `Vector2D::unfilled(2, 4)`, which reserves eight cells without filling them, fails to serialize. | | `count_sketch_with_l2_norm_metadata_rejects_unknown_and_missing_keys` | An unexpected metadata key and a missing required one both fail closed. | Encodes the nine `CountSketchWithL2NormMetadata` fields plus a `bogus_field`, and the same fields with `cols` omitted, and verifies neither decodes as `CountSketchWithL2NormMetadata`. | | `count_sketch_with_l2_norm_empty_has_one_encoding` | A sketch holding nothing has exactly one encoding. | Verifies a fresh `3x32` sketch and one cleared after an insert serialize to identical bytes. | | `count_sketch_with_l2_norm_rejects_too_many_rows` | CountSketchWithL2Norm carries the seed list's row bound on both sides. | Verifies a sketch past `MATRIX_MAX_ROWS` fails to serialize, that a crafted envelope of that geometry fails to decode with a message naming `MATRIX_MAX_ROWS`, and that the boundary row count still serializes. | -| `known_state_matches_golden` | Known counts, L2 accumulators and seed index serialize to the golden bytes. | Builds a `2x4` sketch with counts `[[127, 128, 65535, -32768], [-32, -33, i32::MIN, i64::MIN]]`, row L2 `[65536, i64::MAX]` and seed index `7`, and verifies `to_asapv1` equals `asapv1_golden/count_l2hh_2x4_seed7.hex`. | +| `known_state_matches_golden` | Known counts and seed index serialize to the golden bytes, and the retired fixture decodes to them. | Builds a `2x4` sketch with counts `[[127, 128, 65535, -32768], [-32, -33, i32::MIN, i64::MIN]]`, row L2 `[5368610562, i64::MAX]` (the rows' sums of squares) and seed index `7`, and verifies `to_asapv1` equals `asapv1_golden/count_l2hh_2x4_seed7_no_l2.hex`; then decodes the retired `count_l2hh_2x4_seed7.hex` (`l2 = [65536, i64::MAX]`) and verifies the recomputed `l2` and the re-encoding equal the new fixture. | +| `count_sketch_with_l2_norm_payload_carries_no_l2` | The payload is `[counts]` alone. | Verifies the populated sketch's payload is a one-element msgpack array. | +| `count_sketch_with_l2_norm_retired_kind_decodes_and_recomputes_l2` | The retired `0x19 0x00` decodes, its `l2` dropped and recomputed. | Re-wraps the populated sketch as a `0x19 0x00` `[counts, l2]` payload with its true `l2`, a wrong-length `l2` and a negative one, and verifies each decodes to the same counters and accumulators and re-encodes as the `0x19 0x01` bytes. | +| `count_sketch_with_l2_norm_rejects_a_stale_l2_cache` | The encoder rejects a cached accumulator the decoder would not recompute. | Bumps row 0's cached accumulator by one and verifies `to_asapv1` fails naming row 0. | +| `count_sketch_with_l2_norm_retired_kind_rejects_a_non_integer_l2` | A retired `l2` that is not an integer array is rejected although it is dropped. | Crafts a `0x19 0x00` payload whose `l2` is three strings and verifies decoding fails. | +| `count_sketch_with_l2_norm_l2_stays_derivable_past_saturation` | A saturated row's accumulator returns to the exact sum after a decrement. | Drives a `1x2` sketch's column 0 to `2^32` by deltas, saturating the row, then back to `2^31`, and verifies the accumulator is `2^62 + 1` and survives a round trip. | ### CountMinSketchWithSpaceSaving @@ -929,10 +934,11 @@ Wire tests: [`src/sketch_framework/hydra/wire.rs`](../src/sketch_framework/hydra | `hydra_count_sketch_round_trip_serialization` | A Count Sketch grid round-trips under kind_id `0x07 0x02`. | Runs the same round trip over `2x16` fast-path Count Sketch counters and the same three frequency probes. | | `hydra_hyperloglog_round_trip_serialization` | An HLL grid round-trips under kind_id `0x07 0x03`. | Runs the same round trip over `HyperLogLog` counters fed 300 records, probing two cardinality queries. | | `hydra_kll_round_trip_serialization` | A KLL grid round-trips under kind_id `0x07 0x00`. | Runs the same round trip over default `Kll` counters fed 300 records, probing two quantiles and a CDF. | -| `hydra_univmon_round_trip_serialization` | A UnivMon grid round-trips under kind_id `0x07 0x04`. | Runs the same round trip over `UnivMon::new(4, 2, 16, 3)` counters fed 120 records, probing L1, L2, entropy, and cardinality. | +| `hydra_univmon_round_trip_serialization` | A UnivMon grid round-trips under kind_id `0x07 0x05`. | Runs the same round trip over `UnivMon::new(4, 2, 16, 3)` counters fed 120 records, probing L1, L2, entropy, and cardinality. | +| `hydra_univmon_retired_kind_decodes_and_recomputes_l2` | The retired `0x07 0x04` decodes, each cell's `l2` dropped and recomputed. | Re-wraps a string-keyed UnivMon grid's cells in the retired shape with an empty `l2`, verifies it decodes and re-encodes to the `0x07 0x05` bytes, and that each of the two decoders rejects the other kind. | | `hydra_count_sketch_negative_cells_round_trip` | Signed cells reach the wire and come back unchanged. | Verifies the Count Sketch grid holds a negative counter and that every cell of every counter matches after a decode. | | `hydra_schema_round_trips_exactly` | The key columns round-trip exactly, escaping included. | For a grid whose two labels carry a semicolon, a colon, and a backslash and whose subkeys carry a semicolon, verifies the round trip preserves both frequency probes' answers and that the decoded schema equals the labels. | -| `hydra_variants_reject_each_others_envelopes` | Each variant's decoder owns exactly one kind_id. | Runs all five per-variant decoders against all five variants' envelopes and verifies each succeeds only on its own, and that a plain `2x16` Count-Min envelope is refused by every decoder and by `from_asapv1`. | +| `hydra_variants_reject_each_others_envelopes` | Each variant's decoder owns exactly one current kind_id (the UnivMon decoder also reads the retired `0x07 0x04`, tested separately). | Runs all five per-variant decoders against all five variants' envelopes and verifies each succeeds only on its own, and that a plain `2x16` Count-Min envelope is refused by every decoder and by `from_asapv1`. | | `hydra_rejects_a_mixed_variant_grid` | A grid mixing counter variants has no encoding. | Replaces one cell of a Count-Min grid with an HLL counter and verifies serialization fails with a complaint naming `cell (1, 1)` and both counter types. | | `hydra_rejects_serializing_an_inconsistent_grid` | The encode side refuses states its own decoder would reject. | Verifies four Count-Min grids each fail to serialize: one whose `rows` is `4` against its storage, one whose grid is an unfilled `Vector2D::unfilled(3, 8)`, one holding a `2x8` cell against a `2x16` prototype, and one whose `prototype` holds data. | | `hydra_univmon_rejects_cells_mixing_key_variants` | A UnivMon grid whose cells hold different `OwnedDataInput` variants has no single `cell_key_type`. | Seats a string-keyed UnivMon in a `u64`-keyed grid and verifies serialization fails with a "mix key variants" complaint. | @@ -982,6 +988,8 @@ Wire tests: [`src/sketch_framework/univmon/wire.rs`](../src/sketch_framework/uni | test_name | test_description | what_is_tested | | --- | --- | --- | | `univmon_round_trip_serialization` | Univmon round trip serialization. | After weighted inserts, verifies non-empty serialization and round-trip preservation of configuration fields, `total_weight`, `L1/L2/entropy` (`<1e-6` drift), and cardinality (`< EPSILON` drift). | +| `univmon_retired_kind_decodes_and_recomputes_l2` | The retired `0x10 0x00` decodes, its `l2` dropped and recomputed. | Re-wraps the populated pyramid as an eight-element `0x10 0x00` payload with a stale all-ones `l2` and with an empty one, and verifies each decodes to the same `estimate_l2` and re-encodes to the `0x10 0x01` bytes. | +| `univmon_payload_carries_no_l2` | The pyramid payload has seven fields. | Verifies the populated pyramid's payload is a seven-element msgpack array. | | `update_populates_total_weight_and_heavy_hitters` | Update populates bucket size and heavy hitters. | Inserting one hot key `40` times sets `total_weight=40`, tracks key in heavy-hitter heap with count `>=20`, and yields exact `L1=40` and `cardinality=1`. | | `merge_with_combines_heavy_hitters` | Merge with combines heavy hitters. | Merging sketches with disjoint heavy keys verifies merged left heap contains both contributions (`left=25`, `right=30`) while right heap retains `right=30`. | | `univmon_layers_use_different_seeds` | Univmon layers use different seeds. | Verifies hash outputs for the same key with seed indices `0..3` are all pairwise different. | @@ -1043,7 +1051,8 @@ Wire tests: [`src/sketch_framework/univmon_two_tier/wire.rs`](../src/sketch_fram | `two_tier_top_layer_complete_tracks_the_top_heap` | Top-layer completeness is readable. | On a one-layer heap-4 pyramid, verifies `top_layer_complete()` holds after 4 keys, fails after a fifth, and holds again after `clear()`. | | `two_tier_estimate_l2_meets_its_documented_bound` | `estimate_l2` meets its `(ε, δ)` bound. | 150 seeded trials of 400 fresh random Zipf-weighted keys into `UnivMonTwoTier::new(8, 1, 5, 64, 3, 16, 3)`: trials with `estimate_l2()^2` off by more than `ε = 0.5` relative stay within `n·δ + 4σ + 1` for `δ = P(Bin(5, 0.125) >= 3)`. `_long` (ignored) runs 3000 trials. | | `two_tier_estimate_cardinality_meets_its_documented_bound` | `estimate_cardinality` meets its joint error/completeness bound. | 60 seeded trials of 2000 fresh random keys into a heap-32 pyramid with `3x256` elephant and `2x32` mouse layers, `8 = ceil(log2(2000/32)) + 2` deep: incomplete top layers stay rare, joint events of completeness and error beyond `ε = 0.5` stay within the Chebyshev `2/(kε²)` bound; the ideal estimator (restoring the exact top count) has variance under `2 F0²/k` and mean error within four standard errors of zero. `_long` (ignored) runs 2000 trials. | -| `univmon_two_tier_round_trip_serialization` | The envelope frames a pyramid under kind_id `0x11 0x00`, and the bytes are stable across a round trip. | For `UnivMonTwoTier::new(4, 2, 3, 16, 2, 8, 4)` fed four weighted string keys, verifies the bytes open with the ASAPv1 magic, a `kind_id_len` of `2`, and `0x11 0x00`, that the metadata carries that layout and `key_type` `string`, and that the decode matches on `total_weight`, L1, L2, cardinality, entropy, and the candidate flags, re-serializing identically. | +| `univmon_two_tier_round_trip_serialization` | The envelope frames a pyramid under kind_id `0x11 0x01`, and the bytes are stable across a round trip. | For `UnivMonTwoTier::new(4, 2, 3, 16, 2, 8, 4)` fed four weighted string keys, verifies the bytes open with the ASAPv1 magic, a `kind_id_len` of `2`, and `0x11 0x01`, that the metadata carries that layout and `key_type` `string`, and that the decode matches on `total_weight`, L1, L2, cardinality, entropy, and the candidate flags, re-serializing identically. | +| `univmon_two_tier_retired_kind_decodes_and_recomputes_l2` | The retired `0x11 0x00` decodes, its `l2` dropped and recomputed. | Re-wraps the populated two-tier pyramid as a `0x11 0x00` payload with an all-zero `l2`, and verifies it decodes to the same `estimate_l2` and re-encodes to the `0x11 0x01` bytes. | | `univmon_two_tier_geometry_survives` | The two tiers are derived from the layer's position. | Verifies every decoded layer keeps its tier's dimensions - `3x16` for the two elephant layers and `2x8` for the mouse layers - its own seed index, and its counter array. | | `univmon_two_tier_layers_with_different_heap_loads_round_trip` | Each layer's heap contents survive the round trip. | For `UnivMonTwoTier::new(8, 2, 3, 32, 2, 16, 4)` fed 40 weighted keys, verifies the layers carry different heap loads, that every layer's length and per-key counts match after a decode, and that the bytes re-serialize identically. | | `univmon_two_tier_without_mouse_layers_round_trips` | The one-tier case still round-trips. | For a pyramid whose four layers are all elephants, verifies the decoded layers all read `2x16` and the bytes re-serialize identically. | @@ -1153,6 +1162,7 @@ Wire tests: [`src/sketch_framework/eh_bucket_sketch/wire.rs`](../src/sketch_fram | `eh_bucket_sketch_round_trip_serialization` | The envelope frames a union under kind_id `0x14 0x00`, and the state survives a round trip. | For a `3x8` fast-path Count-Min variant holding one key, verifies the bytes open with the ASAPv1 magic, a `kind_id_len` of `2`, and `0x14 0x00`, and that the decode keeps a `sketch_type` of `CountMinSketch` and answers that key as the source does. | | `eh_bucket_sketch_every_variant_round_trips` | Every variant this build carries round-trips and keeps its type. | For each populated variant, verifies the decode reports the same `sketch_type` and re-serializes to the bytes it came from. | | `eh_bucket_sketch_kind_ids_are_build_independent` | The ten encoded nested kind_ids, plus the retired `0x05 0x00`, and their registry names are the same in every build. | Verifies `variant_name` maps each of the ten ids to its name, the retired `0x05 0x00` to `DDSketch` and `0xff 0xff` to `None`, and that every populated variant emits the id the table gives it. | +| `eh_bucket_sketch_decodes_retired_l2_carrying_kind_ids` | Buckets nested under the retired `0x19 0x00` and `0x10 0x00` decode into their variants. | Nests the retired `count_l2hh_2x4_seed7` and `univmon_str_l3_2x4_h5` golden blocks under their own ids, verifies each decodes and re-encodes under `0x19 0x01` / `0x10 0x01`, and that the retired ids share their current ids' names. | | `eh_bucket_sketch_rejects_an_experimental_kind_id` | An experimental variant's kind_id is refused without the feature. | Verifies crafted triples carrying the `CocoSketch`, `ElasticSketch`, and `UniformSampling` kind_ids each fail to decode, with a message naming the variant and "experimental" in builds without the feature. | | `eh_bucket_sketch_rejects_an_unknown_kind_id` | An unknown kind_id is refused. | Verifies a Count-Min triple relabelled `0xff 0xff` fails with a "not a wire variant" complaint. | | `eh_bucket_sketch_rejects_sibling_algorithm_kind_ids` | The nested ids are pinned to one algorithm each. | Verifies an HLL triple relabelled `0x01 0x01` or `0x01 0x03` fails to decode, and that a KLL triple relabelled `0x06 0x01` fails too. | @@ -2074,7 +2084,7 @@ Test file: [`tests/e2e/wire.rs`](../tests/e2e/wire.rs) | `the_kmv_envelope_round_trips_and_names_its_kind` | A KMV envelope names kind_id `0x0e 0x00` and preserves the estimate. | Fills `Kmv::::new(1_024)` from `uniform_u64(50_000, 10_000_000, 0x5EED_1001)` and verifies the `kind_id_len` of `2`, bytes `0x0e 0x00` and an unchanged `estimate()`. | | `the_heap_backed_matrix_envelopes_round_trip_and_name_their_kinds` | The heap-backed matrix envelopes name their kinds and keep both matrix and heap. | Over the 20,000-draw Zipf stream, verifies `CountMinSketchWithHeap::, FastPath>::new(4, 2_048, 32)` carries `kind_id_len` `2` and bytes `0x03 0x00` and `CountSketchWithHeap::, RegularPath>::new(5, 2_048, 32)` carries `0x0a 0x00`, that both decodes equal their sources on `estimate` for keys `0..1_024`, and that `heap().len()` is unchanged for both. | | `the_matrix_and_quantile_envelopes_carry_their_answers_unchanged` | The four core envelopes name their kinds and carry their answers unchanged. | Over the 20,000-draw Zipf stream, verifies the kind_id bytes `0x02 0x00` for `CountMinSketch, FastPath>` at `4x2048`, `0x04 0x00` for `CountSketch, RegularPath>` at `5x2048`, `0x06 0x00` for `Kll` at `k = 200` seeded `0x5EED_9002`, and `0x01 0x02` for `HyperLogLog`; then that both matrices agree with their sources on `estimate` for keys `0..1_024`, that KLL's `quantile` is bit-identical at q `0.0`, `0.5` and `1.0`, and that the HyperLogLog `estimate()` is unchanged. | -| `the_univmon_family_envelopes_round_trip_and_name_their_kinds` | The three UnivMon envelopes name their kinds and preserve their aggregate answers. | Over the 20,000-draw Zipf stream, verifies `UnivMon::new(32, 5, 512, 8)` carries bytes `0x10 0x00` with unchanged `estimate_l1()`, `estimate_l2()` and `estimate_cardinality()`; `UnivMonTwoTier::with_defaults()` carries `0x11 0x00` with unchanged `estimate_l1()` and `estimate_l2()`; and `UnivMonQ::new(Default::default())` carries `0x1a 0x00` with unchanged `count()`, `estimate_f2()` and `quantile` at q `0.1`, `0.5` and `0.9`. | +| `the_univmon_family_envelopes_round_trip_and_name_their_kinds` | The three UnivMon envelopes name their kinds and preserve their aggregate answers. | Over the 20,000-draw Zipf stream, verifies `UnivMon::new(32, 5, 512, 8)` carries bytes `0x10 0x01` with unchanged `estimate_l1()`, `estimate_l2()` and `estimate_cardinality()`; `UnivMonTwoTier::with_defaults()` carries `0x11 0x01` with unchanged `estimate_l1()` and `estimate_l2()`; and `UnivMonQ::new(Default::default())` carries `0x1a 0x00` with unchanged `count()`, `estimate_f2()` and `quantile` at q `0.1`, `0.5` and `0.9`. | ### ASAPv1 Golden Byte Vectors @@ -2131,7 +2141,8 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `space_saving_nototal_i64_c8_matches_golden` | An `i64`-keyed Space-Saving summary serializes to its golden bytes, its count tie by two's-complement bits. | Verifies a summary of capacity `8` holding `(-1, 70000, 0)`, `(300, 9, 2)`, `(0, 5, 0)`, `(1, 5, 1)`, `(i64::MIN, 5, 4)`, `(-129, 5, 0)` with `discarded_max` `4`, decodes from `asapv1_golden/space_saving_nototal_i64_c8.hex` (`0x18 0x01`) with that capacity, `min_count()` `4` and those triples and re-serializes identically. | | `space_saving_retired_fixtures_decode_to_their_nototal_twins` | The retired `0x18 0x00` fixtures still decode, their `total` dropped. | Verifies `space_saving_i64_c8.hex` and `space_saving_bytes_c3.hex` decode and re-encode as `space_saving_nototal_i64_c8.hex` and `space_saving_nototal_bytes_c3.hex`. | | `uniform_sampling_r0p25_matches_golden` | A UniformSampling reservoir serializes to its golden bytes (`experimental`). | Verifies a sampler at rate `0.25` with `total_seen` `20`, `rng_state` `0x0123456789abcdef` and entries `(3, -0.0)`, `(3, 0.0)`, `(2^63, 1e300)`, `(u64::MAX, NaN)`, decodes from `asapv1_golden/uniform_sampling_r0p25.hex` with that rate, `total_seen` and the samples' bits and re-serializes identically. | -| `univmon_optimized_l5_e2_3x8_1x4_h6_matches_golden` | A two-tier UnivMon pyramid serializes to its golden bytes. | Verifies `UnivMonTwoTier::new(6, 2, 3, 8, 1, 4, 5)` with counters set by `apply_delta`, the heaps `{"alpha": 40, "beta": 7, "gamma": 7}`, `{"delta": 3}`, `{}`, `{"eps": 1}`, `{}` and `bucket_size` `70` serializes to `asapv1_golden/univmon_optimized_l5_e2_3x8_1x4_h6.hex`, and that the decode reports that layout, every layer's cells and heap, and re-serializes identically. | +| `univmon_optimized_l5_e2_3x8_1x4_h6_matches_golden` | A two-tier UnivMon pyramid serializes to its golden bytes. | Verifies `UnivMonTwoTier::new(6, 2, 3, 8, 1, 4, 5)` with counters set by `apply_delta`, the heaps `{"alpha": 40, "beta": 7, "gamma": 7}`, `{"delta": 3}`, `{}`, `{"eps": 1}`, `{}` and `bucket_size` `70` serializes to `asapv1_golden/univmon_optimized_l5_e2_3x8_1x4_h6_no_l2.hex`, and that the decode reports that layout, every layer's cells and heap, and re-serializes identically. | +| `retired_l2_fixtures_decode_to_their_no_l2_twins` | Each retired `l2`-carrying fixture decodes and re-encodes as its `*_no_l2` twin. | Decodes the retired `univmon_{str,i64,empty}_l3_2x4_h5`, `count_l2hh_2x4_seed7`, `univmon_optimized_l5_e2_3x8_1x4_h6` and `hydra_univmon_1x2` fixtures (kinds `10 00`, `19 00`, `11 00`, `07 04`) and verifies each re-encodes to the matching `*_no_l2` fixture (`10 01`, `19 01`, `11 01`, `07 05`). | | `univmon_q_i32_l3_c8_r3_matches_golden` | A UnivMon-Q golden decodes to its state and re-serializes byte-identically. | Verifies `asapv1_golden/univmon_q_i32_l3_c8_r3.hex` decodes to the config `levels` `3`, `width` `8`, `cols_halving_period` `1`, `depth` `3`, `counter_bits` `32`, `candidates` `2`, `ordered_samples` `3`, `seed_index` `5`, `source_id()` `7`, `count()` `12`, `min()` `-2.5` and `max()` `5.0`, and re-serializes to the same bytes. | ### Conformance Kit diff --git a/src/asapv1/kind.rs b/src/asapv1/kind.rs index a8461698..f98c66c0 100644 --- a/src/asapv1/kind.rs +++ b/src/asapv1/kind.rs @@ -29,7 +29,11 @@ pub(crate) const HYDRA_KLL: &[u8] = &[0x07, 0x00]; pub(crate) const HYDRA_COUNT_MIN: &[u8] = &[0x07, 0x01]; pub(crate) const HYDRA_COUNT_SKETCH: &[u8] = &[0x07, 0x02]; pub(crate) const HYDRA_HLL: &[u8] = &[0x07, 0x03]; -pub(crate) const HYDRA_UNIVMON: &[u8] = &[0x07, 0x04]; +/// Hydra over UnivMon: each cell is the §3.15 pyramid without `l2` (spec §3.9). +pub(crate) const HYDRA_UNIVMON: &[u8] = &[0x07, 0x05]; +/// Retired Hydra-over-UnivMon kind whose cells also carried each layer's `l2` +/// accumulators: decoded, with `l2` dropped and recomputed, and never encoded. +pub(crate) const HYDRA_UNIVMON_CARRYING_L2: &[u8] = &[0x07, 0x04]; pub(crate) const SET_AGGREGATOR: &[u8] = &[0x08, 0x00]; pub(crate) const SET_AGGREGATOR_DELTA: &[u8] = &[0x08, 0x80]; pub(crate) const COUNT_SKETCH_WITH_HEAP: &[u8] = &[0x0a, 0x00]; @@ -38,8 +42,16 @@ pub(crate) const ELASTIC: &[u8] = &[0x0b, 0x00]; pub(crate) const COCO: &[u8] = &[0x0c, 0x00]; pub(crate) const UNIFORM_SAMPLING: &[u8] = &[0x0d, 0x00]; pub(crate) const KMV: &[u8] = &[0x0e, 0x00]; -pub(crate) const UNIVMON: &[u8] = &[0x10, 0x00]; -pub(crate) const UNIVMON_TWO_TIER: &[u8] = &[0x11, 0x00]; +/// UnivMon: the layers' counters and heaps, without `l2` (spec §3.15). +pub(crate) const UNIVMON: &[u8] = &[0x10, 0x01]; +/// Retired UnivMon kind that also carried the layers' `l2` accumulators: +/// decoded, with `l2` dropped and recomputed, and never encoded. +pub(crate) const UNIVMON_CARRYING_L2: &[u8] = &[0x10, 0x00]; +/// Two-tier UnivMon, without `l2` (spec §3.16). +pub(crate) const UNIVMON_TWO_TIER: &[u8] = &[0x11, 0x01]; +/// Retired two-tier UnivMon kind that also carried the layers' `l2` +/// accumulators: decoded, with `l2` dropped and recomputed, and never encoded. +pub(crate) const UNIVMON_TWO_TIER_CARRYING_L2: &[u8] = &[0x11, 0x00]; pub(crate) const EXPONENTIAL_HISTOGRAM: &[u8] = &[0x13, 0x00]; pub(crate) const EH_BUCKET_SKETCH: &[u8] = &[0x14, 0x00]; /// Bloom: the packed bit grid (spec §3.4). @@ -52,7 +64,11 @@ pub(crate) const SPACE_SAVING: &[u8] = &[0x18, 0x01]; /// Retired Space-Saving kind that also carried `total`: decoded, with `total` /// dropped, and never encoded. pub(crate) const SPACE_SAVING_WITH_TOTAL: &[u8] = &[0x18, 0x00]; -pub(crate) const COUNT_SKETCH_WITH_L2_NORM: &[u8] = &[0x19, 0x00]; +/// CountSketchWithL2Norm: the counters alone (spec §3.19). +pub(crate) const COUNT_SKETCH_WITH_L2_NORM: &[u8] = &[0x19, 0x01]; +/// Retired CountSketchWithL2Norm kind that also carried the per-row `l2` +/// accumulators: decoded, with `l2` dropped and recomputed, and never encoded. +pub(crate) const COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2: &[u8] = &[0x19, 0x00]; pub(crate) const UNIVMON_Q: &[u8] = &[0x1a, 0x00]; /// Count-Min with a Space-Saving summary, without its `total` (spec §3.24). pub(crate) const COUNT_MIN_WITH_SPACE_SAVING: &[u8] = &[0x1e, 0x01]; diff --git a/src/sketch_framework/eh_bucket_sketch/wire.rs b/src/sketch_framework/eh_bucket_sketch/wire.rs index 1a78597f..68f61b47 100644 --- a/src/sketch_framework/eh_bucket_sketch/wire.rs +++ b/src/sketch_framework/eh_bucket_sketch/wire.rs @@ -23,12 +23,15 @@ //! //! ## The kind_id namespace is fixed in every build //! -//! All eleven ids (ten encoded, plus the retired `0x05 0x00`) dispatch -//! whatever features are on. A decoder built without +//! All thirteen ids (ten encoded, plus the retired `0x05 0x00`, `0x10 0x00` +//! and `0x19 0x00`) dispatch whatever features are on. A decoder built without //! `experimental` rejects `0x0d 0x00` by name with the feature named, and its //! encoder can never emit it. A DDSketch bucket encodes as `0x05 0x01`; the //! retired `0x05 0x00` still decodes into the same variant, its `sum` / `min` / //! `max` dropped. +//! A CountSketchWithL2Norm bucket encodes as `0x19 0x01` and a UnivMon bucket +//! as `0x10 0x01`; the retired `0x19 0x00` and `0x10 0x00`, which also carried +//! `l2`, still decode into the same variants, their `l2` dropped and recomputed. use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError}; use serde::{Deserialize, Serialize}; @@ -46,14 +49,16 @@ pub(crate) fn variant_name(kind_id: &[u8]) -> Option<&'static str> { match kind_id { kind::COUNT_MIN => Some("CountMinSketch"), kind::COCO => Some("Coco"), - kind::COUNT_SKETCH_WITH_L2_NORM => Some("CountSketchWithL2Norm"), + kind::COUNT_SKETCH_WITH_L2_NORM | kind::COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2 => { + Some("CountSketchWithL2Norm") + } kind::COUNT_SKETCH => Some("CountSketch"), kind::DDSKETCH | kind::DDSKETCH_WITH_SCALARS => Some("DDSketch"), kind::ELASTIC => Some("Elastic"), kind::HLL_ERTL => Some("HLL"), kind::KLL => Some("KLL"), kind::UNIFORM_SAMPLING => Some("UniformSampling"), - kind::UNIVMON => Some("UnivMon"), + kind::UNIVMON | kind::UNIVMON_CARRYING_L2 => Some("UnivMon"), _ => None, } } @@ -87,8 +92,8 @@ pub(crate) struct SketchState { pub(crate) state: Vec, } -/// Rejects a kind_id outside the eleven (ten encoded, plus the retired -/// `0x05 0x00`). +/// Rejects a kind_id outside the thirteen (ten encoded, plus the retired +/// `0x05 0x00`, `0x10 0x00` and `0x19 0x00`). pub(crate) fn unknown_variant(kind_id: &[u8]) -> RmpDecodeError { RmpDecodeError::Uncategorized(format!( "ASAPv1 EhBucketSketch: kind_id {kind_id:02x?} is not a wire variant" @@ -186,9 +191,11 @@ fn rebuild_variant(triple: &SketchState) -> Result Ok(EhBucketSketch::CocoSketch(crate::CocoSketch::from_asapv1( &bytes, )?)), - kind::COUNT_SKETCH_WITH_L2_NORM => Ok(EhBucketSketch::CountSketchWithL2Norm( - crate::CountSketchWithL2Norm::from_asapv1(&bytes)?, - )), + kind::COUNT_SKETCH_WITH_L2_NORM | kind::COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2 => { + Ok(EhBucketSketch::CountSketchWithL2Norm( + crate::CountSketchWithL2Norm::from_asapv1(&bytes)?, + )) + } kind::COUNT_SKETCH => Ok(EhBucketSketch::CountSketch( crate::CountSketch::from_asapv1(&bytes)?, )), @@ -206,9 +213,9 @@ fn rebuild_variant(triple: &SketchState) -> Result Ok(EhBucketSketch::Uniform( crate::UniformSampling::from_asapv1(&bytes)?, )), - kind::UNIVMON => Ok(EhBucketSketch::UnivMon(crate::UnivMon::from_asapv1( - &bytes, - )?)), + kind::UNIVMON | kind::UNIVMON_CARRYING_L2 => Ok(EhBucketSketch::UnivMon( + crate::UnivMon::from_asapv1(&bytes)?, + )), other => Err(unknown_variant(other)), } } @@ -450,6 +457,46 @@ pub(crate) mod tests { } } + /// A bucket nested under the retired `0x19 0x00` or `0x10 0x00`, which + /// carried `l2`, decodes into the same variant and re-encodes under the + /// current `0x19 0x01` / `0x10 0x01`. + #[test] + fn eh_bucket_sketch_decodes_retired_l2_carrying_kind_ids() { + use crate::asapv1::envelope::hex; + let retired = [ + ( + kind::COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2, + kind::COUNT_SKETCH_WITH_L2_NORM, + hex(include_str!( + "../../../asapv1_golden/count_l2hh_2x4_seed7.hex" + )), + ), + ( + kind::UNIVMON_CARRYING_L2, + kind::UNIVMON, + hex(include_str!( + "../../../asapv1_golden/univmon_str_l3_2x4_h5.hex" + )), + ), + ]; + for (retired_id, current_id, bytes) in retired { + assert_eq!(variant_name(retired_id), variant_name(current_id)); + let (kind_id, descriptor, state) = envelope::split(&bytes).expect("split"); + assert_eq!(kind_id, retired_id); + let triple = SketchState { + kind_id: kind_id.to_vec(), + descriptor: descriptor.to_vec(), + state: state.to_vec(), + }; + let decoded = EhBucketSketch::from_asapv1(&envelope_for(&triple)).expect("decode"); + assert_eq!( + sketch_state(&decoded).expect("state").kind_id, + current_id, + "{retired_id:02x?}" + ); + } + } + /// The experimental kind_id is rejected without the feature, with an error /// naming the variant. Crafted bytes, so the test runs in both builds. #[test] diff --git a/src/sketch_framework/hydra/wire.rs b/src/sketch_framework/hydra/wire.rs index bd10ddab..024018d5 100644 --- a/src/sketch_framework/hydra/wire.rs +++ b/src/sketch_framework/hydra/wire.rs @@ -10,9 +10,11 @@ //! ## One kind_id per counter variant //! //! `0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, -//! `0x07 0x03` HyperLogLog, `0x07 0x04` UnivMon. The kind_id names the counter, +//! `0x07 0x03` HyperLogLog, `0x07 0x05` UnivMon. The kind_id names the counter, //! so the payload carries no per-cell variant tag and a grid mixing variants -//! has no encoding. +//! has no encoding. The retired `0x07 0x04` UnivMon counter, whose cells also +//! carried each layer's `l2` accumulators, still decodes with `l2` dropped +//! and recomputed, and is never encoded. //! //! ## Counters are inlined, not nested //! @@ -49,11 +51,14 @@ use crate::asapv1::envelope; use crate::asapv1::kind; use crate::asapv1::kind::{ HYDRA_COUNT_MIN, HYDRA_COUNT_SKETCH, HYDRA_HLL, HYDRA_KLL, HYDRA_UNIVMON, + HYDRA_UNIVMON_CARRYING_L2, }; use crate::asapv1::wire_key::exact_vec; use crate::common::hash::check_matrix_rows; use crate::input::HydraCounter; -use crate::sketch_framework::univmon::wire::{PyramidPayload, UnivMonMetadata, univmon_metadata}; +use crate::sketch_framework::univmon::wire::{ + PyramidPayload, PyramidPayloadCarryingL2, UnivMonMetadata, univmon_metadata, +}; use crate::sketches::countminsketch_heap::heap_wire::{EMPTY_KEY_TYPE, check_empty_key_type}; use crate::{ CountMinSketch, CountSketch, DefaultXxHasher, Ertl, FastPath, HashProfile, HllRegistersP14, @@ -146,7 +151,7 @@ struct HydraKllMetadata { cell_item_type: String, } -/// Hydra descriptor metadata for the UnivMon counter (`0x07 0x04`). The four +/// Hydra descriptor metadata for the UnivMon counter (`0x07 0x05`, and the retired `0x07 0x04`). The four /// pyramid dimensions and the heaps' `cell_key_type` are shared by every /// cell. #[derive(Debug, PartialEq, Serialize, Deserialize)] @@ -305,6 +310,13 @@ struct HydraUnivMonPayload { cells: Vec>, } +/// The retired `0x07 0x04` payload: each cell in the retired pyramid shape +/// that also carries `l2`. Decoded only. +#[derive(Debug, Serialize, Deserialize)] +struct HydraUnivMonPayloadCarryingL2 { + cells: Vec>, +} + /// The HyperLogLog counter's own descriptor, rebuilt to hand a cell's registers /// to the HLL decoder. #[derive(Serialize)] @@ -467,7 +479,7 @@ fn univmon_cell_holds_keys(payload: &[u8]) -> Result { // directly. impl Hydra { /// Serializes the grid into an ASAPv1 MessagePack envelope, one kind_id per - /// counter variant (`0x07 0x00`-`0x07 0x04`). The metadata is derived from + /// counter variant (`0x07 0x00`-`0x07 0x03`, and `0x07 0x05` for UnivMon). The metadata is derived from /// [`DefaultXxHasher`]'s [`HashProfile`], the profile the subkey hash at /// [`crate::HYDRA_SEED_INDEX`] uses. /// @@ -488,7 +500,9 @@ impl Hydra { } /// Deserializes a grid from an ASAPv1 MessagePack envelope, routing on the - /// kind_id to the counter variant it names. Any other kind_id is rejected. + /// kind_id to the counter variant it names. The retired UnivMon counter + /// `0x07 0x04` decodes too, each cell's `l2` dropped and recomputed. Any + /// other kind_id is rejected. /// /// Every declared count is measured against what the payload actually /// carries before anything is sized from it, so crafted geometry fails @@ -501,8 +515,9 @@ impl Hydra { id if id == kind::HYDRA_COUNT_SKETCH => Self::decode_cs(bytes), id if id == kind::HYDRA_HLL => Self::decode_hll(bytes), id if id == kind::HYDRA_UNIVMON => Self::decode_univmon(bytes), + id if id == kind::HYDRA_UNIVMON_CARRYING_L2 => Self::decode_univmon_carrying_l2(bytes), other => Err(decode_error(format!( - "Hydra kind_id mismatch: stored {other:?}, expected one of {HYDRA_KLL:?}, {HYDRA_COUNT_MIN:?}, {HYDRA_COUNT_SKETCH:?}, {HYDRA_HLL:?}, {HYDRA_UNIVMON:?}" + "Hydra kind_id mismatch: stored {other:?}, expected one of {HYDRA_KLL:?}, {HYDRA_COUNT_MIN:?}, {HYDRA_COUNT_SKETCH:?}, {HYDRA_HLL:?}, {HYDRA_UNIVMON:?}, or the retired {HYDRA_UNIVMON_CARRYING_L2:?}" ))), } } @@ -1011,7 +1026,22 @@ impl Hydra { } fn decode_univmon(bytes: &[u8]) -> Result { - let (metadata, payload) = split_for(bytes, kind::HYDRA_UNIVMON)?; + Self::decode_univmon_cells(bytes, false) + } + + /// The retired `0x07 0x04`: the same metadata, cells in the pyramid shape + /// that also carries `l2`, which is dropped. + fn decode_univmon_carrying_l2(bytes: &[u8]) -> Result { + Self::decode_univmon_cells(bytes, true) + } + + fn decode_univmon_cells(bytes: &[u8], carries_l2: bool) -> Result { + let expected_kind_id = if carries_l2 { + kind::HYDRA_UNIVMON_CARRYING_L2 + } else { + kind::HYDRA_UNIVMON + }; + let (metadata, payload) = split_for(bytes, expected_kind_id)?; let meta: HydraUnivMonMetadata = envelope::decode_block(metadata)?; let shape = UnivMonShape { layers: meta.cell_layers, @@ -1042,7 +1072,7 @@ impl Hydra { shape.layers, shape.rows, shape.cols, shape.heap_size ))); } - let payloads = unpack_univmon_cells(&meta.cell_key_type, payload)?; + let payloads = unpack_univmon_cells(&meta.cell_key_type, payload, carries_l2)?; if payloads.len() != cells { return Err(decode_error(format!( "Hydra carries {} UnivMon cells != rows*cols {cells}", @@ -1123,14 +1153,29 @@ fn pack_univmon_cells(key_type: &str, payloads: &[&[u8]]) -> Result, Rmp } /// Reads the cells with `keys` typed by the metadata `cell_key_type`, -/// returning each cell's pyramid payload bytes. An unknown key type is -/// rejected, and keys whose msgpack types disagree fail to decode. -fn unpack_univmon_cells(key_type: &str, payload: &[u8]) -> Result>, RmpDecodeError> { +/// returning each cell's pyramid payload bytes in the current §3.15 shape. +/// When `carries_l2`, the cells are in the retired shape and their `l2` is +/// dropped. An unknown key type is rejected, and keys whose msgpack types +/// disagree fail to decode. +fn unpack_univmon_cells( + key_type: &str, + payload: &[u8], + carries_l2: bool, +) -> Result>, RmpDecodeError> { macro_rules! unpack { ($ty:ty) => {{ - let decoded: HydraUnivMonPayload<$ty> = envelope::decode_block(payload)?; - decoded - .cells + let cells: Vec> = if carries_l2 { + let decoded: HydraUnivMonPayloadCarryingL2<$ty> = envelope::decode_block(payload)?; + decoded + .cells + .into_iter() + .map(PyramidPayload::from) + .collect() + } else { + let decoded: HydraUnivMonPayload<$ty> = envelope::decode_block(payload)?; + decoded.cells + }; + cells .iter() .map(|cell| rmp_serde::to_vec(cell).map_err(|e| decode_error(e.to_string()))) .collect() @@ -1363,7 +1408,6 @@ mod tests { .into_iter() .map(|cell| PyramidPayload { counts: cell.counts, - l2: cell.l2, heap_lens: cell.heap_lens, keys: cell .keys @@ -1386,6 +1430,45 @@ mod tests { assert!(Hydra::from_asapv1(&encoded).is_ok()); } + /// The retired `0x07 0x04` still decodes: each cell's `l2` (here empty) is + /// read and dropped, the layers' accumulators recomputed, and the grid + /// re-encodes under `0x07 0x05` byte-identically to the original. + #[test] + fn hydra_univmon_retired_kind_decodes_and_recomputes_l2() { + let mut hydra = grid(HydraCounter::UnivMon(UnivMon::new(4, 2, 16, 3))); + let keys = ["alpha", "beta", "gamma", "delta"]; + feed(&mut hydra, (0..40).map(|i| DataInput::Str(keys[i % 4]))); + let encoded = hydra.to_asapv1().expect("serialize"); + assert_envelope(&encoded, kind::HYDRA_UNIVMON); + let (_, metadata, payload) = envelope::split(&encoded).expect("split"); + let cells: HydraUnivMonPayload = from_slice(payload).expect("payload"); + let retired = HydraUnivMonPayloadCarryingL2 { + cells: cells + .cells + .into_iter() + .map(|cell| PyramidPayloadCarryingL2 { + counts: cell.counts, + l2: crate::sketches::countsketch_l2_norm::wire::RetiredL2::default(), + heap_lens: cell.heap_lens, + keys: cell.keys, + heap_counts: cell.heap_counts, + candidate_complete: cell.candidate_complete, + bucket_size: cell.bucket_size, + update_mode: cell.update_mode, + }) + .collect(), + }; + let bytes = envelope::encode( + kind::HYDRA_UNIVMON_CARRYING_L2, + metadata, + &rmp_serde::to_vec(&retired).expect("payload"), + ); + let decoded = Hydra::from_asapv1(&bytes).expect("decode retired"); + assert_eq!(decoded.to_asapv1().expect("re-encode"), encoded); + assert!(Hydra::decode_univmon(&bytes).is_err()); + assert!(Hydra::decode_univmon_carrying_l2(&encoded).is_err()); + } + /// Count Sketch cells are signed: the payload must carry negatives through /// unchanged. #[test] diff --git a/src/sketch_framework/univmon/wire.rs b/src/sketch_framework/univmon/wire.rs index 567b66d6..488cacc5 100644 --- a/src/sketch_framework/univmon/wire.rs +++ b/src/sketch_framework/univmon/wire.rs @@ -8,14 +8,16 @@ //! and `candidate_complete` fields directly without widening any field //! visibility. See `docs/asapv1_wire_format.md`. //! -//! UnivMon is one algorithm — a single kind_id `0x10 0x00`. The pyramid shape +//! UnivMon is one algorithm, kind_id `0x10 0x01`. The pyramid shape //! (`layers`, `rows`, `cols`, `heap_size`) and the heaps' //! `key_type` are construction config and live in the metadata, so the payload -//! carries the layers' raw state. +//! carries the layers' raw state. The layers' L2 accumulators are recomputed +//! from their counters on decode; the retired kind `0x10 0x00` also carried +//! them and still decodes, its `l2` read and dropped. //! //! ## Layers are inlined, not nested //! -//! Each layer is a `CountSketchWithL2Norm` plus an `TopKHeap`. Their state is written +//! Each layer is a `CountSketchWithL2Norm` plus a `TopKHeap`. Their state is written //! straight into UnivMon's own positional array through //! [`l2_norm_wire::layer_state`] and [`heap_entries`]; no layer carries an //! envelope, a magic or a metadata map of its own. @@ -32,11 +34,12 @@ //! the layer's position already gives. use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError}; +use serde::de::DeserializeOwned; use serde::{Deserialize, Serialize}; use crate::asapv1::envelope; use crate::asapv1::kind; -use crate::asapv1::kind::UNIVMON; +use crate::asapv1::kind::{UNIVMON, UNIVMON_CARRYING_L2}; use crate::asapv1::wire_key::{WireBytes, WireString, exact_vec}; use crate::sketches::countminsketch_heap::heap_wire::{ EMPTY_KEY_TYPE, check_distinct_keys, check_empty_key_type, heap_entries, key_type_of, @@ -97,16 +100,15 @@ pub(crate) fn univmon_metadata( /// The pyramid payload both UnivMon kinds share, a msgpack **array** /// (`to_vec`, positional): -/// `[counts, l2, heap_lens, keys, heap_counts, candidate_complete, -/// bucket_size, update_mode]`. +/// `[counts, heap_lens, keys, heap_counts, candidate_complete, bucket_size, +/// update_mode]`. /// -/// `counts` and `l2` concatenate the layers' `CountSketchWithL2Norm` state in layer order; -/// `heap_lens` cuts the parallel `keys` / `heap_counts` arrays into per-layer -/// runs, with `keys` typed by the metadata `key_type`. +/// `counts` concatenates the layers' `CountSketchWithL2Norm` counters in layer +/// order; `heap_lens` cuts the parallel `keys` / `heap_counts` arrays into +/// per-layer runs, with `keys` typed by the metadata `key_type`. #[derive(Debug, Serialize, Deserialize)] pub(crate) struct PyramidPayload { pub(crate) counts: Vec, - pub(crate) l2: Vec, pub(crate) heap_lens: Vec, #[serde( deserialize_with = "exact_vec", @@ -119,6 +121,53 @@ pub(crate) struct PyramidPayload { pub(crate) update_mode: u8, } +/// The retired pyramid payload of `0x10 0x00`, `0x11 0x00` and the Hydra +/// cells of `0x07 0x04`: [`PyramidPayload`] with the layers' `l2` +/// accumulators at position 1. Decoded only; `l2` is dropped. +#[derive(Debug, Serialize, Deserialize)] +pub(crate) struct PyramidPayloadCarryingL2 { + pub(crate) counts: Vec, + pub(crate) l2: l2_norm_wire::RetiredL2, + pub(crate) heap_lens: Vec, + #[serde( + deserialize_with = "exact_vec", + bound(deserialize = "K: Deserialize<'de>") + )] + pub(crate) keys: Vec, + pub(crate) heap_counts: Vec, + pub(crate) candidate_complete: Vec, + pub(crate) bucket_size: u64, + pub(crate) update_mode: u8, +} + +impl From> for PyramidPayload { + /// Drops `l2`; the decoder recomputes it from `counts`. + fn from(retired: PyramidPayloadCarryingL2) -> Self { + PyramidPayload { + counts: retired.counts, + heap_lens: retired.heap_lens, + keys: retired.keys, + heap_counts: retired.heap_counts, + candidate_complete: retired.candidate_complete, + bucket_size: retired.bucket_size, + update_mode: retired.update_mode, + } + } +} + +/// Reads one pyramid payload: the current layout, or the retired one that +/// carries `l2` when `carries_l2`, its `l2` dropped. +pub(crate) fn decode_pyramid_payload( + payload: &[u8], + carries_l2: bool, +) -> Result, RmpDecodeError> { + if carries_l2 { + Ok(envelope::decode_block::>(payload)?.into()) + } else { + envelope::decode_block(payload) + } +} + /// Wire tag of the update mode: `0` unset, `1` standard, `2` terminal-only. /// The mode is acquired from the stream rather than configured, and it selects /// the query recurrence, so it is payload state. @@ -173,7 +222,6 @@ pub(crate) fn check_univmon_dimensions( /// The layers' state in emitted order, ready to be packed. pub(crate) struct PyramidState<'a> { pub(crate) counts: Vec, - pub(crate) l2: Vec, pub(crate) heap_lens: Vec, pub(crate) entries: Vec<(&'a OwnedDataInput, i64)>, pub(crate) candidate_complete: Vec, @@ -181,7 +229,7 @@ pub(crate) struct PyramidState<'a> { pub(crate) update_mode: u8, } -/// Reads every layer's counters, accumulators and heap entries in the emitted +/// Reads every layer's counters and heap entries in the emitted /// order: layers ascending, and within a layer the heap's own emitted order /// (descending count, ties by the total order over the key). /// @@ -217,7 +265,6 @@ pub(crate) fn pyramid_state<'a>( })?; let mut state = PyramidState { counts: Vec::new(), - l2: Vec::new(), heap_lens: Vec::with_capacity(layers), entries: Vec::new(), candidate_complete: candidate_complete.to_vec(), @@ -239,9 +286,9 @@ pub(crate) fn pyramid_state<'a>( counter.seed_index() ))); } - let (counts, l2) = l2_norm_wire::layer_state(counter)?; - state.counts.extend_from_slice(counts); - state.l2.extend_from_slice(l2); + state + .counts + .extend_from_slice(l2_norm_wire::layer_state(counter)?); let entries = heap_entries(&heaps[layer]); check_distinct_keys(entries.iter().map(|(key, _)| *key)) .map_err(|problem| RmpEncodeError::Syntax(format!("{problem} (layer {layer})")))?; @@ -297,7 +344,6 @@ pub(crate) fn encode_pyramid( } rmp_serde::to_vec(&PyramidPayload { counts: state.counts, - l2: state.l2, heap_lens: state.heap_lens, keys, heap_counts, @@ -331,7 +377,6 @@ pub(crate) fn encode_pyramid( } rmp_serde::to_vec(&PyramidPayload { counts: state.counts, - l2: state.l2, heap_lens: state.heap_lens, keys, heap_counts, @@ -350,7 +395,6 @@ pub(crate) fn encode_pyramid( } rmp_serde::to_vec(&PyramidPayload { counts: state.counts, - l2: state.l2, heap_lens: state.heap_lens, keys, heap_counts, @@ -368,7 +412,6 @@ pub(crate) fn encode_pyramid( /// The layer state one pyramid payload decodes into. pub(crate) struct DecodedPyramid { pub(crate) counts: Vec, - pub(crate) l2: Vec, pub(crate) heap_lens: Vec, pub(crate) entries: Vec<(OwnedDataInput, i64)>, pub(crate) candidate_complete: Vec, @@ -376,19 +419,21 @@ pub(crate) struct DecodedPyramid { pub(crate) update_mode: u8, } -/// Reads the payload with `keys` typed by the metadata `key_type`. Rejects an -/// unknown `key_type` and parallel arrays of unequal length; a `keys` array -/// whose msgpack types do not match `key_type` fails to decode. +/// Reads the payload with `keys` typed by the metadata `key_type`, in the +/// retired layout that carries `l2` when `carries_l2` (its `l2` dropped). +/// Rejects an unknown `key_type` and parallel arrays of unequal length; a +/// `keys` array whose msgpack types do not match `key_type` fails to decode. pub(crate) fn decode_pyramid( key_type: &str, payload: &[u8], + carries_l2: bool, ) -> Result { macro_rules! unpack { ($variant:ident, $ty:ty) => { unpack!($ty, OwnedDataInput::$variant) }; ($ty:ty, $into:expr) => {{ - let decoded: PyramidPayload<$ty> = envelope::decode_block(payload)?; + let decoded: PyramidPayload<$ty> = decode_pyramid_payload(payload, carries_l2)?; ( decoded .keys @@ -397,7 +442,6 @@ pub(crate) fn decode_pyramid( .collect::>(), DecodedPyramid { counts: decoded.counts, - l2: decoded.l2, heap_lens: decoded.heap_lens, entries: Vec::new(), candidate_complete: decoded.candidate_complete, @@ -426,7 +470,7 @@ pub(crate) fn decode_pyramid( key.into_string() )), "bytes" => { - let decoded: PyramidPayload = envelope::decode_block(payload)?; + let decoded: PyramidPayload = decode_pyramid_payload(payload, carries_l2)?; ( decoded .keys @@ -435,7 +479,6 @@ pub(crate) fn decode_pyramid( .collect::>(), DecodedPyramid { counts: decoded.counts, - l2: decoded.l2, heap_lens: decoded.heap_lens, entries: Vec::new(), candidate_complete: decoded.candidate_complete, @@ -497,7 +540,6 @@ pub(crate) fn rebuild_layers( ))); } let mut cells = 0usize; - let mut accumulators = 0usize; for &(rows, cols) in geometry { l2_norm_wire::check_dimensions(rows, cols).map_err(complain)?; cells = cells @@ -505,9 +547,6 @@ pub(crate) fn rebuild_layers( complain(format!("UnivMon layer geometry {rows}x{cols} overflows")) })?) .ok_or_else(|| complain("UnivMon total counter count overflows".to_string()))?; - accumulators = accumulators - .checked_add(rows) - .ok_or_else(|| complain("UnivMon total accumulator count overflows".to_string()))?; } if decoded.counts.len() != cells { return Err(complain(format!( @@ -515,12 +554,6 @@ pub(crate) fn rebuild_layers( decoded.counts.len() ))); } - if decoded.l2.len() != accumulators { - return Err(complain(format!( - "UnivMon l2 length {} != the declared layers' {accumulators} rows", - decoded.l2.len() - ))); - } let mut seated = 0usize; for &len in &decoded.heap_lens { seated = seated @@ -537,19 +570,16 @@ pub(crate) fn rebuild_layers( let mut sketches = Vec::with_capacity(layers); let mut heaps = Vec::with_capacity(layers); let mut entries = decoded.entries.into_iter(); - let (mut cell, mut accumulator) = (0usize, 0usize); + let mut cell = 0usize; for (layer, &(rows, cols)) in geometry.iter().enumerate() { let next_cell = cell + rows * cols; - let next_accumulator = accumulator + rows; sketches.push(l2_norm_wire::rebuild_layer::( rows, cols, layer, &decoded.counts[cell..next_cell], - &decoded.l2[accumulator..next_accumulator], )?); cell = next_cell; - accumulator = next_accumulator; let run: Vec<(OwnedDataInput, i64)> = entries .by_ref() .take(decoded.heap_lens[layer] as usize) @@ -571,7 +601,7 @@ pub(crate) fn rebuild_layers( // so this impl reads the private fields directly. impl UnivMon { /// Serializes the pyramid into an ASAPv1 MessagePack envelope - /// (kind_id `0x10 0x00`). The metadata is derived from + /// (kind_id `0x10 0x01`). The metadata is derived from /// [`DefaultXxHasher`]'s [`HashProfile`], the hasher UnivMon is built on. /// /// Fails when a structural parameter is zero, when a layer's geometry or @@ -619,14 +649,22 @@ impl UnivMon { /// Every state the algorithm could not have produced is rejected with an /// error rather than a panic, and no declared count sizes an allocation /// before the payload is measured against it. + /// + /// Reads the current kind `0x10 0x01` and the retired `0x10 0x00`, whose + /// `l2` is read and dropped; either way the layers' L2 accumulators are + /// recomputed from their counters. pub fn from_asapv1(bytes: &[u8]) -> Result { let (kind_id, metadata, payload) = envelope::split(bytes).map_err(RmpDecodeError::Uncategorized)?; - if kind_id != kind::UNIVMON { - return Err(RmpDecodeError::Uncategorized(format!( - "UnivMon kind_id mismatch: stored {kind_id:?}, expected {UNIVMON:?}" - ))); - } + let carries_l2 = match kind_id { + kind::UNIVMON => false, + kind::UNIVMON_CARRYING_L2 => true, + _ => { + return Err(RmpDecodeError::Uncategorized(format!( + "UnivMon kind_id mismatch: stored {kind_id:?}, expected {UNIVMON:?} or the retired {UNIVMON_CARRYING_L2:?}" + ))); + } + }; let meta: UnivMonMetadata = envelope::decode_block(metadata)?; if meta != univmon_metadata::( @@ -650,17 +688,12 @@ impl UnivMon { check_univmon_dimensions(layers, rows, cols, heap_size) .map_err(RmpDecodeError::Uncategorized)?; check_layers("UnivMon", layers).map_err(RmpDecodeError::Uncategorized)?; - let decoded = decode_pyramid(&meta.key_type, payload)?; + let decoded = decode_pyramid(&meta.key_type, payload, carries_l2)?; check_empty_key_type(&meta.key_type, decoded.entries.is_empty()) .map_err(RmpDecodeError::Uncategorized)?; - // The declared layer count is measured against the accumulators the - // payload actually carries before the geometry is built from it. - if rows.checked_mul(layers) != Some(decoded.l2.len()) { - return Err(RmpDecodeError::Uncategorized(format!( - "UnivMon declares {layers} layers of {rows} rows against a payload carrying {} accumulators", - decoded.l2.len() - ))); - } + // `check_layers` capped `layers` at UNIVMON_MAX_LAYERS, so this + // geometry is small; `rebuild_layers` measures its cells against the + // payload's `counts` before any matrix is built. let geometry = vec![(rows, cols); layers]; let (layer_sketches, layer_heaps, candidate_complete, bucket_size, mode_tag) = rebuild_layers(&geometry, heap_size, decoded)?; @@ -735,7 +768,6 @@ mod tests { assert!(!payload.keys.is_empty()); let as_bin = PyramidPayload { counts: payload.counts, - l2: payload.l2, heap_lens: payload.heap_lens, keys: payload .keys @@ -775,7 +807,6 @@ mod tests { // Sized to the crafted geometry, so the row bound is what bites. let mut payload: PyramidPayload = payload_of(&encoded); payload.counts = vec![0; rows * cols * layers]; - payload.l2 = vec![0; rows * layers]; let problem = UnivMon::from_asapv1(&crafted(&meta, &payload)) .expect_err("layer rows past MATRIX_MAX_ROWS must be rejected") .to_string(); @@ -790,7 +821,7 @@ mod tests { let um = populated(); let encoded = um.to_asapv1().expect("serialize UnivMon"); assert!(encoded.starts_with(b"ASAPv1")); - assert_eq!(&encoded[7..10], &[2u8, 0x10, 0x00]); // kind_id_len=2, kind_id=[0x10,0x00] + assert_eq!(&encoded[7..10], &[2u8, 0x10, 0x01]); // kind_id_len=2, kind_id=[0x10,0x01] let meta = metadata_of(&encoded); assert_eq!(meta.metadata_version, 1); @@ -815,6 +846,50 @@ mod tests { ); } + /// `encoded` rewritten under the retired kind `0x10 0x00`, with `l2` + /// at payload position 1. + fn carrying_l2(encoded: &[u8], l2: Vec) -> Vec { + let (_, metadata, _) = envelope::split(encoded).expect("split"); + let p: PyramidPayload = payload_of(encoded); + let retired = PyramidPayloadCarryingL2 { + counts: p.counts, + l2: l2_norm_wire::RetiredL2(l2), + heap_lens: p.heap_lens, + keys: p.keys, + heap_counts: p.heap_counts, + candidate_complete: p.candidate_complete, + bucket_size: p.bucket_size, + update_mode: p.update_mode, + }; + let payload = rmp_serde::to_vec(&retired).expect("payload"); + envelope::encode(kind::UNIVMON_CARRYING_L2, metadata, &payload) + } + + /// The retired `0x10 0x00` still decodes: its `l2`, here stale or empty, is read and dropped, + /// the layers' accumulators recomputed, and the pyramid re-encodes under + /// `0x10 0x01` byte-identically to the original. + #[test] + fn univmon_retired_kind_decodes_and_recomputes_l2() { + let um = populated(); + let encoded = um.to_asapv1().expect("serialize"); + // A stale `l2` (all ones) and an empty one: both are dropped. + let stale = vec![1; um.layers * um.rows]; + for l2 in [stale, Vec::new()] { + let decoded = UnivMon::from_asapv1(&carrying_l2(&encoded, l2)).expect("decode retired"); + assert_eq!(decoded.estimate_l2(), um.estimate_l2()); + assert_eq!(decoded.to_asapv1().expect("re-encode"), encoded); + } + } + + /// The payload holds no `l2`: seven positional fields. + #[test] + fn univmon_payload_carries_no_l2() { + let encoded = populated().to_asapv1().expect("serialize"); + let (_, _, payload) = envelope::split(&encoded).expect("split"); + let fields: Vec = from_slice(payload).expect("payload array"); + assert_eq!(fields.len(), 7); + } + /// Layers hold different numbers of entries: each layer's heap contents /// and emitted order survive the round trip. #[test] @@ -1042,12 +1117,11 @@ mod tests { ); } - // A zero dimension against an empty payload: the accumulator check - // alone passes (0 rows over any layer count names 0 accumulators), so - // the dimensions must be refused before the geometry is sized. + // A zero dimension against an empty payload: the cell count alone + // passes (0 rows over any layer count names 0 cells), so the + // dimensions must be refused before the geometry is sized. let mut empty = payload_of::(&encoded); empty.counts.clear(); - empty.l2.clear(); empty.keys.clear(); empty.heap_counts.clear(); for meta in [shaped(u32::MAX, 0, 16, 4), shaped(3, 0, 0, 4)] { @@ -1182,7 +1256,6 @@ mod tests { let (rows, cols) = (meta.rows as usize, meta.cols as usize); let mut payload: PyramidPayload = payload_of(&encoded); payload.counts = vec![0; rows * cols * layers]; - payload.l2 = vec![0; rows * layers]; payload.heap_lens = vec![0; layers]; payload.candidate_complete = vec![true; layers]; let problem = UnivMon::from_asapv1(&crafted(&meta, &payload)) diff --git a/src/sketch_framework/univmon_two_tier/wire.rs b/src/sketch_framework/univmon_two_tier/wire.rs index aa33bb51..f3065b55 100644 --- a/src/sketch_framework/univmon_two_tier/wire.rs +++ b/src/sketch_framework/univmon_two_tier/wire.rs @@ -7,13 +7,14 @@ //! and `candidate_complete` fields directly without widening any field //! visibility. See `docs/asapv1_wire_format.md`. //! -//! Two-tier UnivMon is one algorithm — a single kind_id `0x11 0x00`. It +//! Two-tier UnivMon is one algorithm, kind_id `0x11 0x01`. It //! shares [`UnivMon`](crate::UnivMon)'s payload -//! (`[counts, l2, heap_lens, keys, heap_counts, candidate_complete, -//! bucket_size, update_mode]`) and differs only in its metadata: the two-tier +//! (`[counts, heap_lens, keys, heap_counts, candidate_complete, bucket_size, +//! update_mode]`) and differs only in its metadata: the two-tier //! layout gives layer `i` the elephant dimensions while `i < elephant_layers` //! and the mouse dimensions after, so every per-layer geometry is derived and -//! none is stored. +//! none is stored. The retired kind `0x11 0x00` also carried the layers' `l2` +//! accumulators and still decodes, its `l2` read and dropped. //! //! [`UnivMonPool`](crate::UnivMonPool) is a free-list of scratch //! `UnivMon`s rather than a sketch, and has no wire kind. @@ -23,7 +24,7 @@ use serde::{Deserialize, Serialize}; use crate::asapv1::envelope; use crate::asapv1::kind; -use crate::asapv1::kind::UNIVMON_TWO_TIER; +use crate::asapv1::kind::{UNIVMON_TWO_TIER, UNIVMON_TWO_TIER_CARRYING_L2}; use crate::sketch_framework::univmon::wire::{ check_layers, decode_pyramid, encode_pyramid, pyramid_key_type, pyramid_state, rebuild_layers, update_mode_of, update_mode_tag, @@ -131,20 +132,11 @@ fn check_layout_dimensions( Ok(()) } -/// Total accumulators the declared layout implies, one per row per layer. -fn accumulator_count(layers: usize, layout: &TwoTierLayout) -> Option { - let elephants = layers.min(layout.elephant_layers as usize); - let mice = layers - elephants; - elephants - .checked_mul(layout.elephant_rows as usize)? - .checked_add(mice.checked_mul(layout.mouse_rows as usize)?) -} - // Wire serialization for UnivMonTwoTier. `wire` is a descendant of the sketch // module, so this impl reads the private fields directly. impl UnivMonTwoTier { /// Serializes the pyramid into an ASAPv1 MessagePack envelope - /// (kind_id `0x11 0x00`). The metadata is derived from + /// (kind_id `0x11 0x01`). The metadata is derived from /// [`DefaultXxHasher`]'s [`HashProfile`], the hasher the pyramid is built /// on. /// @@ -213,11 +205,15 @@ impl UnivMonTwoTier { pub fn from_asapv1(bytes: &[u8]) -> Result { let (kind_id, metadata, payload) = envelope::split(bytes).map_err(RmpDecodeError::Uncategorized)?; - if kind_id != kind::UNIVMON_TWO_TIER { - return Err(RmpDecodeError::Uncategorized(format!( - "UnivMonTwoTier kind_id mismatch: stored {kind_id:?}, expected {UNIVMON_TWO_TIER:?}" - ))); - } + let carries_l2 = match kind_id { + kind::UNIVMON_TWO_TIER => false, + kind::UNIVMON_TWO_TIER_CARRYING_L2 => true, + _ => { + return Err(RmpDecodeError::Uncategorized(format!( + "UnivMonTwoTier kind_id mismatch: stored {kind_id:?}, expected {UNIVMON_TWO_TIER:?} or the retired {UNIVMON_TWO_TIER_CARRYING_L2:?}" + ))); + } + }; let meta: UnivMonTwoTierMetadata = envelope::decode_block(metadata)?; let layout = TwoTierLayout { elephant_layers: meta.elephant_layers, @@ -255,17 +251,12 @@ impl UnivMonTwoTier { ) .map_err(RmpDecodeError::Uncategorized)?; check_layers("UnivMonTwoTier", layers).map_err(RmpDecodeError::Uncategorized)?; - let decoded = decode_pyramid(&meta.key_type, payload)?; + let decoded = decode_pyramid(&meta.key_type, payload, carries_l2)?; check_empty_key_type(&meta.key_type, decoded.entries.is_empty()) .map_err(RmpDecodeError::Uncategorized)?; - // The declared layout is measured against the accumulators the payload - // actually carries before the geometry is built from it. - if accumulator_count(layers, &layout) != Some(decoded.l2.len()) { - return Err(RmpDecodeError::Uncategorized(format!( - "UnivMonTwoTier declares a layout the payload's {} accumulators do not match", - decoded.l2.len() - ))); - } + // `check_layers` capped `layers` at UNIVMON_MAX_LAYERS, so this + // geometry is small; `rebuild_layers` measures its cells against the + // payload's `counts` before any matrix is built. let geometry = geometry_of(layers, &layout); let (layer_sketches, layer_heaps, candidate_complete, bucket_size, mode_tag) = rebuild_layers(&geometry, heap_size, decoded)?; @@ -330,7 +321,7 @@ mod tests { let pyramid = populated(); let encoded = pyramid.to_asapv1().expect("serialize"); assert!(encoded.starts_with(b"ASAPv1")); - assert_eq!(&encoded[7..10], &[2u8, 0x11, 0x00]); // kind_id_len=2, kind_id=[0x11,0x00] + assert_eq!(&encoded[7..10], &[2u8, 0x11, 0x01]); // kind_id_len=2, kind_id=[0x11,0x01] let meta = metadata_of(&encoded); assert_eq!(meta.metadata_version, 1); @@ -365,6 +356,37 @@ mod tests { ); } + /// The retired `0x11 0x00` still decodes: its `l2` (here all zero, not + /// the counters' sums of squares) is read and dropped, the layers' + /// accumulators recomputed, and the pyramid re-encodes under `0x11 0x01` + /// byte-identically to the original. + #[test] + fn univmon_two_tier_retired_kind_decodes_and_recomputes_l2() { + use crate::sketch_framework::univmon::wire::PyramidPayloadCarryingL2; + let pyramid = populated(); + let encoded = pyramid.to_asapv1().expect("serialize"); + let (_, metadata, _) = envelope::split(&encoded).expect("split"); + let p: PyramidPayload = payload_of(&encoded); + let retired = PyramidPayloadCarryingL2 { + counts: p.counts, + l2: crate::sketches::countsketch_l2_norm::wire::RetiredL2(vec![0; 3 * 2 + 2 * 2]), + heap_lens: p.heap_lens, + keys: p.keys, + heap_counts: p.heap_counts, + candidate_complete: p.candidate_complete, + bucket_size: p.bucket_size, + update_mode: p.update_mode, + }; + let bytes = envelope::encode( + kind::UNIVMON_TWO_TIER_CARRYING_L2, + metadata, + &rmp_serde::to_vec(&retired).expect("payload"), + ); + let decoded = UnivMonTwoTier::from_asapv1(&bytes).expect("decode retired"); + assert_eq!(decoded.estimate_l2(), pyramid.estimate_l2()); + assert_eq!(decoded.to_asapv1().expect("re-encode"), encoded); + } + /// The two tiers are derived from the layer's position: the elephant /// layers keep their dimensions and the mouse layers keep theirs. #[test] @@ -668,12 +690,11 @@ mod tests { ); } - // A zero row count against an empty payload: the accumulator check - // alone passes (0 rows over any layer count names 0 accumulators), so - // the tiers' dimensions must be refused before the geometry is sized. + // A zero row count against an empty payload: the cell count alone + // passes (0 rows over any layer count names 0 cells), so the tiers' + // dimensions must be refused before the geometry is sized. let mut empty = payload_of::(&encoded); empty.counts.clear(); - empty.l2.clear(); empty.keys.clear(); empty.heap_counts.clear(); let zero_rows = TwoTierLayout { @@ -786,7 +807,6 @@ mod tests { meta.layers = layers as u32; let mut payload: PyramidPayload = payload_of(&encoded); payload.counts = vec![0; 2 * (2 * 16) + (layers - 2) * (2 * 8)]; - payload.l2 = vec![0; 2 * layers]; payload.heap_lens = vec![0; layers]; payload.candidate_complete = vec![true; layers]; let problem = UnivMonTwoTier::from_asapv1(&crafted(&meta, &payload)) diff --git a/src/sketches/countsketch_l2_norm.rs b/src/sketches/countsketch_l2_norm.rs index e4ba805b..b046ce92 100644 --- a/src/sketches/countsketch_l2_norm.rs +++ b/src/sketches/countsketch_l2_norm.rs @@ -2,6 +2,11 @@ //! //! [`CountSketchWithL2Norm`] is the per-layer sketch of `UnivMon` and //! `UnivMonTwoTier`; its L2 norm sets their terminal-mode heavy-hitter threshold. +//! +//! Each row's L2 accumulator is a cache of its counters: read through +//! [`clamp_to_i64`] it always equals the row's sum of squared counters, +//! saturated at `i64::MAX`, so the ASAPv1 payload does not carry it and the +//! decoder recomputes it. use crate::common::structures::matrix_storage::RowEstimates; use std::marker::PhantomData; @@ -20,21 +25,53 @@ use crate::common::structures::fixed_structure::{DEFAULT_COLS, DEFAULT_ROWS}; #[derive(Clone, Debug)] pub struct CountSketchWithL2Norm { counts: Vector2D, - row_l2_squared: Vector1D, + /// Per row, [`row_capped_square_sum`] of its counters, kept exact by every + /// write in O(1) so `l2` reads it in O(rows). Read it through + /// [`clamp_to_i64`]. + row_l2_squared: Vector1D, rows: usize, cols: usize, seed_index: usize, _hasher: PhantomData, } +/// The cap on one counter's contribution to its row's accumulator: `2^63`, +/// one past `i64::MAX`. +const SQUARE_CAP: i128 = 1 << 63; + +/// One counter's contribution to its row's accumulator: its square, capped at +/// [`SQUARE_CAP`]. The square fits an `i128` (`i64::MIN^2 = 2^126`). +#[inline(always)] +fn capped_square(counter: i64) -> i128 { + let counter = i128::from(counter); + (counter * counter).min(SQUARE_CAP) +} + +/// The sum of a row's [`capped_square`]s: what a row's accumulator holds. +/// Each term is at most `2^63`, so the sum of fewer than `2^64` terms fits an +/// `i128` and adding or removing one term is exact. Capping a term changes +/// nothing once clamped: a capped term alone already reaches `i64::MAX`, so +/// `clamp_to_i64` of this sum is `min(sum(counts[row]^2), i64::MAX)`. +fn row_capped_square_sum(row: &[i64]) -> i128 { + row.iter().fold(0_i128, |sum, &counter| { + sum.saturating_add(capped_square(counter)) + }) +} + +/// An accumulator as the value it stands for, saturated at `i64::MAX`. +#[inline(always)] +fn clamp_to_i64(accumulator: i128) -> i64 { + accumulator.clamp(0, i128::from(i64::MAX)) as i64 +} + impl CountSketchWithL2Norm { /// Bytes a sketch of `rows * cols` counters holds: [`Self::memory_bytes`] /// of one built with those dimensions, for every hasher. pub fn estimated_memory_bytes(rows: usize, cols: usize) -> usize { size_of::().saturating_add( rows.saturating_mul(cols) - .saturating_add(rows) - .saturating_mul(size_of::()), + .saturating_mul(size_of::()) + .saturating_add(rows.saturating_mul(size_of::())), ) } @@ -121,7 +158,8 @@ impl CountSketchWithL2Norm { /// per-row sums of squares. pub fn memory_bytes(&self) -> usize { size_of::() - + (self.rows * self.cols + self.row_l2_squared.capacity()) * size_of::() + + self.rows * self.cols * size_of::() + + self.row_l2_squared.capacity() * size_of::() } /// Exposes the backing matrix for inspection/testing. @@ -129,11 +167,6 @@ impl CountSketchWithL2Norm { &self.counts } - /// Mutable access used internally for testing scenarios. - pub fn as_storage_mut(&mut self) -> &mut Vector2D { - &mut self.counts - } - /// Merges another sketch while asserting compatible dimensions. pub fn merge(&mut self, other: &Self) { assert_eq!( @@ -147,13 +180,10 @@ impl CountSketchWithL2Norm { ); for i in 0..self.rows { - let mut row_l2 = 0_i128; for j in 0..self.cols { self.counts[i][j] += other.counts[i][j]; - let counter = self.counts[i][j] as i128; - row_l2 = row_l2.saturating_add(counter.saturating_mul(counter)); } - self.row_l2_squared[i] = row_l2.min(i64::MAX as i128) as i64; + self.row_l2_squared[i] = row_capped_square_sum(self.counts.row_slice(i)); } } @@ -181,20 +211,15 @@ impl CountSketchWithL2Norm { } } - /// Writes one counter and carries the row's L2 accumulator with it. - /// - /// Saturating i128 intermediate so extreme counts degrade gracefully - /// instead of wrapping (matches merge-path semantics, which also clamps to - /// [0, i64::MAX]: L2^2 can never go negative, even when a decrement follows - /// prior saturation). + /// Writes one counter and carries the row's accumulator with it in O(1): + /// the old counter's [`capped_square`] leaves the sum and the new one's + /// enters, exactly, so the accumulator stays [`row_capped_square_sum`] of + /// the row through saturation and back. #[inline(always)] fn write_counter(&mut self, row: usize, col: usize, new_value: i64) { let old_value = self.counts.query_one_counter(row, col); self.counts[row][col] = new_value; - let old_l2 = self.row_l2_squared.as_slice()[row] as i128; - let new_l2 = old_l2 + (new_value as i128) * (new_value as i128) - - (old_value as i128) * (old_value as i128); - self.row_l2_squared[row] = new_l2.clamp(0, i64::MAX as i128) as i64; + self.row_l2_squared[row] += capped_square(new_value) - capped_square(old_value); } /// Seed offset this sketch hashes with; UnivMon gives each layer its own. @@ -217,26 +242,6 @@ impl CountSketchWithL2Norm { self.write_counter(row, col, new_value); } - /// Inserts without L2 update using precomputed hash value. - pub fn update_weighted_with_hash_without_l2(&mut self, hashed_val: u128, weight: i64) { - let mask_bits = self.counts.mask_bits() as usize; - let mask = (1u128 << mask_bits) - 1; - let mut shift_amount = 0; - let mut sign_bit_pos = 127; - - for i in 0..self.rows { - let hashed = (hashed_val >> shift_amount) & mask; - let idx = (hashed as usize) % self.cols; - let bit = ((hashed_val >> sign_bit_pos) & 1) as i64; - let sign_bit = -(1 - 2 * bit); - - self.counts[i][idx] += sign_bit * weight; - - shift_amount += mask_bits; - sign_bit_pos -= 1; - } - } - /// Update and estimate with hash optimization. /// due to the limitation of seeds, use fast_insert only pub fn update_weighted_and_estimate(&mut self, val: &DataInput, weight: i64) -> f64 { @@ -249,7 +254,7 @@ impl CountSketchWithL2Norm { pub fn l2_squared(&self) -> f64 { let mut values: RowEstimates = self.row_l2_squared.as_slice()[..self.rows] .iter() - .map(|&v| v as f64) + .map(|&v| clamp_to_i64(v) as f64) .collect(); compute_median_inline_f64(&mut values) } diff --git a/src/sketches/countsketch_l2_norm/wire.rs b/src/sketches/countsketch_l2_norm/wire.rs index c9bcbe87..c8b32634 100644 --- a/src/sketches/countsketch_l2_norm/wire.rs +++ b/src/sketches/countsketch_l2_norm/wire.rs @@ -8,24 +8,27 @@ //! `seed_index` fields directly without widening any field visibility. See //! `docs/asapv1_wire_format.md`. //! -//! CountSketchWithL2Norm is one algorithm — a single kind_id `0x19 0x00`. The dimensions +//! CountSketchWithL2Norm is one algorithm, kind_id `0x19 0x01`. The dimensions //! (`rows` / `cols`) and the seed-list index the sketch hashes with //! (`seed_index`) are construction config and live in the metadata, so the -//! payload is `[counts, l2]`. +//! payload is `[counts]`. //! //! ## The shared sub-payload //! //! [`CountSketchWithL2Norm`] is also the per-layer counter of `UnivMon` and -//! `UnivMonTwoTier`. Those sketches inline the same -//! `(counts, l2)` state into their own positional arrays instead of nesting an -//! envelope per layer, so [`layer_state`] and [`rebuild_layer`] are the one -//! place that state is read and rebuilt. +//! `UnivMonTwoTier`. Those sketches inline the same `counts` into their own +//! positional arrays instead of nesting an envelope per layer, so +//! [`layer_state`] and [`rebuild_layer`] are the one place that state is read +//! and rebuilt. //! -//! ## `l2` is carried, not recomputed +//! ## `l2` is recomputed, not carried //! -//! A row's `l2` is its running sum of squares, clamped to `[0, i64::MAX]`. It -//! is not `sum(counts[row]^2)`: `update_weighted_with_hash_without_l2` -//! moves counters without it and saturation is one-way, so it is state. +//! A row's L2 accumulator always equals +//! [`row_capped_square_sum`](super::row_capped_square_sum) of its counters +//! (read clamped at `i64::MAX`, it is the row's sum of squares), so the +//! decoder recomputes it and the encoder rejects a cached value that disagrees. +//! The retired kind `0x19 0x00` also carried it as `[counts, l2]`; it still +//! decodes, its `l2` read and dropped, and is never encoded. use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError}; use serde::{Deserialize, Serialize}; @@ -33,11 +36,11 @@ use std::marker::PhantomData; use crate::asapv1::envelope; use crate::asapv1::kind; -use crate::asapv1::kind::COUNT_SKETCH_WITH_L2_NORM; +use crate::asapv1::kind::{COUNT_SKETCH_WITH_L2_NORM, COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2}; use crate::common::hash::check_matrix_rows; use crate::{HashProfile, SketchHasher, Vector1D, Vector2D}; -use super::CountSketchWithL2Norm; +use super::{CountSketchWithL2Norm, clamp_to_i64, row_capped_square_sum}; /// CountSketchWithL2Norm descriptor metadata (ASAPv1 §2), a msgpack **map** /// (`to_vec_named`) with keys in this declaration order — the canonical order @@ -83,13 +86,53 @@ pub(crate) fn count_sketch_with_l2_norm_metadata( } } -/// CountSketchWithL2Norm payload (ASAPv1 §3.x), a msgpack **array** (`to_vec`, -/// positional): `[counts, l2]`. `counts` is packed row-major over `rows*cols` -/// signed cells; `l2` is one accumulator per row. +/// CountSketchWithL2Norm payload (ASAPv1 §3.19), a msgpack **array** +/// (`to_vec`, positional): `[counts]`, packed row-major over `rows*cols` +/// signed cells. #[derive(Debug, Serialize, Deserialize)] struct CountSketchWithL2NormPayload { counts: Vec, - l2: Vec, +} + +/// The retired `0x19 0x00` payload, `[counts, l2]`: decoded only, its `l2` +/// dropped. +#[derive(Debug, Serialize, Deserialize)] +struct CountSketchWithL2NormPayloadCarryingL2 { + counts: Vec, + l2: RetiredL2, +} + +/// A retired payload's `l2` array. Decoding reads every element as an `i64`, +/// so anything but an integer array is still rejected, and keeps none of +/// them: the decoded value is always empty and allocates nothing. Encoding +/// writes the held values, which only tests use to craft retired bytes. +#[derive(Debug, Default)] +pub(crate) struct RetiredL2(pub(crate) Vec); + +impl Serialize for RetiredL2 { + fn serialize(&self, serializer: S) -> Result { + self.0.serialize(serializer) + } +} + +impl<'de> Deserialize<'de> for RetiredL2 { + fn deserialize>(deserializer: D) -> Result { + struct SkipI64s; + impl<'de> serde::de::Visitor<'de> for SkipI64s { + type Value = RetiredL2; + fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { + f.write_str("an array of i64 l2 accumulators") + } + fn visit_seq>( + self, + mut seq: A, + ) -> Result { + while seq.next_element::()?.is_some() {} + Ok(RetiredL2::default()) + } + } + deserializer.deserialize_seq(SkipI64s) + } } /// Rejects a zero dimension, a row count past `MATRIX_MAX_ROWS`, and a @@ -122,47 +165,53 @@ pub(crate) fn check_seed_index(seed_index: usize, seed_list_len: usize) -> Resul Ok(()) } -/// The `(counts, l2)` sub-payload one CountSketchWithL2Norm contributes, in the order the -/// wire emits it. Rejects a matrix whose cell count disagrees with its own -/// dimensions, a mis-sized `l2`, and a negative accumulator — states the -/// decoder refuses. +/// The `counts` sub-payload one CountSketchWithL2Norm contributes, in the +/// order the wire emits it. Rejects a matrix whose cell count disagrees with +/// its own dimensions, and a cached accumulator that is not its row's +/// [`row_capped_square_sum`], the value the decoder recomputes; the error +/// names the first such row. pub(crate) fn layer_state( sketch: &CountSketchWithL2Norm, -) -> Result<(&[i64], &[i64]), RmpEncodeError> { +) -> Result<&[i64], RmpEncodeError> { check_dimensions(sketch.rows, sketch.cols).map_err(RmpEncodeError::Syntax)?; let cells = sketch.rows.saturating_mul(sketch.cols); let counts = sketch.counts.as_slice(); - let l2 = sketch.row_l2_squared.as_slice(); if counts.len() != cells { return Err(RmpEncodeError::Syntax(format!( "ASAPv1 CountSketchWithL2Norm state: counts length {} != rows*cols {cells}", counts.len() ))); } - if l2.len() != sketch.rows { + let cached = sketch.row_l2_squared.as_slice(); + if cached.len() != sketch.rows { return Err(RmpEncodeError::Syntax(format!( - "ASAPv1 CountSketchWithL2Norm state: l2 length {} != rows {}", - l2.len(), + "ASAPv1 CountSketchWithL2Norm state: {} cached l2 accumulators over {} rows", + cached.len(), sketch.rows ))); } - if let Some(negative) = l2.iter().find(|&&value| value < 0) { - return Err(RmpEncodeError::Syntax(format!( - "ASAPv1 CountSketchWithL2Norm state: l2 accumulator {negative} is negative" - ))); + for (row, (counters, &held)) in counts.chunks_exact(sketch.cols).zip(cached).enumerate() { + let recomputed = row_capped_square_sum(counters); + if held != recomputed { + return Err(RmpEncodeError::Syntax(format!( + "ASAPv1 CountSketchWithL2Norm state: row {row}'s cached l2 accumulator {held} (l2 squared {}) is not the {recomputed} its counters give (l2 squared {})", + clamp_to_i64(held), + clamp_to_i64(recomputed) + ))); + } } - Ok((counts, l2)) + Ok(counts) } -/// Rebuilds one CountSketchWithL2Norm from a validated sub-payload. The lengths are -/// checked against the declared geometry before the matrix is built, so a -/// crafted `rows`/`cols` never drives an allocation. +/// Rebuilds one CountSketchWithL2Norm from a validated `counts` sub-payload, +/// recomputing each row's L2 accumulator. The length is checked against the +/// declared geometry before the matrix is built, so a crafted `rows`/`cols` +/// never drives an allocation. pub(crate) fn rebuild_layer( rows: usize, cols: usize, seed_index: usize, counts: &[i64], - l2: &[i64], ) -> Result, RmpDecodeError> { check_dimensions(rows, cols).map_err(RmpDecodeError::Uncategorized)?; let cells = rows.saturating_mul(cols); @@ -172,20 +221,14 @@ pub(crate) fn rebuild_layer( counts.len() ))); } - if l2.len() != rows { - return Err(RmpDecodeError::Uncategorized(format!( - "CountSketchWithL2Norm l2 length {} != rows {rows}", - l2.len() - ))); - } - if let Some(negative) = l2.iter().find(|&&value| value < 0) { - return Err(RmpDecodeError::Uncategorized(format!( - "CountSketchWithL2Norm l2 accumulator {negative} is negative" - ))); - } Ok(CountSketchWithL2Norm { counts: Vector2D::from_fn(rows, cols, |r, c| counts[r * cols + c]), - row_l2_squared: Vector1D::from_vec(l2.to_vec()), + row_l2_squared: Vector1D::from_vec( + counts + .chunks_exact(cols) + .map(row_capped_square_sum) + .collect(), + ), rows, cols, seed_index, @@ -197,18 +240,18 @@ pub(crate) fn rebuild_layer( // module, so this impl reads the private fields directly. impl CountSketchWithL2Norm { /// Serializes the sketch into an ASAPv1 MessagePack envelope - /// (kind_id `0x19 0x00`). The metadata is derived from the hasher's + /// (kind_id `0x19 0x01`). The metadata is derived from the hasher's /// [`HashProfile`], so it truthfully describes how the sketch was hashed. /// - /// Fails when the state disagrees with its own dimensions, when an `l2` - /// accumulator is negative, when the geometry needs more hash bits than + /// Fails when the state disagrees with its own dimensions, when a cached + /// L2 accumulator is not its row's sum of squares, when the geometry needs more hash bits than /// the 128-bit key hash carries, when the seed index is past the hasher's /// seed list, or when a dimension or the seed index overflows its `u32` /// metadata field. pub fn to_asapv1(&self) -> Result, RmpEncodeError> { check_dimensions(self.rows, self.cols).map_err(RmpEncodeError::Syntax)?; check_seed_index(self.seed_index, H::seed_list().len()).map_err(RmpEncodeError::Syntax)?; - let (counts, l2) = layer_state(self)?; + let counts = layer_state(self)?; let field = |name: &str, value: usize| { u32::try_from(value).map_err(|_| { RmpEncodeError::Syntax(format!( @@ -223,7 +266,6 @@ impl CountSketchWithL2Norm { ))?; let payload = rmp_serde::to_vec(&CountSketchWithL2NormPayload { counts: counts.to_vec(), - l2: l2.to_vec(), })?; Ok(envelope::encode( kind::COUNT_SKETCH_WITH_L2_NORM, @@ -236,14 +278,22 @@ impl CountSketchWithL2Norm { /// dimensions and the seed index are structural (they are properties of /// the stored sketch, not of the target), so they are echoed back into the /// expected metadata; the hash spec is pinned against this target. + /// + /// Reads the current kind `0x19 0x01` and the retired `0x19 0x00`, whose + /// `l2` is read and dropped; either way each row's accumulator is + /// recomputed from its counters. pub fn from_asapv1(bytes: &[u8]) -> Result { let (kind_id, metadata, payload) = envelope::split(bytes).map_err(RmpDecodeError::Uncategorized)?; - if kind_id != kind::COUNT_SKETCH_WITH_L2_NORM { - return Err(RmpDecodeError::Uncategorized(format!( - "CountSketchWithL2Norm kind_id mismatch: stored {kind_id:?}, expected {COUNT_SKETCH_WITH_L2_NORM:?}" - ))); - } + let carries_l2 = match kind_id { + kind::COUNT_SKETCH_WITH_L2_NORM => false, + kind::COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2 => true, + _ => { + return Err(RmpDecodeError::Uncategorized(format!( + "CountSketchWithL2Norm kind_id mismatch: stored {kind_id:?}, expected {COUNT_SKETCH_WITH_L2_NORM:?} or the retired {COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2:?}" + ))); + } + }; let meta: CountSketchWithL2NormMetadata = envelope::decode_block(metadata)?; if meta != count_sketch_with_l2_norm_metadata::(meta.seed_index, meta.rows, meta.cols) { return Err(RmpDecodeError::Uncategorized( @@ -252,13 +302,16 @@ impl CountSketchWithL2Norm { } check_seed_index(meta.seed_index as usize, meta.seed_list.len()) .map_err(RmpDecodeError::Uncategorized)?; - let p: CountSketchWithL2NormPayload = envelope::decode_block(payload)?; + let counts = if carries_l2 { + envelope::decode_block::(payload)?.counts + } else { + envelope::decode_block::(payload)?.counts + }; rebuild_layer::( meta.rows as usize, meta.cols as usize, meta.seed_index as usize, - &p.counts, - &p.l2, + &counts, ) } } @@ -288,13 +341,130 @@ mod tests { from_slice(metadata).expect("metadata") } - fn crafted(meta: &CountSketchWithL2NormMetadata, counts: Vec, l2: Vec) -> Vec { + fn crafted(meta: &CountSketchWithL2NormMetadata, counts: Vec) -> Vec { let metadata = rmp_serde::to_vec_named(meta).expect("metadata"); - let payload = - rmp_serde::to_vec(&CountSketchWithL2NormPayload { counts, l2 }).expect("payload"); + let payload = rmp_serde::to_vec(&CountSketchWithL2NormPayload { counts }).expect("payload"); envelope::encode(kind::COUNT_SKETCH_WITH_L2_NORM, &metadata, &payload) } + /// `sketch`'s bytes under the retired kind `0x19 0x00`, carrying `l2`. + fn carrying_l2(sketch: &CountSketchWithL2Norm, l2: Vec) -> Vec { + let bytes = sketch.to_asapv1().expect("serialize"); + let (_, metadata, _) = envelope::split(&bytes).expect("split"); + let payload = rmp_serde::to_vec(&CountSketchWithL2NormPayloadCarryingL2 { + counts: sketch.as_storage().as_slice().to_vec(), + l2: RetiredL2(l2), + }) + .expect("payload"); + envelope::encode( + kind::COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2, + metadata, + &payload, + ) + } + + /// The payload is `[counts]` alone: no `l2` rides the wire. + #[test] + fn count_sketch_with_l2_norm_payload_carries_no_l2() { + let encoded = populated().to_asapv1().expect("serialize"); + let (_, _, payload) = envelope::split(&encoded).expect("split"); + let fields: Vec = from_slice(payload).expect("payload array"); + assert_eq!(fields.len(), 1, "payload must be [counts]"); + } + + /// The retired `0x19 0x00` still decodes: its `l2` is read and dropped, + /// each row's accumulator recomputed from the counters, and the sketch + /// re-encodes under `0x19 0x01`. A carried `l2` that disagrees with the + /// counters is not trusted. + #[test] + fn count_sketch_with_l2_norm_retired_kind_decodes_and_recomputes_l2() { + let sketch = populated(); + let true_l2: Vec = sketch + .row_l2_squared + .as_slice() + .iter() + .map(|&v| clamp_to_i64(v)) + .collect(); + for l2 in [true_l2, vec![0, 1, 2], vec![-5]] { + let decoded = Std::from_asapv1(&carrying_l2(&sketch, l2)).expect("decode retired"); + assert_eq!( + decoded.as_storage().as_slice(), + sketch.as_storage().as_slice() + ); + assert_eq!( + decoded.row_l2_squared.as_slice(), + sketch.row_l2_squared.as_slice() + ); + assert_eq!( + decoded.to_asapv1().expect("re-encode"), + sketch.to_asapv1().expect("encode") + ); + } + } + + /// The encoder rejects a cached accumulator the decoder would not + /// recompute. + #[test] + fn count_sketch_with_l2_norm_rejects_a_stale_l2_cache() { + let mut sketch = populated(); + sketch.row_l2_squared[0] += 1; + let problem = sketch + .to_asapv1() + .expect_err("a stale cache must not serialize") + .to_string(); + assert!(problem.contains("row 0"), "got {problem}"); + } + + /// A retired `l2` that is not an integer array is still rejected. + #[test] + fn count_sketch_with_l2_norm_retired_kind_rejects_a_non_integer_l2() { + #[derive(Serialize)] + struct WithStringL2 { + counts: Vec, + l2: Vec, + } + let sketch = populated(); + let bytes = sketch.to_asapv1().expect("serialize"); + let (_, metadata, _) = envelope::split(&bytes).expect("split"); + let payload = rmp_serde::to_vec(&WithStringL2 { + counts: sketch.as_storage().as_slice().to_vec(), + l2: vec!["x".to_string(); 3], + }) + .expect("payload"); + let forged = envelope::encode( + kind::COUNT_SKETCH_WITH_L2_NORM_CARRYING_L2, + metadata, + &payload, + ); + assert!(Std::from_asapv1(&forged).is_err()); + } + + /// A row whose squares saturate the accumulator keeps it equal to the + /// recomputed value once a decrement brings the sum back below `i64::MAX`: + /// the capped-square accumulator never loses the exact sum. + #[test] + fn count_sketch_with_l2_norm_l2_stays_derivable_past_saturation() { + let mut sketch: CountSketchWithL2Norm = CountSketchWithL2Norm::with_dimensions(1, 2); + let delta = + |col: u32, count: i32| crate::octo_delta::CountSketchCellDelta { row: 0, col, count }; + // Column 0 reaches 2^32, whose square 2^64 saturates the row. + sketch.apply_delta(delta(0, 1 << 30)); + for _ in 0..3 { + sketch.apply_delta(delta(0, 1 << 30)); + } + sketch.apply_delta(delta(1, 1)); + assert_eq!(sketch.l2_squared(), i64::MAX as f64); + // Back to 2^31: the row's sum of squares is 2^62 + 1. + sketch.apply_delta(delta(0, -(1 << 30))); + sketch.apply_delta(delta(0, -(1 << 30))); + assert_eq!(sketch.row_l2_squared.as_slice(), [(1_i128 << 62) + 1]); + let decoded = Std::from_asapv1(&sketch.to_asapv1().expect("serialize")).expect("decode"); + assert_eq!( + decoded.row_l2_squared.as_slice(), + sketch.row_l2_squared.as_slice() + ); + } + /// More rows than the seed list has seeds is outside the wire-eligible /// subset, on both sides. #[test] @@ -308,7 +478,6 @@ mod tests { let bytes = crafted( &count_sketch_with_l2_norm_metadata::(0, rows as u32, 32), vec![0; rows * 32], - vec![0; rows], ); let problem = Std::from_asapv1(&bytes) .expect_err("rows past MATRIX_MAX_ROWS must be rejected") @@ -328,7 +497,7 @@ mod tests { let sketch = populated(); let encoded = sketch.to_asapv1().expect("serialize CountSketchWithL2Norm"); assert!(encoded.starts_with(b"ASAPv1")); - assert_eq!(&encoded[7..10], &[2u8, 0x19, 0x00]); // kind_id_len=2, kind_id=[0x19,0x00] + assert_eq!(&encoded[7..10], &[2u8, 0x19, 0x01]); // kind_id_len=2, kind_id=[0x19,0x01] let meta = metadata_of(&encoded); assert_eq!(meta.metadata_version, 1); @@ -499,23 +668,13 @@ mod tests { #[test] fn count_sketch_with_l2_norm_rejects_crafted_geometry() { let meta = count_sketch_with_l2_norm_metadata::(0, 4, 0); - assert!(Std::from_asapv1(&crafted(&meta, Vec::new(), vec![0; 4])).is_err()); + assert!(Std::from_asapv1(&crafted(&meta, Vec::new())).is_err()); let huge = count_sketch_with_l2_norm_metadata::(0, 1, 1 << 24); assert!( - Std::from_asapv1(&crafted(&huge, vec![1, 2, 3], vec![0; 1])).is_err(), + Std::from_asapv1(&crafted(&huge, vec![1, 2, 3])).is_err(), "counts length must match rows*cols" ); - - let meta = count_sketch_with_l2_norm_metadata::(0, 2, 4); - assert!( - Std::from_asapv1(&crafted(&meta, vec![0; 8], vec![0; 5])).is_err(), - "l2 length must match rows" - ); - assert!( - Std::from_asapv1(&crafted(&meta, vec![0; 8], vec![-1, 0])).is_err(), - "a negative l2 accumulator is not a state the algorithm reaches" - ); } /// An unpopulated matrix carries dimensions its cells do not match. @@ -624,7 +783,6 @@ mod tests { let bytes = crafted( &count_sketch_with_l2_norm_metadata::(0, rows as u32, cols as u32), vec![0; rows * cols], - vec![0; rows], ); let problem = Std::from_asapv1(&bytes) .expect_err("a geometry past the hash budget must not decode") @@ -668,7 +826,6 @@ mod tests { let bytes = crafted( &count_sketch_with_l2_norm_metadata::(past as u32, 2, 8), vec![0; 16], - vec![0; 2], ); let problem = Std::from_asapv1(&bytes) .expect_err("a seed index past the seed list must not decode") @@ -700,8 +857,8 @@ mod tests { ); } - /// Known counts, L2 accumulators and seed index serialize to the - /// `asapv1_golden/` bytes. + /// Known counts and seed index serialize to the `asapv1_golden/` bytes. + /// Row 0's squares sum to 5368610562; row 1's saturate at `i64::MAX`. #[test] fn known_state_matches_golden() { use crate::asapv1::envelope::hex; @@ -711,15 +868,39 @@ mod tests { [[127, 128, 65535, -32768], [-32, -33, -2147483648, i64::MIN]]; let sketch: CountSketchWithL2Norm = CountSketchWithL2Norm { counts: Vector2D::from_fn(2, 4, |r, c| COUNTS[r][c]), - row_l2_squared: Vector1D::from_vec(vec![65536, i64::MAX]), + row_l2_squared: Vector1D::from_vec( + COUNTS + .iter() + .map(|row| row_capped_square_sum(row)) + .collect(), + ), rows: 2, cols: 4, seed_index: 7, _hasher: PhantomData, }; let golden = hex(include_str!( - "../../../asapv1_golden/count_l2hh_2x4_seed7.hex" + "../../../asapv1_golden/count_l2hh_2x4_seed7_no_l2.hex" )); assert_eq!(sketch.to_asapv1().unwrap(), golden); + + // The retired fixture carries `l2 = [65536, i64::MAX]`, which is not + // row 0's sum of squares; decoding drops it and recomputes. + let retired = hex(include_str!( + "../../../asapv1_golden/count_l2hh_2x4_seed7.hex" + )); + let decoded = Std::from_asapv1(&retired).expect("decode retired"); + assert_eq!( + decoded.row_l2_squared.as_slice(), + sketch.row_l2_squared.as_slice() + ); + let l2: Vec = decoded + .row_l2_squared + .as_slice() + .iter() + .map(|&v| clamp_to_i64(v)) + .collect(); + assert_eq!(l2, [5_368_610_562, i64::MAX]); + assert_eq!(decoded.to_asapv1().unwrap(), golden); } } diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index 9058f5f0..8ab7a767 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -2,8 +2,8 @@ One `proptest` law per line. Each compares the sketch against an answer computed without it; what that answer is comes first in every section. One -file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 356 laws; -with `--all-features`, 407, adding the three **experimental** modules. Every +file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 357 laws; +with `--all-features`, 408, adding the three **experimental** modules. Every law was mutation-checked: break the implementation, the law goes red. Shared generators, `grid` and `round_trip!` live in `support.rs`. @@ -402,7 +402,7 @@ Compared against: an exact `HashMap` count. Every bound is deterministic. - wire - ASAPv1 at `edge_dimension` keeps `min_count`, 64 `(estimate, error)`; after merging a 1-counter summary whose ceiling no counter holds -## topk_wrappers.rs — 15 +## topk_wrappers.rs — 16 Path laws compare two real implementations: the wrapper's matrix must equal the bare sketch's. @@ -419,9 +419,10 @@ Path laws compare two real implementations: the wrapper's matrix must equal the - every merged resident: `count == merged estimate(k)` - room for the union: merged keys `== a.keys + b.keys` - `CountSketchWithL2Norm`, via hash-taking entry points - - `update_weighted_with_hash_without_l2` writes the cells `CountSketchWithL2Norm::cell_for_row` names, with the row sign + - `update_weighted_with_hash` writes the cells `CountSketchWithL2Norm::cell_for_row` names, with the row sign; its L2 accumulators equal the ones `merge` and the ASAPv1 decoder recompute - `estimate_with_hash == median(sign-corrected cells)` - `l2_squared() == median over rows of sum of squares`, every update + - weights up to `2^33` into saturation and back to zero: `l2_squared() == median over rows of min(sum of squares, i64::MAX)` after every update, equal to the `merge` and ASAPv1-decode recomputations - merge adds cells and recomputes L2 from them - wire - `CountMinSketchWithHeap`, `CountSketchWithHeap` at `edge_rows` x `edge_dimension`: 64 estimates and heap `(key, count)` kept diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index ccc6dbe3..c0cd2017 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -119,15 +119,27 @@ const GOLDEN_DELTA_EMPTY: &str = include_str!("../asapv1_golden/delta_result_emp const GOLDEN_COCO: &str = include_str!("../asapv1_golden/coco_3x7.hex"); const GOLDEN_ELASTIC: &str = include_str!("../asapv1_golden/elastic_4b_2x4.hex"); const GOLDEN_ELASTIC_STALE: &str = include_str!("../asapv1_golden/elastic_4b_2x4_stale.hex"); -const GOLDEN_UNIVMON_STR: &str = include_str!("../asapv1_golden/univmon_str_l3_2x4_h5.hex"); -const GOLDEN_UNIVMON_I64: &str = include_str!("../asapv1_golden/univmon_i64_l3_2x4_h5.hex"); -const GOLDEN_UNIVMON_EMPTY: &str = include_str!("../asapv1_golden/univmon_empty_l3_2x4_h5.hex"); -const GOLDEN_L2HH: &str = include_str!("../asapv1_golden/count_l2hh_2x4_seed7.hex"); +// The retired `l2`-carrying kinds `10 00`, `19 00`, `11 00` and `07 04`, +// decode-only; each has a `*_no_l2` twin under the current kind. +const GOLDEN_UNIVMON_STR_RETIRED: &str = include_str!("../asapv1_golden/univmon_str_l3_2x4_h5.hex"); +const GOLDEN_UNIVMON_I64_RETIRED: &str = include_str!("../asapv1_golden/univmon_i64_l3_2x4_h5.hex"); +const GOLDEN_UNIVMON_EMPTY_RETIRED: &str = + include_str!("../asapv1_golden/univmon_empty_l3_2x4_h5.hex"); +const GOLDEN_L2HH_RETIRED: &str = include_str!("../asapv1_golden/count_l2hh_2x4_seed7.hex"); +const GOLDEN_PYRAMID_RETIRED: &str = + include_str!("../asapv1_golden/univmon_optimized_l5_e2_3x8_1x4_h6.hex"); +const GOLDEN_HYDRA_UNIVMON_RETIRED: &str = include_str!("../asapv1_golden/hydra_univmon_1x2.hex"); +// The current kinds `10 01`, `19 01`, `11 01` and `07 05`. +const GOLDEN_UNIVMON_STR: &str = include_str!("../asapv1_golden/univmon_str_l3_2x4_h5_no_l2.hex"); +const GOLDEN_UNIVMON_I64: &str = include_str!("../asapv1_golden/univmon_i64_l3_2x4_h5_no_l2.hex"); +const GOLDEN_UNIVMON_EMPTY: &str = + include_str!("../asapv1_golden/univmon_empty_l3_2x4_h5_no_l2.hex"); +const GOLDEN_L2HH: &str = include_str!("../asapv1_golden/count_l2hh_2x4_seed7_no_l2.hex"); const GOLDEN_HYDRA_KLL: &str = include_str!("../asapv1_golden/hydra_kll_2x2_k200.hex"); const GOLDEN_HYDRA_CM: &str = include_str!("../asapv1_golden/hydra_cm_2x2_counter_2x2.hex"); const GOLDEN_HYDRA_CS: &str = include_str!("../asapv1_golden/hydra_cs_2x2_counter_2x2.hex"); const GOLDEN_HYDRA_HLL: &str = include_str!("../asapv1_golden/hydra_hll_1x2_p14.hex"); -const GOLDEN_HYDRA_UNIVMON: &str = include_str!("../asapv1_golden/hydra_univmon_1x2.hex"); +const GOLDEN_HYDRA_UNIVMON: &str = include_str!("../asapv1_golden/hydra_univmon_1x2_no_l2.hex"); const GOLDEN_CLASSIC_ADMISSION: &str = include_str!("../asapv1_golden/hll_classic_p12_admission.hex"); const GOLDEN_CMS_ADMISSION: &str = @@ -155,7 +167,7 @@ const GOLDEN_SPACE_SAVING_NOTOTAL_BYTES: &str = #[cfg(feature = "experimental")] const GOLDEN_UNIFORM: &str = include_str!("../asapv1_golden/uniform_sampling_r0p25.hex"); const GOLDEN_PYRAMID: &str = - include_str!("../asapv1_golden/univmon_optimized_l5_e2_3x8_1x4_h6.hex"); + include_str!("../asapv1_golden/univmon_optimized_l5_e2_3x8_1x4_h6_no_l2.hex"); const GOLDEN_UNIVMON_Q: &str = include_str!("../asapv1_golden/univmon_q_i32_l3_c8_r3.hex"); const GOLDEN_EHSL_CM: &str = include_str!("../asapv1_golden/ehsketchlist_cm_i32_fast_2x4.hex"); const GOLDEN_EHSL_DD: &str = include_str!("../asapv1_golden/ehsketchlist_ddsketch_a001.hex"); @@ -1523,6 +1535,37 @@ fn univmon_empty_l3_2x4_h5_matches_golden() { assert_univmon_golden(GOLDEN_UNIVMON_EMPTY, &univmon_empty()); } +/// Each retired `l2`-carrying fixture decodes, its `l2` dropped and +/// recomputed from the counters, and re-encodes as its `*_no_l2` twin. +#[test] +fn retired_l2_fixtures_decode_to_their_no_l2_twins() { + for (retired, twin) in [ + (GOLDEN_UNIVMON_STR_RETIRED, GOLDEN_UNIVMON_STR), + (GOLDEN_UNIVMON_I64_RETIRED, GOLDEN_UNIVMON_I64), + (GOLDEN_UNIVMON_EMPTY_RETIRED, GOLDEN_UNIVMON_EMPTY), + ] { + let decoded = UnivMon::from_asapv1(&decode_hex(retired)).expect("decode retired"); + assert_eq!(decoded.to_asapv1().expect("re-encode"), decode_hex(twin)); + } + let l2hh: CountSketchWithL2Norm = + CountSketchWithL2Norm::from_asapv1(&decode_hex(GOLDEN_L2HH_RETIRED)).expect("retired"); + assert_eq!( + l2hh.to_asapv1().expect("re-encode"), + decode_hex(GOLDEN_L2HH) + ); + let pyramid = + UnivMonTwoTier::from_asapv1(&decode_hex(GOLDEN_PYRAMID_RETIRED)).expect("retired"); + assert_eq!( + pyramid.to_asapv1().expect("re-encode"), + decode_hex(GOLDEN_PYRAMID) + ); + let hydra = Hydra::from_asapv1(&decode_hex(GOLDEN_HYDRA_UNIVMON_RETIRED)).expect("retired"); + assert_eq!( + hydra.to_asapv1().expect("re-encode"), + decode_hex(GOLDEN_HYDRA_UNIVMON) + ); +} + // --------------------------------------------------------------------------- // CountSketchWithL2Norm: golden decodes to the known counts and round-trips. // Encoding the known state is checked in `src/sketches/countsketch_l2_norm/wire.rs`. diff --git a/tests/e2e/wire.rs b/tests/e2e/wire.rs index f9843e5c..3fc3d815 100644 --- a/tests/e2e/wire.rs +++ b/tests/e2e/wire.rs @@ -325,7 +325,7 @@ fn the_univmon_family_envelopes_round_trip_and_name_their_kinds() { } let um_bytes = um.to_asapv1().expect("encode UnivMon"); - assert_kind(&um_bytes, 0x10, 0x00, "UnivMon"); + assert_kind(&um_bytes, 0x10, 0x01, "UnivMon"); let decoded_um = UnivMon::from_asapv1(&um_bytes).expect("decode UnivMon"); assert_eq!(decoded_um.estimate_l1(), um.estimate_l1(), "UnivMon L1"); assert_eq!(decoded_um.estimate_l2(), um.estimate_l2(), "UnivMon L2"); @@ -336,7 +336,7 @@ fn the_univmon_family_envelopes_round_trip_and_name_their_kinds() { ); let pyramid_bytes = pyramid.to_asapv1().expect("encode UnivMonTwoTier"); - assert_kind(&pyramid_bytes, 0x11, 0x00, "UnivMonTwoTier"); + assert_kind(&pyramid_bytes, 0x11, 0x01, "UnivMonTwoTier"); let decoded_pyramid = UnivMonTwoTier::from_asapv1(&pyramid_bytes).expect("decode UnivMonTwoTier"); assert_eq!( diff --git a/tests/pbt/topk_wrappers.rs b/tests/pbt/topk_wrappers.rs index eec656bc..36d8512d 100644 --- a/tests/pbt/topk_wrappers.rs +++ b/tests/pbt/topk_wrappers.rs @@ -466,7 +466,9 @@ proptest! { // ===== CountSketchWithL2Norm lands where the shared cell map says ===== /// An OctoSketch worker addresses cells with `CountSketchWithL2Norm::cell_for_row` and its - /// parent with the sketch's own insert, so the two must name one cell. + /// parent with the sketch's own insert, so the two must name one cell. The + /// running L2 accumulators end where a recomputation from the counters + /// lands: the one `merge` performs, and the one the ASAPv1 decoder performs. #[test] fn count_sketch_with_l2_norm_writes_the_cells_the_shared_map_names( rows in 1usize..6, @@ -476,9 +478,18 @@ proptest! { let mut sketch = CountSketchWithL2Norm::::with_dimensions(rows, cols); let mask_bits = sketch.mask_bits(); for (hash, delta) in &updates { - sketch.update_weighted_with_hash_without_l2(*hash, *delta); + sketch.update_weighted_with_hash(*hash, *delta); } + let mut recomputed = CountSketchWithL2Norm::::with_dimensions(rows, cols); + recomputed.merge(&sketch); + prop_assert_eq!(recomputed.l2_squared(), sketch.l2_squared()); + let decoded = CountSketchWithL2Norm::::from_asapv1( + &sketch.to_asapv1().expect("serialize"), + ) + .expect("decode"); + prop_assert_eq!(decoded.l2_squared(), sketch.l2_squared()); + let mapped = mapped_cells(rows, cols, mask_bits, &updates); for (r, row) in mapped.iter().enumerate() { for (c, cell) in row.iter().enumerate() { @@ -538,6 +549,58 @@ proptest! { } } + /// A first weight of 2^33 into the empty sketch saturates every row (its + /// square is 2^66), and further weights up to 2^33 move rows around and + /// past `i64::MAX`; the same updates negated in reverse bring every + /// counter back to zero. After + /// every step the running accumulator equals `min(sum(counts[row]^2), + /// i64::MAX)` recomputed from the counters, and the recomputation `merge` + /// and the ASAPv1 decoder perform. + #[test] + fn the_l2_accumulator_tracks_rows_into_saturation_and_back( + rows in 1usize..6, + cols in edge_dimension(), + first_hash in any::(), + updates in prop::collection::vec( + (any::(), -(1i64 << 33)..(1i64 << 33)), + 0..48, + ), + ) { + let updates: Vec<(u128, i64)> = + std::iter::once((first_hash, 1i64 << 33)).chain(updates).collect(); + let mut sketch = CountSketchWithL2Norm::::with_dimensions(rows, cols); + let there = updates.iter().copied(); + let back = updates.iter().rev().map(|&(hash, weight)| (hash, -weight)); + let mut saturated = false; + for (step, (hash, weight)) in there.chain(back).enumerate() { + sketch.update_weighted_with_hash(hash, weight); + let capped: Vec = (0..rows) + .map(|r| { + let sum: i128 = (0..cols) + .map(|c| { + let v = i128::from(sketch.as_storage().query_one_counter(r, c)); + v * v + }) + .sum(); + sum.min(i128::from(i64::MAX)) as i64 as f64 + }) + .collect(); + saturated |= capped.contains(&(i64::MAX as f64)); + prop_assert_eq!(sketch.l2_squared(), median(capped), "after step {}", step); + let mut recomputed = + CountSketchWithL2Norm::::with_dimensions(rows, cols); + recomputed.merge(&sketch); + prop_assert_eq!(recomputed.l2_squared(), sketch.l2_squared()); + let decoded = CountSketchWithL2Norm::::from_asapv1( + &sketch.to_asapv1().expect("serialize"), + ) + .expect("decode"); + prop_assert_eq!(decoded.l2_squared(), sketch.l2_squared()); + } + prop_assert!(saturated); + prop_assert_eq!(sketch.l2_squared(), 0.0); + } + #[test] fn merging_count_sketch_with_l2_norm_adds_the_cells_and_re_derives_the_l2( rows in 1usize..6,