diff --git a/CHANGELOG.md b/CHANGELOG.md index c58a26f3..863f6352 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -120,6 +120,14 @@ signals a backwards-compatible change. - **KLL merges require the same `k` and `m`.** `Kll::merge` and `KllDynamic::merge` panic when either differs, where they merged before; `Hydra::merge` and `EhBucketSketch::merge` return `Err` for such KLL states. + +### Docs + +- **A receiver of `f64` deltas matches the producer only up to rounding.** + For Count-Min, Count Sketch and the two heap kinds, the wire format + (§3.23), the `DeltaCounter` docs and the API pages now state the bound; + `i32` / `i64` receivers match exactly. Tests cover both. No behavior change. + ### Fixed - **`Kll::update` regains most of its pre-0.4.0 speed.** With the lazily grown diff --git a/docs/api/api_count_min_sketch.md b/docs/api/api_count_min_sketch.md index 7b3338bc..2c9c8e7a 100644 --- a/docs/api/api_count_min_sketch.md +++ b/docs/api/api_count_min_sketch.md @@ -115,6 +115,8 @@ to each. `new` and the decoder refuse anything else. `merge_delta` adds the delta to the counters; a delta of other dimensions or admission rate, or an integer add that overflows, fails and leaves the sketch unchanged. +With `f64` counters a receiver equals the producer only up to floating-point rounding: each `merge_delta` adds at most `2^-53 * |result|` of rounding to whatever error the delta already carries (at most `2^-53 * |delta|` when the producer formed it as `new - old`), and the errors accumulate over applies (bound in `docs/asapv1_wire_format.md` §3.23). With `i32` / `i64` counters it equals the producer exactly. + ## Admission rate ```rust diff --git a/docs/api/api_count_min_sketch_with_heap.md b/docs/api/api_count_min_sketch_with_heap.md index 7a2faaed..7b487ce1 100644 --- a/docs/api/api_count_min_sketch_with_heap.md +++ b/docs/api/api_count_min_sketch_with_heap.md @@ -136,6 +136,8 @@ replaces the heap with the delta's, its key type included; a delta of other dimensions or heap capacity, or an integer add that overflows, fails and leaves the sketch unchanged. +With `f64` counters a receiver equals the producer only up to floating-point rounding: each `merge_delta` adds at most `2^-53 * |result|` of rounding to whatever error the delta already carries (at most `2^-53 * |delta|` when the producer formed it as `new - old`), and the errors accumulate over applies (bound in `docs/asapv1_wire_format.md` §3.23). With `i32` / `i64` counters it equals the producer exactly. + ## Examples ```rust diff --git a/docs/api/api_count_sketch.md b/docs/api/api_count_sketch.md index dae26a0b..84d8c9fd 100644 --- a/docs/api/api_count_sketch.md +++ b/docs/api/api_count_sketch.md @@ -132,6 +132,8 @@ added to each. `new` and the decoder refuse anything else. `merge_delta` adds the delta to the counters; a delta of other dimensions or admission rate, or an integer add that overflows, fails and leaves the sketch unchanged. +With `f64` counters a receiver equals the producer only up to floating-point rounding: each `merge_delta` adds at most `2^-53 * |result|` of rounding to whatever error the delta already carries (at most `2^-53 * |delta|` when the producer formed it as `new - old`), and the errors accumulate over applies (bound in `docs/asapv1_wire_format.md` §3.23). With `i32` / `i64` counters it equals the producer exactly. + ## Admission rate ```rust diff --git a/docs/api/api_count_sketch_with_heap.md b/docs/api/api_count_sketch_with_heap.md index 42022ed6..740d5489 100644 --- a/docs/api/api_count_sketch_with_heap.md +++ b/docs/api/api_count_sketch_with_heap.md @@ -140,6 +140,8 @@ replaces the heap with the delta's, its key type included; a delta of other dimensions or heap capacity, or an integer add that overflows, fails and leaves the sketch unchanged. +With `f64` counters a receiver equals the producer only up to floating-point rounding: each `merge_delta` adds at most `2^-53 * |result|` of rounding to whatever error the delta already carries (at most `2^-53 * |delta|` when the producer formed it as `new - old`), and the errors accumulate over applies (bound in `docs/asapv1_wire_format.md` §3.23). With `i32` / `i64` counters it equals the producer exactly. + ## Examples ```rust diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index 1f08b2d0..51f0efe1 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1413,6 +1413,8 @@ Rules 3 through 7 are enforced on the **encode** side too, so the format never e 2. The state's metadata equals the delta's, key for key, `key_type` excepted for CountMinSketchWithHeap and CountSketchWithHeap. 3. An integer add that overflows rejects the delta: a `counter_type` cell, or DDSketch's total count (§3.8 rule 6). +**Exactness of a receiver.** A receiver fed a producer's deltas holds the producer's cells exactly when `counter_type` is `"i32"` or `"i64"`: integer adds are exact, and an add that would overflow rejects the delta. When `counter_type` is `"f64"` (Count-Min, Count Sketch, CountMinSketchWithHeap, CountSketchWithHeap) it holds them only up to floating-point rounding, and exactly only in special cases such as every cell and delta being an integer of magnitude below `2^53`. Each apply is one IEEE-754 add rounded to nearest, off by at most `u * |result|` with `u = 2^-53`. For a receiver that starts equal to the producer and applies `n` deltas to a cell, the cell differs from the producer's by at most the sum over `i` of `e_i + u * |r_i|`, where `r_i` is the receiver's cell after apply `i` and `e_i` is how far delta `i` is from the producer's true change in that cell; this holds provided every cell, delta and sum is finite. How large `e_i` is depends on how the producer forms the delta: one that forms `d_i` as `new - old` of its own `f64` cells has `e_i <= u * |d_i|`, so the sum is at most `2 * n * u * M` where `M` bounds every `|d_i|` and `|r_i|`; one that accumulates updates in a separate `f64` buffer carries that buffer's own summation error in `e_i`. Equality is not guaranteed, even for one apply (`old = 1.0`, `new = 1e-17`: the delta rounds to `-1.0`, the receiver holds `0.0`), and `f64` adds are not associative, so the order of applies matters. The heap of the two heap kinds is replaced, not added, and carries no such error. A consumer needing cell-for-cell equality uses `"i32"` or `"i64"` counters. + ### 3.24: CountMinSketchWithSpaceSaving payload (`0x1e 0x01`) `CountMinSketchWithSpaceSaving` is a Count-Min sketch (§3.2) paired with a Space-Saving summary (§3.5). Every update reaches both. A top-k query ranks the summary's monitored keys by their Count-Min estimate: the summary nominates the candidates and the matrix counts them. Both are **inlined**, so the payload is Count-Min's `counts` followed by §3.5's payload, its `counts` renamed `summary_counts`: diff --git a/src/asapv1/delta.rs b/src/asapv1/delta.rs index 07f2929e..6073e007 100644 --- a/src/asapv1/delta.rs +++ b/src/asapv1/delta.rs @@ -7,6 +7,16 @@ use std::ops::RangeBounds; /// A counter a delta adds to. An integer add that overflows has no result; a /// float one always has. +/// +/// Integer adds are exact, so a receiver of `i32` or `i64` deltas equals the +/// producer cell for cell. An `f64` add rounds once to nearest, off by at most +/// `2^-53 * |result|`, so after `n` applies a cell differs from the producer's +/// by at most the sum of `e_i + 2^-53 * |r_i|` (finite values), where `r_i` is +/// the receiver's cell after apply `i` and `e_i` is how far delta `i` is from +/// the producer's true change. A producer forming each delta as `new - old` of +/// its own `f64` cells has `e_i <= 2^-53 * |delta_i|`; one summing updates in a +/// separate buffer has that buffer's summation error. Not guaranteed equal +/// (`docs/asapv1_wire_format.md` §3.23). pub trait DeltaCounter: Copy + PartialEq + Default { /// `self + delta`. fn checked_add(self, delta: Self) -> Option; diff --git a/src/sketches/countminsketch/delta.rs b/src/sketches/countminsketch/delta.rs index 47784bb4..3fe06d58 100644 --- a/src/sketches/countminsketch/delta.rs +++ b/src/sketches/countminsketch/delta.rs @@ -199,4 +199,76 @@ mod tests { CountMinSketch::, RegularPath>::from_storage(Vector2D::unfilled(2, 3)); assert!(unfilled.merge_delta(&delta(vec![0], vec![1])).is_err()); } + + /// Unit roundoff of `f64` under round-to-nearest: one add or subtract is off + /// by at most `u * |result|`. + const U: f64 = f64::EPSILON / 2.0; + + /// A receiver holding `old` after one `f64` delta `new - old`, the way a + /// producer forming the delta from its own cells would send it. + fn receive_one(old: f64, new: f64) -> f64 { + let delta = + CountMinSketchDelta::::new(1, 1, vec![0], vec![new - old]).unwrap(); + let mut receiver = CountMinSketch::, RegularPath>::from_storage( + Vector2D::from_fn(1, 1, |_, _| old), + ); + receiver.merge_delta(&delta).unwrap(); + receiver.as_storage().as_slice()[0] + } + + #[test] + fn f64_receiver_differs_from_producer_within_the_documented_bound() { + // old = 1, new = 1e-17: `new - old` rounds to -1, so old + delta = 0. + let received = receive_one(1.0, 1e-17); + assert_ne!(received, 1e-17); + assert!((received - 1e-17).abs() <= U * 1.0 + U * received.abs()); + + // 1000 applies on a cell whose value jumps between magnitudes 1e-20 and + // 1e3 with mixed signs (seeded): `new - old` then often rounds, and the + // error accumulates. + let mut receiver = CountMinSketch::, RegularPath>::from_storage( + Vector2D::from_fn(1, 1, |_, _| 0.0), + ); + let (mut producer, mut bound, mut state) = (0.0_f64, 0.0_f64, 12345_u64); + let mut next_unit = || { + state = state + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + (state >> 11) as f64 / (1_u64 << 53) as f64 + }; + for _ in 0..1000 { + let magnitude = 10_f64.powf(-20.0 + 23.0 * next_unit()); + let new = if next_unit() < 0.5 { + -magnitude + } else { + magnitude + }; + let d = new - producer; + if d == 0.0 { + continue; // a delta is never zero + } + let delta = + CountMinSketchDelta::::new(1, 1, vec![0], vec![d]).unwrap(); + receiver.merge_delta(&delta).unwrap(); + let received = receiver.as_storage().as_slice()[0]; + bound += U * d.abs() + U * received.abs(); + producer = new; + } + let error = (receiver.as_storage().as_slice()[0] - producer).abs(); + assert!(error > 0.0, "the walk must incur rounding error"); + assert!(error <= bound, "error {error} exceeds bound {bound}"); + } + + #[test] + fn integer_receiver_equals_producer_exactly() { + let (old, new) = ([1_i64, 1 << 53, 7], [0_i64, (1 << 53) + 1, 7_000_000_000]); + let deltas: Vec = old.iter().zip(&new).map(|(o, n)| n - o).collect(); + let delta = + CountMinSketchDelta::::new(1, 3, vec![0, 1, 2], deltas).unwrap(); + let mut receiver = CountMinSketch::, RegularPath>::from_storage( + Vector2D::from_fn(1, 3, |_, c| old[c]), + ); + receiver.merge_delta(&delta).unwrap(); + assert_eq!(receiver.as_storage().as_slice(), &new); + } }