Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,14 @@ signals a backwards-compatible change.
- **KLL merges require the same `k` and `m`.** `Kll::merge` and
`KllDynamic::merge` panic when either differs, where they merged before;
`Hydra::merge` and `EhBucketSketch::merge` return `Err` for such KLL states.

### Docs

- **A receiver of `f64` deltas matches the producer only up to rounding.**
For Count-Min, Count Sketch and the two heap kinds, the wire format
(§3.23), the `DeltaCounter` docs and the API pages now state the bound;
`i32` / `i64` receivers match exactly. Tests cover both. No behavior change.

### Fixed

- **`Kll::update` regains most of its pre-0.4.0 speed.** With the lazily grown
Expand Down
2 changes: 2 additions & 0 deletions docs/api/api_count_min_sketch.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,8 @@ to each. `new` and the decoder refuse anything else. `merge_delta` adds the
delta to the counters; a delta of other dimensions or admission rate, or an
integer add that overflows, fails and leaves the sketch unchanged.

With `f64` counters a receiver equals the producer only up to floating-point rounding: each `merge_delta` adds at most `2^-53 * |result|` of rounding to whatever error the delta already carries (at most `2^-53 * |delta|` when the producer formed it as `new - old`), and the errors accumulate over applies (bound in `docs/asapv1_wire_format.md` §3.23). With `i32` / `i64` counters it equals the producer exactly.

## Admission rate

```rust
Expand Down
2 changes: 2 additions & 0 deletions docs/api/api_count_min_sketch_with_heap.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,8 @@ replaces the heap with the delta's, its key type included; a delta of other
dimensions or heap capacity, or an integer add that overflows, fails and leaves
the sketch unchanged.

With `f64` counters a receiver equals the producer only up to floating-point rounding: each `merge_delta` adds at most `2^-53 * |result|` of rounding to whatever error the delta already carries (at most `2^-53 * |delta|` when the producer formed it as `new - old`), and the errors accumulate over applies (bound in `docs/asapv1_wire_format.md` §3.23). With `i32` / `i64` counters it equals the producer exactly.

## Examples

```rust
Expand Down
2 changes: 2 additions & 0 deletions docs/api/api_count_sketch.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,8 @@ added to each. `new` and the decoder refuse anything else. `merge_delta` adds
the delta to the counters; a delta of other dimensions or admission rate, or an
integer add that overflows, fails and leaves the sketch unchanged.

With `f64` counters a receiver equals the producer only up to floating-point rounding: each `merge_delta` adds at most `2^-53 * |result|` of rounding to whatever error the delta already carries (at most `2^-53 * |delta|` when the producer formed it as `new - old`), and the errors accumulate over applies (bound in `docs/asapv1_wire_format.md` §3.23). With `i32` / `i64` counters it equals the producer exactly.

## Admission rate

```rust
Expand Down
2 changes: 2 additions & 0 deletions docs/api/api_count_sketch_with_heap.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,8 @@ replaces the heap with the delta's, its key type included; a delta of other
dimensions or heap capacity, or an integer add that overflows, fails and leaves
the sketch unchanged.

With `f64` counters a receiver equals the producer only up to floating-point rounding: each `merge_delta` adds at most `2^-53 * |result|` of rounding to whatever error the delta already carries (at most `2^-53 * |delta|` when the producer formed it as `new - old`), and the errors accumulate over applies (bound in `docs/asapv1_wire_format.md` §3.23). With `i32` / `i64` counters it equals the producer exactly.

## Examples

```rust
Expand Down
2 changes: 2 additions & 0 deletions docs/asapv1_wire_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -1413,6 +1413,8 @@ Rules 3 through 7 are enforced on the **encode** side too, so the format never e
2. The state's metadata equals the delta's, key for key, `key_type` excepted for CountMinSketchWithHeap and CountSketchWithHeap.
3. An integer add that overflows rejects the delta: a `counter_type` cell, or DDSketch's total count (§3.8 rule 6).

**Exactness of a receiver.** A receiver fed a producer's deltas holds the producer's cells exactly when `counter_type` is `"i32"` or `"i64"`: integer adds are exact, and an add that would overflow rejects the delta. When `counter_type` is `"f64"` (Count-Min, Count Sketch, CountMinSketchWithHeap, CountSketchWithHeap) it holds them only up to floating-point rounding, and exactly only in special cases such as every cell and delta being an integer of magnitude below `2^53`. Each apply is one IEEE-754 add rounded to nearest, off by at most `u * |result|` with `u = 2^-53`. For a receiver that starts equal to the producer and applies `n` deltas to a cell, the cell differs from the producer's by at most the sum over `i` of `e_i + u * |r_i|`, where `r_i` is the receiver's cell after apply `i` and `e_i` is how far delta `i` is from the producer's true change in that cell; this holds provided every cell, delta and sum is finite. How large `e_i` is depends on how the producer forms the delta: one that forms `d_i` as `new - old` of its own `f64` cells has `e_i <= u * |d_i|`, so the sum is at most `2 * n * u * M` where `M` bounds every `|d_i|` and `|r_i|`; one that accumulates updates in a separate `f64` buffer carries that buffer's own summation error in `e_i`. Equality is not guaranteed, even for one apply (`old = 1.0`, `new = 1e-17`: the delta rounds to `-1.0`, the receiver holds `0.0`), and `f64` adds are not associative, so the order of applies matters. The heap of the two heap kinds is replaced, not added, and carries no such error. A consumer needing cell-for-cell equality uses `"i32"` or `"i64"` counters.

### 3.24: CountMinSketchWithSpaceSaving payload (`0x1e 0x01`)

`CountMinSketchWithSpaceSaving` is a Count-Min sketch (§3.2) paired with a Space-Saving summary (§3.5). Every update reaches both. A top-k query ranks the summary's monitored keys by their Count-Min estimate: the summary nominates the candidates and the matrix counts them. Both are **inlined**, so the payload is Count-Min's `counts` followed by §3.5's payload, its `counts` renamed `summary_counts`:
Expand Down
10 changes: 10 additions & 0 deletions src/asapv1/delta.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ use std::ops::RangeBounds;

/// A counter a delta adds to. An integer add that overflows has no result; a
/// float one always has.
///
/// Integer adds are exact, so a receiver of `i32` or `i64` deltas equals the
/// producer cell for cell. An `f64` add rounds once to nearest, off by at most
/// `2^-53 * |result|`, so after `n` applies a cell differs from the producer's
/// by at most the sum of `e_i + 2^-53 * |r_i|` (finite values), where `r_i` is
/// the receiver's cell after apply `i` and `e_i` is how far delta `i` is from
/// the producer's true change. A producer forming each delta as `new - old` of
/// its own `f64` cells has `e_i <= 2^-53 * |delta_i|`; one summing updates in a
/// separate buffer has that buffer's summation error. Not guaranteed equal
/// (`docs/asapv1_wire_format.md` §3.23).
pub trait DeltaCounter: Copy + PartialEq + Default {
/// `self + delta`.
fn checked_add(self, delta: Self) -> Option<Self>;
Expand Down
72 changes: 72 additions & 0 deletions src/sketches/countminsketch/delta.rs
Original file line number Diff line number Diff line change
Expand Up @@ -199,4 +199,76 @@ mod tests {
CountMinSketch::<Vector2D<i32>, RegularPath>::from_storage(Vector2D::unfilled(2, 3));
assert!(unfilled.merge_delta(&delta(vec![0], vec![1])).is_err());
}

/// Unit roundoff of `f64` under round-to-nearest: one add or subtract is off
/// by at most `u * |result|`.
const U: f64 = f64::EPSILON / 2.0;

/// A receiver holding `old` after one `f64` delta `new - old`, the way a
/// producer forming the delta from its own cells would send it.
fn receive_one(old: f64, new: f64) -> f64 {
let delta =
CountMinSketchDelta::<f64, RegularPath>::new(1, 1, vec![0], vec![new - old]).unwrap();
let mut receiver = CountMinSketch::<Vector2D<f64>, RegularPath>::from_storage(
Vector2D::from_fn(1, 1, |_, _| old),
);
receiver.merge_delta(&delta).unwrap();
receiver.as_storage().as_slice()[0]
}

#[test]
fn f64_receiver_differs_from_producer_within_the_documented_bound() {
// old = 1, new = 1e-17: `new - old` rounds to -1, so old + delta = 0.
let received = receive_one(1.0, 1e-17);
assert_ne!(received, 1e-17);
assert!((received - 1e-17).abs() <= U * 1.0 + U * received.abs());

// 1000 applies on a cell whose value jumps between magnitudes 1e-20 and
// 1e3 with mixed signs (seeded): `new - old` then often rounds, and the
// error accumulates.
let mut receiver = CountMinSketch::<Vector2D<f64>, RegularPath>::from_storage(
Vector2D::from_fn(1, 1, |_, _| 0.0),
);
let (mut producer, mut bound, mut state) = (0.0_f64, 0.0_f64, 12345_u64);
let mut next_unit = || {
state = state
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
(state >> 11) as f64 / (1_u64 << 53) as f64
};
for _ in 0..1000 {
let magnitude = 10_f64.powf(-20.0 + 23.0 * next_unit());
let new = if next_unit() < 0.5 {
-magnitude
} else {
magnitude
};
let d = new - producer;
if d == 0.0 {
continue; // a delta is never zero
}
let delta =
CountMinSketchDelta::<f64, RegularPath>::new(1, 1, vec![0], vec![d]).unwrap();
receiver.merge_delta(&delta).unwrap();
let received = receiver.as_storage().as_slice()[0];
bound += U * d.abs() + U * received.abs();
producer = new;
}
let error = (receiver.as_storage().as_slice()[0] - producer).abs();
assert!(error > 0.0, "the walk must incur rounding error");
assert!(error <= bound, "error {error} exceeds bound {bound}");
}

#[test]
fn integer_receiver_equals_producer_exactly() {
let (old, new) = ([1_i64, 1 << 53, 7], [0_i64, (1 << 53) + 1, 7_000_000_000]);
let deltas: Vec<i64> = old.iter().zip(&new).map(|(o, n)| n - o).collect();
let delta =
CountMinSketchDelta::<i64, RegularPath>::new(1, 3, vec![0, 1, 2], deltas).unwrap();
let mut receiver = CountMinSketch::<Vector2D<i64>, RegularPath>::from_storage(
Vector2D::from_fn(1, 3, |_, c| old[c]),
);
receiver.merge_delta(&delta).unwrap();
assert_eq!(receiver.as_storage().as_slice(), &new);
}
}
Loading