[zzylol]

Part of #250. Golden fixtures: ProjectASAP/sketchlib-golden-bytes#6 (merge first; the submodule pointer then moves to the merged commit).

Why

The HIP payload (0x01 0x03) carried four fields: registers, hip_kxq0, hip_kxq1 and hip_est. kxq0 and kxq1 are a cache of the registers. kxq0 is sum(2^-r) over the registers r below 32, and kxq1 is the same sum over registers of 32 or more. update_with_hash keeps both up to date so that each insert costs O(1). Decode took them as given and did not check them against the registers. A payload could therefore carry sums that no register state produces, and the decoded sketch would take every later estimate increment from those wrong sums. The golden fixtures do exactly this: hll_hip_p12 carries kxq0 = 1.5, kxq1 = 2.5 with 4096 registers. Q-DERIVED says state a decoder can recompute exactly is not carried. This PR is the HLL-HIP row of the "cached values carried on the wire although derivable" table in the umbrella issue.

hip_est stays. HIP adds 2^p / (kxq0 + kxq1) to it each time an insert raises a register, so its value depends on the order the registers rose in, and no walk of the registers reproduces it.

What

• ASAPv1, per Q-VER: HIP encodes as the new kind 0x01 0x04, with payload [registers, hip_est]. The retired 0x01 0x03 ([registers, hip_kxq0, hip_kxq1, hip_est]) still decodes. Its two sums are read as float64, as before, and dropped. It is never encoded.
• On decode, under either kind, kxq0 and kxq1 are recomputed from the registers. The recomputed values equal the incrementally kept ones bit for bit (see Correctness), so a decoded sketch that keeps inserting gives the same hip_est as a sketch that never left memory. There is no last-bit difference. The tests check bit equality, not a tolerance.
• No HIP delta kind exists. Nested kinds are unaffected: Hydra's HLL counter (0x07 0x03) holds HyperLogLog<Ertl> registers, and EhBucketSketch's HLL arm is Ertl-MLE. EhBucketSketch rejects 0x01 0x04 the same way it rejects 0x01 0x03, and its sibling-id test now covers it. HashSketchEnsemble's HllHip member has no wire kind.
• Golden fixtures for the new kind: the submodule moves to the golden-bytes branch hll-hip-0104 (b55ca65), which adds hll_hip_est_p12 and hll_hip_est_p14. hll_hip_p12 and hll_hip_p14 stay as decode-only fixtures for 0x01 0x03.
• Docs:
  • wire format: the §1 registry, the payload diagram, §3.1 (payload table, a "hip_est is carried, kxq0 / kxq1 are not" note, a retired-kind note, decode rule 1), the EhBucketSketch sibling-id paragraph, the Section 4 float-field list, and Q-DERIVED (HIP's kxq0 / kxq1 added under "Recomputed"; the encoder-check sentence now applies to caches an API can set, since HIP's are private and exact by construction)
  • api_hyperloglog.md, api_eh_bucket_sketch.md (paragraph rewrapped), message_pack_format.md, docs/tests.md and CHANGELOG

No public field or method is removed. kxq0 and kxq1 stay in memory as private fields so that an update still costs O(1). No non-test code reads anything this PR orphans.

How

• src/sketches/hll.rs: new private hip_kxq(registers) -> (f64, f64). It sums 2^-r with the same r < 32 split that update_with_hash uses. Each term comes from the new private inverse_power_of_two(r), which builds 2^-r from its bits (f64::from_bits((1023 - r) << 52)), so it is exact by construction rather than relying on powi. update_with_hash now uses it too, in place of 1.0 / ((1 << r) as f64), which fixes a pre-existing shift overflow at r = 64 (precision 1). Its doc comment states the exactness bound. The kxq0, kxq1 and est fields now have doc comments saying which values are derived and which are state.
• src/sketches/hll/wire.rs: HllPayloadHip is { registers, hip_est }. A new HllPayloadHipWithKxq reads the retired four-field payload. HyperLogLogHipImpl::from_asapv1 reads the stored kind id and validates the envelope against that id. Each id fixes its payload length: a 0x01 0x04 payload with four fields, or a 0x01 0x03 payload with two, is rejected. It then rebuilds kxq0 / kxq1 with hip_kxq.
• src/asapv1/kind.rs: HLL_HIP is 0x01 0x04. The new HLL_HIP_WITH_KXQ is 0x01 0x03.
• The new golden fixtures were made by decoding each retired fixture and re-encoding it. Tests check both directions.

Key code interfaces added / modified

src/asapv1/kind.rs

/// HLL HIP: the registers and the running estimate (spec §3.1).
pub(crate) const HLL_HIP: &[u8] = &[HLL_FAMILY, 0x04]; // was [HLL_FAMILY, 0x03]
/// Retired HLL HIP kind that also carried `kxq0` / `kxq1`: decoded, with
/// those two recomputed from the registers, and never encoded.
pub(crate) const HLL_HIP_WITH_KXQ: &[u8] = &[HLL_FAMILY, 0x03]; // new

src/sketches/hll.rs

/// HyperLogLog with the HIP estimator (stays).
pub struct HyperLogLogHipImpl<Registers: HllRegisterStorage> {
    /// One byte per register. // unchanged
    registers: Registers,
    /// `sum(2^-r)` over the registers `r` below 32; derived, kept for O(1) updates. // unchanged, no longer on the wire
    kxq0: f64,
    /// `sum(2^-r)` over the registers `r` of 32 or more. // unchanged, no longer on the wire
    kxq1: f64,
    /// The running HIP estimate; order-dependent state. // unchanged
    est: f64,
    /// Probability each update was admitted. // unchanged
    admission_rate: f64,
}

/// `(kxq0, kxq1)` summed over a register slice; equal bit for bit to the
/// incrementally kept values for precision up to 22. Used by decode.
fn hip_kxq(registers: &[u8]) -> (f64, f64); // new (private)
/// `2^-exponent` built from its bits, exact by construction.
fn inverse_power_of_two(exponent: u8) -> f64; // new (private); also used by update_with_hash

src/sketches/hll/wire.rs

/// HIP payload (`01 04`): the register bin plus the running HIP estimate.
pub(crate) struct HllPayloadHip {
    /// One byte per register, msgpack bin. // unchanged
    pub(crate) registers: Vec<u8>,
    /// Was position 1 and 2 of the payload. // removed
    // pub(crate) hip_kxq0: f64,
    // pub(crate) hip_kxq1: f64,
    /// The running HIP estimate, float64. // unchanged (now position 1)
    pub(crate) hip_est: f64,
}

/// The retired HIP payload (`01 03`), decode-only; `kxq0` / `kxq1` are read and dropped.
pub(crate) struct HllPayloadHipWithKxq { // new
    pub(crate) registers: Vec<u8>,
    pub(crate) hip_kxq0: f64,
    pub(crate) hip_kxq1: f64,
    pub(crate) hip_est: f64,
}

impl<Registers: HllRegisterStorage> HyperLogLogHipImpl<Registers> {
    /// Encodes as `01 04`: `[registers, hip_est]`. // was 01 03 with four fields
    pub fn to_asapv1(&self) -> Result<Vec<u8>, RmpEncodeError>;
    /// Decodes `01 04`, or the retired `01 03`; recomputes `kxq0` / `kxq1`
    /// from the registers either way. // was: took kxq0 / kxq1 from the payload
    pub fn from_asapv1(bytes: &[u8]) -> Result<Self, RmpDecodeError>;
}

ASAPv1 payload, §3.1:

0x01 0x04 (new):     [ registers: bin, hip_est: f64 ]
0x01 0x03 (retired): [ registers: bin, hip_kxq0: f64, hip_kxq1: f64, hip_est: f64 ]  decode-only

Usage

A producer counts distinct keys and ships the sketch. A receiver decodes it and keeps counting the same stream, and its estimate stays equal to the producer's. This was compiled and run as an example binary at c988a01. It printed payload 16731 bytes, estimate 99458.

use asap_sketchlib::DataInput;
use asap_sketchlib::sketches::hll::HyperLogLogHip;

// producer: the P14 HIP sketch that counts the first 50,000 keys
let mut producer = HyperLogLogHip::new();
for key in 0..50_000u64 {
    producer.update(&DataInput::U64(key));
}
// bytes: the ASAPv1 envelope shipped to the receiver; kind 0x01 0x04, payload [registers, hip_est]
let bytes = producer.to_asapv1().expect("encode");
assert_eq!(&bytes[8..10], [0x01, 0x04]);

// receiver: decoded from the bytes; kxq0 / kxq1 are recomputed from the registers
let mut receiver = HyperLogLogHip::from_asapv1(&bytes).expect("decode");
for key in 50_000..100_000u64 {
    producer.update(&DataInput::U64(key));
    receiver.update(&DataInput::U64(key));
}
// both have seen the same 100,000 keys, and their estimates agree exactly
assert_eq!(receiver.estimate(), producer.estimate());
println!("payload {} bytes, estimate {}", bytes.len(), receiver.estimate());

Major algorithm changes

HIP decode: kxq0 / kxq1 recomputed, not read

• Problem: decode took kxq0 and kxq1 from the payload unchecked. The hll_hip_p12 fixture (kxq0 = 1.5, kxq1 = 2.5, 4096 registers that mostly read 0) decoded with those values. Its next estimate increment would have been 4096 / 4.0 = 1024 instead of about 4096 / 4092.6 ≈ 1.0.
• Before / after:
  1. Before: from_asapv1 read (registers, kxq0, kxq1, est) and stored them as given.
  2. After: from_asapv1 reads (registers, est) (or the retired four fields, dropping the middle two), then sets (kxq0, kxq1) = hip_kxq(registers).
• Correctness: every term 2^-r is a power of two.
  • Every partial sum, and every intermediate value of the subtract-then-add update, is a multiple of the smallest term. It is exact while it stays below 2^53 times that term.
  • kxq0: terms run from 2^0 down to 2^-31, and the sum is at most m = 2^p. It is exact for p + 31 <= 53, so for p <= 22.
  • kxq1: terms run from 2^-32 down to 2^-(65 - p), and the sum is at most 2^(p - 32). The ratio to the smallest term is 2^33 at every p, so kxq1 is exact at every precision.
  • So the summed value and the value update_with_hash keeps incrementally are the same f64. The crate builds P12, P14 and P16.
  • hip_decode_then_continue_matches_an_unserialized_sketch checks this at P12, P14 and P16. Every 7th hash is placed at a rank of 32 or more, so kxq1 is non-zero. The test encodes after 0, 1,000 or 40,000 updates, decodes, feeds both sketches the same further hashes, and asserts bit equality of registers, kxq0, kxq1 and est.
• Complexity: decode gains one O(m) pass over the registers. It already makes one O(m) pass for the rank check and another to copy the registers. Updates are unchanged. The payload is 18 bytes smaller (two msgpack float64s).
• Assumptions and limits: bit-exactness is guaranteed up to 2^22 registers (precision 22), and only kxq0 sets that bound. Above it, a custom impl_hll_registers! storage's kxq0 may differ from the incremental value in the last bits.

Before this PR

A HIP sketch encoded as 0x01 0x03 with four fields. Decode trusted kxq0 / kxq1, so a payload with sums inconsistent with its registers made the decoded sketch's future estimate increments wrong.

After this PR

A HIP sketch encodes as 0x01 0x04 with [registers, hip_est]. Decode derives kxq0 / kxq1 from the registers, so they always match the registers, and a decoded sketch continues exactly like the original. Bytes written as 0x01 0x03 still decode, and they re-encode as 0x01 0x04.

Verification

At c988a01 (golden submodule at b55ca65):

• Failing tests written first:
  • hip_encodes_registers_and_estimate_only failed with kind [1, 3] instead of [1, 4].
  • retired_hip_kind_decodes_with_kxq_recomputed failed because the decoded kxq0 was the payload's 1.5 instead of the sum over the registers (about 4092.63).
  • hip_decode_then_continue_matches_an_unserialized_sketch passed before the change too (the sums were carried then). It now guards that recomputing them changes nothing.
• cargo test --lib hll::: 29 passed, including the three tests above, hip_payload_must_match_its_kind_field_count (the two-field 0x01 0x04 payload under 0x01 0x03, and the four-field one under 0x01 0x04, are both rejected), known_state_matches_golden (encodes to the new fixtures; decodes the retired ones with recomputed sums), hll_hip_round_trip_preserves_state (bit equality) and hll_rejects_out_of_range_register_values (both kinds).
• cargo test --lib asapv1::: 12 passed (float_fields_reject_an_integer_and_the_other_float_width covers hip_est on 0x01 0x04 and the three floats on 0x01 0x03; the decode-path table has a row for each kind).
• cargo test --lib eh_bucket_sketch: 20 passed.
• cargo test --test asapv1_golden: 71 passed (default features), 72 passed (--features experimental), including hll_hip_retired_fixtures_decode_to_their_est_twins.
• cargo test --test hll_custom_precision: 5 passed, default and experimental. The P13 HIP round trip now also inserts 2,000 more keys into both sketches and checks that the estimates still agree.
• cargo test --test pbt hll: 102 passed (registers_of reads the two-field payload).
• cargo fmt --check; cargo clippy --all-targets and cargo clippy --all-targets --features experimental: clean.
• Usage example above: compiled and run.

Limitations and follow-ups

• sketchlib-go's ASAPv1 HLL codec is on its open HLL codec PR (asapv1-only/hll). The mirror is a commit stacked on that branch (separate PR body). That branch's golden submodule predates the seed_list_index_scaled fixtures, so it does not yet meet hll_hip_est_*. The Go tests derive the 0x01 0x04 bytes from the retired fixtures instead.
• The legacy Go gob SerializeToBytes for HIP still carries the sums. The Go cleanup PR removes that format.

🤖 Generated with Claude Code