[zzylol]

Part of #250. Golden fixtures: ProjectASAP/sketchlib-golden-bytes#5 (merge first; the submodule pointer then moves to the merged commit).

Why

A Bloom filter answers membership, and every membership answer comes from the bits. Bloom also counted update calls in an inserted field, exposed it through inserted(), and carried it in the ASAPv1 payload. Nothing the filter answers reads it. contains, fill_ratio and estimated_fpp read the bits, and predicted_fpp takes its distinct count from the caller. The count is also not a distinct count: re-inserting a key bumps it but sets no new bit.

The design decision in the umbrella issue: a count of arrivals is a separate exact aggregate, not sketch state. This PR is the Bloom part of that issue.

What

• Bloom's state is the bit grid. The inserted field and the inserted() accessor are removed. update, merge and clear no longer touch a counter.
• ASAPv1, per Q-VER: Bloom encodes as the new kind 0x17 0x01, with payload [words]. The retired 0x17 0x00 ([words, inserted]) still decodes, with the counter read and dropped, and is never encoded.
• Golden fixtures for the new kind: the submodule moves to the golden-bytes branch bloom-bits-1701 (25f8077), which adds bloom_bits_regular_3x128 and bloom_bits_fast_5x32.
• Docs: wire format §1 registry, §3.4 (payload, a "no inserted counter" note, a retired-kind note, decode rule 1), the matrix-row bound paragraph, the integer-scalar list, the KMV note that compared against Bloom's inserted, and Q-DERIVED. Also the Bloom API page, message_pack_format.md, docs/tests.md, tests/PBT_COVERAGE.md and CHANGELOG.

Deleted public API, per the delete-unused-fields rule:

Item	Readers before this PR
Bloom::inserted()	tests only (tests/e2e/membership.rs, tests/e2e/wire.rs, tests/pbt/bloom.rs, tests/asapv1_golden.rs, unit tests)
field Bloom::inserted (private)	inserted(), update, merge, clear, wire.rs

How

• src/sketches/bloom.rs: the field and accessor are removed, along with the counter updates in update (both paths), merge and clear.
• src/sketches/bloom/wire.rs: BloomPayload is { words }. The retired two-field payload is decoded as a (Vec<u64>, u64) tuple: the words are kept and the counter is dropped. No separate payload type is needed, and tests craft retired bytes by encoding a (words, inserted) tuple. from_asapv1 accepts either kind id and picks the payload type by id, so each id fixes its payload length: a 0x17 0x01 payload that still carries a counter, or a 0x17 0x00 payload without one, is rejected.
• src/asapv1/kind.rs: BLOOM is 0x17 0x01, and the new BLOOM_WITH_INSERTED is 0x17 0x00.
• The new golden fixtures were made by decoding each retired fixture and re-encoding it. Tests check both directions.

Key code interfaces added / modified

src/sketches/bloom.rs

/// A partitioned Bloom filter over a packed bit grid (stays). Its state is the
/// bits alone; keep an arrival count in a separate exact aggregate.
pub struct Bloom<Mode = RegularPath, H: SketchHasher = DefaultXxHasher> {
    /// The rows x cols bit grid. // unchanged
    bits: BitMatrix,
    /// Number of update calls, duplicates included. // removed
    // inserted: u64,
    _mode: PhantomData<Mode>,     // unchanged
    _hasher: PhantomData<H>,      // unchanged
}

impl<Mode, H: SketchHasher> Bloom<Mode, H> {
    /// Number of `update` calls this filter has seen, duplicates included. // removed
    // pub fn inserted(&self) -> u64;
    /// Clears every bit. // was: also reset the insert counter
    pub fn clear(&mut self);
    /// Unions `other` into `self`. // was: also summed the insert counters
    pub fn merge(&mut self, other: &Self);
}
// `update` on both paths: sets one bit per slice. // was: also incremented the counter

src/sketches/bloom/wire.rs

impl<Mode: BloomMode, H: SketchHasher + HashProfile> Bloom<Mode, H> {
    /// Encodes kind 0x17 0x01: rows / cols / mode in the metadata; payload [words].
    pub fn to_asapv1(&self) -> Result<Vec<u8>, RmpEncodeError>; // was kind 0x17 0x00, payload [words, inserted]
    /// Decodes kind 0x17 0x01, or the retired 0x17 0x00 with its `inserted`
    /// read and dropped. Fails closed otherwise.
    pub fn from_asapv1(bytes: &[u8]) -> Result<Self, RmpDecodeError>; // was 0x17 0x00 only
}

src/asapv1/kind.rs (crate-private; the wire ids)

/// Bloom: the packed bit grid (spec §3.4).
pub(crate) const BLOOM: &[u8] = &[0x17, 0x01]; // was [0x17, 0x00]
/// Retired kind that also carried the inserted counter: decoded, never encoded.
pub(crate) const BLOOM_WITH_INSERTED: &[u8] = &[0x17, 0x00]; // new

Wire format (docs/asapv1_wire_format.md)

kind_id	Payload	Status
0x17 0x01	[words]	new, encoded
0x17 0x00	[words, inserted]	retired: decoded, counter dropped

The metadata (hash spec, rows, cols, mode) is unchanged.

Usage

A collector pane asks "seen this flow before?" with a Bloom filter. It keeps its arrival count separately. This was compiled and run as an integration test at a7d83f1:

use asap_sketchlib::{Bloom, DataInput, RegularPath};

// Flow keys one collector sees in one pane; "10.0.0.1:443" arrives three times.
let flows = ["10.0.0.1:443", "10.0.0.2:80", "10.0.0.1:443", "10.0.0.3:53", "10.0.0.1:443"];
// The filter answers "seen this flow before?" ...
let mut seen = Bloom::<RegularPath>::with_capacity(10_000, 0.001);
// ... and a separate exact count keeps the number of arrivals, no longer Bloom state.
let mut arrivals: u64 = 0;
for flow in flows {
    seen.update(&DataInput::Str(flow));
    arrivals += 1;
}
assert!(seen.contains(&DataInput::Str("10.0.0.2:80")));
assert_eq!(arrivals, 5);

// On the wire the filter is kind 0x17 0x01: the bit grid alone.
let wire = seen.to_asapv1().unwrap();
assert_eq!(&wire[8..10], &[0x17, 0x01]);
let restored = Bloom::<RegularPath>::from_asapv1(&wire).unwrap();
assert_eq!(restored.to_asapv1().unwrap(), wire);
assert!(flows.iter().all(|f| restored.contains(&DataInput::Str(f))));

// Bytes written before the change (kind 0x17 0x00, with `inserted`) still decode;
// the counter is dropped and the filter re-encodes as 0x17 0x01.
let hex = include_str!("../asapv1_golden/bloom_regular_3x128.hex").trim();
let old: Vec<u8> = (0..hex.len()).step_by(2)
    .map(|i| u8::from_str_radix(&hex[i..i + 2], 16).unwrap()).collect();
assert_eq!(&old[8..10], &[0x17, 0x00]);
let upgraded = Bloom::<RegularPath>::from_asapv1(&old).unwrap();
assert_eq!(&upgraded.to_asapv1().unwrap()[8..10], &[0x17, 0x01]);
println!(
    "bits set={} fill={:.6} arrivals={arrivals} wire_bytes={}",
    seen.as_bits().count_ones(),
    seen.fill_ratio(),
    wire.len()
);
// Printed: bits set=30 fill=0.000183 arrivals=5 wire_bytes=3072

Before this PR, seen.inserted() would have returned 5, and every envelope carried that count.

Before this PR

Bloom counted update calls in inserted, exposed it through inserted(), summed it on merge, and carried it on the wire as 0x17 0x00 [words, inserted].

After this PR

Bloom is the bit grid alone and encodes as 0x17 0x01 [words]. Older 0x17 0x00 bytes still decode, with the counter dropped. Every membership and rate answer is unchanged. The struct is 8 bytes smaller, and each payload is one u64 field smaller.

Verification

Only the tests covering the changed code, plus format and lint. Each command was run with default features and with --features experimental, with the same pass counts unless noted:

• cargo test --lib bloom: 34 passed.
• cargo test --lib envelope: 15 passed.
• cargo test --test asapv1_golden: 71 passed (default), 72 passed (experimental).
• cargo test --test e2e -- membership:: wire:: data_input::: 56 passed (default), 57 passed (experimental).
• cargo test --test pbt bloom: 11 passed. cargo test --test conformance_kit bloom: 2 passed. cargo test --test capability: 3 passed (default).
• cargo fmt --check: clean. cargo clippy --all-targets and --all-targets --features experimental: no warnings.
• Test first: bloom_envelope_structure_and_round_trip was changed to expect 0x17 0x01 and failed ([23, 0] vs [23, 1]) before the change.
• New tests:
  • the_retired_kind_decodes_and_re_encodes_without_the_counter: a crafted 0x17 0x00 envelope decodes to its words and re-encodes as the 0x17 0x01 envelope of the same words.
  • each_kind_rejects_the_other_kinds_payload: 0x17 0x01 with a counter, and 0x17 0x00 without one, are both rejected.
  • bloom_retired_fixtures_decode_to_their_bits_twins: each retired golden fixture decodes to the known grid and re-encodes as its bloom_bits_* twin.
  • bloom_bits_regular_3x128_matches_golden, bloom_bits_fast_5x32_matches_golden: the new fixtures decode to the known grid and round-trip. known_state_matches_golden encodes the known grids to them.
• Renamed unit tests: the_insert_counter_counts_calls_and_clear_returns_it_to_zero became clear_returns_the_filter_to_empty, and merging_identical_geometries_sums_the_insert_counts became merging_identical_geometries_unions_the_members.
• Not run: the full pbt / e2e suites, and benchmarks. CI is not run on this draft.

Limitations and follow-ups

• sketchlib-golden-bytes: the bloom-bits-1701 branch (25f8077) must merge first. Then this submodule pointer moves to the merged commit.
• sketchlib-go has no Bloom codec, only the kind registry. A companion sketchlib-go PR (branch bloom/kind-1701) registers KindBloom = 0x17 0x01 and keeps 0x17 0x00 as KindBloomWithInserted (retired), so its fixture check accepts the new golden files.
• Downstream pin bumps: no downstream caller of Bloom::inserted() was found, so pin bumps need no code change.
• Other sketches with the same pattern are tracked in the umbrella issue.

🤖 Generated with Claude Code