Skip to content

feat(file_index): add Java-compatible Bloom filter index - #744

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:bloom-file-index
Open

feat(file_index): add Java-compatible Bloom filter index#744
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:bloom-file-index

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

The file-index foundation only provided generic predicate plumbing and had no concrete Bloom implementation. Rust therefore could not read or produce Java Paimon BloomFilter64 payloads or use them for conservative file pruning.

Linked issue: N/A

Brief change log

  • Implement Java-compatible bitset layout, double hashing, xxHash64, and Thomas Wang hashing.
  • Add Bloom reader/writer with conservative Eq and In evaluation and strict config/payload validation.
  • Add Java golden bytes, per-type hash fixtures, and no-false-negative tests.

Tests

  • cargo fmt --all -- --check
  • cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
  • cargo test --locked -p paimon --all-targets --features fulltext,vortex
  • Dependency and release-license verification

API and Format

No public API change. The serialized Bloom payload is compatible with Java Paimon. Scan and data-writer integration are not included.

Documentation

No documentation update is required because the feature is not externally exposed yet.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant