perf(filter): batch bitmap extraction for auto-vectorization

[sdf-jkl]

Which issue does this PR close?

Related to #11055. This draft builds on Richard's changes at 4727e6a654d28f6371a9fd352a34d8a77af9c0e8 and includes those commits. The measurements below compare against that PR head, not against main.

Rationale for this change

Filtering a bitmap requires gathering the selected bits from each input word and concatenating the resulting fragments. Finishing one word's variable-length extraction loop before starting the next leaves this work scalar in the inspected builds.

This draft processes eight words together, advancing one selected bit per active lane per iteration. It exposes independent operations across words to LLVM's vectorizer while keeping the implementation in stable, portable Rust. A batch runs until its largest mask population count is exhausted, so shorter lanes can do idle work; the sparse tradeoff still needs evaluation.

What changes are included in this PR?

• Extract eight independent u64 fragments by testing each mask's lowest set bit and conditionally OR-ing a shared destination bit into the lane's result. Empty masks naturally contribute zero.
• Pack each batch immediately using stack arrays and the masks' population counts. There is no input-sized fragment allocation.
• Preallocate the output, write the current partial word, and advance its index when it fills. Handle carry shifts without shifting by 64 and store words in little-endian byte order.
• Retain the original scalar pext64 for leftover words and the final partial word.
• Cover differing lane lengths, empty/full masks, zero-valued fragments, unaligned buffers, longer sources, and word/batch boundaries.

The batch helper is kept as a separate optimization unit with #[inline(never)]. There are no ISA intrinsics, CPU-feature dispatch branches, or unstable Rust APIs.

Are these changes tested?

• cargo test -p arrow-select --lib --release --offline: all 432 tests passed.
• Separate generic/native boundary checks passed 15,300 variant/oracle comparisons per build. Benchmark inputs were also checked before timing.
• Formatting and git diff --check passed.

Local exploratory measurements of full Boolean filtering, 65,536 rows, in microseconds (lower is better):

Build	Selected	Filter reuse	#11055	This draft
Generic x86-64	10%	repeated	3.76	5.94
Generic x86-64	10%	cycling	9.78	6.31
Generic x86-64	50%	repeated	18.54	17.05
Generic x86-64	50%	cycling	20.56	17.23
Generic x86-64	79%	repeated	28.92	24.02
Generic x86-64	79%	cycling	29.27	24.08
Native Ryzen	10%	repeated	4.01	3.34
Native Ryzen	10%	cycling	8.94	3.93
Native Ryzen	50%	repeated	23.33	8.60
Native Ryzen	50%	cycling	25.53	9.19
Native Ryzen	79%	repeated	41.16	11.97
Native Ryzen	79%	cycling	41.39	12.08

Ryzen AI 9 HX PRO 470; rustc 1.98.1 / LLVM 22.1.8. The native build uses RUSTFLAGS="-C target-cpu=native" for the harness and dependencies; the generic build has no CPU override. Each build compares the implementations in one executable, pinned to CPU 2, with rotated order over 21 rounds and approximately 3 ms per variant per round. Numbers are median thread CPU time, including allocation and excluding input/predicate construction. Repeated reuses one bitmap; cycling uses 64 prebuilt bitmaps. These are exploratory harness results, not Criterion confidence intervals, and thread CPU time does not eliminate clock/cache interference.

The generic repeated 10% case regresses. Very sparse direct-gather cases also remain a concern; optimized sparse filtering may bypass this kernel.

Inspection of the extraction loop found scalar conditional moves in the final generic x86 build and ZMM SIMD in the native build. Standalone Rust probes produce YMM SIMD with target-cpu=x86-64-v3 and NEON for both generic AArch64 and target-cpu=neoverse-v2. ARM results are code-generation checks only: no ARM performance measurements or inspection of the bot's compiled binary yet.

Are there any user-facing changes?

Filtering results and public APIs are unchanged. This draft changes performance and needs further evaluation, particularly on the ARM benchmark runner and sparse workloads.

[sdf-jkl]

run benchmark filter_kernels
env:
BENCH_FILTER: NULL

[adriangbot]

Hi @sdf-jkl, your benchmark configuration could not be parsed (#11086 (comment)).

Error: invalid configuration: unknown field BENCH_FILTER, expected one of env, baseline, changed at line 2 column 1

Usage:

run benchmark <name>           # run specific benchmark(s)
run benchmarks                 # run default suite
run benchmarks <name1> <name2> # run specific benchmarks

Any benchmark name is accepted: bench.sh suite names (e.g. tpch, clickbench_partitioned, wide_schema) and Criterion bench targets (e.g. sql_planner) are resolved automatically. A name that matches neither fails on the runner.

Per-side configuration (run benchmark tpch followed by):

env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G

---

File an issue against this benchmark runner

[Rich-T-kid]

run benchmark filter_kernels
env:
BENCH_FILTER: NULL

[adriangbot]

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5672287354-2358-l82s4 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing perf/filter-bits-batched-extraction (2b11fc5) to 2078680 (merge-base) diff

Run configuration

run benchmark filter_kernels
env:
  BENCH_FILTER: "NULL"

BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing perf/filter-bits-batched-extraction (2b11fc5) to 2078680 (merge-base) diff

Run configuration

run benchmark filter_kernels
env:
  BENCH_FILTER: "NULL"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                         main                                   perf_filter-bits-batched-extraction
-----                                                                         ----                                   -----------------------------------
filter context i32 w NULLs (kept 1/2)                                         2.59     82.1±0.81µs        ? ?/sec    1.00     31.7±0.02µs        ? ?/sec
filter context i32 w NULLs high selectivity (kept 1023/1024)                  1.00      5.6±0.02µs        ? ?/sec    1.00      5.6±0.02µs        ? ?/sec
filter context i32 w NULLs low selectivity (kept 1/1024)                      1.02    323.2±3.60ns        ? ?/sec    1.00    316.3±0.80ns        ? ?/sec
filter context string dictionary w NULLs (kept 1/2)                           2.60     81.5±0.09µs        ? ?/sec    1.00     31.3±0.02µs        ? ?/sec
filter context string dictionary w NULLs high selectivity (kept 1023/1024)    1.00      5.6±0.01µs        ? ?/sec    1.01      5.7±0.01µs        ? ?/sec
filter context string dictionary w NULLs low selectivity (kept 1/1024)        1.00    376.3±3.36ns        ? ?/sec    1.00    376.2±3.73ns        ? ?/sec
filter context u8 w NULLs (kept 1/2)                                          2.61     81.3±0.08µs        ? ?/sec    1.00     31.1±0.01µs        ? ?/sec
filter context u8 w NULLs high selectivity (kept 1023/1024)                   1.00      2.8±0.01µs        ? ?/sec    1.02      2.9±0.01µs        ? ?/sec
filter context u8 w NULLs low selectivity (kept 1/1024)                       1.00    307.0±3.37ns        ? ?/sec    1.00    307.4±4.71ns        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	100.0s
Peak memory	23.1 MiB
Avg memory	14.0 MiB
CPU user	94.9s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	90.0s
Peak memory	22.6 MiB
Avg memory	14.4 MiB
CPU user	87.1s
CPU sys	0.0s
Peak spill	0 B

---

File an issue against this benchmark runner