Apache Parquet Java Performance Improvements

[iemejia]

Targeted performance optimizations to parquet-java's encoding, decoding, and compression layers. Each PR addresses an independent codec or subsystem, replacing stream-wrapper abstractions with direct ByteBuffer I/O, eliminating per-value allocations, and adding JMH benchmarks to validate the improvements.

All benchmarks: JDK 25.0.3 (Temurin), JMH 1.37, Linux x86_64, 100K values/iteration.

Pull requests

PR	Status	Scope	Headline result
#3565	Merged	PLAIN encoding/decoding	Decode 2.6–13x, encode 2–3.8x
#3566	Open	DICTIONARY encoding/decoding	Encode 1.5–100x (high-card / long binary)
#3567	Open	DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY	Decode +22–31%, encode +3–7%
#3568	Open	RLE/Bit-Packing hybrid codec	Decode +30%, encode +5%
#3569	Approved	BYTE_STREAM_SPLIT	Decode 1.4–5.1x, encode 1.6–6.9x
#3570	Open	Compression (Snappy, ZSTD, LZ4, GZIP, LZO, Brotli)	Isolated decompress +15–25% (Snappy/Zstd/Gzip), Snappy compress up to ~2.9x; enables LZ4/Brotli/LZO without native libs
#3571	Approved	Row group flush buffer lifecycle	Correct resource management (not a perf win)

PRs 1–6 and 8 are independent and can be reviewed/merged in parallel. Additional PRs for column I/O (par7) and level write batching (par9) will follow once their dependencies (#3565, #3568) land.

Common optimization patterns

• InputStream → ByteBuffer: Direct ByteBuffer (LITTLE_ENDIAN) access for getInt()/getLong()/getFloat()/getDouble() JVM intrinsics instead of per-byte stream reads.
• Buffer reuse: Allocate once, grow lazily — no fresh allocation per page or per run.
• pack32/unpack32 fast paths: Batch 32 values per packer/unpacker call (4x fewer invocations).
• Eliminate intermediate copies: Write directly to output buffers, bypass stream wrappers.

Benchmark highlights

PLAIN (#3565)

Benchmark	Master	Optimized	Speedup
decodeInt	425 M ops/s	5,427 M ops/s	12.8x
decodeBoolean	639	1,642	2.6x
encodeInt	148	559	3.8x
encodeBoolean	850	1,692	2.0x

DICTIONARY (#3566)

Benchmark	Master	Optimized	Speedup
encodeBinary LOW len=1000	1.5 M ops/s	148.3 M ops/s	~100x
encodeBinary LOW len=100	13.2	107.8	8.2x
encodeFlba HIGH len=12	6.3	15.4	2.4x
encodeInt HIGH_CARD	14.9	23.5	1.58x

DELTA (#3567)

Component	Avg improvement
DELTA_BINARY_PACKED decode	+27%
DELTA_BYTE_ARRAY decode	+31%
Long delta decode (TIMESTAMP_MILLIS pattern)	+28%
Encoding (all delta variants)	+3–7%

RLE (#3568)

Category	Avg improvement
Direct decoder (packed data)	+30%
Boolean decode (packed patterns)	+14%
Encoder	+5%

BYTE_STREAM_SPLIT (#3569)

Benchmark	Master	Optimized	Speedup
decodeInt	203 M ops/s	1,034 M ops/s	5.1x
encodeDouble	53	365	6.9x
encodeLong	52	356	6.9x
encodeInt	99	515	5.2x

Compression (#3570)

Isolated CompressionBenchmark, ours/master geomean over 16 realistic encoding shapes (>1 = faster):

Codec	compress	decompress
Snappy	1.38x	1.25x
Zstd	1.00x	1.18x
Gzip	1.00x	1.15x
LZ4_RAW	0.96x	1.06x

Snappy peaks near 2.9x compress / 1.8x decompress on small, highly compressible pages. Compression is ~1–10% of end-to-end write for the fast codecs (up to ~46% for Gzip), so file-level impact scales with codec weight. Also migrates LZO from GPL to Apache 2.0 (aircompressor) and Brotli from abandoned jbrotli to brotli4j (adds aarch64 support); LZ4/Brotli/LZO now work without native Hadoop codecs.

Row group flush (#3571)

Correct resource management — releases column buffers during flush rather than after. Peak memory unchanged (peak occurs during page compression, not flush). Makes buffers GC-eligible sooner.

How to run the benchmarks

# Build the benchmark jar
./mvnw --projects parquet-benchmarks -amd -DskipTests -Denforcer.skip=true clean package

# Quick single-iteration smoke test
./parquet-benchmarks/run.sh all -wi 0 -i 1 -f 1

# Full statistical run
./parquet-benchmarks/run.sh all -wi 5 -i 5 -f 3 -rff /tmp/benchmark-results.json

# Specific benchmark
./parquet-benchmarks/run.sh all "PlainEncodingBenchmark|PlainDecodingBenchmark"

Test validation

Each PR passes the full module test suite:

./mvnw --projects parquet-column,parquet-common,parquet-hadoop -amd verify

[alamb]

Thank you for working on this @iemejia

FWIW this is somewhat similar to some of the work that @HippoBaro has been helping in Rust. More details here

• Column performance: run-proportional read/write cost arrow-rs#9731

[Fokko]

@iemejia Out of curiosity, why everything back to draft?

[iemejia]

@Fokko I am grouping the changes by encoding type and including the benchmarks on each encoding type I hope this will ease the review. It will happen in some mins