Skip to content

perf(parquet): encode dictionary indices in RLE batches - #1328

Merged
zeroshade merged 3 commits into
apache:mainfrom
fallintoplace:perf/parquet-batch-dict-indices
Sep 21, 2026
Merged

zeroshade merged 3 commits into
apache:mainfrom
fallintoplace:perf/parquet-batch-dict-indices

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

What

  • Encode Parquet dictionary indices in RLE batches.

Why

  • WriteIndices currently sends every index through Put, including long repeated runs and complete literal groups.

Implementation

  • Add PutBatchIndices for int32 indices.
  • Encode repeated runs and complete literal groups directly.
  • Pack literal groups through one batched bit-writer call.
  • Keep scalar handling for partial literal groups.
  • Preserve encoded bytes and dictionary state on write errors.

Benchmark

Command, run six times per version with before/after order alternated:

go test -run '^$' -bench '^BenchmarkDictEncoderWriteIndices$' -benchmem -benchtime=500ms -count=1 ./parquet/internal/encoding

Apple M1 Pro, Go 1.26.3, GOMAXPROCS=1. WriteIndices on 65,536 indices.

Input Before After Time change
Constant 187.7 µs 26.0 µs -86.1%
Runs of 8 388.9 µs 200.9 µs -48.3%
Runs of 32 219.4 µs 67.0 µs -69.5%
Runs of 256 197.6 µs 41.9 µs -78.8%
Random, 16 entries 580.7 µs 306.3 µs -47.3%
Random, 256 entries 452.8 µs 302.8 µs -33.1%
Alternating 400.7 µs 270.6 µs -32.5%

Allocations stay at 232 B/op and 4 allocs/op. The benchmark also covers 64-index batches.

Compatibility

  • Encoded bytes match the existing scalar path.
  • No Parquet format or public API changes.

Tests

  • go test ./parquet/...
  • go test -race ./parquet/internal/utils ./parquet/internal/encoding
  • go test -tags noasm ./parquet/internal/utils ./parquet/internal/encoding
  • go vet -composites=false ./parquet/internal/utils ./parquet/internal/encoding

@samueleresca

Copy link
Copy Markdown
Member

Reproduced on linux/amd64 (AMD Ryzen 9 9950X, Go 1.26.0): BenchmarkDictEncoderWriteIndices is 33–86% faster across all 14 cases (geomean −55%), with no change in allocations (232 B/op, 4 allocs/op). six runs per side, alternating order, benchstat p=0.002 on every case.

Small nit: PutBatchLevels int16 and int32 functions can be merged into a generic. Not a blocker.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The batched run/literal transitions, partial groups, width handling, maximum-run splitting, and successful-path byte equivalence look sound.

One non-blocking existing issue remains in WriteIndices: errors that occur only during the final RleEncoder.Flush are ignored, after which dictionary state is cleared. The same behavior is present on main for terminal partial groups/repeated runs, so it is not introduced or widened by this PR and should be handled separately.

@zeroshade
zeroshade merged commit 55d009b into apache:main Sep 21, 2026
27 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants