perf(parquet): encode dictionary indices in RLE batches - #1328
Conversation
|
Reproduced on linux/amd64 (AMD Ryzen 9 9950X, Go 1.26.0): Small nit: |
zeroshade
left a comment
There was a problem hiding this comment.
The batched run/literal transitions, partial groups, width handling, maximum-run splitting, and successful-path byte equivalence look sound.
One non-blocking existing issue remains in WriteIndices: errors that occur only during the final RleEncoder.Flush are ignored, after which dictionary state is cleared. The same behavior is present on main for terminal partial groups/repeated runs, so it is not introduced or widened by this PR and should be handled separately.
What
Why
WriteIndicescurrently sends every index throughPut, including long repeated runs and complete literal groups.Implementation
PutBatchIndicesforint32indices.Benchmark
Command, run six times per version with before/after order alternated:
go test -run '^$' -bench '^BenchmarkDictEncoderWriteIndices$' -benchmem -benchtime=500ms -count=1 ./parquet/internal/encodingApple M1 Pro, Go 1.26.3,
GOMAXPROCS=1.WriteIndiceson 65,536 indices.Allocations stay at 232 B/op and 4 allocs/op. The benchmark also covers 64-index batches.
Compatibility
Tests
go test ./parquet/...go test -race ./parquet/internal/utils ./parquet/internal/encodinggo test -tags noasm ./parquet/internal/utils ./parquet/internal/encodinggo vet -composites=false ./parquet/internal/utils ./parquet/internal/encoding