Skip to content

perf: zstd implement Pooling + DecodeAll with no new reader. - #1331

Draft
samueleresca wants to merge 1 commit into
apache:mainfrom
samueleresca:zstd-pooling
Draft

samueleresca wants to merge 1 commit into
apache:mainfrom
samueleresca:zstd-pooling

Conversation

@samueleresca

@samueleresca samueleresca commented Sep 20, 2026

Copy link
Copy Markdown
Member

Rationale for this change

Performance improvement on the zstd path.

What changes are included in this PR?

  • zstd: decoders come from a sync.Pool instead of being created for every record batch, and each compressed buffer is decoded with one DecodeAll call straight into the destination buffer (decoder built with WithDecoderConcurrency(1) and WithDecodeAllCapLimit(true)) instead of Reset(bytes.NewReader) + io.ReadFull on a streaming decoder.
  • lz4 : no functional change. It keeps its original per-batch lz4.Reader; the library already pools its large block buffers internally. (Only a Decompress method wrapper was added.)

Are these changes tested?

Added tests + benchmark for zstd (the changed path) - every case improves

SMALL buffers (4 KiB per column)
time      : -70.5% to -78.3% in all 8 cases
            1 col,  1 batch  (stream)  9.89 us  ->  2.92 us
            1 col, 64 batches (stream)  592.9 us -> 128.7 us
            16 cols, 64 batches (file)  3.67 ms  ->  894 us
B/op      : -60% (1 col, 1 batch), -64% (1 col, 64 batches), -22% (16 cols)
allocs/op : -45% (1 col, 1 batch) to -70% (16 cols, 64 batches)

LARGE buffers
per column   time                          B/op                        allocs/op
16 KiB       77.6 us  -> 24.0 us  (-69%)   306.9 KiB -> 157.1 KiB (-49%)   184 -> 78  (-58%)
256 KiB      796.7 us -> 191.7 us (-76%)   20.6 MiB  -> 2.05 MiB  (-90%)   185 -> 81  (-56%)
4 MiB        4.63 ms  -> 3.05 ms  (-34%)   51.1 MiB  -> 32.1 MiB  (-37%)   189 -> 79  (-58%)

Are there any user-facing changes?

@samueleresca samueleresca changed the title zstd: implement Pooling + DecodeAll with no new reader. perf: zstd implement Pooling + DecodeAll with no new reader. Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant