Skip to content

perf(parquet): reuse DELTA decoder scratch - #1322

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-delta-byte-array-decoder-scratch
Sep 21, 2026
Merged

zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-delta-byte-array-decoder-scratch

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Reuse DELTA length and prefix scratch buffers across decoder pages.
  • Add tests for scratch reuse and page-to-page correctness.

Why

  • The column reader caches decoders by encoding.
  • Each page currently allocates fresh length and prefix arrays even when the decoder already has enough capacity.

Implementation

  • Keep stable backing buffers for decoded lengths and prefix lengths.
  • Expose sliced views to the existing Decode and Discard paths.
  • Keep the existing page-size behavior when a larger page needs more capacity.
  • The existing repeated SetData + Decode benchmark now shows 18 to 15 allocations/op and about 512 KiB less allocation for a full 65,536-value prefix-heavy page.

Tests:

  • go test ./parquet/internal/encoding -count=1
  • go test ./parquet/file -run '^(TestWithEOFReader|TestInvalidHeaders|TestInvalidFooter|TestIncompleteMetadata|TestDeltaLengthByteArrayPackingWithNulls|TestDeltaBinaryPackedMultipleBatches|TestPageStreaming.*|TestPrimitiveReader|TestFullSeekRow|TestSkipEmptyRepeatedRows)$' -count=1
  • go test -race ./parquet/internal/encoding -run 'TestDelta(ByteArrayDecoderReusesPageScratch|LengthByteArrayDecoderReusesLengthScratch|ByteArrayDecoderKeepsResultsAcrossPages|ByteArrayDecoderRejectsInvalidPrefixes)$' -count=1
  • go vet ./parquet/internal/encoding

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scratch ownership and reset behavior look sound. lengthScratch and prefixScratch are decoder-owned metadata storage, each SetData re-slices from index zero, and decoded byte-array results do not alias either scratch buffer. Page reuse, partial consumption, malformed counts, and growth paths preserve the existing behavior.

@zeroshade
zeroshade merged commit a7a6c19 into apache:main Sep 21, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants