perf(parquet): reuse DELTA_BYTE_ARRAY discard storage - #1321
fallintoplace wants to merge 3 commits into
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
discardScratch can escape through Decode and then be overwritten by a later Discard.
A concrete sequence is values ["aa", "aa", "zz"]: Discard(1) stores "aa" in the scratch; Decode(1) takes the empty-suffix path and returns a prefix backed by that scratch; the next Discard(1) reuses the scratch and changes the already-returned value to "zz".
I reproduced this on the current head: the retained decoded value mutated from "aa" to "zz". This breaks the decoder's existing result-lifetime invariant, including across SetData because the scratch is intentionally retained. Please ensure storage exposed by Decode cannot alias reusable discard scratch and add this Discard→Decode(empty suffix)→Discard transition as a regression case.
What
DELTA_BYTE_ARRAYvalues are discarded.Why
Discardcurrently allocates a new byte slice for every discarded value with a non-empty suffix.Implementation
The benchmark changes from 65,535 allocations to 0 allocations per operation after warm-up, with about 3.3x lower discard time for 65,536 values.
Tests:
go test ./parquet/internal/encoding -count=1go test ./parquet/file -run '^(TestWithEOFReader|TestInvalidHeaders|TestInvalidFooter|TestIncompleteMetadata|TestDeltaLengthByteArrayPackingWithNulls|TestDeltaBinaryPackedMultipleBatches|TestPageStreaming.*|TestPrimitiveReader|TestFullSeekRow|TestSkipEmptyRepeatedRows)$' -count=1go test -race ./parquet/internal/encoding -run 'TestDeltaByteArrayDecoder(DiscardsAllEmptyValues|DiscardCopiesFirstValue|ReusesDiscardScratch|RejectsInvalidPrefixes|KeepsPartialDecodeResults)$' -count=1go vet ./parquet/internal/encoding