perf(arrow/array): copy view buffers directly - #1324
fallintoplace wants to merge 2 commits into
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
concatFixedWidthBuffers introduces a recovered-panic allocation leak for malformed BinaryView/StringView data. It allocates and resizes the destination before slicing each source header buffer. If a non-empty source has a short non-nil header buffer, that slice panics; concat recovers and returns an error, but the local destination has not yet been assigned to out.buffers[1], so out.Release() cannot free it.
I reproduced this on the current head with CheckedAllocator: one malformed one-element BinaryView leaked 64 bytes from concatFixedWidthBuffers. The old path sliced all sources in gatherFixedBuffers before allocating the destination. Please add failure cleanup around the new allocation (or publish it to the output before a bounds-sensitive operation) and cover the recovered short-header path.
What
ArrayDataduring concatenation.Why
memory.Bufferwrapper per input chunk.Implementation
concatFixedWidthBuffershelper for BinaryView headers and ListView/LargeListView sizes.Benchmark on an Apple M1 Pro:
Tests:
go test ./arrow/array -count=1go vet ./arrow/arraygit diff --check