perf(arrow/array): compare approximate list values by valid runs - #1330
fallintoplace wants to merge 8 commits into
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
The new full-range sliceApproxEqual fast path bypasses DataType normalization and changes public behavior.
array.NewBinaryBuilder(mem, arrow.BinaryTypes.String) produces concrete *array.Binary values carrying STRING data types. For two such operands, canEqualDirectly accepts the matching concrete types and the shortcut dispatches to exact binary comparison rather than string approximate comparison. Consequently "a\\x00" versus "a" changes from true to false; LargeString has the analogous issue.
I reproduced this against current head 89944d11: the full-range STRING comparison loses the established trailing-NUL normalization. The previous NewSlice/MakeFromData path normalized these values to *String. Please restrict the shortcut where concrete type and logical DataType semantics agree (or retain the normalization path for these ambiguous Binary/String representations) and add this builder-produced case.
What
Speed up
ApproxEqualfor list arrays.Why
The old path created and released one child slice for every valid list item. This is expensive for large arrays.
Implementation
List,LargeList, andFixedSizeList.Benchmark
Apple M1 Pro, Go 1.26.3,
GOMAXPROCS=1, 65,536 rows, Int32 child values, list size 16. Median of 3 runs. Lowerns/opis better.