Repository navigation
Fix BinaryColumn.getSizeInBytes for over-allocated and shared arrays - #1136
Open
JackieTien97 wants to merge 2 commits into
Open
JackieTien97 wants to merge 2 commits into
JackieTien97 wants to merge 2 commits into
Conversation
getSizeInBytes() is documented as the size of the column as if it was
compacted. BinaryColumn derived it from the retained size of the whole
array, scaled by positionCount / values.length. The unused slots of an
array hold no value, so that scaling divides the size of the values the
column does hold:
- A column built by BinaryColumnBuilder keeps the over-allocated array of
the builder. With a null and a 1024-byte value in 8 slots it reported
294 bytes instead of 1152. A block with a single row, built by
TsBlockBuilder, reports about one eighth of its value this way.
- A region shares the array of the column it was taken from and got an
equal share per position, whatever its own positions hold. Of values
of {4096, 16, 16, 16} bytes, the region with the first one reported
1094 bytes instead of 4224.
- The size was fixed at construction, so it was stale after
setPositionCount(): after shrinking two 1024-byte values to one
position, the column still reported 2208 bytes instead of 1152.
TsBlock.buildTsBlock() creates its columns with zero positions for
readers that fill them in place, so those kept reporting 0.
The size is now the sum over the positions of the column, calculated on
the first call and again after setPositionCount(). A column that fills
its array reports the same value as before. getRegion() stays O(1), and
getRetainedSizeInBytes() is unchanged.
TsBlock.getSizeInBytes() sums the sizes of its columns, so blocks with
TEXT, STRING or BLOB columns report larger and correct sizes. TsBlockSerde
uses that value for the initial capacity of its output buffer and for the
"TsBlock too large" check. Callers that reserve memory by this size
reserve more than before for such blocks.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1136 +/- ##
===========================================
+ Coverage 68.60% 69.13% +0.52%
===========================================
Files 790 790
Lines 58146 58345 +199
Branches 9410 9454 +44
===========================================
+ Hits 39892 40334 +442
+ Misses 16538 16282 -256
- Partials 1716 1729 +13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Column.getSizeInBytes()is documented as the size of the column as if it was compacted.BinaryColumnderived it from the retained size of its whole array, scaled bypositionCount / values.length. The unused slots of an array hold no value, so that scaling divides the size of the values the column does hold.BinaryColumnBuilderkeeps the array of the builder. With a null and a 1024-byte value in 8 slots it reported 294 bytes instead of 1152.TsBlockBuilderallocates 8 slots by default, so a block with one row reported about one eighth of its value.setPositionCount(). The size was fixed at construction. After shrinking two 1024-byte values to one position, the column still reported 2208 bytes instead of 1152.TsBlock.buildTsBlock()creates its columns with zero positions for readers that fill them in place, so those kept reporting 0.The size is now the sum over the positions of the column. It is calculated on the first call of
getSizeInBytes()and again aftersetPositionCount(), sogetRegion()stays O(1). A column that fills its array reports the same value as before, andgetRetainedSizeInBytes()is unchanged.The cost: the first
getSizeInBytes()call on a column walks its positions once. The constructor still walks the whole array once for the retained size, as before.TsBlockcaches its own size, so a block pays this once perBinaryColumn.Effect on callers
TsBlock.getSizeInBytes()sums the sizes of its columns, so blocks with TEXT, STRING or BLOB columns now report larger, correct sizes.TsBlockSerdeuses that value for the initial capacity of its output buffer and for the "TsBlock too large" check. The buffer of such a block no longer starts far too small.TsBlock.getSizeInBytes()when a block is handed to the data exchange (SharedTsBlockQueue,SinkChannel), and its sort operators use the same value. A block with one row holding a 3,000,000-byte TEXT value reserved 375,067 bytes there, about one eighth of what it holds. After this change such blocks reserve what they hold, so queries on large TEXT or BLOB values see more back-pressure than before.Compatibility
java/toolsandcpp/toolsdo not usegetSizeInBytes(). No skill documentation describes this value, soskills/is unchanged.Testing
New tests in
ColumnTestandTsBlockTest. Without the fix they fail as follows:Checks run on this commit (macOS arm64, JDK 21):
./mvnw clean verify -P with-java,with-python -Denable.lzma2=ON: all Java modules pass with 0 failures (common8 tests,tsfile1130 tests,tools258 tests). The C++ module then fails in its own tests on this machine (28 of 1869), so the Python module is skipped../mvnw clean verify -P with-cpp -Denable.asan=ON -Dbuild.type=RelWithDebInfo -Denable.lzma2=ON: fails at test discovery with an AddressSanitizercontainer-overflowreport in a static initializer of the ANTLR runtime (XPathLexer::Initializer)../mvnw spotless:check -P with-java,with-python: Java and C++ sources are clean; the Python step cannot run becausepython/venv/bin/blackis only created by the Python build.This change touches three Java files and nothing under
cpp/orpython/, so the C++ and Python results above do not depend on it.🤖 Generated with Claude Code