Skip to content

Fix BinaryColumn.getSizeInBytes for over-allocated and shared arrays - #1136

Open
JackieTien97 wants to merge 2 commits into
developfrom
ty/fix-binary-column-size-in-bytes
Open

JackieTien97 wants to merge 2 commits into
developfrom
ty/fix-binary-column-size-in-bytes

Conversation

@JackieTien97

Copy link
Copy Markdown
Contributor

Summary

Column.getSizeInBytes() is documented as the size of the column as if it was compacted. BinaryColumn derived it from the retained size of its whole array, scaled by positionCount / values.length. The unused slots of an array hold no value, so that scaling divides the size of the values the column does hold.

  • Over-allocated arrays. A column built by BinaryColumnBuilder keeps the array of the builder. With a null and a 1024-byte value in 8 slots it reported 294 bytes instead of 1152. TsBlockBuilder allocates 8 slots by default, so a block with one row reported about one eighth of its value.
  • Regions. A region shares the array of the column it was taken from and got an equal share per position, whatever its own positions hold. Of values of {4096, 16, 16, 16} bytes, the region with the first one reported 1094 bytes instead of 4224.
  • setPositionCount(). The size was fixed at construction. After shrinking two 1024-byte values to one position, the column still reported 2208 bytes instead of 1152. TsBlock.buildTsBlock() creates its columns with zero positions for readers that fill them in place, so those kept reporting 0.

The size is now the sum over the positions of the column. It is calculated on the first call of getSizeInBytes() and again after setPositionCount(), so getRegion() stays O(1). A column that fills its array reports the same value as before, and getRetainedSizeInBytes() is unchanged.

The cost: the first getSizeInBytes() call on a column walks its positions once. The constructor still walks the whole array once for the retained size, as before. TsBlock caches its own size, so a block pays this once per BinaryColumn.

Effect on callers

TsBlock.getSizeInBytes() sums the sizes of its columns, so blocks with TEXT, STRING or BLOB columns now report larger, correct sizes.

  • TsBlockSerde uses that value for the initial capacity of its output buffer and for the "TsBlock too large" check. The buffer of such a block no longer starts far too small.
  • IoTDB reserves query memory by TsBlock.getSizeInBytes() when a block is handed to the data exchange (SharedTsBlockQueue, SinkChannel), and its sort operators use the same value. A block with one row holding a 3,000,000-byte TEXT value reserved 375,067 bytes there, about one eighth of what it holds. After this change such blocks reserve what they hold, so queries on large TEXT or BLOB values see more back-pressure than before.

Compatibility

  • No change to a public signature or to the file format; this is in-memory accounting only.
  • No CLI change: java/tools and cpp/tools do not use getSizeInBytes(). No skill documentation describes this value, so skills/ is unchanged.

Testing

New tests in ColumnTest and TsBlockTest. Without the fix they fail as follows:

TsBlockTest.testBinaryTsBlockSize:194
ColumnTest.binaryColumnSizeAfterSetPositionCountTest:538 expected:<1152> but was:<2208>
ColumnTest.binaryColumnSizeIgnoresUnusedCapacityTest:503 expected:<1152> but was:<294>
ColumnTest.binaryColumnSizeOfRegionTest:518 expected:<4224> but was:<1094>

Checks run on this commit (macOS arm64, JDK 21):

  • ./mvnw clean verify -P with-java,with-python -Denable.lzma2=ON: all Java modules pass with 0 failures (common 8 tests, tsfile 1130 tests, tools 258 tests). The C++ module then fails in its own tests on this machine (28 of 1869), so the Python module is skipped.
  • ./mvnw clean verify -P with-cpp -Denable.asan=ON -Dbuild.type=RelWithDebInfo -Denable.lzma2=ON: fails at test discovery with an AddressSanitizer container-overflow report in a static initializer of the ANTLR runtime (XPathLexer::Initializer).
  • ./mvnw spotless:check -P with-java,with-python: Java and C++ sources are clean; the Python step cannot run because python/venv/bin/black is only created by the Python build.

This change touches three Java files and nothing under cpp/ or python/, so the C++ and Python results above do not depend on it.

🤖 Generated with Claude Code

getSizeInBytes() is documented as the size of the column as if it was
compacted. BinaryColumn derived it from the retained size of the whole
array, scaled by positionCount / values.length. The unused slots of an
array hold no value, so that scaling divides the size of the values the
column does hold:

- A column built by BinaryColumnBuilder keeps the over-allocated array of
  the builder. With a null and a 1024-byte value in 8 slots it reported
  294 bytes instead of 1152. A block with a single row, built by
  TsBlockBuilder, reports about one eighth of its value this way.
- A region shares the array of the column it was taken from and got an
  equal share per position, whatever its own positions hold. Of values
  of {4096, 16, 16, 16} bytes, the region with the first one reported
  1094 bytes instead of 4224.
- The size was fixed at construction, so it was stale after
  setPositionCount(): after shrinking two 1024-byte values to one
  position, the column still reported 2208 bytes instead of 1152.
  TsBlock.buildTsBlock() creates its columns with zero positions for
  readers that fill them in place, so those kept reporting 0.

The size is now the sum over the positions of the column, calculated on
the first call and again after setPositionCount(). A column that fills
its array reports the same value as before. getRegion() stays O(1), and
getRetainedSizeInBytes() is unchanged.

TsBlock.getSizeInBytes() sums the sizes of its columns, so blocks with
TEXT, STRING or BLOB columns report larger and correct sizes. TsBlockSerde
uses that value for the initial capacity of its output buffer and for the
"TsBlock too large" check. Callers that reserve memory by this size
reserve more than before for such blocks.
@codecov-commenter

codecov-commenter commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.13%. Comparing base (3e63eca) to head (5ed43e2).
⚠️ Report is 3 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1136      +/-   ##
===========================================
+ Coverage    68.60%   69.13%   +0.52%     
===========================================
  Files          790      790              
  Lines        58146    58345     +199     
  Branches      9410     9454      +44     
===========================================
+ Hits         39892    40334     +442     
+ Misses       16538    16282     -256     
- Partials      1716     1729      +13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants