Skip to content

[C++] Bound aligned batch decodes by remaining TsBlock capacity - #1045

Merged
ColinLeeo merged 3 commits into
apache:developfrom
ColinLeeo:claude/tsfile-1035-349bf7
Oct 29, 2026
Merged

ColinLeeo merged 3 commits into
apache:developfrom
ColinLeeo:claude/tsfile-1035-349bf7

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1035.

An aligned table query could hang forever inside the first ResultSet::next() when the internal TsBlock could not hold the decoder's fixed 129-row batch (g_config_value_.tsblock_max_memory_ = 128 in the issue's reproducer). The query returned E_OK; reading its first result never returned.

Three things combined:

  • AlignedChunkReader::decode_tv_batch required row_appender.remaining() >= BATCH (129) and returned E_OVERFLOW before consuming any input. A 128-byte block holds 8 rows for a time+INT64 tuple, so the condition could never be satisfied — even on an empty block.
  • decode_time_value_buf_into_tsblock turned that E_OVERFLOW into E_OK, keeping the unfinished page.
  • TsFileSeriesScanIterator::get_next saw a successful block with zero rows while has_more_data() was still true, and retried. Neither the input cursor nor the block capacity changed, so it spun without progress.

Changes

  • decode_tv_batch now caps each pass at min(BATCH, row_appender.remaining()), mirroring multi_DECODE_TV_BATCH and ChunkReader's per-type batch loops. read_batch_int64 already falls back to per-value decoding when a whole TS2DIFF block does not fit the output array, so a partial batch keeps the stream consistent.
  • An E_OVERFLOW that produced no rows is no longer reported as success: the block is unusable (capacity below one row), so the read surfaces E_INVALID_ARG instead of an empty successful block. The same guard is applied to the multi-column path, which had the same no-progress case.
  • ByteBuffer and BitMap now release their buffer whenever the pointer is set, not only when the reserved size is non-zero. init(0) still allocates an 8-byte header block, so a TsBlock whose capacity is below one row leaked one block per column and a null bitmap per column. LeakSanitizer caught both on Ubuntu through the new test.

Tests

Two new cases in cpp/test/reader/aligned_field_filter_regression_test.cc, run in all six row/batch × serial/parallel parameterizations:

  • SmallTsBlockCapacityResumesAlignedReads reproduces the issue exactly (3 devices × 21 timestamps, ghost declared but never written, device 1 missing v, nulls present, page limit 5, tsblock_max_memory_ = 128) and asserts all 63 rows come back.
  • UnusableTsBlockCapacityReportsError sets a capacity below one row and asserts E_INVALID_ARG rather than an endless empty-success loop.

Verification

  • Standalone reproducer from the issue: with 128 bytes it now prints after scan ret=0 count=63 expected=63 (previously timed out at 5 s); with the default 2 MiB it still returns 63 rows.
  • Reverting only the source change while keeping the new test reproduces the hang (timed out at 20 s on the first case), confirming the test catches the regression.
  • Traced the allocator through the unusable-capacity test: before the ByteBuffer/BitMap fixes 4 allocations (two 8-byte buffers and two 8-byte null bitmaps) were still live at exit; afterwards the count is zero.
  • Full TsFile_Test suite: 1424 passed, 3 skipped, 0 failed.

An aligned table query could hang forever in the first ResultSet::next()
when the internal TsBlock could not hold the decoder's fixed 129-row batch.
decode_tv_batch() required a full batch before consuming any input, so a
small block never filled, E_OVERFLOW was reported as an empty success, and
the scan iterator retried the same decode without progress.

Cap each pass at min(BATCH, remaining) as the multi-column path and
ChunkReader already do, and stop reporting an empty E_OVERFLOW as success:
a block that cannot hold a single row now surfaces E_INVALID_ARG instead of
spinning silently.

Closes apache#1035
ByteBuffer::init(0) still goes through mem_alloc, which allocates an
8-byte header block, but the destructor only released the buffer when
reserved_size_ was non-zero. A TsBlock built with a capacity below one
row therefore leaked one block per column, which LeakSanitizer reported
as a failure of the new small-capacity regression test on Ubuntu.

Free whenever the pointer is set; data_ is only ever assigned by
mem_alloc/mem_realloc, so this cannot release foreign memory.
Same defect as the ByteBuffer fix: BitMap::init(0) rounds its bitmap size
to zero but still allocates an 8-byte header block, and the destructor
only released it when size_ was non-zero. A TsBlock built with a capacity
below one row leaked the null bitmap of every column, which LeakSanitizer
reported on the new small-capacity regression test.

Free on the pointer instead of the size, matching ByteBuffer.
@ColinLeeo
ColinLeeo merged commit f2a71ef into apache:develop Oct 29, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][C++] Aligned table query hangs with a small internal TsBlock capacity

1 participant