Repository navigation
[C++] Bound aligned batch decodes by remaining TsBlock capacity - #1045
Merged
Merged
Conversation
An aligned table query could hang forever in the first ResultSet::next() when the internal TsBlock could not hold the decoder's fixed 129-row batch. decode_tv_batch() required a full batch before consuming any input, so a small block never filled, E_OVERFLOW was reported as an empty success, and the scan iterator retried the same decode without progress. Cap each pass at min(BATCH, remaining) as the multi-column path and ChunkReader already do, and stop reporting an empty E_OVERFLOW as success: a block that cannot hold a single row now surfaces E_INVALID_ARG instead of spinning silently. Closes apache#1035
ByteBuffer::init(0) still goes through mem_alloc, which allocates an 8-byte header block, but the destructor only released the buffer when reserved_size_ was non-zero. A TsBlock built with a capacity below one row therefore leaked one block per column, which LeakSanitizer reported as a failure of the new small-capacity regression test on Ubuntu. Free whenever the pointer is set; data_ is only ever assigned by mem_alloc/mem_realloc, so this cannot release foreign memory.
Same defect as the ByteBuffer fix: BitMap::init(0) rounds its bitmap size to zero but still allocates an 8-byte header block, and the destructor only released it when size_ was non-zero. A TsBlock built with a capacity below one row leaked the null bitmap of every column, which LeakSanitizer reported on the new small-capacity regression test. Free on the pointer instead of the size, matching ByteBuffer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1035.
An aligned table query could hang forever inside the first
ResultSet::next()when the internal TsBlock could not hold the decoder's fixed 129-row batch (g_config_value_.tsblock_max_memory_ = 128in the issue's reproducer). The query returnedE_OK; reading its first result never returned.Three things combined:
AlignedChunkReader::decode_tv_batchrequiredrow_appender.remaining() >= BATCH(129) and returnedE_OVERFLOWbefore consuming any input. A 128-byte block holds 8 rows for a time+INT64 tuple, so the condition could never be satisfied — even on an empty block.decode_time_value_buf_into_tsblockturned thatE_OVERFLOWintoE_OK, keeping the unfinished page.TsFileSeriesScanIterator::get_nextsaw a successful block with zero rows whilehas_more_data()was still true, and retried. Neither the input cursor nor the block capacity changed, so it spun without progress.Changes
decode_tv_batchnow caps each pass atmin(BATCH, row_appender.remaining()), mirroringmulti_DECODE_TV_BATCHandChunkReader's per-type batch loops.read_batch_int64already falls back to per-value decoding when a whole TS2DIFF block does not fit the output array, so a partial batch keeps the stream consistent.E_OVERFLOWthat produced no rows is no longer reported as success: the block is unusable (capacity below one row), so the read surfacesE_INVALID_ARGinstead of an empty successful block. The same guard is applied to the multi-column path, which had the same no-progress case.ByteBufferandBitMapnow release their buffer whenever the pointer is set, not only when the reserved size is non-zero.init(0)still allocates an 8-byte header block, so a TsBlock whose capacity is below one row leaked one block per column and a null bitmap per column. LeakSanitizer caught both on Ubuntu through the new test.Tests
Two new cases in
cpp/test/reader/aligned_field_filter_regression_test.cc, run in all six row/batch × serial/parallel parameterizations:SmallTsBlockCapacityResumesAlignedReadsreproduces the issue exactly (3 devices × 21 timestamps,ghostdeclared but never written, device 1 missingv, nulls present, page limit 5,tsblock_max_memory_ = 128) and asserts all 63 rows come back.UnusableTsBlockCapacityReportsErrorsets a capacity below one row and assertsE_INVALID_ARGrather than an endless empty-success loop.Verification
after scan ret=0 count=63 expected=63(previously timed out at 5 s); with the default 2 MiB it still returns 63 rows.ByteBuffer/BitMapfixes 4 allocations (two 8-byte buffers and two 8-byte null bitmaps) were still live at exit; afterwards the count is zero.TsFile_Testsuite: 1424 passed, 3 skipped, 0 failed.