Repository navigation
C++: optimize batch read/write paths and aligned table null handling - #823
Conversation
69e1658 to
7db1a3a
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #823 +/- ##
===========================================
- Coverage 61.60% 60.52% -1.08%
===========================================
Files 734 735 +1
Lines 45968 48439 +2471
Branches 6897 7679 +782
===========================================
+ Hits 28317 29320 +1003
- Misses 16637 17718 +1081
- Partials 1014 1401 +387 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR consolidates a large optimization branch into develop: it adds batch decode/write APIs across the C++ Decoder/Encoder hierarchy, multi-value aligned read paths with optional parallel decode, columnar tablet write helpers, SIMD fast paths, and a set of correctness fixes for aligned/table null handling (null TAG segments, null FIELD writes, all-null value pages, sparse aligned columns, repeated logical devices, ValuePageWriter::reset state). It also trims the C wrapper API (drops unused metadata export/tag-filter symbols, then re-adds tag-filter helpers in a different section) and removes several regression tests for behaviors it claims to fix.
Changes:
- Add batch decode/encode/write paths through
Decoder/Encoder/page/chunk writers and aMultiAlignedTimeseriesIndexplus single-device aligned fast-path reader. - Several aligned table fixes (null TAG/FIELD, all-null pages, single-device tablet flag,
ValuePageWriter::reset, double-free of first-page buffers viarelease_cur_page_data). - Build/infra: SIMD option, optional
BUILD_EXAMPLES, mem-stat counters widened to 64-bit, newBlockingQueue, removal of several existing regression tests, license-header punctuation churn in multiple CMake files.
Reviewed changes
Copilot reviewed 118 out of 119 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| cpp/src/reader/filter/{and,or}_filter.h, filter.h, time_operator.h | Adds satisfy_batch_time (uses fixed 129-element stack buffer — flagged). |
| cpp/src/encoding/{plain,decoder,encoder,plain_decoder,dictionary_encoder}.h | Batch encode/decode API + dictionary index assignment change (flagged). |
| cpp/src/writer/{value_,time_,}{chunk,page}_writer.{h,cc} | Batch write paths, first-page ownership transfer, larger page buffers. |
| cpp/src/writer/tsfile_table_writer.{h,cc}, tsfile_writer.h | Memoized lowercasing, idempotent close, optional parallel write pool. |
| cpp/src/reader/* | Aligned multi-value batch path, bloom-filter contains, table result-set lifecycle. |
| cpp/src/file/tsfile_io_writer.{h,cc}, restorable_tsfile_io_writer.cc | Recovery cleanup simplified; conditional sync_on_close_ (flagged); chunk-group index for O(1) lookup. |
| cpp/src/file/tsfile_io_reader.h | Device-node cache + multi-SSI alloc. |
| cpp/src/common/allocator/byte_stream.h, alloc_base.h, mem_alloc.cc | Page-mask bitwise modulo + power-of-2 rounding for wrapped buffers (flagged), 64-bit stat counters. |
| cpp/src/common/{tablet,schema,path,global,thread_pool}.* | Single-device flag, string-column uint32_t offsets, Path inlined, config knobs reshuffled. |
| cpp/src/common/container/{bit_map,blocking_queue,byte_buffer}.* | New BlockingQueue, BitMap::may_have_set_bits, bounds asserts. |
| cpp/src/compress/{snappy,lz4,uncompressed}_compressor.* | Safer after_compress ownership handling; Uncompressed now copies. |
| cpp/src/cwrapper/{tsfile_cwrapper.h,arrow_c.cc} | Tag-filter API moved, sliced-Arrow handling reverted (loses prior bug-fix paths). |
| cpp/test/** | Deletes several regression tests (deep path, missing measurement, aligned NULL boundary, dictionary RLE run counts, Arrow slice-with-offset, etc.) and adds new batch/page-boundary tests. |
| python/tsfile/dataset/reader.py + tests | Switches row reads to read_arrow_batch(). |
| cpp/{CMakeLists.txt,examples/**,src/CMakeLists.txt,src/common/CMakeLists.txt,test/CMakeLists.txt} | Build flags, SIMD option, Arrow/Parquet-dependent examples, license-header punctuation regressions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
651f4af to
fcef966
Compare
| // Without the try/catch, a task that throws would: | ||
| // (1) skip the active_-- below → wait_all() blocks forever | ||
| // because active_ never drops to zero, and | ||
| // (2) propagate the exception out of the std::thread function | ||
| // → std::terminate() takes down the whole process. | ||
| // Swallowing the exception is unfortunate but it matches the | ||
| // contract of the public submit(std::function<void()>) overload | ||
| // which has no way to surface the failure back to the caller. | ||
| // submit<F>() callers receive their error via the std::future | ||
| // wrapper installed by std::packaged_task — that path never | ||
| // reaches here, so this catch only fires for fire-and-forget | ||
| // tasks where the alternative is termination. | ||
| try { | ||
| task(); | ||
| } catch (...) { | ||
| // Intentionally suppressed; see comment above. | ||
| } | ||
| { | ||
| std::lock_guard<std::mutex> lk(mu_); |
5ffe889 to
2ad0461
Compare
Batch decode/encode APIs (PLAIN / TS2DIFF / Gorilla) with single-pass TsBlock decode, AVX2/NEON SIMD paths, a single process-wide worker pool for chunk-level parallel read and column-parallel write, and batched NEON statistics. On-disk format unchanged; interoperable with Java/Python.
MultiValueAlignedSkipsBatchPreservesValueAlignment and MultiValueAlignedWideChunkParallelDecode constructed ColIterator locals inside the read loop and called ssi->revert_tsblock() (which frees the TsBlock and its column vectors) while those iterators were still in scope. ~ColIterator() calls vec_->reset_offset(), writing back into the just-freed vector at the closing brace of the loop body — a real use-after-free that Linux Release+ASan flags and whose heap corruption then cascaded into spurious SprintzCodec/CRelease failures later in the same single-process run. Scope the ColIterators in a nested block so they are destroyed before revert_tsblock(). Verified on Linux x86_64 Release+ASan+UBSan: full suite 701/701 pass, 0 ASan/UBSan reports.
TsFileIOReader::get_cached_device_node keyed device_node_cache_ by
IDeviceID::get_device_name(), which renders a null tag segment as the
literal text "null". A device with a real null tag (e.g. tags
(null, b, c)) and a device whose tag value is the string "null"
(("null", b, c)) therefore produce the identical name "a.null.b.c" and
collide in the cache: whichever device is queried first populates the
entry, and every later query for the other device on the same reused
reader gets the first device's cached MetaIndexNode and silently reads
its chunks.
This surfaced through the Python dataset API, where one long-lived
TsFileReader answers many per-device queries: the pytest
test_dataset_null_tag_positions_and_string_null_are_distinct read
(null,b,c)'s data for the ("null",b,c) device. The device metadata
binary search (DeviceIDComparable, segment-based) was always correct;
only the string cache key was lossy.
Key the cache by a collision-free, length-prefixed encoding of the
segment vector that flags null segments explicitly, so a null tag can
never alias the string "null". Add a device_id unit test pinning the
invariant (names collide, segment equality distinguishes them).
Verified: C++ suite 707/707 (Release+ASan+UBSan), Python suite 150/150.
Release builds unconditionally added -march=native on Linux/macOS. Two
problems for artifacts that leave the build host:
1. Portability: a wheel/binary built on a newer CPU can fault with an
illegal instruction on an older target machine.
2. On virtualized macOS arm64 CI runners, -march=native mis-detects the
feature set and drops +crc, while snappy's CRC32 feature probe (run
without -march) still reports crc available and defines
SNAPPY_HAVE_NEON_CRC32=1. snappy.cc then calls the always_inline
__crc32cw intrinsic in a TU compiled without crc support:
error: always_inline function '__crc32cw' requires target feature
'crc', but would be inlined into a function compiled without 'crc'
The default (portable) target for macOS arm64 keeps +crc, so snappy's
fast path stays available while the binary stays portable.
Default Release flags are now -O3 -flto (LTO stays off on MinGW/Windows
as before). -march=native is opt-in via -DTSFILE_ENABLE_NATIVE_ARCH=ON
for local-only builds that never ship. Applies uniformly to CI, wheels,
and manual local builds.
Verified: non-ASan Release (CI config) builds snappy cleanly and passes
705/705; -DTSFILE_ENABLE_NATIVE_ARCH=ON restores -march=native.
| value_matrix_[c].string_col->destroy(); | ||
| value_matrix_[c].string_col->~StringColumn(); |
There was a problem hiding this comment.
Is it possible to merge the two?
Their semantics are similar, using both may lead to confusion.
There was a problem hiding this comment.
Good idea, and fixed.
The SSI is placement-new'd into mem_alloc'd memory and torn down with destroy() + mem_free() (TsFileIOReader::revert_ssi / alloc_ssi / alloc_multi_ssi), so ~TsFileSeriesScanIterator() never runs. Every heap-owning member therefore leaked its backing storage on each query; destroy() only released some of them (itimeseries_index_, chunk_reader_, tsblock_). LeakSanitizer on Linux (Debug+ASan and Release+ASan) reported hundreds of Direct/Indirect leaks routed through the multi-value read path; macOS ASan has no LSan so it stayed hidden locally. Release the remaining members explicitly in destroy(): - tuple_desc_ (std::vector<ColumnSchema>) — add TupleDesc::release() since reset() only clear()s and keeps capacity - value_chunk_meta_cursors_ (std::vector<...::Iterator>) - device_id_ (shared_ptr<IDeviceID>) — the leaked StringArrayDeviceID - measurement_name_ (std::string) Verified on Linux: Debug+ASan and Release+ASan both 705/705 with 0 Direct / 0 Indirect leaks and 0 UBSan reports.
Batch codec APIs (apache#823): - encoder.h: encode_batch() virtual methods for all types - decoder.h: read_batch/ skip/peek block-range methods - plain_encoder.h: SIMD batch encode (simde) for int64/float/double - gorilla_decoder.h: GorillaBitReader raw-pointer fast path TsBlock single-pass decode optimization (apache#823): - tsblock.h: ColAppender/RowIterator/ColIterator helpers - vector.h: add_row_nums/offset methods - variable_length_vector.h: self-contained update_offset - tuple_desc.h: release() for arena-safe cleanup Skipped: ts2diff batch encode (conflicts with CROSS-LANG-FIX), single_device_tsblock re-architecture (stability risk), corrupted file recovery & series stats (not in 2.4.0).
decode_tv_batch<int32_t> replaced this loop in apache#823, leaving the function uncalled. Its NULL branch also predates the time-filter fix applied to the other decode paths in this series.
* [C++] Apply field predicates to aligned table query rows * Preserve aligned NULL rows in table FIELD filtering * Preserve aligned NULL rows outside value statistics A FIELD predicate must see rows whose timestamps lie in the selected time range even when the column's non-NULL values all lie outside it. Replace the all-values-NULL check in the empty-context shortcut with an aligned-source check so such devices fall through to the time-only context instead of returning an empty result. * Remove unused per-value i32 decode in AlignedChunkReader decode_tv_batch<int32_t> replaced this loop in #823, leaving the function uncalled. Its NULL branch also predates the time-filter fix applied to the other decode paths in this series. * [C++] Batch-evaluate FIELD predicates on aligned table rows Add AlignedFieldFilter (a columnar predicate tree) and SelectionMask (a row bitmask), and thread them through the aligned chunk/block readers. Single-threaded flow: page statistics (may_match/all_match) prune pages that cannot match or mark pages that fully match; only filter-referenced columns are decoded first; output columns are decoded lazily and only for pages with at least one matching row; the predicate is evaluated once per page into a bitmask, and the scatter loop only tests bits. NOT is pushed down to leaf comparisons so NULL rows keep SQL three-valued semantics, get_data_type_size is no longer called on string columns, and query() error paths free the field filter. * [C++] Bulk-copy aligned FIELD selections and fix edge cases Copy selected bitmap runs with dense fixed-width bulk appends while preserving NULL/string fallbacks, duplicate projections, post-filter row ranges, and source-block cursor ownership. Keep page-plan decode mode stable across TsBlocks, retain NaN rows when floating statistics cannot prove a match, and evaluate missing aligned columns as NULL. Add focused and integration regression coverage. * [C++] Fix floating NOT, malformed predicates, and reader teardown Negating an ordered FLOAT/DOUBLE comparison flipped the operator (NOT (x < 5) -> x >= 5). NaN makes every ordered comparison false, so the negation must be true for NaN rows while the flipped form is false and dropped them; when the literal itself was NaN, every non-NULL row was lost. Ordered floating-point leaves now keep the operator and record the negation, and evaluation computes !(value <op> literal) for non-NULL rows. Page statistics no longer prove a full-page match or a skip for such leaves. A composite node missing an operand could be classified as a full-page match through its remaining operand. Page statistics then skipped evaluation, so a malformed OR silently returned rows (and did not terminate in batch mode). all_match() now requires both operands, and build_page_plan() rejects a structurally invalid filter with E_INVALID_ARG before classifying pages. AlignedChunkReader is placement-new'd and torn down through destroy() without running its destructor, so the page-plan filter state added for FIELD filtering leaked its backing buffers. destroy() now releases filter_column_mask_ and the current page mask. Tests cover NOT over all six operators and five literals (including NaN and both infinities) for FLOAT and DOUBLE, with NULL rows from a device that lacks the float columns; and a malformed OR that was previously accepted. * [CI] Pin C++ and Python Linux tests to Ubuntu 24.04 * [C++] Validate field filters and restore aligned query fast paths * [C++] Release aligned chunk header names during reader teardown * [C++] Align field predicate literals with Java numeric conversions * [C++] Reject missing NOT operands before query execution * [C++] Evaluate FIELD predicates on non-aligned table rows * Harden FIELD reader cleanup and error state Clear borrowed contexts before deletion, make close idempotent, and publish multi-index ownership before fallible lookup. Keep multi-column read failures terminal until reset, validate page and value-view bounds, and reject residual NOT nodes instead of inverting nullable masks. Add real-file fault injection, malformed page and buffer tests, repeated-close and partial-lookup coverage. Validate reset recovery across output blocks and failures after a result prefix. * Isolate reader failure fixtures for parallel CTest Give each test case and invocation a separate TsFile path so CTest workers cannot truncate or remove one another's input. Reproduce the shared-name failure with four parallel workers, then verify 100 repetitions of every case after isolation. * Give parallel time decode tasks independent state Use each strided task's index for its decoder and compressor slot instead of header-inline worker TLS, which can differ between a caller executable and the reader DLL. This prevents concurrent tasks from sharing slot zero and corrupting timestamps. Complete partially initialized pools when retrying after reset. Validate with the same hidden-symbol caller executable: the old code fails in the first iteration; the fix passes 1000 iterations (2000 fault/recovery tests). Full four-worker CTest also passes with no failures. --------- Co-authored-by: HUANG Xiangdong <1021782+jixuan1989@users.noreply.github.com> Co-authored-by: ColinLee <shuolin_l@163.com>
* perf(cpp): batch and SIMD decode for the RLE bit-packing codecs RLE was the last hot value codec without a batch path: Plain, Gorilla and TS_2DIFF got read_batch_* in #823/#898, while RLE still decoded value by value through read_int and unpacked each bit-packing group with the portable scalar packer. On a wide table the RLE decoder dominates the read profile. - add read_batch_int32/int64 and read_batch_float/double: an RLE run becomes a std::fill, a bit-packing group becomes one memcpy out of the unpacked buffer. - unpack four values per step with a SIMD gather+shuffle kernel and dispatch to native AVX2 intrinsics at runtime (__builtin_cpu_supports) instead of relying on the translation unit being compiled with -mavx2; the simde fallback stays for other targets, and the widths that can cross a ninth byte (59/61/62/63) use a 128-bit buffer where the compiler has one. - read each bit-packing group with one bulk read instead of per-byte read_ui8, and wrap the caller's stream for the pattern length instead of copying it. - the SIMD gather reads up to 7 bytes past the last group, so the RLE and Sprintz decoders pad their buffers by 8 zero bytes; the packer tests do the same and pass the exact packed length. - guard the 128-bit scalar path with __SIZEOF_INT128__ and keep the portable bit loop on MSVC. * fix(cpp): harden RLE batch decode against malformed headers A corrupt bit-packing header could size the new bulk read with a wrapped uint32 product, turn bytes_to_read negative, and write the rest of the pattern past a 7-byte scratch vector. Validate the bit width, group byte count and last_bit_packed_num before sizing anything, and reject patterns whose declared groups do not fit instead of clamping them. - read_bit_packing_buffer: compute group_bytes in 64-bit, bound it by the remaining pattern bytes and mem_alloc's uint32 argument; validate bit_width_ (1..32 / 1..64). - call_read_bit_packing_buffer: require last_bit_packed_num in [1,8], bound the value count by INT32_MAX, and clear the count when the group load fails; read_batch_* now refuses a non-positive count instead of passing it to memcpy. - read_length_and_bitwidth: reject oversized length prefixes, check the bit-width read, and drop the borrowed byte_cache_ view on reset()/destroy(); the int64 decoder deletes the old packer per pattern. - unpack_all_values now decodes the final group from a zero-padded copy so callers no longer need the +8 tail; guard zero-width non-advancing loops. - hoist the simde shuffle mask to a constant byte array, removing the thread-safe-static guard from every kernel call. - Sprintz decoders reject out-of-range block widths. Add differential batch/scalar tests (wrapped and owning streams, capacities 1..1024, mixed run and bit-packed segments, cross-pattern batches, reset and reuse), malformed-header rejection tests, Float/Double RLE batch tests, and exact-size packer tests.
Summary
This PR optimizes the C++ TsFile read/write paths for batch and columnar workloads, and fixes several aligned table null-handling issues uncovered while validating the optimized path.
It consolidates the batch decode/write work from the long-lived optimization branch into a reviewable change for
develop.Supersedes #749, #754, and #774.
Main Changes
Decoderhierarchy and implement batch paths for PLAIN, TS2DIFF, and Gorilla.ChunkReaderandAlignedChunkReader, including shared timestamp decoding for aligned multi-value reads.ByteStream, compressor, and page/chunk writer internals used by the optimized paths.Correctness Fixes
ValuePageWriter::reset()so row count and null bitmap state are reset together.Compatibility Notes
cpp/third_party/is left atdevelopstate so existing platform compatibility fixes are preserved.Verification
cmake --build cpp/target/build --target TsFile_Test -j1ctest --test-dir cpp/target/build/test --output-on-failure -R '^TsFileTableReaderTest\.TestNullInTable4$'ctest --test-dir cpp/target/build/test --output-on-failure -j4cd cpp && mvn spotless:checkcd cpp && mvn apache-rat:checkCurrent full C++ test result:
496/496tests pass.