Repository navigation
perf: cut flush encoding allocations and page-write copies - #1133
luoluoyuyu wants to merge 1 commit into
Conversation
cc79087 to
57f31ff
Compare
Caideyipi
left a comment
There was a problem hiding this comment.
The implementation is otherwise consistent with the compatibility claims, and the PR head checks are green. I found one memory-accounting issue in the retained page scratch buffers; see the inline comment.
| public long estimateMaxMemSize() { | ||
| return timeOut.size() | ||
| + valueOut.size() | ||
| + timeEncoder.getPendingByteCount() |
There was a problem hiding this comment.
[P2] Include retained page scratch in memory estimates. PageBodyBuffer retains its uncompressed and compressed arrays for the lifetime of each page writer. After a page is written and page outputs are reset, these arrays remain reachable, but this estimate (and the chunk writers' estimateMaxSeriesMemSize()) has no capacity term for them. The memory-control path therefore undercounts retained heap as pages and series accumulate, which can defeat the configured memory limit on high-cardinality workloads. Please account for the retained capacities or release them at the memory-control boundary.
|
|
||
| @Override | ||
| public void setExpectedPageCount(int expectedPageCount) { | ||
| this.expectedPageCount = expectedPageCount; |
There was a problem hiding this comment.
[P3] Wire the expected-page count into the writer path. There are no call sites for setExpectedPageCount in this repository, so this field remains 0 for normal writers; growPageBufferAfterFirstPage() therefore always returns and the advertised single pre-sizing optimization never runs. Please set the estimate where the page count is known, or remove this unused path.
c916a9c to
496e668
Compare
Reduce allocations, boxing and buffer copies in the page/chunk write path without changing chunk size, page size or produced bytes. - store chunk pages in ChunkPageBuffer as a sequential slab stream: a slab is filled before the next one is taken, slab size is learned from the first page, and slabs are reused through size-class LIFO pools, so there is no per-page block slack, no linear pool scan and no 2x chunk capacity - keep page writer and page body buffers lazily sized and reused across chunks of a writer - expose retained buffer bytes and use them for the global flush memory threshold, while keeping the logical page/chunk size estimates unchanged - treat expectedPageCount as a clamped layout hint only; it no longer sizes any buffer - remove the per-value lambda and Integer boxing from the dictionary encoder; encode int RLE values from a primitive buffer - write the first page statistics up front for multi-page chunks and compact the first page in place for single-page chunks - batch Gorilla/Chimp bit writes through a 64-bit accumulator and account for buffered bytes when sizing pages - keep ChunkPageBuffer package-private, fix the raw-page copy path for the segmented buffer, and deprecate the test-only snapshot page-buffer getter - simplify LongRleEncoder back to a single buffered implementation and remove the unused page-buffer release and encoder-level retained-byte reporting paths
496e668 to
663b4fe
Compare
Summary
Reduce allocations, boxing and buffer copies in the page/chunk write path, without changing chunk size, page size or produced bytes.
PublicBAOSwithChunkPageBuffer, a sequential slab stream: a slab is filled before the next is taken, slab size is learned from the first page, and slabs are reused through size-class LIFO pools. There is no per-page block slack, no linear pool scan, no 2x chunk capacity and no page-count preallocation.expectedPageCountas a clamped layout hint only (it decides whether the first-page statistics can be written up front); it no longer sizes any buffer.Integerboxing from the dictionary encoder; encode int RLE values from a primitive buffer.LongRleEncoderback to a single buffered implementation.ChunkPageBufferpackage-private, fix the raw-page copy path for the segmented buffer, and deprecate the test-only snapshot page-buffer getter.Results
Local IoTDB flush benchmark (deterministic allocation):
End-to-end throughput is about 57–62 M points/s, roughly 2x the baseline. In a controlled alternating A/B it matches the previous pre-grow version while allocating ~32% less, because the slab stream keeps page bytes exact and still avoids chunk-level doubling.
Compatibility
TsFileLastReaderTestwith-Xmx512mand the newChunkPageBufferTest.Test plan
./mvnw -P with-java -pl java/tsfile -am test(1131 tests)ChunkPageBufferTest(exact page storage, writeTo, replace/truncate, reset/release, slab pool reuse)TsFileLastReaderTestwith-Xmx512mspotless:apply