Repository navigation
Description
Motivation
TsFiles are increasingly read directly from object storage (OSS/S3), mostly through the Python binding: TsFileReader accepts any user-provided seekable file object (#930, #1055), and behind it every read goes through PythonRandomAccessReadFile into the C++ reader. On local disks the current read pattern is fine, but on object storage each read call is a network round trip with 10-100 ms latency, so the number of small reads dominates query time. #1055 covers supporting object storage; this issue covers making the read path fast on it.
What the read path does today
-
Every
read()pays a seek/tell dance.PythonRandomAccessReadFile::read()(python/tsfile/python_random_access_read_file.cc:158) doestell(),seek(offset), one or moreread()calls, thenseek()back to restore the caller position - four or more Python calls per underlying read, and for fsspec-backed sources theread()typically triggers its own ranged GET. -
Reads are small and page-scoped.
ChunkReader::read_from_file_and_rewrap()(cpp/src/reader/chunk_reader.cc:258) reads at least 4096 bytes per call (DEFAULT_READ_SIZE = 4096). The aligned reader has the same pattern (cpp/src/reader/aligned_chunk_reader.cc:482). A sequential chunk scan therefore issues on the order of one read per page, and pages larger than the 4 KiB window cost two. -
Page bodies that exceed the window are fetched twice. When
remaining_size() < compressed_size_,decode_cur_page_data()re-reads from the page start (cpp/src/reader/chunk_reader.cc:320), discarding the bytes already buffered from the previous 4 KiB read. On object storage this overlapping re-read is a wasted round trip per page. -
Nothing overlaps latency. The only concurrency in the read path is aligned multi-column page decode when
parallel_read_enabled_is set and the query is aligned with multiple columns; a scan over many chunks/devices issues its reads strictly serially. -
Metadata open costs two reads per file, and the dataset layer repeats it per file; that is acceptable today but is the floor for the multi-file read cost (Add a selective metadata reader for a set of measurements #1123 already made selective table-metadata reads possible).
Proposed work
- Read-ahead / coalescing layer. Add a configurable read-ahead window (e.g. 1-4 MiB, with
0meaning the current behavior) so sequential page scans turn many small reads into a few large ones. Either aPrefetchingRandomAccessReadFiledecorator over the existing interface (cpp/src/file/random_access_read_file.h) or a window in the chunk readers. This must be opt-in and must not change local-file behavior or memory cost when disabled. - Consume buffered bytes instead of re-reading. When a page body starts inside the current window, read only the missing suffix rather than re-fetching from the page start.
- Cheaper Python adapter. Prefer an offset-based read (
read_at/pread, or fsspeccat_file(start, end)) when the source exposes one, and skip the save/restoreseekwhen it does; avoid the per-readtell(). - Parallel prefetch across chunks/devices. Overlap object-storage latency with decode by prefetching the next chunk(s) on the existing thread pool, gated by an explicit configuration option. This overlaps with the cross-device scan direction discussed on the mailing list.
Acceptance criteria
- A benchmark against an S3-compatible store (MinIO/moto or fsspec with a counting wrapper) showing both request count and wall time for: point query, one-device range scan, multi-device scan.
- Sequential scan issues at least an order of magnitude fewer read requests than today with read-ahead enabled.
- Local-file performance and memory unchanged with the feature off; C++ ASan/LSan clean.
- No change to the file format, and Java/C++ cross-language behavior unaffected.
Related
- [Feature][C++] Object storage (OSS) read and write support #1055 object storage support (feature), [Feature][C++][Python] Support remote random-access reading for TsFile #930 Python seekable file objects
- [Perf][C++] Point queries rebuild index metadata on every call; prepared series queries are ~7x faster but need fixes #1080 point queries rebuild index metadata on every call
- perf(cpp): batch skip for SPRINTZ (skip_exact_* still decodes per value) #1115/perf(cpp): reduce scalar predictor reconstruction in SPRINTZ decode #1114 SPRINTZ decode-side work (the other half of the latency equation)
Activity