Skip to content

[Perf][C++/Python] Reduce object-storage (OSS/S3) read latency: fewer round-trips, read-ahead, and no redundant re-reads #1138

Open

Description

@ColinLeeo

Motivation

TsFiles are increasingly read directly from object storage (OSS/S3), mostly through the Python binding: TsFileReader accepts any user-provided seekable file object (#930, #1055), and behind it every read goes through PythonRandomAccessReadFile into the C++ reader. On local disks the current read pattern is fine, but on object storage each read call is a network round trip with 10-100 ms latency, so the number of small reads dominates query time. #1055 covers supporting object storage; this issue covers making the read path fast on it.

What the read path does today

  1. Every read() pays a seek/tell dance. PythonRandomAccessReadFile::read() (python/tsfile/python_random_access_read_file.cc:158) does tell(), seek(offset), one or more read() calls, then seek() back to restore the caller position - four or more Python calls per underlying read, and for fsspec-backed sources the read() typically triggers its own ranged GET.

  2. Reads are small and page-scoped. ChunkReader::read_from_file_and_rewrap() (cpp/src/reader/chunk_reader.cc:258) reads at least 4096 bytes per call (DEFAULT_READ_SIZE = 4096). The aligned reader has the same pattern (cpp/src/reader/aligned_chunk_reader.cc:482). A sequential chunk scan therefore issues on the order of one read per page, and pages larger than the 4 KiB window cost two.

  3. Page bodies that exceed the window are fetched twice. When remaining_size() < compressed_size_, decode_cur_page_data() re-reads from the page start (cpp/src/reader/chunk_reader.cc:320), discarding the bytes already buffered from the previous 4 KiB read. On object storage this overlapping re-read is a wasted round trip per page.

  4. Nothing overlaps latency. The only concurrency in the read path is aligned multi-column page decode when parallel_read_enabled_ is set and the query is aligned with multiple columns; a scan over many chunks/devices issues its reads strictly serially.

  5. Metadata open costs two reads per file, and the dataset layer repeats it per file; that is acceptable today but is the floor for the multi-file read cost (Add a selective metadata reader for a set of measurements #1123 already made selective table-metadata reads possible).

Proposed work

  • Read-ahead / coalescing layer. Add a configurable read-ahead window (e.g. 1-4 MiB, with 0 meaning the current behavior) so sequential page scans turn many small reads into a few large ones. Either a PrefetchingRandomAccessReadFile decorator over the existing interface (cpp/src/file/random_access_read_file.h) or a window in the chunk readers. This must be opt-in and must not change local-file behavior or memory cost when disabled.
  • Consume buffered bytes instead of re-reading. When a page body starts inside the current window, read only the missing suffix rather than re-fetching from the page start.
  • Cheaper Python adapter. Prefer an offset-based read (read_at/pread, or fsspec cat_file(start, end)) when the source exposes one, and skip the save/restore seek when it does; avoid the per-read tell().
  • Parallel prefetch across chunks/devices. Overlap object-storage latency with decode by prefetching the next chunk(s) on the existing thread pool, gated by an explicit configuration option. This overlaps with the cross-device scan direction discussed on the mailing list.

Acceptance criteria

  • A benchmark against an S3-compatible store (MinIO/moto or fsspec with a counting wrapper) showing both request count and wall time for: point query, one-device range scan, multi-device scan.
  • Sequential scan issues at least an order of magnitude fewer read requests than today with read-ahead enabled.
  • Local-file performance and memory unchanged with the feature off; C++ ASan/LSan clean.
  • No change to the file format, and Java/C++ cross-language behavior unaffected.

Related

Activity

  1. added this to the 2.5.2 milestone on Nov 20, 2026
  2. added
    pythonPull requests that update Python code
    c++Pull requests that update c++ code
    on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    c++Pull requests that update c++ codeperformancepythonPull requests that update Python code

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions