Skip to content

feat(cpp): add a RandomAccessWriteFile abstraction and in-memory write sink - #1094

Merged
ColinLeeo merged 1 commit into
apache:developfrom
ColinLeeo:colin/random-access-write-file-00a587
Nov 12, 2026
Merged

ColinLeeo merged 1 commit into
apache:developfrom
ColinLeeo:colin/random-access-write-file-00a587

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

What

Adds RandomAccessWriteFile, the write-side counterpart of the existing
RandomAccessReadFile byte source, and wires the C++ writers through it. The
first non-local backend is an in-memory sink that lets a complete TsFile be
written and read back without touching the file system.

Part of #1055 (C++ object storage read/write support). This PR is the
backend-independent write path; an object-storage sink is a follow-up.

Design

  • cpp/src/file/random_access_write_file.h — the interface:
    is_opened/file_path/write/get_position/sync/truncate/seek_to_end/close,
    in the project's error-code style. The header documents the contracts,
    notably write-after-close (E_FILE_WRITE_ERR, no-op), truncate extending
    with zeros and not moving the cursor (POSIX ftruncate semantics), and that
    file_path() may be a label for backends without a real path.
  • WriteFile (the existing fd-backed local backend) now implements the
    interface directly — no separate adapter class. This keeps the C API
    (typedef void* WriteFile), the Python binding and all path-based
    constructors working unchanged; the local write path keeps the same
    syscall loop. file_opened() stays as an alias of is_opened(), and
    is_opened() is fd_ >= 0 (the old fd_ > 0 check mis-treated fd 0 as
    closed). The four ASSERT(fd_ > 0) guards became error returns.
  • MemoryRandomAccessWriteFile — growable buffer; constructing it opens it;
    close() keeps the bytes; data()/size()/release_data() expose them. It
    cannot be reopened for append.
  • MemoryRandomAccessReadFile — a RandomAccessReadFile over those bytes so
    TsFileReader::open(std::unique_ptr<RandomAccessReadFile>) reads the
    in-memory file. The read side previously had no in-memory implementation;
    it is required to verify the write path end to end.
  • Wiring: TsFileIOWriter::init, TsFileWriter::init, the
    TsFileTableWriter and TsFileTreeWriter constructors, and
    RestorableTsFileIOWriter::get_write_file() now take/return
    RandomAccessWriteFile*. The three get_fd() liveness checks in
    TsFileIOWriter became is_opened().

Scope

  • Recovery/append (RestorableTsFileIOWriter) remains local-file only:
    self_check reads the existing bytes back by path through a separate
    handle. Documented in both headers; a non-local sink cannot be recovered
    in place.
  • No object-storage client is added here.
  • C API and Python surface are unchanged (follow-ups below).

Tests

New (all pass):

  • cpp/test/file/memory_random_access_write_file_test.cc — sequential
    writes and position tracking, truncate shrink/grow/zero-fill and cursor
    semantics, seek_to_end, write-after-close, invalid arguments,
    release_data, and the local WriteFile driven through the interface.
  • cpp/test/file/write_sink_e2e_test.cc — table-model, non-aligned tree and
    aligned tree round trips written to the memory sink and read back through
    MemoryRandomAccessReadFile; byte-for-byte equality between the memory
    sink and a local file for the same workload; the restorable/recovery path
    still works on a local file.

Verified on macOS arm64, Release:

  • Full TsFile_Test: 1467 passed, 0 failed, 3 pre-existing skips.
  • The 13 new tests also run clean under
    DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib (macOS ASAN is unusable
    on the dev box).
  • clang-format 17.0.6 (the version pinned in spotless) applied to every
    touched file.

Benchmark

cpp/bench_mark/bench_mark_src/write_backend_benchmark.cc, built with
-DBUILD_BENCHMARK=ON -DBUILD_TEST=OFF:

./build/Release/write_backend_benchmark --points=1000000 --repeat=5

Workload: table model, one table with 1 tag + 4 fields
(INT64/INT64/DOUBLE/INT32), 1M rows in 20 tablets of 50k rows, deterministic
LCG values (poorly compressible) and 1..100 ms timestamp gaps. Default
encodings/compression from libtsfile_init(). macOS arm64, Release, page
cache warm.

sink run1 run2 run3 run4 run5 median MB/s Mvalues/s
local WriteFile 0.156 0.159 0.164 0.162 0.185 0.162 s 176.3 24.7
MemoryRandomAccessWriteFile 0.092 0.082 0.075 0.076 0.077 0.077 s 369.0 51.7

The memory sink is ~2.1× faster here because it replaces the write syscalls;
the encoded output is byte-for-byte identical to the local file (asserted by
MemorySinkMatchesLocalFileByteForByte).

Local path: before/after. The same workload was compiled against
develop (same data generation, local WriteFile only, same flags) and run
alternately with this branch's local arm:

build local median
develop baseline 0.157 s / 0.159 s / 0.170 s
this branch 0.162 s / 0.164 s / 0.162 s

The difference is within the run-to-run spread of individual runs
(0.152–0.201 s). The only change on the local path is one virtual dispatch
per TsFileIOWriter::flush_stream_to_file buffer write; an isolated
64 KiB-write microbenchmark on the in-memory sink (concrete call vs. the
interface pointer) shows a difference smaller than its own run-to-run noise
(~2–3 µs per call, ~3 GiB/s). The microbench is included in the benchmark
binary.

Follow-ups (not in this PR)

  • Object-storage sink (upload-on-close or spool-then-upload); no S3/OSS
    client belongs in the C++ core — this interface is the hook.
  • C API: write_file_new still creates local files only; expose a sink
    handle (including the memory sink) for embedders.
  • Python: TsFileWriter accepts a path only; a writable binary file-object
    bridge mirroring python_random_access_read_file.cc is the natural next
    step.
  • Recovery on non-local backends would need self_check to read from the
    sink's matching read source (or a local spool).

…e sink

Introduce RandomAccessWriteFile, the write-side counterpart of the
existing RandomAccessReadFile byte source, and make the fd-backed WriteFile
implement it. TsFileIOWriter, TsFileWriter, TsFileTableWriter,
TsFileTreeWriter and RestorableTsFileIOWriter now reach their sink through
the interface; path-based constructors, the C API and the Python binding
are unchanged, and the local write path keeps the same syscall loop.

Add MemoryRandomAccessWriteFile and MemoryRandomAccessReadFile so a
complete TsFile can be written to memory and read straight back, which
proves the write path is backend-independent. Recovery stays local-file
only: self_check reads the existing bytes back by path.

Replace the get_fd()-based liveness checks with is_opened() (fd >= 0),
keeping file_opened() as an alias.

Add unit and end-to-end tests (table, tree and aligned round trips,
byte-for-byte parity between the memory sink and a local file) and a
write_backend_benchmark.
@ColinLeeo
ColinLeeo merged commit 3173263 into apache:develop Nov 12, 2026
56 of 57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant