Skip to content

[Feature][C++] Object storage (OSS) read and write support #1055

Closed

Description

@ColinLeeo

Motivation

Users want to write TsFiles directly to object storage (OSS/S3) and read them back. Reading has been solved for the Python binding through user-provided seekable file objects, and the C++ reader now accepts a pluggable byte source. Writing has no equivalent hook in C++/C/Python, and the Java object-storage path depends on classes that live outside this repository.

Current state (verified)

Java

  • The file-system abstraction already understands object storage: FSType.OBJECT_STORAGE, the os:// prefix in FSUtils/FSPath, OSFileInputFactory, OSFileOutputFactory, OSFSFactory, and the hybrid factories.
  • Those factories load implementations reflectively from the config strings objectStorageFile, objectStorageTsFileInput, and objectStorageTsFileOutput, whose defaults are org.apache.iotdb.os.fileSystem.OSFile, OSTsFileInput, and OSTsFileOutput. Those classes are not in this repository, so the in-repo os:// path is unverified and requires an external jar.
  • The write path goes through the TsFileOutput interface and FSFactoryProducer.getFileOutputFactory() (TsFileIOWriter, RestorableTsFileIOWriter, ForceAppendTsFileWriter); the read path goes through TsFileInput.

C++

  • Read: RandomAccessReadFile is an abstract byte source, and TsFileReader::open(std::unique_ptr<RandomAccessReadFile>) accepts a caller-provided source. No object-storage implementation ships in the repository.
  • Write: WriteFile is a concrete, fd-based class (create, write, get_position, sync, truncate, seek_to_end, close) used directly by TsFileIOWriter, RestorableTsFileIOWriter, TsFileWriter, TsFileTableWriter, and the C API (write_file_new(pathname, ...)). TsFileIOWriter also calls file_->get_fd() as an "is open" check, which an interface would have to replace.

Python

Proposal

  1. C++ write abstraction: introduce a RandomAccessWriteFile (name TBD) interface mirroring RandomAccessReadFile, with is_opened(), write(...), get_position(), sync(), close(), and — required by recovery/append — truncate(size) and seek_to_end(). Replace get_fd()-based checks with is_opened().
  2. In-memory implementation: a MemoryRandomAccessWriteFile that accumulates bytes and can be handed back or uploaded. This is useful for tests and for "build in memory, then upload" workflows.
  3. Object-storage implementation: either an OSS/S3 backend in the repository, or a spooling sink that writes to a local temporary file and uploads on close (object stores are append-unfriendly). Define explicit behavior for open_for_append, recovery, and truncation on objects; likely these are unsupported and must return a clear error.
  4. Wire the sink through TsFileWriter, TsFileTableWriter, RestorableTsFileIOWriter, the C API, and the Python writer.
  5. Java: decide whether to ship an in-repo OSS implementation (with MinIO integration tests) or keep the external-class contract; either way, add an end-to-end test so the os:// path is covered, and audit TsFileOutput for the operations a remote backend needs (positioning, truncation).
  6. Tests: unit tests with the in-memory sink; integration tests against a local MinIO/S3 mock (gated by environment variables or a Maven profile); cross-language read-back of files written to object storage.

Acceptance criteria

  • C++/C/Python can write a TsFile to a non-local sink through a documented API.
  • In-memory sink tests prove the write path is backend-independent.
  • At least one object-storage integration test writes and reads back a file, or the limitation is documented and returns an explicit error.
  • Java object-storage read/write is either implemented and tested in-repo or explicitly documented as requiring external classes.
  • No change to local-file behavior or performance.

Related

Activity

  1. added this to the 2.5.1 milestone on Nov 2, 2026
  2. added
    javaPull requests that update Java code
    pythonPull requests that update Python code
    c++Pull requests that update c++ code
    on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0c++Pull requests that update c++ codefeatureNew featurejavaPull requests that update Java codepythonPull requests that update Python code

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions