Repository navigation
Closed
Labels
P0c++Pull requests that update c++ codePull requests that update c++ codefeatureNew featureNew featurejavaPull requests that update Java codePull requests that update Java codepythonPull requests that update Python codePull requests that update Python code
Milestone
Description
Motivation
Users want to write TsFiles directly to object storage (OSS/S3) and read them back. Reading has been solved for the Python binding through user-provided seekable file objects, and the C++ reader now accepts a pluggable byte source. Writing has no equivalent hook in C++/C/Python, and the Java object-storage path depends on classes that live outside this repository.
Current state (verified)
Java
- The file-system abstraction already understands object storage:
FSType.OBJECT_STORAGE, theos://prefix inFSUtils/FSPath,OSFileInputFactory,OSFileOutputFactory,OSFSFactory, and the hybrid factories. - Those factories load implementations reflectively from the config strings
objectStorageFile,objectStorageTsFileInput, andobjectStorageTsFileOutput, whose defaults areorg.apache.iotdb.os.fileSystem.OSFile,OSTsFileInput, andOSTsFileOutput. Those classes are not in this repository, so the in-repoos://path is unverified and requires an external jar. - The write path goes through the
TsFileOutputinterface andFSFactoryProducer.getFileOutputFactory()(TsFileIOWriter,RestorableTsFileIOWriter,ForceAppendTsFileWriter); the read path goes throughTsFileInput.
C++
- Read:
RandomAccessReadFileis an abstract byte source, andTsFileReader::open(std::unique_ptr<RandomAccessReadFile>)accepts a caller-provided source. No object-storage implementation ships in the repository. - Write:
WriteFileis a concrete, fd-based class (create,write,get_position,sync,truncate,seek_to_end,close) used directly byTsFileIOWriter,RestorableTsFileIOWriter,TsFileWriter,TsFileTableWriter, and the C API (write_file_new(pathname, ...)).TsFileIOWriteralso callsfile_->get_fd()as an "is open" check, which an interface would have to replace.
Python
TsFileReader(source)accepts a path/bytes/PathLike or a seekable binary file object (create_tsfile_reader_from_python_file), so fsspec-style remote reads already work (see [Feature][C++][Python] Support remote random-access reading for TsFile #930).TsFileWriter(pathname, ...)accepts a path only; there is no write sink.
Proposal
- C++ write abstraction: introduce a
RandomAccessWriteFile(name TBD) interface mirroringRandomAccessReadFile, withis_opened(),write(...),get_position(),sync(),close(), and — required by recovery/append —truncate(size)andseek_to_end(). Replaceget_fd()-based checks withis_opened(). - In-memory implementation: a
MemoryRandomAccessWriteFilethat accumulates bytes and can be handed back or uploaded. This is useful for tests and for "build in memory, then upload" workflows. - Object-storage implementation: either an OSS/S3 backend in the repository, or a spooling sink that writes to a local temporary file and uploads on close (object stores are append-unfriendly). Define explicit behavior for
open_for_append, recovery, and truncation on objects; likely these are unsupported and must return a clear error. - Wire the sink through
TsFileWriter,TsFileTableWriter,RestorableTsFileIOWriter, the C API, and the Python writer. - Java: decide whether to ship an in-repo OSS implementation (with MinIO integration tests) or keep the external-class contract; either way, add an end-to-end test so the
os://path is covered, and auditTsFileOutputfor the operations a remote backend needs (positioning, truncation). - Tests: unit tests with the in-memory sink; integration tests against a local MinIO/S3 mock (gated by environment variables or a Maven profile); cross-language read-back of files written to object storage.
Acceptance criteria
- C++/C/Python can write a TsFile to a non-local sink through a documented API.
- In-memory sink tests prove the write path is backend-independent.
- At least one object-storage integration test writes and reads back a file, or the limitation is documented and returns an explicit error.
- Java object-storage read/write is either implemented and tested in-repo or explicitly documented as requiring external classes.
- No change to local-file behavior or performance.
Related
- [Feature][C++][Python] Support remote random-access reading for TsFile #930 — remote random-access reading for TsFile (read side, already closed)
- [Feature] Support appending to a normally closed TsFile in C++ and C APIs #923 — appending to a normally closed TsFile in C++ and C APIs
Metadata
Metadata
Assignees
Labels
P0c++Pull requests that update c++ codePull requests that update c++ codefeatureNew featureNew featurejavaPull requests that update Java codePull requests that update Java codepythonPull requests that update Python codePull requests that update Python code
Activity