Skip to content

feat(core): open a table over a caller-supplied object store - #1005

Draft
yihua wants to merge 2 commits into
apache:mainfrom
yihua:feat/inject-object-store
Draft

yihua wants to merge 2 commits into
apache:mainfrom
yihua:feat/inject-object-store

Conversation

@yihua

@yihua yihua commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

closes #1001

Description

Stacked on #989; review the last commit.

TableBuilder::with_object_store(Arc<dyn ObjectStore>) routes all of a table's I/O through a caller-supplied store, reads and writes alike: the metadata table, compaction including async, and any URI scheme. Storage options and per-reader storage overrides then do not change where I/O goes. Injection is in the Rust core API only. hudi-core now has pub use object_store so callers implement against the same version, which ties hudi-core's public API to object_store 0.13: moving to 0.14 becomes a breaking change for hudi-core.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

New tests read a table through an injected store under a scheme object_store does not recognise, check that no other store is built and that derived storages reuse the injected one, and run an async compaction whose reads and writes all go through it.

yihua added 2 commits November 8, 2026 13:49
Every Storage::new parsed the base URL into a fresh object store, so one
table open, plan and read built a new HTTP client, connection pool and
credential provider at each call site (table properties, timeline, file
system view, metadata table open and its reader, each file group reader).

Storage::derive builds a Storage for another base path and options that
reuses the parent's store when the options are equal and the base path has
the same store root (scheme and authority, plus the bucket for path-style
URLs). A table keeps the storage it opened with and derives the timeline,
file system view, file group readers and its metadata table from it. A
different bucket or different options still builds its own store, and
sharing stays within one table.
Every Storage built its store from the storage options and environment
through parse_url_opts, so a caller with its own I/O layer had no way
to route a table through it, and tables were limited to the URI schemes
object_store recognises.

TableBuilder::with_object_store(Arc<dyn ObjectStore>) routes all of a
table's I/O through the given store, reads and writes alike: the
timeline, listings, base and log files, the metadata table, and
compaction, including a background one, which now derives its table
from the parent's storage instead of reopening from options. Storage
derive reuses an injected store for every base path under the same
root whatever the options, and accepts any URI scheme for it. Storage
options and per-reader storage overrides then do not change where I/O
goes.

hudi-core re-exports object_store so a caller can implement a store
against the same version. Injection is exposed in the Rust core API
only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Open a table over a caller-supplied object store

1 participant