Repository navigation
Conversation
Every Storage::new parsed the base URL into a fresh object store, so one table open, plan and read built a new HTTP client, connection pool and credential provider at each call site (table properties, timeline, file system view, metadata table open and its reader, each file group reader). Storage::derive builds a Storage for another base path and options that reuses the parent's store when the options are equal and the base path has the same store root (scheme and authority, plus the bucket for path-style URLs). A table keeps the storage it opened with and derives the timeline, file system view, file group readers and its metadata table from it. A different bucket or different options still builds its own store, and sharing stays within one table.
Every Storage built its store from the storage options and environment through parse_url_opts, so a caller with its own I/O layer had no way to route a table through it, and tables were limited to the URI schemes object_store recognises. TableBuilder::with_object_store(Arc<dyn ObjectStore>) routes all of a table's I/O through the given store, reads and writes alike: the timeline, listings, base and log files, the metadata table, and compaction, including a background one, which now derives its table from the parent's storage instead of reopening from options. Storage derive reuses an injected store for every base path under the same root whatever the options, and accepts any URI scheme for it. Storage options and per-reader storage overrides then do not change where I/O goes. hudi-core re-exports object_store so a caller can implement a store against the same version. Injection is exposed in the Rust core API only.
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #1001
Description
Stacked on #989; review the last commit.
TableBuilder::with_object_store(Arc<dyn ObjectStore>)routes all of a table's I/O through a caller-supplied store, reads and writes alike: the metadata table, compaction including async, and any URI scheme. Storage options and per-reader storage overrides then do not change where I/O goes. Injection is in the Rust core API only. hudi-core now haspub use object_storeso callers implement against the same version, which ties hudi-core's public API to object_store 0.13: moving to 0.14 becomes a breaking change for hudi-core.How are the changes test-covered
New tests read a table through an injected store under a scheme
object_storedoes not recognise, check that no other store is built and that derived storages reuse the injected one, and run an async compaction whose reads and writes all go through it.