Skip to content

feat(cpp): forward read options through file slice reads - #767

Open
codope wants to merge 3 commits into
apache:mainfrom
codope:cpp/read-options-637
Open

codope wants to merge 3 commits into
apache:mainfrom
codope:cpp/read-options-637

Conversation

@codope

@codope codope commented Sep 7, 2026 •

Copy link
Copy Markdown
Member

Description

read_file_slice and read_file_slice_from_paths built ReadOptions::new() internally, so a C++ caller got no column projection, no filters and no read-optimized mode even though FileGroupReader supports all of them. A merge-on-read slice was always materialized with every column.

This adds two shared bridge structs and two entry points:

struct HudiFilter      { rust::String field; rust::String op; rust::String value; };
struct HudiReadOptions { rust::Vec<rust::String> projection;   // empty = all columns
                         rust::Vec<HudiFilter>   filters;      // op: = != < <= > >= IN "NOT IN"
                         bool                    read_optimized; // base file only, skip log files
};
read_file_slice_with_options(slice, options);
read_file_slice_from_paths_with_options(base_file_path, log_file_paths, options);

Filters use the same (field, op, value) grammar as the Python binding so the two bindings stay in sync.

read_optimized is a bool rather than a key=value bag because it is the only read config a single slice read acts on. A time window (hoodie.read.as.of.timestamp, start/end.timestamp) is applied from the reader's configs, so it belongs on new_file_group_reader_with_options; the header doc says so. false defers to the reader's own setting rather than forcing the mode off, so a zero-initialized struct cannot undo read-optimized mode on a reader opened with it.

The existing two entry points stay as wrappers over default options.

Closes #637. Related: apache/hudi#18308.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

Unit tests in cpp/src/lib.rs: option mapping (defaults, where read_optimized: false sets nothing; every field; bad operator); a MOR slice with real log files read through read_file_slice_with_options with a projection and an id = <max> filter; the paths entry point driven through real CxxString/CxxVector arguments with a projection and filter; and read-optimized mode proven by a slice whose only log file does not exist, which reads with read_optimized: true and fails naming the log file without it.

@codope
codope requested a review from xushiyan as a code owner September 7, 2026 09:24
@codecov

codecov Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.96%. Comparing base (6ba42e7) to head (eb740f3).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #767   +/-   ##
=======================================
  Coverage   90.96%   90.96%           
=======================================
  Files         154      154           
  Lines       22070    22070           
=======================================
  Hits        20076    20076           
  Misses       1994     1994           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codope
codope requested a review from yihua September 7, 2026 14:12
@codope

codope commented Oct 27, 2026

Copy link
Copy Markdown
Member Author

@yihua Thanks for landing #766 . I will rebase this PR and update it. If both can go in the upcoming 0.6.0 release, that would be great!

@yihua yihua left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. A few comments on how the read configs are read and applied.

Separately, the release only publishes hudi-core, hudi-datafusion and hudi to crates.io. Does hudi-cpp need to be published as well, so C++ consumers can pick up these entry points from a release?

Comment thread cpp/src/lib.rs Outdated
}

/// Options for one read. An empty `projection` reads every column;
/// `hudi_options` are `key=value` read configs such as

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking: For the eager slice reads, the only per-read key the core reader acts on is hoodie.read.use.read_optimized.mode; as.of.timestamp, start/end.timestamp and stream.batch_size set here are accepted and ignored (see the doc on FileGroupReader::read_file_slice_from_paths), so a C++ caller passing a time window per read gets unfiltered rows back with no error. Could the doc here say that read-optimized mode is the one honored per read, and that time-window options belong on new_file_group_reader_with_options? The PR description's "for free" line for batch_size and as.of.timestamp probably wants the same correction.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! the commit-time window comes from the reader's configs, so a per-read as.of/start/end was accepted and ignored. With the field now a bool read_optimized (see the next thread), there's no way to pass those per read anymore, and the struct doc points time-window options to new_file_group_reader_with_options. I've also corrected the "for free" line in the description.

Comment thread cpp/src/lib.rs Outdated
struct HudiReadOptions {
projection: Vec<String>,
filters: Vec<HudiFilter>,
hudi_options: Vec<String>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking: Is hudi_options needed on the per-read struct? A C++ caller can already get read-optimized mode by passing hoodie.read.use.read_optimized.mode=true to new_file_group_reader_with_options, since resolve_read_options carries reader-level read configs into every read, and that is the only per-read key the eager slice reads act on. Dropping it would keep the new surface to projection and filters and avoid a field where most keys are silently ignored; if a per-read toggle on a shared reader is the use case, a bool read_optimized would say so more precisely.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed.. replaced it with bool read_optimized, since a per-read toggle on a shared reader is the use case (one reader per table across many splits).

Comment thread cpp/src/lib.rs
guard("new_file_group_reader_with_options", || {
let base_uri = cxx_str(base_uri)?;

let opt_vec = key_values(cxx_strs(options)?)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking: The constructor now throwing on an option without = is a behavior change to an existing entry point that the read-options feature doesn't depend on. Would you consider splitting it into its own fix(cpp) PR so it shows up on its own in the release notes? Its justification also leans on hudi_options existing on the per-read struct (see the comment on line 54).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. It's out of this PR, and the constructor here drops a malformed option as before. I will create a separate ficx(cpp) PR

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created separate PR as dicussed above #942

@codope

codope commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

Thanks for the contribution. A few comments on how the read configs are read and applied.

Separately, the release only publishes hudi-core, hudi-datafusion and hudi to crates.io. Does hudi-cpp need to be published as well, so C++ consumers can pick up these entry points from a release?

@yihua I have addressed the comments.

On publishing hudi-cpp: I don't think crates.io gets C++ consumers much. They build through cpp/CMakeLists.txt, which runs cargo and the cxx-generated header, so the ASF source release, which already contains cpp/, is what they'd pick up. The crate also sets [lib] name = "hudi", which collides with its hudi dependency (probably that's why cargo test -p hudi-cpp --doc fails with E0464), so that would need sorting out before a publish anyway. I'd rather open a separate issue on how C++ artifacts get distributed (source release only, prebuilt per-platform libs, or a vcpkg/conan port) than grow this PR. Happy to file it if that sounds right.

@yihua

yihua commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution and for splitting out the constructor change into #942! LGTM, with a minor revision pushed in the latest commit that documents the \, and \\ escapes for IN / NOT IN values on HudiFilter.

codope and others added 2 commits November 7, 2026 21:07
read_file_slice and read_file_slice_from_paths built ReadOptions::new()
internally, so a C++ caller got no column projection, no filters and no
read-optimized mode even though the Rust reader supports all of them. A
merge-on-read slice was always materialized with every column.

Add HudiReadOptions (projection, filters, read_optimized) as a shared
bridge struct and *_with_options variants of both reads. Filters use the
same (field, op, value) grammar as the Python binding. read_optimized is
a bool rather than a key=value bag because it is the only read config a
single slice read acts on; a time window belongs on the reader. false
defers to the reader's own setting, so a zero-initialized struct cannot
turn off read-optimized mode on a reader opened with it. The old entry
points stay as wrappers over default options.

Closes apache#637

Signed-off-by: codope <sagarsumit09@gmail.com>
State on the HudiFilter doc, which cxx carries into the generated C++
header, that IN / NOT IN items are trimmed, `\,` is a literal comma and
`\\` a literal backslash, matching what the core filter parser accepts.
@yihua
yihua force-pushed the cpp/read-options-637 branch from a195aa4 to 80155ed Compare November 8, 2026 05:12
@codope

codope commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Thanks @yihua ! The revision looks good. Could you please merge this PR?

@codope
codope requested a review from yihua November 9, 2026 07:37
@codope

codope commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

@yihua gentle ping on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cpp: read_file_slice{,_from_paths} ignore ReadOptions (no projection/filter pushdown)

2 participants