Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #767 +/- ##
=======================================
Coverage 90.96% 90.96%
=======================================
Files 154 154
Lines 22070 22070
=======================================
Hits 20076 20076
Misses 1994 1994 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
yihua
left a comment
There was a problem hiding this comment.
Thanks for the contribution. A few comments on how the read configs are read and applied.
Separately, the release only publishes hudi-core, hudi-datafusion and hudi to crates.io. Does hudi-cpp need to be published as well, so C++ consumers can pick up these entry points from a release?
| } | ||
|
|
||
| /// Options for one read. An empty `projection` reads every column; | ||
| /// `hudi_options` are `key=value` read configs such as |
There was a problem hiding this comment.
non-blocking: For the eager slice reads, the only per-read key the core reader acts on is hoodie.read.use.read_optimized.mode; as.of.timestamp, start/end.timestamp and stream.batch_size set here are accepted and ignored (see the doc on FileGroupReader::read_file_slice_from_paths), so a C++ caller passing a time window per read gets unfiltered rows back with no error. Could the doc here say that read-optimized mode is the one honored per read, and that time-window options belong on new_file_group_reader_with_options? The PR description's "for free" line for batch_size and as.of.timestamp probably wants the same correction.
There was a problem hiding this comment.
Good catch! the commit-time window comes from the reader's configs, so a per-read as.of/start/end was accepted and ignored. With the field now a bool read_optimized (see the next thread), there's no way to pass those per read anymore, and the struct doc points time-window options to new_file_group_reader_with_options. I've also corrected the "for free" line in the description.
| struct HudiReadOptions { | ||
| projection: Vec<String>, | ||
| filters: Vec<HudiFilter>, | ||
| hudi_options: Vec<String>, |
There was a problem hiding this comment.
non-blocking: Is hudi_options needed on the per-read struct? A C++ caller can already get read-optimized mode by passing hoodie.read.use.read_optimized.mode=true to new_file_group_reader_with_options, since resolve_read_options carries reader-level read configs into every read, and that is the only per-read key the eager slice reads act on. Dropping it would keep the new surface to projection and filters and avoid a field where most keys are silently ignored; if a per-read toggle on a shared reader is the use case, a bool read_optimized would say so more precisely.
There was a problem hiding this comment.
agreed.. replaced it with bool read_optimized, since a per-read toggle on a shared reader is the use case (one reader per table across many splits).
| guard("new_file_group_reader_with_options", || { | ||
| let base_uri = cxx_str(base_uri)?; | ||
|
|
||
| let opt_vec = key_values(cxx_strs(options)?)?; |
There was a problem hiding this comment.
non-blocking: The constructor now throwing on an option without = is a behavior change to an existing entry point that the read-options feature doesn't depend on. Would you consider splitting it into its own fix(cpp) PR so it shows up on its own in the release notes? Its justification also leans on hudi_options existing on the per-read struct (see the comment on line 54).
There was a problem hiding this comment.
Done. It's out of this PR, and the constructor here drops a malformed option as before. I will create a separate ficx(cpp) PR
155b774 to
d56aeee
Compare
@yihua I have addressed the comments. On publishing |
d56aeee to
e778fb7
Compare
|
Thanks for the contribution and for splitting out the constructor change into #942! LGTM, with a minor revision pushed in the latest commit that documents the |
read_file_slice and read_file_slice_from_paths built ReadOptions::new() internally, so a C++ caller got no column projection, no filters and no read-optimized mode even though the Rust reader supports all of them. A merge-on-read slice was always materialized with every column. Add HudiReadOptions (projection, filters, read_optimized) as a shared bridge struct and *_with_options variants of both reads. Filters use the same (field, op, value) grammar as the Python binding. read_optimized is a bool rather than a key=value bag because it is the only read config a single slice read acts on; a time window belongs on the reader. false defers to the reader's own setting, so a zero-initialized struct cannot turn off read-optimized mode on a reader opened with it. The old entry points stay as wrappers over default options. Closes apache#637 Signed-off-by: codope <sagarsumit09@gmail.com>
State on the HudiFilter doc, which cxx carries into the generated C++ header, that IN / NOT IN items are trimmed, `\,` is a literal comma and `\\` a literal backslash, matching what the core filter parser accepts.
a195aa4 to
80155ed
Compare
|
Thanks @yihua ! The revision looks good. Could you please merge this PR? |
|
@yihua gentle ping on this PR. |
Description
read_file_sliceandread_file_slice_from_pathsbuiltReadOptions::new()internally, so a C++ caller got no column projection, no filters and no read-optimized mode even thoughFileGroupReadersupports all of them. A merge-on-read slice was always materialized with every column.This adds two shared bridge structs and two entry points:
Filters use the same
(field, op, value)grammar as the Python binding so the two bindings stay in sync.read_optimizedis a bool rather than akey=valuebag because it is the only read config a single slice read acts on. A time window (hoodie.read.as.of.timestamp,start/end.timestamp) is applied from the reader's configs, so it belongs onnew_file_group_reader_with_options; the header doc says so.falsedefers to the reader's own setting rather than forcing the mode off, so a zero-initialized struct cannot undo read-optimized mode on a reader opened with it.The existing two entry points stay as wrappers over default options.
Closes #637. Related: apache/hudi#18308.
How are the changes test-covered
Unit tests in
cpp/src/lib.rs: option mapping (defaults, whereread_optimized: falsesets nothing; every field; bad operator); a MOR slice with real log files read throughread_file_slice_with_optionswith a projection and anid = <max>filter; the paths entry point driven through realCxxString/CxxVectorarguments with a projection and filter; and read-optimized mode proven by a slice whose only log file does not exist, which reads withread_optimized: trueand fails naming the log file without it.