Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #375
User-provided options that contradicted what a table persists in
hoodie.properties(e.g.hoodie.table.type) were silently overwritten by the persisted value. The table builder now returns an error when they conflict.I believe that I have implemented everything that we discussed within the ticket but if there is anything that I missed or that should be changed please let me know!
This also picked up some whitespace fixes in hudi-rs/.github/workflows/ci.yml - feel free to throw that out if you want since it isn't technically related to this PR - this was grabbed when pushing to the repo to get through the checks that are ran!
How it works
hoodie.propertiesis compared against the user options before merging. Keys thatHudiTableConfigmodels are compared after parsing, so equivalent spellings are not flagged. Any other key, such ashoodie.record.merge.mode, is compared as a raw string.hoodie.table.ordering.fieldsagainst a persistedhoodie.table.precombine.field).ConfigError::ConflictingTableProperties.MetadataTableEnabled,MetadataTablePartitionsand the unmodeledhoodie.table.metadata.*keys),BasePath,Checksum,CreateSchema, or an emptyDatabaseName. The persisted value still wins for these, as before.build()resolves on a clone of the resolver, so calling it twice on one builder compares against the original user options each time.build_with_resolved_options(). A later change tohoodie.propertiescan't fail it.HudiConfigValuenow derivesPartialEqto compare parsed values.Behavior change
Code that passes options contradicting the table used to have them silently dropped and now gets an error.
Cross-binding impact
The change adds a
ConfigErrorvariant and aPartialEqderive, both additive. Nothing incrates/datafusion,python/,cpp/orcrates/ffireferencesConfigError, and the Python binding convertsCoreErrorinto aHudiCoreErrormessage string.How are the changes test-covered
Unit tests in
table/builder.rscover conflicts and equivalent spellings; values that fail to parse; exact comparison of string configs; aliases in both directions and a table that persists both a key and its alias; raw comparison of unmodeled keys; sorted multi-conflict reporting; and merge precedence. Builder-level tests against the V6 COW fixture cover a rejected conflict, a matching value (including building twice on one builder), and others.To see what a user gets on error, I ran
test_all_conflicts_are_reported_in_sorted_orderwith a temporaryprintln!of the error message (not included in this PR). The test passes two conflicting options, one modeled (hoodie.table.type) and one unmodeled (hoodie.record.merge.mode), and the output is:Both conflicts are reported at once, sorted by key, each with the user value and the persisted value.
Thanks in advance for the review I really appreciate it!