Skip to content

fix(core): error on user options that conflict with table properties - #1013

Open
oglego wants to merge 1 commit into
apache:mainfrom
oglego:fix/375-conflicting-table-options
Open

oglego wants to merge 1 commit into
apache:mainfrom
oglego:fix/375-conflicting-table-options

Conversation

@oglego

@oglego oglego commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Description

Closes #375

User-provided options that contradicted what a table persists in hoodie.properties (e.g. hoodie.table.type) were silently overwritten by the persisted value. The table builder now returns an error when they conflict.

I believe that I have implemented everything that we discussed within the ticket but if there is anything that I missed or that should be changed please let me know!

This also picked up some whitespace fixes in hudi-rs/.github/workflows/ci.yml - feel free to throw that out if you want since it isn't technically related to this PR - this was grabbed when pushing to the repo to get through the checks that are ran!

How it works

  • Every key in hoodie.properties is compared against the user options before merging. Keys that HudiTableConfig models are compared after parsing, so equivalent spellings are not flagged. Any other key, such as hoodie.record.merge.mode, is compared as a raw string.
  • Aliases are honored in both directions (e.g. hoodie.table.ordering.fields against a persisted hoodie.table.precombine.field).
  • All conflicts are reported together, sorted by key, through a new ConfigError::ConflictingTableProperties.
  • No conflict is raised for metadata-related configs (MetadataTableEnabled, MetadataTablePartitions and the unmodeled hoodie.table.metadata.* keys), BasePath, Checksum, CreateSchema, or an empty DatabaseName. The persisted value still wins for these, as before.
  • build() resolves on a clone of the resolver, so calling it twice on one builder compares against the original user options each time.
  • Background compaction reopens the table from its own resolved options, so that reopen skips the check through an internal build_with_resolved_options(). A later change to hoodie.properties can't fail it.
  • HudiConfigValue now derives PartialEq to compare parsed values.

Behavior change

Code that passes options contradicting the table used to have them silently dropped and now gets an error.

Cross-binding impact

The change adds a ConfigError variant and a PartialEq derive, both additive. Nothing in crates/datafusion, python/, cpp/ or crates/ffi references ConfigError, and the Python binding converts CoreError into a HudiCoreError message string.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
  • Details are described below

Unit tests in table/builder.rs cover conflicts and equivalent spellings; values that fail to parse; exact comparison of string configs; aliases in both directions and a table that persists both a key and its alias; raw comparison of unmodeled keys; sorted multi-conflict reporting; and merge precedence. Builder-level tests against the V6 COW fixture cover a rejected conflict, a matching value (including building twice on one builder), and others.

To see what a user gets on error, I ran test_all_conflicts_are_reported_in_sorted_order with a temporary println! of the error message (not included in this PR). The test passes two conflicting options, one modeled (hoodie.table.type) and one unmodeled (hoodie.record.merge.mode), and the output is:

Config error: Conflicting user options with hoodie.properties: hoodie.record.merge.mode: user 'COMMIT_TIME_ORDERING', persisted 'EVENT_TIME_ORDERING'; hoodie.table.type: user 'MERGE_ON_READ', persisted 'COPY_ON_WRITE'

Both conflicts are reported at once, sorted by key, each with the user value and the persisted value.

Thanks in advance for the review I really appreciate it!

@oglego
oglego requested a review from xushiyan as a code owner November 11, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Handle conflicts between table properties and user-provided option

1 participant