Skip to content

fix(core): avoid pruning NaN values with column stats - #1022

Open
GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:fix/859-nan-column-stats
Open

GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:fix/859-nan-column-stats

Conversation

@GayathriSrividya

@GayathriSrividya GayathriSrividya commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Closes #859

Description

Parquet omits NaN values from float min/max statistics, which can make Spark incorrectly skip files for matching predicates. Mark Float32/Float64 column ranges as unknown when they contain NaN, so those incomplete stats are not used to prune files. Writer paths inspect original batches rather than decoding newly written Parquet again.

How are the changes test-covered

  • Automated tests (unit and/or integration tests)

Regression test: ./build-wrapper.sh cargo test -p hudi-core metadata::table::column_stats::tests::test_column_ranges_from_parquet_nan_values_are_not_prunable --lib — passed (1 passed, 1708 filtered out).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(core): keep NaN in the column stats range of float and double columns

1 participant