Skip to content

Fix pandas missing values becoming "<NA>" or crashing in from_dict (#9117) - #9118

Open
zhengdaodeguL wants to merge 1 commit into
huggingface:mainfrom
zhengdaodeguL:fix/pandas-missing-values-from-dict-9117
Open

zhengdaodeguL wants to merge 1 commit into
huggingface:mainfrom
zhengdaodeguL:fix/pandas-missing-values-from-dict-9117

Conversation

@zhengdaodeguL

Copy link
Copy Markdown

Fixes #9117

When a pandas nullable Series contains a missing value, Dataset.from_dict does not preserve it as a null: with an explicit string feature, pd.NA is converted to the literal string "<NA>"; without explicit features the same input raises ArrowTypeError. Nullable Int64, Float64, and boolean Series raise TypeError during conversion, and timestamps/durations with NaT fail as well. Arrow itself handles all of these correctly (pa.array(series) gives None), so the bug is in datasets' Series-to-Python conversion path.

_cast_to_python_objects now normalizes pd.NA and pd.NaT to None when converting a Series (DataFrame columns use the same path), before recursively casting the values.

Tests: new regression coverage for string, nullable integer/float/boolean, timestamp, and duration Series, DataFrame conversion, and Dataset.from_dict with and without explicit features. The new tests fail on the unpatched source (16 failures) and pass with the fix; the full tests/test_arrow_dataset.py and tests/features/test_features.py run is 643 passed, 52 skipped, 6 subtests passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pandas missing values become "<NA>" or cause Dataset.from_dict to fail

1 participant