Skip to content

Fix silent float-to-int truncation in Dataset.from_dict (#9102) - #9107

Open
zhengdaodeguL wants to merge 2 commits into
huggingface:mainfrom
zhengdaodeguL:fix/from-dict-int64-silent-truncation-9102
Open

zhengdaodeguL wants to merge 2 commits into
huggingface:mainfrom
zhengdaodeguL:fix/from-dict-int64-silent-truncation-9102

Conversation

@zhengdaodeguL

Copy link
Copy Markdown

Fixes #9102

Dataset.from_dict({"x": [1.9, 2.5]}, features=Features({"x": Value("int64")})) silently returned {'x': [1, 2]} — wrong data, no error. The library's own cast_array_to_feature path raises ArrowInvalid for the same values, so this was an inconsistency, not intended semantics.

Root cause: Value.encode_example used plain int(value) for integer features, which truncates floats. The fix routes Python and NumPy floats through Arrow's safe cast (pa.scalar(value).cast(...)), so fractional values raise ArrowInvalid while lossless conversions (1.0 → 1) still work. This covers from_dict, from_list, and nested features, since they share the same encoding path.

Thanks to PCTEJA for reporting the issue with a precise root-cause analysis.

Verification

  • New regression tests in tests/test_arrow_dataset.py: 10 cases (float lists, NumPy float arrays incl. float32, nested lists, nested dicts — via both from_dict and from_list) asserting ArrowInvalid on truncation, plus 3 control cases proving lossless conversions still work. All 13 pass; the 10 regression cases fail on base (verified with a source-only base control).
  • Targeted suites: 315 passed, 7 skipped; ruff lint/format clean.

…9102)

Use Arrow safe casts when encoding Python and NumPy floats as integer features so fractional values raise ArrowInvalid. Cover from_dict, from_list, nested features, and lossless conversions.

Reported by PCTEJA.

@1AbdullahButt 1AbdullahButt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran Dataset.from_dict with features={"x": Value(...)} on main and on this branch (pyarrow 26.0.1, Python 3.11, in Docker).

input target main this PR
[1.9, 2.5] int64 [1, 2] (silent) ArrowInvalid
[1.0, 2.0] int64 [1, 2] [1, 2]
[nan] int64 ValueError ArrowInvalid
[300.0] int8 ArrowInvalid ArrowInvalid
[np.float16(2.5)] int64 [2] ArrowInvalid: Float value 16640 was truncated
[np.longdouble(2.0)] int64 [2] ArrowNotImplementedError: Unsupported numpy type 13
IterableDataset.from_generator yielding 1.9 int64 [{'x': 1}] ArrowInvalid

The main fix works. map, add_item, cast and from_pandas already raised on main, so this also makes from_dict and encode_example agree with them. The new tests pass: pytest tests/test_arrow_dataset.py -k "from_dict or float_to_int or safe_int" gives 17 passed here and 4 on main.

Two things I found:

  1. Regression for np.longdouble. pa.scalar() does not accept np.longdouble, so a whole-number value like np.longdouble(2.0) used to work and now fails. np.longdouble is a subclass of np.floating, so it goes into the new branch.
  2. Wrong value in the error for np.float16. pa.scalar(np.float16(2.5)) makes a half-float scalar, and the cast error prints its raw bits (16640) instead of 2.5. Whole-number float16 values still work.

Converting to a Python float first fixes both for me: pa.scalar(float(value)).cast(self.pa_type).as_py(). With that change, np.longdouble(2.0) gives 2 and np.float16(2.5) reports Float value 2.500000 was truncated. A np.longdouble and a np.float16 case in the new parametrized test would cover this.

(I made this review with help from Claude Code. I ran the table above on main and on this branch, the float(value) change in a local patch, and the filtered test_arrow_dataset.py tests. I did not run the full test suite.)

@zhengdaodeguL

Copy link
Copy Markdown
Author

Thank you for the thorough review — the comparison table and the local verification of the fix made this straightforward to address. Both findings are fixed in the latest commit.

One refinement on the suggested pa.scalar(float(value)): a blanket float() conversion loses precision for extended-precision np.longdouble (63-bit mantissa here). I verified that np.longdouble('9007199254740993') would become 9007199254740992, np.longdouble('9223372036854775807') rounds out of int64 range, and np.longdouble('1.0000000000000000001') would silently become 1. So finite longdouble values now take a separate path: check for a fractional part first (raising ArrowInvalid with str(value) in the message, since an f-string would coerce through Python float and lose precision there too), then convert exactly via a Python integer into the target Arrow integer type. All other floats use your suggested pa.scalar(float(value)) conversion, which is exact for float16/float32/float64.

The parametrized tests now cover: whole-number longdouble, fractional longdouble just above 1, longdouble precision and int64-max cases, and fractional float16 with an assertion on the numeric value in the error message. The new cases fail on the unpatched source (6 failures) and pass with the fix; the full tests/test_arrow_dataset.py and tests/features/test_features.py run is 652 passed, 52 skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataset.from_dict() silently truncates floats to ints — 1.9 becomes 1 with no error, while the library's own safe cast raises

2 participants