Repository navigation
Fix silent float-to-int truncation in Dataset.from_dict (#9102) - #9107
zhengdaodeguL wants to merge 2 commits into
Conversation
…9102) Use Arrow safe casts when encoding Python and NumPy floats as integer features so fractional values raise ArrowInvalid. Cover from_dict, from_list, nested features, and lossless conversions. Reported by PCTEJA.
1AbdullahButt
left a comment
There was a problem hiding this comment.
I ran Dataset.from_dict with features={"x": Value(...)} on main and on this branch (pyarrow 26.0.1, Python 3.11, in Docker).
| input | target | main | this PR |
|---|---|---|---|
[1.9, 2.5] |
int64 | [1, 2] (silent) |
ArrowInvalid |
[1.0, 2.0] |
int64 | [1, 2] |
[1, 2] |
[nan] |
int64 | ValueError |
ArrowInvalid |
[300.0] |
int8 | ArrowInvalid |
ArrowInvalid |
[np.float16(2.5)] |
int64 | [2] |
ArrowInvalid: Float value 16640 was truncated |
[np.longdouble(2.0)] |
int64 | [2] |
ArrowNotImplementedError: Unsupported numpy type 13 |
IterableDataset.from_generator yielding 1.9 |
int64 | [{'x': 1}] |
ArrowInvalid |
The main fix works. map, add_item, cast and from_pandas already raised on main, so this also makes from_dict and encode_example agree with them. The new tests pass: pytest tests/test_arrow_dataset.py -k "from_dict or float_to_int or safe_int" gives 17 passed here and 4 on main.
Two things I found:
- Regression for
np.longdouble.pa.scalar()does not acceptnp.longdouble, so a whole-number value likenp.longdouble(2.0)used to work and now fails.np.longdoubleis a subclass ofnp.floating, so it goes into the new branch. - Wrong value in the error for
np.float16.pa.scalar(np.float16(2.5))makes a half-float scalar, and the cast error prints its raw bits (16640) instead of2.5. Whole-number float16 values still work.
Converting to a Python float first fixes both for me: pa.scalar(float(value)).cast(self.pa_type).as_py(). With that change, np.longdouble(2.0) gives 2 and np.float16(2.5) reports Float value 2.500000 was truncated. A np.longdouble and a np.float16 case in the new parametrized test would cover this.
(I made this review with help from Claude Code. I ran the table above on main and on this branch, the float(value) change in a local patch, and the filtered test_arrow_dataset.py tests. I did not run the full test suite.)
|
Thank you for the thorough review — the comparison table and the local verification of the fix made this straightforward to address. Both findings are fixed in the latest commit. One refinement on the suggested The parametrized tests now cover: whole-number longdouble, fractional longdouble just above 1, longdouble precision and int64-max cases, and fractional float16 with an assertion on the numeric value in the error message. The new cases fail on the unpatched source (6 failures) and pass with the fix; the full |
Fixes #9102
Dataset.from_dict({"x": [1.9, 2.5]}, features=Features({"x": Value("int64")}))silently returned{'x': [1, 2]}— wrong data, no error. The library's owncast_array_to_featurepath raisesArrowInvalidfor the same values, so this was an inconsistency, not intended semantics.Root cause:
Value.encode_exampleused plainint(value)for integer features, which truncates floats. The fix routes Python and NumPy floats through Arrow's safe cast (pa.scalar(value).cast(...)), so fractional values raiseArrowInvalidwhile lossless conversions (1.0→1) still work. This coversfrom_dict,from_list, and nested features, since they share the same encoding path.Thanks to PCTEJA for reporting the issue with a precise root-cause analysis.
Verification
tests/test_arrow_dataset.py: 10 cases (float lists, NumPy float arrays incl. float32, nested lists, nested dicts — via bothfrom_dictandfrom_list) assertingArrowInvalidon truncation, plus 3 control cases proving lossless conversions still work. All 13 pass; the 10 regression cases fail on base (verified with a source-only base control).rufflint/format clean.