Repository navigation
Align features inside lists in concatenate_datasets and map - #9115
Open
Dhruv-Sharma29 wants to merge 1 commit into
Open
Dhruv-Sharma29 wants to merge 1 commit into
Dhruv-Sharma29 wants to merge 1 commit into
Conversation
_check_if_features_can_be_aligned and _align_features recursed into dicts but compared lists as a whole. So when a struct field inside a list was all None in one dataset or map() shard, the null field could not be aligned with the typed one and concatenate_datasets and map(num_proc>1) raised "The features can't be aligned". Recurse into the items of two lists of the same kind (List with the same length, or LargeList).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9114
_check_if_features_can_be_alignedand_align_featuresrecurse into dicts but compared lists as a whole. So when a struct field inside a list was allNonein one dataset ormap()shard,List({..., "y": Value("null")})andList({..., "y": Value("string")})counted as different types andconcatenate_datasets/map(num_proc>1)raised "The features can't be aligned".Both functions now recurse into the items of two lists of the same kind (
Listwith the same length, orLargeList). Lists of different kinds or lengths are still compared as before, and items with different non-null types still raise.Tests:
test_features_alignment: four new cases (null field second and first, in aLargeList, in a list of lists).test_features_alignment_of_different_lists_fails: lists with different item types,ListvsLargeList, and different fixed lengths still can't be aligned (passes before and after this change).test_concatenate_datasets_list_of_structs_with_null_field: both orders, checking the features and the data.test_map_multiprocessing_list_of_structs_with_null_field:map(num_proc=2)where the first shard only hasNonefor the field.The new alignment cases and the two dataset tests fail on main and pass with the fix.
This touches
_check_if_features_can_be_aligned, which #8927 rewrites. The two conflict there, but only that the list case needs its own branch in whichever version lands second.Tests run locally (macOS arm64, Python 3.12):
tests/featuresandtests/test_arrow_dataset.pygive 957 passed, 182 skipped, andtests/test_iterable_dataset.pypasses.ruff checkandruff format --checkpass.