Skip to content

Align features inside lists in concatenate_datasets and map - #9115

Open
Dhruv-Sharma29 wants to merge 1 commit into
huggingface:mainfrom
Dhruv-Sharma29:fix/align-features-inside-lists
Open

Dhruv-Sharma29 wants to merge 1 commit into
huggingface:mainfrom
Dhruv-Sharma29:fix/align-features-inside-lists

Conversation

@Dhruv-Sharma29

Copy link
Copy Markdown
Contributor

Fixes #9114

_check_if_features_can_be_aligned and _align_features recurse into dicts but compared lists as a whole. So when a struct field inside a list was all None in one dataset or map() shard, List({..., "y": Value("null")}) and List({..., "y": Value("string")}) counted as different types and concatenate_datasets / map(num_proc>1) raised "The features can't be aligned".

Both functions now recurse into the items of two lists of the same kind (List with the same length, or LargeList). Lists of different kinds or lengths are still compared as before, and items with different non-null types still raise.

Tests:

  • test_features_alignment: four new cases (null field second and first, in a LargeList, in a list of lists).
  • test_features_alignment_of_different_lists_fails: lists with different item types, List vs LargeList, and different fixed lengths still can't be aligned (passes before and after this change).
  • test_concatenate_datasets_list_of_structs_with_null_field: both orders, checking the features and the data.
  • test_map_multiprocessing_list_of_structs_with_null_field: map(num_proc=2) where the first shard only has None for the field.

The new alignment cases and the two dataset tests fail on main and pass with the fix.

This touches _check_if_features_can_be_aligned, which #8927 rewrites. The two conflict there, but only that the list case needs its own branch in whichever version lands second.

Tests run locally (macOS arm64, Python 3.12): tests/features and tests/test_arrow_dataset.py give 957 passed, 182 skipped, and tests/test_iterable_dataset.py passes. ruff check and ruff format --check pass.

_check_if_features_can_be_aligned and _align_features recursed into
dicts but compared lists as a whole. So when a struct field inside a
list was all None in one dataset or map() shard, the null field could
not be aligned with the typed one and concatenate_datasets and
map(num_proc>1) raised "The features can't be aligned".

Recurse into the items of two lists of the same kind (List with the
same length, or LargeList).
Copilot AI balanced review requested due to automatic review settings November 21, 2026 05:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

concatenate_datasets and map(num_proc>1) fail when a struct field inside a list is all None in one part

2 participants