Skip to content

Align null struct fields inside lists - #9116

Open
mintcoder-cn wants to merge 1 commit into
huggingface:mainfrom
mintcoder-cn:fix-align-null-struct-field-in-list
Open

mintcoder-cn wants to merge 1 commit into
huggingface:mainfrom
mintcoder-cn:fix-align-null-struct-field-in-list

Conversation

@mintcoder-cn

Copy link
Copy Markdown

Fixes #9114

concatenate_datasets rejects a list of structs when one part inferred Value("null") for a field that has a concrete type in another part. A struct that is not inside a list already aligns in that case. _align_features only recursed into dicts, so the two list types were compared as a whole.

List and LargeList features are now merged element by element. A null field takes the other part's type. Two different non-null types still raise. List(Value("null")) alignment is unchanged.

Verified with the reproduction in #9114 (Dataset.from_dict on both orders) and the existing null-list alignment cases.

concatenate_datasets treated a list of structs with one null field as a
different type from the same list with a concrete field. Merge List and
LargeList features element by element, the way bare structs already align.

Fixes huggingface#9114
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

concatenate_datasets and map(num_proc>1) fail when a struct field inside a list is all None in one part

1 participant