fix(variant): reject FixedSizeList shredding

[cakeni]

Which issue does this PR close?

• Closes [Variant] Reconcile FixedSizeList support across shredded Variant APIs #10616.

Rationale for this change

The Variant shredding spec does not define FixedSizeList as a valid shredded typed_value. Keep writes strict by rejecting it in shred_variant, while normalizing legacy Arrow FixedSizeList metadata to List on read for backwards compatibility.

What changes are included in this PR?

• Reject DataType::FixedSizeList when selecting a shredding type.
• Normalize FixedSizeList typed_value to List in VariantArray::try_new.
• Remove the FSL-specific path from unshred_variant.
• Add regression coverage for invalid shredding and persisted Parquet read/unshred.

Are these changes tested?

• cargo test -p parquet-variant-compute --lib
• Persisted Parquet read/unshred regression for legacy FSL metadata.

Are there any user-facing changes?

shred_variant rejects FixedSizeList; existing data with Arrow FixedSizeList metadata is read as a regular List.

[Jefffrey]

the docs dont mention support for fixed size list: https://arrow.apache.org/docs/format/CanonicalExtensions.html#parquet-variant

should we be supporting this?

[cakeni]

You're right. FixedSizeList isn't part of the Variant shredding spec. I changed the PR to reject it in shred_variant instead of extending the reader APIs, and added it to the invalid shredding type coverage.

[Jefffrey]

cc @klion26 @sdf-jkl

[sdf-jkl]

This is a more difficult one because it is still an open question and has a long history. My head is spinning after thinking about it all day.

The Arrow canonical extension documentation does not list FixedSizeList as a supported typed_value type. However, I could not find FixedSizeList being considered or mentioned during the discussion in apache/arrow#47456, so I am hesitant to treat its omission as a deliberate decision to reject it.

At the Parquet level, an Arrow FixedSizeList is still encoded as an ordinary three-level LIST. The fixed length exists only in the Arrow schema metadata, so accepting it is not obviously a question of Parquet Variant specification compliance.

The relevant arrow-rs history is:

• [Variant]: Implement DataType::ListView/LargeListView support for cast_to_variant kernel #8241 introduced the ListLikeArray trait for cast_to_variant.
• [Variant]: Implement DataType::FixedSizeList support for cast_to_variant kernel #8282 implemented it for FixedSizeListArray.
• [Variant] Add list support to unshred_variant #8514 reused it to add unshredding support for all five Arrow list-like types, including FixedSizeList.
• Add support for FixedSizeList to variant_to_arrow #9663 added the FixedSizeList builder used by variant_to_arrow, variant_get, and shred_variant.

Using ListLikeArray for cast_to_variant makes sense: Variant value encoding has only ordinary arrays, so an Arrow FixedSizeList becomes a Variant array and its fixed length is discarded.

I think reusing the same abstraction to determine which types unshred_variant accepts was a mistake. ListLikeArray only says that an Arrow array can expose child values and an element range; it does not say that every implementing type is a valid shredded typed_value.

If we decide that FixedSizeList is not a supported Variant typed_value, then both shred_variant and unshred_variant should reject it. Keeping the FSL unshredding branch does not currently provide backwards-compatible reading because VariantArray::try_new already rejects an FSL typed_value before unshred_variant can be called.

Alternatively, if we want a lenient-reader policy for data previously produced by arrow-rs, we should explicitly accept or normalize FSL on read and add a persisted read/unshred test. Leaving only the existing internal unshredding branch does not accomplish that.

DuckDB provides the clearest comparison. Its fixed-length list equivalent, ARRAY, is explicitly rejected as a shredding target. Its Variant reader has no FSL-specific unshredding path: the physical Parquet LIST is inferred as a variable-length DuckDB LIST, and Arrow-specific fixed-size metadata is not restored. It therefore unshreds the value as an ordinary Variant array without preserving or validating a fixed length.

[cakeni]

I’d keep this strict for now: reject FixedSizeList in shred_variant and remove the unreachable FSL branch in unshred_variant, since VariantArray::try_new already rejects it. If we want to support older data written by arrow-rs, we can handle that separately by normalizing FSL to a regular list when reading it.

[sdf-jkl]

we can handle that separately by normalizing FSL to a regular list when reading it.

💯

[klion26]

Is the separation of this branch and underscore our expected behavior? Seems the error is the same.

[cakeni]

The two branches return the same error, but I kept them separate to make the unsupported type cases explicit. I can merge them if we prefer a single error path here.

[sdf-jkl]

I think we can merge them.

If we keep it explicit it would require a comment, because someone else reading the code would come to the same conclusion of dropping it.

Without the historical context a separate arm provides no value.

[sdf-jkl]

The scope of the issue was normalizing the FSL typed_value handling for Variant, so I propose we add more changes here:

• drop FSL handling from unshred_variant
• add FSL to variable-len list conversion in VariantArray::try_new for backwards compatibility and a more lenient reader

[sdf-jkl]

@cakeni should we add the changes I proposed above here?

[cakeni]

Updated this to normalize FixedSizeList to List in VariantArray::try_new and remove the FSL-specific path from unshred_variant. The persisted Parquet read regression covers the lenient-reader case.

[sdf-jkl]

Thanks @cakeni, LGTM with a nit.

[sdf-jkl]

Suggested change

[alamb]

@cakeni do you have time to address @sdf-jkl 's review?

[sdf-jkl]

Thanks @cakeni @klion26 🚀