fix(table): accept DataSplit version 9 and mirror _WRITE_COLS_SEQUENCES - #745
Open
JunRuiLee wants to merge 4 commits into
Open
fix(table): accept DataSplit version 9 and mirror _WRITE_COLS_SEQUENCES#745JunRuiLee wants to merge 4 commits into
JunRuiLee wants to merge 4 commits into
Conversation
Java bumped `DataSplit#VERSION` from 8 to 9 when it appended `_WRITE_COLS_SEQUENCES` (per-column maximum sequence numbers) to `DataFileMeta.SCHEMA`. The split body itself did not change -- only the per-file `DataFileMeta` row grew by one nullable trailing field -- but `DataSplit::deserialize` accepted version 8 exclusively, so every split produced by current Java was rejected with `Unsupported`. Carry the new field on `DataFileMeta` and make the row codec layout-aware: the 20-field layout (Java `DataFileMetaWriteColsLegacySerializer`) and the 21-field layout (Java `DataFileMetaSerializer`) share slots 0..=19 and differ only in arity, which the enclosing split's version selects. Only the current layout is written, matching Java keeping its legacy serializers deserialize-only. The field is also decoded from manifest entries and declared in the manifest Avro schema, so a file read from a manifest and handed on inside a split no longer silently drops it. Goldens come from the Java compatibility fixtures: `datasplit_v9.bin` is the raw body of `compatibility/datasplit-v9`, and the two `split_v1_*.bin` fixtures are refreshed from their Java counterparts, whose embedded `DataSplit` is now version 9.
The three fixtures live behind the `fulltext` feature, so they were not covered by a default-feature build.
JunRuiLee
marked this pull request as ready for review
August 25, 2026 12:19
JunRuiLee
marked this pull request as draft
August 25, 2026 12:26
The binding test pinned the emitted body version to 8.
Bound the array reservation by the remaining input: a declared element count came straight from the manifest, so a corrupt one could ask for an arbitrary allocation before the first element read failed. Reject a union index other than 0 or 1 for the nullable array instead of reading the following bytes as an array. Reject a DataFileMeta row shorter than the fixed part of the layout it is being read as, rather than letting the row constructor accept it. Also correct a wrong claim in the row codec doc: Java's legacy serializers do expose toRow, so they are not deserialize-only.
JunRuiLee
marked this pull request as ready for review
August 25, 2026 13:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
DataSplit::deserializeaccepted wire version 8 only, so anyDataSplitproduced bycurrent Paimon Java is rejected with
Unsupported.Java bumped
DataSplit#VERSION8 → 9 in apache/paimon#9313 when it appended_WRITE_COLS_SEQUENCES(per-column maximum sequence numbers) toDataFileMeta.SCHEMA.The split body itself is unchanged — the only difference is that the per-file
DataFileMetarow grew by one nullable trailing field:_WRITE_COLSDataFileMetaWriteColsLegacySerializer_WRITE_COLS_SEQUENCESat slot 20DataFileMetaSerializerChanges
DataFileMetacarriescolumn_max_sequence_numbers: Option<Vec<i64>>.DataFileMetaRowLayout. The two layouts shareslots 0..=19 and differ only in arity, which the enclosing split's version selects.
Only the current layout is written, mirroring Java keeping its legacy serializers
deserialize-only.
DataSplit::deserializeaccepts v8 and v9;serializeemits v9.so a file read from a manifest and handed on inside a split no longer silently drops it.
Goldens
All taken from the Java compatibility fixtures rather than generated here:
goldens/datasplit_v9.binis the raw body ofcompatibility/datasplit-v9(the Javafixture is an object stream:
TC_BLOCKDATALONG+ declared length + body +TC_ENDBLOCKDATA). Extracting the same way fromcompatibility/datasplit-v8reproducesthe existing
goldens/datasplit_v8.binbyte for byte, which is what validates theextraction.
goldens/split_v1_data.bin/split_v1_indexed.binare refreshed from their Javacounterparts, whose embedded
DataSplitis now v9. They were still carrying a v8 body.The v8 golden is kept as a legacy-decode test.
Note for reviewers
serializenow emits v9 rather than v8. That follows Java, which no longer writes v8, andv8 stays readable — but it does mean bytes produced here can no longer be read by a Paimon
older than apache/paimon#9313. Happy to gate it behind an explicit version argument instead
if you would rather not move the write side in the same change.