Skip to content

fix: write manifest _FILE as an Avro record - #743

Merged
JingsongLi merged 1 commit into
apache:mainfrom
shyjsarah:fix/manifest-file-record-java-compat
Aug 25, 2026
Merged

fix: write manifest _FILE as an Avro record#743
JingsongLi merged 1 commit into
apache:mainfrom
shyjsarah:fix/manifest-file-record-java-compat

Conversation

@shyjsarah

Copy link
Copy Markdown
Contributor

Summary

Follow up on #742 by aligning the Avro type of ManifestEntry._FILE with Java Paimon.

Java's ManifestAvroReader requires _FILE to be a RECORD, but paimon-rust currently writes it as a nullable UNION (["null", record]). This causes Java to reject Rust-written manifests with:

Unexpected Manifest Avro type for field _FILE: expected RECORD but found UNION.

Changes

  • Write _FILE directly as a non-null Avro record.
  • Add a schema regression test that verifies _FILE is a non-null record.
  • Keep a compatibility test proving paimon-rust can still read manifests written with the previous Rust field order and nullable _FILE union.

Testing

  • cargo fmt --all -- --check
  • cargo test -p paimon --lib (2380 passed, 2 ignored)
  • cargo clippy -p paimon --all-targets -- -D warnings

Notes

  • No public API changes.
  • Existing Rust-written manifests remain readable by paimon-rust.
  • The change only affects the schema used for newly written manifest files.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit e2d16f6 into apache:main Aug 25, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants