Fix v2 referenced data file schema - #3953
Draft
kevinjqliu wants to merge 15 commits into
Draft
Conversation
Add the optional referenced_data_file field to v2 manifests so position delete metadata is preserved. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Expect referenced_data_file in v2 FastAvro and REST manifest output now that field 143 is part of the schema. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
V2 DataFile accessors must be made layout-aware before approval.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds optional referenced_data_file field 143 to the v2 data-file schema and tests manifest round-tripping.
Changes:
- Extends the v2 schema.
- Adds v2 position-delete manifest coverage.
The v2 DataFile accessors still assume the v3 layout, causing incorrect field mapping and an IndexError when accessing referenced_data_file.
File summaries
| File | Summary |
|---|---|
tests/utils/test_manifest.py |
Adds v2 manifest serialization and deserialization coverage. |
pyiceberg/manifest.py |
Adds field 143 to the v2 schema; critical accessor layout issue remains. |
Review details
Suppressed comments (1)
pyiceberg/manifest.py:313
- This schema change also adds
referenced_data_fileto V2 Avro output, but existing compatibility assertions still delete that key from their expected V2 dictionaries (tests/avro/test_file.py:228-231andtests/integration/test_rest_manifest.py:115-117). Those tests will now fail because the reader returns the optional field asNone; update the V2 expectations to retain field 143 while continuing to omit only V3-only fields.
NestedField(
field_id=143,
name="referenced_data_file",
field_type=StringType(),
required=False,
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+309
to
+313
| NestedField( | ||
| field_id=143, | ||
| name="referenced_data_file", | ||
| field_type=StringType(), | ||
| required=False, |
Make the v2 FastAvro test explicitly map the default v3 in-memory projection to the v2 file schema instead of relying on positional alignment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Rename the test to state that it projects the canonical v3 record into a v2 Avro manifest. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Construct the manifest entry with v2 schemas and compare schema-defined fields directly, avoiding v3 projection and manual field removal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Build the v2 record and expected FastAvro output from an independent field fixture so missing schema fields cannot make the test pass circularly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
kevinjqliu
marked this pull request as draft
September 12, 2026 02:24
Keep explicit v2 from_args construction and compare against a literal FastAvro record so the compatibility assertion remains independent of the schema. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Convert positional records with their v2 struct so the FastAvro test compares the constructed entry directly without duplicating expected values. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The full v2 record comparison already verifies referenced_data_file. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Build the expected REST manifest entry with v2 from_args records and compare it directly without reflecting over v3 properties or deleting fields. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the existing todict behavior unchanged and use a separate schema-aware converter for v2 records. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Teach the existing test converter to map positional records with a struct and remove the separate record_to_dict helpers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use one schema-aware test converter for Avro and REST manifest comparisons instead of maintaining two subtly different todict implementations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Read referenced_data_file from its v2 position and keep v3-only accessors null for v2 records. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Project records to versioned writer schemas by Iceberg field ID so DataFile keeps one stable in-memory layout while v2 field ordering remains schema-correct. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add optional
referenced_data_file(field ID 143) to the v2 data-file schema and cover its manifest round trip. The field is permitted for v2 position deletes by the data file fields spec and was previously added only to the v3 projection in #3690.Testing
uv run pytest -q tests/utils/test_manifest.py