You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Exercised all three tables from the consumer side with PyIceberg main (0d584073), via the symlink route.
All three load and scan correctly.v1/partitioned-parquet-data decodes its spec as identity(2) + day(3) and returns 3 rows; v2/nested-types-parquet-data decodes the struct's child ids 5 and 6 with the list and map nulls intact; v2/simple-append-parquet-data returns 4 rows with the one null. Sizes measured 16161, 25967 and 28047 bytes, matching the 16–28 KB in the README. index.json resolved all three metadata paths as documented.
One correction to the Paths section. The substitution route does not work for PyIceberg as described, because the absolute prefix is not only in metadata.json: the manifest lists carry manifest_path and the manifests carry file_path, both absolute (file:/tmp/iceberg-verification/...). Measured:
No symlink, no substitution: StaticTable.from_metadatasucceeds and the scan then fails on the manifest list.
Prefix substituted in metadata.json only: the load succeeds, location is the new prefix, and the scan fails one level deeper, on the manifest_path inside the manifest-list Avro.
PyIceberg has no load-time path-rewriting hook — StaticTable.from_metadata takes a metadata path and properties and nothing else — so on Windows the honest recipe is "copy the corpus and rewrite the prefix in the metadata JSON and in the manifest lists and manifests", which is a different operation from substituting while loading. Worth saying that explicitly, since the two routes are currently presented as equally cheap.
Related, and worth a line in the procedure: a broken path setup surfaces at scan, not at load. A surface that asserts only on decoded metadata passes with no symlink at all, so a consumer can report green on a corpus it never actually opened.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Reference Tables, with additional guidelines in
README.mdon best practices in organizing and writing reference tables for consumer subscription.This fixture was developed using Claude, and is currently in draft mode while being reviewed by human