Skip to content

Add Reference Tables - #6

Draft
sungwy wants to merge 1 commit into
mainfrom
reference-tables
Draft

sungwy wants to merge 1 commit into
mainfrom
reference-tables

Conversation

@sungwy

@sungwy sungwy commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Add Reference Tables, with additional guidelines in README.md on best practices in organizing and writing reference tables for consumer subscription.

This fixture was developed using Claude, and is currently in draft mode while being reviewed by human

@moomindani

Copy link
Copy Markdown

Exercised all three tables from the consumer side with PyIceberg main (0d584073), via the symlink route.

All three load and scan correctly. v1/partitioned-parquet-data decodes its spec as identity(2) + day(3) and returns 3 rows; v2/nested-types-parquet-data decodes the struct's child ids 5 and 6 with the list and map nulls intact; v2/simple-append-parquet-data returns 4 rows with the one null. Sizes measured 16161, 25967 and 28047 bytes, matching the 16–28 KB in the README. index.json resolved all three metadata paths as documented.

One correction to the Paths section. The substitution route does not work for PyIceberg as described, because the absolute prefix is not only in metadata.json: the manifest lists carry manifest_path and the manifests carry file_path, both absolute (file:/tmp/iceberg-verification/...). Measured:

  • No symlink, no substitution: StaticTable.from_metadata succeeds and the scan then fails on the manifest list.
  • Prefix substituted in metadata.json only: the load succeeds, location is the new prefix, and the scan fails one level deeper, on the manifest_path inside the manifest-list Avro.

PyIceberg has no load-time path-rewriting hook — StaticTable.from_metadata takes a metadata path and properties and nothing else — so on Windows the honest recipe is "copy the corpus and rewrite the prefix in the metadata JSON and in the manifest lists and manifests", which is a different operation from substituting while loading. Worth saying that explicitly, since the two routes are currently presented as equally cheap.

Related, and worth a line in the procedure: a broken path setup surfaces at scan, not at load. A surface that asserts only on decoded metadata passes with no symlink at all, so a consumer can report green on a corpus it never actually opened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants