Add experimental modular footer async reader - #11081
Draft
Jiayi-Wang-db wants to merge 2 commits into
Draft
Jiayi-Wang-db wants to merge 2 commits into
Jiayi-Wang-db wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
This is an experimental draft for design review and does not yet close an issue.
Rationale for this change
Parquet's monolithic footer requires an object-store reader to fetch and decode all file metadata before it can plan projected reads. This prototype integrates the modular-footer format with the existing arrow-rs asynchronous reader so metadata for projected columns can be fetched incrementally while preserving the normal data-page decoding path.
What changes are included in this PR?
This PR adds an experimental
ModularFooterReader, a footer-loading abstraction, andParquetRecordBatchStreamBuilder::new_with_modular_footer. It decodes the MFP1 tail, selectively decodes projected placement metadata, reconstructs projected nested schemas, and hands ordinary Parquet column chunks to the existing async record-batch decoder. Retained modular state supports a staged filter path that fetches only requested columns' row-group statistics and installs native Parquet statistics before row-group selection. Optional offset and column indexes are fetched only for projected column chunks, with remote ranges batched throughget_byte_ranges, and are attached to the standard page-pruning path. Optional file metadata preservescreated_by, key/value metadata, and embeddedARROW:schemabehavior when reader options require it.The object-store example supports S3, GCS, Azure, HTTP, and local files. The remaining format-adaptation limitation is that multiple dictionary offsets are represented by the first offset because the existing
ColumnChunkMetaDatamodel exposes only one dictionary-page offset.Are these changes tested?
Focused tests cover MFP1 parsing, projected placement decoding, selective native statistics loading, projected page-index loading and batched request shape, file metadata, nested projected schema reconstruction, and equality between modular-footer and legacy reads through the normal async data-page decoder. An end-to-end filter test loads
c0statistics, retains the matching row group, reads onlyc1through the ordinary async data-page decoder, and matches the legacy result. The focused suite passes 10 tests with one external-fixture test ignored by default.cargo fmt,git diff --check, and strict Clippy for all async/object-store targets pass.On a synthetic file with 10,000 columns, 10 row groups, and an 11,474,640-byte legacy footer, one-column modular metadata loading took approximately 2.83 ms versus 43.95 ms for the full legacy footer. A filter-planning benchmark for
SELECT c1 WHERE c0 >= 5000took approximately 2.65 ms, fetched a 1 MiB tail plus the 160-bytec0statistics module, and pruned 5 of 10 row groups. These figures measure metadata planning rather than end-to-end cloud query latency.Are there any user-facing changes?
This draft adds experimental public async-reader APIs for modular-footer files. Existing Parquet reader behavior is unchanged.