Skip to content

Add experimental modular footer async reader - #11081

Draft
Jiayi-Wang-db wants to merge 2 commits into
apache:mainfrom
Jiayi-Wang-db:modular-footer-goal-1
Draft

Jiayi-Wang-db wants to merge 2 commits into
apache:mainfrom
Jiayi-Wang-db:modular-footer-goal-1

Conversation

@Jiayi-Wang-db

@Jiayi-Wang-db Jiayi-Wang-db commented Sep 14, 2026

Copy link
Copy Markdown

Which issue does this PR close?

This is an experimental draft for design review and does not yet close an issue.

Rationale for this change

Parquet's monolithic footer requires an object-store reader to fetch and decode all file metadata before it can plan projected reads. This prototype integrates the modular-footer format with the existing arrow-rs asynchronous reader so metadata for projected columns can be fetched incrementally while preserving the normal data-page decoding path.

What changes are included in this PR?

This PR adds an experimental ModularFooterReader, a footer-loading abstraction, and ParquetRecordBatchStreamBuilder::new_with_modular_footer. It decodes the MFP1 tail, selectively decodes projected placement metadata, reconstructs projected nested schemas, and hands ordinary Parquet column chunks to the existing async record-batch decoder. Retained modular state supports a staged filter path that fetches only requested columns' row-group statistics and installs native Parquet statistics before row-group selection. Optional offset and column indexes are fetched only for projected column chunks, with remote ranges batched through get_byte_ranges, and are attached to the standard page-pruning path. Optional file metadata preserves created_by, key/value metadata, and embedded ARROW:schema behavior when reader options require it.

The object-store example supports S3, GCS, Azure, HTTP, and local files. The remaining format-adaptation limitation is that multiple dictionary offsets are represented by the first offset because the existing ColumnChunkMetaData model exposes only one dictionary-page offset.

Are these changes tested?

Focused tests cover MFP1 parsing, projected placement decoding, selective native statistics loading, projected page-index loading and batched request shape, file metadata, nested projected schema reconstruction, and equality between modular-footer and legacy reads through the normal async data-page decoder. An end-to-end filter test loads c0 statistics, retains the matching row group, reads only c1 through the ordinary async data-page decoder, and matches the legacy result. The focused suite passes 10 tests with one external-fixture test ignored by default. cargo fmt, git diff --check, and strict Clippy for all async/object-store targets pass.

On a synthetic file with 10,000 columns, 10 row groups, and an 11,474,640-byte legacy footer, one-column modular metadata loading took approximately 2.83 ms versus 43.95 ms for the full legacy footer. A filter-planning benchmark for SELECT c1 WHERE c0 >= 5000 took approximately 2.65 ms, fetched a 1 MiB tail plus the 160-byte c0 statistics module, and pruned 5 of 10 row groups. These figures measure metadata planning rather than end-to-end cloud query latency.

Are there any user-facing changes?

This draft adds experimental public async-reader APIs for modular-footer files. Existing Parquet reader behavior is unchanged.

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant