Skip to content

perf(core): read parquet footers concurrently when pruning files by column stats - #996

Draft
yihua wants to merge 1 commit into
apache:mainfrom
yihua:perf/concurrent-footer-pruning
Draft

yihua wants to merge 1 commit into
apache:mainfrom
yihua:perf/concurrent-footer-pruning

Conversation

@yihua

@yihua yihua commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Description

closes #994

Column-stats pruning read each candidate base file's parquet footer one after another, so planning a filtered read over N files waited for N sequential round trips to storage. A partition's footer reads now go out concurrently, bounded by hoodie.plan.listing.parallelism (the knob the partition listing already uses), and the retained file groups keep their input order. Partitions are still pruned one at a time, so the gain is largest for tables with many files per partition. The pruning decision for each file is unchanged.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

A new test counts footer reads in flight through the test-only counting object store and fails when the reads are serial. The existing footer-pruning tests cover the keep and prune decisions.

…olumn stats

Column-stats pruning read each candidate base file's footer one after
another. A partition's footer reads now go out concurrently, bounded by
hoodie.plan.listing.parallelism, and the retained file groups keep their
input order.
@codecov

codecov Bot commented Nov 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.97%. Comparing base (6ba42e7) to head (4a19ee9).

Files with missing lines Patch % Lines
crates/core/src/table/fs_view.rs 87.50% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #996      +/-   ##
==========================================
+ Coverage   90.96%   90.97%   +0.01%     
==========================================
  Files         154      154              
  Lines       22070    22073       +3     
==========================================
+ Hits        20076    20081       +5     
+ Misses       1994     1992       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Read parquet footers concurrently when pruning files by column stats

1 participant