Skip to content

perf(core): cut serial requests in the column_stats lookup during read planning - #1010

Draft
yihua wants to merge 1 commit into
apache:mainfrom
yihua:perf/column-stats-pruning-reads
Draft

yihua wants to merge 1 commit into
apache:mainfrom
yihua:perf/column-stats-pruning-reads

Conversation

@yihua

@yihua yihua commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Description

closes #1007

A filtered plan prunes files with the metadata table's column_stats partition. The lookup now fences base files with the timeline the planner's metadata table handle already loaded instead of listing the metadata timeline again, and fetches the log files concurrently (bounded by hoodie.read.file.slice.read.concurrency) alongside the base files, which are still read one at a time. The merge order is unchanged. Every whole-file log read also drops the HEAD it sent before its GET, since the GET returns the object. The write path's caller (compute_partition_stats_records) keeps listing the metadata timeline and reads one log file at a time as before.

Impact, from a request trace of a cold filtered plan of TPC-H SF1 lineitem on S3 (non-partitioned COW table, column_stats holds 2 log-only file groups with 2 log files each): a re-plan with the same filter consists only of the column_stats lookup, and goes from 10 requests, each waiting on the previous one (LIST of the column_stats partition, LIST of the metadata timeline, then HEAD + GET per log file), in 372 ms, to 5 requests in 2 sequential steps (LIST, then 4 GETs concurrently) in 85 ms.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

fs_view_mdt_column_stats_pruning_reads_without_head_or_timeline_listing counts the requests of a filtered plan on a table with a metadata table: the column_stats partition is listed, the metadata timeline is not, no HEAD is sent, and the pruned file list matches footer pruning over a storage listing. test_load_column_stats_records_same_across_fence_source_and_read_concurrency checks that fencing with the loaded timeline admits the same slices as listing, and that the records read concurrently equal those read one file at a time. test_load_column_stats_records_merges_in_slice_order merges a key held by a base and two logs, with tombstones before and after newer records, at concurrency 1 and 4. The request counts and latencies above come from a request trace against S3.

…d planning

A filtered snapshot plan prunes files with the metadata table's
column_stats partition. The lookup listed the metadata timeline again,
sent a HEAD before each whole-file log read, and read every file one
after another.

- Fence column_stats base files with the metadata table handle's loaded
  timeline (completed instants, or below the earliest active instant)
  when the planner holds the handle. The write path still lists the
  metadata timeline.
- Drop the HEAD before a whole-file read: the GET returns the object.
- During read planning, fetch the log files concurrently, bounded by
  hoodie.read.file.slice.read.concurrency, alongside the base files,
  which are read one at a time. The write path reads one log file at a
  time as before. The merge order is unchanged: every base, then log
  blocks in slice and file order, with rollback targets collected
  across all logs.
@codecov

codecov Bot commented Nov 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.81443% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.97%. Comparing base (6ba42e7) to head (4f52668).

Files with missing lines Patch % Lines
crates/core/src/write/metadata.rs 91.17% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1010      +/-   ##
==========================================
+ Coverage   90.96%   90.97%   +0.01%     
==========================================
  Files         154      154              
  Lines       22070    22099      +29     
==========================================
+ Hits        20076    20105      +29     
  Misses       1994     1994              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Column stats pruning during read planning issues its requests one after another

1 participant