Skip to content

fix(core): estimate row counts from the sampled file's whole size - #1002

Draft
yihua wants to merge 1 commit into
apache:mainfrom
yihua:fix/row-estimate-file-overhead
Draft

yihua wants to merge 1 commit into
apache:mainfrom
yihua:fix/row-estimate-file-overhead

Conversation

@yihua

@yihua yihua commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

closes #998

Description

FileStatsEstimator derived bytes per row from the sampled file's row-group compressed bytes but divided each file's whole size by it, so small files, where the footer and metadata dominate, got inflated num_records. It now uses the sample's whole file size, taken from the HEAD the footer read already makes, and rounds the estimate. compression_ratio (behind byte_size) has the same overhead skew and is left for a follow-up.

Impact: no extra request, since the size comes from the HEAD the footer read already makes. On TPC-H SF10 (local files, DuckDB with the duckdb-hudi join filters), the planned row estimate for nation goes from 6,037 to 25 (the actual count) and for region from 480 to 1. With the correct sizes, DuckDB's join-order optimizer no longer builds the Q5 plan that produced a 1.8-billion-row intermediate join, and a profiled warm run of Q5 goes from 3,994 ms to 449 ms. That run also had #1004's earlier revision applied, which does not change the plan.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

test_from_parquet_footer_small_file_estimates_own_row_count and test_get_file_slices_single_file_estimates_exact_num_records fail without the fix (33 and 802 estimated rows for files holding 5 and 4).

FileStatsEstimator derived bytes per row from the sampled file's
row-group compressed bytes, but divided each file's whole size by it.
The footer and other file-level metadata counted in the file size and
not in the per-row size, so small files, where that metadata dominates,
got inflated num_records estimates.

Bytes per row now come from the sample's whole file size, taken from the
HEAD the footer read already makes, and the estimate is rounded, so a
file like the sample maps back to its own row count.

compression_ratio, behind byte_size, has the same overhead skew and is
left for a follow-up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FileStatsEstimator inflates num_records for small base files

1 participant