Repository navigation
Conversation
FileStatsEstimator derived bytes per row from the sampled file's row-group compressed bytes, but divided each file's whole size by it. The footer and other file-level metadata counted in the file size and not in the per-row size, so small files, where that metadata dominates, got inflated num_records estimates. Bytes per row now come from the sample's whole file size, taken from the HEAD the footer read already makes, and the estimate is rounded, so a file like the sample maps back to its own row count. compression_ratio, behind byte_size, has the same overhead skew and is left for a follow-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #998
Description
FileStatsEstimatorderived bytes per row from the sampled file's row-group compressed bytes but divided each file's whole size by it, so small files, where the footer and metadata dominate, got inflatednum_records. It now uses the sample's whole file size, taken from the HEAD the footer read already makes, and rounds the estimate.compression_ratio(behindbyte_size) has the same overhead skew and is left for a follow-up.Impact: no extra request, since the size comes from the HEAD the footer read already makes. On TPC-H SF10 (local files, DuckDB with the duckdb-hudi join filters), the planned row estimate for
nationgoes from 6,037 to 25 (the actual count) and forregionfrom 480 to 1. With the correct sizes, DuckDB's join-order optimizer no longer builds the Q5 plan that produced a 1.8-billion-row intermediate join, and a profiled warm run of Q5 goes from 3,994 ms to 449 ms. That run also had #1004's earlier revision applied, which does not change the plan.How are the changes test-covered
test_from_parquet_footer_small_file_estimates_own_row_countandtest_get_file_slices_single_file_estimates_exact_num_recordsfail without the fix (33 and 802 estimated rows for files holding 5 and 4).