[DO NOT MERGE] Test apache/arrow-rs#10852 (adaptive RowSelection construction) with ClickBench - #25422
Draft
alamb wants to merge 16 commits into
Draft
[DO NOT MERGE] Test apache/arrow-rs#10852 (adaptive RowSelection construction) with ClickBench#25422alamb wants to merge 16 commits into
alamb wants to merge 16 commits into
Conversation
Pin arrow/parquet crates to arrow-rs rev 901e69f913bb8a1738009218898959e8f4cd7ec8 via [patch.crates-io] until arrow 60.0.0 is released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tore 0.14, etc) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # datafusion/core/tests/user_defined/user_defined_aggregates.rs # datafusion/core/tests/user_defined/user_defined_scalar_functions.rs # datafusion/core/tests/user_defined/user_defined_window_functions.rs # datafusion/ffi/src/udaf/mod.rs # datafusion/optimizer/src/optimizer.rs
…anges - data_buffers() now returns &Arc<[Buffer]> and GenericByteViewArray::new_unchecked takes Arc<[Buffer]> - ParquetMetaData::column_index()/offset_index() replaced by page_index() returning the new PageIndex struct - PageIndex cannot be constructed outside the parquet crate (apache/arrow-rs#10824): patch serialized index bytes in statistics_order_tests and drop the synthetic page index from single_column_metadata until upstream exposes a constructor - New ColumnOrder/SortOrder variants: INT96 now reports SortOrder::INT96_TIMESTAMP (still treated as untrusted for min/max), and ColumnOrder union fields 2/3 are now recognized
…adapt to API changes
# Conflicts: # Cargo.lock # datafusion/common/src/dfschema.rs # datafusion/physical-expr/src/planner.rs
- Use PageIndexBuilder (apache/arrow-rs#10842) to restore the page index tests in statistics_order_tests.rs - Update RunEndEncoded Display expectation (apache/arrow-rs#10840) - Update decimal formatting expectation (apache/arrow-rs#10869) - Update memory size expectations Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Resolved conflicts in Cargo.lock and datafusion/physical-expr/src/scalar_subquery.rs (arrow 60 field metadata conversion combined with main's destructuring). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…or-60.0.0 @ d4079008) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
github-actions
Bot
added
documentation
github-actions
Bot
added
ffi
Contributor
Author
|
run benchmark clickbench_pushdown |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing adaptive-selector-arrow-pr-10852 (a8cb182) to 140c7c5 (merge-base) diff Run configurationrun benchmark clickbench_pushdownResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing adaptive-selector-arrow-pr-10852 (a8cb182) to 140c7c5 (merge-base) diff Run configurationrun benchmark clickbench_pushdownCPU Details (lscpu)Details
Resource Usageclickbench_pushdown — base (merge-base)
clickbench_pushdown — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
ReadPlanBuilder::with_predicate_optionsarrow-rs#10776DO NOT MERGE: this PR exists only to run the DataFusion benchmarks against a pre-release arrow-rs branch.
Rationale for this change
apache/arrow-rs#10852 changes how the parquet reader builds
RowSelections under theAutorow selection policy when predicate results are scattered. I want to measure its effect on ClickBench withpushdown_filtersenabled.What changes are included in this PR?
Based on #25335 (arrow/parquet 60.0.0), plus a
[patch.crates-io]section pinning all arrow/parquet crates to https://github.com/alamb/arrow-rs/tree/adaptive-selector-60.0.0, which is the 60.0.0 tag with the four commits from apache/arrow-rs#10852 cherry-picked on top (commitd4079008).What is the testing strategy for this PR?
Benchmarks only. Local results for ClickBench q24 with
datafusion.execution.parquet.pushdown_filters = true(5 runs x 15 executions each,profilingprofile) are in the review at apache/arrow-rs#10852 (review): roughly 10% faster with the arrow-rs change.Are there any user-facing changes?
No.