Pluggable Backend Interface with DataFusion for Bounded-Memory Compute - #3716

Closed

qzyu999 wants to merge 37 commits into

apache:mainfrom

qzyu999:pluggable-backend-init

Commits on Jul 25, 2026

Core: Add pluggable execution backend with DataFusion for OOM-resilient operations
qzyu999
committed
d87a050

Commits on Jul 26, 2026

Fix pre-commit check failures: pydocstyle, codespell, markdownlint, uv.lock
qzyu999
committed
1530069

Commits on Jul 27, 2026

Fix lint and type errors for pluggable backend execution module
qzyu999
committed
dbe2a28
fix: add type annotations and fix linting issues in execution module
qzyu999
committed
c14961c
fix: improve type annotations in execution module and tests
qzyu999
committed
2389585
Fix ruff and mypy errors in execution module tests - partial (96 remaining)
qzyu999
committed
a001e8b
Continue fixing mypy errors in execution module tests (88 remaining)
qzyu999
committed
b46ba03
Fix mypy type errors in pluggable backend tests
qzyu999
committed
f7f8646
Restore transforms.py and avro/file.py to main state
qzyu999
committed
9ac1673
Fix import sorting and formatting (ruff)
qzyu999
committed
9fc763e
Fix schema reconciliation to always apply when file_schema is valid
qzyu999
committed
b094122
Fix unbound predicate and file:// URI handling in orchestrate_scan
qzyu999
committed
ea4cd97
Fix schema reconciliation and predicate binding edge cases
qzyu999
committed
c8e5b98
Fix type promotion and ArrowScan deprecation warning in tests
qzyu999
committed
e8537a0
Fix delete to use orchestrate_scan for schema reconciliation
qzyu999
committed
77019d5
Fix test isolation and name mapping for schema evolution
qzyu999
committed
661b423
Fix orchestrate_scan to use row_filter instead of task.residual for read pushdown
qzyu999
committed
9cc3e46
Fix filter logic and add column rename reconciliation
qzyu999
committed
016f07b
Fix partition value projection and post-filter ordering
qzyu999
committed
6234c3e
Fix e2e test: use int32 to match IntegerType schema
qzyu999
committed
7ccb81b
Fix Spark e2e test: don't require positional delete files
qzyu999
committed
14e4f35
Fix to_arrow_batch_reader to respect limit parameter
qzyu999
committed
d5830a4
Skip Puffin delete files (v3 delete vectors) in positional delete handling
qzyu999
committed
9f8741b
Apply ruff formatting
qzyu999
committed
cb5c2c1
Use logger instead of warnings.warn for Puffin DV skip message
qzyu999
committed
b221ef9
Support Puffin delete vectors (DVs) in Iceberg v3 scans
qzyu999
committed
c823bf8
Merge origin/main into pluggable-backend-init
qzyu999
committed
f461d36
Fix post-filter to use task.residual and preserve dictionary encoding
qzyu999
committed
444a67b
Fix post-filter: use row_filter, handle already-bound predicates
qzyu999
committed
ae1cc66
Skip post-filter when pushdown handled the filter (fixes UUID issue)
qzyu999
committed
2b82fb9
Fix post-filter: track actual pushdown instead of inferring from residual
qzyu999
committed
ea07a19

Commits on Jul 28, 2026