Adds lazy Parquet support to blosc2.open(), fetching only the physical fields and row groups needed

[FrancescAlted]

Adds lazy Parquet support to blosc2.open(), fetching only the physical fields and row groups needed.

• Aligns persistent caches with other remote formats: readable source directories and .b2d generations.
• Retains discovery metadata so warm opens require no source requests.
• Reads cached groups directly, avoiding full-group copies into memory.
• Supports shared caches, cache limits, refresh, and portable .b2z exports.
• Adds documentation and regression coverage for cache reuse, concurrency, and archive handling.

Cached sources are treated as immutable until refresh() is called.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved read-locking, cache validation, refresh, persistence, and API compatibility issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 2

Open (3)

• Hold owner lock across the entire Parquet read · New
• Evict oversized entries from the memory cache · New
• Reject shared_cache with eager Parquet loading · New

What changed in this PR

Adds experimental lazy Parquet support to blosc2.open(), including selective row-group/field reads and persistent caching. Several correctness and API issues remain unresolved.

Changes:

• Adds lazy Parquet routing, caching, refresh, and shared-cache support.
• Adds references, archives, documentation, tests, plans, and benchmarks.

File	Description
tests/​ctable/​test_remote_parquet.py	Parquet parity and cache regression tests
src/​blosc2/​schunk.py	Parquet open dispatch
src/​blosc2/​remote_parquet.py	Lazy Parquet reader and cache lifecycle
src/​blosc2/​remote_ctable.py	Parquet constructor and routing
src/​blosc2/​ctable_storage.py	Remote size handling
src/​blosc2/​ctable_remote_read.py	Parquet column reads
src/​blosc2/​b2objects.py	Reference decoding
plans/​remote-parquet.md	Implementation plan and measurements
plans/​remote-parquet-cache-unification.md	Cache lifecycle plan
plans/​remote-parquet-cache-metadata.md	Cache metadata plan
doc/​reference/​remotectable.rst	API reference updates
doc/​guides/​remote_tables.md	User documentation
bench/​remote_parquet_traffic.py	Synthetic traffic benchmark
bench/​remote_parquet_synthetic_optimized.txt	Synthetic benchmark results
bench/​remote_parquet_files.py	File and HTTP benchmark
bench/​remote_parquet_chicago_results.json	Local benchmark results
bench/​remote_parquet_chicago_http_results.json	HTTP benchmark results
bench/​remote_parquet_chicago_http_optimized.jsonl	Optimized HTTP results
bench/​remote_parquet_chicago_http_leaf_projection.jsonl	Leaf projection results
bench/​remote_parquet_chicago_http_capacity_hint.jsonl	Capacity-hint results

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate findings affect refresh correctness, URL identity, cache semantics, and data projection.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3

Open (3)

• Preserve meaningful query parameters in source URLs · New
• Refresh readers when no disk cache marker exists · New
• Reject reader option changes with retained groups · New

Resolved since last review (3)

• Hold owner lock across the entire Parquet read
• Reject shared_cache with eager Parquet loading
• Evict oversized entries from the memory cache

Previously missed (1)

In code that hasn't changed since last review

Normalize save destination and urlpath arguments

src/​blosc2/​remote_parquet.py:1102

RemoteCTable.save accepts urlpath= and rejects specifying both destination and urlpath (remote_ctable.py:460-475), but this override only looks at positional args or destination. Thus remote.save(urlpath=...) falls through with an unexpected keyword, and remote.save(dest, urlpath=...) does not produce the documented conflict. Normalize the two destination arguments before choosing the archive/reference path.

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical reopening and multiple cache, fetch-efficiency, and dependency issues remain unresolved.

Review effort: Lite
Findings: 1 · 1

Open (2)

• RemoteCTable rejects root CTable artifacts · New
• Corrupt disk cache metadata prevents recovery · New

Resolved since last review (3)

• Reject reader option changes with retained groups
• Refresh readers when no disk cache marker exists
• Preserve meaningful query parameters in source URLs