Summary
Cold Storage v2 allows late-arriving rows to update partitions that have already moved to object storage, without placing a full remote Parquet rewrite on the ingest path.
Late writes are captured in immutable local Delta Runs. Queries merge those Runs with the current remote base, while the cold-storage manager periodically compacts accumulated Runs into a new immutable remote Parquet version.
remote base + local Delta Runs → merged reads
→ asynchronous manager compaction
→ next immutable remote base
Motivation
Cold storage currently seals a partition before dropping its local data copy. Rows arriving after that boundary cannot update the immutable remote partition. This is problematic for workloads where data naturally arrives late, including market-data corrections, delayed telemetry, intermittent edge devices, and historical backfills.
Rewriting a complete remote partition for every late write would add object-store latency and substantial write amplification to ingestion. Cold Storage v2 separates late-write capture from remote publication, keeping writes and reads available while compaction and cleanup run asynchronously.
Capabilities
- Accept late writes into cold partitions through local Delta Runs
- Establish the same replicated capture boundary across every cluster instance
- Merge the remote base and visible Delta Runs during reads
- Batch many late writes into one bounded-memory remote compaction
- Publish each compacted base under a new immutable object key
- Activate new base versions for new snapshots without disrupting existing readers
- Retain prior versions for a configurable source-grace period
- Expose Delta debt, compaction lag, failures, active cutoffs, and retained-version metrics
Scope and compatibility
- Enterprise feature, disabled by default behind a coordinated cluster upgrade gate
- Existing Delta-active partitions remain active if the setting is subsequently disabled
- Initial version excludes tables with deduplication enabled until Delta survivor rules are defined
- Compaction scheduling and policy thresholds remain work in progress
Related
Summary
Cold Storage v2 allows late-arriving rows to update partitions that have already moved to object storage, without placing a full remote Parquet rewrite on the ingest path.
Late writes are captured in immutable local Delta Runs. Queries merge those Runs with the current remote base, while the cold-storage manager periodically compacts accumulated Runs into a new immutable remote Parquet version.
Motivation
Cold storage currently seals a partition before dropping its local data copy. Rows arriving after that boundary cannot update the immutable remote partition. This is problematic for workloads where data naturally arrives late, including market-data corrections, delayed telemetry, intermittent edge devices, and historical backfills.
Rewriting a complete remote partition for every late write would add object-store latency and substantial write amplification to ingestion. Cold Storage v2 separates late-write capture from remote publication, keeping writes and reads available while compaction and cleanup run asynchronously.
Capabilities
Scope and compatibility
Related