Skip to content

Cold Storage v2: late writes and remote compaction #130

Description

@bluestreak01

Summary

Cold Storage v2 allows late-arriving rows to update partitions that have already moved to object storage, without placing a full remote Parquet rewrite on the ingest path.

Late writes are captured in immutable local Delta Runs. Queries merge those Runs with the current remote base, while the cold-storage manager periodically compacts accumulated Runs into a new immutable remote Parquet version.

remote base + local Delta Runs → merged reads
                              → asynchronous manager compaction
                              → next immutable remote base

Motivation

Cold storage currently seals a partition before dropping its local data copy. Rows arriving after that boundary cannot update the immutable remote partition. This is problematic for workloads where data naturally arrives late, including market-data corrections, delayed telemetry, intermittent edge devices, and historical backfills.

Rewriting a complete remote partition for every late write would add object-store latency and substantial write amplification to ingestion. Cold Storage v2 separates late-write capture from remote publication, keeping writes and reads available while compaction and cleanup run asynchronously.

Capabilities

  • Accept late writes into cold partitions through local Delta Runs
  • Establish the same replicated capture boundary across every cluster instance
  • Merge the remote base and visible Delta Runs during reads
  • Batch many late writes into one bounded-memory remote compaction
  • Publish each compacted base under a new immutable object key
  • Activate new base versions for new snapshots without disrupting existing readers
  • Retain prior versions for a configurable source-grace period
  • Expose Delta debt, compaction lag, failures, active cutoffs, and retained-version metrics

Scope and compatibility

  • Enterprise feature, disabled by default behind a coordinated cluster upgrade gate
  • Existing Delta-active partitions remain active if the setting is subsequently disabled
  • Initial version excludes tables with deduplication enabled until Delta survivor rules are defined
  • Compaction scheduling and policy thresholds remain work in progress

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    enterpriseFeatures specific to QuestDB Enterprise

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions