Skip to content

fix(core): merge WAL generations across schema changes#175

Merged
beinan merged 3 commits into
mainfrom
fix/wal-schema-backcompat
Jul 23, 2026
Merged

fix(core): merge WAL generations across schema changes#175
beinan merged 3 commits into
mainfrom
fix/wal-schema-backcompat

Conversation

@beinan

@beinan beinan commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • evolve legacy base tables to the latest additive rollout_schema() before WAL merge, adding missing nullable columns as all-null arrays
  • align flushed WAL batches to the evolved base table schema so old generations null-fill missing fields and newer generations preserve populated fields
  • reject missing required columns, type mismatches, and unknown columns instead of silently dropping data
  • preserve the existing base-first scan_one_by_id and lsm_scanner_for_source read path
  • keep legacy WAL point-lookup projections to columns guaranteed across generations; absent post-Add claim-check offloaded message field columns to rollout schema #172 fields decode as None
  • cover old base/old WAL merge, old base/new-schema value preservation, and base/WAL point lookups

Testing

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test -p lance-context-core -p lance-context-master --lib
  • cargo test -p lance-context-core --lib claim_check -- --nocapture

@beinan
beinan force-pushed the fix/wal-schema-backcompat branch from 2038efa to d0abbdd Compare July 23, 2026 01:43
@beinan
beinan force-pushed the fix/wal-schema-backcompat branch from d0abbdd to 33f8e77 Compare July 23, 2026 03:18
@beinan
beinan merged commit c8d3740 into main Jul 23, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant