Skip to content

fix(core): read the metadata table as of a handle's data snapshot - #1012

Draft
yihua wants to merge 1 commit into
apache:mainfrom
yihua:fix/mdt-snapshot-fence
Draft

yihua wants to merge 1 commit into
apache:mainfrom
yihua:fix/mdt-snapshot-fence

Conversation

@yihua

@yihua yihua commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Description

closes #1009

A Table handle now reads the metadata table as of the data snapshot it opened on:

  • The valid-instant set for a read is built from the listing the handle's timeline was loaded from (pending instants of every action, completed rollbacks and restores), not from fresh listings.
  • The metadata table is opened pinned to the snapshot: every metadata compaction that may have folded in an instant the snapshot cannot see is dropped from its timeline, so reads take the file slice from before it. Planning, the public metadata read APIs and table stats all read this pinned table.
  • When the metadata table has archived the instants that earlier slice needs, or its cleaner removed the slice, the handle refuses the metadata table and plans from a storage listing. When a pinned files partition read fails, as after a clean that runs once the table is pinned, that plan lists storage too. Write paths keep the latest view.

One deliberate deviation from Java: the set is capped at the newest instant time the handle listed, so a metadata-only delta commit newer than the snapshot is excluded. Java loads the metadata timeline together with the data timeline and needs no cap; here the metadata table is opened at the first read, and an instant above the cap was created after the snapshot.

Impact: a plan no longer lists the data timeline again for pending and rollback instants, which is 2 fewer LISTs per plan. A handle whose metadata table has nothing past its snapshot does no extra I/O; the metadata partitions are listed for the cleaned-slice check only when pinning drops a compaction.

Known limits: a partially cleaned earlier slice is caught only when reading it fails, so a clean interrupted after removing the slice's logs but not its base reads without error and can miss the files those logs recorded. Storage listing cannot see a base file a later data clean deleted, so a plan that falls back can come back short, as on a table without a metadata table.

Over half of the added lines are tests. This textually conflicts with #985 and #1011 in the timeline loader, and needs a one-character change after #984.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

The snapshot_fence tests cover a stale handle after a commit and clean, after a metadata compaction past its snapshot, with a metadata table archived or cleaned past its snapshot, and with its pinned slice cleaned after the pin (which fails with No schema available for merge output without the planning fallback). They also check that a compaction that ran before an unseen commit completed is kept, that a fresh handle reads the latest metadata, and that a data instant pending at open stays excluded once it completes.

A data table handle pins its timeline at open, but it opened the
metadata table lazily at the first read and rebuilt the valid-instant
set from fresh listings of the data timeline. A commit and a clean that
landed between open and the first read were admitted, so the clean's
deletion of the handle's base file reached the listing while the
handle's timeline still rejected the newer slice, and the file group's
rows disappeared.

- The valid-instant set for a read comes from the listing the handle's
  timeline was loaded from: pending instants of every action and
  completed rollbacks and restores are captured with the completed
  commits, and the set is capped at the newest instant time in that
  listing. Java has no such cap because it loads the metadata timeline
  together with the data one; here a metadata-only delta commit newer
  than the snapshot is excluded.
- The handle opens its metadata table pinned to its snapshot: every
  metadata compaction that may have folded in an instant the snapshot
  cannot see is dropped from the metadata timeline, so reads take the
  slice from before it. Planning, the public metadata read APIs and
  table stats all read that pinned table.
- When the metadata table archived the instants that slice needs, or
  its cleaner removed the slice of a rewritten file group, the handle
  refuses the metadata table, keeps the refusal, and plans from a
  storage listing. When a pinned files-partition read fails, as after a
  clean that runs once the table is pinned, planning lists storage too.
- Write paths keep using the latest metadata table and valid-instant
  set.
@codecov

codecov Bot commented Nov 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.44444% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.02%. Comparing base (6ba42e7) to head (764d18f).

Files with missing lines Patch % Lines
crates/core/src/metadata/table/mod.rs 90.76% 12 Missing ⚠️
crates/core/src/timeline/loader.rs 92.30% 4 Missing ⚠️
crates/core/src/table/fs_view.rs 90.90% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1012      +/-   ##
==========================================
+ Coverage   90.96%   91.02%   +0.06%     
==========================================
  Files         154      154              
  Lines       22070    22229     +159     
==========================================
+ Hits        20076    20234     +158     
- Misses       1994     1995       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A table handle can plan from metadata-table state newer than its snapshot and silently drop a file group

1 participant