Repository navigation
Conversation
A data table handle pins its timeline at open, but it opened the metadata table lazily at the first read and rebuilt the valid-instant set from fresh listings of the data timeline. A commit and a clean that landed between open and the first read were admitted, so the clean's deletion of the handle's base file reached the listing while the handle's timeline still rejected the newer slice, and the file group's rows disappeared. - The valid-instant set for a read comes from the listing the handle's timeline was loaded from: pending instants of every action and completed rollbacks and restores are captured with the completed commits, and the set is capped at the newest instant time in that listing. Java has no such cap because it loads the metadata timeline together with the data one; here a metadata-only delta commit newer than the snapshot is excluded. - The handle opens its metadata table pinned to its snapshot: every metadata compaction that may have folded in an instant the snapshot cannot see is dropped from the metadata timeline, so reads take the slice from before it. Planning, the public metadata read APIs and table stats all read that pinned table. - When the metadata table archived the instants that slice needs, or its cleaner removed the slice of a rewritten file group, the handle refuses the metadata table, keeps the refusal, and plans from a storage listing. When a pinned files-partition read fails, as after a clean that runs once the table is pinned, planning lists storage too. - Write paths keep using the latest metadata table and valid-instant set.
1 task done
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1012 +/- ##
==========================================
+ Coverage 90.96% 91.02% +0.06%
==========================================
Files 154 154
Lines 22070 22229 +159
==========================================
+ Hits 20076 20234 +158
- Misses 1994 1995 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
closes #1009
A
Tablehandle now reads the metadata table as of the data snapshot it opened on:filespartition read fails, as after a clean that runs once the table is pinned, that plan lists storage too. Write paths keep the latest view.One deliberate deviation from Java: the set is capped at the newest instant time the handle listed, so a metadata-only delta commit newer than the snapshot is excluded. Java loads the metadata timeline together with the data timeline and needs no cap; here the metadata table is opened at the first read, and an instant above the cap was created after the snapshot.
Impact: a plan no longer lists the data timeline again for pending and rollback instants, which is 2 fewer LISTs per plan. A handle whose metadata table has nothing past its snapshot does no extra I/O; the metadata partitions are listed for the cleaned-slice check only when pinning drops a compaction.
Known limits: a partially cleaned earlier slice is caught only when reading it fails, so a clean interrupted after removing the slice's logs but not its base reads without error and can miss the files those logs recorded. Storage listing cannot see a base file a later data clean deleted, so a plan that falls back can come back short, as on a table without a metadata table.
Over half of the added lines are tests. This textually conflicts with #985 and #1011 in the timeline loader, and needs a one-character change after #984.
How are the changes test-covered
The
snapshot_fencetests cover a stale handle after a commit and clean, after a metadata compaction past its snapshot, with a metadata table archived or cleaned past its snapshot, and with its pinned slice cleaned after the pin (which fails withNo schema available for merge outputwithout the planning fallback). They also check that a compaction that ran before an unseen commit completed is kept, that a fresh handle reads the latest metadata, and that a data instant pending at open stays excluded once it completes.