Repository navigation
Conversation
… read
Opening a table read .hoodie/hoodie.properties and only then listed the
active timeline directory, so every open paid two sequential round trips
before the timeline was known.
TableBuilder::build now lists the layout-2 active timeline directory
(.hoodie/{hoodie.timeline.path}) concurrently with the properties read.
The timeline loader reuses that listing only when the resolved configs
select layout 2 with the same directory on the same base URL, and only
when the listing holds a file the store stamped strictly after
hoodie.properties: a writer changes the timeline before it rewrites the
properties (a layout upgrade migrates the timeline first), so only such
a file shows the listing reflects the properties that were read. Both
times are the store's, so the reader's clock does not enter. Equal
times, an empty or failed listing, or a properties time at the Unix
epoch (a store that keeps no last-modified time) list the timeline
again. A properties failure is returned as before, never the listing's
error.
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
closes #1008
Opening a table now lists the layout-2 active timeline directory (
.hoodie/{hoodie.timeline.path}) concurrently with thehoodie.propertiesread. The timeline loader reuses that listing only when the resolved configs select the same directory on the same base URL, and the listing holds a file the store stamped strictly afterhoodie.properties. A writer changes the timeline before it rewrites the properties (a layout upgrade migrates the timeline first), so only such a file shows the listing was taken after the properties that were read. Both times come from the store, so the reader's clock does not enter. Equal times, an empty or failed listing, or a properties time at the Unix epoch (a store that keeps no last-modified time) list the timeline again, and a properties failure is still returned as before.Impact: opening a table on S3 whose latest timeline file is newer than
hoodie.properties(TPC-H SF100ordersandregion) goes from 2 sequential requests (~96-140 ms) to 1 step with the GET and LIST in flight together (~50-55 ms), one round trip saved per table open and per metadata table open. When the timeline has no file newer than the properties (a table copied with uniform timestamps, or a properties rewrite after the last commit), the listing is not reused and the open lists again: one extra LIST, with the same latency as before. On a layout-1 table the early listing is one unused LIST.How are the changes test-covered
The
table::buildertests open tables through a test store that records when each GET and LIST starts and ends. They check that the timeline LIST overlaps the properties GET and is not repeated, and that the timeline is listed again for a layout-1 table, a failed listing, an overridden timeline path, a timeline no newer than the properties, and properties stamped at the Unix epoch. A replayed mid-upgrade open (timeline listed before the migration finished, upgraded properties read after) lists again with the store clock in step with the host and a day behind it. Each opened timeline is compared with one loaded without the early listing. The S3 latencies above come from request traces of table opens.