Skip to content

perf(core): list the active timeline concurrently with the properties read - #1011

Draft
yihua wants to merge 1 commit into
apache:mainfrom
yihua:perf/open-timeline-list-concurrent
Draft

yihua wants to merge 1 commit into
apache:mainfrom
yihua:perf/open-timeline-list-concurrent

Conversation

@yihua

@yihua yihua commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Description

closes #1008

Opening a table now lists the layout-2 active timeline directory (.hoodie/{hoodie.timeline.path}) concurrently with the hoodie.properties read. The timeline loader reuses that listing only when the resolved configs select the same directory on the same base URL, and the listing holds a file the store stamped strictly after hoodie.properties. A writer changes the timeline before it rewrites the properties (a layout upgrade migrates the timeline first), so only such a file shows the listing was taken after the properties that were read. Both times come from the store, so the reader's clock does not enter. Equal times, an empty or failed listing, or a properties time at the Unix epoch (a store that keeps no last-modified time) list the timeline again, and a properties failure is still returned as before.

Impact: opening a table on S3 whose latest timeline file is newer than hoodie.properties (TPC-H SF100 orders and region) goes from 2 sequential requests (~96-140 ms) to 1 step with the GET and LIST in flight together (~50-55 ms), one round trip saved per table open and per metadata table open. When the timeline has no file newer than the properties (a table copied with uniform timestamps, or a properties rewrite after the last commit), the listing is not reused and the open lists again: one extra LIST, with the same latency as before. On a layout-1 table the early listing is one unused LIST.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

The table::builder tests open tables through a test store that records when each GET and LIST starts and ends. They check that the timeline LIST overlaps the properties GET and is not repeated, and that the timeline is listed again for a layout-1 table, a failed listing, an overridden timeline path, a timeline no newer than the properties, and properties stamped at the Unix epoch. A replayed mid-upgrade open (timeline listed before the migration finished, upgraded properties read after) lists again with the store clock in step with the host and a day behind it. Each opened timeline is compared with one loaded without the early listing. The S3 latencies above come from request traces of table opens.

… read

Opening a table read .hoodie/hoodie.properties and only then listed the
active timeline directory, so every open paid two sequential round trips
before the timeline was known.

TableBuilder::build now lists the layout-2 active timeline directory
(.hoodie/{hoodie.timeline.path}) concurrently with the properties read.
The timeline loader reuses that listing only when the resolved configs
select layout 2 with the same directory on the same base URL, and only
when the listing holds a file the store stamped strictly after
hoodie.properties: a writer changes the timeline before it rewrites the
properties (a layout upgrade migrates the timeline first), so only such
a file shows the listing reflects the properties that were read. Both
times are the store's, so the reader's clock does not enter. Equal
times, an empty or failed listing, or a properties time at the Unix
epoch (a store that keeps no last-modified time) list the timeline
again. A properties failure is returned as before, never the listing's
error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Opening a table reads hoodie.properties and lists the active timeline one after the other

1 participant