Skip to content

Find a shard table by where tables start - #514

Open
bjmeetsfo wants to merge 2 commits into
mainfrom
oss/find-a-shards-table-by-where-tables-start
Open

Find a shard table by where tables start#514
bjmeetsfo wants to merge 2 commits into
mainfrom
oss/find-a-shards-table-by-where-tables-start

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

A datanode finishing a load asks the metaserver to record it, and the
metaserver first checks the shard's table is not dropped or frozen. Finding
that table walked every table in the cluster -- under the write lock, so
every other call waited behind a walk that got longer as tables were added.

tables    before     after
   500    2.6 us    0.7 us
 2 000    7.3 us    1.1 us
 8 000   38.3 us    1.2 us

Thirty times faster at eight thousand tables, measured before and after back
to back, and it stops growing with the cluster: the answer is a lookup
rather than a search.

A table's shards start at first_shard_id, which is pinned when the table is
created, so where each table starts is something that can be recorded. A
shard belongs to the nearest table starting at or below it, if that table's
range reaches it: nothing else can own it, because every other table starts
above the shard or ends before the nearest one begins.

Two tables may claim overlapping shards, and the walk answered those with
the first in table map order, which is not the nearest start. So an overlap
is noticed when it appears and the walk decides from then on.

The index is derived state, and derived state drifts. It is maintained in
the four places a range can come about: creating a table, growing its shard
count, forgetting it, and restoring a state that arrived whole. A test walks
every shard id in range after each of those and checks the lookup and the
walk agree.

That test earned its place twice. It found that the overlap check only
looked at the nearest table above rather than every table starting inside
the range, so growing one table across another was not noticed. And with the
check removed from the update path entirely it fails, which the first
version of the test did not -- it grew a range that reached nothing.

A datanode finishing a load asks the metaserver to record it, and the
metaserver first checks the shard's table is not dropped or frozen. Finding
that table walked every table in the cluster -- under the write lock, so
every other call waited behind a walk that got longer as tables were added.

    tables    before     after
       500    2.6 us    0.7 us
     2 000    7.3 us    1.1 us
     8 000   38.3 us    1.2 us

Thirty times faster at eight thousand tables, measured before and after back
to back, and it stops growing with the cluster: the answer is a lookup
rather than a search.

A table's shards start at first_shard_id, which is pinned when the table is
created, so where each table starts is something that can be recorded. A
shard belongs to the nearest table starting at or below it, if that table's
range reaches it: nothing else can own it, because every other table starts
above the shard or ends before the nearest one begins.

Two tables may claim overlapping shards, and the walk answered those with
the first in table map order, which is not the nearest start. So an overlap
is noticed when it appears and the walk decides from then on.

The index is derived state, and derived state drifts. It is maintained in
the four places a range can come about: creating a table, growing its shard
count, forgetting it, and restoring a state that arrived whole. A test walks
every shard id in range after each of those and checks the lookup and the
walk agree.

That test earned its place twice. It found that the overlap check only
looked at the nearest table above rather than every table starting inside
the range, so growing one table across another was not noticed. And with the
check removed from the update path entirely it fails, which the first
version of the test did not -- it grew a range that reached nothing.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 31, 2026 07:35
…table-by-where-tables-start

# Conflicts:
#	crates/temporalstore-rust/src/meta.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants