Skip to content

The divergence check takes from a server what it reads - #465

Open
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/the-divergence-check-takes-what-it-reads
Open

The divergence check takes from a server what it reads#465
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/the-divergence-check-takes-what-it-reads

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Each reconciliation round copied every server out of the metadata state so
that the comparison could run without the lock. Copying without the lock is
right; copying the whole record is not.

A server carries its shard-load list and its shard-state list, and a shard
state carries three strings -- the serving state, the table name and the
shard URI. The check reads none of them. It reduces each state to a
ShardHealth, which is Copy and holds a &'static str, and keeps the shard id.
So a datanode holding two thousand shards had six thousand strings copied
every round to answer a question about shard ids and verdicts.

Reduce while the lock is held, and copy that.

servers x shards/server     before      after
          8 x    500      1 795 us     441 us
         16 x  1 000      8 038 us   3 028 us
         32 x  2 000     32 965 us  10 534 us

Three times faster, measured before and after back to back on the same box.

The healthiest-verdict rule for a shard two workers disagree about now
applies where the two reports are still both visible, when the server is
observed. Its test builds the full record with both reports and observes it,
so what it exercises is unchanged.

Each reconciliation round copied every server out of the metadata state so
that the comparison could run without the lock. Copying without the lock is
right; copying the whole record is not.

A server carries its shard-load list and its shard-state list, and a shard
state carries three strings -- the serving state, the table name and the
shard URI. The check reads none of them. It reduces each state to a
ShardHealth, which is Copy and holds a &'static str, and keeps the shard id.
So a datanode holding two thousand shards had six thousand strings copied
every round to answer a question about shard ids and verdicts.

Reduce while the lock is held, and copy that.

    servers x shards/server     before      after
              8 x    500      1 795 us     441 us
             16 x  1 000      8 038 us   3 028 us
             32 x  2 000     32 965 us  10 534 us

Three times faster, measured before and after back to back on the same box.

The healthiest-verdict rule for a shard two workers disagree about now
applies where the two reports are still both visible, when the server is
observed. Its test builds the full record with both reports and observes it,
so what it exercises is unchanged.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 30, 2026 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants