Skip to content

Rank a shard replicas against locations already parsed - #503

Open
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/rank-replicas-against-parsed-locations
Open

Rank a shard replicas against locations already parsed#503
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/rank-replicas-against-parsed-locations

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Ordering a shard's replicas nearest-first measures each one's distance from
the caller, and measuring it parsed that replica's location. Per replica,
per shard, on the call every client and proxy makes when a table's topology
moves.

Those same locations are parsed once at the top of the function to size the
separation ladder. The note above the ranking already says the distance has
to be computed once rather than inside sort_by_key, because the key function
is called O(n log n) times -- and it is. The parse simply stayed where the
distance is computed.

Counted rather than timed, because the machine this was measured on is at
load 22 and the timings swing by more than the effect. For one request
against 64 servers and a table of 1024 shards with 3 replicas each:

Location::parse calls   3 137  ->  65

Sixty-five is the floor: one for each server, one for the caller. The other
3 072 were the same handful of locations parsed again for every shard.

A replica that is not one of those servers -- a route naming a server the
metaserver has not heard about -- is still parsed on the spot.

Four tests cover the ordering this feeds, including that callers equally
close keep the order the load scan chose. Answering a zero distance from the
index fails all four.

Ordering a shard's replicas nearest-first measures each one's distance from
the caller, and measuring it parsed that replica's location. Per replica,
per shard, on the call every client and proxy makes when a table's topology
moves.

Those same locations are parsed once at the top of the function to size the
separation ladder. The note above the ranking already says the distance has
to be computed once rather than inside sort_by_key, because the key function
is called O(n log n) times -- and it is. The parse simply stayed where the
distance is computed.

Counted rather than timed, because the machine this was measured on is at
load 22 and the timings swing by more than the effect. For one request
against 64 servers and a table of 1024 shards with 3 replicas each:

    Location::parse calls   3 137  ->  65

Sixty-five is the floor: one for each server, one for the caller. The other
3 072 were the same handful of locations parsed again for every shard.

A replica that is not one of those servers -- a route naming a server the
metaserver has not heard about -- is still parsed on the spot.

Four tests cover the ordering this feeds, including that callers equally
close keep the order the load scan chose. Answering a zero distance from the
index fails all four.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 31, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants