Skip to content

Listing shards names the tables on the page, not in the cluster - #512

Open
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/list-shards-names-only-the-page
Open

Listing shards names the tables on the page, not in the cluster#512
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/list-shards-names-only-the-page

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Listing shards is paginated: it hands back at most a page and says where to
resume. To put a namespace and a table name against each shard on that page,
it first built a map of every shard in the cluster to its owning table --
walking every table, over every shard that table declares, cloning the
namespace and the name for each -- and then used at most one page of it.

So the cost of asking for a hundred shards was set by how many shards exist
altogether, and a caller paging through them paid it again on every page.

Choose the page first, then name only what is on it. A table's shards are a
contiguous range and the page is sorted, so the entries of the page that
belong to a table are a slice of it, found by two binary searches.

tables   declared shards     before      after
   500             2 000     378 us     244 us
 2 000             8 000   1 449 us     570 us
 8 000            32 000   6 892 us   1 002 us

Nearly seven times faster at the largest, measured before and after back to
back, and what is left grows with the tables rather than with every shard
they declare.

The rule that the first table in map order wins a shard two tables both
claim is kept: the map was built shard by shard in that order and the entry
is only filled if it is empty. Pointing the slice at the wrong end so no
page entry is ever named fails the test that a listed shard names the table
claiming it.

Listing shards is paginated: it hands back at most a page and says where to
resume. To put a namespace and a table name against each shard on that page,
it first built a map of every shard in the cluster to its owning table --
walking every table, over every shard that table declares, cloning the
namespace and the name for each -- and then used at most one page of it.

So the cost of asking for a hundred shards was set by how many shards exist
altogether, and a caller paging through them paid it again on every page.

Choose the page first, then name only what is on it. A table's shards are a
contiguous range and the page is sorted, so the entries of the page that
belong to a table are a slice of it, found by two binary searches.

    tables   declared shards     before      after
       500             2 000     378 us     244 us
     2 000             8 000   1 449 us     570 us
     8 000            32 000   6 892 us   1 002 us

Nearly seven times faster at the largest, measured before and after back to
back, and what is left grows with the tables rather than with every shard
they declare.

The rule that the first table in map order wins a shard two tables both
claim is kept: the map was built shard by shard in that order and the entry
is only filled if it is empty. Pointing the slice at the wrong end so no
page entry is ever named fails the test that a listed shard names the table
claiming it.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 31, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants