Listing shards names the tables on the page, not in the cluster - #512
Open
bjmeetsfo wants to merge 1 commit into
Open
Listing shards names the tables on the page, not in the cluster#512bjmeetsfo wants to merge 1 commit into
bjmeetsfo wants to merge 1 commit into
Conversation
Listing shards is paginated: it hands back at most a page and says where to
resume. To put a namespace and a table name against each shard on that page,
it first built a map of every shard in the cluster to its owning table --
walking every table, over every shard that table declares, cloning the
namespace and the name for each -- and then used at most one page of it.
So the cost of asking for a hundred shards was set by how many shards exist
altogether, and a caller paging through them paid it again on every page.
Choose the page first, then name only what is on it. A table's shards are a
contiguous range and the page is sorted, so the entries of the page that
belong to a table are a slice of it, found by two binary searches.
tables declared shards before after
500 2 000 378 us 244 us
2 000 8 000 1 449 us 570 us
8 000 32 000 6 892 us 1 002 us
Nearly seven times faster at the largest, measured before and after back to
back, and what is left grows with the tables rather than with every shard
they declare.
The rule that the first table in map order wins a shard two tables both
claim is kept: the map was built shard by shard in that order and the entry
is only filled if it is empty. Pointing the slice at the wrong end so no
page entry is ever named fails the test that a listed shard names the table
claiming it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Listing shards is paginated: it hands back at most a page and says where to
resume. To put a namespace and a table name against each shard on that page,
it first built a map of every shard in the cluster to its owning table --
walking every table, over every shard that table declares, cloning the
namespace and the name for each -- and then used at most one page of it.
So the cost of asking for a hundred shards was set by how many shards exist
altogether, and a caller paging through them paid it again on every page.
Choose the page first, then name only what is on it. A table's shards are a
contiguous range and the page is sorted, so the entries of the page that
belong to a table are a slice of it, found by two binary searches.
Nearly seven times faster at the largest, measured before and after back to
back, and what is left grows with the tables rather than with every shard
they declare.
The rule that the first table in map order wins a shard two tables both
claim is kept: the map was built shard by shard in that order and the entry
is only filled if it is empty. Pointing the slice at the wrong end so no
page entry is ever named fails the test that a listed shard names the table
claiming it.