Counting one namespace's tables walked every table in the cluster - #523
Open
bjmeetsfo wants to merge 2 commits into
Open
Counting one namespace's tables walked every table in the cluster#523bjmeetsfo wants to merge 2 commits into
bjmeetsfo wants to merge 2 commits into
Conversation
Listing namespaces reports how many tables each one holds, and it counted them
by filtering the whole table set once per namespace. So the cost was the
namespaces multiplied by the tables, and a metrics scrape pays it on every
tick:
namespaces tables before after
32 1024 171.8us 39.8us
64 4096 2824.8us 384.1us
One pass over the tables tallies every namespace at once. The whole scrape at
the larger size goes from 3639us to 1329us.
A namespace holding no table is absent from the tally rather than present with
a zero, so it is read back through a lookup that falls back to zero. Dropped
tables are excluded when the tally is built, exactly where the filter excluded
them before.
Test: three namespaces holding different numbers of tables, one holding none,
and a dropped table -- with every count checked against counting that
namespace's tables one by one, which is what the listing used to do. It is
also checked against the four counts directly, so it cannot pass by agreeing
on zero everywhere.
Checked against a mutation: with the dropped-table exclusion removed the test
fails, naming the namespace and both counts.
Measured with the arms interleaved, on a shared box, and the smaller arm run
twice to bracket the noise.
…-tables-in-one-pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Listing namespaces reports how many tables each one holds, and it counted them by filtering the whole table set once per namespace. So the cost was the namespaces multiplied by the tables, and a metrics scrape pays it on every tick.
One pass over the tables tallies every namespace at once. The whole scrape at the larger size goes from 3639us to 1329us.
Why it was not caught before
An earlier look at scrape cost varied the number of shards and found it flat. It used one namespace and one table, so it could not have seen a cost that depends on those. The dimension has to be the one the cost is in.
Behaviour
A namespace holding no table is absent from the tally rather than present with a zero, so it is read back through a lookup that falls back to zero. Dropped tables are excluded when the tally is built, exactly where the filter excluded them before.
Testing
Three namespaces holding different numbers of tables, one holding none, and a dropped table. Every count is checked against counting that namespace's tables one by one -- which is what the listing used to do -- and then against the four expected counts directly, so it cannot pass by agreeing on zero everywhere.
Checked against a mutation: with the dropped-table exclusion removed the test fails, naming the namespace and both counts.
Measurements were taken with the arms interleaved, on a shared box, and the smaller arm was run twice to bracket the noise.
Suites: 327 metadata tests, 47 metaserver binary tests, 246 consensus tests, all green.