Say how long a topology answer took - #507
Open
bjmeetsfo wants to merge 2 commits into
Open
Conversation
Every background subsystem here reports what it did and how much of it, and none of them report how long anything took. There is no latency measurement anywhere in the metaserver. That matters most for the topology query, because it is the call every client and every proxy makes and the only one whose slowness a caller feels directly. A metaserver that has become slow to answer -- a datanode holding the write lock through a long heartbeat, a table that has grown, a loaded box -- looks exactly like one that is fast, right up until clients start timing out, and then nothing says which of those it was. temporalstore_meta_topology_query_latency_us is a histogram over the whole answer, in microseconds, with a sum and a count. Timed around the lock as well as the work. A client waiting on a busy metaserver waits for the lock as surely as for the placement, and timing only the placement would report the metaserver as fast while every caller experienced it as slow. A query for a table that is not there is an answer too, and is timed. Buckets are recorded per bucket and added up when rendered, because cumulative is the shape Prometheus wants and per-bucket is the shape that is cheap to record on the hot path.
# Conflicts: # crates/temporalstore-rust/src/meta.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every background subsystem here reports what it did and how much of it, and
none of them report how long anything took. There is no latency measurement
anywhere in the metaserver.
That matters most for the topology query, because it is the call every
client and every proxy makes and the only one whose slowness a caller feels
directly. A metaserver that has become slow to answer -- a datanode holding
the write lock through a long heartbeat, a table that has grown, a loaded
box -- looks exactly like one that is fast, right up until clients start
timing out, and then nothing says which of those it was.
temporalstore_meta_topology_query_latency_us is a histogram over the whole
answer, in microseconds, with a sum and a count.
Timed around the lock as well as the work. A client waiting on a busy
metaserver waits for the lock as surely as for the placement, and timing
only the placement would report the metaserver as fast while every caller
experienced it as slow.
A query for a table that is not there is an answer too, and is timed.
Buckets are recorded per bucket and added up when rendered, because
cumulative is the shape Prometheus wants and per-bucket is the shape that is
cheap to record on the hot path.