Skip to content

Say how long a topology answer took - #507

Open
bjmeetsfo wants to merge 2 commits into
mainfrom
oss/say-how-long-a-topology-answer-took
Open

Say how long a topology answer took#507
bjmeetsfo wants to merge 2 commits into
mainfrom
oss/say-how-long-a-topology-answer-took

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Every background subsystem here reports what it did and how much of it, and
none of them report how long anything took. There is no latency measurement
anywhere in the metaserver.

That matters most for the topology query, because it is the call every
client and every proxy makes and the only one whose slowness a caller feels
directly. A metaserver that has become slow to answer -- a datanode holding
the write lock through a long heartbeat, a table that has grown, a loaded
box -- looks exactly like one that is fast, right up until clients start
timing out, and then nothing says which of those it was.

temporalstore_meta_topology_query_latency_us is a histogram over the whole
answer, in microseconds, with a sum and a count.

Timed around the lock as well as the work. A client waiting on a busy
metaserver waits for the lock as surely as for the placement, and timing
only the placement would report the metaserver as fast while every caller
experienced it as slow.

A query for a table that is not there is an answer too, and is timed.

Buckets are recorded per bucket and added up when rendered, because
cumulative is the shape Prometheus wants and per-bucket is the shape that is
cheap to record on the hot path.

Every background subsystem here reports what it did and how much of it, and
none of them report how long anything took. There is no latency measurement
anywhere in the metaserver.

That matters most for the topology query, because it is the call every
client and every proxy makes and the only one whose slowness a caller feels
directly. A metaserver that has become slow to answer -- a datanode holding
the write lock through a long heartbeat, a table that has grown, a loaded
box -- looks exactly like one that is fast, right up until clients start
timing out, and then nothing says which of those it was.

temporalstore_meta_topology_query_latency_us is a histogram over the whole
answer, in microseconds, with a sum and a count.

Timed around the lock as well as the work. A client waiting on a busy
metaserver waits for the lock as surely as for the placement, and timing
only the placement would report the metaserver as fast while every caller
experienced it as slow.

A query for a table that is not there is an answer too, and is timed.

Buckets are recorded per bucket and added up when rendered, because
cumulative is the shape Prometheus wants and per-bucket is the shape that is
cheap to record on the hot path.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 31, 2026 05:17
# Conflicts:
#	crates/temporalstore-rust/src/meta.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants