Skip to content

feat(service): report hedge delays in place of raw phase latency - #206

Merged
shikhar merged 6 commits into
mainfrom
prune-latency-metrics
Oct 29, 2026
Merged

shikhar merged 6 commits into
mainfrom
prune-latency-metrics

Conversation

@shikhar

@shikhar shikhar commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

cachey_bucket_latency_* reports every latency model (first_byte, body, request) at three quantiles: up to 48 series per bucket per instance. Dashboards read only the request model's median and p99. The first-byte and body models matter for what they set, the early hedge delay, but that delay is the smallest of p99, 1.25 × p95, and 5 × the median, so it could not be read off any one series.

Change

  • Bucket latency, request model only: cachey_bucket_latency_seconds, _model_bytes, _predictions_total, and _predictions_exceeded_total now report only the request model, and latency only at the median and p99. Metric names are unchanged and phase="request" stays, so existing queries keep matching.
  • New cachey_bucket_hedge_delay_seconds{bucket, phase, bytes}: the delay before an early hedge fires, at the reported read sizes up to the largest the model retains. phase is first_byte (timed by requested bytes) or body (timed by body bytes), matching cachey_hedges_total. It is the value the downloader uses.
  • Series count: up to 20 per bucket, down from 48: 12 for the request model and 8 for hedge delays.
  • BucketMetrics: carries hedge delays instead of the first-byte and body estimates.
  • cachey_fetch_request_entries: buckets at 1, 2, and 4 instead of 1 to 32. That's 6 series per kind instead of 9.
  • cachey_fetch_request_bytes: buckets from 4 KiB to 16 MiB, four times apart, as the latency model spaces its bands. That's 10 series per kind instead of 9. The lowest bound was 512 KiB, so every small exact-range read fell in the first bucket.
  • Test removed: the integration test that only checked a ranged GET counts status="success". Other integration tests cover ranged reads.

docs/replica-reads.md documents the new gauge and the request-only latency series.

Validation

  • 204 tests, cargo +nightly fmt, and Clippy with -D warnings pass.
  • A throwaway check, not committed, fit a model to 4 KiB, 64 KiB, and 1 MiB reads. Its reported delays matched hedge_delay at those sizes, and 16 MiB was omitted rather than extrapolated.

🤖 Generated with Claude Code

shikhar and others added 5 commits October 29, 2026 11:29
`cachey_bucket_latency_*` reported three phases (`first_byte`, `body`,
`request`) at three quantiles, up to 48 series per bucket per instance.
Only the request model's median and p99 are read in practice.

Bucket latency metrics now report only the request model, at median and
p99: up to 12 series per bucket. Metric names are unchanged and
`phase="request"` stays, so existing queries keep matching.
`BucketMetrics` drops the first-byte and body estimates, which nothing
else read, so scrapes no longer compute them. Hedging still uses those
models internally.

`cachey_fetch_request_entries` buckets shrink from 1..32 to 1, 2, 4,
which still separate single entries from reads that cross a page
boundary and from small multi-page reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes the test added for bucket latency series, and the integration
test that only checked a ranged GET counts `status="success"`; other
tests cover ranged reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cachey_fetch_request_bytes` bucketed from 512 KiB to 16 MiB in six
steps, so every small exact-range read fell in the first bucket. Three
bounds 16 times apart separate small reads from page-sized ones with 6
series per kind instead of 9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bounds 16 times apart were too coarse. `cachey_fetch_request_bytes` now
buckets from 16 KiB to 16 MiB in steps of four, as the latency model
spaces its bands: 9 series per kind, as before 64 KiB to 16 MiB, but
starting low enough to separate small exact-range reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cachey_fetch_request_bytes` now buckets from 4 KiB to 16 MiB in steps
of four, so reads below 16 KiB are told apart. The lowest bound matches
the smallest size bucket latency is reported at.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

This PR reduces exported bucket latency series and changes fetch-size histogram bounds.

  • Bucket latency metrics show request times at the median and p99.
  • Fetch histograms group bytes and entries into new buckets.
  • The ranged-GET metrics check is no longer in the integration suite.

Acknowledgment: shikhar explicitly described the test removal and cited other ranged-read integration tests as the reason.

Reviews (1) · Last reviewed commit: "feat(service): bucket fetch sizes from 4..." · Reviewed by Greptile

Early hedges wait for a delay from the first-byte model before response
headers, and the body model after them. Dashboards graph hedges by
phase but could not see what triggers them.

`cachey_bucket_hedge_delay_seconds{bucket, phase, bytes}` reports that
delay at the reported read sizes, up to the largest the model retains,
with `phase` values matching `cachey_hedges_total`. It is the value the
downloader uses, so no dashboard has to rebuild the caps from quantiles:
8 series per bucket instead of 24 for the raw quantiles of both models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shikhar shikhar changed the title feat(service): trim bucket latency and fetch size series feat(service): report hedge delays in place of raw phase latency Oct 29, 2026
@shikhar
shikhar merged commit dcaff20 into main Oct 29, 2026
7 checks passed
@shikhar
shikhar deleted the prune-latency-metrics branch October 29, 2026 19:35
@github-actions github-actions Bot mentioned this pull request Oct 29, 2026
shikhar pushed a commit that referenced this pull request Oct 29, 2026
## 🤖 New release

* `cachey`: 0.13.1 -> 0.13.2

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

##
[0.13.2](0.13.1...0.13.2)
- 2026-10-29

### Added

- *(service)* resolve millisecond latencies in download and first chunk
histograms ([#207](#207))
- *(service)* report hedge delays in place of raw phase latency
([#206](#206))

### Fixed

- *(object_store)* probe a stale first bucket before it has a median
([#204](#204))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant