Skip to content

query: keep unreported cache out of the hit-rate denominator - #6

Closed
xiaomai1011 wants to merge 1 commit into
nerv-base:mainfrom
xiaomai1011:fix/cache-hit-ignores-unreported-cache
Closed

xiaomai1011 wants to merge 1 commit into
nerv-base:mainfrom
xiaomai1011:fix/cache-hit-ignores-unreported-cache

Conversation

@xiaomai1011

Copy link
Copy Markdown
Contributor

Implements option A from #5.

What

Tokens.HitRate() divided cache_read by input + cache_read + cache_write, so prompt tokens from a route that never reported a cache field were counted as misses. This change marks those tokens and keeps them out of the denominator.

  • model.Tokens gains CacheMissing (JSON cacheMissing,omitempty). It is a subset of Input, never added on top: Total() is unchanged, per-request and cost figures are unchanged.
  • query.annotateCacheMissing runs once over the loaded events and sets CacheMissing = Input for routes that never logged CacheRead, CacheWrite or CacheWrite1h. The route key is the whole routing identity — harness, model, resolved provider, base URL — not just the model name, because one model name can be served both by a relay that drops the fields and by the provider itself, which reports them.
  • Tokens.HitRate() subtracts CacheMissing from the denominator, and Totals.CacheHit now calls HitRate() instead of recomputing the formula inline, so every surface (GUI, CLI, statusline) shows the same number by construction.

Why

Relayed traffic (local OpenAI-compatible relay in front of the provider) never gets a truthful hit rate today: the relay drops usage.prompt_tokens_details, the events store cache_read = 0, and the headline percentage collapses. From the library in #5: the month read 88.41 % while the routes that do report cache sat above 97 %; excluding the silent input it reads 96.84 %. The silent input was 75.1 % of the month's input.

Nothing about "the cache missed" versus "this route never told us" was representable before, which is why the fix is a token bucket rather than a formatting change.

Behaviour change

The headline percentage moves for anyone whose events include a silent route (88.41 % → 96.84 % on the library above). Total, input, output, cache read, cache write and cost are untouched, so nothing downstream of those numbers changes.

If you would rather keep the denominator and only suppress the misleading 0 % (option B), or show both numbers (option C), say so and I will rework this patch — it is a small, self-contained change either way.

Tests

  • New TestHitRateIgnoresRoutesThatNeverReportCache: a reporting route (100 in / 300 read) plus a silent relay (400 + 200 in, twice) must keep Input == 700, Total() == 1000, CacheMissing == 600, CacheHit == 0.75, and Totals.CacheHit must equal Tokens.HitRate().
  • Updated fixtures that encode the old formula: internal/query/service_test.go, internal/query/provider_ids_test.go (both the alt path and union cases), internal/cli/commands_test.go (summary text cache hit 61.2% → 68.2%, daily/summary token literals).
  • gofmt clean on the touched files, go vet ./... clean, go test ./internal/model/... ./internal/query/... ./internal/cli/... green apart from the two Windows-only failures below.

Two failures that are not from this change

Both reproduce on an untouched checkout of main on Windows:

  • internal/app TestSourceKeyLivesOutsideTheIndex — key file mode = 666, want 600; Windows has no POSIX 0600.
  • internal/cli TestScanIndexesSyntheticClaudeLog — the test's fake home does not stop the scanner from picking up the machine's real harness roots (scan covered 32 sources).

CI on a clean runner should be unaffected by both.

A route that never logs a cache field is silent about the cache, not proof
of a miss, but the hit rate read its prompt tokens as a miss anyway. Anyone
behind a relay that drops usage.prompt_tokens_details ends up with a hit rate
far below what the same library reports on direct routes: on a real library
the month read 88.41%, while the routes that do report cache sat above 97%.

Tokens.CacheMissing now carries the prompt tokens of a route that never
reported any cache field, keyed by the whole routing identity (harness,
model, resolved provider, base URL) so one model name served both by a
silent relay and by its own provider is not lumped together.
Tokens.HitRate subtracts it from the denominator, and Totals.CacheHit now
calls HitRate, so every surface shows the same number. CacheMissing stays a
subset of Input: Total does not change, and the GUI keeps showing the silent
tokens, it just no longer calls them misses.
@xiaomai1011

Copy link
Copy Markdown
Contributor Author

Closing this: my premise was wrong.

I verified afterwards that the usage object on the affected route does carry cache_read_input_tokens / cache_creation_input_tokens — it is simply always 0. The relay forwards usage verbatim, nothing is dropped by it or by the parser. Sending the same 6,862-token prompt three times still returns cache_read_input_tokens: 0 every time, so the route genuinely has no prompt cache.

That means subtracting those tokens from the denominator would have hidden 75% of the input and inflated the headline from 88.4% to 96.8% — a prettier number, not a truer one.

Issue #5 has been rewritten with the real numbers and a presentation-only suggestion (per-provider hit rate, or n/a for routes that never cache). Thanks for the time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant