Repository navigation
query: keep unreported cache out of the hit-rate denominator - #6
xiaomai1011 wants to merge 1 commit into
Conversation
A route that never logs a cache field is silent about the cache, not proof of a miss, but the hit rate read its prompt tokens as a miss anyway. Anyone behind a relay that drops usage.prompt_tokens_details ends up with a hit rate far below what the same library reports on direct routes: on a real library the month read 88.41%, while the routes that do report cache sat above 97%. Tokens.CacheMissing now carries the prompt tokens of a route that never reported any cache field, keyed by the whole routing identity (harness, model, resolved provider, base URL) so one model name served both by a silent relay and by its own provider is not lumped together. Tokens.HitRate subtracts it from the denominator, and Totals.CacheHit now calls HitRate, so every surface shows the same number. CacheMissing stays a subset of Input: Total does not change, and the GUI keeps showing the silent tokens, it just no longer calls them misses.
|
Closing this: my premise was wrong. I verified afterwards that the usage object on the affected route does carry That means subtracting those tokens from the denominator would have hidden 75% of the input and inflated the headline from 88.4% to 96.8% — a prettier number, not a truer one. Issue #5 has been rewritten with the real numbers and a presentation-only suggestion (per-provider hit rate, or |
Implements option A from #5.
What
Tokens.HitRate()dividedcache_readbyinput + cache_read + cache_write, so prompt tokens from a route that never reported a cache field were counted as misses. This change marks those tokens and keeps them out of the denominator.model.TokensgainsCacheMissing(JSONcacheMissing,omitempty). It is a subset ofInput, never added on top:Total()is unchanged, per-request and cost figures are unchanged.query.annotateCacheMissingruns once over the loaded events and setsCacheMissing = Inputfor routes that never loggedCacheRead,CacheWriteorCacheWrite1h. The route key is the whole routing identity — harness, model, resolved provider, base URL — not just the model name, because one model name can be served both by a relay that drops the fields and by the provider itself, which reports them.Tokens.HitRate()subtractsCacheMissingfrom the denominator, andTotals.CacheHitnow callsHitRate()instead of recomputing the formula inline, so every surface (GUI, CLI, statusline) shows the same number by construction.Why
Relayed traffic (local OpenAI-compatible relay in front of the provider) never gets a truthful hit rate today: the relay drops
usage.prompt_tokens_details, the events storecache_read = 0, and the headline percentage collapses. From the library in #5: the month read 88.41 % while the routes that do report cache sat above 97 %; excluding the silent input it reads 96.84 %. The silent input was 75.1 % of the month's input.Nothing about "the cache missed" versus "this route never told us" was representable before, which is why the fix is a token bucket rather than a formatting change.
Behaviour change
The headline percentage moves for anyone whose events include a silent route (88.41 % → 96.84 % on the library above).
Total,input,output,cache read,cache writeand cost are untouched, so nothing downstream of those numbers changes.If you would rather keep the denominator and only suppress the misleading
0 %(option B), or show both numbers (option C), say so and I will rework this patch — it is a small, self-contained change either way.Tests
TestHitRateIgnoresRoutesThatNeverReportCache: a reporting route (100 in / 300 read) plus a silent relay (400 + 200 in, twice) must keepInput == 700,Total() == 1000,CacheMissing == 600,CacheHit == 0.75, andTotals.CacheHitmust equalTokens.HitRate().internal/query/service_test.go,internal/query/provider_ids_test.go(both thealt pathandunioncases),internal/cli/commands_test.go(summary textcache hit 61.2%→68.2%, daily/summary token literals).gofmtclean on the touched files,go vet ./...clean,go test ./internal/model/... ./internal/query/... ./internal/cli/...green apart from the two Windows-only failures below.Two failures that are not from this change
Both reproduce on an untouched checkout of
mainon Windows:internal/app TestSourceKeyLivesOutsideTheIndex—key file mode = 666, want 600; Windows has no POSIX 0600.internal/cli TestScanIndexesSyntheticClaudeLog— the test's fake home does not stop the scanner from picking up the machine's real harness roots (scan covered 32 sources).CI on a clean runner should be unaffected by both.