Skip to content

gui: refute implausible decode windows in the live output rate - #9

Merged
zzstar101 merged 1 commit into
mainfrom
fix/live-tps-flushed-window
Nov 16, 2026
Merged

zzstar101 merged 1 commit into
mainfrom
fix/live-tps-flushed-window

Conversation

@zzstar101

Copy link
Copy Markdown
Collaborator

What

The live card's output speed reads impossible numbers for providers that hand a finished answer over in one flush — Gemini through an Anthropic-messages relay is the worst case, occasionally showing millions of tok/s. This refutes those windows instead of believing them.

Why

outputRate trusts DecodeMS whenever it is greater than zero, and DecodeMS is the span between the first streamed chunk and the last one as they arrived. That span only measures generation while the chunks arrive as the model writes them. A provider that buffers the answer makes the span the flush latency instead, so the numerator is the whole answer's output tokens and the denominator is a few milliseconds:

  • DSH logs 10,713 tokens "decoded" in 4 ms for a Gemini tool call ⇒ 2,678,250 tok/s on the card.
  • In the same session, all 306 tool-call-chunks records carry len(dt) == 0 with exactly one args member: the whole argument payload arrives as a single non-incremental delta.
  • The one text-chunks record there has 212 timed deltas and reads a correct 93 tok/s, so the pathology follows the block type, not the model or the token count.
  • Corpus-wide, 26,054 logged requests read above 300 tok/s; the worst belongs to claude-opus-5-5 at 4,358,000 tok/s (4,358 tokens / 1 ms).

DSH's own pill derives from the same journal (Σ decodeTokens / Σ decodeMs, cumulative over the session), which is why it climbs past 400 tok/s and keeps rising while the session runs. That is upstream of this repo; this change makes MyToken stop inheriting the artifact.

How

A measured window is refuted when it would have to run faster than plausibleRate (1000 tok/s), and that request then falls through to the existing bounded gap estimate instead of being believed. A refuted window is not a measurement, so it does not overrule anything.

The bound is deliberately loose, and is not the estimate's fast (300) ceiling. Over the whole index, cross-checked against "the window covers at least half the gap before it": the fastest corroborated measurement is 514 tok/s (mean 191), while the 1,000–5,000 and >5,000 bands contain zero corroborated samples across 4,071 requests. So 1000 keeps genuine fast measurements (a 500 tok/s window stays 500) and refutes every flushed one.

  • plausibleRate + measuredWindow in internal/gui/load.go.
  • outputRate now also reports whether the number is a measurement, and the card labels it tok/s instead of tok/s (est.) when it is (internal/gui/live.go, new tpsMeasured string in both languages).
  • No schema change, no ParserVersion bump, estimate path untouched (still bounded by fast).

Verification

Effect over the real index, replaying outputRate at 1 Hz across the six Gemini sessions:

before after
worst reading 2,678,250 tok/s 958 tok/s
p50 / p90 / p99 1,049 / 48,583 / — 275 / 453 / 875
readings above 1,000 tok/s 35% above 2k, 18.5% above 10k 0%
deepseek-flash (streams normally, control) p50 ≈ 259 p50 259, max 563 (real window kept)
gofmt -l .                  # empty, bar an ignored scratch file under build/
go vet ./...                # ok
GOOS=windows go vet ./...   # ok
go test ./...               # ok

New tests: TestOutputRateRefutesFlushedWindow, TestOutputRateRefutedWindowDoesNotPoisonMeasurement, TestOutputRateKeepsFastPlausibleMeasurement (a real 500 tok/s window must survive), TestOutputRateReportsProvenance, TestMeasuredWindowBoundary.

Follow-up (not in this PR)

Recording how many timed deltas back a window (DecodeSamples) would catch the case this heuristic cannot see: a flush that happens to land under the bound. That needs a parser version bump and a migration.

The live card's speed divides a request's output tokens by the window the
log timed for it (DecodeMS), and that window is the span between the first
and the last chunk of the answer as they arrived. It only measures
generation while the chunks arrive as the model writes them: a provider
that buffers the answer makes the span the flush latency, so 10,713 tokens
"decoded" in 4 ms read as 2.7M tok/s. Gemini through an Anthropic-messages
relay is the worst case — all 306 of one session's tool-call-chunks records
carry no timed deltas at all — and the same artifact shows up across the
corpus (26,054 requests above 300 tok/s, worst is a 4,358-token answer
logged at 1 ms).

A window implying more than plausibleRate (1000 tok/s) never watched the
model write, so it is refuted rather than believed: the request falls
through to the gap estimate, which is bounded by fast (300). The bound is
loose on purpose and is not fast itself — corpus-wide the fastest window
corroborated by the gap before it is 514 tok/s, while every window in the
thousands is corroborated by nothing, so a genuine 500 tok/s measurement
survives.

outputRate now also reports whether the number is a measurement, and the
card drops the "(est.)" label when it is.
@zzstar101
zzstar101 merged commit 235e38b into main Nov 16, 2026
6 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant