Repository navigation
gui: refute implausible decode windows in the live output rate - #9
Merged
Merged
Conversation
The live card's speed divides a request's output tokens by the window the log timed for it (DecodeMS), and that window is the span between the first and the last chunk of the answer as they arrived. It only measures generation while the chunks arrive as the model writes them: a provider that buffers the answer makes the span the flush latency, so 10,713 tokens "decoded" in 4 ms read as 2.7M tok/s. Gemini through an Anthropic-messages relay is the worst case — all 306 of one session's tool-call-chunks records carry no timed deltas at all — and the same artifact shows up across the corpus (26,054 requests above 300 tok/s, worst is a 4,358-token answer logged at 1 ms). A window implying more than plausibleRate (1000 tok/s) never watched the model write, so it is refuted rather than believed: the request falls through to the gap estimate, which is bounded by fast (300). The bound is loose on purpose and is not fast itself — corpus-wide the fastest window corroborated by the gap before it is 514 tok/s, while every window in the thousands is corroborated by nothing, so a genuine 500 tok/s measurement survives. outputRate now also reports whether the number is a measurement, and the card drops the "(est.)" label when it is.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The live card's output speed reads impossible numbers for providers that hand a finished answer over in one flush — Gemini through an Anthropic-messages relay is the worst case, occasionally showing millions of tok/s. This refutes those windows instead of believing them.
Why
outputRatetrustsDecodeMSwhenever it is greater than zero, andDecodeMSis the span between the first streamed chunk and the last one as they arrived. That span only measures generation while the chunks arrive as the model writes them. A provider that buffers the answer makes the span the flush latency instead, so the numerator is the whole answer's output tokens and the denominator is a few milliseconds:tool-call-chunksrecords carrylen(dt) == 0with exactly oneargsmember: the whole argument payload arrives as a single non-incremental delta.text-chunksrecord there has 212 timed deltas and reads a correct 93 tok/s, so the pathology follows the block type, not the model or the token count.claude-opus-5-5at 4,358,000 tok/s (4,358 tokens / 1 ms).DSH's own pill derives from the same journal (
Σ decodeTokens / Σ decodeMs, cumulative over the session), which is why it climbs past 400 tok/s and keeps rising while the session runs. That is upstream of this repo; this change makes MyToken stop inheriting the artifact.How
A measured window is refuted when it would have to run faster than
plausibleRate(1000 tok/s), and that request then falls through to the existing bounded gap estimate instead of being believed. A refuted window is not a measurement, so it does not overrule anything.The bound is deliberately loose, and is not the estimate's
fast(300) ceiling. Over the whole index, cross-checked against "the window covers at least half the gap before it": the fastest corroborated measurement is 514 tok/s (mean 191), while the 1,000–5,000 and >5,000 bands contain zero corroborated samples across 4,071 requests. So 1000 keeps genuine fast measurements (a 500 tok/s window stays 500) and refutes every flushed one.plausibleRate+measuredWindowininternal/gui/load.go.outputRatenow also reports whether the number is a measurement, and the card labels ittok/sinstead oftok/s (est.)when it is (internal/gui/live.go, newtpsMeasuredstring in both languages).ParserVersionbump, estimate path untouched (still bounded byfast).Verification
Effect over the real index, replaying
outputRateat 1 Hz across the six Gemini sessions:deepseek-flash(streams normally, control)New tests:
TestOutputRateRefutesFlushedWindow,TestOutputRateRefutedWindowDoesNotPoisonMeasurement,TestOutputRateKeepsFastPlausibleMeasurement(a real 500 tok/s window must survive),TestOutputRateReportsProvenance,TestMeasuredWindowBoundary.Follow-up (not in this PR)
Recording how many timed deltas back a window (
DecodeSamples) would catch the case this heuristic cannot see: a flush that happens to land under the bound. That needs a parser version bump and a migration.