Repository navigation
[Docs] Explain the 0.2.13 improvement instead of tabulating it - #5
Merged
Merged
Conversation
The measured block listed numbers a reader had to assemble into a claim.
It now says the claim first, then backs it:
- a "short version" paragraph: first token 11-22 % sooner, three more
answers per 45 s window at four streams, and nothing traded for it
because the engine is at parity on all eight engine-level legs;
- the table gains an absolute column (210 / 541 / 812 ms) next to the
percentage, and a second column showing the same measurement read the
other way ("stock needs 28.4 % longer"), with one sentence saying they
are one number and which to quote;
- a separate table for work done in the fixed window, so throughput is no
longer a row crammed into the latency table;
- "why the gap exists" now contrasts the two servers behaviour at a fresh
prompt instead of naming the router;
- "the same config also changes answers": the 24-token extraction probe
(stock returns '', the fork returns 2024-06-09) and the green suite,
which were only in the report before;
- "what this does not claim" kept to three sentences, with the drift bound
moved into the Measured-on block where the numbers are cited.
The chart's numbers are unchanged. The same two-way framing is applied to
the headline table in dirigo-results/0.2.13-port/README.md, so the front
page and the evidence say the same thing in the same words.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result
The measured block on the front page tabulated the comparison but did not explain it: a reader got four percentages
and a parity row and had to assemble the claim themselves, while the strongest parts of the measurement (the 210 ms
absolute gap, the extra answers per window, the 24-token extraction where stock returns nothing) were either a single
crammed row or absent from the page entirely.
Before: "First token, 1 stream — 739 ms — 949 ms — −22.1 %", with the mechanism in one dense sentence and
four caveats in the surrounding paragraph. After: the claim first, the evidence under it, the two framings of the
same number shown once and explained, and a short "what this does not claim".
Related discussion
none — documentation only.
Approach
streams finish three more answers per 45 s window, and that nothing is traded for it because the engine is at
parity on every leg.
fifth column giving the same measurement the other way round ("stock needs 28.4 % longer"). One sentence says they
are one number and which to quote, so nobody can accuse the page of picking whichever direction flatters it.
with the granularity stated, instead of a row that mixed requests and percentages.
opens the cache and pays the pool work; the fork recognises one-shot traffic, serves with the cache off, switches
back on for a conversation), which also explains why the gap is zero at eight streams.
'', the fork returns2024-06-09with zeroreasoning characters — and the green suite (1,537 tests against stock's 1,512 with one failure) are now on the
page, cited to the report.
the Measured-on block, next to the hardware and the date.
Validation
Passed:
the absolute gaps were computed from those medians (949−739=210, 3163−2622=541, 7500−6688=812); the reciprocal
percentages (28.4 / 20.6 / 12.1 %) are 100·(stock/fork − 1) of the same numbers, printed here to one decimal.
guidellm/*.jsonand the probe output indirigo-results/0.2.13-port/README.md.have consistent column counts (4 and 3 data columns).
parsing as tables.
Failed:
none observed.
Not run:
Limits
request count is reported but not claimed.