From 4dc6f340112946b8b15be3264b5bdacd378fc3c0 Mon Sep 17 00:00:00 2001 From: rwkeyes <86691029+rwkeyes@users.noreply.github.com> Date: Fri, 13 Nov 2026 12:18:46 -0500 Subject: [PATCH] docs: explain the 0.2.13 improvement instead of tabulating it The measured block listed numbers a reader had to assemble into a claim. It now says the claim first, then backs it: - a "short version" paragraph: first token 11-22 % sooner, three more answers per 45 s window at four streams, and nothing traded for it because the engine is at parity on all eight engine-level legs; - the table gains an absolute column (210 / 541 / 812 ms) next to the percentage, and a second column showing the same measurement read the other way ("stock needs 28.4 % longer"), with one sentence saying they are one number and which to quote; - a separate table for work done in the fixed window, so throughput is no longer a row crammed into the latency table; - "why the gap exists" now contrasts the two servers behaviour at a fresh prompt instead of naming the router; - "the same config also changes answers": the 24-token extraction probe (stock returns '', the fork returns 2024-06-09) and the green suite, which were only in the report before; - "what this does not claim" kept to three sentences, with the drift bound moved into the Measured-on block where the numbers are cited. The chart's numbers are unchanged. The same two-way framing is applied to the headline table in dirigo-results/0.2.13-port/README.md, so the front page and the evidence say the same thing in the same words. --- README.md | 64 +++++++++++++++++++--------- dirigo-results/0.2.13-port/README.md | 12 +++--- 2 files changed, 51 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 8a89d4a45fd..433664cb150 100644 --- a/README.md +++ b/README.md @@ -51,8 +51,11 @@ What upstream's releases actually changed, each measured against the release bef End to end on the card this branch was verified on (Arc Pro B70, Qwen3.8-Flash-Next Coder IQ1_M): **1,404 tok/s** prompt at 4K and 1,773 at 16K, **80.8 and 68.1 tok/s** decode, first token after 2.94 s and 9.27 s — -[the measurements](DIRIGO.md#speed-against-stock-measured-with-guidellm-the-standard-tool). The fork's own -changes are much smaller and server-side; they are in [What this fork changes](#what-this-fork-changes). +[the measurements](DIRIGO.md#speed-against-stock-measured-with-guidellm-the-standard-tool). + +The fork's own changes are smaller and server-side. Measured against stock v0.2.13 on the same card and the same +engine version, they make the first token arrive **11–22 % sooner** at one to four streams, and answer short requests +that stock truncates — the engine itself is at parity. [How much, and where it does not apply](#what-it-is-worth-measured-this-fork-vs-stock-strata-v0213). > **Never used Strata before?** Start with [Getting started](docs/GETTING-STARTED.md) — it walks from "what is > this" to your first answer, and explains every word Strata uses in [plain language](docs/GLOSSARY.md). @@ -207,31 +210,54 @@ change; with none of them enabled the server behaves as stock. ### What it is worth, measured: this fork vs stock Strata v0.2.13 -Same card, same engine version (0.1.59), same model and pack, same server config, same requests. GuideLLM 0.8.1, -512 tokens in / 128 out, one pass per arm. +Same card, same engine version (0.1.59), same model and pack, same server config, same request stream. GuideLLM +0.8.1, 512 tokens in / 128 out, one pass per arm. + +**The short version.** On a stream of fresh prompts the first token arrives **11–22 % sooner** — 739 ms against +stock's 949 ms at one stream — and at four streams the same 45-second window finishes **three more answers**. Nothing +is traded for it: prefill and decode are within 1.1 % of stock on all eight engine-level legs, because the engine +version is the same on both sides and the difference is entirely in the server.

Time to first token, Dirigo 0.2.13 vs stock 0.2.13 on an Intel Arc Pro B70: 739 vs 949 ms at one stream, 2622 vs 3163 at two, 6688 vs 7500 at four, 9641 vs 9659 at eight. Engine-level prefill and decode are within 1.1 percent on all eight legs.

-| | this fork | stock v0.2.13 | difference | -| --- | ---: | ---: | ---: | -| First token, 1 stream | **739 ms** | 949 ms | **−22.1 %** | -| First token, 2 streams | **2,622 ms** | 3,163 ms | **−17.1 %** | -| First token, 4 streams | **6,688 ms** | 7,500 ms | **−10.8 %** | -| First token, 8 streams | 9,641 ms | 9,659 ms | equal | -| Requests finished per 45 s window, 1–8 streams | 22 / 22 / 22 / 23 | 21 / 20 / 19 / 21 | +1 / +2 / +3 / +2 | -| Prefill and decode, direct to the engine, 8 legs | — | — | parity, largest gap 1.1 % | - -**The engine is the same speed; the server answers sooner.** On a stream of unique prompts the router serves with -the prompt cache off, so there is no pool work between the request and the first token; a conversation switches it -back on, which is what stock always does. That is also why 8 streams show no gap — queueing dominates there. +| streams | first token, this fork | stock v0.2.13 | arrives sooner by | read the other way: stock needs | +| --- | ---: | ---: | ---: | ---: | +| 1 | **739 ms** | 949 ms | **210 ms (22.1 %)** | 28.4 % longer | +| 2 | **2,622 ms** | 3,163 ms | **541 ms (17.1 %)** | 20.6 % longer | +| 4 | **6,688 ms** | 7,500 ms | **812 ms (10.8 %)** | 12.1 % longer | +| 8 | 9,641 ms | 9,659 ms | 18 ms — equal | equal | + +The last two columns are one measurement read two ways: 739 ms is 22.1 % below 949 ms, and 949 ms is 28.4 % above +739 ms. "Arrives sooner" is the column to quote. + +| work done in the same 45 s window | this fork | stock v0.2.13 | +| --- | ---: | ---: | +| requests finished, 1 / 2 / 4 / 8 streams | 22 / 22 / 22 / 23 | 21 / 20 / 19 / 21 | +| output tokens per second in that window, 1 / 2 / 4 / 8 | 62.6 / 62.6 / 62.6 / 65.4 | 59.7 / 56.9 / 54.0 / 59.7 | + +**Why the gap exists.** A fresh prompt is the one case the two servers handle differently. Stock opens the prompt +cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique prompts as +one-shot traffic and serves it with the cache **off** — no pool work between the request and the first token — then +switches the cache back on the moment the traffic looks like a conversation. That is why the advantage is largest +with requests arriving one at a time, and gone at eight streams where both servers are queue-bound. + +**The same config also changes answers, not just timing.** Asked for a date inside a 24-token budget ("Invoice +INV-7 dated 9 June 2024 → ISO date"), stock spends all 24 tokens on reasoning and returns an empty string; the fork +returns `2024-06-09` with zero reasoning characters. And the port's own suite is green — 1,537 tests, 17 skipped — +where the stock tree has one failure (`test_hip_ordinal`, a test that reads the host's AMD topology; fixed upstream +in [#5602](https://github.com/Niko1221/Strata/pull/5602)). + +**What this does not claim.** A faster engine — it is the same speed, and this table would look identical for any +server-side change. A win on conversational traffic, where the fork turns the cache back on and behaves like stock. +A throughput difference at one stream: one extra request in a 45-second window is the measurement's own granularity +(4.8 %), and it is reported rather than counted as a win. > **Measured on:** Intel Arc Pro B70 32 GB, Ryzen 7 7700X, Qwen3.8-Flash-Next Coder IQ1_M, engine 0.1.59 on both > sides, 2026-11-09. One pass per arm in one session; positional drift in this rig is 2–4 %, so the 10.8–22.1 % -> first-token gaps are larger than the drift and the one-request throughput difference at 1 stream is not. -> Errors: 0 in both arms. Full report, raw JSON and the script that recomputes every figure: -> [`dirigo-results/0.2.13-port/`](dirigo-results/0.2.13-port/). +> first-token gaps sit outside it. Errors: 0 in both arms. Full report, raw JSON and the script that recomputes +> every figure: [`dirigo-results/0.2.13-port/`](dirigo-results/0.2.13-port/). The parts that belong upstream are open as separate branches listed in [`DIRIGO.md`](DIRIGO.md). diff --git a/dirigo-results/0.2.13-port/README.md b/dirigo-results/0.2.13-port/README.md index 91e48b31631..e0bd0ca74eb 100644 --- a/dirigo-results/0.2.13-port/README.md +++ b/dirigo-results/0.2.13-port/README.md @@ -13,12 +13,12 @@ Every figure in the chart is recomputed from the JSON files in this folder by `s ## Headline -| | Dirigo port | stock v0.2.13 | difference | -| --- | ---: | ---: | ---: | -| Time to first token, 1 stream (median) | **739 ms** | 949 ms | **−22.1 %** | -| Time to first token, 2 streams | **2,622 ms** | 3,163 ms | **−17.1 %** | -| Time to first token, 4 streams | **6,688 ms** | 7,500 ms | **−10.8 %** | -| Time to first token, 8 streams | 9,641 ms | 9,659 ms | −0.2 % (equal) | +| | Dirigo port | stock v0.2.13 | first token arrives sooner by | same measurement, other way round | +| --- | ---: | ---: | ---: | ---: | +| Time to first token, 1 stream (median) | **739 ms** | 949 ms | **210 ms (22.1 %)** | stock needs 28.4 % longer | +| Time to first token, 2 streams | **2,622 ms** | 3,163 ms | **541 ms (17.1 %)** | stock needs 20.6 % longer | +| Time to first token, 4 streams | **6,688 ms** | 7,500 ms | **812 ms (10.8 %)** | stock needs 12.1 % longer | +| Time to first token, 8 streams | 9,641 ms | 9,659 ms | 18 ms — equal | equal | | Requests finished in a fixed 45 s window, 1 / 2 / 4 / 8 streams | 22 / 22 / 22 / 23 | 21 / 20 / 19 / 21 | +1 / +2 / +3 / +2 | | Prefill and decode, direct to the engine, 8 legs | — | — | **parity, largest gap 1.1 %** | | Unit suite | **1,537 OK** | 1,512, 1 failure | — |