From 636832792e0154113610bbb3fcd6405089aabd3f Mon Sep 17 00:00:00 2001 From: rwkeyes <86691029+rwkeyes@users.noreply.github.com> Date: Fri, 13 Nov 2026 12:38:49 -0500 Subject: [PATCH] docs: rename "Why it is fast" to what it now has to prove, and prove it The section is the fork's case, so it should be named for it and built like an argument rather than a description. It is now "Why this fork is faster than stock Strata", and it demonstrates in this order: 1. what is NOT faster - both trees build engine 0.1.59 and the 8-leg engine battery lands within 1.1 %, so the reader knows the claim rests on the server before seeing a single gain; 2. the claim and its evidence - 739 vs 949 ms at one stream, the chart, the table with absolute and reciprocal columns, the work finished in the same window; 3. where it comes from - the two servers' behaviour at a fresh prompt; 4. a control - the same router measured shadow (934 ms, stock's number) against on (727 ms) on one build, so the switch rather than the build is shown to move the first token; 5. how to reproduce it - the harness, the per-level JSON, make-chart.py; 6. what it does not show - engine parity, conversations, 8 streams, the one-request granularity at 1 stream; 7. what both versions share underneath - one paragraph, since the engine decisions are stock's and belong to the shared explanation. The drafting table moved to "What this fork changes", where a list of differences belongs, under "the same drafter, more control"; the extraction and suite results moved with it as "two measured differences that are not speed". The old "Why it is fast" bullets are gone: the three engine decisions now appear once, as the closing paragraph of the new section. --- README.md | 121 +++++++++++++++++++++++++++--------------------------- 1 file changed, 60 insertions(+), 61 deletions(-) diff --git a/README.md b/README.md index 7a55e2b2177..3af69fba9dd 100644 --- a/README.md +++ b/README.md @@ -18,49 +18,18 @@ engine **0.1.58**) plus this fork's changes, in the state that served a live mac measurements behind it kept in [`dirigo-results/`](dirigo-results/). What was verified, and how: [`DIRIGO.md`](DIRIGO.md). -## Why it is fast +## Why this fork is faster than stock Strata -Stock Strata v0.2.13 and this fork run the **same engine version (0.1.59)**, so the engine's speed is upstream's — -Niko1221's — and identical on both sides. Three decisions make that engine fast: +"Stock" here is **Niko1221's v0.2.13** — the upstream release — built from the same tag and run on the same card, +model, pack, config and request stream as this fork. Everything below is measured on both sides, not estimated. -- **Only the experts a word needs are computed.** Qwen3.8-Flash-Next is 24,576 small specialists and a word - needs about ten. The busiest live on the card (each free GB holds roughly 700 more), every expert stays in RAM, and the - CPU computes the ones the card does not hold *at the same time* as the card works — neither waits. -- **Long prompts are read in big pieces** (up to 8,192 tokens), which is why a long document is read at over - 1,000 tokens a second, and a follow-up reuses the context already read instead of starting over. -- **Guess, then check.** The model's built-in draft layer guesses up to three tokens and the model verifies all - of them in one pass (2.4–3.2 tokens per pass), so an answer arrives 1.6–1.8× sooner with the same text. +**First, the part that is not faster.** Both trees build engine 0.1.59, and the engine-level battery — eight legs, +prefill and decode, straight to the engine, two interleaved passes — lands within **1.1 %** of each other. If the fork +had only the engine to show, this section would be empty. The difference is in the server: -**All three are stock behaviour.** The expert split, the prompt-chunk reads and guess-then-check (the MTP draft -layer) run the same on both sides. The comparison below starts both arms with `--spec 6 --spec-min-p 0.5` and neither -sends a per-request override, so drafting is not part of the gain. - -What the fork adds is **control over the drafter, not a different drafter**: - -| setting | stock v0.2.13 | this fork | -| --- | --- | --- | -| draft depth | `--spec` at startup, for every request | also `spec=N` per request, clamped by the engine to 2…the startup value | -| prompt lookup | `--suffix-draft` at startup | also `lookup=0\|1` per request | -| draft confidence | `spec_min_p` per request via `strata_tune` | the same, and changeable globally on a live server | -| window policy | — | `spec_window`: serve some traffic at a lower depth (0 in the measurement) | - -None of it was switched on for the numbers below: `spec_window` was 0, and GuideLLM sends no `strata_tune`. The -fork's measured difference is on the **server** side, which is the next section. - -On the card this branch was verified on (Arc Pro B70, Qwen3.8-Flash-Next Coder IQ1_M) that comes out at **1,404 tok/s** -reading a 4K prompt and 1,773 at 16K, **80.8 and 68.1 tok/s** writing, first token after 2.94 s and 9.27 s — -[those measurements](DIRIGO.md#speed-against-stock-measured-with-guidellm-the-standard-tool). How each step of the -engine's speed was won is the [history table in `docs/DETAILS.md`](docs/DETAILS.md); it is not repeated here. - -### Against stock v0.2.13, on the same card - -Same card, same engine version (0.1.59), same model and pack, same server config, same request stream. GuideLLM -0.8.1, 512 tokens in / 128 out, one pass per arm. - -**The short version.** The first token arrives **11–22 % sooner** on a stream of fresh prompts — 739 ms against -stock's 949 ms at one stream — and at four streams the same 45-second window finishes **three more answers**. Nothing -is traded for it: prefill and decode are within 1.1 % of stock on all eight engine-level legs, and the gain is a -server-side one — the engine measures the same on both sides. +**The first token arrives 11–22 % sooner** on a stream of fresh prompts — 739 ms against stock's 949 ms at one +stream — and at four streams the same 45-second window finishes **three more answers**. Nothing is traded for it: the +engine figures are parity, as above.

Time to first token, Dirigo 0.2.13 vs stock 0.2.13 on an Intel Arc Pro B70: 739 vs 949 ms at one stream, 2622 vs 3163 at two, 6688 vs 7500 at four, 9641 vs 9659 at eight. Engine-level prefill and decode are within 1.1 percent on all eight legs. @@ -77,22 +46,30 @@ The last two columns are one measurement read two ways: 739 ms is 22.1 % below 9 739 ms. "Arrives sooner" is the column to quote. In the same 45-second window the fork finishes 22 / 22 / 22 / 23 requests against 21 / 20 / 19 / 21 — 62.6 / 62.6 / 62.6 / 65.4 output tokens a second against 59.7 / 56.9 / 54.0 / 59.7. -**Why the gap exists.** A fresh prompt is the one case the two servers handle differently. Stock opens the prompt -cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique prompts as -one-shot traffic and serves it with the cache **off** — no pool work between the request and the first token — then -switches the cache back on the moment the traffic looks like a conversation. That is why the advantage is largest -with requests arriving one at a time, and gone at eight streams where both servers are queue-bound. +**Where the difference comes from.** A fresh prompt is the one case the two servers handle differently. Stock opens +the prompt cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique +prompts as one-shot traffic and serves it with the cache **off** — no pool work between the request and the first +token — then switches the cache back on the moment the traffic looks like a conversation. That is why the advantage is +largest with requests arriving one at a time, and gone at eight streams where both servers are queue-bound. + +**A control, so the mechanism is not taken on faith.** The same router was measured in both positions on one build: +with `conv_route: shadow` — classify and log, never switch — it lands on the stock number, 934 ms at one stream, and +with `conv_route: on` it lands on 727 ms. Same build, same card, same workload; the switch is what moves the first +token ([the pair](DIRIGO.md)). The two arms of the comparison above also ran one config file each, identical apart +from paths, and the same model file for the 28.8 GB lookup table. -**The same config also changes answers, not just timing.** Asked for a date inside a 24-token budget ("Invoice -INV-7 dated 9 June 2024 → ISO date"), stock spends all 24 tokens on reasoning and returns an empty string; the fork -returns `2024-06-09` with zero reasoning characters. The port's suite is green — 1,537 tests, 17 skipped — where the -stock tree has one failure (`test_hip_ordinal`, a test that reads the host's AMD topology; fixed upstream in -[#5602](https://github.com/Niko1221/Strata/pull/5602)). +**Reproduce it.** `dirigo-results/0.2.13-port/scripts/strata-guidellm-0213-ab.sh` runs both arms; +[the report folder](dirigo-results/0.2.13-port/) carries the per-level JSON, the chart, and `make-chart.py`, which +recomputes every figure on this page from those files. Both arms: 0 errored requests. -**What this does not claim.** A faster engine — it is the same speed, and this table would look identical for any -server-side change. A win on conversational traffic, where the fork turns the cache back on and behaves like stock. -A throughput difference at one stream: one extra request in a 45-second window is the measurement's own granularity -(4.8 %), and it is reported rather than counted as a win. +**What it does not show.** A faster engine — parity, measured above. A win on conversations, where the cache comes +back on and the two converge. Anything at eight streams, where both are queue-bound. A throughput win at one stream: +one extra request in a 45-second window is the measurement's own granularity (4.8 %), reported rather than claimed. + +**What the engine both of us run is doing.** Underneath the server, the speed is stock's: only the experts a word +needs are computed (the card and the CPU working at the same time), prompts are read in 8,192-token pieces and reused +on a follow-up, and the draft layer guesses up to three tokens for one-pass verification. None of that is ours, and +none of it is changed by the fork — [how it works](docs/HOW_IT_WORKS.md). > **Measured on:** Intel Arc Pro B70 32 GB, Ryzen 7 7700X, Qwen3.8-Flash-Next Coder IQ1_M, engine 0.1.59 on both > sides, 2026-11-09. One pass per arm in one session; positional drift in this rig is 2–4 %, so the 10.8–22.1 % @@ -115,7 +92,7 @@ A throughput difference at one stream: one extra request in a 45-second window i | Check my hardware before downloading anything | [What you need](#what-you-need) · [which model fits](docs/MODELS.md#pick-by-memory) | 5 min | | Understand the engine | [How it works](docs/HOW_IT_WORKS.md) · [technical reference](docs/DETAILS.md) | — | | Read the evidence for the numbers | [`DIRIGO.md`](DIRIGO.md) · [`dirigo-results/`](dirigo-results/) | — | -| See how it compares to stock 0.2.13 | [Measured side by side](#against-stock-v0213-on-the-same-card) | — | +| See how it compares to stock 0.2.13 | [Measured side by side](#why-this-fork-is-faster-than-stock-strata) | — | | All guides, page by page | [Documentation index](docs/README.md) | — | ## What you need @@ -244,8 +221,8 @@ More: [Troubleshooting](docs/TROUBLESHOOTING.md) · [engine errors](docs/DETAILS ## What this fork changes -The engine, kernels and model format are upstream's. This fork adds server-level behaviour and one engine -change; with none of them enabled the server behaves as stock. +The engine, kernels and model format are upstream's. This fork adds server-level behaviour and one engine change; +with none of them enabled the server behaves as stock. | Change | What it does | | --- | --- | @@ -254,12 +231,34 @@ change; with none of them enabled the server behaves as stock. | retune between requests | change engine settings on a live server — no restart, no reload | | per-request draft depth | a client can set its own speculation depth for one request | +### Around drafting: the same drafter, more control + +Both versions run the model's MTP draft layer; nothing in the carry replaces it. What the fork adds is how it can be +set — and `spec_min_p` per request is upstream's, not ours: + +| setting | stock v0.2.13 | this fork | +| --- | --- | --- | +| draft depth | `--spec` at startup, for every request | also `spec=N` per request, clamped by the engine to 2…the startup value | +| prompt lookup | `--suffix-draft` at startup | also `lookup=0\|1` per request | +| draft confidence | `spec_min_p` per request via `strata_tune` | the same, and changeable globally on a live server | +| window policy | — | `spec_window`: serve some traffic at a lower depth (0 in the measured config) | + +None of it was switched on for the speed numbers above: `spec_window` was 0, and GuideLLM sends no `strata_tune`. + +### Two measured differences that are not speed + +- **Short answers.** Asked for a date inside a 24-token budget ("Invoice INV-7 dated 9 June 2024 → ISO date"), stock + spends all 24 tokens on reasoning and returns an empty string; the fork returns `2024-06-09` with zero reasoning + characters. +- **A green suite.** The port passes 1,537 tests (17 skipped) where the stock tree has one failure — + `test_hip_ordinal`, a test that reads the host's AMD topology, fixed upstream in + [#5602](https://github.com/Niko1221/Strata/pull/5602). + ### What it is worth, measured -The numbers are above, in [Against stock v0.2.13](#against-stock-v0213-on-the-same-card): first token 11–22 % sooner -at one to four streams, engine at parity, and short requests answered that stock truncates. Full report, raw JSON, -the chart and the script that recomputes every figure in it: -[`dirigo-results/0.2.13-port/`](dirigo-results/0.2.13-port/). +The numbers are above, in [Why this fork is faster than stock Strata](#why-this-fork-is-faster-than-stock-strata): +first token 11–22 % sooner at one to four streams, engine at parity. Full report, raw JSON, the chart and the script +that recomputes every figure: [`dirigo-results/0.2.13-port/`](dirigo-results/0.2.13-port/). The parts that belong upstream are open as separate branches listed in [`DIRIGO.md`](DIRIGO.md).