diff --git a/README.md b/README.md index 7a55e2b2177..3af69fba9dd 100644 --- a/README.md +++ b/README.md @@ -18,49 +18,18 @@ engine **0.1.58**) plus this fork's changes, in the state that served a live mac measurements behind it kept in [`dirigo-results/`](dirigo-results/). What was verified, and how: [`DIRIGO.md`](DIRIGO.md). -## Why it is fast +## Why this fork is faster than stock Strata -Stock Strata v0.2.13 and this fork run the **same engine version (0.1.59)**, so the engine's speed is upstream's — -Niko1221's — and identical on both sides. Three decisions make that engine fast: +"Stock" here is **Niko1221's v0.2.13** — the upstream release — built from the same tag and run on the same card, +model, pack, config and request stream as this fork. Everything below is measured on both sides, not estimated. -- **Only the experts a word needs are computed.** Qwen3.8-Flash-Next is 24,576 small specialists and a word - needs about ten. The busiest live on the card (each free GB holds roughly 700 more), every expert stays in RAM, and the - CPU computes the ones the card does not hold *at the same time* as the card works — neither waits. -- **Long prompts are read in big pieces** (up to 8,192 tokens), which is why a long document is read at over - 1,000 tokens a second, and a follow-up reuses the context already read instead of starting over. -- **Guess, then check.** The model's built-in draft layer guesses up to three tokens and the model verifies all - of them in one pass (2.4–3.2 tokens per pass), so an answer arrives 1.6–1.8× sooner with the same text. +**First, the part that is not faster.** Both trees build engine 0.1.59, and the engine-level battery — eight legs, +prefill and decode, straight to the engine, two interleaved passes — lands within **1.1 %** of each other. If the fork +had only the engine to show, this section would be empty. The difference is in the server: -**All three are stock behaviour.** The expert split, the prompt-chunk reads and guess-then-check (the MTP draft -layer) run the same on both sides. The comparison below starts both arms with `--spec 6 --spec-min-p 0.5` and neither -sends a per-request override, so drafting is not part of the gain. - -What the fork adds is **control over the drafter, not a different drafter**: - -| setting | stock v0.2.13 | this fork | -| --- | --- | --- | -| draft depth | `--spec` at startup, for every request | also `spec=N` per request, clamped by the engine to 2…the startup value | -| prompt lookup | `--suffix-draft` at startup | also `lookup=0\|1` per request | -| draft confidence | `spec_min_p` per request via `strata_tune` | the same, and changeable globally on a live server | -| window policy | — | `spec_window`: serve some traffic at a lower depth (0 in the measurement) | - -None of it was switched on for the numbers below: `spec_window` was 0, and GuideLLM sends no `strata_tune`. The -fork's measured difference is on the **server** side, which is the next section. - -On the card this branch was verified on (Arc Pro B70, Qwen3.8-Flash-Next Coder IQ1_M) that comes out at **1,404 tok/s** -reading a 4K prompt and 1,773 at 16K, **80.8 and 68.1 tok/s** writing, first token after 2.94 s and 9.27 s — -[those measurements](DIRIGO.md#speed-against-stock-measured-with-guidellm-the-standard-tool). How each step of the -engine's speed was won is the [history table in `docs/DETAILS.md`](docs/DETAILS.md); it is not repeated here. - -### Against stock v0.2.13, on the same card - -Same card, same engine version (0.1.59), same model and pack, same server config, same request stream. GuideLLM -0.8.1, 512 tokens in / 128 out, one pass per arm. - -**The short version.** The first token arrives **11–22 % sooner** on a stream of fresh prompts — 739 ms against -stock's 949 ms at one stream — and at four streams the same 45-second window finishes **three more answers**. Nothing -is traded for it: prefill and decode are within 1.1 % of stock on all eight engine-level legs, and the gain is a -server-side one — the engine measures the same on both sides. +**The first token arrives 11–22 % sooner** on a stream of fresh prompts — 739 ms against stock's 949 ms at one +stream — and at four streams the same 45-second window finishes **three more answers**. Nothing is traded for it: the +engine figures are parity, as above.

Time to first token, Dirigo 0.2.13 vs stock 0.2.13 on an Intel Arc Pro B70: 739 vs 949 ms at one stream, 2622 vs 3163 at two, 6688 vs 7500 at four, 9641 vs 9659 at eight. Engine-level prefill and decode are within 1.1 percent on all eight legs. @@ -77,22 +46,30 @@ The last two columns are one measurement read two ways: 739 ms is 22.1 % below 9 739 ms. "Arrives sooner" is the column to quote. In the same 45-second window the fork finishes 22 / 22 / 22 / 23 requests against 21 / 20 / 19 / 21 — 62.6 / 62.6 / 62.6 / 65.4 output tokens a second against 59.7 / 56.9 / 54.0 / 59.7. -**Why the gap exists.** A fresh prompt is the one case the two servers handle differently. Stock opens the prompt -cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique prompts as -one-shot traffic and serves it with the cache **off** — no pool work between the request and the first token — then -switches the cache back on the moment the traffic looks like a conversation. That is why the advantage is largest -with requests arriving one at a time, and gone at eight streams where both servers are queue-bound. +**Where the difference comes from.** A fresh prompt is the one case the two servers handle differently. Stock opens +the prompt cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique +prompts as one-shot traffic and serves it with the cache **off** — no pool work between the request and the first +token — then switches the cache back on the moment the traffic looks like a conversation. That is why the advantage is +largest with requests arriving one at a time, and gone at eight streams where both servers are queue-bound. + +**A control, so the mechanism is not taken on faith.** The same router was measured in both positions on one build: +with `conv_route: shadow` — classify and log, never switch — it lands on the stock number, 934 ms at one stream, and +with `conv_route: on` it lands on 727 ms. Same build, same card, same workload; the switch is what moves the first +token ([the pair](DIRIGO.md)). The two arms of the comparison above also ran one config file each, identical apart +from paths, and the same model file for the 28.8 GB lookup table. -**The same config also changes answers, not just timing.** Asked for a date inside a 24-token budget ("Invoice -INV-7 dated 9 June 2024 → ISO date"), stock spends all 24 tokens on reasoning and returns an empty string; the fork -returns `2024-06-09` with zero reasoning characters. The port's suite is green — 1,537 tests, 17 skipped — where the -stock tree has one failure (`test_hip_ordinal`, a test that reads the host's AMD topology; fixed upstream in -[#5602](https://github.com/Niko1221/Strata/pull/5602)). +**Reproduce it.** `dirigo-results/0.2.13-port/scripts/strata-guidellm-0213-ab.sh` runs both arms; +[the report folder](dirigo-results/0.2.13-port/) carries the per-level JSON, the chart, and `make-chart.py`, which +recomputes every figure on this page from those files. Both arms: 0 errored requests. -**What this does not claim.** A faster engine — it is the same speed, and this table would look identical for any -server-side change. A win on conversational traffic, where the fork turns the cache back on and behaves like stock. -A throughput difference at one stream: one extra request in a 45-second window is the measurement's own granularity -(4.8 %), and it is reported rather than counted as a win. +**What it does not show.** A faster engine — parity, measured above. A win on conversations, where the cache comes +back on and the two converge. Anything at eight streams, where both are queue-bound. A throughput win at one stream: +one extra request in a 45-second window is the measurement's own granularity (4.8 %), reported rather than claimed. + +**What the engine both of us run is doing.** Underneath the server, the speed is stock's: only the experts a word +needs are computed (the card and the CPU working at the same time), prompts are read in 8,192-token pieces and reused +on a follow-up, and the draft layer guesses up to three tokens for one-pass verification. None of that is ours, and +none of it is changed by the fork — [how it works](docs/HOW_IT_WORKS.md). > **Measured on:** Intel Arc Pro B70 32 GB, Ryzen 7 7700X, Qwen3.8-Flash-Next Coder IQ1_M, engine 0.1.59 on both > sides, 2026-11-09. One pass per arm in one session; positional drift in this rig is 2–4 %, so the 10.8–22.1 % @@ -115,7 +92,7 @@ A throughput difference at one stream: one extra request in a 45-second window i | Check my hardware before downloading anything | [What you need](#what-you-need) · [which model fits](docs/MODELS.md#pick-by-memory) | 5 min | | Understand the engine | [How it works](docs/HOW_IT_WORKS.md) · [technical reference](docs/DETAILS.md) | — | | Read the evidence for the numbers | [`DIRIGO.md`](DIRIGO.md) · [`dirigo-results/`](dirigo-results/) | — | -| See how it compares to stock 0.2.13 | [Measured side by side](#against-stock-v0213-on-the-same-card) | — | +| See how it compares to stock 0.2.13 | [Measured side by side](#why-this-fork-is-faster-than-stock-strata) | — | | All guides, page by page | [Documentation index](docs/README.md) | — | ## What you need @@ -244,8 +221,8 @@ More: [Troubleshooting](docs/TROUBLESHOOTING.md) · [engine errors](docs/DETAILS ## What this fork changes -The engine, kernels and model format are upstream's. This fork adds server-level behaviour and one engine -change; with none of them enabled the server behaves as stock. +The engine, kernels and model format are upstream's. This fork adds server-level behaviour and one engine change; +with none of them enabled the server behaves as stock. | Change | What it does | | --- | --- | @@ -254,12 +231,34 @@ change; with none of them enabled the server behaves as stock. | retune between requests | change engine settings on a live server — no restart, no reload | | per-request draft depth | a client can set its own speculation depth for one request | +### Around drafting: the same drafter, more control + +Both versions run the model's MTP draft layer; nothing in the carry replaces it. What the fork adds is how it can be +set — and `spec_min_p` per request is upstream's, not ours: + +| setting | stock v0.2.13 | this fork | +| --- | --- | --- | +| draft depth | `--spec` at startup, for every request | also `spec=N` per request, clamped by the engine to 2…the startup value | +| prompt lookup | `--suffix-draft` at startup | also `lookup=0\|1` per request | +| draft confidence | `spec_min_p` per request via `strata_tune` | the same, and changeable globally on a live server | +| window policy | — | `spec_window`: serve some traffic at a lower depth (0 in the measured config) | + +None of it was switched on for the speed numbers above: `spec_window` was 0, and GuideLLM sends no `strata_tune`. + +### Two measured differences that are not speed + +- **Short answers.** Asked for a date inside a 24-token budget ("Invoice INV-7 dated 9 June 2024 → ISO date"), stock + spends all 24 tokens on reasoning and returns an empty string; the fork returns `2024-06-09` with zero reasoning + characters. +- **A green suite.** The port passes 1,537 tests (17 skipped) where the stock tree has one failure — + `test_hip_ordinal`, a test that reads the host's AMD topology, fixed upstream in + [#5602](https://github.com/Niko1221/Strata/pull/5602). + ### What it is worth, measured -The numbers are above, in [Against stock v0.2.13](#against-stock-v0213-on-the-same-card): first token 11–22 % sooner -at one to four streams, engine at parity, and short requests answered that stock truncates. Full report, raw JSON, -the chart and the script that recomputes every figure in it: -[`dirigo-results/0.2.13-port/`](dirigo-results/0.2.13-port/). +The numbers are above, in [Why this fork is faster than stock Strata](#why-this-fork-is-faster-than-stock-strata): +first token 11–22 % sooner at one to four streams, engine at parity. Full report, raw JSON, the chart and the script +that recomputes every figure: [`dirigo-results/0.2.13-port/`](dirigo-results/0.2.13-port/). The parts that belong upstream are open as separate branches listed in [`DIRIGO.md`](DIRIGO.md).