From 919e10af3971a4d199649f836093a383035d64d3 Mon Sep 17 00:00:00 2001 From: rwkeyes <86691029+rwkeyes@users.noreply.github.com> Date: Fri, 13 Nov 2026 12:27:08 -0500 Subject: [PATCH] docs: say what is shared with stock before naming the fork's one gain Two things read as if the fork invented them. MTP: the four "why it is fast" bullets were presented under a heading that now carries a fork-vs-stock comparison, so "guess, then check" looked like the fork's trick. It is not - both sides run the same engine version and the same draft layer. The section now says so up front, states that the carry changes none of the four, and names what the fork does add around drafting (per-request `spec=N` and `spec_window`, neither used by the measurement) before pointing at the server-side difference. The 28.8 GB table: it belongs to the model, not to either engine. The bullet now says that, the glossary says neither stock nor this fork creates, modifies or re-encodes it, and the one PLE file the carry does touch is named - `ple_parity`, the table's self-test, which now skips instead of failing when the model file is absent. --- README.md | 17 +++++++++++++---- docs/GLOSSARY.md | 6 +++++- 2 files changed, 18 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index c0194da03ec..97bf6e6eaed 100644 --- a/README.md +++ b/README.md @@ -20,8 +20,8 @@ measurements behind it kept in [`dirigo-results/`](dirigo-results/). What was ve ## Why it is fast -The speed is **upstream Strata's** — Niko1221's engine. This fork adds server behaviour on top of it; the -engine's own performance is not ours. It comes from four decisions: +Stock Strata v0.2.13 and this fork run the **same engine version (0.1.59)**, so the engine's speed is upstream's — +Niko1221's — and identical on both sides. Four decisions make that engine fast: - **Only the experts a word needs are computed.** Qwen3.8-Flash-Next is 24,576 small specialists and a word needs about ten. The busiest live on the card (each free GB holds roughly 700 more), every expert stays in RAM, and the @@ -29,11 +29,20 @@ engine's own performance is not ours. It comes from four decisions: - **Long prompts are read in big pieces** (up to 8,192 tokens), which is why a long document is read at over 1,000 tokens a second, and a follow-up reuses the context already read instead of starting over. - **A 28.8 GB lookup table lives on the SSD** and is read a few rows per token, so it costs disk bandwidth - instead of memory. It is the model's second file: one `per_layer_token_embd.weight` tensor of 160 × 320,001,536 - values at 4 bits — 28,800,138,240 bytes, decimal GB, 26.8 GiB. + instead of memory. It belongs to the **model**, not to either engine: the publisher's second file, one + `per_layer_token_embd.weight` tensor of 160 × 320,001,536 values at 4 bits — 28,800,138,240 bytes, decimal GB, + 26.8 GiB. Stock and this fork read the same file. - **Guess, then check.** The model's built-in draft layer guesses up to three tokens and the model verifies all of them in one pass (2.4–3.2 tokens per pass), so an answer arrives 1.6–1.8× sooner with the same text. +**Both versions do all four.** The expert split, the 28.8 GB table and guess-then-check — the MTP draft layer — are +stock behaviour, not fork features, and the carry does not change how any of them runs; that is why the comparison +below builds the same engine version on both sides. (The one PLE file the carry touches is `ple_parity`, that table's +self-test, which now skips instead of failing when the model file is not on the box.) What the fork adds around drafting is two settings a client or a +config can send — a per-request depth (`spec=N`) and a draft window (`spec_window`) — and neither is used by the +measurement: `spec_window` was 0, and GuideLLM never sends `spec`. The fork's measured difference is on the +**server** side, which is the next section. + On the card this branch was verified on (Arc Pro B70, Qwen3.8-Flash-Next Coder IQ1_M) that comes out at **1,404 tok/s** reading a 4K prompt and 1,773 at 16K, **80.8 and 68.1 tok/s** writing, first token after 2.94 s and 9.27 s — [those measurements](DIRIGO.md#speed-against-stock-measured-with-guidellm-the-standard-tool). How each step of the diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index bfcf292eff5..92b8cc6410c 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -53,11 +53,15 @@ An expert used for every token, so it always lives on the card. A small built-in model that guesses the next few words so the big model can check several at once. It only guesses; the big model still decides every word, so answers do not change. See also **spec**. +Every build of Strata has it — stock and this fork run the same draft layer, so it is not a difference between them. +What this fork adds is only the ability to set its depth per request (`spec=N`) and a draft window (`spec_window`). + **PLE table** The large lookup table (28.8 GB for Qwen3.8-Flash-Next) that Strata reads a few rows of per token from the SSD instead of holding in memory. `--ple-gguf` points at it; `--ple-io` chooses how it is read. -The size is not a round number chosen by a doc: it is the model's **second file**, one tensor +It ships **with the model**, not with Strata, and neither the stock engine nor this fork creates, modifies or +re-encodes it. The size is not a round number chosen by a doc: it is the publisher's second GGUF file, one tensor `per_layer_token_embd.weight` of 160 × 320,001,536 values stored as IQ4_NL (18 bytes per 32 values), which is 160 × 320,001,536 × 18/32 = **28,800,138,240 bytes**. "28.8 GB" throughout these guides is decimal GB — the same file reads 26.8 GiB in `df`. A pack that stores the table in another type is bigger: Unsloth's UD-Q6_K_XL keeps it