From c4a5843df7c4adb102ebb458ac8f945317574066 Mon Sep 17 00:00:00 2001 From: rwkeyes <86691029+rwkeyes@users.noreply.github.com> Date: Fri, 13 Nov 2026 12:33:34 -0500 Subject: [PATCH] docs: MTP is shared - say what the fork lets you tune, not that we draft The bullet read as if guess-then-check were the fork's, and the clarifying paragraph named only two of the fork's three drafting settings. Both are fixed, and one of the three turned out to be upstream's already. Both arms in the 0.2.13 comparison started with --spec 6 --spec-min-p 0.5 and sent no per-request override, so drafting contributes nothing to the measured gain. That is now stated with the settings, and a small table gives the real difference - control, not a different drafter: draft depth stock: --spec at startup fork: also spec=N per request, 2..startup prompt lookup stock: --suffix-draft at startup fork: also lookup=0|1 per request draft confidence stock: spec_min_p per request fork: the same, plus live global retune window policy stock: none fork: spec_window (off in the measurement) spec_min_p per request is stock behaviour: upstream's server forwards strata_tune{pcie_frac, spec_min_p, prefill_pipe, aux_cpus} in v0.2.13. The fork's additions are `spec`, `lookup`, the window policy and the TUNE/POST /v1/tune channel that changes settings globally without a restart - checked in both serve/server.py files, not assumed. The glossary's MTP entry now matches. --- README.md | 21 +++++++++++++++------ docs/GLOSSARY.md | 4 +++- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index c7fa6b195e5..7a55e2b2177 100644 --- a/README.md +++ b/README.md @@ -31,12 +31,21 @@ Niko1221's — and identical on both sides. Three decisions make that engine fas - **Guess, then check.** The model's built-in draft layer guesses up to three tokens and the model verifies all of them in one pass (2.4–3.2 tokens per pass), so an answer arrives 1.6–1.8× sooner with the same text. -**All three are stock behaviour.** The expert split, the prompt-chunk reads and guess-then-check (the MTP draft layer) -work the same on both sides — the carry changes nothing in how they run, which is why the comparison below builds the -same engine version for both arms. What the fork adds around drafting is two settings a client or a config can send — -a per-request depth (`spec=N`) and a draft window (`spec_window`) — and neither is used by the measurement: -`spec_window` was 0, and GuideLLM never sends `spec`. The fork's measured difference is on the **server** side, -which is the next section. +**All three are stock behaviour.** The expert split, the prompt-chunk reads and guess-then-check (the MTP draft +layer) run the same on both sides. The comparison below starts both arms with `--spec 6 --spec-min-p 0.5` and neither +sends a per-request override, so drafting is not part of the gain. + +What the fork adds is **control over the drafter, not a different drafter**: + +| setting | stock v0.2.13 | this fork | +| --- | --- | --- | +| draft depth | `--spec` at startup, for every request | also `spec=N` per request, clamped by the engine to 2…the startup value | +| prompt lookup | `--suffix-draft` at startup | also `lookup=0\|1` per request | +| draft confidence | `spec_min_p` per request via `strata_tune` | the same, and changeable globally on a live server | +| window policy | — | `spec_window`: serve some traffic at a lower depth (0 in the measurement) | + +None of it was switched on for the numbers below: `spec_window` was 0, and GuideLLM sends no `strata_tune`. The +fork's measured difference is on the **server** side, which is the next section. On the card this branch was verified on (Arc Pro B70, Qwen3.8-Flash-Next Coder IQ1_M) that comes out at **1,404 tok/s** reading a 4K prompt and 1,773 at 16K, **80.8 and 68.1 tok/s** writing, first token after 2.94 s and 9.27 s — diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index a78eb8efbc2..6b824a96466 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -54,7 +54,9 @@ A small built-in model that guesses the next few words so the big model can chec guesses; the big model still decides every word, so answers do not change. See also **spec**. Every build of Strata has it — stock and this fork run the same draft layer, so it is not a difference between them. -What this fork adds is only the ability to set its depth per request (`spec=N`) and a draft window (`spec_window`). +What this fork adds is control over it: a per-request depth (`spec=N`, clamped to the startup value), a per-request +prompt-lookup switch (`lookup=0|1`) and a window policy that serves some traffic at a lower depth (`spec_window`). +Stock sets `--spec` and `--suffix-draft` once at startup; both sides can retune `spec_min_p` per request. **PLE table** The large lookup table (28.8 GB for Qwen3.8-Flash-Next) that Strata reads a few rows of per token from the SSD