diff --git a/README.md b/README.md index 7eb8f408a77..8a89d4a45fd 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,8 @@ engine's performance is not ours. It comes from four decisions: - **Long prompts are read in big pieces** (up to 8,192 tokens), which is why a long document is read at over 1,000 tokens a second, and a follow-up reuses the context already read instead of starting over. - **A 28.8 GB lookup table lives on the SSD** and is read a few rows per token, so it costs disk bandwidth - instead of memory. + instead of memory. It is the model's second file: one `per_layer_token_embd.weight` tensor of 160 × 320,001,536 + values at 4 bits — 28,800,138,240 bytes, decimal GB, 26.8 GiB. - **Guess, then check.** The model's built-in draft layer guesses up to three tokens and the model verifies all of them in one pass (2.4–3.2 tokens per pass), so an answer arrives 1.6–1.8× sooner with the same text. diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index c05e23c2a0a..bfcf292eff5 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -57,6 +57,12 @@ guesses; the big model still decides every word, so answers do not change. See a The large lookup table (28.8 GB for Qwen3.8-Flash-Next) that Strata reads a few rows of per token from the SSD instead of holding in memory. `--ple-gguf` points at it; `--ple-io` chooses how it is read. +The size is not a round number chosen by a doc: it is the model's **second file**, one tensor +`per_layer_token_embd.weight` of 160 × 320,001,536 values stored as IQ4_NL (18 bytes per 32 values), which is +160 × 320,001,536 × 18/32 = **28,800,138,240 bytes**. "28.8 GB" throughout these guides is decimal GB — the same +file reads 26.8 GiB in `df`. A pack that stores the table in another type is bigger: Unsloth's UD-Q6_K_XL keeps it +as Q8_0 (170 bytes per 160-value row) and the same table becomes 54.4 GB. + ## The engine **Engine**