Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ engine's performance is not ours. It comes from four decisions:
- **Long prompts are read in big pieces** (up to 8,192 tokens), which is why a long document is read at over
1,000 tokens a second, and a follow-up reuses the context already read instead of starting over.
- **A 28.8 GB lookup table lives on the SSD** and is read a few rows per token, so it costs disk bandwidth
instead of memory.
instead of memory. It is the model's second file: one `per_layer_token_embd.weight` tensor of 160 × 320,001,536
values at 4 bits — 28,800,138,240 bytes, decimal GB, 26.8 GiB.
- **Guess, then check.** The model's built-in draft layer guesses up to three tokens and the model verifies all
of them in one pass (2.4–3.2 tokens per pass), so an answer arrives 1.6–1.8× sooner with the same text.

Expand Down
6 changes: 6 additions & 0 deletions docs/GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ guesses; the big model still decides every word, so answers do not change. See a
The large lookup table (28.8 GB for Qwen3.8-Flash-Next) that Strata reads a few rows of per token from the SSD
instead of holding in memory. `--ple-gguf` points at it; `--ple-io` chooses how it is read.

The size is not a round number chosen by a doc: it is the model's **second file**, one tensor
`per_layer_token_embd.weight` of 160 × 320,001,536 values stored as IQ4_NL (18 bytes per 32 values), which is
160 × 320,001,536 × 18/32 = **28,800,138,240 bytes**. "28.8 GB" throughout these guides is decimal GB — the same
file reads 26.8 GiB in `df`. A pack that stores the table in another type is bigger: Unsloth's UD-Q6_K_XL keeps it
as Q8_0 (170 bytes per 160-value row) and the same table becomes 54.4 GB.

## The engine

**Engine**
Expand Down
Loading