From 9e422261631a3498bd34e952fc56e8952660009b Mon Sep 17 00:00:00 2001 From: rwkeyes <86691029+rwkeyes@users.noreply.github.com> Date: Sun, 15 Nov 2026 07:21:19 -0500 Subject: [PATCH] docs: correct the claims the last day of measurement invalidated Four fixes to pages a first-time reader meets before anything else. 1. DIRIGO.md said the batch_after threshold "was not swept" and that the router's switch back was unmeasured. Both were measured on 2026-11-14: the switch back costs a conversation 1.9 s per re-open, so the value stays at 4, and sweeping it found that the config key was never read (fixed in b906f0b5). The page now says that instead of listing a limit that no longer exists. 2. README's change table said the cache goes off "for runs of single-shot prompts" without the number, so a reader could think one request flips it. It says four, and what the other settings cost. 3. The same table claimed the routing is worth -12..-22 % without saying at how many clients: it is 1-4, nothing at 8. 4. Both quick starts named data/expert-profile.bin without saying which model shape it fits. The profile is matched on the expert count, not the quantization: 48x512 for the full model, 48x256 for the pruned Coder - and the wrong one stops the engine with "it is a profile for a different model", which reads like a corrupted pack. Verified from the IQ3_XXS GGUF's own metadata (expert_count 512) before writing it. Also repairs the one dead link a reader could hit: docs/HOW_IT_WORKS.md pointed at ../README.md#how-does-it-work, a heading the README rewrite removed. A second candidate (MODELS.md#a-big-graphics-card-and-little-ram) was left alone after checking: that anchor resolves, and my first attempt at "fixing" it would have broken a working link. --- DIRIGO.md | 8 +++++++- README.md | 7 +++++-- docs/GETTING-STARTED.md | 5 +++++ docs/GLOSSARY.md | 6 ++++-- docs/HOW_IT_WORKS.md | 3 ++- 5 files changed, 23 insertions(+), 6 deletions(-) diff --git a/DIRIGO.md b/DIRIGO.md index 36c3b0b43b6..f145627452a 100644 --- a/DIRIGO.md +++ b/DIRIGO.md @@ -80,7 +80,13 @@ The difference is **this fork's conversation-cache routing, not the upstream tag Raw results: [`dirigo-results/guidellm/`](dirigo-results/guidellm/). Limits are in the write-up: one card, one slot, one quantization, a single-shot workload — it measures the "run of single-shot prompts" side of the router, not the switch -back on a conversational turn, and the `--conv-route-batch-after` threshold was not swept. +back on a conversational turn. + +Both gaps were closed on 2026-11-14 — see [`docs/PERF-NEXT.md`](docs/PERF-NEXT.md) §7. The switch back costs a +conversation **1.9 s** per re-open, so `conv_route_batch_after` stays at 4: a run of one still saves 0.52 s across six +one-shot requests and loses 1.88 s on the next conversation turn. Sweeping that threshold is what found the second +thing: the server read the flag but ignored the `conv_route_batch_after` key in the config, so every deployment ran 4 +whatever the config said. Fixed in `b906f0b5` (PR #16). ## What else was measured, and what was rejected diff --git a/README.md b/README.md index 1ed32f5be45..7072ee39caf 100644 --- a/README.md +++ b/README.md @@ -164,7 +164,10 @@ python tools/iq_pack.py \ ``` Then serve. These are the settings the measured service ran with — replace the paths, and keep -`data/expert-profile*.bin` where the binary can find it: +`data/expert-profile*.bin` where the binary can find it. The profile is matched to the model's **expert count**, not its +quantization: the full model (IQ2/IQ3/Q2 packs, 48×512) takes `data/expert-profile.bin`; the pruned *Coder* model +(48×256) takes `data/expert-profile-coder.bin`. The wrong one is refused at start with +`it is a profile for a different model`: ```sh ./build-sycl-aot/strata --serve \ @@ -244,7 +247,7 @@ with none of them enabled the server behaves as stock. | Change | What it does | | --- | --- | -| conversation-cache routing | turns the prompt cache off for runs of single-shot prompts, back on for a conversation | +| conversation-cache routing | turns the prompt cache off after **four** single-shot prompts in a row; one conversational turn switches it back on. Worth −12…−22 % first-token time at 1–4 clients and nothing at 8, and nothing on a conversation — and a threshold of 1 costs 1.9 s per conversation ([why](docs/PERF-NEXT.md#7-mixed-traffic-why-batch_after-stays-at-4)) | | thinking policy | skips the scratchpad for short, factual asks; keeps it when the answer needs work | | retune between requests | change engine settings on a live server — no restart, no reload | | per-request draft depth | a client can set its own speculation depth for one request | diff --git a/docs/GETTING-STARTED.md b/docs/GETTING-STARTED.md index 0c3a86d516a..77822bf7451 100644 --- a/docs/GETTING-STARTED.md +++ b/docs/GETTING-STARTED.md @@ -159,6 +159,11 @@ python tools/iq_pack.py \ The pack is finished only when it contains `native_experts.txt`; a pack without it will not load. Keep the pack, the tokenizer folder and the GGUF shards together. +Pick the expert profile by the model's **expert count**, which is in the pack's `index.txt`, not by its quantization: +`data/expert-profile.bin` for the full 48×512 model (IQ2_*/IQ3_*/Q2_0), `data/expert-profile-coder.bin` for the pruned +48×256 Coder. Getting this wrong stops the engine with `it is a profile for a different model` — a one-line fix, but it +reads like a corrupted pack. + ### B4. Serve the engine ```sh diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index 6b824a96466..6659dd15bcb 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -155,8 +155,10 @@ A channel for changing engine settings on a running engine, between requests, wi channel. **conv route / conversation-cache routing (this fork)** -A server-side rule that turns the prompt cache off during a run of one-shot prompts (faster prefill) and back -on for a conversation (prefix reuse). Modes: `off`, `shadow` (classify and log only), `on`. +A server-side rule that turns the prompt cache off after **four** single-shot prompts in a row (faster prefill) and +back on at the next conversational turn (prefix reuse). Modes: `off`, `shadow` (classify and log only), `on`. +The four is measured, not arbitrary: at a run length of one, the conversation that follows re-reads its whole context +(**+1.9 s**), while the one-shot side saves only 0.17 s per request — break-even is ~21 requests per conversation. **Carry** This fork's delta on top of an upstream tag: 35 files, +5,024/−264 for v0.2.12. Kept as one patch in diff --git a/docs/HOW_IT_WORKS.md b/docs/HOW_IT_WORKS.md index 7c8f9eecf6a..c5918ec9ded 100644 --- a/docs/HOW_IT_WORKS.md +++ b/docs/HOW_IT_WORKS.md @@ -3,7 +3,8 @@ Models like Qwen3.8-Flash-Next normally run on servers with hundreds of gigabytes of graphics memory. Your graphics card has 12-24 GB. Strata makes it fit by **sharing the work across your whole PC** - the same idea as a kitchen, where the things you use all the time stay on the counter and the rest waits in the pantry. Back to the -[README](../README.md#how-does-it-work). +[README](../README.md), or to [what this fork changes](../README.md#what-this-fork-changes) if you want the delta rather +than the mechanism. ## Who does what