diff --git a/DIRIGO.md b/DIRIGO.md index 36c3b0b43b6..f145627452a 100644 --- a/DIRIGO.md +++ b/DIRIGO.md @@ -80,7 +80,13 @@ The difference is **this fork's conversation-cache routing, not the upstream tag Raw results: [`dirigo-results/guidellm/`](dirigo-results/guidellm/). Limits are in the write-up: one card, one slot, one quantization, a single-shot workload — it measures the "run of single-shot prompts" side of the router, not the switch -back on a conversational turn, and the `--conv-route-batch-after` threshold was not swept. +back on a conversational turn. + +Both gaps were closed on 2026-11-14 — see [`docs/PERF-NEXT.md`](docs/PERF-NEXT.md) §7. The switch back costs a +conversation **1.9 s** per re-open, so `conv_route_batch_after` stays at 4: a run of one still saves 0.52 s across six +one-shot requests and loses 1.88 s on the next conversation turn. Sweeping that threshold is what found the second +thing: the server read the flag but ignored the `conv_route_batch_after` key in the config, so every deployment ran 4 +whatever the config said. Fixed in `b906f0b5` (PR #16). ## What else was measured, and what was rejected diff --git a/README.md b/README.md index 1ed32f5be45..7072ee39caf 100644 --- a/README.md +++ b/README.md @@ -164,7 +164,10 @@ python tools/iq_pack.py \ ``` Then serve. These are the settings the measured service ran with — replace the paths, and keep -`data/expert-profile*.bin` where the binary can find it: +`data/expert-profile*.bin` where the binary can find it. The profile is matched to the model's **expert count**, not its +quantization: the full model (IQ2/IQ3/Q2 packs, 48×512) takes `data/expert-profile.bin`; the pruned *Coder* model +(48×256) takes `data/expert-profile-coder.bin`. The wrong one is refused at start with +`it is a profile for a different model`: ```sh ./build-sycl-aot/strata --serve \ @@ -244,7 +247,7 @@ with none of them enabled the server behaves as stock. | Change | What it does | | --- | --- | -| conversation-cache routing | turns the prompt cache off for runs of single-shot prompts, back on for a conversation | +| conversation-cache routing | turns the prompt cache off after **four** single-shot prompts in a row; one conversational turn switches it back on. Worth −12…−22 % first-token time at 1–4 clients and nothing at 8, and nothing on a conversation — and a threshold of 1 costs 1.9 s per conversation ([why](docs/PERF-NEXT.md#7-mixed-traffic-why-batch_after-stays-at-4)) | | thinking policy | skips the scratchpad for short, factual asks; keeps it when the answer needs work | | retune between requests | change engine settings on a live server — no restart, no reload | | per-request draft depth | a client can set its own speculation depth for one request | diff --git a/docs/GETTING-STARTED.md b/docs/GETTING-STARTED.md index 0c3a86d516a..77822bf7451 100644 --- a/docs/GETTING-STARTED.md +++ b/docs/GETTING-STARTED.md @@ -159,6 +159,11 @@ python tools/iq_pack.py \ The pack is finished only when it contains `native_experts.txt`; a pack without it will not load. Keep the pack, the tokenizer folder and the GGUF shards together. +Pick the expert profile by the model's **expert count**, which is in the pack's `index.txt`, not by its quantization: +`data/expert-profile.bin` for the full 48×512 model (IQ2_*/IQ3_*/Q2_0), `data/expert-profile-coder.bin` for the pruned +48×256 Coder. Getting this wrong stops the engine with `it is a profile for a different model` — a one-line fix, but it +reads like a corrupted pack. + ### B4. Serve the engine ```sh diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index 6b824a96466..6659dd15bcb 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -155,8 +155,10 @@ A channel for changing engine settings on a running engine, between requests, wi channel. **conv route / conversation-cache routing (this fork)** -A server-side rule that turns the prompt cache off during a run of one-shot prompts (faster prefill) and back -on for a conversation (prefix reuse). Modes: `off`, `shadow` (classify and log only), `on`. +A server-side rule that turns the prompt cache off after **four** single-shot prompts in a row (faster prefill) and +back on at the next conversational turn (prefix reuse). Modes: `off`, `shadow` (classify and log only), `on`. +The four is measured, not arbitrary: at a run length of one, the conversation that follows re-reads its whole context +(**+1.9 s**), while the one-shot side saves only 0.17 s per request — break-even is ~21 requests per conversation. **Carry** This fork's delta on top of an upstream tag: 35 files, +5,024/−264 for v0.2.12. Kept as one patch in diff --git a/docs/HOW_IT_WORKS.md b/docs/HOW_IT_WORKS.md index 7c8f9eecf6a..c5918ec9ded 100644 --- a/docs/HOW_IT_WORKS.md +++ b/docs/HOW_IT_WORKS.md @@ -3,7 +3,8 @@ Models like Qwen3.8-Flash-Next normally run on servers with hundreds of gigabytes of graphics memory. Your graphics card has 12-24 GB. Strata makes it fit by **sharing the work across your whole PC** - the same idea as a kitchen, where the things you use all the time stay on the counter and the rest waits in the pantry. Back to the -[README](../README.md#how-does-it-work). +[README](../README.md), or to [what this fork changes](../README.md#what-this-fork-changes) if you want the delta rather +than the mechanism. ## Who does what