Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion DIRIGO.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,13 @@ The difference is **this fork's conversation-cache routing, not the upstream tag

Raw results: [`dirigo-results/guidellm/`](dirigo-results/guidellm/). Limits are in the write-up: one card, one slot, one
quantization, a single-shot workload — it measures the "run of single-shot prompts" side of the router, not the switch
back on a conversational turn, and the `--conv-route-batch-after` threshold was not swept.
back on a conversational turn.

Both gaps were closed on 2026-11-14 — see [`docs/PERF-NEXT.md`](docs/PERF-NEXT.md) §7. The switch back costs a
conversation **1.9 s** per re-open, so `conv_route_batch_after` stays at 4: a run of one still saves 0.52 s across six
one-shot requests and loses 1.88 s on the next conversation turn. Sweeping that threshold is what found the second
thing: the server read the flag but ignored the `conv_route_batch_after` key in the config, so every deployment ran 4
whatever the config said. Fixed in `b906f0b5` (PR #16).

## What else was measured, and what was rejected

Expand Down
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,10 @@ python tools/iq_pack.py \
```

Then serve. These are the settings the measured service ran with — replace the paths, and keep
`data/expert-profile*.bin` where the binary can find it:
`data/expert-profile*.bin` where the binary can find it. The profile is matched to the model's **expert count**, not its
quantization: the full model (IQ2/IQ3/Q2 packs, 48×512) takes `data/expert-profile.bin`; the pruned *Coder* model
(48×256) takes `data/expert-profile-coder.bin`. The wrong one is refused at start with
`it is a profile for a different model`:

```sh
./build-sycl-aot/strata --serve \
Expand Down Expand Up @@ -244,7 +247,7 @@ with none of them enabled the server behaves as stock.

| Change | What it does |
| --- | --- |
| conversation-cache routing | turns the prompt cache off for runs of single-shot prompts, back on for a conversation |
| conversation-cache routing | turns the prompt cache off after **four** single-shot prompts in a row; one conversational turn switches it back on. Worth −12…−22 % first-token time at 1–4 clients and nothing at 8, and nothing on a conversation — and a threshold of 1 costs 1.9 s per conversation ([why](docs/PERF-NEXT.md#7-mixed-traffic-why-batch_after-stays-at-4)) |
| thinking policy | skips the scratchpad for short, factual asks; keeps it when the answer needs work |
| retune between requests | change engine settings on a live server — no restart, no reload |
| per-request draft depth | a client can set its own speculation depth for one request |
Expand Down
5 changes: 5 additions & 0 deletions docs/GETTING-STARTED.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,11 @@ python tools/iq_pack.py \
The pack is finished only when it contains `native_experts.txt`; a pack without it will not load. Keep the pack,
the tokenizer folder and the GGUF shards together.

Pick the expert profile by the model's **expert count**, which is in the pack's `index.txt`, not by its quantization:
`data/expert-profile.bin` for the full 48×512 model (IQ2_*/IQ3_*/Q2_0), `data/expert-profile-coder.bin` for the pruned
48×256 Coder. Getting this wrong stops the engine with `it is a profile for a different model` — a one-line fix, but it
reads like a corrupted pack.

### B4. Serve the engine

```sh
Expand Down
6 changes: 4 additions & 2 deletions docs/GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,10 @@ A channel for changing engine settings on a running engine, between requests, wi
channel.

**conv route / conversation-cache routing (this fork)**
A server-side rule that turns the prompt cache off during a run of one-shot prompts (faster prefill) and back
on for a conversation (prefix reuse). Modes: `off`, `shadow` (classify and log only), `on`.
A server-side rule that turns the prompt cache off after **four** single-shot prompts in a row (faster prefill) and
back on at the next conversational turn (prefix reuse). Modes: `off`, `shadow` (classify and log only), `on`.
The four is measured, not arbitrary: at a run length of one, the conversation that follows re-reads its whole context
(**+1.9 s**), while the one-shot side saves only 0.17 s per request — break-even is ~21 requests per conversation.

**Carry**
This fork's delta on top of an upstream tag: 35 files, +5,024/−264 for v0.2.12. Kept as one patch in
Expand Down
3 changes: 2 additions & 1 deletion docs/HOW_IT_WORKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
Models like Qwen3.8-Flash-Next normally run on servers with hundreds of gigabytes of graphics memory. Your graphics
card has 12-24 GB. Strata makes it fit by **sharing the work across your whole PC** - the same idea as a kitchen, where
the things you use all the time stay on the counter and the rest waits in the pantry. Back to the
[README](../README.md#how-does-it-work).
[README](../README.md), or to [what this fork changes](../README.md#what-this-fork-changes) if you want the delta rather
than the mechanism.

## Who does what

Expand Down
Loading