Skip to content

[Docs] Rename to "Why this fork is faster than stock Strata" and demonstrate it #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rwkeyes merged 1 commit into from
Nov 13, 2026
Merged

[Docs] Rename to "Why this fork is faster than stock Strata" and demonstrate it #11

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 60 additions & 61 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,49 +18,18 @@ engine **0.1.58**) plus this fork's changes, in the state that served a live mac
measurements behind it kept in [`dirigo-results/`](dirigo-results/). What was verified, and how:
[`DIRIGO.md`](DIRIGO.md).

## Why it is fast
## Why this fork is faster than stock Strata

Stock Strata v0.2.13 and this fork run the **same engine version (0.1.59)**, so the engine's speed is upstream's —
Niko1221's — and identical on both sides. Three decisions make that engine fast:
"Stock" here is **Niko1221's v0.2.13** — the upstream release — built from the same tag and run on the same card,
model, pack, config and request stream as this fork. Everything below is measured on both sides, not estimated.

- **Only the experts a word needs are computed.** Qwen3.8-Flash-Next is 24,576 small specialists and a word
needs about ten. The busiest live on the card (each free GB holds roughly 700 more), every expert stays in RAM, and the
CPU computes the ones the card does not hold *at the same time* as the card works — neither waits.
- **Long prompts are read in big pieces** (up to 8,192 tokens), which is why a long document is read at over
1,000 tokens a second, and a follow-up reuses the context already read instead of starting over.
- **Guess, then check.** The model's built-in draft layer guesses up to three tokens and the model verifies all
of them in one pass (2.4–3.2 tokens per pass), so an answer arrives 1.6–1.8× sooner with the same text.
**First, the part that is not faster.** Both trees build engine 0.1.59, and the engine-level battery — eight legs,
prefill and decode, straight to the engine, two interleaved passes — lands within **1.1 %** of each other. If the fork
had only the engine to show, this section would be empty. The difference is in the server:

**All three are stock behaviour.** The expert split, the prompt-chunk reads and guess-then-check (the MTP draft
layer) run the same on both sides. The comparison below starts both arms with `--spec 6 --spec-min-p 0.5` and neither
sends a per-request override, so drafting is not part of the gain.

What the fork adds is **control over the drafter, not a different drafter**:

| setting | stock v0.2.13 | this fork |
| --- | --- | --- |
| draft depth | `--spec` at startup, for every request | also `spec=N` per request, clamped by the engine to 2…the startup value |
| prompt lookup | `--suffix-draft` at startup | also `lookup=0\|1` per request |
| draft confidence | `spec_min_p` per request via `strata_tune` | the same, and changeable globally on a live server |
| window policy | — | `spec_window`: serve some traffic at a lower depth (0 in the measurement) |

None of it was switched on for the numbers below: `spec_window` was 0, and GuideLLM sends no `strata_tune`. The
fork's measured difference is on the **server** side, which is the next section.

On the card this branch was verified on (Arc Pro B70, Qwen3.8-Flash-Next Coder IQ1_M) that comes out at **1,404 tok/s**
reading a 4K prompt and 1,773 at 16K, **80.8 and 68.1 tok/s** writing, first token after 2.94 s and 9.27 s —
[those measurements](DIRIGO.md#speed-against-stock-measured-with-guidellm-the-standard-tool). How each step of the
engine's speed was won is the [history table in `docs/DETAILS.md`](docs/DETAILS.md); it is not repeated here.

### Against stock v0.2.13, on the same card

Same card, same engine version (0.1.59), same model and pack, same server config, same request stream. GuideLLM
0.8.1, 512 tokens in / 128 out, one pass per arm.

**The short version.** The first token arrives **11–22 % sooner** on a stream of fresh prompts — 739 ms against
stock's 949 ms at one stream — and at four streams the same 45-second window finishes **three more answers**. Nothing
is traded for it: prefill and decode are within 1.1 % of stock on all eight engine-level legs, and the gain is a
server-side one — the engine measures the same on both sides.
**The first token arrives 11–22 % sooner** on a stream of fresh prompts — 739 ms against stock's 949 ms at one
stream — and at four streams the same 45-second window finishes **three more answers**. Nothing is traded for it: the
engine figures are parity, as above.

<p align="center">
<img src="dirigo-results/0.2.13-port/chart.svg" width="1000" alt="Time to first token, Dirigo 0.2.13 vs stock 0.2.13 on an Intel Arc Pro B70: 739 vs 949 ms at one stream, 2622 vs 3163 at two, 6688 vs 7500 at four, 9641 vs 9659 at eight. Engine-level prefill and decode are within 1.1 percent on all eight legs.">
Expand All @@ -77,22 +46,30 @@ The last two columns are one measurement read two ways: 739 ms is 22.1 % below 9
739 ms. "Arrives sooner" is the column to quote. In the same 45-second window the fork finishes 22 / 22 / 22 / 23
requests against 21 / 20 / 19 / 21 — 62.6 / 62.6 / 62.6 / 65.4 output tokens a second against 59.7 / 56.9 / 54.0 / 59.7.

**Why the gap exists.** A fresh prompt is the one case the two servers handle differently. Stock opens the prompt
cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique prompts as
one-shot traffic and serves it with the cache **off** — no pool work between the request and the first token — then
switches the cache back on the moment the traffic looks like a conversation. That is why the advantage is largest
with requests arriving one at a time, and gone at eight streams where both servers are queue-bound.
**Where the difference comes from.** A fresh prompt is the one case the two servers handle differently. Stock opens
the prompt cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique
prompts as one-shot traffic and serves it with the cache **off** — no pool work between the request and the first
token — then switches the cache back on the moment the traffic looks like a conversation. That is why the advantage is
largest with requests arriving one at a time, and gone at eight streams where both servers are queue-bound.

**A control, so the mechanism is not taken on faith.** The same router was measured in both positions on one build:
with `conv_route: shadow` — classify and log, never switch — it lands on the stock number, 934 ms at one stream, and
with `conv_route: on` it lands on 727 ms. Same build, same card, same workload; the switch is what moves the first
token ([the pair](DIRIGO.md)). The two arms of the comparison above also ran one config file each, identical apart
from paths, and the same model file for the 28.8 GB lookup table.

**The same config also changes answers, not just timing.** Asked for a date inside a 24-token budget ("Invoice
INV-7 dated 9 June 2024 → ISO date"), stock spends all 24 tokens on reasoning and returns an empty string; the fork
returns `2024-06-09` with zero reasoning characters. The port's suite is green — 1,537 tests, 17 skipped — where the
stock tree has one failure (`test_hip_ordinal`, a test that reads the host's AMD topology; fixed upstream in
[#5602](https://github.com/Niko1221/Strata/pull/5602)).
**Reproduce it.** `dirigo-results/0.2.13-port/scripts/strata-guidellm-0213-ab.sh` runs both arms;
[the report folder](dirigo-results/0.2.13-port/) carries the per-level JSON, the chart, and `make-chart.py`, which
recomputes every figure on this page from those files. Both arms: 0 errored requests.

**What this does not claim.** A faster engine — it is the same speed, and this table would look identical for any
server-side change. A win on conversational traffic, where the fork turns the cache back on and behaves like stock.
A throughput difference at one stream: one extra request in a 45-second window is the measurement's own granularity
(4.8 %), and it is reported rather than counted as a win.
**What it does not show.** A faster engine — parity, measured above. A win on conversations, where the cache comes
back on and the two converge. Anything at eight streams, where both are queue-bound. A throughput win at one stream:
one extra request in a 45-second window is the measurement's own granularity (4.8 %), reported rather than claimed.

**What the engine both of us run is doing.** Underneath the server, the speed is stock's: only the experts a word
needs are computed (the card and the CPU working at the same time), prompts are read in 8,192-token pieces and reused
on a follow-up, and the draft layer guesses up to three tokens for one-pass verification. None of that is ours, and
none of it is changed by the fork — [how it works](docs/HOW_IT_WORKS.md).

> **Measured on:** Intel Arc Pro B70 32 GB, Ryzen 7 7700X, Qwen3.8-Flash-Next Coder IQ1_M, engine 0.1.59 on both
> sides, 2026-11-09. One pass per arm in one session; positional drift in this rig is 2–4 %, so the 10.8–22.1 %
Expand All @@ -115,7 +92,7 @@ A throughput difference at one stream: one extra request in a 45-second window i
| Check my hardware before downloading anything | [What you need](#what-you-need) · [which model fits](docs/MODELS.md#pick-by-memory) | 5 min |
| Understand the engine | [How it works](docs/HOW_IT_WORKS.md) · [technical reference](docs/DETAILS.md) | — |
| Read the evidence for the numbers | [`DIRIGO.md`](DIRIGO.md) · [`dirigo-results/`](dirigo-results/) | — |
| See how it compares to stock 0.2.13 | [Measured side by side](#against-stock-v0213-on-the-same-card) | — |
| See how it compares to stock 0.2.13 | [Measured side by side](#why-this-fork-is-faster-than-stock-strata) | — |
| All guides, page by page | [Documentation index](docs/README.md) | — |

## What you need
Expand Down Expand Up @@ -244,8 +221,8 @@ More: [Troubleshooting](docs/TROUBLESHOOTING.md) · [engine errors](docs/DETAILS

## What this fork changes

The engine, kernels and model format are upstream's. This fork adds server-level behaviour and one engine
change; with none of them enabled the server behaves as stock.
The engine, kernels and model format are upstream's. This fork adds server-level behaviour and one engine change;
with none of them enabled the server behaves as stock.

| Change | What it does |
| --- | --- |
Expand All @@ -254,12 +231,34 @@ change; with none of them enabled the server behaves as stock.
| retune between requests | change engine settings on a live server — no restart, no reload |
| per-request draft depth | a client can set its own speculation depth for one request |

### Around drafting: the same drafter, more control

Both versions run the model's MTP draft layer; nothing in the carry replaces it. What the fork adds is how it can be
set — and `spec_min_p` per request is upstream's, not ours:

| setting | stock v0.2.13 | this fork |
| --- | --- | --- |
| draft depth | `--spec` at startup, for every request | also `spec=N` per request, clamped by the engine to 2…the startup value |
| prompt lookup | `--suffix-draft` at startup | also `lookup=0\|1` per request |
| draft confidence | `spec_min_p` per request via `strata_tune` | the same, and changeable globally on a live server |
| window policy | — | `spec_window`: serve some traffic at a lower depth (0 in the measured config) |

None of it was switched on for the speed numbers above: `spec_window` was 0, and GuideLLM sends no `strata_tune`.

### Two measured differences that are not speed

- **Short answers.** Asked for a date inside a 24-token budget ("Invoice INV-7 dated 9 June 2024 → ISO date"), stock
spends all 24 tokens on reasoning and returns an empty string; the fork returns `2024-06-09` with zero reasoning
characters.
- **A green suite.** The port passes 1,537 tests (17 skipped) where the stock tree has one failure —
`test_hip_ordinal`, a test that reads the host's AMD topology, fixed upstream in
[#5602](https://github.com/Niko1221/Strata/pull/5602).

### What it is worth, measured

The numbers are above, in [Against stock v0.2.13](#against-stock-v0213-on-the-same-card): first token 11–22 % sooner
at one to four streams, engine at parity, and short requests answered that stock truncates. Full report, raw JSON,
the chart and the script that recomputes every figure in it:
[`dirigo-results/0.2.13-port/`](dirigo-results/0.2.13-port/).
The numbers are above, in [Why this fork is faster than stock Strata](#why-this-fork-is-faster-than-stock-strata):
first token 11–22 % sooner at one to four streams, engine at parity. Full report, raw JSON, the chart and the script
that recomputes every figure: [`dirigo-results/0.2.13-port/`](dirigo-results/0.2.13-port/).

The parts that belong upstream are open as separate branches listed in [`DIRIGO.md`](DIRIGO.md).

Expand Down
Loading