Strata is a local AI engine. It runs a very large language model on one ordinary PC by sharing the work
between the graphics card, the CPU, the RAM and the SSD — no account, no cloud. It speaks the OpenAI and
Anthropic APIs on localhost, so the coding tools you already use can talk to it.
This repository is Dirigo Agents' fork of Niko1221/Strata (MIT).
It is source only: no installers, no engine binaries, no model files. Upstream's own README is preserved at
docs/UPSTREAM-README.md.
Strata running a 24,576-expert model locally, on one graphics card.
The default branch, dirigo/0.2.12-verified, is one snapshot: upstream tag v0.2.12 (commit 146df3c4,
engine 0.1.58) plus this fork's changes, in the state that served a live machine on 2026-11-08 — with the
measurements behind it kept in dirigo-results/. What was verified, and how:
DIRIGO.md.
"Stock" here is Niko1221's v0.2.13 — the upstream release — built from the same tag and run on the same card, model, pack, config and request stream as this fork. Everything below is measured on both sides, not estimated.
First, the part that is not faster. Both trees build engine 0.1.59, and the engine-level battery — eight legs, prefill and decode, straight to the engine, two interleaved passes — lands within 1.1 % of each other. If the fork had only the engine to show, this section would be empty. The difference is in the server:
The first token arrives 11–22 % sooner on a stream of fresh prompts — 739 ms against stock's 949 ms at one stream — and at four streams the same 45-second window finishes three more answers. Nothing is traded for it: the engine figures are parity, as above.
The gain shrinks as more clients share the box. This is a single-user number: one machine, one conversation. The advantage falls from 22 % to 17 % to 11 % as concurrent clients go from one to two to four, and at eight there is no difference at all (9,641 ms against 9,659 ms) — the engine serves one request at a time, so beyond one user both servers are mostly queueing. If several people or agents will use the machine, size it for that with
--batch/--slots(batching) and treat 22 % as the figure for one user.
| streams | first token, this fork | stock v0.2.13 | arrives sooner by | read the other way: stock needs |
|---|---|---|---|---|
| 1 | 739 ms | 949 ms | 210 ms (22.1 %) | 28.4 % longer |
| 2 | 2,622 ms | 3,163 ms | 541 ms (17.1 %) | 20.6 % longer |
| 4 | 6,688 ms | 7,500 ms | 812 ms (10.8 %) | 12.1 % longer |
| 8 | 9,641 ms | 9,659 ms | 18 ms — equal | equal |
The last two columns are one measurement read two ways: 739 ms is 22.1 % below 949 ms, and 949 ms is 28.4 % above 739 ms. "Arrives sooner" is the column to quote. In the same 45-second window the fork finishes 22 / 22 / 22 / 23 requests against 21 / 20 / 19 / 21 — 62.6 / 62.6 / 62.6 / 65.4 output tokens a second against 59.7 / 56.9 / 54.0 / 59.7.
Which of those rows are you? GuideLLM's "streams" are concurrent clients against one server.
- One stream is the ordinary case: one person at one keyboard, or one agent loop — ask, read the answer, ask again. That is where the 22 % applies, and it is most desktop use.
- Two to four is a few agents or tools sharing the box: an editor's assistant beside a terminal agent, or two people on one machine. 17 % and 11 % still apply.
- Eight is a queue: a batch job, an evaluation harness, a team of people. Both servers ran these tests at
--batch 0— one request at a time — so eight clients wait their turn and the two lines meet, as the note above says. For several requests actually in flight,--batch/--slots(2–8 in the usual case;--parallelin the Terminal) is the setting, and the numbers would be a different experiment.
Where the difference comes from. A fresh prompt is the one case the two servers handle differently. Stock opens the prompt cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique prompts as one-shot traffic and serves it with the cache off — no pool work between the request and the first token — then switches the cache back on the moment the traffic looks like a conversation. That is why the advantage is largest with requests arriving one at a time, and gone at eight streams where both servers are queue-bound.
A control, so the mechanism is not taken on faith. The same router was measured in both positions on one build:
with conv_route: shadow — classify and log, never switch — it lands on the stock number, 934 ms at one stream, and
with conv_route: on it lands on 727 ms. Same build, same card, same workload; the switch is what moves the first
token (the pair). The two arms of the comparison above also ran one config file each, identical apart
from paths, and the same model file for the 28.8 GB lookup table.
Reproduce it. dirigo-results/0.2.13-port/scripts/strata-guidellm-0213-ab.sh runs both arms;
the report folder carries the per-level JSON, the chart, and make-chart.py, which
recomputes every figure on this page from those files. Both arms: 0 errored requests.
What it does not show. A faster engine — parity, measured above. A win on conversations, where the cache comes
back on and the two converge. Anything about multi-slot serving: the tests ran --batch 0, one request at a time. A
throughput win at one stream: one extra request in a 45-second window is the measurement's own granularity (4.8 %),
reported rather than claimed.
What the engine both of us run is doing. Underneath the server, the speed is stock's: only the experts a word needs are computed (the card and the CPU working at the same time), prompts are read in 8,192-token pieces and reused on a follow-up, and the draft layer guesses up to three tokens for one-pass verification. None of that is ours, and none of it is changed by the fork — how it works.
Measured on: Intel Arc Pro B70 32 GB, Ryzen 7 7700X, Qwen3.8-Flash-Next Coder IQ1_M, engine 0.1.59 on both sides, 2026-11-09. One pass per arm in one session; positional drift in this rig is 2–4 %, so the 10.8–22.1 % first-token gaps sit outside it. Errors: 0 in both arms. Full report, raw JSON and the script that recomputes every figure:
dirigo-results/0.2.13-port/.
Never used Strata before? Start with Getting started — it walks from "what is this" to your first answer, and explains every word Strata uses in plain language.
Just want to run Strata, not build it? Use the stock release instead — one line in the terminal and no compiler: Download the release · install guide.
| I want to… | Go to | Time |
|---|---|---|
| Run Strata on my own PC | Install (stock release) — installer, or one line in the terminal | 10 min + model download |
| Reproduce the build this fork verified | Quick start below · evidence | 30–60 min of compiling |
| Connect Claude Code, Codex, OpenCode, Hermes… | strata connect, or tool connections |
5 min |
| Check my hardware before downloading anything | What you need · which model fits | 5 min |
| Understand the engine | How it works · technical reference | — |
| Read the evidence for the numbers | DIRIGO.md · dirigo-results/ |
— |
| See how it compares to stock 0.2.13 | Measured side by side | — |
| All guides, page by page | Documentation index | — |
Strata is not a small-model tool. The smallest recommended Qwen3.8-Flash-Next pack needs about 37.6 GB of memory (RAM plus VRAM); the coding-only pack runs in about 32 GB. Unified memory (Apple Silicon, Strix Halo) counts once. Check which model fits before you download tens of gigabytes.
| Requirement | |
|---|---|
| Graphics card | NVIDIA RTX 20/30/40/50 · AMD Radeon RX 7000/9000 and AI PRO R9700 (RX 6000 experimental) · Intel Arc on Linux · Apple Silicon |
| Memory | 38–55 GB of RAM + VRAM for the Qwen3.8-Flash-Next packs (37.6 GB for the smallest), with room left for the OS; 46–64 GB of system RAM plus a 12 GB card for DeepSeek V4.1-Flash |
| Disk | SSD: 66–84 GB for Qwen3.8-Flash-Next — the pack plus the model's 28.8 GB lookup table — and 239–274 GB for DeepSeek V4.1-Flash |
| To build from source | Python 3, CMake 3.24+, Ninja, a C++20 compiler, and your GPU's toolkit (CUDA, ROCm, oneAPI or Xcode) |
The 28.8 GB in that disk figure is the model's lookup table, not the engine's: it is the publisher's second file, read from the SSD a few rows at a time instead of being held in memory. It belongs to every pack of this model, stock or fork alike — what it is and where the size comes from.
The card-specific details are in NVIDIA, AMD, Intel Arc, Mac, multiple GPUs.
This is the build the branch was verified with: Intel Arc, SYCL AOT bmg-g31, Intel oneAPI 2026.2. AMD and
NVIDIA build from the same sources with their own guides below.
# 1. source (the default branch is the verified snapshot)
git clone --branch dirigo/0.2.12-verified https://github.com/rwkeyes/Strata.git
cd Strata
# 2. Python side
python3 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
# 3. build — needs oneAPI's icpx and oneMKL on PATH: source /opt/intel/oneapi/setvars.sh
cmake -S sycl -B build-sycl-aot -DSTRATA_SYCL_AOT=bmg-g31
cmake --build build-sycl-aot --target strata -j"$(nproc)"
# 4. prove what you built before going further
./build-sycl-aot/strata --version # must print: strata 0.1.58The engine reads a pack (a prepared model folder) plus the model's GGUF shards. Build the pack from the GGUF files you downloaded, once per model:
python tools/iq_pack.py \
--gguf /path/to/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf \
--out packs/iq3xxsThen serve. These are the settings the measured service ran with — replace the paths, and keep
data/expert-profile*.bin where the binary can find it. The profile is matched to the model's expert count, not its
quantization: the full model (IQ2/IQ3/Q2 packs, 48×512) takes data/expert-profile.bin; the pruned Coder model
(48×256) takes data/expert-profile-coder.bin. The wrong one is refused at start with
it is a profile for a different model:
./build-sycl-aot/strata --serve \
--pack packs/iq3xxs --native <shard-1.gguf> --ple-gguf <shard-2.gguf> \
--expert-profile data/expert-profile.bin --expert-cache auto --stream-experts --prefill auto \
--prompt-cache 6 --prompt-cache-every 4096 --batch 0 --spec 6 --spec-min-p 0.5 \
--mtp <draft> --mtp-draft-vocab data/draft_vocab_en.bin --mtp-window 8192 \
--max-context 131072 --kv int8 --kv-resident 32768 --vram-reserve-mib 700--serve is the engine's own token protocol. For a browser or an OpenAI-compatible API, run the bundled
server with those arguments in its config (args), and check it answers:
.venv/bin/python -m serve.server --engine strata --config strata.json --port 8080
curl -s http://127.0.0.1:8080/v1/modelsOther builds: AMD / HIP · NVIDIA / CUDA ·
Mac / Metal · the guided ./setup.sh route.
If you want Strata running rather than built, the released app and terminal do the engine choice, download and model preparation for you:
# Linux / Mac
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
strata launch# Windows PowerShell, then open a new terminal
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iexstrata launch --plan prints what would be downloaded and how much memory it needs, and changes nothing.
Full install guide · Terminal launch.
| Word | Plain meaning |
|---|---|
| pack | a prepared model folder the engine reads |
| shard | one part of a model file split across several downloads |
| quantization (Q2_0, IQ3_XXS, IQ1_M) | how much the model was compressed — smaller is faster and lighter, larger is better |
| expert | one small specialist network inside the model; only a few are needed per word |
| PLE table | the big lookup table Strata reads from your SSD instead of holding in memory |
| KV cache | what the model remembers about the conversation; costs memory per token of context |
| slot | one conversation being served at a time (the engine serves one at a time by design) |
| prefill / decode | reading your prompt / writing the answer |
| TTFT | time to first token — how long before the answer starts |
| spec / MTP | the helper that guesses the next words so the big model can check several at once |
| TUNE | the fork's channel for changing engine settings between requests, without a restart |
Full list, including the fork-specific terms: Glossary.
| Symptom | What to do |
|---|---|
strata is not found |
Open a new terminal; on Linux/Mac add ~/.local/bin to PATH |
| "Couldn't find an engine for this computer" | strata update, then strata doctor --live (Troubleshooting) |
Build stops at icpx: not found |
source /opt/intel/oneapi/setvars.sh before cmake |
| Engine exits before it is ready, after a source build | The expert profile is missing — data/expert-profile.bin must be beside the binary's working directory |
| It used the wrong graphics card | strata-device --list-devices; pick one with --device or ROCR_VISIBLE_DEVICES / CUDA_VISIBLE_DEVICES |
| Loads, then runs out of memory | Smaller pack, lower --max-context, or lower --kv-resident |
| Very slow first answer | First start warms the file cache; measure the second request |
More: Troubleshooting · engine errors.
The engine, kernels and model format are upstream's. This fork adds server-level behaviour and one engine change; with none of them enabled the server behaves as stock.
| Change | What it does |
|---|---|
| conversation-cache routing | turns the prompt cache off after four single-shot prompts in a row; one conversational turn switches it back on. Worth −12…−22 % first-token time at 1–4 clients and nothing at 8, and nothing on a conversation — and a threshold of 1 costs 1.9 s per conversation (why) |
| thinking policy | skips the scratchpad for short, factual asks; keeps it when the answer needs work |
| retune between requests | change engine settings on a live server — no restart, no reload |
| per-request draft depth | a client can set its own speculation depth for one request |
Both versions run the model's MTP draft layer; nothing in the carry replaces it. What the fork adds is how it can be
set — and spec_min_p per request is upstream's, not ours:
| setting | stock v0.2.13 | this fork |
|---|---|---|
| draft depth | --spec at startup, for every request |
also spec=N per request, clamped by the engine to 2…the startup value |
| prompt lookup | --suffix-draft at startup |
also lookup=0|1 per request |
| draft confidence | spec_min_p per request via strata_tune |
the same, and changeable globally on a live server |
| window policy | — | spec_window: serve some traffic at a lower depth (0 in the measured config) |
None of it was switched on for the speed numbers above: spec_window was 0, and GuideLLM sends no strata_tune.
- Short answers. Asked for a date inside a 24-token budget ("Invoice INV-7 dated 9 June 2024 → ISO date"), stock
spends all 24 tokens on reasoning and returns an empty string; the fork returns
2024-06-09with zero reasoning characters. - A green suite. The port passes 1,537 tests (17 skipped) where the stock tree has one failure —
test_hip_ordinal, a test that reads the host's AMD topology, fixed upstream in #5602.
The numbers are above, in Why this fork is faster than stock Strata:
first token 11–22 % sooner at one to four streams, engine at parity. Full report, raw JSON, the chart and the script
that recomputes every figure: dirigo-results/0.2.13-port/.
The parts that belong upstream are open as separate branches listed in DIRIGO.md.
| Path | What lives there |
|---|---|
sycl/ |
the Intel Arc build of the engine |
src/, include/ |
the engine core (CUDA, and the shared C++ sources) |
metal/ |
the Apple Silicon backend |
serve/ |
the OpenAI / Anthropic compatible server, model preparation, monitoring |
strata_cli/, strata_control/ |
the strata terminal command |
apps/ |
the desktop app |
tools/ |
pack building, calibration and measurement tools |
bench/ |
benchmark protocol and community results |
docs/ |
all guides — index |
dirigo-results/ |
this fork's measurements: the 0.2.12 evidence, the 0.2.13 fork-vs-stock report and chart, the carry patch |
MIT — upstream Strata plus the Dirigo changes; see LICENSE. Model files are not part of this
repository and carry their own licences.
The translated READMEs (README.de.md, README.es.md, README.fr.md, README.ja.md, README.pl.md,
README.pt-BR.md, README.zh-CN.md) are upstream's and describe stock Strata, not this fork.