diff --git a/README.md b/README.md index 2db2be7b6d9..1ed32f5be45 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,12 @@ had only the engine to show, this section would be empty. The difference is in t stream — and at four streams the same 45-second window finishes **three more answers**. Nothing is traded for it: the engine figures are parity, as above. +> **The gain shrinks as more clients share the box.** This is a single-user number: one machine, one conversation. +> The advantage falls from 22 % to 17 % to 11 % as concurrent clients go from one to two to four, and at eight there is +> no difference at all (9,641 ms against 9,659 ms) — the engine serves one request at a time, so beyond one user both +> servers are mostly queueing. If several people or agents will use the machine, size it for that with +> `--batch`/`--slots` ([batching](docs/BATCHING.md)) and treat 22 % as the figure for one user. +

Time to first token, Dirigo 0.2.13 vs stock 0.2.13 on an Intel Arc Pro B70: 739 vs 949 ms at one stream, 2622 vs 3163 at two, 6688 vs 7500 at four, 9641 vs 9659 at eight. Engine-level prefill and decode are within 1.1 percent on all eight legs.

@@ -52,10 +58,10 @@ requests against 21 / 20 / 19 / 21 — 62.6 / 62.6 / 62.6 / 65.4 output tokens a That is where the 22 % applies, and it is most desktop use. - **Two to four** is a few agents or tools sharing the box: an editor's assistant beside a terminal agent, or two people on one machine. 17 % and 11 % still apply. -- **Eight** is a queue: a batch job, an evaluation harness, a team. Both servers ran these tests at `--batch 0` — one - request at a time — so eight clients mostly wait for their turn, and the two lines meet. If you need several - requests actually in flight, the engine takes `--batch`/`--slots` (2–8 in the usual case; `--parallel` in the - Terminal) and the numbers would be a different experiment — [batching](docs/BATCHING.md) covers that configuration. +- **Eight** is a queue: a batch job, an evaluation harness, a team of people. Both servers ran these tests at + `--batch 0` — one request at a time — so eight clients wait their turn and the two lines meet, as the note above + says. For several requests actually in flight, `--batch`/`--slots` (2–8 in the usual case; `--parallel` in the + Terminal) is the setting, and the numbers would be a different experiment. **Where the difference comes from.** A fresh prompt is the one case the two servers handle differently. Stock opens the prompt cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique diff --git a/dirigo-results/0.2.13-port/README.md b/dirigo-results/0.2.13-port/README.md index e0bd0ca74eb..eef5f7b9e40 100644 --- a/dirigo-results/0.2.13-port/README.md +++ b/dirigo-results/0.2.13-port/README.md @@ -176,8 +176,10 @@ have, and is **10.8–22.1 % sooner to first token at 1–4 concurrent streams** the same model. Does not show: anything about other GPUs or other engines; a full statistical treatment of the GuideLLM gap (one pass -per arm); any benefit from 0.2.13's bf16 determinism change, which this hardware did not need; or a speed win for -conversational traffic, where the fork turns the cache back on and behaves like stock. +per arm); any benefit from 0.2.13's bf16 determinism change, which this hardware did not need; a speed win for +conversational traffic, where the fork turns the cache back on and behaves like stock; or a speed win as clients are +added — the advantage falls from 22 % to 17 % to 11 % at one, two and four concurrent clients and is gone at eight, +because every arm here ran one request at a time (`--batch 0`). ## Reproducing diff --git a/dirigo-results/0.2.13-port/chart.svg b/dirigo-results/0.2.13-port/chart.svg index 163dc036611..ba402382fc8 100644 --- a/dirigo-results/0.2.13-port/chart.svg +++ b/dirigo-results/0.2.13-port/chart.svg @@ -1,5 +1,5 @@ - - + + Dirigo 0.2.13 vs stock 0.2.13 — same card, same requests Intel Arc Pro B70 32 GB · engine 0.1.59 on both sides · Qwen3.8-Flash-Next Coder IQ1_M · GuideLLM 0.8.1, 512 in / 128 out, 45 s per point @@ -101,7 +101,8 @@ 23 requests · 65.4 tok/s 21 requests · 59.7 tok/s - -Source: dirigo-results/0.2.13-port/guidellm/guidellm-0213ab-d0213.json and -s0213.json, medians of the per-request samples; battery/*.json. Recompute with scripts/make-chart.py. -One pass per arm in one session; the companion 0.2.12→0.2.13 run put positional drift at 2–4 %. GuideLLM sends unique prompts, so the fork's cache-off path applies — a conversation switches the cache back on. + +Source: dirigo-results/0.2.13-port/guidellm/guidellm-0213ab-d0213.json and -s0213.json, medians of the per-request samples; battery/*.json. Recompute with scripts/make-chart.py. +One pass per arm in one session; the companion 0.2.12→0.2.13 run put positional drift at 2–4 %. GuideLLM sends unique prompts, so the fork's cache-off path applies — a conversation switches the cache back on. +Single-user figures: both arms ran one request at a time, so the advantage falls to zero at eight concurrent clients, where the two servers are queueing. \ No newline at end of file diff --git a/dirigo-results/0.2.13-port/scripts/make-chart.py b/dirigo-results/0.2.13-port/scripts/make-chart.py index aa3b05dc523..9df52a9d4c0 100644 --- a/dirigo-results/0.2.13-port/scripts/make-chart.py +++ b/dirigo-results/0.2.13-port/scripts/make-chart.py @@ -78,7 +78,7 @@ def main(): g = guidellm() b = battery() D, S = g["d0213"], g["s0213"] - W, H = 1180, 1000 + W, H = 1180, 1032 s = [] a = s.append a(f"") @@ -164,6 +164,7 @@ def yof(v): a(f"") a(f"Source: dirigo-results/0.2.13-port/guidellm/guidellm-0213ab-d0213.json and -s0213.json, medians of the per-request samples; battery/*.json. Recompute with scripts/make-chart.py.") a(f"One pass per arm in one session; the companion 0.2.12→0.2.13 run put positional drift at 2–4 %. GuideLLM sends unique prompts, so the fork's cache-off path applies — a conversation switches the cache back on.") + a(f"Single-user figures: both arms ran one request at a time, so the advantage falls to zero at eight concurrent clients, where the two servers are queueing.") a("") out = os.path.join(BENCH, "chart.svg") open(out, "w").write("\n".join(s))