From 34b4a9ba0e06f0280f818612e49f770dfb8d23f3 Mon Sep 17 00:00:00 2001 From: rwkeyes <86691029+rwkeyes@users.noreply.github.com> Date: Fri, 13 Nov 2026 12:57:56 -0500 Subject: [PATCH] docs: make "this is a single-user number" part of the claim, not a footnote The 22 % reads as a property of the fork. It is a property of the fork serving one user: add concurrent clients and it falls to 17 %, then 11 %, then nothing at eight, because every arm ran one request at a time. It is now said twice, both above the fold: - a callout directly under the bold claim, before the chart, with the three percentages, the eight-stream tie in milliseconds, the reason (one request at a time) and what to do instead (--batch/--slots); - the chart footer carries it in the image itself, since the chart is what gets screenshotted and reposted: "Single-user figures: both arms ran one request at a time, so the advantage falls to zero at eight concurrent clients, where the two servers are queueing." The eight-stream bullet in the "which row are you" list is shortened so it points at the callout instead of repeating it, and the report's "does not show" list gains the same scaling in words. --- README.md | 14 ++++++++++---- dirigo-results/0.2.13-port/README.md | 6 ++++-- dirigo-results/0.2.13-port/chart.svg | 11 ++++++----- dirigo-results/0.2.13-port/scripts/make-chart.py | 3 ++- 4 files changed, 22 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 2db2be7b6d9..1ed32f5be45 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,12 @@ had only the engine to show, this section would be empty. The difference is in t stream — and at four streams the same 45-second window finishes **three more answers**. Nothing is traded for it: the engine figures are parity, as above. +> **The gain shrinks as more clients share the box.** This is a single-user number: one machine, one conversation. +> The advantage falls from 22 % to 17 % to 11 % as concurrent clients go from one to two to four, and at eight there is +> no difference at all (9,641 ms against 9,659 ms) — the engine serves one request at a time, so beyond one user both +> servers are mostly queueing. If several people or agents will use the machine, size it for that with +> `--batch`/`--slots` ([batching](docs/BATCHING.md)) and treat 22 % as the figure for one user. +

Time to first token, Dirigo 0.2.13 vs stock 0.2.13 on an Intel Arc Pro B70: 739 vs 949 ms at one stream, 2622 vs 3163 at two, 6688 vs 7500 at four, 9641 vs 9659 at eight. Engine-level prefill and decode are within 1.1 percent on all eight legs.

@@ -52,10 +58,10 @@ requests against 21 / 20 / 19 / 21 — 62.6 / 62.6 / 62.6 / 65.4 output tokens a That is where the 22 % applies, and it is most desktop use. - **Two to four** is a few agents or tools sharing the box: an editor's assistant beside a terminal agent, or two people on one machine. 17 % and 11 % still apply. -- **Eight** is a queue: a batch job, an evaluation harness, a team. Both servers ran these tests at `--batch 0` — one - request at a time — so eight clients mostly wait for their turn, and the two lines meet. If you need several - requests actually in flight, the engine takes `--batch`/`--slots` (2–8 in the usual case; `--parallel` in the - Terminal) and the numbers would be a different experiment — [batching](docs/BATCHING.md) covers that configuration. +- **Eight** is a queue: a batch job, an evaluation harness, a team of people. Both servers ran these tests at + `--batch 0` — one request at a time — so eight clients wait their turn and the two lines meet, as the note above + says. For several requests actually in flight, `--batch`/`--slots` (2–8 in the usual case; `--parallel` in the + Terminal) is the setting, and the numbers would be a different experiment. **Where the difference comes from.** A fresh prompt is the one case the two servers handle differently. Stock opens the prompt cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique diff --git a/dirigo-results/0.2.13-port/README.md b/dirigo-results/0.2.13-port/README.md index e0bd0ca74eb..eef5f7b9e40 100644 --- a/dirigo-results/0.2.13-port/README.md +++ b/dirigo-results/0.2.13-port/README.md @@ -176,8 +176,10 @@ have, and is **10.8–22.1 % sooner to first token at 1–4 concurrent streams** the same model. Does not show: anything about other GPUs or other engines; a full statistical treatment of the GuideLLM gap (one pass -per arm); any benefit from 0.2.13's bf16 determinism change, which this hardware did not need; or a speed win for -conversational traffic, where the fork turns the cache back on and behaves like stock. +per arm); any benefit from 0.2.13's bf16 determinism change, which this hardware did not need; a speed win for +conversational traffic, where the fork turns the cache back on and behaves like stock; or a speed win as clients are +added — the advantage falls from 22 % to 17 % to 11 % at one, two and four concurrent clients and is gone at eight, +because every arm here ran one request at a time (`--batch 0`). ## Reproducing diff --git a/dirigo-results/0.2.13-port/chart.svg b/dirigo-results/0.2.13-port/chart.svg index 163dc036611..ba402382fc8 100644 --- a/dirigo-results/0.2.13-port/chart.svg +++ b/dirigo-results/0.2.13-port/chart.svg @@ -1,5 +1,5 @@ - - + + Dirigo 0.2.13 vs stock 0.2.13 — same card, same requests Intel Arc Pro B70 32 GB · engine 0.1.59 on both sides · Qwen3.8-Flash-Next Coder IQ1_M · GuideLLM 0.8.1, 512 in / 128 out, 45 s per point @@ -101,7 +101,8 @@ 23 requests · 65.4 tok/s 21 requests · 59.7 tok/s - -Source: dirigo-results/0.2.13-port/guidellm/guidellm-0213ab-d0213.json and -s0213.json, medians of the per-request samples; battery/*.json. Recompute with scripts/make-chart.py. -One pass per arm in one session; the companion 0.2.12→0.2.13 run put positional drift at 2–4 %. GuideLLM sends unique prompts, so the fork's cache-off path applies — a conversation switches the cache back on. + +Source: dirigo-results/0.2.13-port/guidellm/guidellm-0213ab-d0213.json and -s0213.json, medians of the per-request samples; battery/*.json. Recompute with scripts/make-chart.py. +One pass per arm in one session; the companion 0.2.12→0.2.13 run put positional drift at 2–4 %. GuideLLM sends unique prompts, so the fork's cache-off path applies — a conversation switches the cache back on. +Single-user figures: both arms ran one request at a time, so the advantage falls to zero at eight concurrent clients, where the two servers are queueing. \ No newline at end of file diff --git a/dirigo-results/0.2.13-port/scripts/make-chart.py b/dirigo-results/0.2.13-port/scripts/make-chart.py index aa3b05dc523..9df52a9d4c0 100644 --- a/dirigo-results/0.2.13-port/scripts/make-chart.py +++ b/dirigo-results/0.2.13-port/scripts/make-chart.py @@ -78,7 +78,7 @@ def main(): g = guidellm() b = battery() D, S = g["d0213"], g["s0213"] - W, H = 1180, 1000 + W, H = 1180, 1032 s = [] a = s.append a(f"") @@ -164,6 +164,7 @@ def yof(v): a(f"") a(f"Source: dirigo-results/0.2.13-port/guidellm/guidellm-0213ab-d0213.json and -s0213.json, medians of the per-request samples; battery/*.json. Recompute with scripts/make-chart.py.") a(f"One pass per arm in one session; the companion 0.2.12→0.2.13 run put positional drift at 2–4 %. GuideLLM sends unique prompts, so the fork's cache-off path applies — a conversation switches the cache back on.") + a(f"Single-user figures: both arms ran one request at a time, so the advantage falls to zero at eight concurrent clients, where the two servers are queueing.") a("") out = os.path.join(BENCH, "chart.svg") open(out, "w").write("\n".join(s))