Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ had only the engine to show, this section would be empty. The difference is in t
stream — and at four streams the same 45-second window finishes **three more answers**. Nothing is traded for it: the
engine figures are parity, as above.

> **The gain shrinks as more clients share the box.** This is a single-user number: one machine, one conversation.
> The advantage falls from 22 % to 17 % to 11 % as concurrent clients go from one to two to four, and at eight there is
> no difference at all (9,641 ms against 9,659 ms) — the engine serves one request at a time, so beyond one user both
> servers are mostly queueing. If several people or agents will use the machine, size it for that with
> `--batch`/`--slots` ([batching](docs/BATCHING.md)) and treat 22 % as the figure for one user.

<p align="center">
<img src="dirigo-results/0.2.13-port/chart.svg" width="1000" alt="Time to first token, Dirigo 0.2.13 vs stock 0.2.13 on an Intel Arc Pro B70: 739 vs 949 ms at one stream, 2622 vs 3163 at two, 6688 vs 7500 at four, 9641 vs 9659 at eight. Engine-level prefill and decode are within 1.1 percent on all eight legs.">
</p>
Expand All @@ -52,10 +58,10 @@ requests against 21 / 20 / 19 / 21 — 62.6 / 62.6 / 62.6 / 65.4 output tokens a
That is where the 22 % applies, and it is most desktop use.
- **Two to four** is a few agents or tools sharing the box: an editor's assistant beside a terminal agent, or two
people on one machine. 17 % and 11 % still apply.
- **Eight** is a queue: a batch job, an evaluation harness, a team. Both servers ran these tests at `--batch 0` — one
request at a time — so eight clients mostly wait for their turn, and the two lines meet. If you need several
requests actually in flight, the engine takes `--batch`/`--slots` (2–8 in the usual case; `--parallel` in the
Terminal) and the numbers would be a different experiment — [batching](docs/BATCHING.md) covers that configuration.
- **Eight** is a queue: a batch job, an evaluation harness, a team of people. Both servers ran these tests at
`--batch 0` — one request at a time — so eight clients wait their turn and the two lines meet, as the note above
says. For several requests actually in flight, `--batch`/`--slots` (2–8 in the usual case; `--parallel` in the
Terminal) is the setting, and the numbers would be a different experiment.

**Where the difference comes from.** A fresh prompt is the one case the two servers handle differently. Stock opens
the prompt cache and pays the pool work before it can emit a token. The fork's router recognises a run of unique
Expand Down
6 changes: 4 additions & 2 deletions dirigo-results/0.2.13-port/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,8 +176,10 @@ have, and is **10.8–22.1 % sooner to first token at 1–4 concurrent streams**
the same model.

Does not show: anything about other GPUs or other engines; a full statistical treatment of the GuideLLM gap (one pass
per arm); any benefit from 0.2.13's bf16 determinism change, which this hardware did not need; or a speed win for
conversational traffic, where the fork turns the cache back on and behaves like stock.
per arm); any benefit from 0.2.13's bf16 determinism change, which this hardware did not need; a speed win for
conversational traffic, where the fork turns the cache back on and behaves like stock; or a speed win as clients are
added — the advantage falls from 22 % to 17 % to 11 % at one, two and four concurrent clients and is gone at eight,
because every arm here ran one request at a time (`--batch 0`).

## Reproducing

Expand Down
11 changes: 6 additions & 5 deletions dirigo-results/0.2.13-port/chart.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion dirigo-results/0.2.13-port/scripts/make-chart.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ def main():
g = guidellm()
b = battery()
D, S = g["d0213"], g["s0213"]
W, H = 1180, 1000
W, H = 1180, 1032
s = []
a = s.append
a(f"<svg xmlns='http://www.w3.org/2000/svg' width='{W}' height='{H}' viewBox='0 0 {W} {H}'>")
Expand Down Expand Up @@ -164,6 +164,7 @@ def yof(v):
a(f"<line x1='34' y1='{fy-18}' x2='{W-34}' y2='{fy-18}' stroke='{GRID}'/>")
a(f"<text x='34' y='{fy}' font-size='11.5' fill='{MUTED}'>Source: dirigo-results/0.2.13-port/guidellm/guidellm-0213ab-d0213.json and -s0213.json, medians of the per-request samples; battery/*.json. Recompute with scripts/make-chart.py.</text>")
a(f"<text x='34' y='{fy+17}' font-size='11.5' fill='{MUTED}'>One pass per arm in one session; the companion 0.2.12→0.2.13 run put positional drift at 2–4 %. GuideLLM sends unique prompts, so the fork's cache-off path applies — a conversation switches the cache back on.</text>")
a(f"<text x='34' y='{fy+34}' font-size='11.5' font-weight='600' fill='{INK}'>Single-user figures: both arms ran one request at a time, so the advantage falls to zero at eight concurrent clients, where the two servers are queueing.</text>")
a("</g></svg>")
out = os.path.join(BENCH, "chart.svg")
open(out, "w").write("\n".join(s))
Expand Down
Loading