[BobClawblaw]

What

--drafter for the llama family: a smaller Llama on the same tokenizer proposes a stream's next tokens, and the served model verifies them in its windows, so replies keep the model's bits (builds on #1415). It was brought up with Nanbeige/Nanbeige4.1-3B drafting for Nanbeige4.3-33B, both served from their bf16 releases as published. The two share the tokenizer exactly (166,100 ids, merges, added tokens, bos and eos); only their vocab_sizes pad differently. The 3B is admitted as a drafter only: serving it on its own is refused by name. --drafter-bits 4 and quantized drafter checkpoints are refused by name too: the drafter is the bf16 release, nothing prepared at load.

How

zig/src/families/llama/drafter.zig is the lane host's priced copy proposer (LaneHost.proposers). A round asks it for up to 15 tokens after the stream's context. The window verifies them row by row and keeps the prefix the model's own draws agree with, as it keeps any copy draft. There is no lane-core change and no new kernel: the drafter runs the family's own forward on the served model's queue (Model.loadOn) and in its residency set.

• One budget. The streams' caches and the drafter's share one ledger (budget.Ledger). A stream's admission (budget.admitStream) checks its peak first (its caches, its drafter cache's growth and the old state's copy), kept drafter caches shedding first; trims the kept prompt states to what that peak leaves, freeing a restore entry that can't stay beside it; and only then allocates. A freed drafter cache keeps its tokens for the next request sharing its prefix, counted, and gives way first.
• Drafts. Greedy steps chained on the GPU through token slots (a step's argmax is the next step's row), so one proposal is one chain of command buffers. A chain stops at an eos.
• Depth. It follows measured rates: the stream's decayed acceptance, a chained step's wall time, and a line fitted to the round's time by its window's rows. A stream alone drafts the depth with the most expected tokens a millisecond; with several streams in a round, each drafts the depth with the most tokens a millisecond for the whole round (pricing.depthShared), nothing only where drafting costs the round more than it lands.
• Start and stop. Drafter.start owns the model from the call and frees the drafter if its warm-up fails; the host frees the drafter before the backend whose ledger it gives back to.
• Check mode. tf-llama-run <model> <tokens> draft <drafter>,<n> decodes greedily through the drafter's windows and prints the tokens with the rounds and drafts kept.

Measured (M5 Ultra, the bf16 33B and the bf16 3B, 28 Oct 2026, one model process on the Mac)

Identity. On eight chat prompts (128 tokens, thinking on and off, English and Chinese) and after a 1,500-token prompt (256 tokens), the drafted greedy tokens equal the plain greedy tokens exactly: 21.1-40.7 tok/s drafted against 15.8-16.1 plain, 1.73-4.54 tokens landing a round, a chained bf16 step 8.5-8.9 ms. The recipe doc has each prompt.

Two streams. On fresh servers (--context 32768 --parallel 2 --prompt-cache-gib 8), eight new prompts sent two at a time (greedy, 256 tokens, thinking off) decode together at 30.1-40.9 tok/s (36.1 on average) with --drafter, against 26.6-30.1 (29.0) without; every reply's token ids are equal. Streams joining, leaving, queued, refused (HTTP 400) and cancelled each return the ids of their plain solo run, a reply alone after a cancel is unchanged, and shared rounds keep drafting (199 drafts offered, 125 kept).

Receipt

• Environment: TensorFold at this branch (main 01e25d3 plus these commits), Zig 0.17.0, one Apple M5 Ultra (256 GB unified, macOS 27, tensor units). Nanbeige/Nanbeige4.3-33B @ c1093c501cc6460427c0ac634db7223d0450c0ee with --drafter Nanbeige/Nanbeige4.1-3B @ 9c4555c37921c0982af8fffeee1e0a6e6b953c4f (--drafter-bits 0), LLAMA_LOAD_LIMIT_GB=150.
• Tests run: On these commits over main ea01ae1: zig build test and test-kernel-contract exit 0 on llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415's head; on llama: a smaller Llama drafting for the served one (--drafter), replies unchanged (builds on #1415) #1417's head, which carries llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0; zig fmt --check is clean, the lean check finds 0 new problems (426 on main and on each head), llama_teacher_judge_test.py passes, and tf-kernel-contract --only llama on the M5 Ultra gives SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. Rebased onto main 01e25d3 for this push (one conflict, in docs/kernel-contract.md: main's row-split seam paragraph and this family's, both kept; the family's own files are unchanged): on llama: a smaller Llama drafting for the served one (--drafter), replies unchanged (builds on #1415) #1417's head zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, check-release-paths and the security tests exit 0, fmt is clean and lean finds 0 new; on llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415's head zig build test and test-kernel-contract exit 0. Five greedy chats (1,102 tokens) are byte-identical between the previous head and this one, both plain and with --drafter.
• Not run: the harness quality suite; the harness's serving plans with the bf16 drafter (they passed with an earlier 4-bit drafter, whose drafts change no reply either); CUDA, HIP, M1-M4.

Checks

zig build test covers the depth rule and its shared-round pricing, the cost fit, the acceptance history, the drafter role, the roll call, the one budget (two drafted 240,000-token requests that each fit a 100 GiB budget alone are refused together), admission's order with an allocation recorder (kept prompt states trimmed before anything is made, a failed growth giving every byte back) and a failed warm-up freeing the drafter. Without --drafter, the served path is #1415's: the 1,500-position logits are bit-identical to its judged run.

[ashhart]

Drafted equals plain per the receipts, and there's no lane-core change or new kernel. That's good. Two things:

• The default --drafter-bits 4 refuses a bf16 drafter and needs a llama_convert.py --bits 4 conversion, so the default depends on llama: published MLX affine quants (8-bit and 4-bit) read in place, beside the bf16 default (builds on #1415) #1416. Please make bf16 the default, or accept MLX's layout once llama: published MLX affine quants (8-bit and 4-bit) read in place, beside the bf16 default (builds on #1415) #1416 does.
• Shared rounds. With the drafter, two-stream greedy drops from 31.0 to 27.8 tok/s (chat) and from 34.1 to 29.5 (code) on an M5 Ultra by the PR's own numbers. Yet the recipe keeps the drafter on. Either skip drafts in shared rounds when they cost more than they land, or don't recommend the drafter above --parallel 1.

[BobClawblaw]

Thanks. Pushed 5d4a3af (one commit, on #1416's new head):

--drafter-bits 4. We took the second route: with #1416 reading MLX's layout in place, the default now serves a published MLX 4-bit drafter as it is on the Hub, mlx-community/Nanbeige4.1-3B-4bit @ 4d6c41d88f1416777bbda005aad92798e56e5ba9. There is no conversion and nothing depends on llama_convert.py, which is gone. --drafter-bits 0 serves the bf16 release. With the published 4-bit drafter, drafted greedy tokens equal plain greedy tokens on all nine check prompts, at 23.2-46.4 tok/s against 16.0-16.3. The bf16 drafter gives the same tokens on two of them, at 8.5 ms a step against 4.0-4.2 ms.

Shared rounds. Both of your options, in turn:

1. While more than one stream holds a drafter cache, the drafter proposes nothing (Drafter.sharing), so shared rounds decode undrafted. On fresh servers (bench_concurrent.py, 128 greedy tokens, 8 runs each), two drafted streams give 31.2-31.3 tok/s together on chat against 31.3 without --drafter, and 31.4-31.6 on code against 34.4. On the harness ruler they give 31.9-32.5 against 28.0-31.8.
2. Code is still 8% behind, because a server without --drafter keeps the lane core's own copy drafts and shared rounds give those up. Getting them back would mean copying the lane core's copy policy into the family, so instead the recipe no longer recommends the drafter above --parallel 1. Its drafted command is now --parallel 1, and the doc says a server for several streams leaves the drafter off. The harness's quick and speed plans pass on that server (one stream 99.5-116.8 tok/s on the ruler).

One correction to the old receipt: the 87-117 tok/s single-stream figures from bench_openai.py and bench_concurrent.py --alone overstate ordinary text. Those tools send the same prompt more than once, and a freed drafter cache keeps its tokens for a request sharing its prefix, so a greedy repeat drafts the earlier reply back whole. On a fresh server one stream gives 30.0-32.8 tok/s on bench_concurrent.py's code and 24.9-25.3 on its chat, against 20.2 and 16.4. The PR body now quotes those.

Also: the 3B is admitted as a drafter only (config.zig's qualified list has a role). Serving it on its own is refused by name.

On these commits over main a1e6d34: zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security, test-pull-security all exit 0; zig fmt --check is clean; the lean check finds 0 new problems (428 on main and on the head); tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 23 kernels: 23 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. After the rebase, the Fibonacci prompt's drafted tokens still equal the plain ones. Main then took MOSS-Image (#1457), DFlash2 and MTP fixes for other families, a GLM CUDA log message and an opt-in prompt-cache rule for CUDA families (off by default): none of it in this family's files or the paths it serves through. Rebased onto main 15a0b2a for this push, the stack's top head (#1417's, which carries these commits) reran the build, tf-kernel-contract --only llama, test-kernel-contract, zig fmt --check and the bf16, quant and drafted bit-identity checks, and all pass.

[ashhart]

Thanks @BobClawblaw. The drafter's default now reads the published 4-bit layout, or bf16 with an explicit width of 0 (zig/src/native/llama.zig:158), and the recipe pins both checkpoints.

Three asks:

1. One memory budget. Startup counts both drafter caches (zig/src/native/llama.zig:94), then hands the whole remaining budget to the target (:110). Runtime admission doesn't count the drafter's live and kept caches, its growth copies or kept prompt states. On the production formulas, with a 100 GiB cache budget, context 262,144 and two streams, startup passes, yet two admissible 240,000-token requests together exceed the budget by 25.3 GiB. Please admit the target and the drafter against one budget, and add a host test where two requests that each fit alone are refused together.
2. Shared rounds. Thanks for taking up the shared-round point. Our suggestion was to skip a shared round's drafts when they cost more than they land; the head skips them in every shared round (zig/src/families/llama/drafter.zig:141, :345), so drafting stops as soon as a second stream joins. Our recipe rules count that as a drafts-off fallback, which is a bug for an advertised drafted path (docs/recipes/SKILL.md, the concurrency rules). Please keep drafting on for every admitted stream and price each shared round, then check drafted against plain and concurrent against solo as streams join, leave, are refused and are cancelled.
3. Warm-up failure leaks the drafter. If warm-up fails (zig/src/native/llama.zig:117), the drafter made just before it is never freed: its errdefer comes after, at :121. Please register the cleanup before the warm-up, and add a test that fails it.

[BobClawblaw]

Thanks. Pushed b652456 (one commit on #1416's new head):

1. One memory budget. The backend's admission now runs on one ledger (budget.Ledger) for every cache the load holds. That covers the streams' caches, the drafter's slots (taken or kept) and a slot's growth copy while it lives. When a stream is admitted (backend.prefillFn), its drafter slot is made or grown to the stream's capacity in the same check (Drafter.slotsHook: growth, prepare, shed). If the two don't fit beside what is held, kept drafter slots give way first; if they still don't fit, the stream waits (error.LaneMemory) until another finishes. Slots no longer grow during decoding, so nothing allocates outside admission. Kept prompt states fit in what the ledger leaves. The host test (llama budget: two drafted requests that each fit alone are refused together, and pass in turn) takes the 33B's and the 3B's configs: a 100 GiB budget, a 262,144-token context and two streams pass startup. One 240,000-token request fits, the second waits (together they would pass the budget by 24.5 GiB on these formulas), and after the first ends the second fits beside the first's kept slot. growth has its own test (the new cache plus the old state's copy, nothing when the slot holds enough).

2. Shared rounds are priced, and drafting stays on. pricing.depthShared gives each stream in a shared round the depth with the most tokens a millisecond for the whole round. Each stream's drafts land 1 + p + ... + p^k tokens, for a window of all the streams' rows plus every stream's chained steps, against the round undrafted. It returns 0 only for a round where drafting costs more than it lands. Shared rounds don't feed the fitted line of a round's time by its rows. Checked on a drafted --parallel 2 server (--context 32768), greedy, token ids from return_token_ids:

• A drafts alone; B joins after A's first 40 tokens (shared rounds); C arrives while two streams hold the server and queues; D starts and its client disconnects after 20 tokens (cancelled); E sends a prompt past the context and is refused with HTTP 400. A, B and C each return the token ids of their plain solo run ("draft": false), and A alone after the cancel is unchanged.
• Two streams from the first token to the last: both equal their plain solo runs, and the shared rounds offered 189 drafts and kept 127.
• Speed on fresh servers, eight new prompts sent two at a time (greedy, 256 tokens): two drafted streams decode together at 34.4-42.4 tokens/s (38.4 on average) against 26.6-30.0 (29.0) without --drafter, all eight replies' ids equal to the undrafted ones.
• bench_concurrent.py --levels 1,2 --alone --serial on a fresh drafted server: every reply equals its request alone and undrafted (alone 48/48, serial 12/12). Its speeds aren't quoted, because its repeated prompts let a freed slot draft the earlier reply back.

3. Warm-up failure. Drafter.start owns the model from the call: if init fails it frees the model, and the drafter it makes is freed if its warm-up fails (started, with the cleanup registered before the warm-up). The host frees the drafter before the backend, whose ledger the drafter's slots give back to. The test llama drafter: a warm-up that fails frees the drafter it was made for makes the warm-up fail and checks the drafter is freed. tf-llama-run's draft mode uses Drafter.start and Drafter.prepare too.

The harness's quick plan on the drafted --parallel 1 server, this build (its ruler with no other model loaded): structured 18/18 with drafted == plain 12/12, tools 5/5, sampling 9/9, turns resumed 5,493 of 5,649 tokens (== cold), soak 60/60, drafts 12/12, concurrent 12/12, streaming 4/4, multi-turn 8/8, stop 3/3, thinking 10/10, copy 2/2, max_tokens 3/3. Its ruler gives 113.5 / 110.7 / 109.2 / 101.8 tokens/s on one stream (texts the drafter continues almost whole).

The doc's drafter section is rewritten for the budget and the priced rounds. --parallel 2 drafts both streams.

CI on these commits over main 03d08f1 (the pushed heads are rebased onto d0c9dc9, which adds only a CI workflow time limit): on #1417's head, which carries all three, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0, and on #1415's and #1416's heads zig build test and test-kernel-contract exit 0; zig fmt --check is clean and the lean check finds 0 new problems (428 on main and on each head); llama_teacher_judge_test.py passes; tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 23 kernels: 23 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems (21 kernels on #1415's head). Rebuilt on the rebased heads, the bf16 release's 1,500-position logits are bit-identical to the judged run on all three, both quants' to theirs, and a drafted greedy chat (128 tokens) equals the plain one. The pushed heads differ from the ones CI ran on only in comments and docstrings (the lean check's one-line rule); fmt, lean, the judge's tests, the build and zig build test were rerun on them.

[ashhart]

Thanks @BobClawblaw. On b6524565, the drafter's cache and growth copies are in the common ledger (budget.zig:24), shared rounds are priced against their undrafted cost instead of skipped (pricing.zig:60), and the drafter's cleanup is registered before warm-up, with a failed-warm test.

One ordering problem is left. backend.prefillFn reserves the target and calls the drafter's prepare (zig/src/families/llama/backend.zig:216-218) before room.fit (:224) trims the prompt store. But prepare has already allocated the new cache and the old-state copy (drafter.zig:220, :226). Retained prompt snapshots can still fill the old budget, so the peak can pass it. Worked through with this PR's own formulas at an 8 GiB budget, old drafter cache, prompt states, new cache and copy reach 9.61 GB, about 1 GB over, before the trim runs. Please trim or reserve that peak before allocating, keep a protected restore entry or refuse safely, and add an admission test with retained prompt states and an allocation recorder that checks the peak, including a failed prepare.

The checker and float64 requests on #1415 and #1416 also apply to this stack.

[BobClawblaw]

Thanks, that ordering was wrong. Pushed 50fe245. This PR now builds on #1415 alone: we've withdrawn #1416 (the quants; the reasons are there), so the drafter is served from its bf16 release as published.

Admission before allocation. budget.admitStream now runs a stream's admission in one order, and backend.prefillFn goes through it:

1. the stream's peak (its caches, plus its drafter slot's growth with the old state's copy) is checked against the ledger, kept drafter slots shed first, LaneMemory if it still doesn't fit;
2. the room that peak leaves is computed, and a restore entry that can't stay beside the peak is freed (the stream prefills cold) — otherwise it stays protected;
3. the kept prompt states are trimmed to that room (room.fit, never evicting the protected entry);
4. only then is the stream's cache reserved and the drafter's prepare allowed to allocate; after the cache is made, the store gets back what the copy and the old slot gave up.

The host tests drive the same function with a fake store, drafter slot and allocation recorder that tracks the most the ledger and the kept states hold at once:

• your shape at an 8 GiB budget (slot 1, kept states 3 with a 1 GiB restore entry, a 3 GiB stream, a slot growing to 2 with a 0.5 copy): allocating before the trim would reach 9.5 GiB; the recorded peak is 8.0, and the protected entry stays;
• a restore entry too big to stay beside the peak is dropped, and the peak stays inside;
• a failed prepare gives every byte back (the ledger returns to the old slot alone) with the peak inside;
• a peak that can't fit at all waits before anything is trimmed or made.

The drafter without #1416. --drafter takes Nanbeige/Nanbeige4.1-3B @ 9c4555c37921c0982af8fffeee1e0a6e6b953c4f as published (bf16, 7.33 GiB); --drafter-bits 4 and quantized drafter checkpoints are refused by name before anything is served ("a Llama drafter is served from its bf16 release as published"). Measured again on this head, one model process on the Mac:

• tf-llama-run ... draft: on eight chat prompts (128 tokens, thinking on and off) and after a 1,500-token prompt (256), the drafted tokens equal plain greedy exactly, at 21.1-40.7 tok/s against 15.8-16.1 (1.73-4.54 tokens a round; a chained bf16 step 8.5-8.9 ms).
• Served, two streams (--parallel 2), fresh servers, eight new prompts two at a time: 30.1-40.9 tok/s together (36.1 on average) against 26.6-30.1 (29.0) undrafted, every reply's ids equal.
• The streams scenario (join, leave, queue, refuse with HTTP 400, cancel) returns each reply's plain solo ids, and shared rounds keep drafting: 199 drafts offered, 125 kept.

CI on these commits over main ea01ae1: zig build test and test-kernel-contract exit 0 on #1415's head; on #1417's head, which carries #1415, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0; zig fmt --check is clean, the lean check finds 0 new problems (426 on main and on each head), llama_teacher_judge_test.py passes, and tf-kernel-contract --only llama on the M5 Ultra gives SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. Rebased onto main e4cc630 for this push (its lane-host commits change only TENSORFOLD_PREFILL_SHARE, off by default, and a flag only GLM's CUDA backend reads): both heads rebuilt, zig build test exits 0 on both and test-kernel-contract on #1415's, fmt is clean, lean finds 0 new, the bf16 1,500-position logits are bit-identical to the judged run on both, and a drafted greedy chat equals the plain one.

[ashhart]

Thanks @BobClawblaw. The admission order now works: the stream's peak is checked and the kept prompt states are trimmed before anything is allocated, a protected restore entry stays or is dropped safely, and the recorder tests cover the 8 GiB shape and a failed prepare. Thanks also for the rebase notes.

One thing left, from our first review. With the quantized drafter withdrawn, --drafter-bits still defaults to 4, and zig/src/native/llama.zig refuses any width other than 0. So the plain command --drafter models/nanbeige4.1-3b is refused at startup, and only works with --drafter-bits 0 added. We asked for bf16 to be the default. Since the Llama drafter now has only one width, please don't refuse the default. Either ignore --drafter-bits for this family, as Gemma 4 does, or refuse only a width the user actually passed. Please also update the recipe's drafted command and the doc to match, and add a small test showing the default options serve the bf16 drafter.

A small one while you're there: in open, if building the drafter's refusal message fails, the drafter model just loaded isn't freed yet. Assigning it to drafter_model before drafterRefusal would let the errdefer cover it.

Also, if you can, please author future commits with your id-prefixed noreply address (288991054+BobClawblaw@users.noreply.github.com). Our checks expect that form.

Once the default is sorted, we'll run the device checks on our side and land this right after #1415.

[BobClawblaw]

Thanks @ashhart. Done in c3ee3db:

• Default drafter width. --drafter-bits is no longer read for Llama, as Gemma 4's host does it, so the plain --drafter models/nanbeige4.1-3b serves the bf16 drafter. A quantized drafter checkpoint is still refused by name at load. The doc's drafted command drops --drafter-bits 0, and so does the recipe's (re-pinned to this head).
• Test. llama host: the default options serve the drafter, whatever --drafter-bits holds in zig/src/native/llama.zig checks that the default api.Open (drafter_bits 4), --drafter-bits 0 and --no-drafts give the expected drafter directory. It runs under zig build test-native (a planted failure fails there). The bf16-only admission is covered by the config tests already there.
• open's errdefer. drafter_model is now set before drafterRefusal, so a failed allocation while its message is built frees the drafter.
• Identity. This commit and future ones are authored as 288991054+BobClawblaw@users.noreply.github.com.

Checked on the M5 Ultra on this head, with the served command carrying no --drafter-bits:

• The server log shows the drafter loaded: llama drafter: ...Nanbeige4.1-3B: 32 layers, bf16, 7.33 GiB of weights, 2 caches of up to 131088 tokens.
• Serving checks all pass: drafts 12/12 equal to undrafted, sampling 9/9, structured 18/18 (drafted == plain 12/12), tools 5/5, soak 60/60, concurrent 12/12, and turns resumed == cold.
• One stream decodes at 81.9-114.1 tokens/s, with 85% of drafts kept.
• CI set green: zig build test, test-native, test-kernel-contract, test-golden, the server and pull security tests, fuzz-cortheon, check-release-paths, dev-tools, fmt, and the lean check (0 new).

[BobClawblaw]

Rebased onto main e5b50c8 with #1415, which it builds on; the head is now 09cb746.

This branch's own conflict was the --drafter refusal in native/metal.zig. It keeps this branch's wording, which names Llama, beside main's list of models refused across two Macs, which now includes MiniMax-M3. git range-diff shows every commit otherwise unchanged.

Checks on the M5 Ultra, run on the same commits rebased onto main fdd8b6d, one main earlier, against the previous head (80bf371):

• Greedy tokens are identical on a short chat and after a 1,743-token prompt.
• The drafted tokens with Nanbeige4.1-3B are identical to the previous head's and equal to plain greedy; drafted decode is 20.9 tok/s against 20.5 before.
• zig fmt, zig build test, test-kernel-contract, test-golden, both security tests, fuzz-cortheon and dev-tools pass, and lean_check.py reports 0 new problems.

zig fmt and zig build test pass on 09cb746 itself.

[BobClawblaw]

Rebased onto main 6f9dfa2; the head is now 08379d5, carrying #1415's seven commits plus this PR's five.

One change beyond the rebase, in the last commit: main's new serving line names a server's draft sources (3dfd41f). With --drafter it said drafts: copies only, because the Llama drafter proposes through the lane core's copy proposer, which DraftSources.ofLanes does not name. native/llama.zig now sets the drafter where it registers it as the proposer, and the line reads drafts: Llama drafter (<dir>) + copies. git range-diff shows the other four commits unchanged.

Checks on the M5 Ultra, on this head over main 6f9dfa2:

• zig build test and test-native exit 0, and zig fmt --check is clean.
• The served startup line reads as above, the harness's tool cases pass 5/5, and a resumed turn equals a cold one after a restart.

On the same commits one main earlier (d377fcc):

• The lean check found 0 new problems (424 on main and on the head).
• tf-kernel-contract --only llama passed every kernel.
• The bf16 release's greedy tokens drafted equal its plain greedy tokens.
• The 1,500-position logits are byte-identical to the previous head's build.

[ashhart]

On the drafter: sampled at 0.7 it's 35.5 and 29.9 tok/s against about 39.7 without, and level on ordinary greedy chats. Our rule is that --drafter is never slower than no drafter. Could it stand aside on sampled requests, or on rounds where its acceptance doesn't pay, so it never costs, and keep its gain on repetitive text?

[BobClawblaw]

Thanks, reproduced: on the previous head, sampled replies on Nanbeige4.3-33B-INT4 with the 3B drafter ran at about 30 tok/s against 46.6 without. The new commit (llama: --drafter never costs: ...) makes --drafter stand aside wherever it doesn't pay:

• Copies first. With --drafter a request no longer loses the lane host's copy proposals: it keeps its own lanes.SuffixLookup (at the host's min_match) and asks it first, behind the lane core's gate (two tokens, enter_match ids matched).
• The drafter only where it pays. pricing.depth now prices a plain round beside each drafting depth, from the same timed line of drafted rounds, and returns 0 when a plain round lands more tokens a millisecond. A request then takes the lane core's plain step.
• Measured at little cost. A new request starts from the recent requests' acceptance. While it stands aside, a round drafts after 64 plain rounds, then twice as many each time, up to 1,024, so its acceptance stays measured.

Rebased onto main 7b65d60; the head is now 33a44db.

Speed, on an M5 Ultra: one stream, 16k context, thinking off, 384 tokens a reply, decode tok/s after the first token, the mean of two passes, each from a fresh server. Measured on the same commits over main 6f9dfa2. The rebases since change only the refusal line beside main's new --slide check in native/metal.zig, and on this head a served run of every request below on INT4 with --drafter gives the same replies byte for byte. Every reply is byte-identical with and without --drafter, and to the previous head.

Nanbeige4.3-33B-INT4 with the bf16 Nanbeige4.1-3B drafter:

request	--drafter	without
chat, greedy (the server's first request)	45.6	46.8
chat, greedy, repeated	46.9	46.8
chat, 0.7, seeds 1 and 2	46.6, 46.4	46.7, 46.7
story, greedy	46.8	47.0
story, 0.7, seeds 1 and 2	46.8, 46.5	46.9, 46.9
copy (repeat a text three times)	84.1	83.7

Nanbeige4.3-33B bf16 with the same drafter, where drafting pays:

request	previous head	--drafter	without
chat, greedy	19.7	19.8	13.5
chat, 0.7, seeds 1 and 2	15.1, 14.7	15.0, 14.8	13.5, 13.5
story, greedy	16.6	16.6	13.5
story, 0.7, seeds 1 and 2	14.9, 13.7	13.5, 13.7	13.6, 13.5
copy	47.5	86.1	63.1
chat, greedy, repeated	19.3	13.6, then 19.2	13.5

Two limits, so you can judge them:

• On INT4, the server's first request runs 2.5% under no drafter. A new server has no recent acceptance yet, so the first request drafts until its own rounds show drafting doesn't pay.
• On bf16, the repeated greedy chat ran right after the sampled requests in the first pass. It started from their lower acceptance (about 0.42 against its own 0.48, near the break-even) and stood aside: level with no drafter, where the previous head gained 1.4×. In the second pass it drafted (19.2). It is never slower, but it can miss a gain near the break-even.

Checked on the pushed head: zig fmt --check, zig build test (including two new drafter tests in drafter_test.zig), test-native, test-kernel-contract, test-golden, the server and pull security tests, and lean (0 new).

[ashhart]

Thanks @BobClawblaw, commit 14 reads like exactly what we asked for. Could you post --drafter against no drafter on this head, greedy and sampled at 0.7, at 1 and 2 streams on fresh prompts? The last sampled numbers (35.5 and 29.9 tok/s against about 39.7) and commit 9's two-stream line (31.5 against 34.4) both predate it. We'll gate it after today's 1.2.1 release.

[BobClawblaw]

Here are --drafter against no drafter on this head (33a44db), on an M5 Ultra. The target is Nanbeige4.3-33B bf16 and the drafter is Nanbeige4.1-3B. Each config ran on a fresh server with --parallel 2 --context 16384 --prompt-cache-gib 0, and the same binary served both. The prompts are tools/bench_prompts.json in six windows: window k sends prompt k alone, then prompts k and k+1 together. Every request carries a fixed seed (greedy and 0.7), and no prompt repeats within a cell, so nothing is served from cache.

Natural stops (no ignore_eos, up to 1,024 tokens). One stream is decode tok/s from the server's own done line; two streams is both replies' tokens over the wall time of the pair. Medians of the six windows:

	drafter	no drafter
greedy, 1 stream	49.6	27.1	+83%
greedy, 2 streams	27.9	22.8	+22%
0.7, 1 stream	25.9	17.6	+47%
0.7, 2 streams	21.4	19.4	+10%

The drafter is behind in 0 of the 24 window pairs; the closest is a tie at 19.8 (0.7, 2 streams). Per-window spread at 0.7 with 1 stream: 14.3-44.5 against 12.9-37.9. Without a drafter the server still drafts from the request's own copies, which is why the raw-completion prompts reach 60 tok/s greedy on both sides.

bench_concurrent.py (--levels 1,2 --tokens 256 --reps 1 --temperatures 0,0.7 --prompts <window file> --alone, plus --serial on the drafter server). Aggregate tok/s, medians of the six windows:

	drafter	no drafter
greedy, 1 stream	40.5	14.1
greedy, 2 streams	36.5	25.2
0.7, 1 stream	54.7	12.8
0.7, 2 streams	37.2	21.1

These overstate the drafter. ignore_eos keeps short answers going past their end, and that repetitive tail drafts almost perfectly; one request accepted 237 of 237 drafts. The natural-stop table is the one to go by.

Replies are unchanged. Every reply equals its request run alone (72/72) and "draft": false (48/48). Reply hashes are identical between the two servers: 33 distinct replies in the bench_concurrent runs and 36 in the natural-stop runs.

On this head, --drafter is ahead in every cell, by +10% to +83%, and no window is slower.

[ashhart]

Thanks @BobClawblaw, this answers the 1 Nov ask cell for cell. On head 33a44db, natural stops, medians over the six windows:

Cell	--drafter	none	Change
greedy, 1 stream	49.6	27.1	+83%
greedy, 2 streams	27.9	22.8	+22%
0.7, 1 stream	25.9	17.6	+47%
0.7, 2 streams	21.4	19.4	+10%

None of the 24 window pairs is slower with the drafter (the closest is a tie at 19.8). Replies match with and without it, and without --drafter nothing changes for any other family. Commit 14's stand-aside pricing reads as asked.

One question, not blocking: the 1 Nov INT4 tables show the first request on a new server 2.5% slower with the drafter (45.6 against 46.8). This PR refuses quantized Llama configs and drafters (families/llama/config.zig:187, native/llama.zig:64). Which build were those INT4 numbers from? When INT4 is served (#1782), that cold first request must not cost anything.

Next we run the same matrix on our own M5, alternating the two servers. Then we gate it.

[BobClawblaw]

Thanks @ashhart. The 1 Nov INT4 numbers came from #1782's branch: its INT4 commits on top of this PR's commits, commit 14's stand-aside included, over main 6f9dfa2. This PR alone refuses quantized configs, so INT4 runs only on #1782's stack. Both branches were then rebased onto 7b65d60 (#1782's head became 97c6e12); the rebase changed only the refusal line beside main's new --slide check in native/metal.zig, and a served INT4 run with --drafter on the rebased head gave the same replies byte for byte.

The cold first request can no longer cost anything. Since 70817a8, #1782 refuses --drafter for Nanbeige4.3-33B-INT4 by name, before the drafter is opened, so an INT4 server never drafts and its first request takes the plain path. The bf16 release keeps --drafter, and there the server's first request gains (19.8 against 13.5 tok/s in the same 1 Nov run). The Mac run posted on #1782 checks the refusal.