[BobClawblaw]

What

Fixes #1414. A llama family for the native Metal engine, brought up on Nanbeige/Nanbeige4.3-33B (model_type llama: a dense 33B with 52 layers, 56 query and 8 key heads of 128, a 166,912-token vocabulary, 512K context, thinking and XML tool calls in its chat template). It serves the bf16 release as published, pinned to revision c1093c501cc6460427c0ac634db7223d0450c0ee: no conversion, bf16 keys and values, no fast math. Recipe doc: docs/recipes/nanbeige-4.3-33b-metal.md.

How

zig/src/families/llama/ follows the Spark-X2.6 Metal family with HF's modeling_llama in bf16 as the reference, at its rounding points: a bf16 residual with each add rounded, bf16(w * bf16(x * rsqrt(mean(x^2) + eps))), RoPE bf16(bf16(x c) + bf16(o s)) with bf16 cos and sin of the fp32 angle, bf16(bf16(silu(g)) * u). Projections read the release's bf16 rows in place from the mapped shards (no copies, no stacking): prompt chunks of up to 2,048 rows on the tensor-unit GEMM (gemm.metal), decode windows on ll_matvec with fp32 sums rounded once. Attention: prompt chunks on the tensor units over a position-major cache (attention_nax.metal), decode rows on sdpa_vector lanes over key blocks at absolute positions. The lane host's shared rounds, pipelined lone steps drawn on the GPU (argmax or tf_sample_full), copy drafts and the prompt cache; a --context/LLAMA_LOAD_LIMIT_GB budget (75% of RAM by default) sizes the caches. tf-llama-run checks greedy tokens, logits, forced steps and window exactness on real weights.

Every kernel is registered with the kernel contract (zig/tests/contract/metal_llama*.zig, zig/src/families/llama/contracts.zig): the glue, the six row-projection forms, the four GEMM forms and the prompt attention at both served geometries, gates 1-5 on this Mac:

SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 432063 ms, limits 105 ms

Measured (M5 Ultra, 256 GB, macOS 27)

Against HF's own code (transformers 4.57.1, bf16 on the CPU): 1,500 teacher-forced positions, mean KL 0.0015, top-1 99.1%; 64 positions after a 9,000-token prompt, mean KL 0.0013, top-1 100%. Three of six 64-token greedy chats equal HF's token for token; the other three part at an exact bf16 tie of the two best logits. One-row steps and 7- or 16-row windows give the same logits bit for bit (tf-llama-run ... exact fails on any differing row and checks a planted one every run) after 64- and 1,500-token prompts; a turn resumed from a kept prompt state equals the same request after a restart.

Served (--context 131072 --parallel 2 --prompt-cache-gib 8): one stream 16.4 tok/s (61 ms a step, the weights' 62 GiB read at 1.08 TB/s), two streams 28-32 together, each reply equal to its run alone; cold prompts 1,259 / 1,313 / 1,074 / 866 tok/s at 2.1k / 8.3k / 33k / 61k tokens, a 131,132-token prompt at 563 tok/s; a resumed turn starts in 0.34 s against 4.5 s cold. Structured output 18/18, tools 5/5, seeded sampling 9/9, a 60-turn soak beside a decoding stream 60/60, 12 concurrent replies equal to the request alone, needles at 10/50/90% of 128,550 tokens exact. The recipe repo with the harness evidence: github.com/BobClawblaw/nanbeige43-33b-bf16-m5-ultra.

Receipt

• Environment: TensorFold at this branch (main 01e25d3 plus these commits), Zig 0.17.0, one Apple M5 Ultra (256 GB unified, macOS 27, tensor units), Nanbeige/Nanbeige4.3-33B revision c1093c501cc6460427c0ac634db7223d0450c0ee served with --context 131072 --parallel 2 --prompt-cache-gib 8 and LLAMA_LOAD_LIMIT_GB=150, an idle 93 GiB server of another model loaded beside it for the speed lines below; the one-process receipt follows them.
• Exactness, tools/bench_concurrent.py --levels 1,2 --tokens 128 --temperatures 0,0.7 --alone --serial: every reply equals its request alone and "draft": false (alone equal 12/12, serial equal 12/12, failed 0).
• Decode speed, tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0,0.7: 16.3 tok/s on the chat prompt and 18.2 on the raw prompt greedy, 16.1-16.2 sampled (before: n/a, no engine served this model). Two streams together: 31.0 (chat) and 34.1 (code) greedy.
• Prompt speed, tools/bench_cache.py --require-cold on a 9,000-token prompt: the cold first request's ttft 6.93 s (1,299 tok/s); before: n/a. Repeated raw-token prompts in bench_cache.py report cached_tokens 0 (the family's prompt cache resumes chat turns: the harness's turns check resumes 5,493 of 5,649 tokens in 0.34 s against 4.4 s cold); the cold first request is what the receipt asks for.
• No other model server or GPU job on the Mac (28 Oct 2026; a CPU float64 reference run overlapped the first minutes): the 1,500-position logits bit-identical to the run judged below, the six greedy chats equal to the earlier run, tf-llama-run ... exact 0 differing rows after 64- and 1,500-token prompts at windows of 16 and 7 (the planted row reported each time); six prompts two at a time on --parallel 2 return each one's token ids alone; three chats with an 8,500-token first turn resume their second turn from the prompt cache (8,521-8,528 of 8,554-8,568 tokens kept) with the ids of a server without the cache.
• Fidelity judged by tools/zig/llama_teacher_judge.py (the repository's near-tie rule, qwen27_margins.py; exits 1 otherwise, and refuses a row holding NaN or ±Inf on either side): against HF's bf16 run 13 of 1,500 positions differ, 425 and 1370 red (three tokens within a bf16 step); against tools/zig/llama_f64_reference.py's float64 pass of the release the engine passes (1,483 equal, 17 near-ties) and HF's run is red at 425. The recipe lists every differing position with both verdicts.
• Tests run: On these commits over main ea01ae1: zig build test and test-kernel-contract exit 0 on llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415's head; on llama: a smaller Llama drafting for the served one (--drafter), replies unchanged (builds on #1415) #1417's head, which carries llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0; zig fmt --check is clean, the lean check finds 0 new problems (426 on main and on each head), llama_teacher_judge_test.py passes, and tf-kernel-contract --only llama on the M5 Ultra gives SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. Rebased onto main 01e25d3 for this push (one conflict, in docs/kernel-contract.md: main's row-split seam paragraph and this family's, both kept; the family's own files are unchanged): on llama: a smaller Llama drafting for the served one (--drafter), replies unchanged (builds on #1415) #1417's head zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, check-release-paths and the security tests exit 0, fmt is clean and lean finds 0 new; on llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415's head zig build test and test-kernel-contract exit 0. Five greedy chats (1,102 tokens) are byte-identical between the previous head and this one, both plain and with --drafter.
• Pushed head: rebased onto main 01e25d3; see the line above.
• Not run: the harness quality suite (not asked for); CUDA and HIP (a Metal family); M1-M4.

Checks

zig build test, zig fmt --check, the lean check (no new problems), test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security, test-pull-security: all exit 0 on this branch. Only the M5's tensor units were checked; M1-M4 Macs and smaller RAM are untested. Logprobs are refused by name (the release's tokenizer decodes through a Strip step the engine's tokenizer does not count as token-local). Only the checkpoints in qualified (config.zig: Nanbeige4.3-33B) are admitted, by the geometry their configs name; other llama checkpoints are refused by name, as are Macs without tensor units (M1-M4). Tied heads, other head sizes, Llama 3's scaled RoPE and biased projections are refused at load. Nanbeige4.3-33B is Apache-2.0 at the pinned revision.

Follow-up: a smaller Llama drafting for the served one, from its bf16 release (#1417). The published MLX quants (#1416) are withdrawn: no implementation of the reference's rounding points meets the near-tie rule against float64 at every position on them (details there).

[ashhart]

This is sound and exact by design:

• It's off unless model_type is llama, which had no native backend before.
• Weights are read in place from the bf16 shards, and rounding follows modeling_llama.
• All 21 kernels are registered with the contract; the 15 an M3 can run pass here.
• The receipts show windows equal to one-row steps, concurrent equal to solo, and resumed equal to fresh.

Before it lands:

• Teacher-forced misses. Top-1 is 99.1% over 1,500 positions, so about 13 positions differ. CONTRIBUTING asks for each one's contenders and margins. Please list them.
• Admission. It takes any bf16 llama checkpoint with 128-wide heads, but only Nanbeige is validated. Restrict admission, or state clearly that other checkpoints are unqualified.
• No tensor units. There's no named refusal on Macs without tensor units (M1 to M4). The prompt path needs the tensor-unit GEMM and attention, so a load there would fail at pipeline compile without a clear message.
• Licence. Please state the model licence in the recipe: Nanbeige4.3-33B and Nanbeige4.1-3B are Apache-2.0 at the pinned revisions.
• Test machine. The receipts were taken beside an idle 93 GiB server of another model. CONTRIBUTING asks for one model process per test machine; we'll run the device check on a clean M5.

The branch merges cleanly into main. fmt and lean are clean, host tests pass, and the llama contract passes 15/15 on an M3; the 6 tensor-unit forms need an M5.

[BobClawblaw]

Thanks. Pushed f92916d (rebased onto main 15a0b2a, a new third commit; the first two are unchanged apart from the rebase):

Teacher-forced misses. 13 of the 1,500 positions. In every one, HF's and the engine's contenders are within one bf16 step on both sides (0.125 below 32, 0.25 from 32 to 64). HF's two tie at 3 of them, the engine's at 8, and neither ties at 22 or 166. Both sides break ties to the lower id. Position 1370 has three contenders, not two: no also ties at the top on both sides. Row n predicts token n + 1. The rebased build's logits are bit-identical to the ones measured. The table is now in the recipe doc:

position	HF's top-1 (a)	engine's top-1 (b)	HF's logits a, b	engine's logits a, b	KL
22	en 271	see 3574	20.75, 20.625	20.625, 20.75	0.0047
83	the 269	Python 9120	26.25, 26.25	26.25, 26.375	0.0022
85	developed 3260	- 152369	35.5, 35.5	35.5, 35.75	0.0134
166	"; 2651	")) 9441	36.25, 36	36.25, 36.5	0.0234
224	( 152374	(' 1091	39.75, 39.5	39.5, 39.5	0.0078
401	( 152374	({' 50468	39.5, 39.25	39.25, 39.25	0.0076
425	{ 152405	1 152367	40, 39.75	40, 40	0.0163
586	representation 7443	encoding 13571	30, 30	29.875, 30	0.0027
1302	If 3934	`` 2283	32, 31.875	32, 32	0.0022
1304	indent 28799	allow 13471	37.25, 37	37, 37	0.0078
1370	it 388	the 269	38.25, 38 (no 754: 38.25)	38, 38 (no 754: 38)	0.0076
1384	`` 16411	circular 13380	30.875, 30.75	30.75, 30.75	0.0025
1447	instead 62222	with 3629	35, 34.75	34.5, 34.5	0.0129

Admission. The family now admits only the checkpoints listed in qualified (config.zig), matched on the geometry the config names: hidden, intermediate, layers, heads, key heads, vocabulary, rope theta and context. Today that list is Nanbeige4.3-33B. Any other llama checkpoint parses, then fails at load with the native Llama engine serves only the checkpoints checked against Hugging Face (Nanbeige4.3-33B): this llama checkpoint's geometry is not one of them. I checked that message on the release with only its context changed. A unit test covers five geometry changes.

No tensor units. Model.load checks device.tensorUnits() right after the device opens, before any shard is mapped or pipeline compiled. It refuses with the native Llama engine needs Metal 4 tensor units (an M5 or later) for its prompt GEMM and attention: this Mac's GPU (M1 to M4) has none. There is no M1 to M4 here, so this path has not run on a device.

Licence. The recipe doc now has a Licence section: Nanbeige4.3-33B is Apache-2.0 at c1093c501cc6460427c0ac634db7223d0450c0ee. The download command is pinned to that revision. Nanbeige4.1-3B (Apache-2.0 at 9c4555c37921c0982af8fffeee1e0a6e6b953c4f) goes in #1417's doc, since the drafter comes in there.

Test machine. Understood, and thanks for running the device check on a clean M5. The receipts in the PR were taken beside the other server, as the PR says.

Also removed: tf-llama-run's two environment switches (LLAMA_LAYERS, LLAMA_WINDOW). The exact mode now takes the window width as an argument (exact <cont.json>,7). On the rebuilt tool, 64 continuation tokens give bit-identical logits as one-row steps and as 16-row and 7-row windows (0 of 64 rows differ).

On these commits over main a1e6d34: zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0. zig fmt --check is clean, and the lean check finds 0 new problems. tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. The 1,500- and 64-token logits and a 64-token greedy chat are bit-identical to the measured build. Main then took MOSS-Image (#1457), DFlash2 and MTP fixes for other families, a GLM CUDA log message and an opt-in prompt-cache rule for CUDA families (off by default): none of it in this family's files or the paths it serves through. Rebased onto main 15a0b2a for this push, the stack's top head (#1417's, which carries these commits) reran the build, tf-kernel-contract --only llama, test-kernel-contract, zig fmt --check and the bf16, quant and drafted bit-identity checks, and all pass.

[ashhart]

Thanks @BobClawblaw. The full list of teacher-forced misses is a real help, and the qualification gate (zig/src/families/llama/config.zig:34) and the early refusal without tensor units (zig/src/families/llama/model.zig:41) are both in.

Three things before we run it on a device:

1. Row 1370 in the recipe's table (docs/recipes/nanbeige-4.3-33b-metal.md:82) isn't a two-way tie. The reference picks 388 at 38.25, with 754 also at 38.25 and 269 at 38. The engine gives all three 38 and picks 269. So the engine is one bf16 step below the reference on 388 and 754, and the two-contender rule doesn't cover it. Please find where that row's arithmetic departs from the reference, or show with a float64 pass that the reference is the one off.
2. The exactness mode can't fail. zig/tests/llama/run.zig counts differing rows (:93) but prints the count and exits 0 (:107-108). Please make it fail when any row differs, and add a planted differing row that must fail.
3. A clean receipt. The recipe's device receipt (:86) still comes from a run with another server resident. Please attach one with a single model process: reference parity, resumed against fresh, and concurrent against solo, with the token arrays and margins for every differing choice.

With those in, we'll repeat the checks on an M5 Ultra.

[BobClawblaw]

Thanks. Pushed 2e91754 (two commits on the last head, rebased onto main d0c9dc9):

1. Row 1370. You're right: it's not a two-way tie, and neither is 425 (three tokens within a bf16 step there too). tools/zig/llama_f64_reference.py runs the release in float64 (its bf16 weights cast up exactly, modeling_llama's arithmetic, one rounding at the end) and settles both:

1370	float64	rounded once	HF bf16	engine
it 388	38.1266	38.25	38.25	38.0
no 754	38.1037	38.0	38.25	38.0
the 269	38.0106	38.0	38.0	38.0

The three lie within 0.116 of each other, under half a bf16 step (0.25 at 32-64), and 388 sits 0.0016 above the rounding midpoint 38.125. HF's run puts 754 a step high; the engine puts 388 a step low. So each side is one step off on one contender, and the reference is off too.

To find where the engine's arithmetic departs, I stopped its forward after each layer count at row 1370 (a local probe, not in the PR) and compared the residual with HF's output_hidden_states and the float64 one. Both bf16 paths drift from float64 alike: 0.30% after the first layer, 1.8% after the 51st, 0.80% (engine) and 0.82% (HF) on average over the layers. The engine's largest excess over HF is 1.5-1.7 times HF's error at layers 20-22. That is one channel, 4086, a massive activation near 427 where a bf16 step is 2: both hold 430 after layer 19, and at layer 20 HF's add lands on 428 while the engine's stays at 430. Nothing departs from the reference's rounding points; it is one rounding landing a step apart.

Judged against the float64 pass over all 1,500 positions with the repository's near-tie rule (qwen27_margins.py), the engine passes: 1,483 equal, all 17 differences near-ties. HF's bf16 run fails it at 425, where it rounds two contenders a step high. The recipe's table now lists every position where the engine differs from HF or from float64, with both verdicts.

2. The exactness mode fails. tf-llama-run ... exact now exits with error.RowsDiffer when any row differs. Every run also plants a one-bit difference in a copy of the steps' logits and fails with error.PlantedRowMissed unless that row, and only it, is reported (zig/src/families/llama/exact.zig, with a host test). The new judge, tools/zig/llama_teacher_judge.py, exits 1 on any difference that isn't a near-tie; its unit test plants a wider margin and a third contender.

3. A clean receipt, with no other model server or GPU job on the Mac, on this branch's build (a CPU float64 reference run, one layer's weights at a time, overlapped its first minutes):

• Reference parity: the 1,500-position logits are bit-identical to the run judged above, and the six greedy chats' tokens equal the earlier run (three equal HF's; the other three part at the positions in the doc).
• exact after 64- and 1,500-token prompts, windows of 16 and 7: 0 differing rows each, and the planted row reported each time.
• Concurrent against solo: six prompts, greedy, 128 tokens, return_token_ids, sent two at a time on --parallel 2: every token array equals the request alone.
• Resumed against fresh: three chats whose first turn carries an 8,500-token log. The second turn resumed from the prompt cache (8,521-8,528 of 8,554-8,568 prompt tokens kept) returns the same token ids as on a server without the cache (0 kept).
• No choice differed, so there are no margins to list. The token arrays:

Token ids

prompt 0, alone and two at a time: [2359, 825, 3324, 2028, 152402, 152350, 2921, 3002, 152382, 14634, 261, 9120, 873, 349, 7605, 269, 311, 152369, 397, 58625, 1053, 72009, 152363, 358, 261, 6312, 2488, 152361, 14650, 4721, 2603, 3324, 2888, 2392, 849, 404, 7615, 152361, 14650, 1871, 48299, 21374, 152361, 17654, 6168, 311, 152369, 397, 58625, 358, 375, 152374, 152368, 5318, 152368, 152363, 375, 152374, 152367, 5318, 152367, 152361, 130712, 152361, 41076, 6312, 2488, 152361, 13172, 6602, 3741, 152470, 17654, 8873, 37226, 152361, 14650, 14583, 152361, 18628, 38503, 152361, 28972, 754, 1565, 4230, 152470, 2603, 849, 152361, 13, 166104, 13, 13, 7255, 8792, 13, 1537, 4182, 42936, 152374, 152349, 1123, 13, 264, 7292, 13, 264, 11859, 269, 311, 152369, 397, 58625, 1053, 72009, 152361, 13, 13, 264, 120809, 269, 21374, 152382, 13, 295, 152403, 152374]
prompt 1, alone and two at a time: [2359, 825, 298, 6595, 298, 2028, 152382, 471, 129236, 289, 1358, 18446, 2542, 269, 13690, 322, 6361, 1229, 14650, 2603, 5969, 1358, 18446, 2888, 152361, 11888, 38503, 152361, 40194, 55146, 11577, 152382, 27997, 102359, 1480, 13969, 25842, 602, 152397, 6361, 15686, 6720, 152397, 1582, 1545, 6361, 442, 6050, 589, 13690, 6361, 152361, 14650, 1358, 18446, 152361, 28972, 754, 5083, 152361, 13, 166104, 13, 13, 31952, 3745, 4722, 1094, 10434, 152363, 297, 724, 388, 24225, 6558, 152440, 152350, 10569, 152363, 3545, 7660, 9744, 13969, 25842, 602, 8221, 703, 3961, 4819, 152361, 675, 11577, 152363, 2403, 55146, 11577, 152363, 22417, 6361, 1831, 289, 1094, 11280, 2665, 269, 13690, 152361, 1107, 261, 816, 152363, 724, 457, 1277, 3030, 442, 269, 7542, 152363, 457, 1125, 269, 13690, 24870, 6361, 152361, 166101]
prompt 2, alone and two at a time: [2092, 1445, 152441, 152443, 152700, 9004, 5157, 14636, 29362, 2442, 67613, 10097, 8215, 152372, 152930, 5562, 53083, 152389, 1112, 24621, 4227, 29362, 152473, 3278, 50126, 6615, 4323, 152372, 152593, 957, 87446, 133816, 27067, 957, 152624, 1008, 89615, 2441, 10617, 152372, 140125, 109281, 152389, 4586, 23537, 152700, 9004, 152389, 13, 166104, 13, 13, 29362, 4268, 152803, 3278, 50126, 6615, 4323, 152468, 20239, 5848, 152389, 82372, 152836, 92108, 154119, 152464, 133816, 27067, 957, 152372, 9329, 4224, 10617, 2441, 152449, 140125, 109281, 152389, 166101]
prompt 3, alone and two at a time: [2359, 825, 298, 6595, 298, 2028, 152382, 471, 1886, 2053, 4443, 290, 261, 13695, 4055, 152363, 609, 1326, 950, 1229, 15813, 152361, 14650, 2603, 358, 2053, 3517, 152361, 28972, 609, 1326, 950, 152361, 17654, 341, 38503, 152361, 13, 166104, 13, 13, 152367, 152361, 583, 3786, 1920, 152369, 2565, 9558, 337, 4299, 37496, 152361, 13, 152375, 152361, 68456, 80991, 404, 37496, 10845, 152361, 13, 152385, 152361, 84984, 11428, 290, 3348, 15040, 152361, 13, 152395, 152361, 12272, 286, 773, 337, 3777, 2021, 415, 24751, 152361, 166101]
prompt 4, alone and two at a time: [2359, 825, 3324, 2028, 152402, 152350, 2921, 3002, 152382, 10793, 261, 13505, 1637, 358, 12645, 471, 19276, 152383, 297, 471, 20722, 152383, 337, 269, 5463, 290, 4624, 152361, 14650, 2603, 849, 13505, 2888, 152361, 17562, 290, 4624, 322, 20150, 152363, 2951, 4624, 152361, 28972, 3486, 13505, 1637, 152361, 13, 166104, 13, 13, 152405, 13, 259, 152383, 19276, 1832, 471, 89344, 748, 13, 259, 152383, 20722, 1832, 471, 31974, 152383, 13, 152406, 166101]
prompt 5, alone and two at a time: [2359, 825, 3324, 2921, 25528, 152361, 14650, 2603, 7615, 152361, 16354, 152382, 36054, 152343, 152385, 337, 486, 152375, 152361, 152343, 152367, 152375, 36054, 374, 152343, 152395, 2432, 290, 152343, 152385, 152363, 1838, 152343, 152395, 152412, 152375, 11924, 152411, 152361, 14770, 20089, 152361, 13, 166104, 13, 13, 152422, 152411, 166101]
chat 0 turn 1: [2359, 825, 3324, 2028, 152402, 152350, 2251, 152382, 471, 21301, 14504, 10824, 2617, 152343, 152385, 152413, 6005, 14650, 1151, 289, 2753, 1962, 152361, 14650, 10357, 152361, 363, 1962, 3517, 152382, 471, 152368, 152368, 152368, 152367, 14504, 152369, 152367, 10824, 2617, 152343, 152385, 152413, 289, 152343, 152375, 270, 358, 3947, 9917, 2577, 1768, 14504, 152369, 152367, 152361, 14650, 2603, 38503, 152361, 28972, 754, 5083, 152361, 13, 166104, 13, 13, 32604, 152369, 152367, 10824, 2617, 152343, 152385, 152413, 152361, 166101]
chat 0 turn 2, fresh and resumed (8521 of 8558 kept): [764, 2028, 322, 10227, 689, 820, 1119, 388, 4145, 337, 14504, 152369, 152367, 298, 11985, 2617, 152343, 152385, 152413, 152361, 20691, 417, 269, 1962, 6934, 152382, 13, 13, 152383, 152368, 152368, 152368, 152367, 14504, 152369, 152367, 10824, 2617, 152343, 152385, 152413, 289, 152343, 152375, 270, 358, 3947, 9917, 152383, 13, 13, 6114, 388, 4145, 152343, 152375, 8905, 152361, 13, 166104, 13, 13, 1874, 4145, 1472, 152375, 8905, 454, 337, 14504, 152369, 152367, 298, 11985, 2617, 152343, 152385, 152413, 326, 3629, 3947, 9917, 472, 166101]
chat 1 turn 1: [2359, 825, 3324, 2028, 152402, 152350, 2251, 152382, 471, 4894, 1094, 7643, 11795, 358, 3947, 67904, 2351, 269, 865, 152343, 152394, 152368, 3517, 6005, 14650, 1275, 3517, 152343, 152368, 152368, 152368, 152368, 298, 152343, 152368, 152368, 152395, 152398, 152470, 1672, 1861, 865, 152343, 152394, 152368, 3517, 152361, 14650, 11401, 2753, 1962, 3517, 152361, 14650, 1275, 3947, 67904, 2351, 865, 152343, 152394, 152368, 3517, 152361, 2269, 152402, 152350, 14480, 11925, 152361, 13, 13, 40563, 152382, 13, 152368, 152368, 152368, 152368, 14504, 152369, 152368, 10824, 2617, 152343, 152368, 289, 152343, 152367, 270, 358, 3947, 67904, 13, 152368, 152368, 152368, 152367, 14504, 152369, 152367, 10824, 2617, 152343, 152385, 152413, 289, 152343, 152375, 270, 358, 3947, 9917, 13, 152368, 152368, 152368, 152375, 14504, 152369, 152375, 10824, 2617, 152343, 152413, 152395, 289, 152343]
chat 1 turn 2, fresh and resumed (8528 of 8554 kept): [764, 2028, 7717, 298, 1014, 820, 1094, 7643, 11795, 358, 3947, 471, 82440, 152383, 2351, 269, 865, 152343, 152394, 152368, 3517, 152363, 297, 952, 1494, 597, 1326, 3615, 152361, 13, 13, 4546, 532, 611, 918, 269, 865, 152343, 152394, 152368, 3517, 326, 7707, 152343, 152368, 152368, 152368, 152368, 298, 152343, 152368, 152368, 152395, 152398, 1123, 13, 13, 152369, 152343, 152368, 152368, 152368, 152368, 152382, 14504, 152369, 152368, 10824, 2617, 152343, 152368, 289, 152343, 152367, 270, 358, 3947, 1472, 82440, 454, 13, 152369, 152343, 152368, 152368, 152368, 152367, 152382, 14504, 152369, 152367, 10824, 2617, 152343, 152385, 152413, 289, 152343, 152375, 270, 358, 3947, 9917, 13, 152369, 152343, 152368, 152368, 152368, 152375, 152382, 14504, 152369, 152375, 10824, 2617, 152343, 152413, 152395, 289, 152343, 152385, 270, 358, 3947, 9917, 13, 152369]
chat 2 turn 1: [2359, 825, 3324, 2028, 152402, 152350, 2251, 152382, 471, 3512, 322, 269, 3947, 343, 1326, 152343, 152368, 152368, 152398, 152368, 6005, 14650, 11401, 2753, 1962, 3517, 152361, 14650, 2888, 1326, 152343, 152368, 152368, 152398, 152368, 5969, 152382, 471, 152368, 152368, 152398, 152368, 14504, 152369, 152408, 10824, 2617, 152343, 152385, 152385, 152368, 289, 152343, 152367, 152385, 270, 358, 3947, 67904, 2577, 14650, 3324, 3947, 152382, 67904, 152361, 14650, 341, 38503, 152361, 18628, 3324, 152361, 13, 166104, 13, 13, 764, 3947, 343, 1326, 152343, 152368, 152368, 152398, 152368, 322, 1472, 82440, 35906, 166101]
chat 2 turn 2, fresh and resumed (8525 of 8568 kept): [764, 2028, 322, 10227, 689, 269, 3947, 343, 1326, 152343, 152368, 152368, 152398, 152367, 152361, 2269, 532, 1277, 417, 269, 773, 2753, 152361, 13, 13, 6161, 152343, 152368, 152368, 152398, 152367, 152382, 1495, 152368, 152368, 152398, 152367, 14504, 152369, 152368, 10824, 2617, 152343, 152385, 152408, 152413, 289, 152343, 152367, 270, 358, 3947, 9917, 152608, 13, 13, 764, 3947, 322, 1472, 569, 35906, 13, 166104, 13, 13, 764, 3947, 343, 1326, 152343, 152368, 152368, 152398, 152367, 322, 1472, 569, 35906, 166101]

CI on these commits over main 03d08f1 (the pushed heads are rebased onto d0c9dc9, which adds only a CI workflow time limit): on #1417's head, which carries all three, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0, and on #1415's and #1416's heads zig build test and test-kernel-contract exit 0; zig fmt --check is clean and the lean check finds 0 new problems (428 on main and on each head); llama_teacher_judge_test.py passes; tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 23 kernels: 23 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems (21 kernels on #1415's head). Rebuilt on the rebased heads, the bf16 release's 1,500-position logits are bit-identical to the judged run on all three, both quants' to theirs, and a drafted greedy chat (128 tokens) equals the plain one. The pushed heads differ from the ones CI ran on only in comments and docstrings (the lean check's one-line rule); fmt, lean, the judge's tests, the build and zig build test were rerun on them.

[ashhart]

Thanks @BobClawblaw. On 2e917540, exact mode now fails on a planted differing row (zig/tests/llama/run.zig:108-120). The float64 reference script, with its table of contenders, answers the row 1370 question in substance. We'll reproduce both on our side.

One more change, to the teacher-forced checker. judge() (tools/zig/llama_teacher_judge.py:23) counts two records with the same winning id as equal without looking at their scores, so {ids: [1, 2], logits: [NaN, 0]} on both sides passes. top2() (:12) has no finite check either. Please reject a row holding any NaN or ±Inf before choosing or comparing tokens. Add NaN and infinity controls on each side, including one where the winning token is unchanged, and make each one fail the checker's exit.

[BobClawblaw]

Thanks. Pushed 567379e (two commits on the last head, rebased onto main e4cc630):

Non-finite rows. llama_teacher_judge.py now refuses a row holding NaN or ±Inf before choosing or comparing tokens: top2() raises on any non-finite logit in the row (not only the winner's), judge() re-checks both sides' records, and main() reports refused: <side> position <n>: ... and exits 1. New controls in llama_teacher_judge_test.py, each failing the checker:

• NaN, +Inf and -Inf, each on the native side and on the reference side;
• {ids: [1, 2], logits: [NaN, 0]} on both sides (the winning id unchanged), and an unchanged winner with a -Inf runner-up;
• top2() with the non-finite value away from the winner;
• the script's exit: bf16 files with NaN, +Inf or -Inf planted off the winner, on either side, exit 1 with refused; the same files all finite exit 0.

llama_f64_reference.py also gained --bf16-points, which rounds to bf16 at every one of the reference's rounding points (each operation exact before it), for #1416's close.

Re-judging the real 1,500 positions against the float64 pass gives the same result as before (1,483 equal, 17 near-ties, pass).

CI on these commits over main ea01ae1: zig build test and test-kernel-contract exit 0 on #1415's head; on #1417's head, which carries #1415, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0; zig fmt --check is clean, the lean check finds 0 new problems (426 on main and on each head), llama_teacher_judge_test.py passes, and tf-kernel-contract --only llama on the M5 Ultra gives SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. Rebased onto main e4cc630 for this push (its lane-host commits change only TENSORFOLD_PREFILL_SHARE, off by default, and a flag only GLM's CUDA backend reads): both heads rebuilt, zig build test exits 0 on both and test-kernel-contract on #1415's, fmt is clean, lean finds 0 new, the bf16 1,500-position logits are bit-identical to the judged run on both, and a drafted greedy chat equals the plain one.

[ashhart]

Hi @BobClawblaw, a status update on #1415.

The engine side holds up as far as we can tell from the code. One CLI change is needed before it can land, and
we have it ready if you'd like us to carry it. Registering llama makes every llama checkpoint look servable to
tensorfold pull, models, plan and info. So pull downloads any Llama repo in full and serve then refuses
it, where main refuses it from config.json first. Our fix gives the family an optional admission check that runs
the same qualification as load, so those commands accept only Nanbeige4.3-33B's geometry. It adds a test that a
Llama-3.1-8B-shaped config is refused.

What's still owed is our device run on an M5 Ultra with the bf16 release at revision c1093c50: the llama kernel
contract, the exactness runs, the teacher-forced judge, the serve cells, and the named refusal on an M3. It waits
on that checkpoint being in place on our test Mac.

One request: the changelog's 563 tok/s at 131k came from a run beside another resident server, and no planned
cell reproduces it. Could you either drop that figure, or post a cold 131k run on an otherwise idle Mac?

[BobClawblaw]

Thanks, please carry the admission-check fix; refusing a Llama-3.1-8B-shaped config from pull, models, plan and info the same way load does is the right behavior.

On the 563 tok/s at 131k: you're right that it came from a run beside another resident server, so it isn't a cold, idle-machine number. Please drop that figure from the changelog when you carry the fix (or tell me and I'll push the removal myself).

[BobClawblaw]

Rebased onto main e5b50c8; the head is now 3696681, still seven commits.

The conflicts were with MiniMax-M3 landing beside Llama in the same lists: tensorfold.zig, cli/hub.zig, build/metal_targets.zig and native/metal.zig. Llama joins MiniMax-M3 in main's lists of models refused across two Macs, and the moved --drafter refusal stays in refusal(). git range-diff shows every commit otherwise unchanged.

Checks on the M5 Ultra with the bf16 release, run on the same commits rebased onto main fdd8b6d, one main earlier. The later main commits don't touch these files.

• tf-llama-run built at this head and at the previous one (87e6bfb) gives identical greedy tokens on a short chat (128 tokens) and after a 1,743-token prompt (64 tokens).
• Speed is level at 13.5 tok/s.
• zig fmt, zig build test, test-kernel-contract, test-golden, both security tests, fuzz-cortheon and dev-tools pass, and lean_check.py reports 0 new problems.

zig fmt and zig build test pass on 3696681 itself.

[BobClawblaw]

Rebased onto main 6f9dfa2; the head is now 4f8e13f, still seven commits, with no conflicts. git range-diff shows every commit unchanged apart from the base.

Checks on the M5 Ultra:

• On this head, zig build test and test-kernel-contract exit 0. They ran on the same commits one main earlier (d377fcc). The one later main commit with code, the DeepSeek-V4.x sampling defaults, touches no file here.
• On llama: a smaller Llama drafting for the served one (--drafter), replies unchanged (builds on #1415) #1417's head over main 6f9dfa2, zig build test and test-native exit 0 and zig fmt --check is clean. One main earlier (d377fcc), the lean check found 0 new problems (424 on main and on the head) and tf-kernel-contract --only llama passed every kernel.
• The bf16 release's 1,500-position teacher-forced logits are byte-identical to the previous head's build.

[BobClawblaw]

Following your note on #1782 (0 planted controls for #1415's kernels too): a new commit, llama: a planted control for every kernel group, each red at its gate as planted, gives each of this PR's kernel groups a control that fails as planted. Rebased onto main 7b65d60; the head is now 30cde27, the seven commits unchanged apart from the base and main's new --slide refusal line beside ours in native/metal.zig.

tf-kernel-contract --only llama on this head: SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 12 controls: 12 red as planted, 0 missed; 0 limit problems.

Checked on this head: zig fmt --check, zig build test, test-native, test-kernel-contract, test-golden, the server and pull security tests, and lean (0 new).