[drowzeys]

Thank you, Ash, for the invitation to send this for review, and for the Metal runtime it builds on.

This is the first video and audio generation model on TensorFold's Zig + Metal runtime. MiniMax H3 / FastH3 is a 21B-parameter joint video and audio diffusion transformer: 50 blocks denoise one packed sequence of text, audio and video rows. Its whole denoising forward now runs on the M5's tensor units through TensorFold's runtime, with no Python and no MLX in the forward, in the checkpoint's own precision. It is the case for TensorFold beyond text: the same runtime, kernel contract and lean code carry a model whose step is a denoising pass over 15,000 rows instead of a token.

On a Mac Studio (M5 Ultra, 256 GB), 864x480, 124 frames (5 s), 15,435 packed rows, FastH3 8-Step V2:

• 10.1 s a forward (median), against 27.9 s for the MLX reference (the same model in bf16 on MLX 0.32.3 with FastVideo's fp Metal sparse-attention kernel), alternated on the same machine: 2.75× faster.
• Parity at the reference's own noise floor. At step 5 of 8 the video velocity matches the MLX reference at cosine 0.99838 (relative error 0.057). The same binary with only the fp32 summation order of its tile scores reversed matches itself at cosine 0.99834 (relative error 0.058). The audio rows, whose attention is dense, match at cosine 0.99980 to 0.99987.
• Memory: 4.3 GB peak process footprint, plus the 42 GB bf16 checkpoint mapped read-only in place. The MLX reference peaks at 79.9 GB.
• Deterministic: two runs give byte-identical final video and audio rows.
• Kernel contract: 13 of 13 new kernels pass gates 1–4, and gate 5 has 0 limit problems. The products are upstream's own tfp::gemm_nax, unchanged.
• Lean check: 0 new findings. Host suite: identical to main.
• A decoded clip from these rows is clean (a fox crossing snow, judged by eye).

Precision is the checkpoint's own:

• bf16 weights, read in place from the checkpoint's shards.
• bf16 activations.
• fp32 sums.
• fp32 tile means, scores, top-k and the pooled branch.

The routed attention rounds its softmax weights to bf16 before the value product, as MLX's attention does. Nothing is quantized. An earlier int8 variant (closed as #586) stays on the fork and is not part of this change.

What comes next, for the gate we agree on:

• A first-frame (image to video) follow-up, stacked on this one: upstream/h3-firstframe.
• Exercising it from this repository alone. The case file (the prompt's rows, the schedule and the tile map) is written today by an exporter that uses the Python 0.6 H3 family on the fork. The next pieces to bring to Zig are the time embedder and modulation tables, so that a step count is not fixed at export, and then the text encoder and decoders.
• A tensorfold generate path. Today this is a test tool beside the engine, with no lane round. How a denoiser should sit beside or inside the lane core is a design question I would like to settle with you.
• Speed. The products are about 70% of a forward (profile below). Fused projections and a bf16 attention with fewer passes over the scores are the obvious levers.

I did not open an issue first, since this came by invitation. Happy to move the gate discussion into one if you prefer.

What it adds

These are new files, plus the minimal wiring: 13 lines in build.zig, 1 import in zig/build/kernel_contract.zig, 17 lines in the Metal contract runner, 7 in metal_limits.zig, and 15 launch records in registry.zig.

• zig/src/families/h3/:
  • h3.zig loads the model and the case file, and holds the forward around the blocks.
  • block.zig is one block: modulated norm, q/k/v and the routing gate, head layout, routing, attention, the pooled branch, the out projection and the SwiGLU MLP.
  • tensors.zig maps the safetensors shards whole into Metal buffers, finding tensors by name.
• zig/kernels/metal/h3/:
  • h3_rows.metal: the rows in and out, the norms, the gated adds and the SwiGLU.
  • h3_routing.metal: the head layout with q/k norm and rotary, tile means, tile scores, top-k and the pooled branch.
  • h3_attention.metal: tile-routed attention on the tensor units, with fp32 scores and an online softmax.
  • The projections use the prefill source's gemm_nax entry points as they are.
• zig/tests/h3_dit.zig (zig build tf-h3-dit): every forward for a case file, the first forward against the reference velocities, a per-stage profile, and the rows as raw f32.
• zig/tests/contract/metal_h3*.zig: the 13 kernels' float64 references and launches, plus the top-k against a CPU oracle.
• docs/recipes/minimax-h3-metal.md: the arithmetic, the case file and the receipt.

How it fits the engine

• No lane round. H3 has no token stream, so nothing is drafted. The tool runs beside the engine. The lane core, the server and every existing family are untouched.
• Precision. As listed above, with no precision trade.
• Chips. It needs the M5's tensor units and refuses to load without them.

Receipts

Mac Studio, Apple M5 Ultra, 256 GB, macOS 27.0.1, Zig 0.17.0, Xcode Metal toolchain.

• Branch at the head of upstream/h3-metal on main 46a08c18.
• Checkpoint FastVideo/FastVideo-FastH3-8-Step-V2, 14 shards.
• 864x480, 124 frames, 8 steps: 15,435 rows (36 text, 414 audio, 14,985 video), 288 tiles (8 prefix, 56 video tiles kept a query).
• Public prompt: "A red fox trots across a snowy meadow at sunrise, its breath visible in the cold air. The camera tracks it from the side. Soft wind and crunching snow.", seed 42.
• Reference: the Python 0.6 H3 family (fork) in bf16 on MLX 0.32.3, with FastVideo's reference routing.
• Every timed run held the machine's shared GPU lock. No other GPU job ran, and no Ollama model was loaded except as noted.

Speed, alternated on the same machine:

Run (in order)	A forward, steps 2–8 (wall)	Notes
tf-h3-dit, run 1	10.14–10.30 s	step 1: 9.80 s
MLX reference, run 1	27.98, 27.95, 34.03 s	three forwards
tf-h3-dit, run 2	8.85–10.16 s	step 1: 17.1 s wall, 8.9 s on the GPU (first-touch of the mapped weights)
MLX reference, run 2	27.70, 27.82, 27.84 s	an idle 73 GB Ollama model was resident during this run (not ours)
Median	tf-h3-dit 10.15 s, MLX 27.9 s	2.75×

The GPU's own time a forward is within 0.01 s of wall time from step 2 on.

zig build tf-h3-dit -Dcpu=apple_m1 -Doptimize=ReleaseFast
zig-out/bin/tf-h3-dit <FastH3>/transformer 14 case.safetensors out
# reference: the fork's h3_case.py forward with route(..., "simd"), mx.eval per forward, timed

Per stage, from one profiled forward (each stage in its own command buffer, so they sum to more than a forward):

Stage	s	Stage	s
q, k, v, gate products	4.00	attention out	1.02
head layout	0.51	MLP in	4.54
routing	0.46	MLP out	2.17
attention	2.61	norms, mix, rows	0.42

Parity: the first forward's velocities against the reference:

Case	Video	Audio
Step 1 of 8 (all video tiles pure noise)	cosine 0.98981, rel. error 0.143, max abs 3.52 (reference max 7.47)	cosine 0.99987, rel. error 0.017
Step 5 of 8 (reference ran steps 1–4)	cosine 0.99838, rel. error 0.057, max abs 2.98 (reference max 7.58)	cosine 0.99980, rel. error 0.020
Self-noise: same binary, tile-score sum reversed, step 1	cosine 0.99021, rel. error 0.140	cosine 0.99992, rel. error 0.012
Self-noise, step 5	cosine 0.99834, rel. error 0.058, max abs 3.38	cosine 0.99963, rel. error 0.027
For scale: the fork's int8 engine, step 5	cosine 0.99574, rel. error 0.092	cosine 0.99690, rel. error 0.079

At step 1, every video tile is noise. FastH3's top-k among them is decided by differences far below bf16, so any two correct implementations differ there. Video against the reference sits at the engine's own self-noise floor at both steps. The audio rows, which attend densely, show the arithmetic agreement directly. Per row (step 5): median relative error 0.047, 99th percentile 0.138.

Memory (/usr/bin/time -l, the 8-step run):

	Peak memory footprint	Max resident
tf-h3-dit	4.26 GB, plus the checkpoint mapped read-only (42 GB on disk)	1.21 GB
MLX reference, 3 forwards	79.9 GB	42.8 GB

Determinism: two runs, byte-identical final video and audio rows (cmp).

Kernel contract

zig build tf-kernel-contract -Dcpu=apple_m1 -Doptimize=ReleaseFast, then zig-out/bin/tf-kernel-contract --offline DIR (with M1/M2 compiles) on the M5 Ultra:

main 46a08c18: SUMMARY 26 kernels: 26 pass gates 1-4, 0 fail, 2 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems
this branch:   SUMMARY 39 kernels: 39 pass gates 1-4, 0 fail, 2 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems
               PASS limits: 92 registered Metal launches against M1-M5, 0 problems

The two known findings are main's affine_mm q4 entries.

The registered kernels: h3_rows_in, h3_rows_out, h3_norm, h3_final_norm, h3_gate_add, h3_gate_mix, h3_swiglu, h3_heads, h3_pool, h3_tile_scores, h3_coarse, h3_attention, and h3_topk.

• Each one is checked against its formula in float64.
• Intermediate bf16 roundings are declared as mids. Inputs are of one sign where a sum could cancel, so no rounding hides behind a small result.
• h3_topk is checked byte for byte against a CPU oracle under a stated rule: a NaN score ranks as −inf, and ties go to the lower tile. Gate 1 found that a NaN score let a row keep one tile too many and write past its row. The kernel now ranks NaN as −inf, and the test keeps that failure covered.
• Gate 4 is exact for the dot products (h3_rows_in, h3_rows_out, h3_tile_scores). The softmax and norm kernels state why their bf16 output hides the sums' error, as main's softmax and RMS norm do.
• Launch records: M1/M2 thread limits come from --offline. h3_attention is a tensor-op kernel at 256 threads, and the GEMMs reserve 256.
• Gate 3 runs at the engine's widths (5,376, 7,168, 14,336) with tens of rows, and at 2 heads and 16 tiles for the attention, not at 15,000 rows: a float64 host reference cannot keep up at that size.

Checks

Check	main 46a08c1	This branch
python3 tools/zig/lean_check.py	517 problems	517 (zero added)
zig build test -Dcpu=apple_m1	204/204 steps, 2989 of 2999 passed, 10 skipped	204/204 steps, 2989 of 2999 passed, 10 skipped
zig fmt --check on changed Zig		clean
zig build tf-h3-dit on the M5 Ultra		builds and runs the receipts above

The 10 skips are the same on both: JPEG pixel cases without libjpeg-turbo, and others the suite skips here.

Not tested

• Any Mac but the M5 Ultra; chips without tensor units are refused.
• Any checkpoint but FastH3 8-Step V2. Any size but 864x480 at 124 frames.
• A case file from this repository: the exporter lives on the fork (tools/zig/h3_case.py, branch h3-1.0.4). I can attach the receipt's case file (186 MB) wherever you prefer.
• Clip quality by metric. Clips were judged by eye.

Licenses

The code is Apache-2.0, as TensorFold is. MiniMax H3 and FastH3 weights are under the MiniMax H3 Community License, which limits territories. No weights are included.

The CUDA side of the same model is #947, and the Studio that drives both is #941.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PfjmufXm4L7qsVNFyRw9ZK

[ashhart]

Thank you, @drowzeys, we will carry the H3 Metal denoiser work with your credit into the shared H3 CUDA/Metal family; the landing order and remaining checks are in our reply on #961.

Posted by Ash's AI Assistant Synthena

[drowzeys]

Rebased on main 6ee828c2; head 458be74c is now a428b8b7. The pull request's diff against main is patch-identical to before (main's new commits only touch docs); zig fmt --check clean on the changed Zig files; python3 tools/zig/lean_check.py reports no new findings against main.

[ashhart]

Thank you, @drowzeys, for H3/FastH3 on Metal. The selected work from #949 is included through #961 (013b75368) in 1.1.0, within one family and the bf16 tf-h3-dit test tool. Geometry/media checks and transactional schedule replacement accompany the integration. Full scope and numerical, resource and reference-media results are on #961: Apple M5 Ultra H3 kernels passed 15/15 with controls; full FastH3/H3 checkpoint/media-tool cells remain unrun.

Posted by Ash's AI Assistant Synthena

[ashhart]

The bf16 H3 Metal component shipped in 1.1.0 as 013b75368; thanks @drowzeys. We're closing it as landed.

Posted by Ash's AI Assistant Synthena