[bri-prism]

Not token-identical to tip at verify widths ≥ 5. On Hopper, PTQ1_0 batches of 5-8 columns now run on the PT mat-vec instead of MMQ, so the arithmetic for those batches changes. @bri-prism decides on non-exact changes; this one is flagged the same way as #380 and #381.

What

On Hopper, PTQ1_0 mat-muls with 5-8 columns (MMVQ_MAX_BATCH_SIZE) stay on the PT mat-vec (mul_mat_vec_ptq1_0_pt) instead of crossing to MMQ at 5. The change is 6 lines in ggml_cuda_should_use_mmvq (mmvq.cu). Ada and every other architecture keep the 4-column crossover.

The gate is cc >= GGML_CUDA_CC_HOPPER && cc < GGML_CUDA_CC_BLACKWELL. GGML_CUDA_CC_BLACKWELL is 1200 in this tree, so sm_100/sm_103/sm_110 fall inside the gate too. Only H200 was measured.

Why

The existing comment explains the crossover at 4 with an Ada measurement: there, the PTQ1_0 MMQ tile path overtakes the mat-vec at 5+ columns. On H200 it is the other way round. MMQ at N=5 costs about 2.5× a single-column forward, while the PT mat-vec grows smoothly. This affects MTP / speculative verify at depth 4-7 (5-8 columns).

Results

H200, CUDA 12.8, Release sm_90. Ternary Bonsai 2 27B PTQ1_0 with an MTP head.

One forward of N tokens at depth 4096 (llama-bench -fa 1 -ub 512 -b 2048 -d 4096 -r 10, ABBA, 4 pairs, tip vs this branch):

N	tip ms	this PR ms	ratio	per-pair range
1-4	7.13 / 8.18 / 9.04 / 9.94	7.14 / 8.15 / 9.01 / 9.96	0.997-1.002	0.987-1.013
5	17.99	12.55	0.698	0.697-0.699
6	18.15	13.27	0.731	0.729-0.733
7	18.52	14.35	0.775	0.773-0.778
8	18.75	15.69	0.837	0.833-0.844
9-16			0.997-1.000	0.995-1.002

So pp N=5 is −30% and N=8 is −16%. Outside 5-8 nothing changes.

End to end, llama-server, MTP greedy decode, 4 prompts × 256 tokens × 2 reps, paired against tip. The same routing was selected through an environment override on an experimental build:

• MTP depth 5 (6-column verify): median ×1.24 tok/s (geomean ×1.21, all 8 pairs ≥ 1.03).
• MTP depth 1-3 and plain decode: median ×0.999 to ×1.004, i.e. unchanged (verify batches are ≤ 4 columns, so the path is the same).

Exactness

• Within a batch: the PT kernel's per-column arithmetic does not depend on the column count. With the 8-column cap, columns at N=2-8 are bitwise equal to each other on four 27B shapes (K×M = 5120×17408, 17408×5120, 5120×6144, 6144×5120). On tip, N=5-8 (MMQ) differ from N=2-4. The 1-column result comes from a different kernel and differs from N≥2 both on tip and here.
• Tokens vs tip: 8/8 identical at MTP depth 0-3, because verify widths ≤ 4 are unchanged. At depth 5, 4/8 runs differ from tip (first difference at token 20-24 on 2 of 4 prompts, deterministic across reps), because the MMQ arithmetic is replaced.
• MTP output vs plain decode at depth 5 improves from 4/8 identical on tip to 6/8. The same 6/8 is what depth 1-3 already gets.

Related PRs

• CUDA: fast, batch-invariant small-batch PTQ1_0 mat-vec (1.5x decode on Ampere) #218 (@sudoingX) added the batch-invariant PT PTQ1_0 mat-vec that this relies on; that kernel is already in prism. This PR only moves the Hopper crossover.
• cuda: BATCH_INVARIANT: occupancy-independent FA KV split, PTQ1_0 mat-vec up to 8 columns #322 (@professorpalmer) raises the PTQ1_0 cap to 8 columns only under GGML_CUDA_BATCH_INVARIANT=1, plus a GGML_CUDA_PTQ1_MMVQ_MAX override. Its measurement (pp5 MMQ 172 vs mat-vec 162 tok/s) supports 4 as the default on its hardware, which is consistent with keeping 4 off Hopper here. The two PRs edit the same lines in ggml_cuda_should_use_mmvq and will conflict textually. Whichever lands second needs a small rebase. The natural combination is cuda: BATCH_INVARIANT: occupancy-independent FA KV split, PTQ1_0 mat-vec up to 8 columns #322's override, then this Hopper default, then 4.

Test plan

• Rebased on current prism. Builds Release sm_90.
• test-backend-ops -b CUDA0 -o MUL_MAT: 1630/1630 at e8db221 (job 29679), the same (empty) failure set as tip 73bfe6cc.
• Speed and token checks above (run on the pre-rebase commit; the rebase onto the 2026-10-24 merges was clean and touched no CUDA files).

Not verified

• Blackwell (sm_100/sm_103/sm_120) and GB10: no hardware here. sm_100/sm_103/sm_110 fall inside the cc < GGML_CUDA_CC_BLACKWELL gate, so they get the new rule unmeasured.