[professorpalmer]

One of the five pieces of #285, split per bri-prism's request on #317. Single commit on current prism (eaecb50c7), applies without conflicts. Only active under GGML_CUDA_BATCH_INVARIANT=1.

What it does

• The flash-attention vector path's KV split no longer depends on the occupancy of the kernel instance that runs, so the same query produces the same split whether it is decoded alone or inside a verify batch.
• PTQ1_0 batches of 5-8 columns stay on the PT mat-vec path instead of crossing to MMQ: a 5-column verify on MMQ did not match a single decode. GGML_CUDA_PTQ1_MMVQ_MAX overrides the crossover; the default of 4 is confirmed by measurement (pp5: 172 tok/s on MMQ vs 162 on the mat-vec, so 4 is where MMQ starts to win and where the verify columns of a draft of 4 still match).

Receipts

• test-backend-ops -b CUDA0 with GGML_CUDA_BATCH_INVARIANT=1: FLASH_ATTN_EXT 2994/2994, MUL_MAT 1516/1516.
• Keeps the weight path exact between drafted and undrafted decode. Attention is still not batch-invariant under drafting in general (its KV split follows the kernel instance and the padded KV length), which is stated in cuda+server: q8_0 K/V at the full 262k window on 12 GB (tiered VMM KV cache), MTP drafting at every depth #285 and in the Bonsai repo's docs.

[bri-prism]

Agent review: posted by the maintainer's coding agent at their request.

H200 (sm_90, CUDA 12.8), this diff applied to prism 73bfe6c, Ternary-Bonsai-2-9B PTQ1_0:

• With GGML_CUDA_BATCH_INVARIANT=1: MUL_MAT ptq1_0 180/180, FLASH_ATTN_EXT with hsk=256 372/372.
• Default mode (the env var unset): llama-server greedy output, 4 prompts × 128 tokens, is token-identical to stock prism; decode is within noise (+0.0% to +0.8%).

So the change stays inside the opt-in path, as described. I didn't measure 5-8-column verify throughput under BATCH_INVARIANT on this card.

[bri-prism]

Retested on an H200 against the current prism tip. With GGML_CUDA_BATCH_INVARIANT unset, nothing changes: tests pass and greedy output and speed match base. I couldn't confirm the invariance this is for, though. On the 9B PQ2_0 with the MTP head and BI on, plain and MTP-n4 output already match on base with 512-token prompts. At about 29k tokens of context they diverge at token 65 on both base and this PR, at the same token, so the occupancy-independent split doesn't close that case. Do you have a repro where base under BI diverges and this PR fixes it? Under BI the PTQ1_0 change costs about 23% decode time at 8 sequences (9B, batched-bench), which is expected, but worth stating in the description. Two small things: clamp GGML_CUDA_PTQ1_MMVQ_MAX to MMVQ_MAX_BATCH_SIZE (the MUL_MAT caller doesn't bound ne11 first, so a value above 8 would send prefill batches to the mat-vec), and document it next to the other GGML_CUDA_* knobs.

[professorpalmer]

Thank you for testing this. I looked for a repro, and I did not find one. So I am closing this PR.

Since #221, prism already covers batch-invariant mode in the three places this PR touches: the vector kernel for 1-8 queries, the KV split sized as for one query tile (ntiles_dst_eff), and the PTQ1_0 PT kernel with the batch-invariant flag. On an RTX 4070 (Ada, where the default sends 5+ column PTQ1_0 batches to MMQ), prism c96baf4ab under GGML_CUDA_BATCH_INVARIANT=1 gives the same tokens with MTP drafting at n_max 4 as plain decode: Bonsai 2 27B with its MTP head, q8_0 K/V, a 600-token prompt, 256 greedy tokens. That matches your H200 result. The divergence that you saw at about 29k is in the stream-k split, which this PR does not change either.

If it is useful, I can open a small docs-only PR for GGML_CUDA_BATCH_INVARIANT (docs/build.md has no entry for it). Tell me and I will.