Skip to content

hip: RDNA3.5 MMDQ prefill and quantized FFN fusions - #205

Draft
liangliangchang wants to merge 3 commits into
gfx11from
lichang.mmdq-prefill-fusions-gfx11
Draft

liangliangchang wants to merge 3 commits into
gfx11from
lichang.mmdq-prefill-fusions-gfx11

Conversation

@liangliangchang

@liangliangchang liangliangchang commented Nov 13, 2026 •

Copy link
Copy Markdown

Adds opt-in FP16 WMMA multiplication on quantized weights (MMDQ), including Q5_0 support, and fuses eligible gate/up projections, SwiGLU or GEGLU, and the conversion feeding the down projection. The down multiplication remains a separate launch that reads the FP16 intermediate directly. This reduces repeated activation conversion and materialization during prefill.

Q5_0 support, the MMDQ foundation, SwiGLU fusion and GEGLU are combined in one PR because the extensions depend on the same kernel and dispatch. MMQ producer fusions are included here as well: their eligibility must give MMDQ priority to avoid intercepting a faster full FFN fusion. The branch is based directly on gfx11 (f70aba1cb6), with no dependency on PRs #192, #195, #201 or #204.

Included work:

  • Dense Q4_K, Q5_0, Q6_K and eligible Q8_0 MMDQ, plus streaming expert MMDQ for supported quantizations.
  • SwiGLU and GEGLU FFN fusion, output bias fusion, and reuse of converted input across eligible nearby Q/K/V projections.
  • MMQ fallback paths that combine RMSNorm and weight multiplication with Q8 packing, combine GLU with Q8 packing, and interleave the Q4_K activation-packing grid.
  • Q6_K scale reuse for RDNA3.5 J64 MMQ tiles.

The graph matchers check types, layouts, shape, graph consumers and applicable memory alias constraints. The kernels retain FP32 accumulation and reject explicit unsupported operand-precision overrides. MMDQ converts inputs and decoded weights to FP16, so numerical equivalence is bounded by tests rather than a claim of bitwise identity. The dispatch is restricted to RDNA3.5.

Fresh results from the final combined build on Strix Halo (gfx1151):

Model Input gfx11 (ms) This PR (ms) Time saved
Qwen2.5-0.5B 3968 tokens 285.77 209.19 26.8%
Qwen2.5-3B 128 tokens 45.79 37.26 18.6%
Qwen2.5-3B 3968 tokens 1223.88 958.24 21.7%
Gemma-2B 3968 tokens 894.92 719.97 19.5%
Gemma-3-12B 128 tokens 145.69 116.52 20.0%
Gemma-3-4B VLM 779 positions 1104.85 1054.25 4.6%
Qwen2.5-VL-3B 1606 positions 2309.61 2226.04 3.6%

Language rows are llama-bench prefill, batch 4096, FP16 KV and flash attention. Microbatch is 2048 for Qwen and 1024 for Gemma. VLM rows are HTTP time to first content token, using microbatch 512, context 4096, identical images/prompts, 32 output tokens, no checkpoints, and no prompt/RAM caching. Each arm has two process starts, two warmups and five retained samples per start, ordered A/B/B/A; values are medians of ten samples. These results compare against gfx11; vLLM was not rerun.

Component ablations used separately built MMDQ-only or MMQ-only extractions on the same gfx11 base, with two starts and seven samples per arm. They identify contributions, but are not additive:

Component disabled for comparison Representative workload Time saved when enabled
Q5_0 MMDQ Qwen2.5-0.5B, 3968 27.1%
SwiGLU FFN fusion Qwen2.5-3B, 3968 14.5%
GEGLU extension Gemma-2B, 3968 11.4%
Q6_K MMDQ, including dependent fusion eligibility Gemma-3-12B, 3968 6.4%
Streaming expert MMDQ Gemma-4-26B-A4B, 3968 9.3%
Dense Q8_0 MMDQ Gemma-4-26B-A4B, 3968 2.9%
Converted-input reuse Qwen3-4B, 3968 0.8%
Output bias fusion Qwen2.5-0.5B, 3968 0.4%
RMSNorm/Q8 packing Qwen2.5-3B, 3968 0.6-0.9% across two boards
GLU/Q8 packing Qwen2.5-3B, 3968 2.2-2.3% across two boards
Packing-grid interleave Qwen2.5-3B, 3968 0.5-0.8% across two boards
Q6_K MMQ scale reuse (other MMQ optimizations off) Gemma-3-12B, 128 3.5%

The Q6_K MMQ scale-reuse result uses A/B/B/A with five retained samples per start. On Qwen2.5-7B at 128 tokens, its saving varied from 0.5% to 3.9% across two boards. The input-reuse and bias effects are small; the main gains come from the GEMM and FFN paths. MMDQ normalization fusion, dense Q5_K dispatch and block swizzling are excluded because their additional benefit was not established or they regressed previous tests. Q5_K decoding remains where required by the supported expert path.

Enable MMDQ with GGML_CUDA_MMDQ=1. The reported full configuration also sets GGML_CUDA_MMDQ_Q5_0, GGML_CUDA_MMDQ_Q8_0, GGML_CUDA_MMDQ_GLU, GGML_CUDA_MMDQ_GEGLU, GGML_CUDA_MMDQ_MOE, GGML_CUDA_MMDQ_FUSE_BIAS and GGML_CUDA_MMDQ_SHARE_SRC1 to 1. GGML_CUDA_MMDQ_NO_Q6K=1 disables dense Q6_K MMDQ. The MMQ packing/interleave switches are GGML_CUDA_FUSE_NORM_PACK, GGML_CUDA_FUSE_GLU_PACK and GGML_CUDA_QUANTIZE_K_INTERLEAVE, defaulting to enabled for eligible RDNA3.5 calls and accepting 0 for ablation. GGML_CUDA_MMQ_Q6_K_SCALE_REUSE=0 is the compile-time switch for the Q6_K MMQ specialization.

Validation: Release HIP build passed; 120/120 focused backend tests passed with MMDQ enabled, and 40/40 MMQ producer tests passed with it disabled. Coverage includes quantization types, batch and row tails, SwiGLU/GEGLU, biases, shared inputs, extra consumers, residuals, expert routing, and explicit FP32 accumulation. The relevant HIP code passed warning-as-error compilation on gfx1151 and gfx942, including compilation with the Q6_K scale specialization disabled. Tests use a maximum normalized mean-square error of 5e-4 where applicable; this is numerical kernel validation, not a downstream model-quality evaluation.

Assisted-by: Codex

Optional token-tile grouping

Restores GGML_CUDA_MMDQ_SWIZZLE=n for dense matrices whose FP16 input exceeds 8 MiB. Grouping a few token tiles changes block scheduling to improve cache reuse. It does not change quantization or arithmetic. The default remains 0 (ordinary grid order); this is deliberately opt-in because grouping regressed the larger Qwen3.6 workload in the earlier audit.

Fresh Gemma-2B measurements at 3968 input tokens, A/B/B/A with two warmups per process and ten retained samples per arm:

Configuration Grouping off Grouping = 2 Time saved
This MMDQ PR alone 719.90 ms 703.61 ms 2.26%
Best optimized branch, isolated setting 671.46 ms 653.72 ms 2.64%

The isolated best-branch gains were smaller for Qwen2.5-3B (0.96%) and Qwen3-4B (0.40%). This is a per-workload tuning option, not a universal default recommendation.

Validation: 37/37 focused MMDQ cases passed at grouping 0 and 2, including an input above the size threshold with a partial final tile group. The source compiles with warnings treated as errors for gfx1151 and gfx942. This remains part of the MMDQ PR; there is no new dependency on another draft.

Keep the default grid order; enable grouping only when requested and the FP16 input exceeds 8 MiB. Gemma-2B long prompts benefit, while larger models require their measured setting.

Assisted-by: Codex
@jimw567

jimw567 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

🤖 /pr skill detected an infra issue. Restarting: GHA runs: AMD-Ecosystem/llama.cpp=43352321448

Detected: Infrastructure-eligible GHA rerun(s) dispatched

1 similar comment
@jimw567

jimw567 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

🤖 /pr skill detected an infra issue. Restarting: GHA runs: AMD-Ecosystem/llama.cpp=43352321448

Detected: Infrastructure-eligible GHA rerun(s) dispatched

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants