Repository navigation
hip: RDNA3.5 MMDQ prefill and quantized FFN fusions - #205
Draft
liangliangchang wants to merge 3 commits into
Draft
liangliangchang wants to merge 3 commits into
liangliangchang wants to merge 3 commits into
Conversation
Assisted-by: Codex
Keep the default grid order; enable grouping only when requested and the FP16 input exceeds 8 MiB. Gemma-2B long prompts benefit, while larger models require their measured setting. Assisted-by: Codex
Assisted-by: Codex
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
1 similar comment
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds opt-in FP16 WMMA multiplication on quantized weights (MMDQ), including Q5_0 support, and fuses eligible gate/up projections, SwiGLU or GEGLU, and the conversion feeding the down projection. The down multiplication remains a separate launch that reads the FP16 intermediate directly. This reduces repeated activation conversion and materialization during prefill.
Q5_0 support, the MMDQ foundation, SwiGLU fusion and GEGLU are combined in one PR because the extensions depend on the same kernel and dispatch. MMQ producer fusions are included here as well: their eligibility must give MMDQ priority to avoid intercepting a faster full FFN fusion. The branch is based directly on
gfx11(f70aba1cb6), with no dependency on PRs #192, #195, #201 or #204.Included work:
The graph matchers check types, layouts, shape, graph consumers and applicable memory alias constraints. The kernels retain FP32 accumulation and reject explicit unsupported operand-precision overrides. MMDQ converts inputs and decoded weights to FP16, so numerical equivalence is bounded by tests rather than a claim of bitwise identity. The dispatch is restricted to RDNA3.5.
Fresh results from the final combined build on Strix Halo (gfx1151):
Language rows are
llama-benchprefill, batch 4096, FP16 KV and flash attention. Microbatch is 2048 for Qwen and 1024 for Gemma. VLM rows are HTTP time to first content token, using microbatch 512, context 4096, identical images/prompts, 32 output tokens, no checkpoints, and no prompt/RAM caching. Each arm has two process starts, two warmups and five retained samples per start, ordered A/B/B/A; values are medians of ten samples. These results compare against gfx11; vLLM was not rerun.Component ablations used separately built MMDQ-only or MMQ-only extractions on the same gfx11 base, with two starts and seven samples per arm. They identify contributions, but are not additive:
The Q6_K MMQ scale-reuse result uses A/B/B/A with five retained samples per start. On Qwen2.5-7B at 128 tokens, its saving varied from 0.5% to 3.9% across two boards. The input-reuse and bias effects are small; the main gains come from the GEMM and FFN paths. MMDQ normalization fusion, dense Q5_K dispatch and block swizzling are excluded because their additional benefit was not established or they regressed previous tests. Q5_K decoding remains where required by the supported expert path.
Enable MMDQ with
GGML_CUDA_MMDQ=1. The reported full configuration also setsGGML_CUDA_MMDQ_Q5_0,GGML_CUDA_MMDQ_Q8_0,GGML_CUDA_MMDQ_GLU,GGML_CUDA_MMDQ_GEGLU,GGML_CUDA_MMDQ_MOE,GGML_CUDA_MMDQ_FUSE_BIASandGGML_CUDA_MMDQ_SHARE_SRC1to1.GGML_CUDA_MMDQ_NO_Q6K=1disables dense Q6_K MMDQ. The MMQ packing/interleave switches areGGML_CUDA_FUSE_NORM_PACK,GGML_CUDA_FUSE_GLU_PACKandGGML_CUDA_QUANTIZE_K_INTERLEAVE, defaulting to enabled for eligible RDNA3.5 calls and accepting0for ablation.GGML_CUDA_MMQ_Q6_K_SCALE_REUSE=0is the compile-time switch for the Q6_K MMQ specialization.Validation: Release HIP build passed; 120/120 focused backend tests passed with MMDQ enabled, and 40/40 MMQ producer tests passed with it disabled. Coverage includes quantization types, batch and row tails, SwiGLU/GEGLU, biases, shared inputs, extra consumers, residuals, expert routing, and explicit FP32 accumulation. The relevant HIP code passed warning-as-error compilation on gfx1151 and gfx942, including compilation with the Q6_K scale specialization disabled. Tests use a maximum normalized mean-square error of 5e-4 where applicable; this is numerical kernel validation, not a downstream model-quality evaluation.
Assisted-by: Codex
Optional token-tile grouping
Restores
GGML_CUDA_MMDQ_SWIZZLE=nfor dense matrices whose FP16 input exceeds 8 MiB. Grouping a few token tiles changes block scheduling to improve cache reuse. It does not change quantization or arithmetic. The default remains0(ordinary grid order); this is deliberately opt-in because grouping regressed the larger Qwen3.6 workload in the earlier audit.Fresh Gemma-2B measurements at 3968 input tokens, A/B/B/A with two warmups per process and ten retained samples per arm:
The isolated best-branch gains were smaller for Qwen2.5-3B (0.96%) and Qwen3-4B (0.40%). This is a per-workload tuning option, not a universal default recommendation.
Validation: 37/37 focused MMDQ cases passed at grouping 0 and 2, including an input above the size threshold with a partial final tile group. The source compiles with warnings treated as errors for gfx1151 and gfx942. This remains part of the MMDQ PR; there is no new dependency on another draft.