Repository navigation
hip: accelerate D64/D72/D80 vision attention with WMMA - #195
Open
liangliangchang wants to merge 5 commits into
Open
liangliangchang wants to merge 5 commits into
liangliangchang wants to merge 5 commits into
Conversation
Assisted-by: Codex
Assisted-by: Codex
liangliangchang
marked this pull request as ready for review
November 12, 2026 00:46
Preserve D80 full-tile tuning and extend it to D64. Keep the native dispatch switch independent of the D72 adapter. Assisted-by: Codex
This was referenced Nov 12, 2026
Account for the KV_rows_ptr argument added in 1b6935b. Update only the 20 existing kernel specializations reported by the gfx908 check; keep the spill threshold unchanged. Assisted-by: Codex
Assisted-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change enables the existing WMMA attention kernel for eligible unmasked D64/D72/D80 vision heads on RDNA3.5. Native D64/D80 inputs use the existing kernel directly. D72 inputs are padded to D80, processed with their original attention scale, and cropped back to D72. Complete dense KV tiles skip redundant partial-tile bounds handling; the final tile retains its checks.
Implementation
1/sqrt(72). Its 4096 vision patch positions still attend globally.The path requires equal Q/K/V sequence lengths, head counts and batch dimensions; FP32 Q/output and FP16 K/V; channel-contiguous inputs and contiguous output. D72 row counts are bounded before allocation and index narrowing. The existing WMMA configuration, null-mask support and D80 FP32 accumulators are reused. Q/output storage remains FP32 and K/V remains FP16.
The D72 adapter uses pooled temporary buffers. For 4096 positions and 16 heads it adds three padding launches, one crop launch and 60 MiB of logical temporary storage, reused across layers. Native D64/D80 needs none of those adapter copies. Window batching, QKV fusion, language GEMM and checkpoint optimizations are separate from this change.
Combined branch: fresh end-to-end TTFT
Combined commit
ec56b7427is compared with clean gfx11f70aba1cb. All rows below are fresh measurements of this combined implementation with D64 and D80 full-tile tuning enabled. Each model uses the same board, input, model/projector files and server settings for both builds.Weights are Q4_K_M, except Gemma-4-26B-A4B uses UD-Q4_K_M and Gemma-4-E2B uses Q4_0. The projector files and original image dimensions are recorded in
input_conditions.json. BF16 projector weights still produce the FP32 Q / FP16 K/V tensors required by this attention path.Timing is HTTP time to the first content token from
llama-server. Each value is the median of ten measured requests across two server starts, with two warmups per start. Order per model: gfx11 / combined / combined / gfx11. No profiled requests or historical timing values are mixed into this table. All accepted samples, including outliers, are retained. Samples with detected competing GPU/CPU activity are rejected and retried.Full GPU offload, 8 CPU threads, flash attention on, context 4096, batch 2048, microbatch 512, one server slot, FP16 K/V cache, 32-token greedy output cap, HIP graphs on, unsafe math off. Checkpoints and prompt reuse are disabled:
--ctx-checkpoints 0 --no-cache-prompt, requestcache_prompt=false. Every accepted request has zero reused prompt positions. RAM prompt-cache capacity is left at the server default; these runs do not pass--cache-ram 0. Native image/text boundaries are preserved.Gemma-4 variants use the same
--image-max-tokens 280on both builds. Gemma-3 retains its normal 896 x 896 vision input and 4096 patch positions, producing 256 image embeddings. The images/messages are the saved original workloads for each model; prompt-position counts and model/projector/request hashes are checked for equality across the two builds. These are per-model comparisons, not comparisons between different model workloads.Qwen2.5-VL still uses the original masked path for its window-attention layers. This PR accelerates its unmasked full-attention layers. Combining it with the separate window-batching PR may expose additional eligible calls; that combination is not part of these results. MiniCPM also contains other attention head widths, which retain their existing path.
Controls and isolated tuning evidence
GGML_CUDA_FATTN_WMMA_D72GGML_CUDA_FATTN_WMMA_NO_GQAGGML_CUDA_FATTN_D64_FULL_TILEGGML_CUDA_FATTN_D80_FULL_TILEThe two full-tile controls must be set at compilation, for example through
CMAKE_HIP_FLAGS; process environment variables cannot change compiled loop bodies. Runtime dispatch switches are read once per process.The following are earlier isolated measurements, retained to explain the tuning choices. They are not additional samples in the fresh combined table:
D80 full-tile tuning stays enabled to preserve the measured Gemma benefit. The small native Qwen2.5 penalty does not eliminate the gain from WMMA dispatch. There is one shared implementation and one D80 default; this change does not reverse the earlier D80 tuning. These timings alone do not identify the compiler/resource cause of the native D80 difference.
Validation and numerical behavior
The combined Release HIP build passes for
llama-server,llama-benchandtest-backend-ops. All 21 focused CPU-reference comparisons pass: D64/D80 dispatch boundaries, partial tiles and permuted batches, plus D72 thresholds and feature fallbacks. The existing test tolerance is unchanged. Separate dispatcher builds with each dispatch macro disabled pass; the affected template also builds with native dispatch disabled, preserving D72 support. These checks use-Werrorwith the recorded suppression for existing tautological-compare warnings. Source whitespace checks pass.Fresh generated-output comparison for the combined branch:
One result needs explicit qualification: Gemma-4-E2B gfx11 samples alternate between describing the image and asking for an image, while all combined samples ask for an image on this repeated-prompt workload. The output check does not establish equivalent image understanding for this model. The saved responses make this visible; the TTFT improvement should not be presented as proof of unchanged model quality.
Existing numerical evidence for the unchanged D72 adapter remains applicable to that adapter: a controlled 4096-position, 16-head D72 probe matched the original optimization branch bitwise. Against FP64 stable-softmax reference, relative L2 error was 0.03620% for padded D80 WMMA versus 0.54033% for the generic D72 kernel; maximum absolute error was 0.00010253 versus 0.00188810. The D80 full-tile-only addition previously produced bitwise-identical tested attention outputs and Gemma-3-4B vision embeddings.
The earlier Gemma-3-4B request changed one generated word at token 28, also reproduced by the original optimization branch. End-of-vision embeddings differed from the generic path by about 3% relative L2 after the complete tower, despite lower attention error on the FP64 probe. Numerical differences can alter greedy output; neither those earlier checks nor the fresh text comparisons establish unchanged task accuracy across a broad dataset. The current tests support operator correctness on the tested shapes.
Scope and provenance
Catalog directions: #19 native non-GQA D64/D80 WMMA, #20 D72-to-D80 adaptation, #58 D80 full-tile tuning, plus the D64 extension of the full-tile rule. These share the same existing attention implementation. No dedicated replacement attention kernel is added.
Branch:
lichang.gemma3-d72-wmma-gfx11. Existing PR: #195.AI assistance: Codex assisted with implementation integration, test execution and this description.