Repository navigation
hip: fuse Gemma vision RoPE and enable D512 WMMA attention - #208
Draft
liangliangchang wants to merge 1 commit into
Draft
liangliangchang wants to merge 1 commit into
liangliangchang wants to merge 1 commit into
Conversation
Keep the existing grid loops for oversized tensors and specialize one-row-per-block vision RoPE. Enable the existing D512 4x8 tile on RDNA3.5 with FP16 KV and no logit softcap. Assisted-by: Codex
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
1 similar comment
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two related RDNA3.5 attention improvements for Gemma-4: fuse RMSNorm, channel weights, and the two vision RoPE axes into one kernel; and enable the existing D512 4x8 WMMA attention tile for eligible language attention.
The two-axis fusion requires matching NeoX RoPE parameters, complementary offsets, FP32 tensors, and no extra consumers or unsafe aliases. The common case uses one row per block without grid-loop state or a trailing barrier. Tensors exceeding the grid limits retain the existing loops and synchronization. This avoids the register increase that made the initial port slower: the E2B vision kernel returns from 48 to 32 VGPRs, with no scratch spills.
D512 selection is limited to RDNA3.5, GQA ratio 8, at least 32 queries, FP16 K/V, and zero logit softcap. It uses the existing WMMA implementation and keeps Q outside registers for this tile. Other attention shapes retain their current selection.
Controls:
GGML_CUDA_FUSE_NORM_ROPE_2AXIS=0,GGML_CUDA_NORM_ROPE_SMALL_BLOCK=0, andGGML_CUDA_FATTN_WMMA_D512=0independently disable the corresponding changes. The shape guards apply when enabled.Fresh standalone results:
In the combined optimization stack, D512 WMMA alone saves 1.51% on E2B. Specializing the two-axis grid further saves 1.44% on E2B and 1.42% on A4B. A retained-prefill trace measures the final E2B two-axis kernel at 7.116 ms versus 7.156 ms in the best optimized branch.
Timing uses HTTP time to first content on Strix Halo. Each comparison uses identical model files, requests, image budget 280, microbatch size, and checkpoints/cache reuse disabled. A/B/B/A processes provide two warmups per process and ten retained samples per arm. The isolated changes are not additive percentages.
Validation: independent gfx11 build; 259/259 focused backend cases passed with the changes on and off. Coverage includes both RoPE axes, an extra consumer that must prevent fusion, an oversized grid, D512 query tails, masks, and multiple batches. Relevant sources compile with warnings treated as errors for gfx1151 and gfx942. These checks do not replace a model-quality evaluation.
This branch starts directly from gfx11 and does not depend on the existing D72/non-GQA attention, dedicated prefill attention, BLAS fusion, or tensor-traffic PRs. Those PRs edit nearby dispatch code, so their changes must be retained when resolving source conflicts. This groups catalog directions #17, #44, and #46 into one draft.
AI assistance: Codex helped with the implementation, validation, and description. The change is submitted to the AMD fork for human review.