Skip to content

hip: fuse Gemma vision RoPE and enable D512 WMMA attention - #208

Draft
liangliangchang wants to merge 1 commit into
gfx11from
lichang.gemma-attention-gfx11
Draft

liangliangchang wants to merge 1 commit into
gfx11from
lichang.gemma-attention-gfx11

Conversation

@liangliangchang

Copy link
Copy Markdown

Adds two related RDNA3.5 attention improvements for Gemma-4: fuse RMSNorm, channel weights, and the two vision RoPE axes into one kernel; and enable the existing D512 4x8 WMMA attention tile for eligible language attention.

The two-axis fusion requires matching NeoX RoPE parameters, complementary offsets, FP32 tensors, and no extra consumers or unsafe aliases. The common case uses one row per block without grid-loop state or a trailing barrier. Tensors exceeding the grid limits retain the existing loops and synchronization. This avoids the register increase that made the initial port slower: the E2B vision kernel returns from 48 to 32 VGPRs, with no scratch spills.

D512 selection is limited to RDNA3.5, GQA ratio 8, at least 32 queries, FP16 K/V, and zero logit softcap. It uses the existing WMMA implementation and keeps Q outside registers for this tile. Other attention shapes retain their current selection.

Controls: GGML_CUDA_FUSE_NORM_ROPE_2AXIS=0, GGML_CUDA_NORM_ROPE_SMALL_BLOCK=0, and GGML_CUDA_FATTN_WMMA_D512=0 independently disable the corresponding changes. The shape guards apply when enabled.

Fresh standalone results:

Model / configuration Disabled Enabled Time saved
Gemma-4-E2B, this PR alone on gfx11 346.48 ms 336.89 ms 2.77%
Gemma-4-26B-A4B, this PR alone on gfx11 803.41 ms 785.76 ms 2.20%

In the combined optimization stack, D512 WMMA alone saves 1.51% on E2B. Specializing the two-axis grid further saves 1.44% on E2B and 1.42% on A4B. A retained-prefill trace measures the final E2B two-axis kernel at 7.116 ms versus 7.156 ms in the best optimized branch.

Timing uses HTTP time to first content on Strix Halo. Each comparison uses identical model files, requests, image budget 280, microbatch size, and checkpoints/cache reuse disabled. A/B/B/A processes provide two warmups per process and ten retained samples per arm. The isolated changes are not additive percentages.

Validation: independent gfx11 build; 259/259 focused backend cases passed with the changes on and off. Coverage includes both RoPE axes, an extra consumer that must prevent fusion, an oversized grid, D512 query tails, masks, and multiple batches. Relevant sources compile with warnings treated as errors for gfx1151 and gfx942. These checks do not replace a model-quality evaluation.

This branch starts directly from gfx11 and does not depend on the existing D72/non-GQA attention, dedicated prefill attention, BLAS fusion, or tensor-traffic PRs. Those PRs edit nearby dispatch code, so their changes must be retained when resolving source conflicts. This groups catalog directions #17, #44, and #46 into one draft.

AI assistance: Codex helped with the implementation, validation, and description. The change is submitted to the AMD fork for human review.

Keep the existing grid loops for oversized tensors and specialize one-row-per-block vision RoPE. Enable the existing D512 4x8 tile on RDNA3.5 with FP16 KV and no logit softcap.

Assisted-by: Codex
@jimw567

jimw567 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

🤖 /pr skill detected an infra issue. Restarting: GHA runs: AMD-Ecosystem/llama.cpp=43354086638

Detected: Infrastructure-eligible GHA rerun(s) dispatched

1 similar comment
@jimw567

jimw567 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

🤖 /pr skill detected an infra issue. Restarting: GHA runs: AMD-Ecosystem/llama.cpp=43354086638

Detected: Infrastructure-eligible GHA rerun(s) dispatched

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants