Skip to content

hip: fuse Gemma scalar and GELU multiplications on RDNA3.5 - #209

Draft
liangliangchang wants to merge 1 commit into
gfx11from
lichang.gemma-elementwise-gfx11
Draft

liangliangchang wants to merge 1 commit into
gfx11from
lichang.gemma-elementwise-gfx11

Conversation

@liangliangchang

Copy link
Copy Markdown

Gemma-4 prefill launches separate elementwise kernels for GELU followed by multiplication in its per-layer embedding path, and for the scalar multiplication after a normalized residual. This change folds those operations into the existing gated-unary and RMSNorm kernels on RDNA3.5.

The two fusions keep FP32 storage and arithmetic. GELU fusion accepts matching shapes with supported row layouts, allows an intervening metadata-only view, and rejects extra consumers or unsafe aliases. Post-residual scaling preserves the order RMSNorm, channel weights, residual addition, then scalar multiplication. Its single-use chain checks and external-operand overlap checks preserve fallback for unsupported graphs. Exact contiguous output aliases are allowed; partial and broadcast overlaps are rejected.

Both fusions are enabled for eligible RDNA3.5 graphs. GGML_CUDA_FUSE_GELU_MUL=0 and GGML_CUDA_FUSE_RMS_POST_SCALE=0 independently restore the existing paths. Current gfx11 already implements RMSNorm + SCALE, so this PR does not duplicate that optimization.

Fresh paired measurements:

Model Configuration Disabled Enabled Time saved
Gemma-4-E2B This PR alone on gfx11 345.56 ms 343.90 ms 0.48%
Gemma-4-26B-A4B This PR alone on gfx11 803.34 ms 801.89 ms 0.18%
Gemma-4-E2B Combined optimization stack 270.01 ms 266.81 ms 1.19%

A diagnostic E2B request trace with graphs disabled confirms activation: broadcast multiplication launches fall from 278 to 9 and standalone unary launches from 137 to 5; 137 RMSNorm launches execute the post-residual scale variant. These launch counts are diagnostic and are not substituted for the normal-graph TTFT measurements.

The A4B result is small and should be treated as neutral. These are incremental savings after earlier optimizations, not additive percentages or a claim of a large independent speedup.

Timing uses HTTP time to first content on Strix Halo, with identical models, requests, image budget 280, microbatch size 512, and checkpoints/cache reuse disabled. Each row uses A/B/B/A processes, two warmups per process, and ten retained samples per arm. Loading and compilation are excluded.

Validation: independent gfx11 build; 112/112 focused numerical cases passed with both fusions enabled and disabled. Coverage includes scalar tails, in-place fallback, extra consumers, GELU operand order, padded/strided layouts, and unsupported broadcasts. The affected sources compile with warnings treated as errors for gfx1151 and gfx942. These checks do not replace a model-quality evaluation.

This branch starts directly from gfx11 and has no dependency on the other optimization PRs. It groups catalog directions #34 and #36 into one draft. The combined branch also validates their interaction with the existing BLAS and MMDQ fusions.

AI assistance: Codex helped with implementation, validation, and description. This draft is submitted to the AMD fork for human review.

Preserve FP32 arithmetic and the existing numerical kernels. Fuse eligible GELU products and normalized residual scaling with consumer, layout, and alias checks. Retain fallbacks for unsupported graphs.

Assisted-by: Codex
@jimw567

jimw567 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

🤖 /pr skill detected an infra issue. Restarting: GHA runs: AMD-Ecosystem/llama.cpp=43357774235

Detected: Infrastructure-eligible GHA rerun(s) dispatched

1 similar comment
@jimw567

jimw567 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

🤖 /pr skill detected an infra issue. Restarting: GHA runs: AMD-Ecosystem/llama.cpp=43357774235

Detected: Infrastructure-eligible GHA rerun(s) dispatched

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants