Repository navigation
hip: fuse Gemma scalar and GELU multiplications on RDNA3.5 - #209
Draft
liangliangchang wants to merge 1 commit into
Draft
liangliangchang wants to merge 1 commit into
liangliangchang wants to merge 1 commit into
Conversation
Preserve FP32 arithmetic and the existing numerical kernels. Fuse eligible GELU products and normalized residual scaling with consumer, layout, and alias checks. Retain fallbacks for unsupported graphs. Assisted-by: Codex
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
1 similar comment
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gemma-4 prefill launches separate elementwise kernels for GELU followed by multiplication in its per-layer embedding path, and for the scalar multiplication after a normalized residual. This change folds those operations into the existing gated-unary and RMSNorm kernels on RDNA3.5.
The two fusions keep FP32 storage and arithmetic. GELU fusion accepts matching shapes with supported row layouts, allows an intervening metadata-only view, and rejects extra consumers or unsafe aliases. Post-residual scaling preserves the order RMSNorm, channel weights, residual addition, then scalar multiplication. Its single-use chain checks and external-operand overlap checks preserve fallback for unsupported graphs. Exact contiguous output aliases are allowed; partial and broadcast overlaps are rejected.
Both fusions are enabled for eligible RDNA3.5 graphs.
GGML_CUDA_FUSE_GELU_MUL=0andGGML_CUDA_FUSE_RMS_POST_SCALE=0independently restore the existing paths. Current gfx11 already implements RMSNorm + SCALE, so this PR does not duplicate that optimization.Fresh paired measurements:
A diagnostic E2B request trace with graphs disabled confirms activation: broadcast multiplication launches fall from 278 to 9 and standalone unary launches from 137 to 5; 137 RMSNorm launches execute the post-residual scale variant. These launch counts are diagnostic and are not substituted for the normal-graph TTFT measurements.
The A4B result is small and should be treated as neutral. These are incremental savings after earlier optimizations, not additive percentages or a claim of a large independent speedup.
Timing uses HTTP time to first content on Strix Halo, with identical models, requests, image budget 280, microbatch size 512, and checkpoints/cache reuse disabled. Each row uses A/B/B/A processes, two warmups per process, and ten retained samples per arm. Loading and compilation are excluded.
Validation: independent gfx11 build; 112/112 focused numerical cases passed with both fusions enabled and disabled. Coverage includes scalar tails, in-place fallback, extra consumers, GELU operand order, padded/strided layouts, and unsupported broadcasts. The affected sources compile with warnings treated as errors for gfx1151 and gfx942. These checks do not replace a model-quality evaluation.
This branch starts directly from gfx11 and has no dependency on the other optimization PRs. It groups catalog directions #34 and #36 into one draft. The combined branch also validates their interaction with the existing BLAS and MMDQ fusions.
AI assistance: Codex helped with implementation, validation, and description. This draft is submitted to the AMD fork for human review.