Repository navigation
hip: reduce tensor traffic and tune small RoPE launches - #206
Draft
liangliangchang wants to merge 1 commit into
Draft
liangliangchang wants to merge 1 commit into
liangliangchang wants to merge 1 commit into
Conversation
Assisted-by: Codex
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
1 similar comment
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduces tensor traffic and launch overhead in eligible normalization, strided copies, RoPE and expert reduction operations. LayerNorm, affine weight and bias execute in one pass; strided FP32-to-FP16 copies move eight values per thread; small norm/RoPE and standalone RoPE use smaller blocks; common MoE reductions process four columns per thread.
This branch starts directly from
gfx11(f70aba1cb6) and builds and improves performance independently. It does not require the MMDQ, dedicated-attention, window-batching or BLAS-fusion PRs. When integrating it with #201, the larger BLAS producer fusion should retain priority over standalone LayerNorm affine fusion; their gains overlap.Fresh HTTP time-to-first-content-token results from the final build on Strix Halo:
Each model pair used one board, identical model/image/prompt inputs, microbatch 512, context 4096, 32 output tokens, no checkpoints, and no prompt/RAM cache. Gemma4 used image budget 280. Each arm has two process starts, two warmups and five retained samples per start, ordered A/B/B/A. Values are medians of ten samples. vLLM was not rerun.
Component ablations from separately preserved extraction builds used on/off/off/on, two warmups and seven retained samples per start. These conditional gains are not additive:
The copy and affine gains are small. They were retained because both process-start comparisons had the same direction. Their benefit may change when combined with other fusions.
Rejected candidates are excluded from this branch: early vision K/V casting was 0.26% slower; Gemma two-axis RoPE was slower with 256-, 128- and 32-thread variants; CPU image preprocessing threads did not show a reliable gain. The older generic RMSNorm and ordinary FP16 conversion changes are superseded by current gfx11 code and were not copied.
LayerNorm/copy/automatic RoPE dispatch is limited to eligible RDNA3.5 shapes, layouts and alignment. Existing large-grid loops and synchronization are preserved. The LayerNorm fusion retains FP32 intermediate rounding. The copy converts to the same FP16 destination format as the original operation; this PR does not broadly change activation precision.
The measured configuration sets
GGML_CUDA_NORM_ROPE_WARP=1,GGML_CUDA_ROPE_BLOCK=64andGGML_CUDA_MOE_REDUCE_X4=1; those explicit launch options are opt-in. Eligible automatic paths useGGML_CUDA_FUSE_NORM_MUL_ADD,GGML_CUDA_CPY_F32_F16_VEC8andGGML_CUDA_ROPE_SMALL_BLOCK, accepting0to disable them for ablation.Validation: Release HIP build passed; 982/982 focused backend cases passed for copies, RoPE, norm/RoPE, LayerNorm affine and MoE reduction. Added LayerNorm cases cover width tails, broadcast affine parameters and extra graph consumers. All changed HIP units passed warning-as-error compilation for gfx1151 and gfx942. These are numerical kernel tests, not a downstream model-quality evaluation.
Assisted-by: Codex