Skip to content

hip: reduce tensor traffic and tune small RoPE launches - #206

Draft
liangliangchang wants to merge 1 commit into
gfx11from
lichang.rdna35-tensor-traffic-gfx11
Draft

liangliangchang wants to merge 1 commit into
gfx11from
lichang.rdna35-tensor-traffic-gfx11

Conversation

@liangliangchang

Copy link
Copy Markdown

Reduces tensor traffic and launch overhead in eligible normalization, strided copies, RoPE and expert reduction operations. LayerNorm, affine weight and bias execute in one pass; strided FP32-to-FP16 copies move eight values per thread; small norm/RoPE and standalone RoPE use smaller blocks; common MoE reductions process four columns per thread.

This branch starts directly from gfx11 (f70aba1cb6) and builds and improves performance independently. It does not require the MMDQ, dedicated-attention, window-batching or BLAS-fusion PRs. When integrating it with #201, the larger BLAS producer fusion should retain priority over standalone LayerNorm affine fusion; their gains overlap.

Fresh HTTP time-to-first-content-token results from the final build on Strix Halo:

Model Input gfx11 (ms) This PR (ms) Time saved
Qwen2.5-VL-3B 1606 positions 2247.14 2224.37 1.0%
Gemma-3-4B 779 positions 1027.67 1014.48 1.3%
Gemma-4-26B-A4B 779 positions 833.34 805.92 3.3%
Qwen3.5-35B-A3B 414 positions 399.22 385.79 3.4%

Each model pair used one board, identical model/image/prompt inputs, microbatch 512, context 4096, 32 output tokens, no checkpoints, and no prompt/RAM cache. Gemma4 used image budget 280. Each arm has two process starts, two warmups and five retained samples per start, ordered A/B/B/A. Values are medians of ten samples. vLLM was not rerun.

Component ablations from separately preserved extraction builds used on/off/off/on, two warmups and seven retained samples per start. These conditional gains are not additive:

Component Workload Time saved when enabled
LayerNorm affine fusion Gemma-3-4B VLM 0.71%
Strided copy vectorization, without early K/V casting Qwen2.5-VL-3B 0.34%
One-warp norm/RoPE Qwen3-4B, 3968-token llama-bench prefill 2.24%
Standalone RoPE block tuning Qwen2.5-VL-3B 0.79%
Vectorized MoE reduction Qwen3.5-35B-A3B VLM, 414 positions 1.59%

The copy and affine gains are small. They were retained because both process-start comparisons had the same direction. Their benefit may change when combined with other fusions.

Rejected candidates are excluded from this branch: early vision K/V casting was 0.26% slower; Gemma two-axis RoPE was slower with 256-, 128- and 32-thread variants; CPU image preprocessing threads did not show a reliable gain. The older generic RMSNorm and ordinary FP16 conversion changes are superseded by current gfx11 code and were not copied.

LayerNorm/copy/automatic RoPE dispatch is limited to eligible RDNA3.5 shapes, layouts and alignment. Existing large-grid loops and synchronization are preserved. The LayerNorm fusion retains FP32 intermediate rounding. The copy converts to the same FP16 destination format as the original operation; this PR does not broadly change activation precision.

The measured configuration sets GGML_CUDA_NORM_ROPE_WARP=1, GGML_CUDA_ROPE_BLOCK=64 and GGML_CUDA_MOE_REDUCE_X4=1; those explicit launch options are opt-in. Eligible automatic paths use GGML_CUDA_FUSE_NORM_MUL_ADD, GGML_CUDA_CPY_F32_F16_VEC8 and GGML_CUDA_ROPE_SMALL_BLOCK, accepting 0 to disable them for ablation.

Validation: Release HIP build passed; 982/982 focused backend cases passed for copies, RoPE, norm/RoPE, LayerNorm affine and MoE reduction. Added LayerNorm cases cover width tails, broadcast affine parameters and extra graph consumers. All changed HIP units passed warning-as-error compilation for gfx1151 and gfx942. These are numerical kernel tests, not a downstream model-quality evaluation.

Assisted-by: Codex

@jimw567

jimw567 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

🤖 /pr skill detected an infra issue. Restarting: GHA runs: AMD-Ecosystem/llama.cpp=43337881360

Detected: Infrastructure-eligible GHA rerun(s) dispatched

1 similar comment
@jimw567

jimw567 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

🤖 /pr skill detected an infra issue. Restarting: GHA runs: AMD-Ecosystem/llama.cpp=43337881360

Detected: Infrastructure-eligible GHA rerun(s) dispatched

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants