Repository navigation
ggml-cuda: fuse FP16/BF16 BLAS producers and outputs on RDNA3.5 - #201
Open
liangliangchang wants to merge 2 commits into
Open
liangliangchang wants to merge 2 commits into
liangliangchang wants to merge 2 commits into
Conversation
Keep compatible FP16/BF16 FFN intermediates in pooled BLAS buffers and combine conversion with bias, residual, GLU/GELU, RMSNorm, and clamp work. Preserve GEMM precision, FP32 rounding boundaries, existing dispatch fallbacks, and gfx11 normalization/output-stride fixes. Include runtime family and component switches plus graph tests for precision overrides, extra consumers, mixed types, odd sizes, and the normalized, plain, gated, and clamped FFN patterns. Assisted-by: Codex
liangliangchang
marked this pull request as ready for review
November 12, 2026 22:13
This was referenced Nov 13, 2026
…sions-gfx11 Co-Authored-By: Claude Code <noreply@anthropic.com> # Conflicts: # ggml/src/ggml-cuda/ggml-cuda.cu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eligible BLAS graphs write FP32 intermediates, run separate bias/activation/clamp passes, and convert back to the 16-bit format required by the next GEMM. This change combines those passes and keeps internal FFN buffers in the format BLAS already uses. Across six VLMs, end-to-end TTFT falls by 4.84%-9.04%, with a 1.068x geometric-mean speedup over clean
gfx11.Changes
These are twelve related directions from the optimization branch, including the FP16/BF16 parent paths and their refinements. The GEMMs remain separate library calls. Existing GEMM precision settings and FP32 arithmetic rounding boundaries are preserved, and final graph tensors remain FP32.
Dispatch is limited to eligible RDNA3.5 contiguous 2D BLAS graphs with F16/BF16 weights, at least 32 rows, default precision, and no compute override. Existing MMVF/MMF choices, quantized GEMMs, unsupported layouts, and graphs with extra intermediate consumers retain their fallback paths.
GGML_CUDA_FUSE_CUBLAS_16BIT=0disables the complete family; component controls support individual ablation.Performance
Fresh end-to-end HTTP TTFT to the first content token, comparing clean
gfx11with this PR alone. The other window and attention optimizations are absent from both builds.Each pair uses the same Strix Halo board, model, image, prompt, and settings. Results are medians of 10 samples per variant: two server starts per variant in A/B/B/A order, with two warmups and five measured requests per start. Profiled requests are excluded.
Release HIP on gfx1151, TheRock
10.1.0a20260731, HIP graphs enabled, VMM/RCCL/unsafe math disabled. Common settings:-ngl 999 -t 8 -fa on -c 4096 -b 2048 -ub 512 -np 1 -ctk f16 -ctv f16 --ctx-checkpoints 0 --no-cache-prompt --cache-ram 0. Temperature is zero, with up to 32 output tokens; cached prompt positions were verified zero. Gemma4 uses an image-token budget of 280 in both builds. Gemma3-12B uses a 640x480 image; the other models use 1024x800.Validation and limits
-Werror -Wno-tautological-compareusing the local SDK; this does not represent the full CI matrix.Private pooled buffers reduce memory traffic and launches but can add scratch allocation. Model-wide activation precision, KV-cache precision, attention, and checkpoint behavior are unchanged.
Relationship to earlier PRs
Neither #192 (Qwen window batching/copy avoidance) nor #195 (D64/D72/D80 WMMA attention) is required or duplicated. All results above were measured directly on
gfx11without either PR.A dry-run merge with #195 is clean. #192 has a textual conflict because both PRs add independent fusion handlers; resolve it by retaining both complete handlers. There is no code dependency or required merge order. Performance interactions mean the separate gains should not be added arithmetically.
Codex assisted with extraction, adaptation, and validation; the commit includes an
Assisted-by: Codextrailer.