Skip to content

ggml-cuda: fuse FP16/BF16 BLAS producers and outputs on RDNA3.5 - #201

Open
liangliangchang wants to merge 2 commits into
gfx11from
lichang.fp16-blas-fusions-gfx11
Open

liangliangchang wants to merge 2 commits into
gfx11from
lichang.fp16-blas-fusions-gfx11

Conversation

@liangliangchang

Copy link
Copy Markdown

Eligible BLAS graphs write FP32 intermediates, run separate bias/activation/clamp passes, and convert back to the 16-bit format required by the next GEMM. This change combines those passes and keeps internal FFN buffers in the format BLAS already uses. Across six VLMs, end-to-end TTFT falls by 4.84%-9.04%, with a 1.068x geometric-mean speedup over clean gfx11.

Changes

  • Share the converted input between gated FFN up/gate GEMMs and retain compatible gated, plain GELU, and clamped FFN intermediates in pooled FP16/BF16 buffers.
  • Fuse RMSNorm plus weight multiplication into the BLAS input conversion.
  • Fuse output conversion with bias and optional residual addition.
  • Fuse GLU/GELU/clamp with the required conversions, with paired loads/stores and scalar/tail fallbacks.

These are twelve related directions from the optimization branch, including the FP16/BF16 parent paths and their refinements. The GEMMs remain separate library calls. Existing GEMM precision settings and FP32 arithmetic rounding boundaries are preserved, and final graph tensors remain FP32.

Dispatch is limited to eligible RDNA3.5 contiguous 2D BLAS graphs with F16/BF16 weights, at least 32 rows, default precision, and no compute override. Existing MMVF/MMF choices, quantized GEMMs, unsupported layouts, and graphs with extra intermediate consumers retain their fallback paths. GGML_CUDA_FUSE_CUBLAS_16BIT=0 disables the complete family; component controls support individual ablation.

Performance

Fresh end-to-end HTTP TTFT to the first content token, comparing clean gfx11 with this PR alone. The other window and attention optimizations are absent from both builds.

Model Projector weights Prompt positions gfx11 (ms) This PR (ms) TTFT reduction
Qwen2.5-VL-3B FP16 1606 2245.61 2127.12 5.28%
Qwen3-VL-4B FP16 1322 1023.54 974.03 4.84%
Gemma3-4B FP16 779 1027.79 964.24 6.18%
Gemma3-12B FP16 367 1211.52 1146.06 5.40%
Gemma4-E2B BF16 775 354.63 328.06 7.49%
Gemma4-26B-A4B FP16 779 803.42 730.77 9.04%

Each pair uses the same Strix Halo board, model, image, prompt, and settings. Results are medians of 10 samples per variant: two server starts per variant in A/B/B/A order, with two warmups and five measured requests per start. Profiled requests are excluded.

Release HIP on gfx1151, TheRock 10.1.0a20260731, HIP graphs enabled, VMM/RCCL/unsafe math disabled. Common settings: -ngl 999 -t 8 -fa on -c 4096 -b 2048 -ub 512 -np 1 -ctk f16 -ctv f16 --ctx-checkpoints 0 --no-cache-prompt --cache-ram 0. Temperature is zero, with up to 32 output tokens; cached prompt positions were verified zero. Gemma4 uses an image-token budget of 280 in both builds. Gemma3-12B uses a 640x480 image; the other models use 1024x800.

Validation and limits

  • 166/166 CPU-reference graph tests passed, covering FP16/BF16, odd dimensions, scalar/vector paths, small-batch fallback, extra consumers, explicit F32 precision, mixed weight types, swapped gates, residuals, and normalized/plain/clamped FFNs.
  • 166/166 deterministic fusion-on/off cases matched bit-for-bit across 4,693,884 final FP32 values; maximum absolute error and normalized MSE were zero.
  • Five models produced an identical response across builds. Gemma4-E2B has pre-existing response instability, alternating between describing the image and asking for it; both builds produced the same sequence and both response variants improved in TTFT. These checks are targeted validation, not a full task-accuracy evaluation.
  • Kernel traces confirm the FP16/BF16 fusion paths execute; profiled new helpers use at most 48 VGPRs and zero scratch.
  • Release HIP build passed for gfx1151. The four changed HIP translation units also compiled for gfx942 with -Werror -Wno-tautological-compare using the local SDK; this does not represent the full CI matrix.

Private pooled buffers reduce memory traffic and launches but can add scratch allocation. Model-wide activation precision, KV-cache precision, attention, and checkpoint behavior are unchanged.

Relationship to earlier PRs

Neither #192 (Qwen window batching/copy avoidance) nor #195 (D64/D72/D80 WMMA attention) is required or duplicated. All results above were measured directly on gfx11 without either PR.

A dry-run merge with #195 is clean. #192 has a textual conflict because both PRs add independent fusion handlers; resolve it by retaining both complete handlers. There is no code dependency or required merge order. Performance interactions mean the separate gains should not be added arithmetically.

Codex assisted with extraction, adaptation, and validation; the commit includes an Assisted-by: Codex trailer.

Keep compatible FP16/BF16 FFN intermediates in pooled BLAS buffers and
combine conversion with bias, residual, GLU/GELU, RMSNorm, and clamp work.
Preserve GEMM precision, FP32 rounding boundaries, existing dispatch
fallbacks, and gfx11 normalization/output-stride fixes.

Include runtime family and component switches plus graph tests for
precision overrides, extra consumers, mixed types, odd sizes, and the
normalized, plain, gated, and clamped FFN patterns.

Assisted-by: Codex
…sions-gfx11

Co-Authored-By: Claude Code <noreply@anthropic.com>

# Conflicts:
#	ggml/src/ggml-cuda/ggml-cuda.cu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant