Skip to content

hip: accelerate D64/D72/D80 vision attention with WMMA - #195

Open
liangliangchang wants to merge 5 commits into
gfx11from
lichang.gemma3-d72-wmma-gfx11
Open

liangliangchang wants to merge 5 commits into
gfx11from
lichang.gemma3-d72-wmma-gfx11

Conversation

@liangliangchang

@liangliangchang liangliangchang commented Nov 11, 2026 •

Copy link
Copy Markdown

This change enables the existing WMMA attention kernel for eligible unmasked D64/D72/D80 vision heads on RDNA3.5. Native D64/D80 inputs use the existing kernel directly. D72 inputs are padded to D80, processed with their original attention scale, and cropped back to D72. Complete dense KV tiles skip redundant partial-tile bounds handling; the final tile retains its checks.

Implementation

  • Enable the existing 64-query, one-head WMMA specialization for D64 and D80.
  • Route eligible native D64/D80 self-attention to that specialization without padding or extra copies.
  • Adapt D72 by appending eight zero channels to Q/K/V, keeping the original scale, and copying the first 72 output channels back. For Gemma-3 the scale remains 1/sqrt(72). Its 4096 vision patch positions still attend globally.
  • Keep one shared full-tile bounds optimization for D64/D80. D80 tuning remains enabled, including for D72-padded Gemma inputs.
  • Keep existing fallback paths for unsupported shapes, masks, GQA, sinks, indexed KV, bias/softcap and other GPU architectures. Native dispatch requires at least 64 positions; D72 requires at least 256 positions and 4096 total query rows.
  • Keep both dispatch controls independent. Disabling native D64/D80 dispatch does not compile out the D80 kernel required by the D72 adapter.

The path requires equal Q/K/V sequence lengths, head counts and batch dimensions; FP32 Q/output and FP16 K/V; channel-contiguous inputs and contiguous output. D72 row counts are bounded before allocation and index narrowing. The existing WMMA configuration, null-mask support and D80 FP32 accumulators are reused. Q/output storage remains FP32 and K/V remains FP16.

The D72 adapter uses pooled temporary buffers. For 4096 positions and 16 heads it adds three padding launches, one crop launch and 60 MiB of logical temporary storage, reused across layers. Native D64/D80 needs none of those adapter copies. Window batching, QKV fusion, language GEMM and checkpoint optimizations are separate from this change.

Combined branch: fresh end-to-end TTFT

Combined commit ec56b7427 is compared with clean gfx11 f70aba1cb. All rows below are fresh measurements of this combined implementation with D64 and D80 full-tile tuning enabled. Each model uses the same board, input, model/projector files and server settings for both builds.

Model Vision head width Prompt positions gfx11 TTFT, ms Combined TTFT, ms TTFT reduction Board
Gemma-3-4B-IT 72 779 1029.64 667.70 35.15% Halo06
Gemma-3-12B-IT 72 367 1144.24 788.10 31.12% Halo06
Qwen3-VL-4B-Instruct 64 1322 1020.10 836.41 18.01% Halo05
Qwen2.5-VL-3B-Instruct 80 1606 2242.25 2166.48 3.38% Halo05
Qwen2.5-VL-7B-Instruct 80 1606 2703.89 2621.59 3.04% Halo05
Gemma-4-E2B-IT 64 775 347.63 312.09 10.22% Halo06
Gemma-4-26B-A4B-IT 72 779 829.85 716.41 13.67% Halo41
Gemma-4-31B-IT 72 779 2616.86 2504.01 4.31% Halo06
Gemma-4-31B-IT OpenNav 72 473 1676.98 1657.01 1.19% Halo41
Qwen3.5-35B-A3B 72 414 407.64 381.05 6.52% Halo41
Qwen3-Omni-30B-A3B-Instruct 72 1322 1324.70 1093.07 17.49% Halo41
MiniCPM-o-2_6 72 1004 1433.27 1280.68 10.65% Halo05

Weights are Q4_K_M, except Gemma-4-26B-A4B uses UD-Q4_K_M and Gemma-4-E2B uses Q4_0. The projector files and original image dimensions are recorded in input_conditions.json. BF16 projector weights still produce the FP32 Q / FP16 K/V tensors required by this attention path.

Timing is HTTP time to the first content token from llama-server. Each value is the median of ten measured requests across two server starts, with two warmups per start. Order per model: gfx11 / combined / combined / gfx11. No profiled requests or historical timing values are mixed into this table. All accepted samples, including outliers, are retained. Samples with detected competing GPU/CPU activity are rejected and retried.

Full GPU offload, 8 CPU threads, flash attention on, context 4096, batch 2048, microbatch 512, one server slot, FP16 K/V cache, 32-token greedy output cap, HIP graphs on, unsafe math off. Checkpoints and prompt reuse are disabled: --ctx-checkpoints 0 --no-cache-prompt, request cache_prompt=false. Every accepted request has zero reused prompt positions. RAM prompt-cache capacity is left at the server default; these runs do not pass --cache-ram 0. Native image/text boundaries are preserved.

Gemma-4 variants use the same --image-max-tokens 280 on both builds. Gemma-3 retains its normal 896 x 896 vision input and 4096 patch positions, producing 256 image embeddings. The images/messages are the saved original workloads for each model; prompt-position counts and model/projector/request hashes are checked for equality across the two builds. These are per-model comparisons, not comparisons between different model workloads.

Qwen2.5-VL still uses the original masked path for its window-attention layers. This PR accelerates its unmasked full-attention layers. Combining it with the separate window-batching PR may expose additional eligible calls; that combination is not part of these results. MiniCPM also contains other attention head widths, which retain their existing path.

Controls and isolated tuning evidence

Control Default Scope
GGML_CUDA_FATTN_WMMA_D72 1 Compile-time and runtime switch for D72 padding dispatch.
GGML_CUDA_FATTN_WMMA_NO_GQA 1 Compile-time and runtime switch for native D64/D80 dispatch.
GGML_CUDA_FATTN_D64_FULL_TILE 1 Compile-time switch for complete D64 tiles.
GGML_CUDA_FATTN_D80_FULL_TILE 1 Compile-time switch for complete D80 tiles, including padded D72.

The two full-tile controls must be set at compilation, for example through CMAKE_HIP_FLAGS; process environment variables cannot change compiled loop bodies. Runtime dispatch switches are read once per process.

The following are earlier isolated measurements, retained to explain the tuning choices. They are not additional samples in the fresh combined table:

Isolated addition Model Before, ms After, ms Incremental effect
D80 full-tile tuning on D72 padding Gemma-3-4B, Halo06 684.05 666.24 2.60% lower TTFT
D64 full-tile tuning on native WMMA Qwen3-VL-4B, Halo05 857.55 839.36 2.12% lower TTFT
D80 full-tile tuning on native WMMA Qwen2.5-VL-3B, Halo05 2162.00 2170.03 0.37% higher TTFT

D80 full-tile tuning stays enabled to preserve the measured Gemma benefit. The small native Qwen2.5 penalty does not eliminate the gain from WMMA dispatch. There is one shared implementation and one D80 default; this change does not reverse the earlier D80 tuning. These timings alone do not identify the compiler/resource cause of the native D80 difference.

Validation and numerical behavior

The combined Release HIP build passes for llama-server, llama-bench and test-backend-ops. All 21 focused CPU-reference comparisons pass: D64/D80 dispatch boundaries, partial tiles and permuted batches, plus D72 thresholds and feature fallbacks. The existing test tolerance is unchanged. Separate dispatcher builds with each dispatch macro disabled pass; the affected template also builds with native dispatch disabled, preserving D72 support. These checks use -Werror with the recorded suppression for existing tautological-compare warnings. Source whitespace checks pass.

Fresh generated-output comparison for the combined branch:

Model Generated text versus gfx11 on these requests
Gemma-3-4B-IT Different; outputs retained in the run artifacts
Gemma-3-12B-IT Identical
Qwen3-VL-4B-Instruct Identical
Qwen2.5-VL-3B-Instruct Identical
Qwen2.5-VL-7B-Instruct Different; outputs retained in the run artifacts
Gemma-4-E2B-IT Different; outputs retained in the run artifacts
Gemma-4-26B-A4B-IT Different; outputs retained in the run artifacts
Gemma-4-31B-IT Identical
Gemma-4-31B-IT OpenNav Identical
Qwen3.5-35B-A3B Different; outputs retained in the run artifacts
Qwen3-Omni-30B-A3B-Instruct Different; outputs retained in the run artifacts
MiniCPM-o-2_6 Identical

One result needs explicit qualification: Gemma-4-E2B gfx11 samples alternate between describing the image and asking for an image, while all combined samples ask for an image on this repeated-prompt workload. The output check does not establish equivalent image understanding for this model. The saved responses make this visible; the TTFT improvement should not be presented as proof of unchanged model quality.

Existing numerical evidence for the unchanged D72 adapter remains applicable to that adapter: a controlled 4096-position, 16-head D72 probe matched the original optimization branch bitwise. Against FP64 stable-softmax reference, relative L2 error was 0.03620% for padded D80 WMMA versus 0.54033% for the generic D72 kernel; maximum absolute error was 0.00010253 versus 0.00188810. The D80 full-tile-only addition previously produced bitwise-identical tested attention outputs and Gemma-3-4B vision embeddings.

The earlier Gemma-3-4B request changed one generated word at token 28, also reproduced by the original optimization branch. End-of-vision embeddings differed from the generic path by about 3% relative L2 after the complete tower, despite lower attention error on the FP64 probe. Numerical differences can alter greedy output; neither those earlier checks nor the fresh text comparisons establish unchanged task accuracy across a broad dataset. The current tests support operator correctness on the tested shapes.

Scope and provenance

Catalog directions: #19 native non-GQA D64/D80 WMMA, #20 D72-to-D80 adaptation, #58 D80 full-tile tuning, plus the D64 extension of the full-tile rule. These share the same existing attention implementation. No dedicated replacement attention kernel is added.

Branch: lichang.gemma3-d72-wmma-gfx11. Existing PR: #195.

AI assistance: Codex assisted with implementation integration, test execution and this description.

@liangliangchang
liangliangchang marked this pull request as ready for review November 12, 2026 00:46
Preserve D80 full-tile tuning and extend it to D64. Keep the native dispatch switch independent of the D72 adapter.

Assisted-by: Codex
@liangliangchang liangliangchang changed the title hip: accelerate D72 vision attention with D80 WMMA hip: accelerate D64/D72/D80 vision attention with WMMA Nov 12, 2026
Account for the KV_rows_ptr argument added in 1b6935b. Update only the 20 existing kernel specializations reported by the gfx908 check; keep the spill threshold unchanged.

Assisted-by: Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant