Repository navigation
hip: add dedicated prefill and vision attention with mask classification - #204
Draft
liangliangchang wants to merge 5 commits into
Draft
liangliangchang wants to merge 5 commits into
liangliangchang wants to merge 5 commits into
Conversation
Classify mask tiles and specialize D256 query groups. Keep the new paths opt-in with conservative shape and precision guards. Assisted-by: Codex
Reserve classification scratch before node temporaries and release it after graph evaluation. Reuse leaf masks only when no graph node writes overlapping storage, and keep reuse on the main stream. Assisted-by: Codex
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
1 similar comment
Collaborator
|
🤖 Detected: Infrastructure-eligible GHA rerun(s) dispatched |
Assisted-by: Codex
Disable UI provisioning in the Ubuntu and Windows behavior builds so a changing external asset bundle cannot prevent the C++ and Python tests from running. Assisted-by: Codex
Replace the deprecated setter in the prefill mask reuse test with ggml_prec_set_acc. Both set the same FP32 accumulation parameter; the supported API also builds with fatal deprecation warnings. Assisted-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds opt-in RDNA3.5 attention kernels for vision and long-prompt language processing. Eligible attention calls reuse Q/K/V tiles and classify mask tiles as fully allowed, fully blocked, or mixed. Fully blocked tiles skip their attention work; fully allowed tiles avoid per-element mask loads. Other shapes keep the existing dispatch.
This PR combines catalog directions 12, 13, 18 and 62 because they share the dedicated kernel. It is based directly on
gfx11(f70aba1cb6) and has no dependency on the other optimization PRs. PR #195 improves the existing general WMMA path; this PR adds an opt-in dedicated alternative and does not copy or remove #195's changes. The mask-prefix fallback and mask-classification cache are excluded: they have no demonstrated additional benefit with this selected path, and the cache would introduce shared scratch-state coupling.The supported paths are unmasked, non-GQA D64/D72/D80 vision attention, and masked D64/D128/D256 prefill with GQA. Dispatch checks types, strides, alignment and shape. Attention with sinks, sparse indices, softcap, or ALiBi keeps the existing implementation.
Fresh results on Strix Halo (gfx1151), using Release HIP builds and the same model files and settings for both arms:
Language rows use
llama-benchprefill with batch size 4096, FP16 KV cache and flash attention. Microbatch size is 2048 for Qwen and 1024 for Gemma. VLM rows use HTTP time to first content token, microbatch 512, context 4096, the same image and prompt, 32 output tokens, checkpoints disabled, and prompt/RAM caching disabled. Each arm has two process starts with two warmups and five measured samples per start, ordered A/B/B/A. The table reports the median of the ten accepted samples. vLLM was not rerun for this PR.Enable with
GGML_CUDA_FATTN_PREFILL=1andGGML_CUDA_FATTN_VISION=1. The measurements also enableGGML_CUDA_FATTN_PREFILL_D256_SXH=1, which uses FP16 score exchange between warps while retaining FP32 accumulation. D256 defaults to eight query groups;GGML_CUDA_FATTN_PREFILL_D256_NG=4selects the smaller configuration. On Gemma-2B, score exchange alone saved about 0.8%; the eight-group configuration with score exchange saved about 3% against four groups with FP32 exchange. These gains overlap and should not be added.Validation: Release HIP build passed; 37/37 focused backend cases passed against the reference, covering mask classes, finite mask bias, tails, GQA, batches and fallback cases. Changed HIP code also passed warning-as-error compilation for gfx1151 and gfx942. The tests validate numerical error within their stated tolerance; they do not establish bitwise identity or replace a downstream model-quality evaluation. The new paths remain opt-in.
Assisted-by: Codex
Graph-scoped mask reuse
The combined PR comparison exposed repeated classification of the same immutable mask. This update classifies each eligible leaf mask once per graph and reuses its classes across attention layers. It rejects masks with overlapping graph writes, limits reuse to the main stream, and reserves the classification allocation before node scratch. There is no dependency on the MMDQ PR.
Fresh paired Qwen2.5-0.5B tests at 3968 tokens, two warmups and ten retained samples per arm:
A separate three-prefill trace reduced classification from 144 launches / 17.32 ms to 6 launches / 0.63 ms. The standalone end-to-end gain is small; the larger combined gain should not be interpreted as additive to every other optimization.
Validation: 49/49 focused attention cases passed with reuse enabled and disabled. The tests include repeated masks and computed-mask fallback. Relevant sources compile with warnings treated as errors for gfx1151 and gfx942. Set
GGML_CUDA_FATTN_PREFILL_MASK_CACHE=0to disable reuse; the dedicated prefill path remains opt-in throughGGML_CUDA_FATTN_PREFILL=1.