Skip to content

split FA for minimal 256 KV rows at long context for gfx11 family and… - #220

Open
Annieren wants to merge 1 commit into
gfx11from
annier.fa-split-rdna35
Open

Annieren wants to merge 1 commit into
gfx11from
annier.fa-split-rdna35

Conversation

@Annieren

Copy link
Copy Markdown

… number of wgp larger than 8

Overview

Original design on gfx1151 at long context it can pick about 100 splits. Each block then reads only a few dozen KV rows, which is too little to hide memory latency. The new changes work when the GPU is RDNA3.5, number of wgp ≥ 8 (≥ 16 CUs), single-token decode, and n_kv ≥ 512. Everything else, including other GPUs, prefill, short context and small RDNA3.5 parts like gfx1153, is unchanged.

Additional information

  1. Perplexity is within rounding
  2. Performance improvement:
model depth base t/s PR t/s speedup
Llama-2-7B 128 49.68 49.65 0.999x
Llama-2-7B 1920 36.71 37.63 1.025x
Llama-2-7B 4096 26.32 28.86 1.097x
Llama-3.1-8B 128 45.19 45.15 0.999x
Llama-3.1-8B 4096 39.13 40.79 1.042x
Qwen2.5-7B 128 48.20 48.18 1.000x
Qwen2.5-7B 4096 43.35 45.79 1.056x
Qwen2.5-7B 8000 39.19 43.46 1.109x
Qwen3-30B-A3B 128 87.90 87.85 0.999x
Qwen3-30B-A3B 4096 73.78 74.56 1.011x
Qwen3-4B 128 79.23 79.26 1.000x
Qwen3-4B 4096 60.34 65.21 1.081x
Qwen3-4B 8000 49.67 56.67 1.141x
Qwen3.5-9B 128 38.21 38.16 0.999x
Qwen3.5-9B 4096 37.06 37.21 1.004x
gemma-2b 128 121.22 121.22 1.000x
gemma-2b 4096 113.91 116.00 1.018x
gemma-3-12b 128 27.94 28.26 1.011x
gemma-3-12b 4096 24.88 25.65 1.031x

Requirements

Yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants