Skip to content

Port the persistent BatchAttention kernel to ROCm (#101) - #404

Open
PMukund219 wants to merge 1 commit into
AMD-Ecosystem:amd-integrationfrom
PMukund219:persistent-batch-attention
Open

PMukund219 wants to merge 1 commit into
AMD-Ecosystem:amd-integrationfrom
PMukund219:persistent-batch-attention

Conversation

@PMukund219

Copy link
Copy Markdown

Summary

Ports upstream's persistent BatchAttention (flashinfer-ai#1137) to ROCm for #101: mixed prefill + decode batches run on one fixed grid, so one captured graph serves every batch shape. Upstream's flashinfer/attention/_core.py runs unchanged on top of a new HIP kernel, planner, binding and JIT generator.

What changed

  • include/flashinfer/rocm/attention/persistent.cuh — new MFMA 16×16×16 wave64 kernel.
  • include/flashinfer/rocm/attention/scheduler.cuh — upstream's TwoStageHolisticPlan / HolisticPlanInfo, appended.
  • csrc/rocm/batch_attention*.cu, batch_attention_customize_config.jinja — plan/run binding with the argument lists _core.py passes, plus tensor validation.
  • flashinfer/jit/rocm/modules.py — gen_batch_attention_module; refuses fp8/NVFP4 KV, the profiler, positional encoding, unsupported head dims and the attention-sink variant before compiling.
  • flashinfer/rocm/__init__.py, flashinfer/rocm/api.py, flashinfer/jit/rocm/api.py, flashinfer/jit/attention/__init__.py — un-gate flashinfer.attention and re-export.
  • tests/rocm/test_batch_attention.py — 200 cases; benchmarks/rocm/bench_mixed_attention.py — BatchAttention arm; arch_caps.py, README.md, docs/rocm/backends.md — docs and support matrix.

Architecture / design notes

Choice Why
Sᵀ = K·Qᵀ, Oᵀ = Vᵀ·Pᵀ MFMA's C layout is the B layout Pᵀ needs, so P stays in registers (no LDS transpose).
Two launches instead of cooperative grid.sync() On ROCm 7.2 a cooperative launch from a torch extension segfaults the process at exit; a minimal hipLaunchCooperativeKernel + grid.sync() kernel, unrelated to FlashInfer, reproduces it once torch has used the GPU. Stream order is the barrier; both launches are in the same graph.
fp32 partials, capacity capped to the workspace Upstream's bound overflows the 384 MB workspace at 304 CTAs (MI300X) × 8 KV heads; a plan needing more now raises instead of writing out of bounds.
Own holistic_packed_causal_kv_end It is upstream's current packed_causal_kv_end (ceiling, clamped), which the kernel matches. The fork's function of that name, used by the MLA planner, floors and does not clamp; MLA is left untouched.

Benchmark results

MI300X, vllm/vllm-openai-rocm:v0.29.0 image (ROCm 7.2.3, torch 2.12), bf16, causal; baseline is the in-tree fa2 prefill + decode wrappers. Not compared with AITER: the image's AITER 0.1.19 is below the 0.1.21 floor, so auto fell back to fa2.

Benchmark BatchAttention speedup vs separate prefill + decode
benchmarks/bench_batch_attention.py (Qwen-8B / Llama-3.1-70B / Qwen-MoE-235B; 128 decodes + one 4k prefill chunk) 1.88× / 1.84× / 2.38× mixed; 1.61–2.15× decode-only; 1.78–2.40× prefill-only
benchmarks/rocm/bench_mixed_attention.py (1k/4k prefill + 1–128 decodes) median 1.50× (0.91–1.73×; the one loss is 1k prefill + 128 decodes); POD is 0.65×

bench_mixed_attention.py --accuracy: matches the separate wrappers to one bf16 rounding step. Untuned (no async KV pipeline, one CTA per CU).

Test plan

GPU runs on MI300X in the image above, at base 9c078ef4; the patch applies cleanly on ba03cba5. Since the GPU run the only changes are pre-commit formatting and the removal of two unused helpers and unused includes; the compiled kernels are identical before and after.

  • tests/rocm/test_batch_attention.py — 200/200 (upstream's workload mixes, every axis value, split-KV, graph replay across batch shapes, refusals)
  • Existing tests covering what this PR touches (imports, API parity, module gating, JIT codegen): test_api_parity_runtime, test_package_imports, test_comm_import_gate, test_rocm_dispatch, test_jit_env, test_modules_hip_codegen, test_customize_prefill_signature, test_redistribution_licences, test_benchmark_harness, test_rocm_small_surface — pass
  • Compile-only, ROCm 10.1.0 and 7.2.3, gfx942 and gfx950
  • Not run on a GPU: the full pytest -m "not slow" suite, the ROCm 10.1 dev image, gfx950, AITER comparison
  • pre-commit run -a

Upstream's flashinfer.BatchAttention (the persistent "holistic" kernel,
flashinfer-ai#1137) runs a mixed prefill + decode batch on one fixed grid, so
one captured graph serves every batch shape. On ROCm the module was gated as
CUDA-only. This adds the HIP kernel, planner, binding and JIT generator so
upstream's Python class runs unchanged on gfx942 / gfx950.

- include/flashinfer/rocm/attention/persistent.cuh: MFMA 16x16x16 wave64
  kernel. Computes S^T = K Q^T and O^T = V^T P^T so the softmax probabilities
  feed the next MFMA from registers. Attention and the merge of split-KV
  partials are two ordinary launches: a cooperative grid.sync() launch from a
  torch extension crashes the process at exit on ROCm 7.2.
- scheduler.cuh: TwoStageHolisticPlan / HolisticPlanInfo. fp32 partials with a
  capacity capped to the float workspace and checked (upstream's bound
  overflows 384 MB at 304 CTAs and 8 KV heads). Own causal-end helper; the
  existing packed_causal_kv_end (MLA) uses an older formula.
- csrc/rocm/batch_attention*.cu, config template: plan/run with the argument
  lists _core.py passes; tensor validation; NVFP4 and soft-cap mismatch refused.
- jit/rocm/modules.py: gen_batch_attention_module, refusing fp8/NVFP4 KV, the
  profiler, positional encoding and unsupported head dims before compiling;
  the AttentionSink variant is refused, so the sink wrapper fails at
  construction. flashinfer.attention un-gated; BatchAttention re-exported.
- tests/rocm/test_batch_attention.py: 200 cases against an fp32 torch
  reference, including graph replay across batch shapes and the refusals.
- benchmarks/rocm/bench_mixed_attention.py: BatchAttention as a third arm.

Regression verdict: additive. No existing kernel or dispatch path changes; the
one shared-code edit is a refusal for the AttentionSink prefill variant, which
had no ROCm kernel. On MI300X BatchAttention is 1.12-2.40x faster than the
in-tree prefill + decode kernels in 16 of 17 configurations (0.91x at a 1k
prefill with 128 decodes). Not compared with AITER (unavailable).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant