Repository navigation
Port the persistent BatchAttention kernel to ROCm (#101) - #404
Open
PMukund219 wants to merge 1 commit into
Open
PMukund219 wants to merge 1 commit into
PMukund219 wants to merge 1 commit into
Conversation
Upstream's flashinfer.BatchAttention (the persistent "holistic" kernel, flashinfer-ai#1137) runs a mixed prefill + decode batch on one fixed grid, so one captured graph serves every batch shape. On ROCm the module was gated as CUDA-only. This adds the HIP kernel, planner, binding and JIT generator so upstream's Python class runs unchanged on gfx942 / gfx950. - include/flashinfer/rocm/attention/persistent.cuh: MFMA 16x16x16 wave64 kernel. Computes S^T = K Q^T and O^T = V^T P^T so the softmax probabilities feed the next MFMA from registers. Attention and the merge of split-KV partials are two ordinary launches: a cooperative grid.sync() launch from a torch extension crashes the process at exit on ROCm 7.2. - scheduler.cuh: TwoStageHolisticPlan / HolisticPlanInfo. fp32 partials with a capacity capped to the float workspace and checked (upstream's bound overflows 384 MB at 304 CTAs and 8 KV heads). Own causal-end helper; the existing packed_causal_kv_end (MLA) uses an older formula. - csrc/rocm/batch_attention*.cu, config template: plan/run with the argument lists _core.py passes; tensor validation; NVFP4 and soft-cap mismatch refused. - jit/rocm/modules.py: gen_batch_attention_module, refusing fp8/NVFP4 KV, the profiler, positional encoding and unsupported head dims before compiling; the AttentionSink variant is refused, so the sink wrapper fails at construction. flashinfer.attention un-gated; BatchAttention re-exported. - tests/rocm/test_batch_attention.py: 200 cases against an fp32 torch reference, including graph replay across batch shapes and the refusals. - benchmarks/rocm/bench_mixed_attention.py: BatchAttention as a third arm. Regression verdict: additive. No existing kernel or dispatch path changes; the one shared-code edit is a refusal for the AttentionSink prefill variant, which had no ROCm kernel. On MI300X BatchAttention is 1.12-2.40x faster than the in-tree prefill + decode kernels in 16 of 17 configurations (0.91x at a 1k prefill with 128 decodes). Not compared with AITER (unavailable).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports upstream's persistent
BatchAttention(flashinfer-ai#1137) to ROCm for #101: mixed prefill + decode batches run on one fixed grid, so one captured graph serves every batch shape. Upstream'sflashinfer/attention/_core.pyruns unchanged on top of a new HIP kernel, planner, binding and JIT generator.What changed
include/flashinfer/rocm/attention/persistent.cuh— new MFMA 16×16×16 wave64 kernel.include/flashinfer/rocm/attention/scheduler.cuh— upstream'sTwoStageHolisticPlan/HolisticPlanInfo, appended.csrc/rocm/batch_attention*.cu,batch_attention_customize_config.jinja— plan/run binding with the argument lists_core.pypasses, plus tensor validation.flashinfer/jit/rocm/modules.py—gen_batch_attention_module; refuses fp8/NVFP4 KV, the profiler, positional encoding, unsupported head dims and the attention-sink variant before compiling.flashinfer/rocm/__init__.py,flashinfer/rocm/api.py,flashinfer/jit/rocm/api.py,flashinfer/jit/attention/__init__.py— un-gateflashinfer.attentionand re-export.tests/rocm/test_batch_attention.py— 200 cases;benchmarks/rocm/bench_mixed_attention.py—BatchAttentionarm;arch_caps.py,README.md,docs/rocm/backends.md— docs and support matrix.Architecture / design notes
Sᵀ = K·Qᵀ,Oᵀ = Vᵀ·PᵀPᵀneeds, soPstays in registers (no LDS transpose).grid.sync()hipLaunchCooperativeKernel+grid.sync()kernel, unrelated to FlashInfer, reproduces it once torch has used the GPU. Stream order is the barrier; both launches are in the same graph.holistic_packed_causal_kv_endpacked_causal_kv_end(ceiling, clamped), which the kernel matches. The fork's function of that name, used by the MLA planner, floors and does not clamp; MLA is left untouched.Benchmark results
MI300X,
vllm/vllm-openai-rocm:v0.29.0image (ROCm 7.2.3, torch 2.12), bf16, causal; baseline is the in-tree fa2 prefill + decode wrappers. Not compared with AITER: the image's AITER 0.1.19 is below the 0.1.21 floor, soautofell back to fa2.benchmarks/bench_batch_attention.py(Qwen-8B / Llama-3.1-70B / Qwen-MoE-235B; 128 decodes + one 4k prefill chunk)benchmarks/rocm/bench_mixed_attention.py(1k/4k prefill + 1–128 decodes)bench_mixed_attention.py --accuracy: matches the separate wrappers to one bf16 rounding step. Untuned (no async KV pipeline, one CTA per CU).Test plan
GPU runs on MI300X in the image above, at base
9c078ef4; the patch applies cleanly onba03cba5. Since the GPU run the only changes arepre-commitformatting and the removal of two unused helpers and unused includes; the compiled kernels are identical before and after.tests/rocm/test_batch_attention.py— 200/200 (upstream's workload mixes, every axis value, split-KV, graph replay across batch shapes, refusals)test_api_parity_runtime,test_package_imports,test_comm_import_gate,test_rocm_dispatch,test_jit_env,test_modules_hip_codegen,test_customize_prefill_signature,test_redistribution_licences,test_benchmark_harness,test_rocm_small_surface— passpytest -m "not slow"suite, the ROCm 10.1 dev image, gfx950, AITER comparisonpre-commit run -a