You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(attention): refuse causal qo_len > kv_len on the fa2 batch routes - #402
fa2 batch prefill silently returned wrong output for a causal request whose query is longer than its non-empty KV (qo_len > kv_len > 0): the causal bounds in prefill.cuh are uint32 and wrap. The fa2 batch routes now refuse that request with a ValueError, as single prefill, single POD and speculative decode already did; AITER computes it correctly and keeps running it. The PR also removes 1,502 test skips per arch that were stale or guarded only the tests' own reference.
What changed
The refusal (fa2 only)
shape (qo/kv)
gfx942 fa2
gfx950 fa2
AITER, both
37/54
5.0e-4
5.0e-4
≤ 1.4e-3
127/54
2.64
9.2e-4
≤ 1.4e-3
127/97
3.28
3.28
≤ 1.4e-3
Max abs error at the base commit against a bottom-right causal torch reference (fp16, page 16, hd 128). Ragged, batch POD and block-sparse are wrong at the same shapes (batch POD on gfx942 only, at 127/97). A request with no KV is not affected: fa2 returns zero rows for it, so it is not refused.
include/flashinfer/rocm/attention/scheduler.cuh — PrefillPlanInfo records the first short request (16 fields, was 15).
csrc/rocm/batch_prefill.cu — BatchPrefillWithKVCachePlan raises for a causal plan before it writes anything; paged and ragged run() raise when run with a causal mask (e.g. forward(causal=True) after plan(causal=False)).
csrc/rocm/batch_pod.cu, csrc/rocm/pytorch_extension_utils.h — the same run-time check on batch POD's prefill arm, whose causal flag only arrives at run().
flashinfer/rocm/prefill.py — the planner is asked to refuse only when run() will apply the causal mask (keyed on the mask buffer run() reads), and gets the page-metadata lengths the kernel reads rather than a caller's seq_lens. CUDA-graph buffers and the graph row capacity are written only after the planner accepts the batch, so a refused replan leaves a captured graph's inputs as they were.
flashinfer/rocm/decode.py — the tensor-core multi-token path passes the kernel's lengths and opts into the plan-time refusal. Decode and paged prefill widen a uint32seq_lens (the documented dtype) to int64 where they read it in: the planner accepts int32/int64 only, and torch cannot max() a uint32 tensor.
docs/rocm/backends.md — listed under "Refused rather than ignored".
Skips removed (per arch)
Skip
Cases
Why it was wrong
AITER causal, test_batch_prefill_kernels.py
1,440
Written against the Python aiter.ops path the C++ shims replaced; never revisited.
fp8-KV bf16 at hd 64/256, test_fp8_kv_prefill.py
32
The warm-up only built bf16 at hd 128.
top-k k > vocab_size, test_sampling.py, test_logits_processor.py
30
The reference indexed sorted_prob[:, k - 1] out of range; the kernels are correct there.
Architecture / design notes
The check sits in the fa2 C++ planner because every fa2 batch route passes through it (paged, ragged incl. deprecated forward(), batch POD, block-sparse, decode with tensor cores) and AITER does not — its modules bind a no-op plan. So backend="auto" that resolves to AITER still runs the shape; a call auto demotes to fa2 now raises instead of returning wrong output. Upstream's sparse.py passes causal even with a custom mask, so the plan adapter only refuses at plan time for callers that set causal_mask_applies; block-sparse is refused by run() when the causal mask applies. Fixing the kernel's unsigned arithmetic instead would change device code on every fa2 prefill; this change leaves the fa2 fatbin byte-identical (checked on gfx950).
SGLang's EAGLE verify plans kv = seq + draft ≥ qo, and vLLM plans prefill with real lengths, so neither reaches the refusal.
Known, left for follow-ups: the RoPE query position in fa2 batch prefill wraps the same way at qo_len > kv_len for any mask mode; and a graph wrapper planned non-causal but captured through forward(causal=True) is not refused on a later short replan (replay skips run()), as on the base commit.
Benchmark results
Host time per call (µs), min of 3 interleaved base/head rounds × 5,000 calls; causal, H=8, D=128, page 16, qo 4 / kv 64 per request. single_decode is an untouched control. Kernel time is unchanged: the fa2 device code is byte-identical.
config
gfx950 base
gfx950 head
Δ
gfx942 base
gfx942 head
Δ
paged plan bs=1
71.66
72.11
+0.6%
127.85
125.06
−2.2%
paged run bs=1
11.92
11.86
−0.5%
20.67
20.61
−0.3%
ragged plan bs=1
44.34
44.11
−0.5%
68.84
65.80
−4.4%
ragged run bs=1
10.24
10.25
+0.1%
16.04
15.46
−3.6%
paged plan bs=64
142.96
143.38
+0.3%
247.94
233.72
−5.7%
paged run bs=64
14.74
14.75
+0.1%
24.30
24.26
−0.2%
ragged plan bs=64
44.76
44.95
+0.4%
71.18
65.03
−8.6%
ragged run bs=64
12.46
12.46
+0.0%
20.19
20.20
+0.0%
single_decode (control)
7.68
7.49
−2.5%
11.94
11.19
−6.3%
gfx950 is within noise everywhere. On gfx942 the control moved 6%, so the base arm was contended; head is not slower anywhere. A caller that passes seq_lens to the paged plan pays 7–11 µs more per plan() for the page-metadata lengths. The fp8-KV test file compiles 4:42 longer on a cold cache for its 32 new cases.
Test plan
Probe of all four fa2 batch routes at 37/54, 127/54, 127/97: wrong at base, ValueError at head, valid shape unchanged — gfx942, gfx950
AITER at qo_len > kv_len, fp16 and bf16, page 16 and 5 (flat-gather route), incl. a kv 0 request: correct — gfx942, gfx950
A/B: every new refusal and regression test fails with its fix reverted and passes with it — gfx942, gfx950
Full ROCm suite (92 files) at bbdeeca, gfx950: 35,598 passed, 0 failed, 2,514 skipped
Full ROCm suite (92 files) at bbdeeca, gfx942: 35,587 passed, 0 failed, 2,525 skipped
uint32seq_lens (39d6cef): the decode/prefill files at 39d6cef, 9,858 passed, and the new tests fail on bbdeeca — gfx950 only; gfx942 not run (Python dtype handling, no arch-specific path)
pre-commit run --from-ref origin/amd-integration --to-ref HEAD
fa2 batch prefill accepted a causal request whose query is longer than its
KV and returned wrong numbers for it: the causal bounds in prefill.cuh are
uint32 (`kv_len - qo_len + ...`), so they wrap and the mask stops masking.
Measured on the paged route against a bottom-right causal torch reference
(AITER 0.1.21.post2, ROCm 10.1, page 16, hd 128):
| shape (qo/kv) | gfx942 fa2 | gfx950 fa2 | AITER, both |
|---|---|---|---|
| 37/54 | 5.0e-4 | 5.0e-4 | <= 8.0e-4 |
| 127/54 | 2.64 | 9.2e-4 | <= 2.4e-3 |
| 127/97 | 3.28 | 3.28 | <= 1.2e-3 |
A request with no KV at all is not affected: fa2 returns zero rows for it
on ragged and block-sparse (5.3e-4 and 4.7e-4 against the reference on
gfx950), which is also what an empty block row is, so it is not refused.
Single prefill, single POD and speculative decode already refuse the shape;
the batch routes did not. The check sits in the fa2 C++ planner, which every
fa2 batch route (paged, ragged, batch POD, block-sparse) passes through and
AITER does not (its modules bind a no-op plan), so AITER keeps computing it:
- PrefillPlanInfo records the first short request (16 fields, was 15).
- BatchPrefillWithKVCachePlan raises for a causal plan, before it writes.
- The fa2 paged/ragged run() and batch POD's prefill arm raise when run with
a causal mask, which catches forward(causal=True) after plan(causal=False)
(SGLang's call shape) and batch POD, whose causal flag only arrives at run().
- The plan-time refusal applies only when the caller passes
causal_mask_applies: upstream's sparse.py passes causal even with a custom
mask, which replaces it in run(), so block-sparse is refused by run() alone.
The wrappers key it on the mask buffer run() reads. ROCm's ragged run()
ignores prefix_len_ptr and stays causal, so that no longer exempts a plan.
The paged planner gets the page-metadata lengths the kernel reads, never a
caller's seq_lens, which fa2 has never read: an overstated one would hide a
short request and an understated one refuse a good batch. That costs 7-11 us
per plan() only for a caller that passes seq_lens.
- The planner counts a request with no pages as having no KV, whatever its
last_page_len slot holds.
- The paged and ragged plans write the CUDA-graph buffers and pin the row
capacity only after the planner accepts the batch, so a refused plan leaves
a captured graph's inputs as they were (replay never reaches run()'s check)
and a refused first plan does not cap later ones.
The paged tests' CUDA-graph warm-up planned one page against qo_len up to
127, so it now plans enough pages for a causal case, and the use_softmax test
read split_kv as the last plan-info field, so it now indexes field 14.
Rollout: SGLang's EAGLE verify plans kv = seq + draft >= qo even at its
graph-capture fill value, and vLLM plans prefill with real lengths, so
neither hits this. With backend="auto", a batch that auto demotes to fa2
(short-query or native-paging probe) now raises for such a request rather
than returning wrong output.
Co-Authored-By: Claude <noreply@anthropic.com>
Both skips were stale. The AITER causal skip in the two paged grids came in
with the first AITER batch-prefill integration (2fe9bcd), against a Python
aiter.ops path that the C++ shims replaced (77bf911); it was never
revisited. With it removed, 1,440 cases per arch run and pass on gfx942 and
gfx950 (AITER 0.1.21.post2, ROCm 10.1) -- causal AITER prefill is what
`auto` routes most serving prefill to. The kv_layout arm of the same guard
was dead: both tests parametrize kv_layout as NHD only.
The fp8-KV bf16 skip at head_dim 64/256 only reflected the warm-up, which
built bf16 at 128 alone. It now builds those head dims too, unwindowed only
as the test uses them; the 32 cases pass on both arches.
Co-Authored-By: Claude <noreply@anthropic.com>
The eight `k > vocab_size` skips guarded the tests' own reference, which
indexed the pivot at `sorted_prob[:, k - 1]` and went out of range (on the
device, the variable-k test aborted the worker). The kernels themselves are
right there: top_k_renorm_probs, top_k_mask_logits and
top_k_sampling_from_probs (scalar and per-row k) all leave the input
untouched for k >= vocab_size on gfx942 and gfx950, while k = vocab_size - 1
masks exactly one entry per row. That is the path a serving engine takes with
top-k disabled, so the reference now clamps the pivot to the vocabulary and
the 30 cases per arch run.
Co-Authored-By: Claude <noreply@anthropic.com>
This PR adds safeguards for causal requests whose query length exceeds their non-empty KV length on ROCm’s fa2 batch-attention routes, while preserving AITER support for those shapes.
Changes:
Records short-KV requests in plan metadata and adds plan-time and runtime rejection checks.
Uses kernel-visible lengths for validation and delays graph-buffer updates until planning succeeds.
Adds regression coverage and removes stale attention, fp8-KV, and sampling skips.
File
Description
tests/rocm/test_sampling.py
Fixes top-k references and enables oversized-k cases.
tests/rocm/test_logits_processor.py
Enables oversized-k coverage with corrected references.
Both wrappers document seq_lens as uint32, but torch 2.12 cannot max() a
uint32 tensor ("gt_cpu" not implemented for 'UInt32'), so a plan with two or
more requests already raised on the base commit. The fa2 planner now also
reads these lengths and accepts only int32 or int64, which broke the
single-request tensor-core decode plan that used to work. seq_lens is now
widened to int64 where each wrapper reads it in, which covers the planner,
both max() calls and AITER decode's sizing.
Co-Authored-By: Claude <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The test never captures or replays a graph, so it cannot detect whether a refused replan changes replay output on either route.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Test refused replans preserve replayed causal output
tests/rocm/test_causal_short_kv.py:296
This test never captures or replays a graph, so a refused replan could change the scheduler workspace without failing it. For both routes, capture and replay a valid causal run, then verify that replay after the refused replan preserves the output.
Review of 39d6cef, previously-missed item (test at test_causal_short_kv.py:296): Declined — the refusal raises at csrc/rocm/batch_prefill.cu:93, before PrefillPlan writes the workspace at :108, and the test already asserts every captured input buffer is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fa2 batch prefill silently returned wrong output for a causal request whose query is longer than its non-empty KV (
qo_len > kv_len > 0): the causal bounds inprefill.cuhareuint32and wrap. The fa2 batch routes now refuse that request with aValueError, as single prefill, single POD and speculative decode already did; AITER computes it correctly and keeps running it. The PR also removes 1,502 test skips per arch that were stale or guarded only the tests' own reference.What changed
The refusal (fa2 only)
Max abs error at the base commit against a bottom-right causal torch reference (fp16, page 16, hd 128). Ragged, batch POD and block-sparse are wrong at the same shapes (batch POD on gfx942 only, at 127/97). A request with no KV is not affected: fa2 returns zero rows for it, so it is not refused.
include/flashinfer/rocm/attention/scheduler.cuh—PrefillPlanInforecords the first short request (16 fields, was 15).csrc/rocm/batch_prefill.cu—BatchPrefillWithKVCachePlanraises for a causal plan before it writes anything; paged and raggedrun()raise when run with a causal mask (e.g.forward(causal=True)afterplan(causal=False)).csrc/rocm/batch_pod.cu,csrc/rocm/pytorch_extension_utils.h— the same run-time check on batch POD's prefill arm, whose causal flag only arrives atrun().flashinfer/rocm/prefill.py— the planner is asked to refuse only whenrun()will apply the causal mask (keyed on the mask bufferrun()reads), and gets the page-metadata lengths the kernel reads rather than a caller'sseq_lens. CUDA-graph buffers and the graph row capacity are written only after the planner accepts the batch, so a refused replan leaves a captured graph's inputs as they were.flashinfer/rocm/decode.py— the tensor-core multi-token path passes the kernel's lengths and opts into the plan-time refusal. Decode and paged prefill widen auint32seq_lens(the documented dtype) to int64 where they read it in: the planner accepts int32/int64 only, and torch cannotmax()auint32tensor.docs/rocm/backends.md— listed under "Refused rather than ignored".Skips removed (per arch)
test_batch_prefill_kernels.pyaiter.opspath the C++ shims replaced; never revisited.test_fp8_kv_prefill.pyk > vocab_size,test_sampling.py,test_logits_processor.pysorted_prob[:, k - 1]out of range; the kernels are correct there.Architecture / design notes
The check sits in the fa2 C++ planner because every fa2 batch route passes through it (paged, ragged incl. deprecated
forward(), batch POD, block-sparse, decode with tensor cores) and AITER does not — its modules bind a no-op plan. Sobackend="auto"that resolves to AITER still runs the shape; a callautodemotes to fa2 now raises instead of returning wrong output. Upstream'ssparse.pypassescausaleven with a custom mask, so the plan adapter only refuses at plan time for callers that setcausal_mask_applies; block-sparse is refused byrun()when the causal mask applies. Fixing the kernel's unsigned arithmetic instead would change device code on every fa2 prefill; this change leaves the fa2 fatbin byte-identical (checked on gfx950).SGLang's EAGLE verify plans
kv = seq + draft ≥ qo, and vLLM plans prefill with real lengths, so neither reaches the refusal.Known, left for follow-ups: the RoPE query position in fa2 batch prefill wraps the same way at
qo_len > kv_lenfor any mask mode; and a graph wrapper planned non-causal but captured throughforward(causal=True)is not refused on a later short replan (replay skipsrun()), as on the base commit.Benchmark results
Host time per call (µs), min of 3 interleaved base/head rounds × 5,000 calls; causal, H=8, D=128, page 16, qo 4 / kv 64 per request.
single_decodeis an untouched control. Kernel time is unchanged: the fa2 device code is byte-identical.gfx950 is within noise everywhere. On gfx942 the control moved 6%, so the base arm was contended; head is not slower anywhere. A caller that passes
seq_lensto the paged plan pays 7–11 µs more perplan()for the page-metadata lengths. The fp8-KV test file compiles 4:42 longer on a cold cache for its 32 new cases.Test plan
ValueErrorat head, valid shape unchanged — gfx942, gfx950qo_len > kv_len, fp16 and bf16, page 16 and 5 (flat-gather route), incl. a kv 0 request: correct — gfx942, gfx950uint32seq_lens(39d6cef): the decode/prefill files at 39d6cef, 9,858 passed, and the new tests fail on bbdeeca — gfx950 only; gfx942 not run (Python dtype handling, no arch-specific path)pre-commit run --from-ref origin/amd-integration --to-ref HEAD