Skip to content

fix(attention): refuse causal qo_len > kv_len on the fa2 batch routes - #402

Merged
demandal25 merged 4 commits into
amd-integrationfrom
fa2-causal-kv-guard
Nov 17, 2026
Merged

demandal25 merged 4 commits into
amd-integrationfrom
fa2-causal-kv-guard

Conversation

@demandal25

@demandal25 demandal25 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

fa2 batch prefill silently returned wrong output for a causal request whose query is longer than its non-empty KV (qo_len > kv_len > 0): the causal bounds in prefill.cuh are uint32 and wrap. The fa2 batch routes now refuse that request with a ValueError, as single prefill, single POD and speculative decode already did; AITER computes it correctly and keeps running it. The PR also removes 1,502 test skips per arch that were stale or guarded only the tests' own reference.

What changed

The refusal (fa2 only)

shape (qo/kv) gfx942 fa2 gfx950 fa2 AITER, both
37/54 5.0e-4 5.0e-4 ≤ 1.4e-3
127/54 2.64 9.2e-4 ≤ 1.4e-3
127/97 3.28 3.28 ≤ 1.4e-3

Max abs error at the base commit against a bottom-right causal torch reference (fp16, page 16, hd 128). Ragged, batch POD and block-sparse are wrong at the same shapes (batch POD on gfx942 only, at 127/97). A request with no KV is not affected: fa2 returns zero rows for it, so it is not refused.

  • include/flashinfer/rocm/attention/scheduler.cuh — PrefillPlanInfo records the first short request (16 fields, was 15).
  • csrc/rocm/batch_prefill.cu — BatchPrefillWithKVCachePlan raises for a causal plan before it writes anything; paged and ragged run() raise when run with a causal mask (e.g. forward(causal=True) after plan(causal=False)).
  • csrc/rocm/batch_pod.cu, csrc/rocm/pytorch_extension_utils.h — the same run-time check on batch POD's prefill arm, whose causal flag only arrives at run().
  • flashinfer/rocm/prefill.py — the planner is asked to refuse only when run() will apply the causal mask (keyed on the mask buffer run() reads), and gets the page-metadata lengths the kernel reads rather than a caller's seq_lens. CUDA-graph buffers and the graph row capacity are written only after the planner accepts the batch, so a refused replan leaves a captured graph's inputs as they were.
  • flashinfer/rocm/decode.py — the tensor-core multi-token path passes the kernel's lengths and opts into the plan-time refusal. Decode and paged prefill widen a uint32 seq_lens (the documented dtype) to int64 where they read it in: the planner accepts int32/int64 only, and torch cannot max() a uint32 tensor.
  • docs/rocm/backends.md — listed under "Refused rather than ignored".

Skips removed (per arch)

Skip Cases Why it was wrong
AITER causal, test_batch_prefill_kernels.py 1,440 Written against the Python aiter.ops path the C++ shims replaced; never revisited.
fp8-KV bf16 at hd 64/256, test_fp8_kv_prefill.py 32 The warm-up only built bf16 at hd 128.
top-k k > vocab_size, test_sampling.py, test_logits_processor.py 30 The reference indexed sorted_prob[:, k - 1] out of range; the kernels are correct there.

Architecture / design notes

The check sits in the fa2 C++ planner because every fa2 batch route passes through it (paged, ragged incl. deprecated forward(), batch POD, block-sparse, decode with tensor cores) and AITER does not — its modules bind a no-op plan. So backend="auto" that resolves to AITER still runs the shape; a call auto demotes to fa2 now raises instead of returning wrong output. Upstream's sparse.py passes causal even with a custom mask, so the plan adapter only refuses at plan time for callers that set causal_mask_applies; block-sparse is refused by run() when the causal mask applies. Fixing the kernel's unsigned arithmetic instead would change device code on every fa2 prefill; this change leaves the fa2 fatbin byte-identical (checked on gfx950).

SGLang's EAGLE verify plans kv = seq + draft ≥ qo, and vLLM plans prefill with real lengths, so neither reaches the refusal.

Known, left for follow-ups: the RoPE query position in fa2 batch prefill wraps the same way at qo_len > kv_len for any mask mode; and a graph wrapper planned non-causal but captured through forward(causal=True) is not refused on a later short replan (replay skips run()), as on the base commit.

Benchmark results

Host time per call (µs), min of 3 interleaved base/head rounds × 5,000 calls; causal, H=8, D=128, page 16, qo 4 / kv 64 per request. single_decode is an untouched control. Kernel time is unchanged: the fa2 device code is byte-identical.

config gfx950 base gfx950 head Δ gfx942 base gfx942 head Δ
paged plan bs=1 71.66 72.11 +0.6% 127.85 125.06 −2.2%
paged run bs=1 11.92 11.86 −0.5% 20.67 20.61 −0.3%
ragged plan bs=1 44.34 44.11 −0.5% 68.84 65.80 −4.4%
ragged run bs=1 10.24 10.25 +0.1% 16.04 15.46 −3.6%
paged plan bs=64 142.96 143.38 +0.3% 247.94 233.72 −5.7%
paged run bs=64 14.74 14.75 +0.1% 24.30 24.26 −0.2%
ragged plan bs=64 44.76 44.95 +0.4% 71.18 65.03 −8.6%
ragged run bs=64 12.46 12.46 +0.0% 20.19 20.20 +0.0%
single_decode (control) 7.68 7.49 −2.5% 11.94 11.19 −6.3%

gfx950 is within noise everywhere. On gfx942 the control moved 6%, so the base arm was contended; head is not slower anywhere. A caller that passes seq_lens to the paged plan pays 7–11 µs more per plan() for the page-metadata lengths. The fp8-KV test file compiles 4:42 longer on a cold cache for its 32 new cases.

Test plan

  • Probe of all four fa2 batch routes at 37/54, 127/54, 127/97: wrong at base, ValueError at head, valid shape unchanged — gfx942, gfx950
  • AITER at qo_len > kv_len, fp16 and bf16, page 16 and 5 (flat-gather route), incl. a kv 0 request: correct — gfx942, gfx950
  • A/B: every new refusal and regression test fails with its fix reverted and passes with it — gfx942, gfx950
  • Full ROCm suite (92 files) at bbdeeca, gfx950: 35,598 passed, 0 failed, 2,514 skipped
  • Full ROCm suite (92 files) at bbdeeca, gfx942: 35,587 passed, 0 failed, 2,525 skipped
  • uint32 seq_lens (39d6cef): the decode/prefill files at 39d6cef, 9,858 passed, and the new tests fail on bbdeeca — gfx950 only; gfx942 not run (Python dtype handling, no arch-specific path)
  • pre-commit run --from-ref origin/amd-integration --to-ref HEAD

demandal25 and others added 3 commits November 16, 2026 17:04
fa2 batch prefill accepted a causal request whose query is longer than its
KV and returned wrong numbers for it: the causal bounds in prefill.cuh are
uint32 (`kv_len - qo_len + ...`), so they wrap and the mask stops masking.
Measured on the paged route against a bottom-right causal torch reference
(AITER 0.1.21.post2, ROCm 10.1, page 16, hd 128):

| shape (qo/kv) | gfx942 fa2 | gfx950 fa2 | AITER, both |
|---|---|---|---|
| 37/54  | 5.0e-4 | 5.0e-4 | <= 8.0e-4 |
| 127/54 | 2.64   | 9.2e-4 | <= 2.4e-3 |
| 127/97 | 3.28   | 3.28   | <= 1.2e-3 |

A request with no KV at all is not affected: fa2 returns zero rows for it
on ragged and block-sparse (5.3e-4 and 4.7e-4 against the reference on
gfx950), which is also what an empty block row is, so it is not refused.

Single prefill, single POD and speculative decode already refuse the shape;
the batch routes did not. The check sits in the fa2 C++ planner, which every
fa2 batch route (paged, ragged, batch POD, block-sparse) passes through and
AITER does not (its modules bind a no-op plan), so AITER keeps computing it:

- PrefillPlanInfo records the first short request (16 fields, was 15).
- BatchPrefillWithKVCachePlan raises for a causal plan, before it writes.
- The fa2 paged/ragged run() and batch POD's prefill arm raise when run with
  a causal mask, which catches forward(causal=True) after plan(causal=False)
  (SGLang's call shape) and batch POD, whose causal flag only arrives at run().
- The plan-time refusal applies only when the caller passes
  causal_mask_applies: upstream's sparse.py passes causal even with a custom
  mask, which replaces it in run(), so block-sparse is refused by run() alone.
  The wrappers key it on the mask buffer run() reads. ROCm's ragged run()
  ignores prefix_len_ptr and stays causal, so that no longer exempts a plan.
  The paged planner gets the page-metadata lengths the kernel reads, never a
  caller's seq_lens, which fa2 has never read: an overstated one would hide a
  short request and an understated one refuse a good batch. That costs 7-11 us
  per plan() only for a caller that passes seq_lens.
- The planner counts a request with no pages as having no KV, whatever its
  last_page_len slot holds.
- The paged and ragged plans write the CUDA-graph buffers and pin the row
  capacity only after the planner accepts the batch, so a refused plan leaves
  a captured graph's inputs as they were (replay never reaches run()'s check)
  and a refused first plan does not cap later ones.

The paged tests' CUDA-graph warm-up planned one page against qo_len up to
127, so it now plans enough pages for a causal case, and the use_softmax test
read split_kv as the last plan-info field, so it now indexes field 14.

Rollout: SGLang's EAGLE verify plans kv = seq + draft >= qo even at its
graph-capture fill value, and vLLM plans prefill with real lengths, so
neither hits this. With backend="auto", a batch that auto demotes to fa2
(short-query or native-paging probe) now raises for such a request rather
than returning wrong output.

Co-Authored-By: Claude <noreply@anthropic.com>
Both skips were stale. The AITER causal skip in the two paged grids came in
with the first AITER batch-prefill integration (2fe9bcd), against a Python
aiter.ops path that the C++ shims replaced (77bf911); it was never
revisited. With it removed, 1,440 cases per arch run and pass on gfx942 and
gfx950 (AITER 0.1.21.post2, ROCm 10.1) -- causal AITER prefill is what
`auto` routes most serving prefill to. The kv_layout arm of the same guard
was dead: both tests parametrize kv_layout as NHD only.

The fp8-KV bf16 skip at head_dim 64/256 only reflected the warm-up, which
built bf16 at 128 alone. It now builds those head dims too, unwindowed only
as the test uses them; the 32 cases pass on both arches.

Co-Authored-By: Claude <noreply@anthropic.com>
The eight `k > vocab_size` skips guarded the tests' own reference, which
indexed the pivot at `sorted_prob[:, k - 1]` and went out of range (on the
device, the variable-k test aborted the worker). The kernels themselves are
right there: top_k_renorm_probs, top_k_mask_logits and
top_k_sampling_from_probs (scalar and per-row k) all leave the input
untouched for k >= vocab_size on gfx942 and gfx950, while k = vocab_size - 1
masks exactly one entry per row. That is the path a serving engine takes with
top-k disabled, so the reference now clamps the pivot to the vocabulary and
the 30 cases per arch run.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 03:29
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Graph replay can use the wrong causal mask, stale AOT artifacts can mismatch the new plan format, and valid uint32 decode lengths are rejected.

3 open issues
What changed in this PR

This PR adds safeguards for causal requests whose query length exceeds their non-empty KV length on ROCm’s fa2 batch-attention routes, while preserving AITER support for those shapes.

Changes:

  • Records short-KV requests in plan metadata and adds plan-time and runtime rejection checks.
  • Uses kernel-visible lengths for validation and delays graph-buffer updates until planning succeeds.
  • Adds regression coverage and removes stale attention, fp8-KV, and sampling skips.
File Description
tests/​rocm/​test_sampling.py Fixes top-k references and enables oversized-k cases.
tests/​rocm/​test_logits_processor.py Enables oversized-k coverage with corrected references.
tests/​rocm/​test_fp8_kv_prefill.py Expands bf16 warm-up and removes related skips.
tests/​rocm/​test_customize_prefill_use_softmax.py Updates the split-KV assertion for the extended plan layout.
tests/​rocm/​test_causal_short_kv.py Adds short-KV, mask, empty-cache, and graph-replan regressions.
tests/​rocm/​test_batch_prefill_kernels.py Removes stale AITER skips and adjusts causal warm-up shapes.
tests/​rocm/​test_batch_pod.py Tests causal refusal on the batch POD prefill arm.
tests/​rocm/​test_api_parity_runtime.py Updates planner-adapter expectations for causal validation.
include/​flashinfer/​rocm/​attention/​scheduler.cuh Extends plan metadata with the first short-KV request.
flashinfer/​rocm/​prefill.py Integrates validation and defers persistent graph-buffer writes.
flashinfer/​rocm/​decode.py Passes kernel lengths and enables causal planner validation.
docs/​rocm/​backends.md Documents the fa2 short-KV refusal.
csrc/​rocm/​pytorch_extension_utils.h Adds the shared causal short-KV runtime guard.
csrc/​rocm/​batch_prefill.cu Detects short requests and guards planning and execution.
csrc/​rocm/​batch_pod.cu Applies the runtime guard to batch POD prefill.

Review effort: Balanced

Comment thread flashinfer/rocm/prefill.py Resolved
Comment thread flashinfer/rocm/decode.py Resolved
Both wrappers document seq_lens as uint32, but torch 2.12 cannot max() a
uint32 tensor ("gt_cpu" not implemented for 'UInt32'), so a plan with two or
more requests already raised on the base commit. The fa2 planner now also
reads these lengths and accepts only int32 or int64, which broke the
single-request tensor-core decode plan that used to work. seq_lens is now
widened to int64 where each wrapper reads it in, which covers the planner,
both max() calls and AITER decode's sizing.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 14:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The test never captures or replays a graph, so it cannot detect whether a refused replan changes replay output on either route.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Test refused replans preserve replayed causal output

tests/​rocm/​test_causal_short_kv.py:296

This test never captures or replays a graph, so a refused replan could change the scheduler workspace without failing it. For both routes, capture and replay a valid causal run, then verify that replay after the refused replan preserves the output.

3 resolved since last review

Review effort: Balanced

@demandal25

Copy link
Copy Markdown
Collaborator Author

Review of 39d6cef, previously-missed item (test at test_causal_short_kv.py:296): Declined — the refusal raises at csrc/rocm/batch_prefill.cu:93, before PrefillPlan writes the workspace at :108, and the test already asserts every captured input buffer is unchanged.

@demandal25
demandal25 merged commit 10dc7e0 into amd-integration Nov 17, 2026
15 checks passed
@demandal25
demandal25 deleted the fa2-causal-kv-guard branch November 17, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants