Skip to content

fix(decode): size AITER decode from the page table, not seq_lens - #403

Merged
demandal25 merged 1 commit into
amd-integrationfrom
aiter-decode-seq-lens
Nov 17, 2026
Merged

demandal25 merged 1 commit into
amd-integrationfrom
aiter-decode-seq-lens

Conversation

@demandal25

Copy link
Copy Markdown
Collaborator

Summary

AITER batch decode sized its launch from max(seq_lens) while the kernel reads each request's length from the page table, so a seq_lens shorter than the page table left partitions unlaunched and returned NaN, inf or wrong rows with no error. plan() now sizes from the page table, as the max_seq_len graph path already did. Dropping the builtin max() over the seq_lens tensor also cuts AITER decode plan() from 342 to 57 µs at bs=256.

What changed

seq_lens for one 2048-token request gfx950 AITER, base gfx950 AITER, head fa2
none / exact 6.1e-5 6.1e-5 6.3e-5
512 inf 6.1e-5 6.3e-5
256 or 1 NaN 6.1e-5 6.3e-5
4096 (overstated) 6.1e-5 6.1e-5 6.3e-5

Max abs error against a torch reference, fp16, page 16, AITER 0.1.21.post2.

  • flashinfer/rocm/decode.py, plan(): max_kv_len = max(max_pages, 1) * page_size on both AITER paths. The one-page floor is needed because a batch whose requests all have length 0 asked AITER for a zero-partition kernel, which fails to compile.
  • flashinfer/rocm/decode.py, run(): on AITER, an empty batch returns its empty output instead of launching an empty grid (base raised from max() in plan()), and a batch where no request has a page returns zeros, as fa2 does, instead of launching a reduce that reads partition −1.
  • tests/rocm/test_batch_decode_aiter.py: understated and overstated seq_lens against fa2; batches with no pages; an empty batch with and without return_lse.

Known, not changed: in a batch mixing requests with and without pages, AITER's row for the request without pages is undefined. That is AITER's reduce kernel, unchanged from the base commit, and the test compares only rows with pages.

Benchmark results

Host time per call (µs), gfx950, page 16, H=32 KVH=8 D=128, 2048 tokens per request, min of 9 interleaved base/head rounds; GPU idle before and after.

bs plan base plan head run base run head
1 58.57 56.60 16.63 16.54
64 128.19 56.48 111.30 111.30
256 342.29 57.33 412.17 412.17

Test plan

  • Probe of the seq_lens cases above: wrong at base, correct at head — gfx950
  • A/B: the new tests fail on the base commit (5 of 7) and pass at ee21828 — gfx950
  • Decode files at ee21828 (test_batch_decode_aiter, _kernels, _speculative, _vllm, test_decode_guards, test_aiter_auto_fallback): 2,616 passed, 0 failed — gfx950
  • gfx942: unverified. The gfx942 node available could not initialise HSA inside the ROCm 10.1 development image. The change is host-side Python with no arch-specific path, but it has not been run there.
  • pre-commit run --from-ref origin/amd-integration --to-ref HEAD

Without a declared max_seq_len, the AITER decode plan sized its launch from
max(seq_lens) while the kernel reads each request's length from the page
table, so a seq_lens below the page-table length left partitions unlaunched
and returned NaN, inf or wrong rows with no error. Measured against a torch
reference, one 2048-token request, page 16, fp16, AITER 0.1.21.post2:

| seq_lens          | gfx950 AITER | fa2      |
|-------------------|--------------|----------|
| none / exact      | 6.1e-5       | 6.3e-5   |
| 512               | inf          | 6.3e-5   |
| 256 or 1          | NaN          |          |
| 4096 (overstated) | 6.1e-5       |          |

It now sizes from max pages * page_size, as the max_seq_len path already
does, floored at one page: a batch whose requests all have length 0 asked
AITER for a zero-partition kernel, which fails to compile. With the floor
an empty batch plans too, so run() returns its empty output rather than
launching an empty grid; before, plan() raised from max() on an empty
tensor.

The page count is computed by plan() anyway, so the builtin max() over the
seq_lens tensor goes away. Host plan()/run() time, gfx950, page 16, H=32
KVH=8 D=128, 2048 tokens per request, min of 9 interleaved rounds (us):

| bs  | plan base | plan new | run base | run new |
|-----|-----------|----------|----------|---------|
| 1   | 58.57     | 56.60    | 16.63    | 16.54   |
| 64  | 128.19    | 56.48    | 111.30   | 111.30  |
| 256 | 342.29    | 57.33    | 412.17   | 412.17  |

Verified on gfx950 only. gfx942 is unverified: the gfx942 node available
could not initialise HSA inside the ROCm 10.1 development image.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 21:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The CUDA graph path still permits an unsafe partition −1 reduction for pageless batches because the guard checks capacity and is bypassed on replay; device-side handling and capture/replay tests are needed.

1 open issue
What changed in this PR

This PR fixes AITER batch-decode launch sizing in the ROCm backend by using page-table bounds rather than caller-supplied seq_lens, improving correctness and reducing planning overhead.

Changes:

  • Derives launch bounds from page counts or declared graph capacity, with a one-page minimum.
  • Adds early returns for empty batches and zero-width page tables.
  • Adds regression tests for mismatched sequence lengths, pageless requests, and empty batches.
File Description
tests/​rocm/​test_batch_decode_aiter.py Adds length-mismatch and empty-input regression tests.
flashinfer/​rocm/​decode.py Updates AITER launch sizing and empty-input handling.

Review effort: Balanced

Comment thread flashinfer/rocm/decode.py Resolved
@github-actions

Copy link
Copy Markdown
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@demandal25
demandal25 merged commit ba03cba into amd-integration Nov 17, 2026
14 checks passed
@demandal25
demandal25 deleted the aiter-decode-seq-lens branch November 17, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants