Repository navigation
fix(decode): size AITER decode from the page table, not seq_lens - #403
Merged
Merged
Conversation
Without a declared max_seq_len, the AITER decode plan sized its launch from max(seq_lens) while the kernel reads each request's length from the page table, so a seq_lens below the page-table length left partitions unlaunched and returned NaN, inf or wrong rows with no error. Measured against a torch reference, one 2048-token request, page 16, fp16, AITER 0.1.21.post2: | seq_lens | gfx950 AITER | fa2 | |-------------------|--------------|----------| | none / exact | 6.1e-5 | 6.3e-5 | | 512 | inf | 6.3e-5 | | 256 or 1 | NaN | | | 4096 (overstated) | 6.1e-5 | | It now sizes from max pages * page_size, as the max_seq_len path already does, floored at one page: a batch whose requests all have length 0 asked AITER for a zero-partition kernel, which fails to compile. With the floor an empty batch plans too, so run() returns its empty output rather than launching an empty grid; before, plan() raised from max() on an empty tensor. The page count is computed by plan() anyway, so the builtin max() over the seq_lens tensor goes away. Host plan()/run() time, gfx950, page 16, H=32 KVH=8 D=128, 2048 tokens per request, min of 9 interleaved rounds (us): | bs | plan base | plan new | run base | run new | |-----|-----------|----------|----------|---------| | 1 | 58.57 | 56.60 | 16.63 | 16.54 | | 64 | 128.19 | 56.48 | 111.30 | 111.30 | | 256 | 342.29 | 57.33 | 412.17 | 412.17 | Verified on gfx950 only. gfx942 is unverified: the gfx942 node available could not initialise HSA inside the ROCm 10.1 development image. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The CUDA graph path still permits an unsafe partition −1 reduction for pageless batches because the guard checks capacity and is bypassed on replay; device-side handling and capture/replay tests are needed.
1 open issue
What changed in this PR
This PR fixes AITER batch-decode launch sizing in the ROCm backend by using page-table bounds rather than caller-supplied seq_lens, improving correctness and reducing planning overhead.
Changes:
- Derives launch bounds from page counts or declared graph capacity, with a one-page minimum.
- Adds early returns for empty batches and zero-width page tables.
- Adds regression tests for mismatched sequence lengths, pageless requests, and empty batches.
| File | Description |
|---|---|
tests/rocm/test_batch_decode_aiter.py |
Adds length-mismatch and empty-input regression tests. |
flashinfer/rocm/decode.py |
Updates AITER launch sizing and empty-input handling. |
Review effort: Balanced
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
AITER batch decode sized its launch from
max(seq_lens)while the kernel reads each request's length from the page table, so aseq_lensshorter than the page table left partitions unlaunched and returned NaN, inf or wrong rows with no error.plan()now sizes from the page table, as themax_seq_lengraph path already did. Dropping the builtinmax()over theseq_lenstensor also cuts AITER decodeplan()from 342 to 57 µs at bs=256.What changed
seq_lensfor one 2048-token requestMax abs error against a torch reference, fp16, page 16, AITER 0.1.21.post2.
flashinfer/rocm/decode.py,plan():max_kv_len = max(max_pages, 1) * page_sizeon both AITER paths. The one-page floor is needed because a batch whose requests all have length 0 asked AITER for a zero-partition kernel, which fails to compile.flashinfer/rocm/decode.py,run(): on AITER, an empty batch returns its empty output instead of launching an empty grid (base raised frommax()inplan()), and a batch where no request has a page returns zeros, as fa2 does, instead of launching a reduce that reads partition −1.tests/rocm/test_batch_decode_aiter.py: understated and overstatedseq_lensagainst fa2; batches with no pages; an empty batch with and withoutreturn_lse.Known, not changed: in a batch mixing requests with and without pages, AITER's row for the request without pages is undefined. That is AITER's reduce kernel, unchanged from the base commit, and the test compares only rows with pages.
Benchmark results
Host time per call (µs), gfx950, page 16, H=32 KVH=8 D=128, 2048 tokens per request, min of 9 interleaved base/head rounds; GPU idle before and after.
Test plan
seq_lenscases above: wrong at base, correct at head — gfx950test_batch_decode_aiter,_kernels,_speculative,_vllm,test_decode_guards,test_aiter_auto_fallback): 2,616 passed, 0 failed — gfx950pre-commit run --from-ref origin/amd-integration --to-ref HEAD