You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
preserve every top-k/top-p boundary tie, matching flashinfer across renormalization, sampling, and combined top-k -> top-p filtering
retry the same exact kernel with one CTA per row when cooperative launch is unavailable
scope cooperative-launch capability by device and operation, and propagate unrelated Triton failures
remove the older approximate multi-launch fallback and redundant workspace/target conversions
Dropping the token-id tie clipping also removes its per-CTA tie workspace and one synchronization stage from the fused path. This keeps the final PR to one production file; no test files are included.
Validation
RTX A1000 tie regression covering cooperative multi-CTA and single-CTA fallback paths
bf16-logit cutoff ties for top-k/top-p renormalization and top-k, top-p, and combined sampling; every boundary tie remained reachable
tie-free mixed per-row top-k/top-p sweep against PyTorch references
all six fused kernel variants cross-compiled for sm_75, sm_80, sm_86, sm_89, sm_90, sm_100, and sm_120
Ruff, Python byte-compilation, and diff whitespace checks
Credit
This revision builds directly on #345. Thank you, @jason-fxz, for taking the original correctness report seriously, replacing the fragile estimator with a much stronger exact design, and carefully reviewing #329. The core single-launch histogram approach here is your work; the follow-up changes retain its exactness while making device handling and fallback behavior more robust.
Thanks for folding #345 in and for the follow-up work — the per-device _num_sm, the in-kernel clamp and the single-CTA retry are all improvements, and everything passed on my side (no-tie sweep, tie stress, sm_75–sm_120 cross-compile).
One change before that: the tie clipping in _tie_prefix / _keep_mask. We'd rather keep the Triton path aligned with flashinfer, which keeps every boundary tie on every op (>= pivot for top-k renorm, "fewer than k strictly greater" for top-k sampling, > low for top-p). With bf16 logits exact ties at the cutoff are common, so the two paths would otherwise sample from different distributions on the same model. Could you drop the clipping and keep >= thr as in #345?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dropping the token-id tie clipping also removes its per-CTA tie workspace and one synchronization stage from the fused path. This keeps the final PR to one production file; no test files are included.
Validation
Credit
This revision builds directly on #345. Thank you, @jason-fxz, for taking the original correctness report seriously, replacing the fragile estimator with a much stronger exact design, and carefully reviewing #329. The core single-launch histogram approach here is your work; the follow-up changes retain its exactness while making device handling and fallback behavior more robust.