Repository navigation
cuda: 5120-wide fused norm + Hadamard with PDL, and two-row PQ2_0 decode tiles on Hopper - #417
Merged
Merged
Conversation
Add 5120 (five 1024-element segments) to the widths the fused norm + Hadamard kernel handles, and launch that width with programmatic dependent launch so it overlaps the previous kernel's tail. Bit-identical to the unfused path.
Make the rows per warp of the PQ2_0 multi-column mat-vec a template parameter. On Hopper, one-column launches with at most 17408 rows use two rows per warp, so the smaller projections launch more CTAs; the vocabulary head keeps four-row activation reuse. The per-row accumulation order is unchanged.
…da/pq2-decode-tune
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #352 (stacked on its branch; the diff here is only the two commits below).
What
width with programmatic dependent launch.
launch more CTAs; the vocabulary head keeps four rows per warp.
Why
5120-wide norm ran unfused, and the four-row tile left SMs idle on the smaller projections.
How
of the row cutoff with partial warps and CTAs.
Testing
unchanged. Hopper only for the row tile.