Skip to content

cuda: keep PTQ1_0 on the PT mat-vec for 5-8 columns on Hopper (pp N=5 -30%, MTP depth 5 x1.24 on H200; not token-identical at verify width >= 5) #382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bri-prism merged 1 commit into from
Oct 24, 2026
Merged

cuda: keep PTQ1_0 on the PT mat-vec for 5-8 columns on Hopper (pp N=5 -30%, MTP depth 5 x1.24 on H200; not token-identical at verify width >= 5) #382

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ggml/src/ggml-cuda/mmvq.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,12 @@ bool ggml_cuda_should_use_mmvq(enum ggml_type type, int cc, int64_t ne11) {
// The PT mat-vec path shares the weight decode across columns; with the branch-free PTQ1_0
// MMQ tile loader the tile path overtakes it at 5+ columns on Ada (RTX 4070, Bonsai 2 27B:
// mat-vec 155 t/s vs MMQ 244 t/s at n=8).
// On Hopper it is the other way round: H200, Bonsai 2 27B, one forward of n tokens at depth 4096,
// MMQ 17.99 / 18.76 ms vs PT mat-vec 12.64 / 15.71 ms at n = 5 / 8. The PT kernel's per-column
// arithmetic does not depend on the column count, so 5-8 column results equal the 2-4 column ones.
if (cc >= GGML_CUDA_CC_HOPPER && cc < GGML_CUDA_CC_BLACKWELL) {
return ne11 <= MMVQ_MAX_BATCH_SIZE;
}
return ne11 <= 4;
}
#endif
Expand Down
Loading