Load global scales in qmm_t kernels - #4483
Conversation
1209877 to
0390b68
Compare
|
A quick helper to load the model in mlx-lm |
|
Very nice! I am a very surprised it gives such a big improvement for prefill.. So before scaling was expressed as multiplication of the output after matmul I guess. Do you see any improvements on decode? Probably not, right? |
| bits, | ||
| aligned, | ||
| batched); | ||
| kernel = global_scale ? get_quantized_kernel_wrapped( |
There was a problem hiding this comment.
Nit:
I think you can simplify the big if clause with:
kernel = get_qmm_nax_kernel_wrapped(
...
global_scale.has_value());
0390b68 to
e4b3a5c
Compare
|
To clarify, both this plus #4481 are where the larger win happens, and yes, it's really just prefill. Decode is effectively unchanged. I could be mistaken, but I think the CI failure on the other PR is exposing existing platform-specific CUDA numerical variance, so not sure how you'd like me to address that. |
Compliments #4481 by adding global scale to the qmm_t kernels.
Performance
Using the NVIDIA Model Optimizer on Qwen/Qwen3.6-35B-A3B with nvfp4_mlp_only p2048/g128