Skip to content

Load global scales in qmm_t kernels - #4483

Open
dhiltgen wants to merge 2 commits into
ml-explore:mainfrom
dhiltgen:g-qqmm-plain
Open

Load global scales in qmm_t kernels#4483
dhiltgen wants to merge 2 commits into
ml-explore:mainfrom
dhiltgen:g-qqmm-plain

Conversation

@dhiltgen

@dhiltgen dhiltgen commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Compliments #4481 by adding global scale to the qmm_t kernels.

Performance

Using the NVIDIA Model Optimizer on Qwen/Qwen3.6-35B-A3B with nvfp4_mlp_only p2048/g128

GPU upstream/main prompt tps this PR prompt tps this + #4481
M5 Max 1,267.5 1,365.9 4,256.2
M3 Ultra 1,603.2 1,683.5 2,819.9
  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: co-developed with a coding agent

@dhiltgen dhiltgen changed the title Use matrix kernels for global-scale qqmm Load global scales in qmm_t kernels Sep 9, 2026
@dhiltgen

dhiltgen commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

A quick helper to load the model in mlx-lm
modelopt_qwen36_mlx_lm.py

@nastya236

nastya236 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Very nice! I am a very surprised it gives such a big improvement for prefill.. So before scaling was expressed as multiplication of the output after matmul I guess. Do you see any improvements on decode? Probably not, right?
Do you have hf nvfp4 model in mind already quantized that I can use to test the speed up?

Comment thread mlx/backend/metal/quantized.cpp Outdated
bits,
aligned,
batched);
kernel = global_scale ? get_quantized_kernel_wrapped(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit:
I think you can simplify the big if clause with:

  kernel = get_qmm_nax_kernel_wrapped(
        ...
        global_scale.has_value());

@dhiltgen

Copy link
Copy Markdown
Contributor Author

To clarify, both this plus #4481 are where the larger win happens, and yes, it's really just prefill. Decode is effectively unchanged. I could be mistaken, but I think the CI failure on the other PR is exposing existing platform-specific CUDA numerical variance, so not sure how you'd like me to address that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants