Skip to content

Improve metal memory usage for SDPA D512 - #4487

Open
dhiltgen wants to merge 1 commit into
ml-explore:mainfrom
dhiltgen:sdpa-nax-d512
Open

Improve metal memory usage for SDPA D512#4487
dhiltgen wants to merge 1 commit into
ml-explore:mainfrom
dhiltgen:sdpa-nax-d512

Conversation

@dhiltgen

Copy link
Copy Markdown
Contributor

This helps improve the memory usage of D512 attention on NAX GPUs with models such as gemma4

Testing with mlx-community/gemma-4-31b-it-nvfp4 and mlx-community/gemma-4-12B-it-nvfp4

Model Prompt main peak this branch peak Memory saved
Gemma 4 31B 16K 23.9515 GB 22.0368 GB 1.9147 GB (7.99%)
Gemma 4 12B 16K 9.7047 GB 8.7595 GB 0.9452 GB (9.74%)
Gemma 4 12B 65K 13.8004 GB 9.5775 GB 4.2228 GB (30.60%)

Both prompt and generation performance are unchanged.

  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: co-developed with coding agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant