Repository navigation
Conversation
Fold query tokens into GQA rows and share K/V loads across up to four tokens. Skip masked cache blocks and balance split-K work over the visible range. Preserve partial-token bounds and resource fallbacks. Add focused attention correctness and performance cases. Assisted-by: Codex (cherry picked from commit 86ede78b27ca0754e48ba0c5dc6b843a9af53e31)
Factor the shared integer scale out of the four dot-product terms and apply the minimum scale after summing activation bytes. Preserve exact integer results. Add dense and MoE correctness cases. The destination already covers the Q2_K MoE performance shapes, so do not duplicate them. Assisted-by: Codex (cherry picked from commit 9e3bfc0b957e0d2898019ac66ded7076f52e0a6b)
Use the minimum subgroup size for integer MMVQ with AMD's proprietary driver. Use two output rows for eight-column Q2_K/Q3_K/Q4_K there, and for Q3_K on RADV. Keep the tuning gated to Strix Halo and add Q3_K/Q4_K tail tests. Co-authored-by: zz002 <zz002@users.noreply.github.com> Assisted-by: Codex (cherry picked from commit 3167fb3dcc337e3ef7b38d3d14443cbcef62db47)
Return false for invalid coopmat1 layouts instead of asserting. Preserve valid layouts and shared-memory checks. Clarify tuning names and comments, and restore unrelated upstream formatting. Assisted-by: Codex (cherry picked from commit dc43ae86c7575847b6381d47c9200b5defcd9c74)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Model-agnostic Vulkan optimizations for long-context, small-batch decoding.
Additional information
Model: MAI-Code-1.1-Flash-On-Device.
Vulkan build, host checks, style checks and port-equivalence review passed. GPU numerical validation, performance measurements, full CI and perplexity for this revision are pending; the earlier attention tolerance miss and sparse-attention slowdowns remain open.
Requirements