Skip to content

vulkan : improve long-context speculative decoding - #217

Closed
fhanuman wants to merge 4 commits into
masterfrom
halo-perf-opt-vulkan
Closed

fhanuman wants to merge 4 commits into
masterfrom
halo-perf-opt-vulkan

Conversation

@fhanuman

Copy link
Copy Markdown
Collaborator

Overview

Model-agnostic Vulkan optimizations for long-context, small-batch decoding.

  1. Share K/V across up to four query tokens per workgroup in GQA attention. Skip fully masked blocks and split visible KV work evenly across workgroups.
  2. Factor Q2_K integer scales out of dot-product sums, preserving exact integer results.
  3. Tune Strix Halo matvec scheduling: two output rows per workgroup for eight-column Q3_K on RADV; wave32 and two rows for eight-column Q2_K/Q3_K/Q4_K on AMD's proprietary driver.

Additional information

Model: MAI-Code-1.1-Flash-On-Device.

Vulkan build, host checks, style checks and port-equivalence review passed. GPU numerical validation, performance measurements, full CI and perplexity for this revision are pending; the earlier attention tolerance miss and sparse-attention slowdowns remain open.

Requirements

  • I have read and agree with the contributing guidelines.
  • AI usage disclosure: YES. Assisted by Codex with code, tests, review and this port.

fhanuman and others added 4 commits November 16, 2026 10:00
Fold query tokens into GQA rows and share K/V loads across up to four tokens. Skip masked cache blocks and balance split-K work over the visible range.

Preserve partial-token bounds and resource fallbacks. Add focused attention correctness and performance cases.

Assisted-by: Codex
(cherry picked from commit 86ede78b27ca0754e48ba0c5dc6b843a9af53e31)
Factor the shared integer scale out of the four dot-product terms and apply the minimum scale after summing activation bytes. Preserve exact integer results.

Add dense and MoE correctness cases. The destination already covers the Q2_K MoE performance shapes, so do not duplicate them.

Assisted-by: Codex
(cherry picked from commit 9e3bfc0b957e0d2898019ac66ded7076f52e0a6b)
Use the minimum subgroup size for integer MMVQ with AMD's proprietary driver. Use two output rows for eight-column Q2_K/Q3_K/Q4_K there, and for Q3_K on RADV.

Keep the tuning gated to Strix Halo and add Q3_K/Q4_K tail tests.

Co-authored-by: zz002 <zz002@users.noreply.github.com>
Assisted-by: Codex
(cherry picked from commit 3167fb3dcc337e3ef7b38d3d14443cbcef62db47)
Return false for invalid coopmat1 layouts instead of asserting. Preserve valid layouts and shared-memory checks.

Clarify tuning names and comments, and restore unrelated upstream formatting.

Assisted-by: Codex
(cherry picked from commit dc43ae86c7575847b6381d47c9200b5defcd9c74)
@fhanuman fhanuman closed this Nov 16, 2026
@fhanuman
fhanuman deleted the halo-perf-opt-vulkan branch November 16, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant