Repository navigation
ggml-hrx: support Gemma routed IDs and expert scales - #183
Merged
AaronStGeorge merged 5 commits intoNov 9, 2026
Merged
AaronStGeorge merged 5 commits into
AaronStGeorge merged 5 commits into
Conversation
Bind exact router ID view spans and gather scalar expert scales with runtime ID bounds checks. Validated on gfx1151 with focused operator/backend tests, full paired HRX/Vulkan perplexity, and generation. High absolute perplexity and CPU REPEAT fallback remain documented limitations.
AaronStGeorge
added a commit
to ROCm/ggml-staging-automation
that referenced
this pull request
Nov 10, 2026
The ten Batch 1 architecture-gallery variants have been qualified locally but are missing from maintained CI coverage. Add their exact model pins to the ordinary `all` tier, including models that needed no backend changes, and select the normal full-model PR CI through the `ci:all-models` label. Closes #129. - Add all ten variants with immutable model revisions, filenames, sizes, and SHA-256 checksums; generation uses `mtp:off`. - Pin llama.cpp to `9bf5871310a8d7de74f4cb0c57009835fbf91c27`, including the Gemma scale fusion from AMD-Ecosystem/llama.cpp#183 and the separately reviewed test correction from AMD-Ecosystem/llama.cpp#186. - Pin HRX to validated revision `82214dfb029975205d1c3a59164b678e2e21fc41`, which supports the kernel corpus's typed numerical expectations. - Keep Muse's HRX generation and perplexity report-only while the [weight-residency work](#122) following AMD-Ecosystem/llama.cpp#160 lands. Vulkan remains gating. Existing Qwen3.8-27B and Mistral-Small-3.2-24B memory exceptions are retained. - Honor HTTP `Retry-After` on model download errors, accepting seconds and HTTP dates. Without a valid header, HTTP 429 retries back off for 60 then 120 seconds. Log the delay and retain three attempts, SHA-256 verification, and atomic publication. Downloader validation: eight local checks passed for date formats, malformed headers, repeated throttling, successful verification, and exhaustion without a final sleep. `git diff --check` passes. Full-tier CI results for the Retry-After change have not yet been reviewed. Full validation: [normal PR CI run 42661313071](https://github.com/ROCm/ggml-staging-automation/actions/runs/42661313071), selected by `ci:all-models` on commit `5be755ff9bbeb0c2ec3105e26356c92b01f3551c`, failed on model downloads. The build and all 563 packaged HRX cases passed. Four of the 40 models downloaded and passed generation and both perplexity regimes on HRX and Vulkan (five generation cases because Qwen3.5 MTP has on/off modes). The remaining 36 models were untested after all three download attempts returned HTTP 429 Too Many Requests. No model benchmark failure was recorded. Full-tier qualification remains incomplete; the immediate blocker is download rate limiting. Prior qualification: [ten-model run 42645448512](https://github.com/ROCm/ggml-staging-automation/actions/runs/42645448512) and [regular CI run 42645459974](https://github.com/ROCm/ggml-staging-automation/actions/runs/42645459974) passed with the same source pins. Earlier ten-model artifacts recorded 120 measured generation requests without request failures and 40 successful perplexity measurements. All four packaged HRX suites also passed locally with the corrected backend test and unchanged CI runtime: buffer 8/8, backend 159/159, Loom JIT 6/6, operations 390/390 (563 total). `git diff --check` passes. The existing profiles use context 4096 for generation and context 512 for prefill-like/decode-like perplexity. These results do not repeat the original full-corpus context-2048 survey or establish fit on a 24 GiB GPU. Muse reached approximately 28.95 GB peak VRAM on HRX versus 15.54 GB on Vulkan in the ten-model qualification. Gemma 26B perplexity remains gating through the existing relative HRX/Vulkan check. Its absolute WikiText perplexity is unusually high on both backends; the earlier prefill difference was +7.93%, within the current 10% limit. Follow-up checks using the CI artifact produced sensible chat answers, with 13 of 14 final answers identical between backends. Both backends degenerated on an unframed WikiText prefix and continued coherently when the identical prefix was framed as an assistant response. This supports a raw-input/chat-format mismatch but does not explain the precise backend gap. Both also made the same sampled arithmetic error and added unwanted Markdown fences around JSON, so successful requests are not a blanket answer-quality pass. The manifest currently has no separate reason field for expectations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reopens #180 after its revert in #182. This PR brings up Gemma 4 26B-A4B expert routing and scale selection on HRX.
Changes
(tokens - 1) * route_stride + 8IDs. The router kernel uses the same bound, so Gemma's eight selected IDs per token work with its 128-ID row stride.REPEATand F32 scalarGET_ROWSdispatches for the expert-scale shape. They cover graphs with exposed intermediates or otherwise unmatched matmul fusion; these paths no longer require CPU execution.MUL_MAT_ID + REPEAT + GET_ROWS + MULfor Gemma's routed down experts. Scaled skinny and tiled entry points reuse the existing matmul core and multiply each result by the selected expert's F32 scale as it is stored. The matcher requires matching route IDs, transient single-consumer intermediates, and Q5_1, Q8_0, Q4_K, or Q6_K weights. This GGUF has 29 Q5_1 and one Q8_0 down-expert tensors. All 30 use the fused path in the full Gemma graph, with no separate scale repeat, gather, or multiply launch.REPEAT → GET_ROWSmatcher and its unused zero-stride gather option. The standalone GPU kernels remain. Ordinary matmul entry points have no added scale branch, load, or binding; five representative ordinary specializations produced byte-identical linked Loom, final Loom, and HSACO. The new matcher still adds graph-build work and binary size.Validation
f4f4f8216613226341c8bea092e93e17e7b4cdd0(treefd7b04eb58bcc7a748e32c5d744b08cb4ad91620) with HRX82214dfb029975205d1c3a59164b678e2e21fc41on Radeon 8060Sgfx1151, pinned withROCR_VISIBLE_DEVICES=0. Current PR head92f3fe648a353fed154f3b65939100c62cbc8d9eadds only test changes: named scalar-gather scenarios and host coverage for scaled matmul dispatch, stride-128 bindings, and fallback selection.unsloth/gemma-4-26B-A4B-it-GGUFrevisionc099eb48e663fd284577b04978a94ffccb261841, filegemma-4-26B-A4B-it-UD-Q4_K_M.gguf, 16,947,541,728 bytes, SHA256f2c28b3dc4776931ac6f879e11f203dec637ea0f14267a86ec8f6165f63f293f.173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08): one sequence, context 2048, flash attention ON, F16 K/V, batch/ubatch 256, mmap. All 144 chunks and 147,312 scored tokens produced PPL20142.1478 ± 383.15903; every chunk matched the previous PR-head HRX run. Matched Vulkan PPL was20050.1927 ± 382.20831.ggml-staging-automationhad identical baseline/candidate two-chunk PPL, including every chunk value. Their decode benchmark pairs were within 0.7%, and alternating-order rechecks found no repeatable prefill slowdown. None of the automation manifest's 30 models hasffn_down_exps.scale; these checks exercise the ordinary path. They preceded removal of the smaller matcher, which none selected.752.68 → 760.11tokens/s and decode means34.08 → 34.21tokens/s. The final build's full-PPL prompt rate was 752.97 tokens/s. Matched HRX/Vulkan/Vulkan/HRX decode measured34.183, 34.203versus52.150, 51.555tokens/s before the removal.Limits
The high absolute Gemma perplexity and HRX/Vulkan difference predate this PR. A synthetic two-token padded
VIEWID case returns zero for token one on both ordinary and scaled matmul paths; the model's router-produced stride-128 IDs passed the full run. Vulkan remains faster for decode on thisgfx1151host. WithGGML_HRX_LOG_WEIGHT_STATS=1, final teardown reported 659 weight materializations and 17,716,051,064 cached GPU weight bytes, identical to the previous PR head, with no duplicate native/packed warning. This PR does not change weight repacking or cache accounting.