Skip to content

ggml-hrx: support Gemma routed IDs and expert scales - #183

Merged
AaronStGeorge merged 5 commits into
hrx-graph-develop-v2from
users/astgeorg/gemma-hrx-bringup-reopened
Nov 9, 2026
Merged

AaronStGeorge merged 5 commits into
hrx-graph-develop-v2from
users/astgeorg/gemma-hrx-bringup-reopened

Conversation

@AaronStGeorge

@AaronStGeorge AaronStGeorge commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Reopens #180 after its revert in #182. This PR brings up Gemma 4 26B-A4B expert routing and scale selection on HRX.

Changes

  • Bind routed ID views to the exact span of (tokens - 1) * route_stride + 8 IDs. The router kernel uses the same bound, so Gemma's eight selected IDs per token work with its 128-ID row stride.
  • Add narrow GPU REPEAT and F32 scalar GET_ROWS dispatches for the expert-scale shape. They cover graphs with exposed intermediates or otherwise unmatched matmul fusion; these paths no longer require CPU execution.
  • Fuse MUL_MAT_ID + REPEAT + GET_ROWS + MUL for Gemma's routed down experts. Scaled skinny and tiled entry points reuse the existing matmul core and multiply each result by the selected expert's F32 scale as it is stored. The matcher requires matching route IDs, transient single-consumer intermediates, and Q5_1, Q8_0, Q4_K, or Q6_K weights. This GGUF has 29 Q5_1 and one Q8_0 down-expert tensors. All 30 use the fused path in the full Gemma graph, with no separate scale repeat, gather, or multiply launch.
  • Remove the earlier REPEAT → GET_ROWS matcher and its unused zero-stride gather option. The standalone GPU kernels remain. Ordinary matmul entry points have no added scale branch, load, or binding; five representative ordinary specializations produced byte-identical linked Loom, final Loom, and HSACO. The new matcher still adds graph-build work and binary size.

Validation

  • Production source validated at f4f4f8216613226341c8bea092e93e17e7b4cdd0 (tree fd7b04eb58bcc7a748e32c5d744b08cb4ad91620) with HRX 82214dfb029975205d1c3a59164b678e2e21fc41 on Radeon 8060S gfx1151, pinned with ROCR_VISIBLE_DEVICES=0. Current PR head 92f3fe648a353fed154f3b65939100c62cbc8d9e adds only test changes: named scalar-gather scenarios and host coverage for scaled matmul dispatch, stride-128 bindings, and fallback selection.
  • Model: unsloth/gemma-4-26B-A4B-it-GGUF revision c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-Q4_K_M.gguf, 16,947,541,728 bytes, SHA256 f2c28b3dc4776931ac6f879e11f203dec637ea0f14267a86ec8f6165f63f293f.
  • Focused cases at PR head: 10/10 scaled matmul device cases against a dequantized CPU reference, 8/8 scalar gather cases, 10/10 standalone repeat/gather device cases, and 2/2 focused host dispatch cases. The new host case checks scaled skinny and tiled selection, exact scale and ID bindings, and fallback when the matmul is exposed or gather IDs differ. Gemma completed nonempty prefill and autoregressive decode with 31/31 layers offloaded on the same production source.
  • Full WikiText-2 raw test split (SHA256 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08): one sequence, context 2048, flash attention ON, F16 K/V, batch/ubatch 256, mmap. All 144 chunks and 147,312 scored tokens produced PPL 20142.1478 ± 383.15903; every chunk matched the previous PR-head HRX run. Matched Vulkan PPL was 20050.1927 ± 382.20831.
  • Six representative models from ggml-staging-automation had identical baseline/candidate two-chunk PPL, including every chunk value. Their decode benchmark pairs were within 0.7%, and alternating-order rechecks found no repeatable prefill slowdown. None of the automation manifest's 30 models has ffn_down_exps.scale; these checks exercise the ordinary path. They preceded removal of the smaller matcher, which none selected.
  • Gemma A/B/B/A pp512/tg128 benchmarks before that removal measured HRX prefill means 752.68 → 760.11 tokens/s and decode means 34.08 → 34.21 tokens/s. The final build's full-PPL prompt rate was 752.97 tokens/s. Matched HRX/Vulkan/Vulkan/HRX decode measured 34.183, 34.203 versus 52.150, 51.555 tokens/s before the removal.

Limits

The high absolute Gemma perplexity and HRX/Vulkan difference predate this PR. A synthetic two-token padded VIEW ID case returns zero for token one on both ordinary and scaled matmul paths; the model's router-produced stride-128 IDs passed the full run. Vulkan remains faster for decode on this gfx1151 host. With GGML_HRX_LOG_WEIGHT_STATS=1, final teardown reported 659 weight materializations and 17,716,051,064 cached GPU weight bytes, identical to the previous PR head, with no duplicate native/packed warning. This PR does not change weight repacking or cache accounting.

Bind exact router ID view spans and gather scalar expert scales with runtime ID bounds checks. Validated on gfx1151 with focused operator/backend tests, full paired HRX/Vulkan perplexity, and generation. High absolute perplexity and CPU REPEAT fallback remain documented limitations.
@AaronStGeorge
AaronStGeorge merged commit 5acec43 into hrx-graph-develop-v2 Nov 9, 2026
4 of 10 checks passed
@AaronStGeorge
AaronStGeorge deleted the users/astgeorg/gemma-hrx-bringup-reopened branch November 9, 2026 03:17
AaronStGeorge added a commit to ROCm/ggml-staging-automation that referenced this pull request Nov 10, 2026
The ten Batch 1 architecture-gallery variants have been qualified locally but are missing from maintained CI coverage. Add their exact model pins to the ordinary `all` tier, including models that needed no backend changes, and select the normal full-model PR CI through the `ci:all-models` label.

Closes #129.

- Add all ten variants with immutable model revisions, filenames, sizes, and SHA-256 checksums; generation uses `mtp:off`.
- Pin llama.cpp to `9bf5871310a8d7de74f4cb0c57009835fbf91c27`, including the Gemma scale fusion from AMD-Ecosystem/llama.cpp#183 and the separately reviewed test correction from AMD-Ecosystem/llama.cpp#186.
- Pin HRX to validated revision `82214dfb029975205d1c3a59164b678e2e21fc41`, which supports the kernel corpus's typed numerical expectations.
- Keep Muse's HRX generation and perplexity report-only while the [weight-residency work](#122) following AMD-Ecosystem/llama.cpp#160 lands. Vulkan remains gating. Existing Qwen3.8-27B and Mistral-Small-3.2-24B memory exceptions are retained.

- Honor HTTP `Retry-After` on model download errors, accepting seconds and HTTP dates. Without a valid header, HTTP 429 retries back off for 60 then 120 seconds. Log the delay and retain three attempts, SHA-256 verification, and atomic publication.

Downloader validation: eight local checks passed for date formats, malformed headers, repeated throttling, successful verification, and exhaustion without a final sleep. `git diff --check` passes. Full-tier CI results for the Retry-After change have not yet been reviewed.

Full validation: [normal PR CI run 42661313071](https://github.com/ROCm/ggml-staging-automation/actions/runs/42661313071), selected by `ci:all-models` on commit `5be755ff9bbeb0c2ec3105e26356c92b01f3551c`, failed on model downloads. The build and all 563 packaged HRX cases passed. Four of the 40 models downloaded and passed generation and both perplexity regimes on HRX and Vulkan (five generation cases because Qwen3.5 MTP has on/off modes). The remaining 36 models were untested after all three download attempts returned HTTP 429 Too Many Requests. No model benchmark failure was recorded. Full-tier qualification remains incomplete; the immediate blocker is download rate limiting.

Prior qualification: [ten-model run 42645448512](https://github.com/ROCm/ggml-staging-automation/actions/runs/42645448512) and [regular CI run 42645459974](https://github.com/ROCm/ggml-staging-automation/actions/runs/42645459974) passed with the same source pins. Earlier ten-model artifacts recorded 120 measured generation requests without request failures and 40 successful perplexity measurements. All four packaged HRX suites also passed locally with the corrected backend test and unchanged CI runtime: buffer 8/8, backend 159/159, Loom JIT 6/6, operations 390/390 (563 total). `git diff --check` passes.

The existing profiles use context 4096 for generation and context 512 for prefill-like/decode-like perplexity. These results do not repeat the original full-corpus context-2048 survey or establish fit on a 24 GiB GPU. Muse reached approximately 28.95 GB peak VRAM on HRX versus 15.54 GB on Vulkan in the ten-model qualification.

Gemma 26B perplexity remains gating through the existing relative HRX/Vulkan check. Its absolute WikiText perplexity is unusually high on both backends; the earlier prefill difference was +7.93%, within the current 10% limit. Follow-up checks using the CI artifact produced sensible chat answers, with 13 of 14 final answers identical between backends. Both backends degenerated on an unframed WikiText prefix and continued coherently when the identical prefix was framed as an assistant response. This supports a raw-input/chat-format mismatch but does not explain the precise backend gap. Both also made the same sampled arithmetic error and added unwanted Markdown fences around JSON, so successful requests are not a blanket answer-quality pass. The manifest currently has no separate reason field for expectations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant