Repository navigation
ggml-hrx: support Mellum routed experts and sliding-window attention - #177
Merged
AaronStGeorge merged 10 commits intoNov 5, 2026
Merged
AaronStGeorge merged 10 commits into
AaronStGeorge merged 10 commits into
Conversation
Remove metadata regeneration and compact-mask substitution. Reuse the merged RoPE factor fixture and graph-reuse checks for the mask and ring-cache regression; explicit factors now exercise the corrected production path. Consolidates the metadata fix and its fixture/scope follow-ups without the reverted long-context fallback. Bead: lemonade-ws-fnxc.4 Assisted-by: Codex
AaronStGeorge
force-pushed
the
users/astgeorg/pr-mellum-routed-support
branch
from
November 5, 2026 02:40
e9ce71a to
a4d8c6a
Compare
Keep explicit factors fixed in the metadata regression and share input initialization. Restore the independent RoPE loop without a metadata mode or skipped scalar assertion. Assisted-by: Codex
AaronStGeorge
marked this pull request as ready for review
November 5, 2026 05:42
This was referenced Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Enable Mellum 2.1 12B-A2.5B Thinking Q4_K_M text inference on HRX. Its graph needs full-row max reduction, a bounded clamp, and routed expert-down multiplication with K=896. After those operators executed, full-model validation exposed a separate attention bug: Qwen metadata regeneration overwrote Mellum's sliding-window masks and physical ring-cache indices. At absolute positions 1280–1535, valid slots 0–255 became out-of-range slots 1280–1535.
This draft targets Mellum contexts up to 2048, with operator support and the attention correctness fix in separate commits:
Rebase and cleanup
Rebased onto
59f141f2826218aea4245abf70336b34deb0afe9, the merged explicit RoPE factor fix (#179). Current head:e6e1b5f7ef0e219a0386ad815f0ad0d0e4c268f3.git range-diff). A separate test-cleanup follow-up brought the total to seven commits; the scaling fusion, its boundary tests, and standalone-kernel cleanup bring the total to ten.Activation scaling fusion
Fusion introduced in
534c30d3c; boundary tests in7ba92fa50; standalone-kernel cleanup ine6e1b5f7e. The fused kernel and matcher are unchanged by the cleanup.0f65778f7and fusion534c30d3c: all 147 printed cumulative PPL rows agree; final PPL is 9.5873 +/- 0.07421, with 29/29 layers offloaded. Mean prompt-evaluation time was 76.969 s -> 75.520 s (1.88% reduction); wall time was 85.029 s -> 83.522 s (1.77% reduction). Same GGUF/corpus, gfx1100, HRX82214df, c2048, b/ub256, F16 KV, flash attention, diagnostics off. Two sequential repeats are preliminary prefill measurements; no new decode-performance claim.e6e1b5f7e: PPL 9.5873 +/- 0.07421, 147 chunks / 150,381 scored tokens, 29/29 layers offloaded, c2048. All 147 printed cumulative rows exactly match the original fusion run. Clean rebuild, both focused tests, andgit diff --checkpass. Evidence:cleanup.jsonandcleanup-gates/.pool_1d_full_max_f32andclamp_f32, their manifest entries, dispatchers, and dedicated tests. The remaining dispatcher is nameddispatch-amax-scale. Current model graphs use only the fused chain. Graphs with exposed/shared intermediates or changed constants now reject HRX graph scheduling; the PR no longer adds standalone pool/clamp support. Placement admission remains necessary to assemble the fused graph.Affected-consumer inventory: the 45 local GGUFs contain only Mellum among architectures using this graph-level scaling. Laguna is not an established current HRX model: its attention SoftPlus is unsupported here. Historical Flint validation used a separate unmerged branch with attention-sink support absent from this baseline. Mistral4 has no established current-baseline validation in this survey. These remain unvalidated potential consumers, not newly claimed model support. No architecture-name gate was added to the generic matcher.
Evidence:
investigation/mellum-scaling-fusion-experiment/in thehrx-survey-mellum-implementationworkspace thread, including exact provenance, command dumps, model inventory, numerical logs, and repeated comparison results.Earlier metadata test cleanup
0f65778f7separatesrun_attention_postprocess_metadata_reuse_casefrom the RoPE CPU-reference helper. Explicit factors stay fixed across the metadata test's three executions; RoPE scalar assertions run independently, with no metadata-mode exception. Graph construction and fixed input/weight/cache setup are reused.Fresh build of clean head
0f65778f7: all five focused attention/RoPE tests passed (CPU unit factors, ordinary factor reference, fused decode factors, metadata reuse, decode scheduling).git diff --checkpassed. Production code is unchanged from the full-model validation below; no new perplexity run was needed for this test-only change.Rebase validation before the test-only cleanup
Fresh build of exact clean head
a4d8c6a47, HRX runtime source82214df, RX 7900 XTX (gfx1100), with verified runtime hashes.git diff --checkpassed. No new throughput claim is made by this rebase.Historical comparison and consumer validation
These results remain attributed to their earlier tested revisions; they were not all rerun for this rebase.
1bc440388, context 2048, full WikiText-2: HRX 9.5873 ± 0.07421, Vulkan 9.5729 ± 0.07400 (+0.1504%); 147 chunks / 150,381 scored tokens and 29/29 layers offloaded. Before the metadata correction, HRX PPL was 16,204.03. Aggregate PPL does not establish token-level equivalence.Scope
Text inference at contexts up to 2048, non-MXFP4, flash attention enabled. Long-context support and CPU-fallback changes are deferred to a separate workstream; vision, MTP execution and non-flash attention are not claimed. Prior large-context experiments are historical evidence only and do not qualify this revised PR.
Earlier-head Mellum startup and 16-token generation also passed at context 2048 with
-fa auto, which enabled flash attention.Requirements