Skip to content

ggml-hrx: support Mellum routed experts and sliding-window attention - #177

Merged
AaronStGeorge merged 10 commits into
hrx-graph-develop-v2from
users/astgeorg/pr-mellum-routed-support
Nov 5, 2026
Merged

AaronStGeorge merged 10 commits into
hrx-graph-develop-v2from
users/astgeorg/pr-mellum-routed-support

Conversation

@AaronStGeorge

@AaronStGeorge AaronStGeorge commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

Enable Mellum 2.1 12B-A2.5B Thinking Q4_K_M text inference on HRX. Its graph needs full-row max reduction, a bounded clamp, and routed expert-down multiplication with K=896. After those operators executed, full-model validation exposed a separate attention bug: Qwen metadata regeneration overwrote Mellum's sliding-window masks and physical ring-cache indices. At absolute positions 1280–1535, valid slots 0–255 became out-of-range slots 1280–1535.

This draft targets Mellum contexts up to 2048, with operator support and the attention correctness fix in separate commits:

  • Reuse existing reduction/unary primitives inside the exact activation-scaling fusion; no standalone pool or clamp kernels are added.
  • Handle partial K blocks in the six shared routed-down exports using format-aware row strides and masked tail loads, preserving compact weights and existing arithmetic.
  • Preserve caller-supplied attention positions, cache indices, and masks. Remove unsafe dense/no-ring metadata regeneration and compact-mask substitution.
  • Fuse the exact activation maximum/scale/division chain into one F32 row kernel, retaining the protective scale and the existing down matmul/output rescale.

Rebase and cleanup

Rebased onto 59f141f2826218aea4245abf70336b34deb0afe9, the merged explicit RoPE factor fix (#179). Current head: e6e1b5f7ef0e219a0386ad815f0ad0d0e4c268f3.

  • Removed the implicit-frequency workaround from the metadata regression. It now exercises corrected explicit RoPE factors. The metadata regression has its own execution loop with fixed factors, input-preservation checks and graph-cache checks; input/weight/cache initialization is shared with the RoPE tests.
  • Kept the upstream explicit/implicit unit-factor parity, non-unit factors, scalar rotation, fused decode, and graph-reuse coverage.
  • Consolidated the three attention fix/follow-up commits into one; the five operator commits are unchanged by the rebase (git range-diff). A separate test-cleanup follow-up brought the total to seven commits; the scaling fusion, its boundary tests, and standalone-kernel cleanup bring the total to ten.
  • Current PR diff: 21 files, +976/-280. Removing the superseded standalone kernels, dispatchers, and tests reduces the previous head by 468 net lines.

Activation scaling fusion

Fusion introduced in 534c30d3c; boundary tests in 7ba92fa50; standalone-kernel cleanup in e6e1b5f7e. The fused kernel and matcher are unchanged by the cleanup.

  • One workgroup per row computes max(abs(x)), follows the existing scale/log/ceil/clamp/exp arithmetic, and publishes both F32 x/s and s. It reuses the existing unary motif unchanged. The down matmul and final output rescale are unchanged.
  • The matcher requires the exact graph constants and shapes, private single-consumer intermediates, and separate input/output storage. Live scale consumers remain supported. Changed constants, observed intermediates, shared consumers, and aliases do not silently lose values.
  • Two focused checks pass after cleanup: fusion dispatch/rejection and CPU-reference finite numerics at widths 1, 64, 257, 896, and 32768, with one and sixteen tokens. Coverage includes zeros, adjacent scaling thresholds, large finite values, tail maxima, alias exclusions, and explicit NaN/Inf expectations. Finite tolerances remain 3e-6 relative + 1e-7 absolute. The removed standalone-operator checks and fused/unfused comparison depended on the deleted implementations.
  • Explicit nonfinite checks cover all-NaN and mixed-NaN rows, signed infinity, finite saturated scales, and maximum finite input. CPU std::max can propagate a final NaN, while HRX maxnum ignores it. Historical fused/unfused comparisons established that the fusion preserved the prior HRX behavior; those logs remain in the evidence.
  • Independent review of the fusion and the exact cleanup commit found no blocking correctness issue. Instrumented model execution confirms 27 or 28 fused dispatches per batch (896-wide rows); the difference is last-layer output cropping in unscored PPL batches, not a missed eligible chain.
  • Two full WikiText-2 runs each for baseline 0f65778f7 and fusion 534c30d3c: all 147 printed cumulative PPL rows agree; final PPL is 9.5873 +/- 0.07421, with 29/29 layers offloaded. Mean prompt-evaluation time was 76.969 s -> 75.520 s (1.88% reduction); wall time was 85.029 s -> 83.522 s (1.77% reduction). Same GGUF/corpus, gfx1100, HRX82214df, c2048, b/ub256, F16 KV, flash attention, diagnostics off. Two sequential repeats are preliminary prefill measurements; no new decode-performance claim.
  • Fresh full WikiText-2 validation of cleanup head e6e1b5f7e: PPL 9.5873 +/- 0.07421, 147 chunks / 150,381 scored tokens, 29/29 layers offloaded, c2048. All 147 printed cumulative rows exactly match the original fusion run. Clean rebuild, both focused tests, and git diff --check pass. Evidence: cleanup.json and cleanup-gates/.
  • The cleanup removes pool_1d_full_max_f32 and clamp_f32, their manifest entries, dispatchers, and dedicated tests. The remaining dispatcher is named dispatch-amax-scale. Current model graphs use only the fused chain. Graphs with exposed/shared intermediates or changed constants now reject HRX graph scheduling; the PR no longer adds standalone pool/clamp support. Placement admission remains necessary to assemble the fused graph.

Affected-consumer inventory: the 45 local GGUFs contain only Mellum among architectures using this graph-level scaling. Laguna is not an established current HRX model: its attention SoftPlus is unsupported here. Historical Flint validation used a separate unmerged branch with attention-sink support absent from this baseline. Mistral4 has no established current-baseline validation in this survey. These remain unvalidated potential consumers, not newly claimed model support. No architecture-name gate was added to the generic matcher.

Evidence: investigation/mellum-scaling-fusion-experiment/ in the hrx-survey-mellum-implementation workspace thread, including exact provenance, command dumps, model inventory, numerical logs, and repeated comparison results.

Earlier metadata test cleanup

0f65778f7 separates run_attention_postprocess_metadata_reuse_case from the RoPE CPU-reference helper. Explicit factors stay fixed across the metadata test's three executions; RoPE scalar assertions run independently, with no metadata-mode exception. Graph construction and fixed input/weight/cache setup are reused.

Fresh build of clean head 0f65778f7: all five focused attention/RoPE tests passed (CPU unit factors, ordinary factor reference, fused decode factors, metadata reuse, decode scheduling). git diff --check passed. Production code is unchanged from the full-model validation below; no new perplexity run was needed for this test-only change.

Rebase validation before the test-only cleanup

Fresh build of exact clean head a4d8c6a47, HRX runtime source 82214df, RX 7900 XTX (gfx1100), with verified runtime hashes.

  • 51 focused checks passed: 42 routed-tail/aligned cases across six exports (Q8_0, Q5_1, Q1_0), three scaling checks, one attention dispatch check, and five RoPE/metadata/decode checks.
  • Full WikiText-2 at context 2048, batch/microbatch 256, F16 KV and flash attention: Mellum PPL 9.5873 ± 0.07421, 147 chunks / 150,381 scored tokens, 29/29 layers offloaded. This matches the earlier full-run result at the printed precision. The run completed in 85.022 s; this was a correctness run, not a matched performance comparison.
  • git diff --check passed. No new throughput claim is made by this rebase.

Historical comparison and consumer validation

These results remain attributed to their earlier tested revisions; they were not all rerun for this rebase.

  • Mellum at PR head 1bc440388, context 2048, full WikiText-2: HRX 9.5873 ± 0.07421, Vulkan 9.5729 ± 0.07400 (+0.1504%); 147 chunks / 150,381 scored tokens and 29/29 layers offloaded. Before the metadata correction, HRX PPL was 16,204.03. Aggregate PPL does not establish token-level equivalence.
  • At that earlier head, peak HRX VRAM 8.23 GiB; short greedy outputs matched. Five-repeat median throughput: pp256 4568.73 tok/s, tg128 100.251 tok/s. These are absolute measurements, not a speedup against the broken model.
  • Existing routed-kernel consumer regressions passed for Ornith 35B, Qwen3.5 MTP, Qwen3-Coder, Qwen3-30B, and Qwen3.6 on the worker's frozen combined baseline/candidate: unchanged PPL, worst median throughput drop 0.33%.
  • The attention fix was also built independently of Mellum operators. Qwen3-Coder PPL 8.1441 unchanged, pp/tg -0.192% / -0.051%; Qwen3-30B PPL 6.5567 unchanged, pp/tg +0.227% / -0.059%. Both completed 146 chunks; matched throughput used five-repeat medians.

Scope

Text inference at contexts up to 2048, non-MXFP4, flash attention enabled. Long-context support and CPU-fallback changes are deferred to a separate workstream; vision, MTP execution and non-flash attention are not claimed. Prior large-context experiments are historical evidence only and do not qualify this revised PR.

Earlier-head Mellum startup and 16-token generation also passed at context 2048 with -fa auto, which enabled flash attention.

Requirements

  • Contributing guidelines have been read by the agent; human review and acceptance remain pending for this draft.
  • AI usage disclosure: YES — Codex assisted with implementation, independent review, validation, and this PR description.

Remove metadata regeneration and compact-mask substitution. Reuse the merged RoPE factor fixture and graph-reuse checks for the mask and ring-cache regression; explicit factors now exercise the corrected production path.

Consolidates the metadata fix and its fixture/scope follow-ups without the reverted long-context fallback.

Bead: lemonade-ws-fnxc.4

Assisted-by: Codex
@AaronStGeorge
AaronStGeorge force-pushed the users/astgeorg/pr-mellum-routed-support branch from e9ce71a to a4d8c6a Compare November 5, 2026 02:40
Keep explicit factors fixed in the metadata regression and share input initialization. Restore the independent RoPE loop without a metadata mode or skipped scalar assertion.

Assisted-by: Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant