Skip to content

hrx: prefer row64 weights for dense Qwen3 - #196

Merged
AaronStGeorge merged 18 commits into
hrx-graph-develop-v2from
users/astgeorg/qwen3-row64-161
Nov 17, 2026
Merged

AaronStGeorge merged 18 commits into
hrx-graph-develop-v2from
users/astgeorg/qwen3-row64-161

Conversation

@AaronStGeorge

@AaronStGeorge AaronStGeorge commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Implements ROCm/ggml-staging-automation#161 by enabling HRX row64 weight preference for dense Qwen3 alongside Qwen3.5 and completing the readers needed to avoid redundant native weight copies.

  • Support native and row64 Q4/Q6 tied-embedding readers, including plain/fused GET_ROWS, and Q6 vocabulary projection. Preserve native host aliases where required.
  • Select M64 Q6 row64 tiles for short prefills (64-127 tokens, N=1024, K=1024/3072), retaining F32 accumulation.
  • Reuse Q6 row64 block scales and packed loads across four decode groups, with F32 centering in the F32-activation reader.
  • Route single-token Qwen Q6/Q8 endpoint projection through the shared vector interface; multi-token endpoints use the common prefill path. Keep CPU-reference coverage for both native and row64 T17 fallback.
  • Consolidate the Q6/Q8 endpoint ABI and vector entry body. Native and row64 arithmetic live in separate reusable motifs, with thin target-specific wrappers. Keep the pre-PR public names loom_libs:ggml_mul_mat_vector_f32_f32 and qwen3_moe:ggml_linear_q6k_q8_1_x4. This structural refactor is distinct from the layout and scheduling changes; the sweep below compares the entire PR against its base and does not isolate a refactor speedup.
  • Retain routing, layout, tied-embedding, residual, and full 151,936-row vocabulary numerical tests.

Fresh gfx1151 validation

Fresh sweep on AMD Radeon 8060S (gfx1151), 2026-11-16 UTC. Exact clean PR head at sweep start 02a8bcbe2d1d5b369b5e39ad305c660ccbc8350d versus exact base 4a5e53048ebc9db933a40b8f091c61dd5c302fea. Later test-only head 53dcc975f81395950fb970a5e0909786c10d088b restores native entry checks and removes a stale test comment; the production Loom-library files and export recipes are unchanged. Both measured builds use HRX 8d78611874afe8b7aaaf0227e28e4c3bd632bc99, ROCm SDK 10.1.0a20260822-1151, and RelWithDebInfo (-O2 -g -DNDEBUG). Separate builds through workspace sources/build.py --gfx 1151 --gpu-selection 0; model SHA256s and loaded-library identities recorded. Uncommitted vocabulary/FFN tuning was excluded.

Pinned ROCR_VISIBLE_DEVICES=0, exclusive shared GPU locks, temporary profile_standard restored afterward. Settings: --device HRX0 -ngl 99 -t 4 -fa on -p 66,128,512 -n 32 -b 512 -ub 512 -r 5; default warmup enabled. Two ABBA blocks per model, five repeats per process: 20 samples per revision/scenario, 48 timed processes and 960 timed samples in total. Graph capture runs are separate and excluded from all throughput statistics.

Values below are pooled median throughput changes, positive meaning faster. ppN processes N prompt tokens; tg32 generates 32 tokens. Repeats within a process are correlated; small deltas and differences between ABBA blocks are observations from this machine, not statistical significance or universal no-regression claims.

Model / quantization pp66 pp128 pp512 tg32
PromptBridge 0.6B Alpha Q4_K_M +7.97% +4.51% -0.86% -6.13%
Qwen3 1.7B Q4_0 +0.56% +0.44% +0.07% +3.09%
Qwen3 4B Instruct-2507 Q4_K_M +30.11% +33.74% -2.17% -1.18%
Qwen3.5 0.8B UD-Q4_K_XL +1.10% -0.49% -0.30% -1.75%
Qwen3.5 2B UD-Q4_K_XL +0.05% +0.70% +0.07% +2.14%
Qwen3.5 9B UD-Q4_K_XL +0.34% -0.37% -0.74% +1.07%

PromptBridge tg32 regressed in both ABBA blocks (-5.86% and -6.79%). The short-prefill gains therefore do not establish an across-the-board win.

The focused GPU suite at that measured revision passed 24/24 tests with unchanged tolerances: full-vocabulary projection, Q4/Q6 row64 GET_ROWS and fused RMSNorm readers, native/row64 single-token endpoints and tied embedding, native/row64 T17 fallback, M64 boundaries, and short/long residual cases. The later test-only commit adds four compact nonzero cases for the actual native wave32/wave64 entries at K=768/2048, N=9, T=1. All four passed normally and with access sanitization through Loom correctness preflight (8/8 checks); all four manifest plans and embedded-corpus generation passed. Expected values were independently verified with GGML scalar dequantization, and emitted artifacts confirm both wave widths. The original T2 reference case remains. This is focused coverage, not a rerun of every backend test or a new perplexity/quality sweep.

Separate Qwen3.5 0.8B UD-Q4_K_XL captures retained all graph variants: both base and head produced 424-, 380-, and 344-command programs. The two single-token variants coexist within each run; their 36-command difference is not a row64-policy delta. Earlier Q4_K_M captures had 427/381/345 commands and are a different model-file configuration.

Separate Qwen MoE coverage

Qwen3-30B-A3B-Instruct-2507 Q4_K_M, existing shared unsloth snapshot eea7b2be5805a5f151f8847ede8e5f9a9284bf77; SHA256 verified against that snapshot's LFS metadata. This is a 128-expert/8-active model with a native Q6_K vocabulary weight of K=2048, N=151936. It uses the same exact base/implementation revisions, build/runtime, GPU policy and benchmark settings as above, in a separate campaign: two ABBA blocks, 20 samples/revision/scenario, eight timed processes and 160 samples. Capture and generation runs are excluded.

Model / quantization pp66 pp128 pp512 tg32
Qwen3-30B-A3B-Instruct-2507 Q4_K_M +0.23% -0.19% -0.44% -0.56%

Captured graphs contain 705 commands for pp128 and 302 for decode at both revisions. All preceding command records match exactly. At the measured revision, only the final command record changes, using the consolidated native endpoint under the temporary name loom_libs:ggml_mul_mat_vector_q6_q8_f32 (native weight format 6, Q8 activation format 9). The later naming-only commit 66353815b restores its original public identity, qwen3_moe:ggml_linear_q6k_q8_1_x4, while retaining the common ABI and reusable motifs. This confirms that the shared native endpoint refactor affects compatible MoE vocabulary consumers even though the new default row64 preference is for dense Qwen3; it does not add a MoE expert-layout policy.

The base and implementation produced byte-identical text for one 32-token greedy completion (The capital of France is, temperature 0, seed 1, context 256). The focused numerical suite above includes the corresponding native Q6/Q8 endpoint CPU-reference case. This is one-model routing/performance coverage plus a short output smoke, not a MoE perplexity sweep or validation of every expert configuration.

Isolated M64 versus M128 on gfx1151

Direct comparison of the existing PR kernels at implementation 02a8bcbe2, HRX 8d78611874, Radeon 8060S. Same Q6 row64 weights, F32 inputs, F32 accumulation, N=1024, and exact token capacity; only the M64/M128 entry changes. These are kernel-invocation measurements, not whole-model improvements.

Tokens K M128 completion (us) M64 completion (us) Reduction
64 1024 110.36 105.77 4.2%
64 3072 306.25 287.64 6.1%
66 1024 151.20 112.86 25.4%
66 3072 496.09 312.06 37.1%
96 1024 169.66 114.61 32.4%
96 3072 436.11 308.12 29.3%
127 1024 205.24 117.89 42.6%
127 3072 664.85 353.56 46.8%

Values are medians of six per-run host completion p50s per kernel, normalized by batch8. Three ABBA blocks per shape,16 independent binding sets,200ms warmup,minimum100 iterations/400ms; shared GPU locks and temporary standard profiling power restored after execution. M64 lowered mean host completion in all24 balanced blocks. Gains vary substantially between blocks, so the table supports the direction of improvement rather than a precise portable percentage. Separate eight-dispatch device replays had latency spikes/instability warnings; raw means and medians are retained but not used as precise device-only gain claims.

All96 confirmation and8 pilot timing processes passed a nonzero independent CPU reference at unchanged atol0.0001,rtol0; four pp66 access-sanitized checks passed. Fixtures use independently decoded random Q6 payload/scales and dyadic inputs/scales to avoid conflating layout/scheduling with F16 operand-rounding error. At pp66, VGPRs116->84, no spills; modeled residency remains8 waves/SIMD, limited by LDS. M64 launches16 rather than8 workgroups at65–127tokens; at64tokens both launch8. Evidence supports retaining the existing M64 selection on this gfx1151, not broadening its shape domain. Raw evidence remains local in artifacts/m64-gfx1151/.

Historical gfx1100 evidence

These results describe earlier implementation stages, not a fresh test of 02a8bcbe2 on gfx1100:

  • RX 7900 XTX, HRX 82214dfb029975205d1c3a59164b678e2e21fc41, ROCm 10.1.0a20260822-1100: the earlier six-model sweep at source matching b47fa040f versus original pre-row64 baseline 010b2ee82630562b71ea9232a1c5b2f69bc02bae used ABBA, 10 samples/revision/scenario. PromptBridge pp66 was -2.06%, pp128 +5.48%, pp512 +0.33%, tg32 +0.21%. These earlier numbers must not be combined with the current gfx1151 table.
  • The retained Q6/F32 row64 load-reuse change reduced isolated device time from 164.550 to 145.218 us on that machine, with independent numerical checks. This is an F32-activation vocabulary-kernel result, not evidence for the later Q6/Q8 structural refactor.
  • Later shared-endpoint work at caf562616 retained a measured T1 kernel cost: old row64 288.003 us versus shared 305.645 us (+6.13%); corresponding host p50 303.831 versus 319.944 us (+5.31%). T17 moved to the common tiled prefill path. These isolated earlier measurements do not substitute for current end-to-end results.

The earlier six-model quality comparison isolated the load-reuse change against pre-reuse PR commit 2fc6ec87b: four 256-token WikiText chunks, batch 256, ubatch 1, four threads, flash attention and GPU offload. All six had 100% top-token agreement, matching displayed perplexity, and mean KL within 1e-6 of zero. Candidate PPL was 45.691690 / 28.136606 / 14.662836 / 27.263403 / 18.983960 / 11.016162 in the six-model dense table order above. Saved-logit quantization explains tiny negative KL or reconstructed baseline-PPL differences. This does not establish current-head PPL or bitwise equality of all outputs.

Limits and retained evidence

  • Naming-only follow-up 66353815b builds through build.py and passes 14 focused gfx1151 GPU cases, four native wave32/wave64 correctness cases, the JIT materialization case, and four host scheduling cases. The host test qwen_terminal_q6k_q8_command.tokens18 fails its temporary-buffer-count assertion on both this edit and unchanged head 0bad06b7f; the numerical T17 cases pass. No new model sweep was run for this naming-only change.
  • The current sweep covers one gfx1151 machine, six dense model files, and one separately measured MoE model. The remaining throughput tradeoffs are tracked in execution bead lemonade-ws-s9tq; the PR does not claim universal improvement or absence of regressions.
  • An inherited Q4 tiled F16-accumulation accuracy issue at K=2560 remains tracked in lemonade-ws-ve03. Earlier layout tests found native and row64 outputs bitwise identical; the independent CPU-reference reproducer remains and tolerances were not relaxed.
  • Current raw commands, source/model/binary/library identities, all samples, telemetry, correctness logs, graph variants, and summaries are retained locally under artifacts/pr196-refresh/. Historical gfx1100 investigation evidence remains under investigation/qwen3-row64-161/. These experiment artifacts are outside the source PR.

AI-assisted implementation and investigation with Codex.

Measured direct Q6 kernel gains at the captured PromptBridge shapes and verified independent CPU output references before routing. Preserve M128 for other dimensions and prompt lengths.

Bead: lemonade-ws-aj0j.6

Assisted-by: Codex
@AaronStGeorge
AaronStGeorge force-pushed the users/astgeorg/qwen3-row64-161 branch from b47fa04 to a27a4be Compare November 16, 2026 00:42
Restore the general vector and native Q6/Q8 public identities while retaining the reusable motifs, common endpoint ABI, and launch schedules. Update catalog references and existing tests together.
@AaronStGeorge
AaronStGeorge marked this pull request as ready for review November 17, 2026 17:13
@AaronStGeorge
AaronStGeorge merged commit 083d1b6 into hrx-graph-develop-v2 Nov 17, 2026
3 of 12 checks passed
@AaronStGeorge
AaronStGeorge deleted the users/astgeorg/qwen3-row64-161 branch November 17, 2026 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant