Repository navigation
hrx: prefer row64 weights for dense Qwen3 - #196
Merged
AaronStGeorge merged 18 commits intoNov 17, 2026
Merged
Conversation
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Measured direct Q6 kernel gains at the captured PromptBridge shapes and verified independent CPU output references before routing. Preserve M128 for other dimensions and prompt lengths. Bead: lemonade-ws-aj0j.6 Assisted-by: Codex
Assisted-by: Codex
AaronStGeorge
force-pushed
the
users/astgeorg/qwen3-row64-161
branch
from
November 16, 2026 00:42
b47fa04 to
a27a4be
Compare
Assisted-by: Codex
Restore the general vector and native Q6/Q8 public identities while retaining the reusable motifs, common endpoint ABI, and launch schedules. Update catalog references and existing tests together.
AaronStGeorge
marked this pull request as ready for review
November 17, 2026 17:13
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements ROCm/ggml-staging-automation#161 by enabling HRX row64 weight preference for dense Qwen3 alongside Qwen3.5 and completing the readers needed to avoid redundant native weight copies.
loom_libs:ggml_mul_mat_vector_f32_f32andqwen3_moe:ggml_linear_q6k_q8_1_x4. This structural refactor is distinct from the layout and scheduling changes; the sweep below compares the entire PR against its base and does not isolate a refactor speedup.Fresh gfx1151 validation
Fresh sweep on AMD Radeon 8060S (gfx1151), 2026-11-16 UTC. Exact clean PR head at sweep start
02a8bcbe2d1d5b369b5e39ad305c660ccbc8350dversus exact base4a5e53048ebc9db933a40b8f091c61dd5c302fea. Later test-only head53dcc975f81395950fb970a5e0909786c10d088brestores native entry checks and removes a stale test comment; the production Loom-library files and export recipes are unchanged. Both measured builds use HRX8d78611874afe8b7aaaf0227e28e4c3bd632bc99, ROCm SDK10.1.0a20260822-1151, and RelWithDebInfo (-O2 -g -DNDEBUG). Separate builds through workspacesources/build.py --gfx 1151 --gpu-selection 0; model SHA256s and loaded-library identities recorded. Uncommitted vocabulary/FFN tuning was excluded.Pinned
ROCR_VISIBLE_DEVICES=0, exclusive shared GPU locks, temporaryprofile_standardrestored afterward. Settings:--device HRX0 -ngl 99 -t 4 -fa on -p 66,128,512 -n 32 -b 512 -ub 512 -r 5; default warmup enabled. Two ABBA blocks per model, five repeats per process: 20 samples per revision/scenario, 48 timed processes and 960 timed samples in total. Graph capture runs are separate and excluded from all throughput statistics.Values below are pooled median throughput changes, positive meaning faster.
ppNprocesses N prompt tokens;tg32generates 32 tokens. Repeats within a process are correlated; small deltas and differences between ABBA blocks are observations from this machine, not statistical significance or universal no-regression claims.PromptBridge tg32 regressed in both ABBA blocks (-5.86% and -6.79%). The short-prefill gains therefore do not establish an across-the-board win.
The focused GPU suite at that measured revision passed 24/24 tests with unchanged tolerances: full-vocabulary projection, Q4/Q6 row64 GET_ROWS and fused RMSNorm readers, native/row64 single-token endpoints and tied embedding, native/row64 T17 fallback, M64 boundaries, and short/long residual cases. The later test-only commit adds four compact nonzero cases for the actual native wave32/wave64 entries at K=768/2048, N=9, T=1. All four passed normally and with access sanitization through Loom correctness preflight (8/8 checks); all four manifest plans and embedded-corpus generation passed. Expected values were independently verified with GGML scalar dequantization, and emitted artifacts confirm both wave widths. The original T2 reference case remains. This is focused coverage, not a rerun of every backend test or a new perplexity/quality sweep.
Separate Qwen3.5 0.8B UD-Q4_K_XL captures retained all graph variants: both base and head produced 424-, 380-, and 344-command programs. The two single-token variants coexist within each run; their 36-command difference is not a row64-policy delta. Earlier Q4_K_M captures had 427/381/345 commands and are a different model-file configuration.
Separate Qwen MoE coverage
Qwen3-30B-A3B-Instruct-2507 Q4_K_M, existing shared
unslothsnapshoteea7b2be5805a5f151f8847ede8e5f9a9284bf77; SHA256 verified against that snapshot's LFS metadata. This is a 128-expert/8-active model with a native Q6_K vocabulary weight of K=2048, N=151936. It uses the same exact base/implementation revisions, build/runtime, GPU policy and benchmark settings as above, in a separate campaign: two ABBA blocks, 20 samples/revision/scenario, eight timed processes and 160 samples. Capture and generation runs are excluded.Captured graphs contain 705 commands for pp128 and 302 for decode at both revisions. All preceding command records match exactly. At the measured revision, only the final command record changes, using the consolidated native endpoint under the temporary name
loom_libs:ggml_mul_mat_vector_q6_q8_f32(native weight format 6, Q8 activation format 9). The later naming-only commit66353815brestores its original public identity,qwen3_moe:ggml_linear_q6k_q8_1_x4, while retaining the common ABI and reusable motifs. This confirms that the shared native endpoint refactor affects compatible MoE vocabulary consumers even though the new default row64 preference is for dense Qwen3; it does not add a MoE expert-layout policy.The base and implementation produced byte-identical text for one 32-token greedy completion (
The capital of France is, temperature 0, seed 1, context 256). The focused numerical suite above includes the corresponding native Q6/Q8 endpoint CPU-reference case. This is one-model routing/performance coverage plus a short output smoke, not a MoE perplexity sweep or validation of every expert configuration.Isolated M64 versus M128 on gfx1151
Direct comparison of the existing PR kernels at implementation
02a8bcbe2, HRX8d78611874, Radeon 8060S. Same Q6 row64 weights, F32 inputs, F32 accumulation, N=1024, and exact token capacity; only the M64/M128 entry changes. These are kernel-invocation measurements, not whole-model improvements.Values are medians of six per-run host completion p50s per kernel, normalized by batch8. Three ABBA blocks per shape,16 independent binding sets,200ms warmup,minimum100 iterations/400ms; shared GPU locks and temporary standard profiling power restored after execution. M64 lowered mean host completion in all24 balanced blocks. Gains vary substantially between blocks, so the table supports the direction of improvement rather than a precise portable percentage. Separate eight-dispatch device replays had latency spikes/instability warnings; raw means and medians are retained but not used as precise device-only gain claims.
All96 confirmation and8 pilot timing processes passed a nonzero independent CPU reference at unchanged atol0.0001,rtol0; four pp66 access-sanitized checks passed. Fixtures use independently decoded random Q6 payload/scales and dyadic inputs/scales to avoid conflating layout/scheduling with F16 operand-rounding error. At pp66, VGPRs116->84, no spills; modeled residency remains8 waves/SIMD, limited by LDS. M64 launches16 rather than8 workgroups at65–127tokens; at64tokens both launch8. Evidence supports retaining the existing M64 selection on this gfx1151, not broadening its shape domain. Raw evidence remains local in
artifacts/m64-gfx1151/.Historical gfx1100 evidence
These results describe earlier implementation stages, not a fresh test of
02a8bcbe2on gfx1100:82214dfb029975205d1c3a59164b678e2e21fc41, ROCm10.1.0a20260822-1100: the earlier six-model sweep at source matchingb47fa040fversus original pre-row64 baseline010b2ee82630562b71ea9232a1c5b2f69bc02baeused ABBA, 10 samples/revision/scenario. PromptBridge pp66 was -2.06%, pp128 +5.48%, pp512 +0.33%, tg32 +0.21%. These earlier numbers must not be combined with the current gfx1151 table.caf562616retained a measured T1 kernel cost: old row64 288.003 us versus shared 305.645 us (+6.13%); corresponding host p50 303.831 versus 319.944 us (+5.31%). T17 moved to the common tiled prefill path. These isolated earlier measurements do not substitute for current end-to-end results.The earlier six-model quality comparison isolated the load-reuse change against pre-reuse PR commit
2fc6ec87b: four 256-token WikiText chunks, batch 256, ubatch 1, four threads, flash attention and GPU offload. All six had 100% top-token agreement, matching displayed perplexity, and mean KL within 1e-6 of zero. Candidate PPL was 45.691690 / 28.136606 / 14.662836 / 27.263403 / 18.983960 / 11.016162 in the six-model dense table order above. Saved-logit quantization explains tiny negative KL or reconstructed baseline-PPL differences. This does not establish current-head PPL or bitwise equality of all outputs.Limits and retained evidence
66353815bbuilds throughbuild.pyand passes 14 focused gfx1151 GPU cases, four native wave32/wave64 correctness cases, the JIT materialization case, and four host scheduling cases. The host testqwen_terminal_q6k_q8_command.tokens18fails its temporary-buffer-count assertion on both this edit and unchanged head0bad06b7f; the numerical T17 cases pass. No new model sweep was run for this naming-only change.lemonade-ws-s9tq; the PR does not claim universal improvement or absence of regressions.lemonade-ws-ve03. Earlier layout tests found native and row64 outputs bitwise identical; the independent CPU-reference reproducer remains and tolerances were not relaxed.artifacts/pr196-refresh/. Historical gfx1100 investigation evidence remains underinvestigation/qwen3-row64-161/. These experiment artifacts are outside the source PR.AI-assisted implementation and investigation with Codex.