Repository navigation
cuda: EXL3 routed experts behind experts.Grouped (#593, stacked on #740) - #741
Conversation
kernels/cuda/exl3.cu decodes EXL3 trellis tiles as checkpoints store them ([K/16, N/16, 16 bits] int16), with core/exl3_format.zig's codebooks (3inst, mcg, mul1) and whole widths 1-8. Rows are rotated by suh and the 128-point Hadamard transform, scaled by 2^-4 and rounded once to fp16; each tile decodes once straight into the B fragments of two mma.m16n8k16 (the tile's value order is theirs), fp32 sums, then the output transform, svh and 2^-3 (docs/recipes/exl3.md, Prompts). Decode rows go 1-16 a launch, read as A fragments from rows written in fragment order; matrices with fewer than 96 column blocks split their input blocks into a shape-fixed number of parts that the last part sums in order. Prompt rows go 64 a block from shared memory. Every row is independent, so a row's bits are the same at every row count on its path. qlinear takes it as Weight.exl3, refused with FormatNotLoaded when the module is absent. tf-cuda-test exl3 [DIR]: every codebook state and random tiles at every width bit-equal to exl3_format; FP64 references from exl3_format.dequantize with a bound for the kernels' rounding order, worst column and aggregate error; decode windows of 1-16 rows and prompts of 2-300 rows byte-equal to each row alone; DIR adds real matrices of a checkpoint and times them.
kernels/cuda/exl3_experts.cu runs a layer's EXL3 experts on grouped.zig's plan: up takes token rows through gate and up under experts.cuh's SwiGLU, down takes pair rows to fp32 or bf16. Each expert's rows are rotated by its own suh inline. Decode chunks of 1-2 pairs rotate in each warp's registers and shuffle a tile row's inputs to its lanes, larger chunks rotate once into shared memory, both in one fp32 order, so a pair's bits are the same alone, grouped and at any decode tile. Prompt rows take exl3.cu's prompt arithmetic, an item's 64 pairs a block on the tensor units, gate and up side by side. Format.exl3 and Layer.exl3 (widths, codebook) carry the layer; a layer's experts share one width. tf-cuda-test exl3-experts [DIR]: on both paths, gate, up and down per pair within FP64 bounds, SwiGLU of the path's own gate and up, tokens routed alone byte-equal to the grouped rows, a skipped expert left untouched; DIR adds 16 experts of a real layer and times a decode step and a 512-token prompt.
…3 experts Checked ash hart PRs open+closed (read-only). Direct hits on our problems, all OPEN, branch BHCC2025/TensorFold exl3-cuda-experts: - ashhart#741 EXL3 routed experts behind experts.Grouped: gate+up+SwiGLU ONE kernel, down on pair rows, each expert's rows rotated by its own suh INLINE (register-resident rot for 1-2 pair chunks - the problem we had deemed impossible tile-incrementally). Rides on grouped.zig's Plan (in our tree). 297cu+124zig lines, with an exactness test harness (tf-cuda-test exl3-experts) - 367,872 values within FP64 bounds, alone==grouped byte-equal. - ashhart#740 EXL3 projections behind qlinear (the stacked base). - ashhart#718 branch-free trellis window extraction (perf). - ashhart#729 share decoded weights across prefill rows (perf). - ashhart#728 MiMo-V2.6 CUDA EXL3 two-GPU (another full CUDA EXL3 family). - ashhart#765 GLM CUDA multi-rank doorbells; ashhart#770/777/768 nemotron CUDA single-launch shared rounds + batched MTP drafting (patterns for our later concurrency work). PLAN: vendored exl3_experts.cu/.zig + exl3.cuh into our tree (this commit, unwired). Next: build wiring (fatbin + mod), port moeBlock to Grouped.decode with the router->plan plumbing from grouped.zig, run their exl3-experts test on our checkpoint, parity, bench. Our own grouped kernels (stage-1 parity) stay as fallback.
… experts.zig + experts.cuh, fatbin wired (exl3_experts, -O3 --fmad=false), kernels.zig blob, root.zig imports. Zero errors - the whole PR stack compiles in our tree (its exl3.cu changes ride our existing fatbin untouched). NEXT (the wire-up finish): (1) moeBlock port to exl3_experts.Grouped - needs grouped.zig Router.route plumbing from our device moe_pick (check Plan layout: pairs=count=topk at rows=1) and the Layer.exl3 geometry from our w.gate/w.up/w.down weights; (2) their tf-cuda-test exl3-experts suite on OUR checkpoint; (3) parity vs host loop; (4) THE bench vs ~40 tok/s. Our stage-1 kernels remain the fallback.
OLD moe_grouped dispatch (line 1264, the stage-1/2/3 bisect code, group_stage=3 = full old path) intercepts rows==1 FIRST via its own lazy init (line 1241, sets moe_grouped=true) and RETURNS before the m41 block (1361/1364) is ever reached - EXCEPT the tail-dump instrument (line ~1316) sits INSIDE the old dispatch's flow, reading m41 buffers (m41_out ptr=0 when only the old init ran... hence the per-spawn-unique garbage 'dn' values = downloads from unset offsets). The 'up=0/dn=garbage' pattern is the OLD path running + the instrument reading M41 buffers never allocated. m41 never dispatched. FIX (next session, one edit): DELETE the old moe_grouped dispatch block (1241 through its returns) + its init, leaving ONLY: setup-guard -> m41 setup -> m41 dispatch. Then the trusted binary runs the real ashhart#741 path for the first time, the instrument reads live buffers, and the decider fields finally carry meaning.
|
A receipt from a second GB10, on Setup. GB10 sm_121, driver 580.178.04, kernel 7.0.0-1019-nvidia, Zig 0.17.0, CUDA 13.0. Tested this PR at
This adds a passing layer-3 receipt on real 4-bit |
…ght, and fails when none is EXL3
|
Thank you, @BHCC2025, for #740 and #741 together: the EXL3 projection behind As agreed on #806, the module is chosen head to head with #837 on the same tensors, and these two are your side of it. Two notes from our read to carry into that comparison, both about one row computed two ways:
Both are fine under the engine's rule that replies are prefilled again in the background, but the family has to know it, so the module should say which entries share an order. Thank you again, and see you in #832. |
|
@ashhart, thank you, both are right: in #740 a split decode adds finished part totals where the prompt keeps one accumulator, and in #741 decode keeps the rotation in fp32 ( |
|
@BHCC2025, a note on expert widths before the EXL3 module is chosen in #832, from reading #741 (
So a mixed-width pack loads without an error, then decodes most experts at the wrong offset and width. I haven't run it through your branch; this is from the code. turboderp's packs use one width per layer, which is why the receipts so far don't show it. Two ways to handle it:
Would you rather take either into #741 yourself, or have them from me? |
|
Thank you, @BHCC2025, the Grouped integration and explicit prompt/decode-order discussion are useful pieces of the common EXL3 module. We will use #837 as the consolidation base, first repairing its loader/ownership cases, then carry your Grouped integration and #806's FP32 contract into it with both CUDA authors and @vcruz305 credited. Per-expert widths stay in the representation. We will replace the layer-wide stride where needed or refuse unsupported mixed layouts until the descriptors are connected. The same-tensor/equal-bit comparison still chooses the faster correct implementation per operation, with prompt/decode and resumed/fresh gates retained. We can make the integration guards at landing. Posted by Ash's AI Assistant Synthena |
…pert's gate, up and down widths its own experts.Grouped takes an .exl3 layer, as #741 designed it, through exl3_grouped.zig: the plan's triples and pair indices drive #837's per-expert pointer tables, gate, up and down keep their own widths, half widths included, and skipped outputs stay untouched. Up comes out in bf16, as the kernel rounds it, and a call asking for the combined fp32 activation pipeline is refused before any device work; down comes out in fp32 or bf16. A compile-only object step builds the loader, the uploads, every qlinear entry, both dense implementations and the Grouped entries against the real types without running CUDA work. Co-authored-by: BHCC2025 <241330250+BHCC2025@users.noreply.github.com> Co-authored-by: Justin Casey <4348807+justincasey@users.noreply.github.com>
|
Thank you, @BHCC2025, for EXL3 routed experts behind We merged #741 for 1.1.0 in efc980b, retaining your commits; that merge leaves the existing tree unchanged. Your design then lands on #837's per-expert tables in 8531225, co-authored with you and @justincasey. Gate, up and down retain each expert's own width, including half widths, and skipped outputs remain untouched. Up produces bf16, so a call requiring the FP32 activation pipeline is refused before device work. NVIDIA GB10/RTX PRO 6000 operator checks passed; whole-layer and two-rank oracle captures remain unavailable. Posted by Ash's AI Assistant Synthena |
The tree is the one before this merge. ashhart#741's projections are an earlier copy of ashhart#806's, which landed above, and both ashhart#741 and ashhart#837 added the EXL3 expert kernel and its tests, ashhart#837's with each expert's own width and ashhart#741's with one stride a layer; ashhart#837's stay. ashhart#741's design, EXL3 routed experts behind experts.Grouped, is what the next commits build on ashhart#837's tables.
The tree is the one before this merge. ashhart#740's projections are its shared commit with ashhart#741, an earlier copy of ashhart#806's, which landed above, and its test change, skipping a matrix a checkpoint stores as plain .weight, is in ashhart#806's tests as kept.
…each expert's gate, up and down widths its own experts.Grouped takes an .exl3 layer, as ashhart#741 designed it, through exl3_grouped.zig: the plan's triples and pair indices drive ashhart#837's per-expert pointer tables, gate, up and down keep their own widths, half widths included, and skipped outputs stay untouched. Up comes out in bf16, as the kernel rounds it, and a call asking for the combined fp32 activation pipeline is refused before any device work; down comes out in fp32 or bf16. A compile-only object step builds the loader, the uploads, every qlinear entry, both dense implementations and the Grouped entries against the real types without running CUDA work. Co-authored-by: BHCC2025 <241330250+BHCC2025@users.noreply.github.com> Co-authored-by: Justin Casey <4348807+justincasey@users.noreply.github.com>
…lamaV3's notice One line for the CUDA module (ashhart#837, ashhart#806, ashhart#741, ashhart#740) and one for Metal's shared prefill tiles (ashhart#729). The third-party notice names the native scan, the CUDA kernels and their Zig side beside the decoders it already lists, under the same Turboderp attribution.
EXL3 routed experts on the native CUDA engine, behind
experts.Grouped. Stacked on #740 (EXL3 projections behindqlinear); part of #593's step 4. No model uses it yet.What it adds
kernels/cuda/exl3_experts.cu: a layer's EXL3 experts ongrouped.zig's plan.uptakes token rows through gate and up underexperts.cuh's SwiGLU (bf16 out);downtakes pair rows to fp32 or bf16. Each expert's rows are rotated by its ownsuhinline.cuda/exl3_experts.zig: the layer geometry and kernels;experts.Format.exl3, andLayer.exl3for the layer's widths and codebook (a layer's experts share one width; a family keeps its shared experts onqlinear).tf-cuda-test exl3-experts [DIR].Arithmetic
Tests (GB10)
tf-cuda-test exl3-experts /path/to/GLM-5.3-Flash-exl3-4.05bpw: 6 pass, 0 fail, 0 skip. On both paths:experts.cuh's formula over the kernel's own gate and up rows (all 50,432 values on each path exact);tf-cuda-test exl3still passes (12 of 12),zig build test test-cuda-hostpasses,zig fmt --checkis clean, andlean_check.pyreports the base's 539.Timings (GB10, GLM-5.3-Flash layer 3, 4-bit experts)
The expert fatbin is 10.6 MB for the three release targets (sm_86, sm_120, sm_121), 3.5 MB each; the projections fatbin of the first PR is 3.1 MB.
Environment and not run
As in #740. No served model, so no
bench_concurrent,prefill_coldor drafted-versus-plain receipt yet; sm_86 and sm_120 compiled, not run; Metal untouched.This branch carries #740's commit underneath; only the second commit is new here.