Repository navigation
cuda: EXL3 linear and grouped routed experts, every codebook and width, a width per expert (#593) - #837
cuda: EXL3 linear and grouped routed experts, every codebook and width, a width per expert (#593)#837justincasey wants to merge 1 commit into
Conversation
…h, a width per expert Hand-written ports of the Python engine's EXL3 kernels: - the trellis decode into mma.m16n8k16 B fragments; - a row-invariant linear for 1 to 128 rows (mul1 half steps included), with prompts in 128-row blocks and qlinear's .exl3 arm; - grouped routed experts with each expert's own width; - the checkpoint's trellis groups from its safetensors headers, and their upload. tf-cuda-test exl3 checks them against float64 references, row and chunk invariance, and, with fixtures, the Python engine's bytes. tf-cuda-test exl3-tp2 checks the GLM split rules on two ranks over NCCL.
|
Thank you, @justincasey. This is careful work: hand-written EXL3 kernels for the linear and the grouped routed experts at every codebook and width, a width per expert, and a receipt that holds a row's bytes equal across row counts 1 to 128, K splits, member counts and prompt chunking, with real RED-SNOW layers byte for byte against the Python engine's kernels. That is exactly the bar we want. Two things before it lands. Your headers say these port the Python engine's We are gathering the EXL3 work in #832, where Cruz is invited to lead. It would be great to have you there. |
|
Ok sounds good. Let's make sure that the mixed widths don't get lost. Happy to bench or compare anything. I have 4 GB10s I help with if needed. |
|
Thank you, @justincasey, the per-expert descriptors and broader format coverage make #837 the common consolidation base. We checked 224406d. Before composition, we will repair the packed-scale scan/load mismatch, allocate scratch replacements before releasing their owner, and refund upload buffers when list insertion fails. We can make those bounded fixes with failure regressions at landing. Then we will carry #806's FP32 contract, #741's Grouped integration and #729's Metal path, preserving your work, @BHCC2025, @mrv777 and @vcruz305's credit. The equal-bit comparison remains required to choose the faster correct implementation per operation; selecting a coverage base does not supply that timing result. Mixed expert widths and full-token cache/draft gates stay together. Posted by Ash's AI Assistant Synthena |
…rts for every codebook and width The merged commit is #837's with its author, date and message, without its five fixture scripts under zig/tests/cuda/oracle/: they import the Python engine, which does not land here. The tests still read recorded bytes from a folder when one is given. The kernel list, the runtime's exports, the test runner's usage and qlinear's weight views take both sides. #837's kernels port the Python engine's EXL3 module from #42. Co-authored-by: vcruz305 <16494159+vcruz305@users.noreply.github.com>
…s, with fp32 outputs #806's kernels and their Zig side land as exl3.cu and exl3_linear.zig beside #837's EXL3 module, which keeps qlinear's .exl3 weight view. #806's GPU tests, which went through its own qlinear view, are kept as exl3_linear_tests.zig, to run once the tiles sit behind #837's Dense. Both PRs added zig/tests/cuda/exl3_tests.zig; #837's keeps that name.
The tree is the one before this merge. #741's projections are an earlier copy of #806's, which landed above, and both #741 and #837 added the EXL3 expert kernel and its tests, #837's with each expert's own width and #741's with one stride a layer; #837's stay. #741's design, EXL3 routed experts behind experts.Grouped, is what the next commits build on #837's tables.
…r's EXL3 rows can end in fp32 #806's kernels and Zig side move to exl3_candidate.cu and exl3_candidate.zig unchanged. Dense keeps #837's reference kernels by default and takes the tiles when its implementation says so, on the same uploaded weight; a layout the tiles cannot read, with a bias, half widths or other strides, is refused by name. qlinear's decodeF32 and promptF32 take EXL3 weights beside NVFP4, and check loading and row strides before any launch. #806's GPU tests run as tf-cuda-test exl3-tiles, their qlinear rows now through Dense with the tiles chosen. Which implementation a family gets stays a choice until both are compared on the same checkpoint tensors. Co-authored-by: BHCC2025 <241330250+BHCC2025@users.noreply.github.com> Co-authored-by: Justin Casey <4348807+justincasey@users.noreply.github.com>
…pert's gate, up and down widths its own experts.Grouped takes an .exl3 layer, as #741 designed it, through exl3_grouped.zig: the plan's triples and pair indices drive #837's per-expert pointer tables, gate, up and down keep their own widths, half widths included, and skipped outputs stay untouched. Up comes out in bf16, as the kernel rounds it, and a call asking for the combined fp32 activation pipeline is refused before any device work; down comes out in fp32 or bf16. A compile-only object step builds the loader, the uploads, every qlinear entry, both dense implementations and the Grouped entries against the real types without running CUDA work. Co-authored-by: BHCC2025 <241330250+BHCC2025@users.noreply.github.com> Co-authored-by: Justin Casey <4348807+justincasey@users.noreply.github.com>
|
Thank you, @justincasey, for the common CUDA EXL3 operators, with @vcruz305 credited for the original Python port. We carried #837 into 1.1.0 in c360607. Our follow-up admits packed i16 sign planes and preserves ownership when scans, scratch growth or uploads fail. #806's tiles remain explicitly selectable, and 8531225 integrates #741's grouped experts with each expert's own gate, up and down widths. Python oracle-recording helpers remain external; the tests still accept their frozen bytes. Whole-layer and two-rank oracle comparisons require those captures, alongside the retained operator checks. NVIDIA GB10/RTX PRO 6000 operators passed; external whole-layer and two-rank oracle captures remain unrun. Posted by Ash's AI Assistant Synthena |
…ed experts for every codebook and width The merged commit is ashhart#837's with its author, date and message, without its five fixture scripts under zig/tests/cuda/oracle/: they import the Python engine, which does not land here. The tests still read recorded bytes from a folder when one is given. The kernel list, the runtime's exports, the test runner's usage and qlinear's weight views take both sides. ashhart#837's kernels port the Python engine's EXL3 module from #42. Co-authored-by: vcruz305 <16494159+vcruz305@users.noreply.github.com>
…mpt rows, with fp32 outputs ashhart#806's kernels and their Zig side land as exl3.cu and exl3_linear.zig beside ashhart#837's EXL3 module, which keeps qlinear's .exl3 weight view. ashhart#806's GPU tests, which went through its own qlinear view, are kept as exl3_linear_tests.zig, to run once the tiles sit behind ashhart#837's Dense. Both PRs added zig/tests/cuda/exl3_tests.zig; ashhart#837's keeps that name.
The tree is the one before this merge. ashhart#741's projections are an earlier copy of ashhart#806's, which landed above, and both ashhart#741 and ashhart#837 added the EXL3 expert kernel and its tests, ashhart#837's with each expert's own width and ashhart#741's with one stride a layer; ashhart#837's stay. ashhart#741's design, EXL3 routed experts behind experts.Grouped, is what the next commits build on ashhart#837's tables.
…se, and qlinear's EXL3 rows can end in fp32 ashhart#806's kernels and Zig side move to exl3_candidate.cu and exl3_candidate.zig unchanged. Dense keeps ashhart#837's reference kernels by default and takes the tiles when its implementation says so, on the same uploaded weight; a layout the tiles cannot read, with a bias, half widths or other strides, is refused by name. qlinear's decodeF32 and promptF32 take EXL3 weights beside NVFP4, and check loading and row strides before any launch. ashhart#806's GPU tests run as tf-cuda-test exl3-tiles, their qlinear rows now through Dense with the tiles chosen. Which implementation a family gets stays a choice until both are compared on the same checkpoint tensors. Co-authored-by: BHCC2025 <241330250+BHCC2025@users.noreply.github.com> Co-authored-by: Justin Casey <4348807+justincasey@users.noreply.github.com>
…each expert's gate, up and down widths its own experts.Grouped takes an .exl3 layer, as ashhart#741 designed it, through exl3_grouped.zig: the plan's triples and pair indices drive ashhart#837's per-expert pointer tables, gate, up and down keep their own widths, half widths included, and skipped outputs stay untouched. Up comes out in bf16, as the kernel rounds it, and a call asking for the combined fp32 activation pipeline is refused before any device work; down comes out in fp32 or bf16. A compile-only object step builds the loader, the uploads, every qlinear entry, both dense implementations and the Grouped entries against the real types without running CUDA work. Co-authored-by: BHCC2025 <241330250+BHCC2025@users.noreply.github.com> Co-authored-by: Justin Casey <4348807+justincasey@users.noreply.github.com>
…lamaV3's notice One line for the CUDA module (ashhart#837, ashhart#806, ashhart#741, ashhart#740) and one for Metal's shared prefill tiles (ashhart#729). The third-party notice names the native scan, the CUDA kernels and their Zig side beside the decoders it already lists, under the same Turboderp attribution.
What this changes
EXL3 on the native CUDA engine (#593): hand-written kernels for the linear and the grouped routed experts, at every codebook (
mul1,mcg,3inst) and width. Each routed expert keeps its own width.exl3_decode.cuh,exl3_linear.cu,cuda/exl3.zig:mma.m16n8k16B fragments, and the input rotation;mul1half steps included), with the K split planned from the shape alone;.exl3arm inqlinear.exl3_experts.cu,cuda/exl3_experts.zig: grouped experts (member lists, each expert's rotation, gate and up, GLM's SwiGLU roundings, down, the weighted combine).vcruz305/RED-SNOW-5.3-FLASH-EXL3-SAGE-4.91bpwmixes 4- to 8-bit experts in 41 of its 43 MoE layers.core/exl3_load.zig,cuda/exl3_upload.zig: a checkpoint's trellis groups from its safetensors headers, and their upload to the device.tf-cuda-test exl3 [DIR]andexl3-tp2 RANK HOST:PORT [DIR], plus a section indocs/recipes/exl3.md.No family loads these yet. One commit on
main(c126fe16), with no conflicts.Receipt
c126fe16; DGX Spark (GB10, sm_121), CUDA 13.0; checkpointvcruz305/RED-SNOW-5.3-FLASH-EXL3-SAGE-4.91bpw.core/exl3_format.zig: the linear at every width and codebook, and the experts at 4/5/6 bits and mixed widths, each against a worst-element bound.DIR, real RED-SNOW layers byte for byte against the Python engine's EXL3 kernels: five linears, a dense prefill, an expert layer, and whole KDA and DSA layers with all 288 experts under real routing.exl3-tp2: the GLM split rules on two GB10s over NCCL, the same way.tf-cuda-test exl3: 28 pass, 0 fail, 0 skip.zig build test-cuda-host: pass.zig fmt --check zig: clean.tools/zig/lean_check.py: 538, the same asc126fe16.DIR, the Python-engine comparisons SKIP and the float64 and invariance checks still run.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (n/a until a family uses it)