cuda: EXL3 projections write fp32 too (qlinear decodeF32 and promptF32) by BHCC2025 · Pull Request #806 · ashhart/TensorFold

BHCC2025 · 2026-10-13T20:59:51Z

Based on #757's head 9a14da6a (qlinear decodeF32 / promptF32). The first two commits are #740's two (EXL3 projections behind qlinear) rebased onto it: a35cc231 is the same patch as #740's 76ac2867, and 241ac1bb differs from 7b6dc024 only in the import lists that met #757's. The last commit, ff00ed30, is this PR. For #593: the GLM-5.3-Flash family sums its ranks' partial rows before rounding, so an EXL3 projection has to hand out its fp32 sums as #757 does for NVFP4.

Change. The EXL3 decode kernel already took an fp32 output flag; the prompt kernel gains the same flag (tf_exl3_store), and qlinear.decodeF32 / promptF32 route .exl3 to them instead of EpilogueNotBuilt. decode and prompt keep their bf16 bits: the bf16 out is the fp32 out rounded. No other format changes.

Test. tf-cuda-test exl3 gains a check of both entries through qlinear on a 2048x256 4-bit mcg matrix: every fp32 value within its FP64 bound, the same bits at every row count (decode 1-16 rows, prompt 1-300), and the bf16 out equal to the fp32 out rounded. With only the qlinear routing reverted it fails with EpilogueNotBuilt.

Receipt. GB10 (sm_121), CUDA 13, Zig 0.17.0, -Dsm=121; checkpoint turboderp GLM-5.3-Flash-exl3 4.05 bpw.

tf-cuda-test exl3 <ckpt>: all PASS. New check: 80,896 FP64 values, max 0.0031 of the bound. The 7 real matrices unchanged (max 0.0308 of the bound).
tf-cuda-test quant-ref: 4/4 PASS (cuda: the NVFP4 lane matmul and prompt GEMM write fp32 too (qlinear decodeF32 and promptF32) #757's NVFP4 fp32 checks).
zig build test test-cuda-host, zig fmt --check zig/: clean.

Those GPU runs were on #757's previous head (2f3988f). After rebasing onto its current head (9a14da6, on 1.0.9) the build, host tests and zig fmt --check pass, and the GPU checks were re-run on this head (ff00ed30) with the same results: #806 (comment)

Not run: Metal (no Mac change), server benches (no served family uses EXL3 fp32 out yet; the family PR will).

…ecodeF32 and promptF32) A tensor-parallel family sums its ranks' partial rows before rounding, so it needs a projection's fp32 sums as the kernel holds them. qmmf.cu and prompt16.cu already take F32 as a template argument; this instantiates FP4's fp32 tiles (and reduce_kernel<true> for the split path), loads them beside the bf16 ones, and adds Lane.matmulOut, Prompt.matmulOut and qlinear's decodeF32 and promptF32. Other formats refuse fp32 out with EpilogueNotBuilt, as the experts do. matmul, decode and prompt keep their signatures and bf16 bits.

kernels/cuda/exl3.cu decodes EXL3 trellis tiles as checkpoints store them ([K/16, N/16, 16 bits] int16), with core/exl3_format.zig's codebooks (3inst, mcg, mul1) and whole widths 1-8. Rows are rotated by suh and the 128-point Hadamard transform, scaled by 2^-4 and rounded once to fp16; each tile decodes once straight into the B fragments of two mma.m16n8k16 (the tile's value order is theirs), fp32 sums, then the output transform, svh and 2^-3 (docs/recipes/exl3.md, Prompts). Decode rows go 1-16 a launch, read as A fragments from rows written in fragment order; matrices with fewer than 96 column blocks split their input blocks into a shape-fixed number of parts that the last part sums in order. Prompt rows go 64 a block from shared memory. Every row is independent, so a row's bits are the same at every row count on its path. qlinear takes it as Weight.exl3, refused with FormatNotLoaded when the module is absent. tf-cuda-test exl3 [DIR]: every codebook state and random tiles at every width bit-equal to exl3_format; FP64 references from exl3_format.dequantize with a bound for the kernels' rounding order, worst column and aggregate error; decode windows of 1-16 rows and prompts of 2-300 rows byte-equal to each row alone; DIR adds real matrices of a checkpoint and times them.

…ght, and fails when none is EXL3

A tensor-parallel family sums its ranks' partial rows before rounding, so an EXL3 projection must hand out its fp32 sums as ashhart#757 does for NVFP4. The decode kernel already takes an fp32 output flag; the prompt kernel gains the same flag (tf_exl3_store), and qlinear's decodeF32 and promptF32 route .exl3 to them. decode and prompt keep their bf16 bits: the bf16 out is the fp32 out rounded. The new exl3 test checks both entries within FP64 bounds, the same bits at every row count, and the bf16 out as their rounding; it fails with EpilogueNotBuilt before this change.

BHCC2025 · 2026-10-13T23:46:12Z

GPU checks on this head (ff00ed3, on #757's 9a14da6): GB10 sm_121, driver 580.178.04, CUDA 13, -Dsm=121, turboderp GLM-5.3-Flash-exl3 4.05 bpw.

tf-cuda-test exl3 <ckpt>: all PASS. The fp32 check covers decodeF32 (1–16 rows) and promptF32 (1–300 rows): 80,896 FP64 values within bounds (max 0.0031 of the bound), the same bits at every row count, and the bf16 out equal to the fp32 out rounded. The 7 real matrices: 21,504 FP64 values, max 0.0308 of the bound. Both figures are unchanged from the previous head.
tf-cuda-test quant-ref: PASS, with every product within 0.510 of its float64 tolerance (cuda: the NVFP4 lane matmul and prompt GEMM write fp32 too (qlinear decodeF32 and promptF32) #757's NVFP4 fp32 checks included).

ashhart · 2026-10-14T09:53:45Z

Hello @BHCC2025,

Thank you for EXL3 projections behind qlinear. decode and prompt stay bf16, and that bf16 is the fp32 sum rounded once. decodeF32 and promptF32 hand out the fp32 sum, for a rank that adds before it rounds. A missing module returns error.FormatNotLoaded. The test skips a selected matrix that the checkpoint stores as plain .weight, and fails the real-matrix pass when none of the selected names is EXL3.

Your body reports tf-cuda-test exl3 passing on a GB10 (sm_121): 80,896 fp32 values, the worst 0.0031 of the bound, and the seven real matrices at 0.0308. quant-ref is 4 of 4. Those GPU runs are on #757's previous head 2f3988fe, which is not in 1.0.10. After the rebase onto 9a14da6a (#757's head, not in 1.0.10) you say the build, the host tests and zig fmt --check pass, and the GPU checks will be re-run on this head. Please post that re-run for ff00ed307, your head, which is not in 1.0.10. We have not run it.

Metal was not run, and the server benches were not run. No served family calls Weight.exl3 on this head. #804's GLM family reads ModelOpt NVFP4 only (config.zig sets head_unit when quant_method is exl3, and does not call Weight.exl3). This lands with the first CUDA family that projects EXL3 through qlinear.

The body says the branch is stacked on #740, and that the first two commits belong to #757 and #740. The parent of your commits is #757 (9a14da6a3, that pull request's head, not in 1.0.10). The three commits after it are yours (241ac1bbc, a35cc2314, ff00ed307), and none of them is in 1.0.10. #740's head 7b6dc024b is not in 1.0.10, and it is not an ancestor of ff00ed307.

The head includes #757. The change sits on v1.0.9, and the trial merge onto that base is clean. This pull request is not in 1.0.10.

We will wrap the lines past 120 columns at landing, including exl3_linear.zig:76. The gate is tf-cuda-test exl3 on a CUDA GPU with tensor cores, before a family calls it.

Thank you again.

BHCC2025 · 2026-10-14T12:45:23Z

@ashhart, thank you. The GPU re-run on ff00ed307 is posted above (#806 (comment)): tf-cuda-test exl3 all PASS (fp32 check 80,896 values, max 0.0031 of the bound; the 7 real matrices 21,504 values, max 0.0308) and quant-ref PASS, on a GB10 (sm_121), driver 580.178.04, CUDA 13, -Dsm=121.

On the stacking, the body was loose and I've corrected it. The parent is #757's head 9a14da6a3. 241ac1bbc and a35cc2314 are #740's two commits (7b6dc024b, 76ac28670, #740's head) rebased onto it: a35cc2314 is the same patch as 76ac28670, and 241ac1bbc differs from 7b6dc024b only where the import lists met #757's. ff00ed307 is this pull request's own change. So #740's head is not an ancestor; its change is carried here rebased.

Agreed on the gate. The first family to call Weight.exl3 through qlinear is our GLM EXL3 glue on #804's branch (glm-exl3-wip on our fork, 9a099a7c), which we'll open once #804 lands. Its glm-shared-check and glm-lanes-concurrent pass at TP=2 and TP=4 on the turboderp 4.05 bpw checkpoint (#593).

ashhart · 2026-10-14T14:00:36Z

@BHCC2025, thank you for the GPU re-run on ff00ed307 and for tightening the stacking notes. 80,896 fp32 values within 0.0031 of the bound, with the real matrices unchanged, is a clean receipt.

@justincasey's #837 builds the same EXL3 projection and grouped experts from the Python engine's kernels, so there are now two implementations of one piece. We want one module behind qlinear: we will run yours and #837 on the same tensors, bits against the float64 oracle and bandwidth against ExLlamaV3's LinearEXL3, keep the faster at equal bits, and credit both of you on it. Both follow @vcruz305's format from #42, so his name goes on it too.

If you and Justin want to compare notes first, #832 is the place.

BHCC2025 · 2026-10-14T14:06:21Z

@ashhart, thank you, that plan works for us: one module behind qlinear, chosen on the same tensors against the float64 oracle and LinearEXL3, with both of us credited and @vcruz305 on everything that follows #42.

One note for the family side: the GLM EXL3 glue we've carried is already on #804's branch (glm-exl3-wip on our fork, 9a099a7c, on 53309a72), with the family's glm-shared-check and glm-lanes-concurrent passing at TP=2 and TP=4 on turboderp's 4.05 bpw pack. So it can take whichever module is kept.

@justincasey, glad to compare notes in #832.

ashhart · 2026-10-17T00:19:54Z

Thank you, @BHCC2025, the FP32 qlinear entry points supply the output contract the shared EXL3 integration needs.

We will use #837 as the common base, repair its loader/ownership cases, then carry this contract and #741's Grouped integration onto it. The current output API and each family's rounding points stay explicit, with both CUDA contributors and @vcruz305's #42 work credited.

We will compare the candidate kernels on identical tensors and output bits before choosing implementations. Your GLM glue receipt remains useful evidence for its own branch; final default-math, mixed-width, prompt/cache and lane gates run on the combined module.

Posted by Ash's AI Assistant Synthena

The tree is the one before this merge. #741's projections are an earlier copy of #806's, which landed above, and both #741 and #837 added the EXL3 expert kernel and its tests, #837's with each expert's own width and #741's with one stride a layer; #837's stay. #741's design, EXL3 routed experts behind experts.Grouped, is what the next commits build on #837's tables.

The tree is the one before this merge. #740's projections are its shared commit with #741, an earlier copy of #806's, which landed above, and its test change, skipping a matrix a checkpoint stores as plain .weight, is in #806's tests as kept.

…r's EXL3 rows can end in fp32 #806's kernels and Zig side move to exl3_candidate.cu and exl3_candidate.zig unchanged. Dense keeps #837's reference kernels by default and takes the tiles when its implementation says so, on the same uploaded weight; a layout the tiles cannot read, with a bias, half widths or other strides, is refused by name. qlinear's decodeF32 and promptF32 take EXL3 weights beside NVFP4, and check loading and row strides before any launch. #806's GPU tests run as tf-cuda-test exl3-tiles, their qlinear rows now through Dense with the tiles chosen. Which implementation a family gets stays a choice until both are compared on the same checkpoint tensors. Co-authored-by: BHCC2025 <241330250+BHCC2025@users.noreply.github.com> Co-authored-by: Justin Casey <4348807+justincasey@users.noreply.github.com>

…lamaV3's notice One line for the CUDA module (#837, #806, #741, #740) and one for Metal's shared prefill tiles (#729). The third-party notice names the native scan, the CUDA kernels and their Zig side beside the decoders it already lists, under the same Turboderp attribution.

ashhart · 2026-10-18T10:49:13Z

Thank you, @BHCC2025, for the EXL3 trellis tiles, FP32 outputs and row-count comparisons.

We merged #806 for 1.1.0 in 1bef815. The unchanged kernels live as exl3_candidate, with 4cec0eb integrating them behind #837's Dense as an explicitly selected implementation on the same uploaded weights. Unsupported layouts and biases are refused by name; qlinear's FP32 decode and prompt methods accept EXL3.

The supplied sm_120/sm_121 tile comparisons cover 80,896 FP64 values and row-count bit equality. The GB10 GLM-5.3-Flash comparison covers the two listed matrices actually stored as EXL3. Choosing a family default still needs a matched comparison of both implementations.

NVIDIA GB10/RTX PRO 6000 tile checks passed; the two listed EXL3 matrices were checked, but a matched implementation-speed comparison remains unrecorded.

Posted by Ash's AI Assistant Synthena

… ring What: a prompt on an EXL3 checkpoint with offloaded experts now runs in deep chunks (TF_FLASHNEXT_EXL3_PREFILL_ROWS, default 8192 rows): each layer runs the batched pass's own blocks on sub-blocks of the engine's chunk over device rows kept for the whole deep chunk, then its routed experts run expert-major over all rows: each expert is copied once per deep chunk and computed on every pair that picked it with a new 64-member prompt-tile kernel for any mul1 width (kernels/cuda/flashnext_exl3_prefill.cu, after upstream's exl3_prompt.cuh). During the prefill every decoder layer's pool lines form one ring the copy engine streams the coming layers' hottest experts into (cuda_exl3_prefill_ring.zig); the pools are left empty afterwards. Optional ashhart#806 tiles for the dense prompt projections (TF_FLASHNEXT_EXL3_PREFILL_TILES=1). New files: cuda_exl3_prefill{,_k,_ring}.zig, flashnext_exl3_prefill.cu, tests/cuda/exl3_prefill_tests.zig (tf-cuda-test exl3-prefill). One-line hooks: Engine.forward / deinit (cuda_prompt.zig), cuda_exl3_pass.part (tiles), cuda_lanes_load (reserve before the pools), kernel list, kernels.zig, barrel, main.zig. Why: the batched pass copied about every expert of every layer once per 256-row block (115 rows/s on real text, 13+ minutes for 200k tokens). Tested (par1-cs13, one idle 5060 Ti under gpurun, real text = the repo docs tokenized): - expert machinery bit-identical: with runSlots in the passes (KERNEL=slots), last-row logits 100% bit-equal to the batched pass at 2048 and 4096 tokens (per-layer lines) and at 2048 (ring); golden greedy 24 identical to S9 on all three prompts through the ring prefill (MIN=1) and decode after the pool reset - the new kernel vs runSlots on one layer: max |dy| / peak 4.3e-5 (fp32 sum order), 2.6x at 1024 rows, 3.3 us/row/layer at 8192 rows (runSlots 20 us/row at its 1408-row max) - golden tf: top-1 16/16, same numbers as S9 (path not taken for logits .all) - prefill rows/s (real text, default pool): 2048: 566 (batched 114), 8192: 1292 (tiles 1526), 32768: 1442 (tiles), 200000 tokens in 168 s = 1190 rows/s (tiles, chunk 512, 20 lines a layer) - zig build test-flashnext-exl3-host Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…hunk-size sweeps in exl3-prefill What: a prompt splits into equal deep chunks (a short last chunk paid a whole chunk's expert copies: 3392 rows took as long as 8192); the deep chunks' dense projections use ashhart#806's prompt tiles by default (TF_FLASHNEXT_EXL3_PREFILL_TILES=0: reference); exl3-prefill engine sweeps deep-chunk sizes and measures decode after the prefill; a host test of the pass build. Why: measured +17% at 8192 tokens with tiles (1292 -> 1526 rows/s); agreement over 16 real-text prompt lengths 520..985 vs the batched pass: top-1 16/16 with and without tiles, mean |d| on the batched top-5 0.148 (tiles) / 0.179 (reference), the same size as the batched pass's own distance to the Python reference golden (0.171). Tested: zig build test-flashnext-exl3-host; exl3-prefill agree/engine on par1-cs13 (idle card). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…mpt rows, with fp32 outputs ashhart#806's kernels and their Zig side land as exl3.cu and exl3_linear.zig beside ashhart#837's EXL3 module, which keeps qlinear's .exl3 weight view. ashhart#806's GPU tests, which went through its own qlinear view, are kept as exl3_linear_tests.zig, to run once the tiles sit behind ashhart#837's Dense. Both PRs added zig/tests/cuda/exl3_tests.zig; ashhart#837's keeps that name.

The tree is the one before this merge. ashhart#741's projections are an earlier copy of ashhart#806's, which landed above, and both ashhart#741 and ashhart#837 added the EXL3 expert kernel and its tests, ashhart#837's with each expert's own width and ashhart#741's with one stride a layer; ashhart#837's stay. ashhart#741's design, EXL3 routed experts behind experts.Grouped, is what the next commits build on ashhart#837's tables.

The tree is the one before this merge. ashhart#740's projections are its shared commit with ashhart#741, an earlier copy of ashhart#806's, which landed above, and its test change, skipping a matrix a checkpoint stores as plain .weight, is in ashhart#806's tests as kept.

…se, and qlinear's EXL3 rows can end in fp32 ashhart#806's kernels and Zig side move to exl3_candidate.cu and exl3_candidate.zig unchanged. Dense keeps ashhart#837's reference kernels by default and takes the tiles when its implementation says so, on the same uploaded weight; a layout the tiles cannot read, with a bias, half widths or other strides, is refused by name. qlinear's decodeF32 and promptF32 take EXL3 weights beside NVFP4, and check loading and row strides before any launch. ashhart#806's GPU tests run as tf-cuda-test exl3-tiles, their qlinear rows now through Dense with the tiles chosen. Which implementation a family gets stays a choice until both are compared on the same checkpoint tensors. Co-authored-by: BHCC2025 <241330250+BHCC2025@users.noreply.github.com> Co-authored-by: Justin Casey <4348807+justincasey@users.noreply.github.com>

…lamaV3's notice One line for the CUDA module (ashhart#837, ashhart#806, ashhart#741, ashhart#740) and one for Metal's shared prefill tiles (ashhart#729). The third-party notice names the native scan, the CUDA kernels and their Zig side beside the decoders it already lists, under the same Turboderp attribution.

…ision What: tf-cuda-test exl3-pack DIR [REPS] [cold|experts] runs v1.1.0's exl3.Dense (reference and ashhart#806 tiles) and exl3_experts.Experts on a real EXL3 pack's shapes (Qwen3.8-Flash-Next 3.05bpw): error against an exact float64 product (bf16 and fp32 outputs), warm and cold-L2 cudaEvent timing, and routed experts with some trellis in mapped pinned host memory against a pinned H2D copy. tf-cuda-test nccl-lat <rank> <id file> times fp32 ncclAllReduce at decode/prefill sizes. Why: decide whether our EXL3 kernels or upstream's go forward, and how rank sums behave on two PCIe 5060 Ti without peer access (docs/plans/exl3-on-upstream-1.1.0.md). Tested: zig build -Dnvcc=nvcc -Dsm=120 on par1-cs13 (code=0); exl3-pack, cold, experts and nccl-lat (NCCL 2.32.3, SHM) ran under ~/bin/gpurun, exit 0; zig fmt --check clean.

… ring What: a prompt on an EXL3 checkpoint with offloaded experts now runs in deep chunks (TF_FLASHNEXT_EXL3_PREFILL_ROWS, default 8192 rows): each layer runs the batched pass's own blocks on sub-blocks of the engine's chunk over device rows kept for the whole deep chunk, then its routed experts run expert-major over all rows: each expert is copied once per deep chunk and computed on every pair that picked it with a new 64-member prompt-tile kernel for any mul1 width (kernels/cuda/flashnext_exl3_prefill.cu, after upstream's exl3_prompt.cuh). During the prefill every decoder layer's pool lines form one ring the copy engine streams the coming layers' hottest experts into (cuda_exl3_prefill_ring.zig); the pools are left empty afterwards. Optional ashhart#806 tiles for the dense prompt projections (TF_FLASHNEXT_EXL3_PREFILL_TILES=1). New files: cuda_exl3_prefill{,_k,_ring}.zig, flashnext_exl3_prefill.cu, tests/cuda/exl3_prefill_tests.zig (tf-cuda-test exl3-prefill). One-line hooks: Engine.forward / deinit (cuda_prompt.zig), cuda_exl3_pass.part (tiles), cuda_lanes_load (reserve before the pools), kernel list, kernels.zig, barrel, main.zig. Why: the batched pass copied about every expert of every layer once per 256-row block (115 rows/s on real text, 13+ minutes for 200k tokens). Tested (par1-cs13, one idle 5060 Ti under gpurun, real text = the repo docs tokenized): - expert machinery bit-identical: with runSlots in the passes (KERNEL=slots), last-row logits 100% bit-equal to the batched pass at 2048 and 4096 tokens (per-layer lines) and at 2048 (ring); golden greedy 24 identical to S9 on all three prompts through the ring prefill (MIN=1) and decode after the pool reset - the new kernel vs runSlots on one layer: max |dy| / peak 4.3e-5 (fp32 sum order), 2.6x at 1024 rows, 3.3 us/row/layer at 8192 rows (runSlots 20 us/row at its 1408-row max) - golden tf: top-1 16/16, same numbers as S9 (path not taken for logits .all) - prefill rows/s (real text, default pool): 2048: 566 (batched 114), 8192: 1292 (tiles 1526), 32768: 1442 (tiles), 200000 tokens in 168 s = 1190 rows/s (tiles, chunk 512, 20 lines a layer) - zig build test-flashnext-exl3-host

…hunk-size sweeps in exl3-prefill What: a prompt splits into equal deep chunks (a short last chunk paid a whole chunk's expert copies: 3392 rows took as long as 8192); the deep chunks' dense projections use ashhart#806's prompt tiles by default (TF_FLASHNEXT_EXL3_PREFILL_TILES=0: reference); exl3-prefill engine sweeps deep-chunk sizes and measures decode after the prefill; a host test of the pass build. Why: measured +17% at 8192 tokens with tiles (1292 -> 1526 rows/s); agreement over 16 real-text prompt lengths 520..985 vs the batched pass: top-1 16/16 with and without tiles, mean |d| on the batched top-5 0.148 (tiles) / 0.179 (reference), the same size as the batched pass's own distance to the Python reference golden (0.171). Tested: zig build test-flashnext-exl3-host; exl3-prefill agree/engine on par1-cs13 (idle card).

…ision What: tf-cuda-test exl3-pack DIR [REPS] [cold|experts] runs v1.1.0's exl3.Dense (reference and ashhart#806 tiles) and exl3_experts.Experts on a real EXL3 pack's shapes (Qwen3.8-Flash-Next 3.05bpw): error against an exact float64 product (bf16 and fp32 outputs), warm and cold-L2 cudaEvent timing, and routed experts with some trellis in mapped pinned host memory against a pinned H2D copy. tf-cuda-test nccl-lat <rank> <id file> times fp32 ncclAllReduce at decode/prefill sizes. Why: decide whether our EXL3 kernels or upstream's go forward, and how rank sums behave on two PCIe 5060 Ti without peer access (docs/plans/exl3-on-upstream-1.1.0.md). Tested: zig build -Dnvcc=nvcc -Dsm=120 on par1-cs13 (code=0); exl3-pack, cold, experts and nccl-lat (NCCL 2.32.3, SHM) ran under ~/bin/gpurun, exit 0; zig fmt --check clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0179oxN37AvEBF1nsVXmybqg

…ision What: tf-cuda-test exl3-pack DIR [REPS] [cold|experts] runs v1.1.0's exl3.Dense (reference and ashhart#806 tiles) and exl3_experts.Experts on a real EXL3 pack's shapes (Qwen3.8-Flash-Next 3.05bpw): error against an exact float64 product (bf16 and fp32 outputs), warm and cold-L2 cudaEvent timing, and routed experts with some trellis in mapped pinned host memory against a pinned H2D copy. tf-cuda-test nccl-lat <rank> <id file> times fp32 ncclAllReduce at decode/prefill sizes. Why: decide whether our EXL3 kernels or upstream's go forward, and how rank sums behave on two PCIe 5060 Ti without peer access (docs/plans/exl3-on-upstream-1.1.0.md). Tested: zig build -Dnvcc=nvcc -Dsm=120 on par1-cs13 (code=0); exl3-pack, cold, experts and nccl-lat (NCCL 2.32.3, SHM) ran under ~/bin/gpurun, exit 0; zig fmt --check clean.

… ring What: a prompt on an EXL3 checkpoint with offloaded experts now runs in deep chunks (TF_FLASHNEXT_EXL3_PREFILL_ROWS, default 8192 rows): each layer runs the batched pass's own blocks on sub-blocks of the engine's chunk over device rows kept for the whole deep chunk, then its routed experts run expert-major over all rows: each expert is copied once per deep chunk and computed on every pair that picked it with a new 64-member prompt-tile kernel for any mul1 width (kernels/cuda/flashnext_exl3_prefill.cu, after upstream's exl3_prompt.cuh). During the prefill every decoder layer's pool lines form one ring the copy engine streams the coming layers' hottest experts into (cuda_exl3_prefill_ring.zig); the pools are left empty afterwards. Optional ashhart#806 tiles for the dense prompt projections (TF_FLASHNEXT_EXL3_PREFILL_TILES=1). New files: cuda_exl3_prefill{,_k,_ring}.zig, flashnext_exl3_prefill.cu, tests/cuda/exl3_prefill_tests.zig (tf-cuda-test exl3-prefill). One-line hooks: Engine.forward / deinit (cuda_prompt.zig), cuda_exl3_pass.part (tiles), cuda_lanes_load (reserve before the pools), kernel list, kernels.zig, barrel, main.zig. Why: the batched pass copied about every expert of every layer once per 256-row block (115 rows/s on real text, 13+ minutes for 200k tokens). Tested (par1-cs13, one idle 5060 Ti under gpurun, real text = the repo docs tokenized): - expert machinery bit-identical: with runSlots in the passes (KERNEL=slots), last-row logits 100% bit-equal to the batched pass at 2048 and 4096 tokens (per-layer lines) and at 2048 (ring); golden greedy 24 identical to S9 on all three prompts through the ring prefill (MIN=1) and decode after the pool reset - the new kernel vs runSlots on one layer: max |dy| / peak 4.3e-5 (fp32 sum order), 2.6x at 1024 rows, 3.3 us/row/layer at 8192 rows (runSlots 20 us/row at its 1408-row max) - golden tf: top-1 16/16, same numbers as S9 (path not taken for logits .all) - prefill rows/s (real text, default pool): 2048: 566 (batched 114), 8192: 1292 (tiles 1526), 32768: 1442 (tiles), 200000 tokens in 168 s = 1190 rows/s (tiles, chunk 512, 20 lines a layer) - zig build test-flashnext-exl3-host

…hunk-size sweeps in exl3-prefill What: a prompt splits into equal deep chunks (a short last chunk paid a whole chunk's expert copies: 3392 rows took as long as 8192); the deep chunks' dense projections use ashhart#806's prompt tiles by default (TF_FLASHNEXT_EXL3_PREFILL_TILES=0: reference); exl3-prefill engine sweeps deep-chunk sizes and measures decode after the prefill; a host test of the pass build. Why: measured +17% at 8192 tokens with tiles (1292 -> 1526 rows/s); agreement over 16 real-text prompt lengths 520..985 vs the batched pass: top-1 16/16 with and without tiles, mean |d| on the batched top-5 0.148 (tiles) / 0.179 (reference), the same size as the batched pass's own distance to the Python reference golden (0.171). Tested: zig build test-flashnext-exl3-host; exl3-prefill agree/engine on par1-cs13 (idle card).

Bizuayeu and others added 4 commits October 14, 2026 03:45

tests: exl3 skips a selected matrix a checkpoint stores as plain .wei…

a35cc23

…ght, and fails when none is EXL3

BHCC2025 mentioned this pull request Oct 13, 2026

GLM-5.3-Flash and full GLM-5.3 on the native CUDA engine (tracker) #593

Closed

Bizuayeu mentioned this pull request Oct 13, 2026

glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804

Merged

8 tasks done

ashhart mentioned this pull request Oct 14, 2026

EXL3 checkpoints on the native engine (tracker) #832

Closed

ashhart mentioned this pull request Oct 14, 2026

cuda: EXL3 routed experts behind experts.Grouped (#593, stacked on #740) #741

Merged

MiaAI-Lab mentioned this pull request Oct 14, 2026

Proposal: EAQ packs, an EXL3 checkpoint with a manifest, and a check the native engine can run before it loads one #857

Closed

This was referenced Oct 17, 2026

EXL3: share decoded weights across prefill rows #729

Merged

Landing next #286

Open

ashhart merged commit 1bef815 into ashhart:main Oct 18, 2026
4 checks passed

ashhart mentioned this pull request Oct 18, 2026

cuda: EXL3 projections behind qlinear (#593) #740

Merged

Navigation Menu

Repository navigation

cuda: EXL3 projections write fp32 too (qlinear decodeF32 and promptF32) - #806

cuda: EXL3 projections write fp32 too (qlinear decodeF32 and promptF32)#806
ashhart merged 4 commits into
ashhart:mainfrom
BHCC2025:exl3-cuda-f32-r2

BHCC2025 commented Oct 13, 2026 •

edited

Loading

Uh oh!

BHCC2025 commented Oct 13, 2026

Uh oh!

ashhart commented Oct 14, 2026

Uh oh!

BHCC2025 commented Oct 14, 2026

Uh oh!

ashhart commented Oct 14, 2026

Uh oh!

BHCC2025 commented Oct 14, 2026

Uh oh!

ashhart commented Oct 17, 2026

Uh oh!

Uh oh!

ashhart commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

BHCC2025 commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BHCC2025 commented Oct 13, 2026

Uh oh!

ashhart commented Oct 14, 2026

Uh oh!

BHCC2025 commented Oct 14, 2026

Uh oh!

ashhart commented Oct 14, 2026

Uh oh!

BHCC2025 commented Oct 14, 2026

Uh oh!

ashhart commented Oct 17, 2026

Uh oh!

Uh oh!

ashhart commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BHCC2025 commented Oct 13, 2026 •

edited

Loading