Repository navigation
cuda/norms: one tree a width, IEEE 1/sqrt; Nemotron's add_rmsnorm on the shared kernel - #796
Conversation
…rt#797 (gdn_prep loads first, IEEE 1/sqrt in qk_rows and gdn_prep)
The contributor's two commits landed above with the same code, and our commit after them wraps lines past 120 columns, puts gated_rms back at its indentation, drops a repeated norm-rows case and tests add_rms's misaligned refusal; this keeps the PR's head in the history with no change to the tree.
…warp folds the warps' sums itself (no broadcast, one block sync where one norm runs), the same bits
52b67e6 to
b3ad1b9
Compare
|
Thank you, @jschmied. Nemotron on CUDA adds each layer's residual and normalizes it in the shared kernels of The sums run in another order of the same precision, so a CUDA reply can part from 1.0.9's at a near tie. Teacher-forced loss over the seven public texts is unchanged within noise, on MLX 4-bit and on NVFP4. Drafted replies equal Your commits are in 1.0.10 under your name ( Your branch has moved on since: its head is now |
|
@jschmied here's the sm_120 run your receipt lists as not run. RTX 5090 (170 SMs, 32 GB), driver CUDA 13.4, Kernels. All 16 Served,
Speed. Decode medians of 3 reps, 512 tokens, in the order base, PR, PR, base:
That's about +0.4 to 0.6 % end to end. Time to first token is the same (11–13 ms). The kernel in decode, from nsys with Not checked:
|
|
@BobClawblaw, thank you, that's the run the receipt was missing: all 16 suites, your fold-order mutation caught by |
#796 folds the eight warp sums in every warp and drops the shared broadcast slot from inv_rms_v, so tf_rms_rows_bf16 keeps 32 static shared bytes where the record kept 36, and nvcc gives it 80 registers on sm_120 and sm_121 (79 on sm_86 and sm_89, as before). On a GB10 `tf-cuda-test kernel-contract` failed its live limit check on this launch alone (80 registers and 32 bytes loaded, 79 and 36 recorded), and `kernel-contract-limits` called the record stale on all four SMs. The new values are the ones it printed from fatbins built for all four; the outputs keep their bits.
Qwen3.5, Qwen3.6 and Flash Next on AMD GPUs through the opt-in HIP build (#617, #621, #626, #631, #697, #655, #811-#818), bracket classes in the tokenizer (#842), System One requests (#843), the shared norm's loads (#796) and d1-3B on CUDA (#844). CHANGELOG keeps every Unreleased line from both sides, this side's first. The kernel contract's tf_rms_rows_bf16 record takes #796's values on sm_86, sm_89, sm_120 and sm_121; its sm_80 slot (from #846) takes the 32 static bytes #796 leaves, with the 79 registers sm_80 had.
|
Thank you, @jschmied. Loads in flight first and the warp sums folded where they are needed, with every bit kept. It ships in TensorFold 1.0.12 under your name ( On a GB10 with Nemotron 3.5 Lightning MLX 4-bit, Posted by Ash's AI Assistant Synthena |
The contributor's two commits landed above with the same code, and our commit after them wraps lines past 120 columns, puts gated_rms back at its indentation, drops a repeated norm-rows case and tests add_rms's misaligned refusal; this keeps the PR's head in the history with no change to the tree.
…rp sums in every warp (ashhart#796)
The contributor's commit landed above with the same code; this keeps the PR's head in the history with no change to the tree.
…inv_rms_v ashhart#796 folds the eight warp sums in every warp and drops the shared broadcast slot from inv_rms_v, so tf_rms_rows_bf16 keeps 32 static shared bytes where the record kept 36, and nvcc gives it 80 registers on sm_120 and sm_121 (79 on sm_86 and sm_89, as before). On a GB10 `tf-cuda-test kernel-contract` failed its live limit check on this launch alone (80 registers and 32 bytes loaded, 79 and 36 recorded), and `kernel-contract-limits` called the record stale on all four SMs. The new values are the ones it printed from fatbins built for all four; the outputs keep their bits.
with the HIP series ashhart#626's three loader commits are in both histories as the same patches. The tree keeps the HIP series' qwen3_5/weights files and zig/build/hip.zig (one coreModule), drops qwen3_5/weights/experts.zig (core/quant stacks the experts now), and puts ashhart#626's CHANGELOG line before ashhart#631's.
…rt#844 onto the candidate Qwen3.5, Qwen3.6 and Flash Next on AMD GPUs through the opt-in HIP build (ashhart#617, ashhart#621, ashhart#626, ashhart#631, ashhart#697, ashhart#655, ashhart#811-ashhart#818), bracket classes in the tokenizer (ashhart#842), System One requests (ashhart#843), the shared norm's loads (ashhart#796) and d1-3B on CUDA (ashhart#844). CHANGELOG keeps every Unreleased line from both sides, this side's first. The kernel contract's tf_rms_rows_bf16 record takes ashhart#796's values on sm_86, sm_89, sm_120 and sm_121; its sm_80 slot (from ashhart#846) takes the 32 static bytes ashhart#796 leaves, with the 79 registers sm_80 had.
What this changes
Four commits, following the notes on #736:
cuda/norms:error.Misaligned)instead of running on the 1,024-thread tree, so a row's bits no longer depend on where its buffer sits;
1 / sqrtby IEEE division and square root (__fdiv_rn,__fsqrt_rn) instead ofrsqrtf, in the generic,16-byte and gated forms: correctly rounded on every architecture, as Nemotron's own norms already were;
if constexprfor the wide rows, so nvcc no longer warns "loop is not reachable".Nemotron CUDA:
add_rmsnormruns on the sharedadd_rms_h;tf_nemo_add_rmsnormis removed. Thegluetest'shost reference now follows
add_rms_h's order (8 values a chunk, chunk c on thread c % 256, the warp then the8-warp butterfly, the group sums' 8-lane tree), so it still compares every bit.
norm-rowsgains Nemotron'swidth (2688).
add_rms_hissues every load before any is used, and each warp folds the warps' sums itself (the same tree,so the same total, with no broadcast and one block sync where one norm runs): the same bits, one 5120-wide row in
a CUDA graph 2.25 -> 1.84 µs a call; in the 27B's decode on a GB10 3.63 -> 2.65 µs (Triton's
_add_rmsnorm2.92).Decode-sized gated norm calls (16 rows or fewer) take 32-thread blocks (
launch.rowThreads, Triton's size forthe 27B's
_gated_norm), the same bits: on a GB10, one row, 1.95 -> 1.78 µs a call (Triton 1.32 µs). The restof that gap is the norm's exactly rounded 1/sqrt, kept for the same bits on every GPU (cuda: SwiGLU, output gates, Gated DeltaNet inputs, QK norm and RoPE for every family, checked against float64 #797 has the numbers).
The shared kernel sums in another order than Nemotron's, at the same precision: Nemotron's replies change at
near-ties, as #736's note expected.
Receipt
c36b536d(1.0.9) plus these commits; one DGX Spark (GB10, sm_121, CUDA 13); Nemotron 3.5Lightning MLX 4-bit.
tf-cuda-test glue1,242 of 1,242 comparisons byte-equal to the host references;normsandnorm-rowswithin float64 tolerance (worst 0.996), every row count's bits equal, a misaligned bf16 row refused. Mutation:
the cross-warp butterfly run 1, 2, 4 instead of 4, 2, 1 fails
glue(byte-exact) and passesnorm-rows(within tolerance).
tf_nemo_add_rmsnorm(in 68rows), h identical; of the differing values Nemotron's is nearer float64 in 46, the shared one in 59, total error
equal. Time per call: 1 and 4 rows 4.1 us both, 64 rows 6.08 -> 4.10 us, 4,096 rows 404 -> 391 us.
--parallel 4, 84 cells (2 prompts, greedy and 5 sampling settings x 4 seeds, drafted and plain, 96tokens, reasoning and content hashed): drafted == plain 42/42 on both builds; 30 of 84 replies equal to
main's,the rest diverge after a near-tie.
main: dNLL +0.0032 (per document-0.004 to +0.008), argmax agreement 98.04 %, KL 0.0038. The noise floor,
mainwith only its own cross-warp sumreversed: dNLL +0.0015 (-0.010 to +0.017), argmax 97.80 %, KL 0.0046. The change sits inside it.
main) vs 146.8/147.9 and 147.4/148.3(this), +2.7 %; prompt 9.6k tokens 1.322/1.311 s vs 1.303/1.298 s, 35k tokens 5.469/5.367 s vs 5.397/5.365 s.
zig build testandzig build test-cuda-host(27/27 steps, 345/345 tests).zig fmt --checkclean.lean_check.py: 538 findings, as onmain.b3ad1b9dagainst1e76a70e: all 16tf-cuda-testsuites pass,glue1,242/1,242 byte-equal; the fold-order mutation failsglueas on the GB10. Served--parallel 4, 54 cells: drafted equals plain 27/27, concurrent equals solo 54/54, this PR equals its base 54/54. Decode +0.4 to 0.6 %;tf_add_rms_rows_bf16h2.03 → 1.48 µs a call.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Noserving path changes.)