[jschmied]

What this changes

Stacked on #746 (the window test checks against its core/attention_ref.zig, and the prompt kernel takes its
attn_odd.cuh). Operator-only, no model calls them yet:

• window_attention.cu/.zig: sliding-window attention, decode rows over a ring of keys (split by key tile, merged
  in tile order, in window_split.cuh; or on tensor cores) and prompt rows, grouped KV heads, window 0 = full.
• moe_glue.cu/.zig: top-k by logit + bias (ties: lower id), sigmoid weights, optionally normalised and scaled,
  a shared-expert slot; a warp form up to 512 experts, a block form beyond; the weighted combine of the slots.
• dense_rows.cu/.zig: bf16 rows times bf16 weight rows into fp32 (routers, output heads), a row's bits at any row
  count. core/moe_ref.zig, core/dense_ref.zig: the float64 references.

NaN and infinities in window_attention.cu (found by the kernel contract's gate 1, fixed here):

• tf_window_decode_tc: the flag that lets a tile move the running max was taken from the first of the two heads a
  thread holds and applied to both, so a head whose scores were all NaN stopped another head's max. Each head now
  takes its own max with no flag: a tile with no key for a head leaves its max as it was, and exp2(+0) is exactly 1
  (ex2.approx.f32 gives +1.0 for +0 and -0 on every target, PTX ISA 9.7.3.21; checked on sm_121), so the kernel
  keeps the old arithmetic and the old speed.
• tf_window_part: tile slots outside the window held a copy of the window's first key, with weight 0, so an
  infinite value at that key gave 0 x inf = NaN in the partial. Those slots now hold zeros.
• Prompt rows (wattn_kernel, both forms): keys past a row's position (and before its window) sit in the same tile
  and mma as rows that see them, with P = 0, so a later row's infinite value made earlier rows NaN. As in cuda: tree attention and DFlash block attention for every family, checked against float64 #746's
  block attention: a block whose output holds a NaN or an infinity runs again (out of line), with every NaN and
  infinite value of V set to 0 before the mma, and each row adding back the ones it sees at the key's float64 weight
  (cuda: tree attention and DFlash block attention for every family, checked against float64 #746's attn_odd.cuh: an infinite value whose float64 weight is 0 adds 0 x inf = NaN, as float64 does; only
  where its weights sum is positive).
• The decode kernels: a visible infinite value whose weight underflows in float32 (more than ~103 below the max)
  but not in float64 (745) went into a sum as 0 x inf, or met a fold factor of 0: NaN, where float64 gives inf. Now
  the same replay, out of line. tf_window_decode_tc: a NaN output (the only class a 0 x inf can get wrong: an
  infinite output holds a weight above float32's least, so above float64's) runs the block, or the one warp that saw
  keys, again with V's NaN and infinite values out of the mma, each head adding them back after its last tile at the
  key's float64 weight, and the warps' fold keeping them. tf_window_part: a tile with a NaN or infinite output
  recomputes those outputs from the finite values and adds each NaN or infinite value at its key's float64 weight
  against the row's float64 max over its window; tf_window_merge folds a NaN result again with those infinities
  unscaled.

Users: families with sliding-window layers or sigmoid-routed MoE on CUDA (Kolibri 1's shapes in the tests; GLM and
DeepSeek-style routing are the same top-k form).

Receipt

• Environment: TensorFold 46a08c18 (1.0.12) plus cuda: tree attention and DFlash block attention for every family, checked against float64 #746 and these commits; one DGX Spark (GB10, sm_121, CUDA 13). No
  checkpoint.
• tf-cuda-test window: 48/4x128 window 513 over rings of 640 and 600 (a ring end inside a tile), 8/2x64 window 100
  with 3 streams, 16/1x128 no window; prompt rows from position 0 and across the ring's wrap: worst 0.465, bits at
  any row count and chunk.
• tf-cuda-test moe-glue: 384 experts top 6 + shared, 256 top 8 normalised x2.5, 512 top 32 (warp form's widest),
  513 top 6 (block form's narrowest), 4096 top 64; picks exact, weights and sums worst 0.996.
• tf-cuda-test dense-rows: [384, 2560], [4103, 2560], [257, 7168]: worst 0.007, bits at every row count.
• Kernel contract (docs/kernel-contract.md), CUDA on GB10 (sm_121), zig build kernel-contract -Dnvcc=/usr/local/cuda/bin/nvcc, exit 0 (cuda: tree attention and DFlash block attention for every family, checked against float64 #746's 21 kernels plus this PR's 16):

  SUMMARY 37 kernels: 37 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems; gates 95658 ms, limits 2 ms
  

  • Before the fixes here and in cuda: tree attention and DFlash block attention for every family, checked against float64 #746: SUMMARY 37 kernels: 27 pass gates 1-4, 10 fail, all at gate 1.
  • Registered (zig/tests/cuda/contract_window.zig, contract_moe.zig): window attention in five calls (tensor-core
    decode, 48/4 heads, window 513, ring 640, two streams; tensor-core decode, 16/1 heads, no window, ring 300;
    split decode tf_window_part + tf_window_merge, 8/2 heads of 64, window 100, ring 128, three streams; prompt
    rows wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2,624; wattn_kernel<128, 8, 8, 8>, 16/2 heads,
    no window); sigmoid top-k in three (warp form, Kolibri's 384 experts top 6 + shared; warp form, 256 top 8
    normalized x2.5; block form, 513 to 4,096 experts); the combine in four (fp32, bf16 and fp16 rows, bf16, fp32
    and fp16 out, the generic loop included); dense rows in each of the four row tiles (1, 4, 16, 32).
  • Served shapes: decode 16 and 64 rows at positions up to 3,000 and 1,000 (300 for the ring of 300); prompt
    chunks of 128 rows from position 0 and 96 rows across the ring's wrap at 2,600, 64 and 48 rows for the unwindowed
    form; top-k 16 and 300 rows (33 rows at 513 experts, 16 at 4,096 top 64); the combine 16 and 300 rows of 2,560 with
    7 slots and 9 rows of 7,168 with 9; dense rows over Kolibri's router (384 x 2,560), an output head's 4,103 columns
    and 257 x 7,168 at each tile's row counts. Tile edges: 32- and 64-key tiles and 16- and 32-row prompt blocks; for top-k
    the 8-row block and 32-expert lane (warp form) and 256 experts (block form); 256 columns and 8 slots for the
    combine; the row tile, 8 outputs and a 256-deep k for dense rows.
  • NaN rules: attention, the combine and dense rows as float64 (propagate); gate 1 plants a NaN query and key,
    +inf and -inf values at a row's own key, a later row's key and the window's first key, and both in one column;
    for prompt rows also +inf at a row's own key at a float64 weight of 0 (48/4 form), and +inf over a row's keys at
    weights below float32's least but not float64's (both forms; in the unwindowed form's gate shape some of those
    keys sit in tiles every row of the block sees); for each decode call, a row's key p all 8 and its window's other
    keys all -8 with +inf, one head querying all 2 (inf, below float32's least) and the next all 8 (NaN, float64
    weight 0), the keys spread over several warps' tiles (48/4) or tiles (split decode). Top-k
    states its rule: picks by fp32 logit + bias, larger first, ties to the lower id, a NaN never picked; weights
    sigmoid(logit).
  • Bounds: the attention entries check gates 1 and 3 against core/attention_ref.zig's float64 bound through their
    own check hook (the framework cannot express P rounded to bf16 before P x V; the split decode's fp32 weights reach
    it through exp the same way). The combine and dense rows run the framework's fp32 bound (worst 0.997 and 0.228 of
    it). Gate 4: exact sums at the served depths, where bf16 sums would miss by 256 to 1.0 million times (attention),
    2,177 to 102 million (combine), 504 to 801 million (dense rows); top-k sums nothing (N/A).
  • Gate 5: all 53 launches are in registry.zig, with compile records from zig build fatbins kernel-contract-fatbins -Dsm=80,86,89,120,121 and tf-cuda-test kernel-contract-limits zig-out/fatbin (PASS kernel-contract-limits: every registered launch fits sm_80, sm_86, sm_89, sm_120 and sm_121, every control red). The fixes moved register counts, re-recorded: both wattn_kernel forms 203 to 223 (sm_80/86/89) and 218
    to 222 (sm_120/121), tf_window_part 52 to 51 (sm_80), plus cuda: tree attention and DFlash block attention for every family, checked against float64 #746's (tf_block_attention now 230 and 228); the
    float64-weight replay leaves every window record as it was; tf_window_decode_tc keeps its records
    (164 and 168). The decode replay re-records tf_window_decode_tc 164 to 176 (sm_80/86/89) and 168 to 178
    (sm_120/121), tf_window_part 51/44/44/48/48 to 55/50/50/56/56 with 128 more static shared bytes (45,824). On the
    GB10 the live check reads PASS limits: 53 registered CUDA launches against sm_80/86/89/120/121, 0 problems.
  • Mutations: each fix reverted alone turns gate 1 red: the shared flag (both tensor-core decode calls, "head 8
    element 0: nan where float64 gives finite"), the window's first key in the outside slots (the split decode, "nan
    where float64 gives pos_inf"), the prompt kernel without its second pass (both forms, "nan where float64 gives
    finite"). A stale register record for wattn_kernel<128, 8, 4, 8> (218 where the compile gives 222) turns gate 5
    red. All restored before the run above. The prompt replay as reviewed (7d599181): "head 252 element 17: pos_inf
    where float64 gives nan" (48/4 form, the weight-0 key) and "head 112 element 13: nan where float64 gives pos_inf"
    (unwindowed form: an infinite value in a tile every row sees, at a weight below float32's least, through the mma),
    besides cuda: tree attention and DFlash block attention for every family, checked against float64 #746's two block forms. On 462d952c the decode plants are red: "head 360 element 21: nan where float64
    gives pos_inf" (48/4), "head 64 element 7" (16/1) and "head 56 element 13" (split decode), the same class, besides
    cuda: tree attention and DFlash block attention for every family, checked against float64 #746's new tree and block plants. The float64 weight test forced true: the three decode calls, "head 361 element
    21", "head 65 element 7" and "head 57 element 13: pos_inf where float64 gives nan".
  • Not run: Metal, which has no kernels in this PR. sm_80, sm_86, sm_89 and sm_120 are covered by compile records
    only, not run on a GPU.
• The fixes keep every finite output's bytes. The fork heads before them (df0ef6bc) and after, the same inputs:
  the five window calls over the registration's served shapes and tile edges with 12 seeds in two finite fills, plus
  sweeps with 2 seeds: decode rows 1 to 64 at position bounds 1 to 3,000 (both sides of 32, 64, 100, 128, 300, 513,
  640), prompt chunks of 1 to 128 rows from positions 0 to 2,400 (both sides of 32, 64, 450, 512, 513). 2,260 runs,
  694,272,000 output bytes, 0 differ (with cuda: tree attention and DFlash block attention for every family, checked against float64 #746's kernels on this branch: 5,808 runs, 1,545,863,168 bytes, 0 differ).
  • The tensor-core decode again after dropping its flag (7d599181 against df0ef6bc): both decode calls over
    their served shapes and edges with 12 seeds in both fills, and rows 1 to 64 at position bounds 1 to 3,000 (both
    sides of 32, 64, 128, 300, 513, 640 and 2,048) with 2 seeds: 1,038 runs, 169,549,824 bytes, 0 differ; all five
    window calls: 2,958 runs, 751,878,144 bytes, 0 differ.
  • The decode's max: where the old shared flag was set, the new code does the same operations. Where it was not
    (no key for the row's heads in the tile), the max is unchanged, so its factor is exp2(+0) = 1 exactly; a head
    with no max yet gets 0 instead of 1, and its sum and values are +0 or NaN, which both factors leave as they are.
    With finite inputs every tile a warp takes has a key. The split decode's outside slots had weight 0, so each added a signed
    zero to a sum that starts at +0 and is never -0; zeros add the same. The prompt kernel's first pass is the old
    kernel; a finite output skips the second.
  • tf-cuda-test window gives the same numbers as before (worst 0.465).
  • The float64-weight replay against the reviewed head (462d952c against 7d599181): cuda: tree attention and DFlash block attention for every family, checked against float64 #746's two block forms (as
    in cuda: tree attention and DFlash block attention for every family, checked against float64 #746, 2,080 runs) and the five window calls over their served shapes with 12 seeds in both finite fills, plus
    with 2 seeds decode rows 1, 2, 3, 16 and 64 at position bounds on both sides of 32, 64, 100, 128, 300, 513, 640
    and 1,280 up to 3,000, and prompt chunks of 1, 15 to 17, 31 to 33, 64, 96 and 128 rows from positions 0, 1, 63 to
    65, 511 to 514, 1,000, 2,600, 2,623 to 2,625 (the ring's wrap) and 5,000: 5,016 runs, 1,179,664,384 output bytes, 0
    differ. A one-ulp change to the window kernels' log2(e) differs in 159,376 bytes.
  • The final decode fold (7d599181) on its own: the tensor-core decode on this head against 8f933773 (the
    per-head flag) over the same decode sweep: 992 runs, 174,522,368 bytes, 0 differ. Its machine code on sm_121 is
    the same on 7d599181 and this head (cuobjdump -sass, tf_window_decode_tc, tf_window_part,
    tf_window_merge). NaN heads: for every query head of each of the five window calls, a NaN in that head's query
    (one element, or all of it) in every row, at the served shapes and decode rows 3 and 17 at bounds 33, 300 and 641
    or prompt chunks of 17 and 33 rows from 0, 513 and 2,600: that head's outputs are all NaN and every other output
    equals the run without the NaN, 2,080 runs, 317,394,944 neighbouring outputs, 0 differ. The shared flag put back
    into the decode fold: 688,128 outputs differ.
  • The decode replay (8cba69e1; 74b749fa moves the split decode to window_split.cuh and joins comments, the same
    SASS on all five targets) against 462d952c: cuda: tree attention and DFlash block attention for every family, checked against float64 #746's eight forms (tree and block, as in cuda: tree attention and DFlash block attention for every family, checked against float64 #746) and the five window
    calls over their served shapes with 4 seeds, plus decode rows 1, 2, 5, 16, 17 and 64 at position bounds 1, 2, 31
    to 33, 63 to 65, 99 to 101, 127 to 129, 299, 300, 512 to 514, 639 to 641, 1,000, 1,300 and 3,000, and prompt chunks
    of 1 to 3, 15 to 17, 31 to 33, 63 to 65 and 128 rows from positions 0, 1, 99 to 101, 511 to 514, 600, 2,600,
    2,623, 2,624 and 2,700, both finite fills: 3,224 runs, 795,961,344 output bytes, 0 differ. A one-ulp change to the
    tensor-core kernels' log2(e) differs in 69,839 bytes. NaN heads as above on this head: 2,080 runs, 325,156,864
    outputs, 0 bad; a replay that makes every head NaN: 22,364,160 bad. Speed (best of 7 x 100 launches with sync,
    5 alternating runs): tensor-core decode 20.4 to 18.9, 47.2 to 44.0 and 11.0 to 10.5 µs, split
    decode 17.1 to 17.0 and 34.0 to 34.3 µs, prompt rows -0.5 to +0.6 %.
  • Speed of the replay change, GB10, best of 5 x 100 launches with sync: prompt rows 17.1 to 17.3, 47.9 to 47.9,
    18.7 to 18.8 and 12.4 to 12.1 µs at the served shapes; the decode kernels' code is unchanged.
  • Speed of the fixes, GB10, 200 launches: prompt rows 45.0 to 45.1 µs (96 rows across the wrap), 14.3 to 14.3-15.3
    µs (128 rows), 574 to 586 µs (a 2,048-row chunk, +2 %); tensor-core decode 12.3 to 12.3 µs (1 and 4 rows), 22.5 to
    22.5 µs (16 rows), 258.5 to 259.0 and 257.9 to 258.4 µs (64 rows, two runs); split decode unchanged.
• Mutations: window bound one key wider (74x split decode, 22x tensor-core decode, 6.4x prompt rows), sigmoid
  exponent x1.0625 (5,527x block form, 6,798x warp form), tie order (wrong pick), the combine dropping a slot, a dense
  row's first product dropped, the dense warp sum one step short.
• sm_120: BobClawblaw ran window, dense-rows, moe-glue on an RTX 5090, all passing with mutations (Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608).
• Speed in this PR: none claimed, nothing calls it.
• Tests run: zig build test from an empty cache on 74b749fa (86/86 steps, 1,380 of 1,387 tests, 7 skipped) and zig build test-cuda-host (41/41 steps), all passing, with the fixes and the registration; tf-cuda-test tree (0.470),
  block (0.380), window (0.465), moe-glue (0.996) and dense-rows (0.007) pass on this branch. zig fmt --check clean. lean_check.py: 517 findings, as
  on main.
• Not run, and why: Metal (unchanged); no model on main uses it.

Until #746 lands, this branch carries its commits (its eight new ones cherry-picked, the same changes); this
PR's own are df0ef6bc, ce160986, 63d15071, 40de67dd, 8f933773, 7d599181, 462d952c, 8cba69e1 and
74b749fa.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (No
  serving path changes.)
• No precision traded for speed. If the bits change, the description says which sums change and why.
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's
  SUMMARY line is in the receipt. (CUDA; no Metal kernels here.)
• Prompt processing is no slower than the last release.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[jschmied]

Rebased onto main 46a08c18 (1.0.12), now at df0ef6bc. The conflicts were only in the build and test registration lists, and the diff against main is otherwise unchanged. It still sits on #746, now on its rebased head 591065e0. The GPU checks give the same numbers as before, and the description has the new test and lint counts.

[jschmied]

The kernels are now registered with the kernel contract in 8f933773. Gate 1 found three NaN bugs in the window kernels, and all three are fixed (ce160986, 63d15071, 40de67dd). #746's fixes are cherry-picked in as well. On finite inputs the outputs are byte-identical to df0ef6bc. The SUMMARY line and the checks are in the description.

[ashhart]

Thanks @jschmied for the three window repairs and the final decode-fold update. This still needs changes before landing.

It inherits #746's visible zero-weight infinity replay mismatch and reference-constructor allocation leak. The prompt replay uses the same helper, so both failing host controls apply. Please repair those once in the shared code, preserving the split-slot and per-head regressions.

The final 7d59918 fold needs its own finite-bit and NaN-head checks; earlier receipts cannot simply qualify it. Keep all 27 launch forms pending, counting #746's inherited fifteen once. Then rerun window boundaries, ring wrap, partial tiles, neighboring heads and numerical/resource gates on the combined head.

Posted by Ash's AI Assistant Synthena

[jschmied]

Repaired once, in the shared code: #746's f4de6c3b and ff6e5c0e are cherry-picked here, and the prompt replay uses the helper in 462d952c. The new head is 462d952c.

• The prompt replay now adds every NaN and infinite value back at the key's float64 weight, so weight 0 gives NaN. Like the block kernel, it now cleans every tile. Before, an infinite value in a tile that every row sees went through the mma at a below-float32 weight.

• Controls (gate 1 plants in contract_window.zig), red on 7d599181:

  • 48/4 form, a weight-0 key at a row's own position: "head 252 element 17: pos_inf where float64 gives nan".
  • Unwindowed form, below-float32 weights in a tile every row sees: "head 112 element 13: nan where float64 gives pos_inf".
  • cuda: tree attention and DFlash block attention for every family, checked against float64 #746's block plants are red there too, and the leak control is the same host test.

  All of them pass on 462d952c. The split-slot and per-head regressions (gate 1's window-first-key and NaN-query plants) still pass.

• Checks on the 7d599181 fold itself:

  • Finite bits: the tensor-core decode on 462d952c against 8f933773 (the per-head flag): 992 runs, 174,522,368 bytes, 0 differ. Its sm_121 SASS is the same as on 7d599181.
  • NaN heads: each query head of all five window calls is made NaN in turn (one element, or the whole query). That head's outputs are all NaN, and every other output equals the run without the NaN: 2,080 runs, 317,394,944 outputs, 0 differ. With the shared flag put back, 688,128 outputs differ.

• Reruns on 462d952c, GB10:

  • Kernel contract: SUMMARY 37 kernels: 37 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems.
  • Limits: kernel-contract-limits passes for all five targets, and the window records are unchanged.
  • Finite-output identity against 7d599181 (both block forms, all five window calls): 5,016 runs, 1,179,664,384 bytes, 0 differ. The runs cover window boundaries (both sides of 513 and 100), ring wrap (300, 640, 128, and prompt chunks across 2,624), partial tiles (1 to 3, 15 to 17 and 31 to 33 rows, bounds on both sides of 32 and 64) and neighbouring heads.
  • Tests: tf-cuda-test window (0.465), block, tree, moe-glue and dense-rows pass, and zig build test from an empty cache passes 86/86 steps.
  • Speed: prompt rows change by -2.4 to +1.2 % at the served shapes.

• All 27 launch forms stay pending, with cuda: tree attention and DFlash block attention for every family, checked against float64 #746's fifteen counted once.

Not changed: the decode kernels have no replay. A visible infinite value whose weight underflows in float32 but not in float64 still gives NaN there, where float64 gives inf (the same as #746's tree attention).

[jschmied]

The open case is closed: the decode kernels now have the replay, in 8cba69e1. #746's new commits are cherry-picked, and the new head is 74b749fa.

• tf_window_decode_tc: a NaN output runs the block (or its one warp with keys) again, with V's NaN and infinite values out of the mma, added back at each key's float64 weight. An infinite output there already holds a weight above float32's least, so it needs no replay.
• tf_window_part: a tile with a NaN or infinite output recomputes those outputs, adding each odd value at its key's float64 weight against the row's float64 max. tf_window_merge folds a NaN result again with those infinities unscaled.
• 74b749fa moves the split decode to window_split.cuh, since the file was past 600 lines. The SASS is the same as 8cba69e1's on all five targets.
• Controls (gate 1 plants, one per decode call), red on 462d952c: "head 360 element 21" (48/4), "head 64 element 7" (16/1) and "head 56 element 13" (split decode), each "nan where float64 gives pos_inf". All of them pass now. With the weight test forced to true, the weight-0 heads give "pos_inf where float64 gives nan".
• Reruns on the new head, GB10 (the identity, NaN-head and speed runs were on 8cba69e1, which has the same SASS):
  • Kernel contract: SUMMARY 37 kernels: 37 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems.
  • Limits: kernel-contract-limits passes for all five targets. tf_window_decode_tc and tf_window_part are re-recorded.
  • Finite-output identity against 462d952c (cuda: tree attention and DFlash block attention for every family, checked against float64 #746's eight forms, all five window calls, window boundaries, ring wrap, partial tiles): 3,224 runs, 795,961,344 bytes, 0 differ. A one-ulp change to log2(e) changes 69,839 bytes.
  • NaN heads: 2,080 runs, 325,156,864 outputs, 0 bad. A replay that makes every head NaN gives 22,364,160 bad.
  • Tests: tf-cuda-test window, block, tree, moe-glue and dense-rows pass, and zig build test from an empty cache passes 86/86 steps.
  • Speed: the tensor-core decode is 4 to 7 % faster (20.4 to 18.9 µs at 16 rows), the split decode changes by -0.6 to +0.8 %, and the prompt rows by -0.5 to +0.6 %.
• All 27 launch forms stay pending.

[ashhart]

Thank you, @jschmied, for sliding-window attention, sigmoid top-k routing, combine and dense CUDA rows.

We merged the reviewed head df0ef6bca for 1.1.0 in a73be1a. The later window/NaN repairs and final decode fold are separate work for 1.1.1, the next patch release.

They share #746's zero-weight-infinity replay mismatch and reference-constructor leak. Please repair that common code once, then cover the final 7d599181 fold with finite-output identity and neighbouring-head controls. The faulty new helper and constructor are absent from the 1.1.0 candidate.

NVIDIA GB10/RTX PRO 6000 release contracts passed 55/55 with five controls red; these operator checks do not establish family adoption.

Posted by Ash's AI Assistant Synthena

[jschmied]

This is already done on this branch for the 1.1.1 increment. The head is 74b749fa.

• The shared helper and constructor repairs from cuda: tree attention and DFlash block attention for every family, checked against float64 #746 are carried here once.
• The window prompt and decode kernels have their own replay fixes in 462d952c and 8cba69e1.
• The final 7d599181 fold is covered by its own finite-output identity, NaN-head and neighbouring-head controls.

Results are in #798 (comment) and #798 (comment).

[ashhart]

Thanks @jschmied for extending the replay across prompt and decode paths and adding the isolation controls. The old constructor/zero-weight findings are repaired in source; the final split-file move preserves the function bodies. This is a 1.1.2 candidate: carry #746's shared changes once, then the remaining window/test changes. Please supply the raw exact-head contract, mutation, finite-byte, resource and timing outputs for the merged build. Kolibri now consumes the operators, so its drafted/plain, resumed/fresh and concurrent/solo checks and prompt/decode comparisons are required too. The shared registry/build merge also needs review before landing.

Posted by Ash's AI Assistant Synthena

[jschmied]

A correction first, about the tree attention kernels this PR carries from #746. On #746 I wrote that the tree launches have "no local memory on any target". I read that from cuobjdump's LOCAL field, which is 0. ptxas -v shows small register spills, into the stack frame, in eight tree attention entry points on some targets. Block, window, moe and dense have no spills. The contract's resource gate (gate 5) records registers, shared memory and bounds, not spills, so it passes either way.

Spills from ptxas -v (nvcc 13.0, -O3, the build's flags), as registers and bytes stored/loaded. Main's register count is in parentheses, and main spills nowhere here except tf_tree_shared_w1/w1s3 (4/4 B on sm_120/121). The SASS is identical at both PR heads and in the merge.

kernel	sm_80	sm_86	sm_89	sm_120	sm_121
tf_tree_merge	114 regs (97), none	114 regs (97), none	114 regs (97), none	128 regs (148), 24/32 B	128 regs (148), 24/32 B
tf_tree_part_256_w1s3	255 regs (218), 16/32 B	255 regs (218), 16/32 B	255 regs (218), 16/32 B	251 regs (231), none	251 regs (231), none
tf_tree_part_256_w1	255 regs (208), 16/24 B	255 regs (208), 16/24 B	255 regs (208), 16/24 B	253 regs (233), none	253 regs (233), none
tf_tree_part_w1	197 regs (168), none	197 regs (168), none	197 regs (168), none	168 regs (168), 56/52 B	168 regs (168), 56/52 B
tf_tree_part_256	255 regs (229), 16/12 B	255 regs (229), 16/12 B	255 regs (229), 16/12 B	255 regs (231), 44/40 B	255 regs (231), 44/40 B
tf_tree_shared_256_w1s3	255 regs (220), 12/32 B	255 regs (220), 4/0 B	255 regs (220), 4/0 B	250 regs (236), none	250 regs (236), none
tf_tree_shared_256_w1	255 regs (210), 12/24 B	255 regs (210), 4/0 B	255 regs (210), 4/0 B	249 regs (237), none	249 regs (237), none
tf_tree_shared_256	255 regs (230), 36/76 B	255 regs (230), 36/76 B	255 regs (230), 36/76 B	255 regs (243), 48/48 B	255 regs (243), 48/48 B

On the GB10 (sm_121), the tf_tree_merge spill is one 8-byte store at the kernel's start and one load after the main loop, per thread: outside the loops. Earlier timing showed tree attention 0 to 15.6 % faster than the head before the tree replay, and the merge times the same as the heads (below), so I see no slowdown. If you want the dim-256 and merge kernels back under the register cap before 1.1.2, we can do that.

A test fix only the merge needs (PR branches unchanged): main's test "the six served MoE and MTP launches are in the live registry" looks records up by name alone. tf_moe_combine now exists in both moe (main's Qwen engine) and moe_glue (#798), so it finds two and fails. With name and image matched (045985b8 in my scratch merge), zig build test passes 154/154 steps and test-cuda-host 82/82:

--- a/zig/src/core/contract/qwen_served_records.zig
+++ b/zig/src/core/contract/qwen_served_records.zig
@@ -20,7 +20,7 @@
     for (records) |record| {
         var found: usize = 0;
-        for (registry.all_cuda) |active| if (std.mem.eql(u8, active.launch.symbol, record.launch.symbol)) {
+        for (registry.all_cuda) |active| if (std.mem.eql(u8, active.launch.symbol, record.launch.symbol) and std.mem.eql(u8, active.launch.image, record.launch.image)) {
             try std.testing.expectEqualDeep(record, active);
             found += 1;
         };

Merged build: main d46d17a (1.1.0) + #746 b502747c + #798 74b749fa, merged locally (not pushed; branches untouched). For Kolibri, #991 d2f62303 on top.

Conflicts and how I resolved them:

• cuda: tree attention and DFlash block attention for every family, checked against float64 #746 into main
  • zig/build/cuda.zig, hostTests: main's longer module list kept; cuda: tree attention and DFlash block attention for every family, checked against float64 #746's contract_attn.zig host test added after it.
  • zig/src/core/contract/registry.zig: cuda: tree attention and DFlash block attention for every family, checked against float64 #746's 15 tree/block records go into main's cuda array, before the ++ registry_media/nemotron_prompt/glm imports. Main's 13.2 and 13.3 sets are unchanged.
  • zig/tests/cuda/contract_tests.zig: imports, the image list and the run body are a union (main's kernels first, then tree/block).
• cuda: sliding-window attention, sigmoid top-k routing and combine, dense rows for every family, checked against float64 #798 on top
  • zig/kernels/cuda/attn_odd.cuh (add/add): cuda: sliding-window attention, sigmoid top-k routing and combine, dense rows for every family, checked against float64 #798's file is cuda: tree attention and DFlash block attention for every family, checked against float64 #746's plus the run-time d overload of score64, so cuda: sliding-window attention, sigmoid top-k routing and combine, dense rows for every family, checked against float64 #798's is taken.
  • cuda.zig, registry.zig, contract_tests.zig: as above, plus cuda: sliding-window attention, sigmoid top-k routing and combine, dense rows for every family, checked against float64 #798's 12 window/moe/dense records and their kernel setup.
  • Every kernel, src/cuda and contract test file is byte-identical to 74b749fa. The registry has every cuda: sliding-window attention, sigmoid top-k routing and combine, dense rows for every family, checked against float64 #798 record line. The one exception is vit full_kernel<72>, where main's newer record wins.
• A semantic conflict that git did not flag: main's test "the six served MoE and MTP launches are in the live registry" (qwen_served_records.zig) looks up records by symbol alone. tf_moe_combine is now in two images, moe (main's Qwen engine) and moe_glue (cuda: sliding-window attention, sigmoid top-k routing and combine, dense rows for every family, checked against float64 #798), so the test found 2 and failed. In the merge I made the lookup match symbol and image. With that, zig build test 154/154 steps and test-cuda-host 82/82 pass. This fix has to go into the merge commit.
• kolibri1: Kolibri-1 on CUDA through the lane host, drafts off (#481) #991 on top (Kolibri only)
  • native/cuda.zig: the family list is main's plus kolibri1.native. Main's kv_pages paging stays. kolibri1: Kolibri-1 on CUDA through the lane host, drafts off (#481) #991's growRoom feeds the unpaged admit and the TooMany count, with served.stream_bytes as on main.
  • cuda_refusal.zig: main's vision, tp and tp-draft refusals stay, and the drafter refusal takes kolibri1: Kolibri-1 on CUDA through the lane host, drafts off (#481) #991's drafter_refusal.
  • root.zig: main's exports. tests/cuda/main.zig: plus the two kolibri suites. build/cuda.zig: plus the kolibri1 import and its host test. registry.zig: plus cuda: SwiGLU, output gates, Gated DeltaNet inputs, QK norm and RoPE for every family, checked against float64 #797's 15 act/gdn/qk records. contract_tests.zig: plus their setup. docs/recipes/README.md: both rows.

Results, GB10 (all raw outputs below):

• Kernel contract, ReleaseFast: merge 80/80 pass, 5/5 controls red; merge with kolibri1: Kolibri-1 on CUDA through the lane host, drafts off (#481) #991 103/103. At the heads, cuda: tree attention and DFlash block attention for every family, checked against float64 #746 21/21 and cuda: sliding-window attention, sigmoid top-k routing and combine, dense rows for every family, checked against float64 #798 37/37. Our kernels' verdicts are the same as at the heads; only the random-input ratios move, because each kernel's seed depends on its place in the list. Debug and ReleaseSafe still panic at contract_ck_lane.zig:108 on main (a read past the end that is already there). I left that alone.
• Limits: kernel-contract-limits passes on sm_80, 86, 89, 120 and 121, every record current. Only nvcc 13.0 is on this machine, so these launches have no 13.2 records yet.
• SASS of tree, block, window, moe and dense is identical to the heads on all five targets.
• Finite bytes: the merge equals cuda: sliding-window attention, sigmoid top-k routing and combine, dense rows for every family, checked against float64 #798's head (3,224 runs, 795,961,344 bytes, 0 differ) and cuda: tree attention and DFlash block attention for every family, checked against float64 #746's for tree and block (1,632 runs, 461,963,264 bytes, 0 differ). NaN heads: 2,080 runs, 0 bad.
• Mutations, each red: float64 weight test forced true (11 kernels and 8 limit records), a moe combine tail slot dropped (3), the dense rows lane sum halved (4), log2(e) one ulp up (170,756 bytes differ).
• Timing: merge vs heads within -0.8 to +2.8 %. One 11 us decode launch reads -7.1 %, but the merge's own three rounds spread 10.4 to 11.4 us.

Kolibri-1 on the GB10, --context 32768 --parallel 8, #991's head against the merge with #991:

Kolibri-1, GB10, --context 32768 --parallel 8	#991 head d2f62303	merged build (main + #746 + #798 + #991)
engine: kolibri-reference, kolibri-layers, kolibri-full	pass	pass, the same result lines
drafted ("draft": true) vs plain, 7 prompts × 160 tokens	7/7 equal	7/7 equal, the same 7 SHAs as d2f62303
cached/resumed vs fresh (reuse_served.py, cache 8 GiB vs 0)	35/35 equal	35/35 equal
kept states at 2048/3001/4096 vs fresh (kolibri-full)	equal	equal
concurrent (4) vs alone, t=0 and t=1.0	12/12 equal per cell	12/12 equal per cell
2,048-token prompt, first request TTFT	1.007 s	1.005 s
8,192-token prompt, first request TTFT	3.755 s	3.760 s
decode, 1 stream (code / chat, t=0)	52.3 / 50.6 tok/s	52.0 / 50.3 tok/s
decode, 4 streams aggregate (code / chat, t=0)	124.8 / 126.6 tok/s	124.6 / 126.3 tok/s
load line	model 85.23 GiB, 8 streams × 1.29 GiB	the same

The cache benchmark (bench_cache.py) reports 0 cached tokens for repeated raw-token prompts on both builds, so its --require-reuse rows are not qualified (outputs equal). Resumed vs fresh is covered by the reuse_served.py rows in the table.

kernel-contract SUMMARY lines

merge (main + #746 + #798):
SUMMARY 80 kernels: 80 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems; gates 318538 ms, limits 7089 ms
merge + #991:
SUMMARY 103 kernels: 103 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems; gates 317467 ms, limits 7378 ms
#746 head b502747c:
SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems; gates 86132 ms, limits 2 ms
#798 head 74b749fa:
SUMMARY 37 kernels: 37 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems; gates 96612 ms, limits 2 ms

kernel-contract, merge, this PR's kernels

PASS nonfinite cuda nemotron mamba scan tf_nemo_scan: the stated rule holds: glue_ref.scan's bytes (y, the window's dt, the state after the kept rows), NaN wherever it gives NaN
PASS nonfinite cuda fn_shift_window contiguous: the stated rule holds: copies all bf16 bits, including NaN payloads, infinities and signed zero
PASS repeat cuda fn_shift_window contiguous: five runs byte-identical (184320 bytes), inputs and guards untouched, no overlap
PASS shapes cuda fn_shift_window contiguous: 18 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
N/A  sums cuda fn_shift_window contiguous: declares none sums
PASS nonfinite cuda fn_shift_window strided: the stated rule holds: copies all bf16 bits, including NaN payloads, infinities and signed zero
PASS repeat cuda fn_shift_window strided: five runs byte-identical (184320 bytes), inputs and guards untouched, no overlap
PASS shapes cuda fn_shift_window strided: 18 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
N/A  sums cuda fn_shift_window strided: declares none sums
PASS nonfinite cuda tree attention tf_tree_paths: the stated rule holds: integers only: each row's depth (at most 128) and its nearest 128 ancestors root first; other slots untouched
PASS repeat cuda tree attention tf_tree_paths: five runs byte-identical (8256 bytes), inputs and guards untouched, no overlap
PASS shapes cuda tree attention tf_tree_paths: 6 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
N/A  sums cuda tree attention tf_tree_paths: declares none sums
PASS nonfinite cuda tree attention dim 128, four-warp blocks (tf_tree_shared, tf_tree_part, tf_tree_merge): the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda tree attention dim 128, four-warp blocks (tf_tree_shared, tf_tree_part, tf_tree_merge): five runs byte-identical (196608 bytes), inputs and guards untouched, no overlap
PASS shapes cuda tree attention dim 128, four-warp blocks (tf_tree_shared, tf_tree_part, tf_tree_merge): 10 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda tree attention dim 128, four-warp blocks (tf_tree_shared, tf_tree_part, tf_tree_merge): exact sums at k 2100, where fp16 sums would miss by infx and bf16 by 1839101x
PASS nonfinite cuda tree attention dim 128, one-warp blocks (tf_tree_shared_w1, tf_tree_part_w1, sliced merge): the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda tree attention dim 128, one-warp blocks (tf_tree_shared_w1, tf_tree_part_w1, sliced merge): five runs byte-identical (196608 bytes), inputs and guards untouched, no overlap
PASS shapes cuda tree attention dim 128, one-warp blocks (tf_tree_shared_w1, tf_tree_part_w1, sliced merge): 9 shapes exact by its own check (worst 0.000 of the bound), 1 edges refused by the kernel
PASS sums cuda tree attention dim 128, one-warp blocks (tf_tree_shared_w1, tf_tree_part_w1, sliced merge): exact sums at k 2100, where fp16 sums would miss by infx and bf16 by 1642027x
PASS nonfinite cuda tree attention dim 128, one-warp blocks, 3 stages (tf_tree_shared_w1s3, tf_tree_part_w1s3): the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda tree attention dim 128, one-warp blocks, 3 stages (tf_tree_shared_w1s3, tf_tree_part_w1s3): five runs byte-identical (196608 bytes), inputs and guards untouched, no overlap
PASS shapes cuda tree attention dim 128, one-warp blocks, 3 stages (tf_tree_shared_w1s3, tf_tree_part_w1s3): 9 shapes exact by its own check (worst 0.000 of the bound), 1 edges refused by the kernel
PASS sums cuda tree attention dim 128, one-warp blocks, 3 stages (tf_tree_shared_w1s3, tf_tree_part_w1s3): exact sums at k 2100, where fp16 sums would miss by infx and bf16 by 1544002x
PASS nonfinite cuda tree attention dim 256, four-warp blocks (tf_tree_shared_256, tf_tree_part_256, tf_tree_merge): the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda tree attention dim 256, four-warp blocks (tf_tree_shared_256, tf_tree_part_256, tf_tree_merge): five runs byte-identical (196608 bytes), inputs and guards untouched, no overlap
PASS shapes cuda tree attention dim 256, four-warp blocks (tf_tree_shared_256, tf_tree_part_256, tf_tree_merge): 10 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda tree attention dim 256, four-warp blocks (tf_tree_shared_256, tf_tree_part_256, tf_tree_merge): exact sums at k 2100, where fp16 sums would miss by infx and bf16 by 1708231x
PASS nonfinite cuda tree attention dim 256, one-warp blocks (tf_tree_shared_256_w1, tf_tree_part_256_w1): the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda tree attention dim 256, one-warp blocks (tf_tree_shared_256_w1, tf_tree_part_256_w1): five runs byte-identical (196608 bytes), inputs and guards untouched, no overlap
PASS shapes cuda tree attention dim 256, one-warp blocks (tf_tree_shared_256_w1, tf_tree_part_256_w1): 9 shapes exact by its own check (worst 0.000 of the bound), 1 edges refused by the kernel
PASS sums cuda tree attention dim 256, one-warp blocks (tf_tree_shared_256_w1, tf_tree_part_256_w1): exact sums at k 2100, where fp16 sums would miss by infx and bf16 by 1806768x
PASS nonfinite cuda tree attention dim 256, one-warp blocks, 3 stages (tf_tree_shared_256_w1s3): the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda tree attention dim 256, one-warp blocks, 3 stages (tf_tree_shared_256_w1s3): five runs byte-identical (196608 bytes), inputs and guards untouched, no overlap
PASS shapes cuda tree attention dim 256, one-warp blocks, 3 stages (tf_tree_shared_256_w1s3): 9 shapes exact by its own check (worst 0.000 of the bound), 1 edges refused by the kernel
PASS sums cuda tree attention dim 256, one-warp blocks, 3 stages (tf_tree_shared_256_w1s3): exact sums at k 2100, where fp16 sums would miss by infx and bf16 by 1741076x
PASS nonfinite cuda block attention tf_block_attention (DFlash2 blocks of 16): the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda block attention tf_block_attention (DFlash2 blocks of 16): five runs byte-identical (131072 bytes), inputs and guards untouched, no overlap
PASS shapes cuda block attention tf_block_attention (DFlash2 blocks of 16): 6 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda block attention tf_block_attention (DFlash2 blocks of 16): exact sums at k 2100, where fp16 sums would miss by infx and bf16 by 1644519x
PASS nonfinite cuda block attention tf_block_attention, causal blocks of 7: the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda block attention tf_block_attention, causal blocks of 7: five runs byte-identical (57344 bytes), inputs and guards untouched, no overlap
PASS shapes cuda block attention tf_block_attention, causal blocks of 7: 6 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda block attention tf_block_attention, causal blocks of 7: exact sums at k 2100, where fp16 sums would miss by infx and bf16 by 1868403x
PASS nonfinite cuda window decode tf_window_decode_tc, 48/4 heads, window 513, ring 640: the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda window decode tf_window_decode_tc, 48/4 heads, window 513, ring 640: five runs byte-identical (196608 bytes), inputs and guards untouched, no overlap
PASS shapes cuda window decode tf_window_decode_tc, 48/4 heads, window 513, ring 640: 6 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda window decode tf_window_decode_tc, 48/4 heads, window 513, ring 640: exact sums at k 3000, where fp16 sums would miss by infx and bf16 by 171083x
PASS nonfinite cuda window decode tf_window_decode_tc, 16/1 heads, no window, ring 300: the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda window decode tf_window_decode_tc, 16/1 heads, no window, ring 300: five runs byte-identical (65536 bytes), inputs and guards untouched, no overlap
PASS shapes cuda window decode tf_window_decode_tc, 16/1 heads, no window, ring 300: 5 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda window decode tf_window_decode_tc, 16/1 heads, no window, ring 300: exact sums at k 300, where fp16 sums would miss by infx and bf16 by 98558x
PASS nonfinite cuda window decode tf_window_part + tf_window_merge, 8/2 heads of 64, window 100, ring 128: the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda window decode tf_window_part + tf_window_merge, 8/2 heads of 64, window 100, ring 128: five runs byte-identical (16384 bytes), inputs and guards untouched, no overlap
PASS shapes cuda window decode tf_window_part + tf_window_merge, 8/2 heads of 64, window 100, ring 128: 6 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda window decode tf_window_part + tf_window_merge, 8/2 heads of 64, window 100, ring 128: exact sums at k 3000, where fp16 sums would miss by infx and bf16 by 256x
PASS nonfinite cuda window prompt wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2624: the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda window prompt wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2624: five runs byte-identical (786432 bytes), inputs and guards untouched, no overlap
PASS shapes cuda window prompt wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2624: 10 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda window prompt wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2624: exact sums at k 2600, where fp16 sums would miss by infx and bf16 by 886194x
PASS nonfinite cuda window prompt wattn_kernel<128, 8, 8, 8>, 16/2 heads, no window, a row a position: the stated rule holds: outputs equal the float64 reference's bytes
PASS repeat cuda window prompt wattn_kernel<128, 8, 8, 8>, 16/2 heads, no window, a row a position: five runs byte-identical (262144 bytes), inputs and guards untouched, no overlap
PASS shapes cuda window prompt wattn_kernel<128, 8, 8, 8>, 16/2 heads, no window, a row a position: 10 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
PASS sums cuda window prompt wattn_kernel<128, 8, 8, 8>, 16/2 heads, no window, a row a position: exact sums at k 100, where fp16 sums would miss by infx and bf16 by 48895x
PASS nonfinite cuda moe top-k tf_moe_topk_sigmoid_warp (Kolibri: 384 experts, top 6 + shared): the stated rule holds: picks by fp32 logit + bias, larger first, ties to the lower id, a NaN never picked; weights sigmoid(logit)
PASS repeat cuda moe top-k tf_moe_topk_sigmoid_warp (Kolibri: 384 experts, top 6 + shared): five runs byte-identical (896 bytes), inputs and guards untouched, no overlap
PASS shapes cuda moe top-k tf_moe_topk_sigmoid_warp (Kolibri: 384 experts, top 6 + shared): 10 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
N/A  sums cuda moe top-k tf_moe_topk_sigmoid_warp (Kolibri: 384 experts, top 6 + shared): declares none sums
PASS nonfinite cuda moe top-k tf_moe_topk_sigmoid_warp (256 experts, top 8 normalized x2.5): the stated rule holds: picks by fp32 logit + bias, larger first, ties to the lower id, a NaN never picked; weights sigmoid(logit)
PASS repeat cuda moe top-k tf_moe_topk_sigmoid_warp (256 experts, top 8 normalized x2.5): five runs byte-identical (1024 bytes), inputs and guards untouched, no overlap
PASS shapes cuda moe top-k tf_moe_topk_sigmoid_warp (256 experts, top 8 normalized x2.5): 10 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
N/A  sums cuda moe top-k tf_moe_topk_sigmoid_warp (256 experts, top 8 normalized x2.5): declares none sums
PASS nonfinite cuda moe top-k tf_moe_topk_sigmoid (block form, 513 to 4,096 experts): the stated rule holds: picks by fp32 logit + bias, larger first, ties to the lower id, a NaN never picked; weights sigmoid(logit)
PASS repeat cuda moe top-k tf_moe_topk_sigmoid (block form, 513 to 4,096 experts): five runs byte-identical (8320 bytes), inputs and guards untouched, no overlap
PASS shapes cuda moe top-k tf_moe_topk_sigmoid (block form, 513 to 4,096 experts): 3 shapes exact by its own check (worst 0.000 of the bound), 0 edges refused by the kernel
N/A  sums cuda moe top-k tf_moe_topk_sigmoid (block form, 513 to 4,096 experts): declares none sums
PASS nonfinite cuda moe combine tf_moe_combine (fp32 rows into bf16): 4 non-finite and 20476 finite outputs match the float64 classes
PASS repeat cuda moe combine tf_moe_combine (fp32 rows into bf16): five runs byte-identical (81920 bytes), inputs and guards untouched, no overlap
PASS shapes cuda moe combine tf_moe_combine (fp32 rows into bf16): 11 shapes within the float64 bound (worst 0.996 of the bound), 0 edges refused by the kernel
PASS sums cuda moe combine tf_moe_combine (fp32 rows into bf16): exact sums at k 9, where fp16 sums would miss by infx and bf16 by 2837x
PASS nonfinite cuda moe combine tf_moe_combine (bf16 rows into bf16): 4 non-finite and 20476 finite outputs match the float64 classes
PASS repeat cuda moe combine tf_moe_combine (bf16 rows into bf16): five runs byte-identical (81920 bytes), inputs and guards untouched, no overlap
PASS shapes cuda moe combine tf_moe_combine (bf16 rows into bf16): 11 shapes within the float64 bound (worst 0.996 of the bound), 0 edges refused by the kernel
PASS sums cuda moe combine tf_moe_combine (bf16 rows into bf16): exact sums at k 9, where fp16 sums would miss by infx and bf16 by 2980x
PASS nonfinite cuda moe combine tf_moe_combine (fp32 rows into fp32): 4 non-finite and 20476 finite outputs match the float64 classes
PASS repeat cuda moe combine tf_moe_combine (fp32 rows into fp32): five runs byte-identical (163840 bytes), inputs and guards untouched, no overlap
PASS shapes cuda moe combine tf_moe_combine (fp32 rows into fp32): 11 shapes within the float64 bound (worst 0.577 of the bound), 0 edges refused by the kernel
PASS sums cuda moe combine tf_moe_combine (fp32 rows into fp32): exact sums at k 9, where fp16 sums would miss by infx and bf16 by 112882174x
PASS nonfinite cuda moe combine tf_moe_combine (fp16 rows into fp16, the generic loop): 4 non-finite and 20476 finite outputs match the float64 classes
PASS repeat cuda moe combine tf_moe_combine (fp16 rows into fp16, the generic loop): five runs byte-identical (81920 bytes), inputs and guards untouched, no overlap
PASS shapes cuda moe combine tf_moe_combine (fp16 rows into fp16, the generic loop): 11 shapes within the float64 bound (worst 0.997 of the bound), 0 edges refused by the kernel
PASS sums cuda moe combine tf_moe_combine (fp16 rows into fp16, the generic loop): exact sums at k 9, where fp16 sums would miss by infx and bf16 by 463979x
PASS nonfinite cuda dense rows tf_dense_rows_1: 1536 non-finite and 384 finite outputs match the float64 classes
PASS repeat cuda dense rows tf_dense_rows_1: five runs byte-identical (1536 bytes), inputs and guards untouched, no overlap
PASS shapes cuda dense rows tf_dense_rows_1: 12 shapes within the float64 bound (worst 0.188 of the bound), 0 edges refused by the kernel
PASS sums cuda dense rows tf_dense_rows_1: exact sums at k 7168, where fp16 sums would miss by 128410230x and bf16 by 699050667x
PASS nonfinite cuda dense rows tf_dense_rows_4: 1536 non-finite and 384 finite outputs match the float64 classes
PASS repeat cuda dense rows tf_dense_rows_4: five runs byte-identical (6144 bytes), inputs and guards untouched, no overlap
PASS shapes cuda dense rows tf_dense_rows_4: 15 shapes within the float64 bound (worst 0.167 of the bound), 0 edges refused by the kernel
PASS sums cuda dense rows tf_dense_rows_4: exact sums at k 7168, where fp16 sums would miss by 154807948x and bf16 by 764125929x
PASS nonfinite cuda dense rows tf_dense_rows_16: 1536 non-finite and 1536 finite outputs match the float64 classes
PASS repeat cuda dense rows tf_dense_rows_16: five runs byte-identical (24576 bytes), inputs and guards untouched, no overlap
PASS shapes cuda dense rows tf_dense_rows_16: 15 shapes within the float64 bound (worst 0.198 of the bound), 0 edges refused by the kernel
PASS sums cuda dense rows tf_dense_rows_16: exact sums at k 7168, where fp16 sums would miss by 178677350x and bf16 by 730173618x
PASS nonfinite cuda dense rows tf_dense_rows_32: 1536 non-finite and 1536 finite outputs match the float64 classes
PASS repeat cuda dense rows tf_dense_rows_32: five runs byte-identical (49152 bytes), inputs and guards untouched, no overlap
PASS shapes cuda dense rows tf_dense_rows_32: 17 shapes within the float64 bound (worst 0.246 of the bound), 0 edges refused by the kernel
PASS sums cuda dense rows tf_dense_rows_32: exact sums at k 7168, where fp16 sums would miss by 150994944x and bf16 by 801311793x
PASS limits live flashnext_batch fn_shift_window: 256 threads launched, the driver allows 1024; 40 registers, 0 + 0 shared bytes (recorded 40, 0)
PASS limits live tree attention tf_tree_paths: 128 threads launched, the driver allows 1024; 30 registers, 0 + 0 shared bytes (recorded 30, 0)
PASS limits live tree attention tf_tree_shared: 128 threads launched, the driver allows 128; 212 registers, 0 + 65536 shared bytes (recorded 212, 0)
PASS limits live tree attention tf_tree_shared_w1: 32 threads launched, the driver allows 32; 195 registers, 0 + 32768 shared bytes (recorded 195, 0)
PASS limits live tree attention tf_tree_shared_w1s3: 32 threads launched, the driver allows 32; 198 registers, 0 + 49152 shared bytes (recorded 198, 0)
PASS limits live tree attention tf_tree_part: 128 threads launched, the driver allows 128; 228 registers, 0 + 65536 shared bytes (recorded 228, 0)
PASS limits live tree attention tf_tree_part_w1: 32 threads launched, the driver allows 32; 168 registers, 0 + 32768 shared bytes (recorded 168, 0)
PASS limits live tree attention tf_tree_part_w1s3: 32 threads launched, the driver allows 32; 202 registers, 0 + 49152 shared bytes (recorded 202, 0)
PASS limits live tree attention tf_tree_shared_256: 128 threads launched, the driver allows 128; 255 registers, 0 + 73728 shared bytes (recorded 255, 0)
PASS limits live tree attention tf_tree_shared_256_w1: 32 threads launched, the driver allows 32; 249 registers, 0 + 40960 shared bytes (recorded 249, 0)
PASS limits live tree attention tf_tree_shared_256_w1s3: 32 threads launched, the driver allows 32; 250 registers, 0 + 57344 shared bytes (recorded 250, 0)
PASS limits live tree attention tf_tree_part_256: 128 threads launched, the driver allows 128; 255 registers, 0 + 73728 shared bytes (recorded 255, 0)
PASS limits live tree attention tf_tree_part_256_w1: 32 threads launched, the driver allows 32; 253 registers, 0 + 40960 shared bytes (recorded 253, 0)
PASS limits live tree attention tf_tree_part_256_w1s3: 32 threads launched, the driver allows 32; 251 registers, 0 + 57344 shared bytes (recorded 251, 0)
PASS limits live tree attention tf_tree_merge: 256 threads launched, the driver allows 256; 128 registers, 0 + 0 shared bytes (recorded 128, 0)
PASS limits live block attention tf_block_attention (up to 8 query heads a key head): 256 threads launched, the driver allows 256; 228 registers, 0 + 16384 shared bytes (recorded 228, 0)
PASS limits live window attention tf_window_part: 256 threads launched, the driver allows 256; 56 registers, 45824 + 0 shared bytes (recorded 56, 45824)
PASS limits live window attention tf_window_merge: 256 threads launched, the driver allows 256; 48 registers, 0 + 0 shared bytes (recorded 48, 0)
PASS limits live window attention tf_window_decode_tc: 128 threads launched, the driver allows 128; 178 registers, 0 + 65536 shared bytes (recorded 178, 0)
PASS limits live window attention wattn_kernel<128, 8, 4, 8>: 256 threads launched, the driver allows 256; 222 registers, 0 + 65536 shared bytes (recorded 222, 0)
PASS limits live window attention wattn_kernel<128, 8, 8, 8>: 256 threads launched, the driver allows 256; 222 registers, 0 + 65536 shared bytes (recorded 222, 0)
PASS limits live moe tf_moe_topk_sigmoid: 256 threads launched, the driver allows 256; 48 registers, 16704 + 0 shared bytes (recorded 48, 16704)
PASS limits live moe tf_moe_topk_sigmoid_warp: 256 threads launched, the driver allows 256; 64 registers, 0 + 0 shared bytes (recorded 64, 0)
PASS limits live moe tf_moe_combine: 256 threads launched, the driver allows 256; 40 registers, 0 + 0 shared bytes (recorded 40, 0)
PASS limits live dense rows tf_dense_rows_1: 256 threads launched, the driver allows 256; 38 registers, 0 + 0 shared bytes (recorded 38, 0)
PASS limits live dense rows tf_dense_rows_4: 256 threads launched, the driver allows 256; 45 registers, 0 + 0 shared bytes (recorded 45, 0)
PASS limits live dense rows tf_dense_rows_16: 256 threads launched, the driver allows 256; 98 registers, 0 + 0 shared bytes (recorded 98, 0)
PASS limits live dense rows tf_dense_rows_32: 256 threads launched, the driver allows 256; 180 registers, 0 + 0 shared bytes (recorded 180, 0)

kernel-contract-limits, merge, this PR's records (sm_80, 86, 89, 120, 121)

RECORD flashnext_batch fn_shift_window: .usage = .{ .{ .regs = 32, .static = 0, .bound = 0 }, .{ .regs = 40, .static = 0, .bound = 0 }, .{ .regs = 40, .static = 0, .bound = 0 }, .{ .regs = 40, .static = 0, .bound = 0 }, .{ .regs = 40, .static = 0, .bound = 0 }, }
RECORD tree attention tf_tree_paths: .usage = .{ .{ .regs = 32, .static = 0, .bound = 0 }, .{ .regs = 40, .static = 0, .bound = 0 }, .{ .regs = 40, .static = 0, .bound = 0 }, .{ .regs = 30, .static = 0, .bound = 0 }, .{ .regs = 30, .static = 0, .bound = 0 }, }
RECORD tree attention tf_tree_shared: .usage = .{ .{ .regs = 202, .static = 0, .bound = 128 }, .{ .regs = 202, .static = 0, .bound = 128 }, .{ .regs = 202, .static = 0, .bound = 128 }, .{ .regs = 212, .static = 0, .bound = 128 }, .{ .regs = 212, .static = 0, .bound = 128 }, }
RECORD tree attention tf_tree_shared_w1: .usage = .{ .{ .regs = 193, .static = 0, .bound = 32 }, .{ .regs = 193, .static = 0, .bound = 32 }, .{ .regs = 193, .static = 0, .bound = 32 }, .{ .regs = 195, .static = 0, .bound = 32 }, .{ .regs = 195, .static = 0, .bound = 32 }, }
RECORD tree attention tf_tree_shared_w1s3: .usage = .{ .{ .regs = 190, .static = 0, .bound = 32 }, .{ .regs = 190, .static = 0, .bound = 32 }, .{ .regs = 190, .static = 0, .bound = 32 }, .{ .regs = 198, .static = 0, .bound = 32 }, .{ .regs = 198, .static = 0, .bound = 32 }, }
RECORD tree attention tf_tree_part: .usage = .{ .{ .regs = 202, .static = 0, .bound = 128 }, .{ .regs = 202, .static = 0, .bound = 128 }, .{ .regs = 202, .static = 0, .bound = 128 }, .{ .regs = 228, .static = 0, .bound = 128 }, .{ .regs = 228, .static = 0, .bound = 128 }, }
RECORD tree attention tf_tree_part_w1: .usage = .{ .{ .regs = 197, .static = 0, .bound = 32 }, .{ .regs = 197, .static = 0, .bound = 32 }, .{ .regs = 197, .static = 0, .bound = 32 }, .{ .regs = 168, .static = 0, .bound = 32 }, .{ .regs = 168, .static = 0, .bound = 32 }, }
RECORD tree attention tf_tree_part_w1s3: .usage = .{ .{ .regs = 197, .static = 0, .bound = 32 }, .{ .regs = 197, .static = 0, .bound = 32 }, .{ .regs = 197, .static = 0, .bound = 32 }, .{ .regs = 202, .static = 0, .bound = 32 }, .{ .regs = 202, .static = 0, .bound = 32 }, }
RECORD tree attention tf_tree_shared_256: .usage = .{ .{ .regs = 255, .static = 0, .bound = 128 }, .{ .regs = 255, .static = 0, .bound = 128 }, .{ .regs = 255, .static = 0, .bound = 128 }, .{ .regs = 255, .static = 0, .bound = 128 }, .{ .regs = 255, .static = 0, .bound = 128 }, }
RECORD tree attention tf_tree_shared_256_w1: .usage = .{ .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 249, .static = 0, .bound = 32 }, .{ .regs = 249, .static = 0, .bound = 32 }, }
RECORD tree attention tf_tree_shared_256_w1s3: .usage = .{ .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 250, .static = 0, .bound = 32 }, .{ .regs = 250, .static = 0, .bound = 32 }, }
RECORD tree attention tf_tree_part_256: .usage = .{ .{ .regs = 255, .static = 0, .bound = 128 }, .{ .regs = 255, .static = 0, .bound = 128 }, .{ .regs = 255, .static = 0, .bound = 128 }, .{ .regs = 255, .static = 0, .bound = 128 }, .{ .regs = 255, .static = 0, .bound = 128 }, }
RECORD tree attention tf_tree_part_256_w1: .usage = .{ .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 253, .static = 0, .bound = 32 }, .{ .regs = 253, .static = 0, .bound = 32 }, }
RECORD tree attention tf_tree_part_256_w1s3: .usage = .{ .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 255, .static = 0, .bound = 32 }, .{ .regs = 251, .static = 0, .bound = 32 }, .{ .regs = 251, .static = 0, .bound = 32 }, }
RECORD tree attention tf_tree_merge: .usage = .{ .{ .regs = 114, .static = 0, .bound = 256 }, .{ .regs = 114, .static = 0, .bound = 256 }, .{ .regs = 114, .static = 0, .bound = 256 }, .{ .regs = 128, .static = 0, .bound = 256 }, .{ .regs = 128, .static = 0, .bound = 256 }, }
RECORD block attention tf_block_attention (up to 8 query heads a key head): .usage = .{ .{ .regs = 230, .static = 0, .bound = 256 }, .{ .regs = 230, .static = 0, .bound = 256 }, .{ .regs = 230, .static = 0, .bound = 256 }, .{ .regs = 228, .static = 0, .bound = 256 }, .{ .regs = 228, .static = 0, .bound = 256 }, }
RECORD window attention tf_window_part: .usage = .{ .{ .regs = 55, .static = 45824, .bound = 256 }, .{ .regs = 50, .static = 45824, .bound = 256 }, .{ .regs = 50, .static = 45824, .bound = 256 }, .{ .regs = 56, .static = 45824, .bound = 256 }, .{ .regs = 56, .static = 45824, .bound = 256 }, }
RECORD window attention tf_window_merge: .usage = .{ .{ .regs = 32, .static = 0, .bound = 256 }, .{ .regs = 40, .static = 0, .bound = 256 }, .{ .regs = 40, .static = 0, .bound = 256 }, .{ .regs = 48, .static = 0, .bound = 256 }, .{ .regs = 48, .static = 0, .bound = 256 }, }
RECORD window attention tf_window_decode_tc: .usage = .{ .{ .regs = 176, .static = 0, .bound = 128 }, .{ .regs = 176, .static = 0, .bound = 128 }, .{ .regs = 176, .static = 0, .bound = 128 }, .{ .regs = 178, .static = 0, .bound = 128 }, .{ .regs = 178, .static = 0, .bound = 128 }, }
RECORD window attention wattn_kernel<128, 8, 4, 8>: .usage = .{ .{ .regs = 223, .static = 0, .bound = 256 }, .{ .regs = 223, .static = 0, .bound = 256 }, .{ .regs = 223, .static = 0, .bound = 256 }, .{ .regs = 222, .static = 0, .bound = 256 }, .{ .regs = 222, .static = 0, .bound = 256 }, }
RECORD window attention wattn_kernel<128, 8, 8, 8>: .usage = .{ .{ .regs = 223, .static = 0, .bound = 256 }, .{ .regs = 223, .static = 0, .bound = 256 }, .{ .regs = 223, .static = 0, .bound = 256 }, .{ .regs = 222, .static = 0, .bound = 256 }, .{ .regs = 222, .static = 0, .bound = 256 }, }
RECORD moe tf_moe_topk_sigmoid: .usage = .{ .{ .regs = 32, .static = 16704, .bound = 256 }, .{ .regs = 48, .static = 16704, .bound = 256 }, .{ .regs = 48, .static = 16704, .bound = 256 }, .{ .regs = 48, .static = 16704, .bound = 256 }, .{ .regs = 48, .static = 16704, .bound = 256 }, }
RECORD moe tf_moe_topk_sigmoid_warp: .usage = .{ .{ .regs = 63, .static = 0, .bound = 256 }, .{ .regs = 63, .static = 0, .bound = 256 }, .{ .regs = 63, .static = 0, .bound = 256 }, .{ .regs = 64, .static = 0, .bound = 256 }, .{ .regs = 64, .static = 0, .bound = 256 }, }
RECORD moe tf_moe_combine: .usage = .{ .{ .regs = 32, .static = 0, .bound = 256 }, .{ .regs = 40, .static = 0, .bound = 256 }, .{ .regs = 40, .static = 0, .bound = 256 }, .{ .regs = 40, .static = 0, .bound = 256 }, .{ .regs = 40, .static = 0, .bound = 256 }, }
RECORD dense rows tf_dense_rows_1: .usage = .{ .{ .regs = 30, .static = 0, .bound = 256 }, .{ .regs = 40, .static = 0, .bound = 256 }, .{ .regs = 40, .static = 0, .bound = 256 }, .{ .regs = 38, .static = 0, .bound = 256 }, .{ .regs = 38, .static = 0, .bound = 256 }, }
RECORD dense rows tf_dense_rows_4: .usage = .{ .{ .regs = 38, .static = 0, .bound = 256 }, .{ .regs = 38, .static = 0, .bound = 256 }, .{ .regs = 38, .static = 0, .bound = 256 }, .{ .regs = 45, .static = 0, .bound = 256 }, .{ .regs = 45, .static = 0, .bound = 256 }, }
RECORD dense rows tf_dense_rows_16: .usage = .{ .{ .regs = 79, .static = 0, .bound = 256 }, .{ .regs = 79, .static = 0, .bound = 256 }, .{ .regs = 79, .static = 0, .bound = 256 }, .{ .regs = 98, .static = 0, .bound = 256 }, .{ .regs = 98, .static = 0, .bound = 256 }, }
RECORD dense rows tf_dense_rows_32: .usage = .{ .{ .regs = 127, .static = 0, .bound = 256 }, .{ .regs = 127, .static = 0, .bound = 256 }, .{ .regs = 127, .static = 0, .bound = 256 }, .{ .regs = 180, .static = 0, .bound = 256 }, .{ .regs = 180, .static = 0, .bound = 256 }, }
INFO limits fatbins: images built by nvcc 13.0, checked against nvcc 13.0's records
PASS kernel-contract-limits: every registered launch fits sm_80, sm_86, sm_89, sm_120 and sm_121, every control red

ptxas -v per entry point, merge: regs/stack/spill store/spill load (bytes)

function: sm_80 | sm_86 | sm_89 | sm_120 | sm_121, each regs/stack/spill-store/spill-load (bytes)
tf_tree_merge: 114/0/0/0 | 114/0/0/0 | 114/0/0/0 | 128/24/24/32 | 128/24/24/32
tf_tree_part_256_w1s3: 255/64/16/32 | 255/48/16/32 | 255/48/16/32 | 251/0/0/0 | 251/0/0/0
tf_tree_part_w1s3: 197/0/0/0 | 197/0/0/0 | 197/0/0/0 | 202/0/0/0 | 202/0/0/0
tf_tree_part_256_w1: 255/72/16/24 | 255/48/16/24 | 255/48/16/24 | 253/0/0/0 | 253/0/0/0
tf_tree_part_w1: 197/0/0/0 | 197/0/0/0 | 197/0/0/0 | 168/64/56/52 | 168/64/56/52
tf_tree_part_256: 255/80/16/12 | 255/80/16/12 | 255/80/16/12 | 255/48/44/40 | 255/48/44/40
tf_tree_part: 202/0/0/0 | 202/0/0/0 | 202/0/0/0 | 228/0/0/0 | 228/0/0/0
tf_tree_shared_256_w1s3: 255/88/12/32 | 255/32/4/0 | 255/32/4/0 | 250/0/0/0 | 250/0/0/0
tf_tree_shared_w1s3: 190/0/0/0 | 190/0/0/0 | 190/0/0/0 | 198/0/0/0 | 198/0/0/0
tf_tree_shared_256_w1: 255/88/12/24 | 255/32/4/0 | 255/32/4/0 | 249/0/0/0 | 249/0/0/0
tf_tree_shared_w1: 193/0/0/0 | 193/0/0/0 | 193/0/0/0 | 195/0/0/0 | 195/0/0/0
tf_tree_shared_256: 255/128/36/76 | 255/128/36/76 | 255/128/36/76 | 255/48/48/48 | 255/48/48/48
tf_tree_shared: 202/0/0/0 | 202/0/0/0 | 202/0/0/0 | 212/0/0/0 | 212/0/0/0
tf_tree_paths: 32/0/0/0 | 40/0/0/0 | 40/0/0/0 | 30/0/0/0 | 30/0/0/0
tf_block_attention: 230/0/0/0 | 230/0/0/0 | 230/0/0/0 | 228/0/0/0 | 228/0/0/0
tf_window_decode_tc: 176/0/0/0 | 176/0/0/0 | 176/0/0/0 | 178/0/0/0 | 178/0/0/0
tf_window_merge: 32/0/0/0 | 40/0/0/0 | 40/0/0/0 | 48/0/0/0 | 48/0/0/0
tf_window_part: 55/0/0/0 | 50/0/0/0 | 50/0/0/0 | 56/0/0/0 | 56/0/0/0
tf_moe_combine: 32/0/0/0 | 40/0/0/0 | 40/0/0/0 | 40/0/0/0 | 40/0/0/0
tf_moe_topk_sigmoid_warp: 63/0/0/0 | 63/0/0/0 | 63/0/0/0 | 64/0/0/0 | 64/0/0/0
tf_moe_topk_sigmoid: 32/256/0/0 | 48/256/0/0 | 48/256/0/0 | 48/256/0/0 | 48/256/0/0
tf_dense_rows_32: 127/0/0/0 | 127/0/0/0 | 127/0/0/0 | 180/0/0/0 | 180/0/0/0
tf_dense_rows_16: 79/0/0/0 | 79/0/0/0 | 79/0/0/0 | 98/0/0/0 | 98/0/0/0
tf_dense_rows_4: 38/0/0/0 | 38/0/0/0 | 38/0/0/0 | 45/0/0/0 | 45/0/0/0
tf_dense_rows_1: 30/0/0/0 | 40/0/0/0 | 40/0/0/0 | 38/0/0/0 | 38/0/0/0

ptxas -v per entry point, main d46d17a

function: sm_80 | sm_86 | sm_89 | sm_120 | sm_121, each regs/stack/spill-store/spill-load (bytes)
tf_tree_merge: 97/0/0/0 | 97/0/0/0 | 97/0/0/0 | 148/0/0/0 | 148/0/0/0
tf_tree_part_256_w1s3: 218/0/0/0 | 218/0/0/0 | 218/0/0/0 | 231/0/0/0 | 231/0/0/0
tf_tree_part_w1s3: 168/0/0/0 | 168/0/0/0 | 168/0/0/0 | 167/0/0/0 | 167/0/0/0
tf_tree_part_256_w1: 208/0/0/0 | 208/0/0/0 | 208/0/0/0 | 233/0/0/0 | 233/0/0/0
tf_tree_part_w1: 168/0/0/0 | 168/0/0/0 | 168/0/0/0 | 168/0/0/0 | 168/0/0/0
tf_tree_part_256: 229/0/0/0 | 229/0/0/0 | 229/0/0/0 | 231/0/0/0 | 231/0/0/0
tf_tree_part: 168/0/0/0 | 168/0/0/0 | 168/0/0/0 | 162/0/0/0 | 162/0/0/0
tf_tree_shared_256_w1s3: 220/0/0/0 | 220/0/0/0 | 220/0/0/0 | 236/0/0/0 | 236/0/0/0
tf_tree_shared_w1s3: 168/0/0/0 | 168/0/0/0 | 168/0/0/0 | 168/8/4/4 | 168/8/4/4
tf_tree_shared_256_w1: 210/0/0/0 | 210/0/0/0 | 210/0/0/0 | 237/0/0/0 | 237/0/0/0
tf_tree_shared_w1: 168/0/0/0 | 168/0/0/0 | 168/0/0/0 | 168/8/4/4 | 168/8/4/4
tf_tree_shared_256: 230/0/0/0 | 230/0/0/0 | 230/0/0/0 | 243/0/0/0 | 243/0/0/0
tf_tree_shared: 168/0/0/0 | 168/0/0/0 | 168/0/0/0 | 167/0/0/0 | 167/0/0/0
tf_tree_paths: 32/0/0/0 | 40/0/0/0 | 40/0/0/0 | 30/0/0/0 | 30/0/0/0
tf_block_attention: 166/0/0/0 | 166/0/0/0 | 166/0/0/0 | 170/0/0/0 | 170/0/0/0
tf_window_decode_tc: 164/0/0/0 | 164/0/0/0 | 164/0/0/0 | 168/0/0/0 | 168/0/0/0
tf_window_merge: 32/0/0/0 | 40/0/0/0 | 40/0/0/0 | 48/0/0/0 | 48/0/0/0
tf_window_part: 52/0/0/0 | 44/0/0/0 | 44/0/0/0 | 48/0/0/0 | 48/0/0/0
tf_moe_combine: 32/0/0/0 | 40/0/0/0 | 40/0/0/0 | 40/0/0/0 | 40/0/0/0
tf_moe_topk_sigmoid_warp: 63/0/0/0 | 63/0/0/0 | 63/0/0/0 | 64/0/0/0 | 64/0/0/0
tf_moe_topk_sigmoid: 32/256/0/0 | 48/256/0/0 | 48/256/0/0 | 48/256/0/0 | 48/256/0/0
tf_dense_rows_32: 127/0/0/0 | 127/0/0/0 | 127/0/0/0 | 180/0/0/0 | 180/0/0/0
tf_dense_rows_16: 79/0/0/0 | 79/0/0/0 | 79/0/0/0 | 98/0/0/0 | 98/0/0/0
tf_dense_rows_4: 38/0/0/0 | 38/0/0/0 | 38/0/0/0 | 45/0/0/0 | 45/0/0/0
tf_dense_rows_1: 30/0/0/0 | 40/0/0/0 | 40/0/0/0 | 38/0/0/0 | 38/0/0/0

SASS, merge vs heads

tree_attention: sass 7f110c7348c15ebc vs 7f110c7348c15ebc IDENTICAL (5 images)
block_attention: sass d462152e749a07a2 vs d462152e749a07a2 IDENTICAL (5 images)
tree_attention: sass 7f110c7348c15ebc vs 7f110c7348c15ebc IDENTICAL (5 images)
block_attention: sass d462152e749a07a2 vs d462152e749a07a2 IDENTICAL (5 images)
window_attention: sass 61640a82459ce061 vs 61640a82459ce061 IDENTICAL (5 images)
moe_glue: sass af4323610f70b73f vs af4323610f70b73f IDENTICAL (5 images)
dense_rows: sass 9729df25627cd387 vs 9729df25627cd387 IDENTICAL (5 images)

numerical suites, merge

$ tf-cuda-test tree   (M)
PASS tree attention: dim 128 and 256, committed keys and paths within float64 tolerance (worst 0.470), a stream's bits alone, any group slots, four-warp or one-warp blocks
$ tf-cuda-test block   (M)
PASS block attention: windowed contexts then the block, within float64 tolerance (worst 0.380), a stream's bits alone
$ tf-cuda-test window   (M)
PASS window attention: decode and prompt rows within float64 tolerance (worst 0.465), bits at any row count and chunk
$ tf-cuda-test moe-glue   (M)
PASS moe glue: picks exact, weights and sums within float64 tolerance (worst 0.996), every row count's bits equal
$ tf-cuda-test dense-rows   (M)
PASS dense rows: within float64 tolerance (worst 0.007), every row count's bits equal

finite bytes: merge vs heads, and the log2(e) mutation

merge vs #746 head:
tree0: runs 216 bytes 65273856 differ 0 nonfinite elems 0
tree1: runs 212 bytes 62128128 differ 0 nonfinite elems 0
tree2: runs 212 bytes 62128128 differ 0 nonfinite elems 0
tree3: runs 216 bytes 65273856 differ 0 nonfinite elems 0
tree4: runs 212 bytes 62128128 differ 0 nonfinite elems 0
tree5: runs 212 bytes 62128128 differ 0 nonfinite elems 0
block0: runs 176 bytes 57671680 differ 0 nonfinite elems 0
block1: runs 176 bytes 25231360 differ 0 nonfinite elems 0
TOTAL runs 1632 bytes 461963264 differing 0 nonfinite 0

merge vs #798 head:
block0: runs 176 bytes 57671680 differ 0 nonfinite elems 0
block1: runs 176 bytes 25231360 differ 0 nonfinite elems 0
tree0: runs 216 bytes 65273856 differ 0 nonfinite elems 0
tree1: runs 212 bytes 62128128 differ 0 nonfinite elems 0
tree2: runs 212 bytes 62128128 differ 0 nonfinite elems 0
tree3: runs 216 bytes 65273856 differ 0 nonfinite elems 0
tree4: runs 212 bytes 62128128 differ 0 nonfinite elems 0
tree5: runs 212 bytes 62128128 differ 0 nonfinite elems 0
window0: runs 316 bytes 72376320 differ 0 nonfinite elems 0
window1: runs 200 bytes 14286848 differ 0 nonfinite elems 0
window2: runs 316 bytes 6031360 differ 0 nonfinite elems 0
window3: runs 380 bytes 183730176 differ 0 nonfinite elems 0
window4: runs 380 bytes 57573376 differ 0 nonfinite elems 0
TOTAL runs 3224 bytes 795961344 differing 0 nonfinite 0

merge vs merge with LOG2E one ulp up (tree, block, window):
block0: runs 176 bytes 57671680 differ 18604 nonfinite elems 0
block1: runs 176 bytes 25231360 differ 8096 nonfinite elems 0
tree0: runs 216 bytes 65273856 differ 14107 nonfinite elems 0
tree1: runs 212 bytes 62128128 differ 12618 nonfinite elems 0
tree2: runs 212 bytes 62128128 differ 12618 nonfinite elems 0
tree3: runs 216 bytes 65273856 differ 12618 nonfinite elems 0
tree4: runs 212 bytes 62128128 differ 11128 nonfinite elems 0
tree5: runs 212 bytes 62128128 differ 11128 nonfinite elems 0
window0: runs 316 bytes 72376320 differ 6534 nonfinite elems 0
window1: runs 200 bytes 14286848 differ 763 nonfinite elems 0
window2: runs 316 bytes 6031360 differ 0 nonfinite elems 0
window3: runs 380 bytes 183730176 differ 41728 nonfinite elems 0
window4: runs 380 bytes 57573376 differ 20814 nonfinite elems 0
TOTAL runs 3224 bytes 795961344 differing 170756 nonfinite 0

NANHEAD cuda window decode tf_window_decode_tc, 48/4 heads, window 513, ring 640: 768 runs, 0 bad so far
NANHEAD cuda window decode tf_window_decode_tc, 16/1 heads, no window, ring 300: 160 runs, 0 bad so far
NANHEAD cuda window decode tf_window_part + tf_window_merge, 8/2 heads of 64, window 100, ring 128: 128 runs, 0 bad so far
NANHEAD cuda window prompt wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2624: 768 runs, 0 bad so far
NANHEAD cuda window prompt wattn_kernel<128, 8, 8, 8>, 16/2 heads, no window, a row a position: 256 runs, 0 bad so far
NANHEAD total runs 2080 outputs 325156864 bad 0

mutations, merge (controls lines omitted)

weight test forced true (attn_odd.cuh: `>= least() || true`):
FAIL nonfinite cuda tree attention dim 128, four-warp blocks (tf_tree_shared, tf_tree_part, tf_tree_merge): head 13 element 21: pos_inf where float64 gives nan
FAIL nonfinite cuda tree attention dim 128, one-warp blocks (tf_tree_shared_w1, tf_tree_part_w1, sliced merge): head 13 element 21: pos_inf where float64 gives nan
FAIL nonfinite cuda tree attention dim 128, one-warp blocks, 3 stages (tf_tree_shared_w1s3, tf_tree_part_w1s3): head 13 element 21: pos_inf where float64 gives nan
FAIL nonfinite cuda tree attention dim 256, four-warp blocks (tf_tree_shared_256, tf_tree_part_256, tf_tree_merge): head 7 element 21: pos_inf where float64 gives nan
FAIL nonfinite cuda tree attention dim 256, one-warp blocks (tf_tree_shared_256_w1, tf_tree_part_256_w1): head 7 element 21: pos_inf where float64 gives nan
FAIL nonfinite cuda tree attention dim 256, one-warp blocks, 3 stages (tf_tree_shared_256_w1s3): head 7 element 21: pos_inf where float64 gives nan
FAIL nonfinite cuda block attention tf_block_attention (DFlash2 blocks of 16): head 20 element 17: pos_inf where float64 gives nan
FAIL nonfinite cuda block attention tf_block_attention, causal blocks of 7: head 20 element 17: pos_inf where float64 gives nan
FAIL nonfinite cuda window decode tf_window_decode_tc, 48/4 heads, window 513, ring 640: head 361 element 21: pos_inf where float64 gives nan
FAIL nonfinite cuda window decode tf_window_decode_tc, 16/1 heads, no window, ring 300: head 65 element 7: pos_inf where float64 gives nan
FAIL nonfinite cuda window prompt wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2624: head 252 element 17: pos_inf where float64 gives nan
FAIL limits live tree attention tf_tree_shared: 128 threads launched, the driver allows 128; 208 registers, 0 + 65536 shared bytes (recorded 212, 0)
FAIL limits live tree attention tf_tree_part: 128 threads launched, the driver allows 128; 224 registers, 0 + 65536 shared bytes (recorded 228, 0)
FAIL limits live tree attention tf_tree_part_w1: 32 threads launched, the driver allows 32; 191 registers, 0 + 32768 shared bytes (recorded 168, 0)
FAIL limits live tree attention tf_tree_part_w1s3: 32 threads launched, the driver allows 32; 216 registers, 0 + 49152 shared bytes (recorded 202, 0)
FAIL limits live tree attention tf_tree_shared_256_w1s3: 32 threads launched, the driver allows 32; 255 registers, 0 + 57344 shared bytes (recorded 250, 0)
FAIL limits live tree attention tf_tree_part_256_w1: 32 threads launched, the driver allows 32; 255 registers, 0 + 40960 shared bytes (recorded 253, 0)
FAIL limits live tree attention tf_tree_part_256_w1s3: 32 threads launched, the driver allows 32; 255 registers, 0 + 57344 shared bytes (recorded 251, 0)
FAIL limits live block attention tf_block_attention (up to 8 query heads a key head): 256 threads launched, the driver allows 256; 173 registers, 0 + 16384 shared bytes (recorded 228, 0)
FAIL kernel-contract: TestFailed

moe combine tail drops a slot (`k + i + 1 < slots`):
FAIL nonfinite cuda moe combine tf_moe_combine (fp32 rows into bf16): 1 of 20480 outputs differ in class; first at 7680: pos_inf where float64 gives nan
FAIL shapes cuda moe combine tf_moe_combine (fp32 rows into bf16): at 16x2560x7 output 0 is -1.162500e1 where float64 gives -1.060424e1: 24.64 of its bound
FAIL nonfinite cuda moe combine tf_moe_combine (bf16 rows into bf16): 1 of 20480 outputs differ in class; first at 7680: pos_inf where float64 gives nan
FAIL shapes cuda moe combine tf_moe_combine (bf16 rows into bf16): at 16x2560x7 output 1 is -1.039063e0 where float64 gives -9.287561e-1: 30.40 of its bound
FAIL nonfinite cuda moe combine tf_moe_combine (fp32 rows into fp32): 1 of 20480 outputs differ in class; first at 7680: pos_inf where float64 gives nan
FAIL shapes cuda moe combine tf_moe_combine (fp32 rows into fp32): at 16x2560x7 output 0 is 1.601148e0 where float64 gives 1.020296e0: 303258.95 of its bound
FAIL kernel-contract: TestFailed

dense rows lane sum from m = 8 (half the lanes):
FAIL nonfinite cuda dense rows tf_dense_rows_1: 597 of 1920 outputs differ in class; first at 768: finite where float64 gives neg_inf
FAIL shapes cuda dense rows tf_dense_rows_1: at 1x384x2560 output 0 is -2.068653e1 where float64 gives 8.145527e0: 4527.61 of its bound
FAIL nonfinite cuda dense rows tf_dense_rows_4: 577 of 1920 outputs differ in class; first at 768: finite where float64 gives pos_inf
FAIL shapes cuda dense rows tf_dense_rows_4: at 4x384x2560 output 0 is 4.029803e0 where float64 gives -1.770996e1: 3450.39 of its bound
FAIL nonfinite cuda dense rows tf_dense_rows_16: 564 of 3072 outputs differ in class; first at 768: finite where float64 gives neg_inf
FAIL shapes cuda dense rows tf_dense_rows_16: at 16x384x2560 output 0 is 8.500683e0 where float64 gives -2.342310e1: 4907.93 of its bound
FAIL nonfinite cuda dense rows tf_dense_rows_32: 584 of 3072 outputs differ in class; first at 768: finite where float64 gives neg_inf
FAIL shapes cuda dense rows tf_dense_rows_32: at 32x384x2560 output 0 is 2.288755e1 where float64 gives 5.905413e1: 5588.89 of its bound
FAIL kernel-contract: TestFailed

timing, best of 3 rounds x 7 x 100 calls, #798 head -> merge (us a call, launch + sync)

  +0.2%     92.45 ->    92.60  cuda tree attention dim 128, four-warp blocks (tf_tree_shared, tf_tree_part, tf_tree_merge) rows 16 k 2100
  +0.2%    129.62 ->   129.85  cuda tree attention dim 128, four-warp blocks (tf_tree_shared, tf_tree_part, tf_tree_merge) rows 64 k 700
  -0.8%     85.63 ->    84.97  cuda tree attention dim 128, one-warp blocks (tf_tree_shared_w1, tf_tree_part_w1, sliced merge) rows 16 k 2100
  -0.2%    109.78 ->   109.61  cuda tree attention dim 128, one-warp blocks (tf_tree_shared_w1, tf_tree_part_w1, sliced merge) rows 64 k 700
  +1.4%    101.99 ->   103.44  cuda tree attention dim 128, one-warp blocks, 3 stages (tf_tree_shared_w1s3, tf_tree_part_w1s3) rows 16 k 2100
  -0.3%    130.81 ->   130.48  cuda tree attention dim 128, one-warp blocks, 3 stages (tf_tree_shared_w1s3, tf_tree_part_w1s3) rows 64 k 700
  -0.5%     93.31 ->    92.85  cuda tree attention dim 256, four-warp blocks (tf_tree_shared_256, tf_tree_part_256, tf_tree_merge) rows 16 k 2100
  +0.0%    144.46 ->   144.48  cuda tree attention dim 256, four-warp blocks (tf_tree_shared_256, tf_tree_part_256, tf_tree_merge) rows 64 k 700
  +0.3%    100.95 ->   101.30  cuda tree attention dim 256, one-warp blocks (tf_tree_shared_256_w1, tf_tree_part_256_w1) rows 16 k 2100
  -0.1%    155.32 ->   155.19  cuda tree attention dim 256, one-warp blocks (tf_tree_shared_256_w1, tf_tree_part_256_w1) rows 64 k 700
  +0.7%    170.46 ->   171.73  cuda tree attention dim 256, one-warp blocks, 3 stages (tf_tree_shared_256_w1s3) rows 16 k 2100
  +0.3%    253.08 ->   253.94  cuda tree attention dim 256, one-warp blocks, 3 stages (tf_tree_shared_256_w1s3) rows 64 k 700
  +0.2%     87.63 ->    87.82  cuda block attention tf_block_attention (DFlash2 blocks of 16) rows 1 k 2100
  -0.8%     35.35 ->    35.05  cuda block attention tf_block_attention (DFlash2 blocks of 16) rows 4 k 700
  +0.2%     84.28 ->    84.47  cuda block attention tf_block_attention, causal blocks of 7 rows 1 k 2100
  +0.7%     33.35 ->    33.57  cuda block attention tf_block_attention, causal blocks of 7 rows 4 k 700
  +0.9%     18.81 ->    18.98  cuda window decode tf_window_decode_tc, 48/4 heads, window 513, ring 640 rows 16 k 3000
  -0.2%     44.19 ->    44.08  cuda window decode tf_window_decode_tc, 48/4 heads, window 513, ring 640 rows 64 k 1000
  -7.1%     11.20 ->    10.41  cuda window decode tf_window_decode_tc, 16/1 heads, no window, ring 300 rows 16 k 300
  +0.7%     17.16 ->    17.28  cuda window decode tf_window_part + tf_window_merge, 8/2 heads of 64, window 100, ring 128 rows 16 k 3000
  +1.2%     34.24 ->    34.64  cuda window decode tf_window_part + tf_window_merge, 8/2 heads of 64, window 100, ring 128 rows 64 k 1000
  +2.8%     17.00 ->    17.48  cuda window prompt wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2624 rows 128 k 0
  +0.4%     47.31 ->    47.48  cuda window prompt wattn_kernel<128, 8, 4, 8>, 48/4 heads, window 513, ring 2624 rows 96 k 2600
  +0.2%     18.67 ->    18.70  cuda window prompt wattn_kernel<128, 8, 8, 8>, 16/2 heads, no window, a row a position rows 64 k 100
  +0.1%     12.72 ->    12.73  cuda window prompt wattn_kernel<128, 8, 8, 8>, 16/2 heads, no window, a row a position rows 48 k 0

Kolibri raw comparison

reference.log: 7 result lines, identical; 1 PASS 0 FAIL
layers.log: 6 result lines, identical; 1 PASS 0 FAIL
full.log: 18 result lines, identical; 1 PASS 0 FAIL
K991: drafted==plain 7/7, shas ['34e0f8a1b2c0', 'c1657fc2f1b6', '3b3596014050', 'cc206f8c50ad', '24865c85fd4c', '947b0722b165', 'e619651b8e23']
MK: drafted==plain 7/7, shas ['34e0f8a1b2c0', 'c1657fc2f1b6', '3b3596014050', 'cc206f8c50ad', '24865c85fd4c', '947b0722b165', 'e619651b8e23']
plain SHAs baseline==merged: True
concurrent vs solo (alone equal/unequal, serial), aggregate tok/s baseline -> merged:
  code t=0.0 c=1: alone 3/3 serial 1/1; 52.3 -> 52.0 (-0.6%), ttft max 0.07 -> 0.07
  code t=0.0 c=4: alone 12/12 serial 1/1; 124.8 -> 124.6 (-0.2%), ttft max 0.36 -> 0.35
  chat t=0.0 c=1: alone 3/3 serial 1/1; 50.6 -> 50.3 (-0.6%), ttft max 0.05 -> 0.05
  chat t=0.0 c=4: alone 12/12 serial 1/1; 126.6 -> 126.3 (-0.2%), ttft max 0.28 -> 0.28
  code t=1.0 c=1: alone 3/3 serial 4/4; 50.1 -> 49.8 (-0.6%), ttft max 0.07 -> 0.07
  code t=1.0 c=4: alone 12/12 serial 4/4; 111.4 -> 111.2 (-0.2%), ttft max 0.36 -> 0.36
  chat t=1.0 c=1: alone 3/3 serial 4/4; 50.5 -> 50.2 (-0.6%), ttft max 0.05 -> 0.05
  chat t=1.0 c=4: alone 12/12 serial 4/4; 115.2 -> 115.0 (-0.2%), ttft max 0.27 -> 0.27
K991 reuse: ['PASS: 35/35 replies equal and as cached as planned', 'timing: 0 resumed replies no sooner than fresh']
MK reuse: ['PASS: 35/35 replies equal and as cached as planned', 'timing: 0 resumed replies no sooner than fresh']
fresh_2k: prompt 2048, first-request ttft 1.007 -> 1.005 s (-0.2%), decode 49.3 -> 49.4 tok/s; repeats+concurrent equal first & baseline tokens: True
fresh_8k: prompt 8192, first-request ttft 3.755 -> 3.760 s (+0.1%), decode 45.9 -> 45.9 tok/s; repeats+concurrent equal first & baseline tokens: True
cached_8k: prompt 8192, first-request ttft 3.747 -> 3.754 s (+0.2%), decode 46.0 -> 45.7 tok/s; repeats+concurrent equal first & baseline tokens: True

[ashhart]

The shared CUDA sliding-window attention, sigmoid top-k routing and combine work is included in 1.1.2 (https://github.com/ashhart/TensorFold/releases/tag/v1.1.2), with bounded window and routing fixtures. Closing this as carried, with your coauthor credit retained. Thank you for the contribution.

It landed as ffe63cb.

- Synthema, Ash's AI Assistant.