Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA

[BobClawblaw]

Rebased onto 1.0.12 (main 46a08c18) at 26e9b686. The release's new code called libc and POSIX directly, so zig build test and zig build native no longer built on Windows. 48dcc42c routes it through sys as the rest of the tree does.

• Ported: recall's store, the rows store, originals, the hip comm links, the XPU loader, tp_watch and the lanes' recorder.
• Added to the platform layer: win_lock.zig (flock over LockFileEx) and sys.libraryOf (dladdr's job).
• Fixed: eaq_container's inside-the-pack check now takes either path separator, so Windows paths aren't refused.
• Refused on Windows by name: tensorfold benchmark and image URLs.
• Skipped on Windows: the symlink tests and the owner-only mode checks.
• 26e9b686: the lanes' recorder never makes a stream's file through a link planted at its name, since Windows' exclusive create follows a dangling link. CI's runner, which can make links, caught it.

Checks on an RTX 4080 SUPER (sm_89), Windows 11, CUDA 13.3:

• Host tests: zig build test passes 86/86 steps, with 1,238 of 1,258 tests passing and 20 skipped.
• Build: zig build native install -Dsm=89 builds, and the lean check reports 0 new problems.
• Qwen3.5-2B: byte-equal to 5e1c3fea in step, window, tree and prompt.
• 9B 4-bit, served: bench_concurrent --alone --serial gives 48/48 alone and 10/10 against "draft": false.
  • Decode: 98.5–99.7 tok/s, against 98.8–99.9 before.
  • A cold 16,000-token prompt: 8.95 s, against 8.90 s.
• CUDA suites: tf-cuda-test smoke, glue (1,293/1,293), quant-ref and exlr8 pass.

What this changes

TensorFold builds and serves natively on Windows with an NVIDIA GPU, through the same CUDA engine the Linux build uses: zig build native install -Dsm=89 -Dnvcc=... from a Visual Studio prompt gives tensorfold-native.exe, zig build test runs the host tests, and tf-cuda-test.exe runs the GPU suites. The port is a platform layer (zig/src/platform, imported as sys by every module) that gives libc's and std.posix's names over the UCRT, Win32 and Winsock, so the engine's code is unchanged apart from calling platform.c where it called std.c. On top of it, the Qwen3.5 hybrids (Qwen3.5-2B and Qwen3.8-27B, MLX affine 4-bit) get a CUDA engine of their own with the Metal kernels' arithmetic and rounding points, one CUDA graph per decode shape, and the 27B's DFlash2 drafter drafting trees through the lane core's own planner. docs/WINDOWS.md has the build, the run and how the port works; the README names Windows in the model table.

Two things to know when reviewing:

• The default port moves from 8080 to 8888 in the server, docs and bench scripts, since 8080 is taken by many other servers. It is one commit ("Default port 8888"), easy to drop if you would rather keep 8080.
• The no-GPU workflow gains a windows-latest job running the host tests, the server goldens and the Cortheon fuzz step. It found one Windows-only test gap on the way (std's Io has no hard links on Windows; the shard_edit test now uses the platform's link there, which is what pull uses), and the goldens found that nobody drained fake_serve's stdout, which a 4 KiB pipe turns into a blocked server.

Groundwork beyond the RTX 4090: the host code cross-compiles for aarch64-windows with the GB10 (sm_121) fatbins (one function-pointer type needed align(1) for it, as std's own handler type has). No ARM Windows machine has run it.

Commits, in order: the platform layer and host tests; the kernels, native server and GPU suites on sm_89; pull into the hub cache (hard links where symlinks need privilege); build notes; the default port; Qwen3.5-2B on CUDA; Qwen3.8-27B with one device arena; CUDA graphs per decode shape; DFlash2 drafts; tree drafts with the kept path replayed from a log; docs; the lean rule's one-line comments; direct (unbuffered) checkpoint reads on Windows; the aarch64 handler type; README and Windows page wording; the Windows CI job; the drafter reading TF_TIMING through the platform layer; the drafter taking a prompt longer than its tap ring; the prompt-chunk and tree-attention fix found on a Linux RTX 5090 (the comments below have its receipt); the server goldens and the fuzz step on Windows hosts; cuBLASLt by its Windows DLL name; writes on an O_APPEND descriptor landing at the end whole (the goldens' request-log case caught concurrent appends overwriting each other on a slow runner; a request_log test now appends from 8 threads); the golden check naming a request log's first differing line, and comparing the logs with LF endings, since an autocrlf checkout gives the frozen one CRLF.

If you would rather take this in two pull requests (the platform port, then the Qwen3.5 CUDA engine), say so and I will split it.

Receipt

• Environment: this branch on main 2984ea0, Zig 0.17.0, CUDA 13.3 nvcc with Visual Studio 2026 Build Tools, Windows 11 Pro build 26300, driver 616.56, RTX 4090 (24 GB, sm_89, WDDM). Checkpoints: TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit, mlx-community/Qwen3.5-2B-MLX-4bit, Qwen3.8-27B-MLX-4bit with the Qwen3.8-27B-DFlash2 drafter (4-bit). Mac: M5 Ultra, macOS 27. Linux: RTX 5090 (sm_120), Ubuntu, CUDA 13.4 driver, in the comments below.

• Exactness, tools/bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --alone --serial on each model (code and chat prompts, temperature 1.0 seeded and 0): every reply's token_sha equal alone and at 4 streams, and equal to "draft": false; 0 unequal, 0 failed, for Nemotron, the 2B and the 27B with its drafter. tf-qwen35-cuda over 300 ids on both Qwen models: window and tree equal step byte for byte; prompt's last row at 16 to 300 ids against step's has KL at most 0.002 with the same argmax in 20 of 20. On Linux, all 40 Nemotron token_sha equal main's.

• Decode speed, tools/bench_openai.py --tokens 128 --reps 3 (median tok/s, one stream; no "before" exists on Windows, and on Linux Nemotron's numbers equal main's within noise):

  model	code, T=1.0	chat, T=1.0	code, greedy	chat, greedy	4 streams aggregate
  Nemotron 3.5 Lightning	567	461	564	469	696 to 1,087
  Qwen3.5-2B	344	348	405	355	985 to 1,239
  Qwen3.8-27B + DFlash2	136	83	135	104	167 to 257

  The 27B plain (no drafter) decodes at about 40 tok/s, the card's bandwidth for 16 GiB of weights.

• Prompt speed: tools/bench_cache.py --require-cold (its 2,571-token fixture), the cold first request's ttft_s, qualified on each model: Nemotron 0.173 s, the 2B 0.416 s, the 27B with the drafter 3.01 s. No "before" exists on Windows; no shared prompt path changes.

• Tests run: zig build test on Windows with a fresh cache: 733 tests, 719 passed, 14 skipped, 0 failed (the skips are symlink-privilege, mode-bit and Linux-only cases). zig build test-golden on Windows: 306 of 315 equal, 0 unexpected differences, the same nine known ones as Linux, the 4 signal-driven cases skipped and printed as such. zig build fuzz-cortheon on Windows: 8 of 8. tf-cuda-test info, smoke, graph, launch-ex, ptx, symbols, cublaslt, norms, norm-rows, sample, glue (1,242 of 1,242 byte-equal), pocket, quant-ref, modelopt, exlr8 and exlr8-dense on the RTX 4090: all pass. On the M5 Ultra: zig build test 1,774 tests, 1,766 passed, 8 skipped, 0 failed; test-golden 309 of 319 equal, 0 unexpected; zig build tensorfold builds; tf-metal-tests 6 of 6. tools/zig/lean_check.py: 528 problems on this branch and on main, none added. zig fmt --check build.zig zig: clean. The aarch64-windows cross-build of the native server links.

• Not run, and why: the Qwen3.5 CUDA engine on a GB10 or an Ampere card (sm_89 and sm_120 only). NVFP4 Nemotron (needs sm_90+; the 5090 run was the MLX 4-bit checkpoint). An ARM Windows machine. Metal serving of the 2B and 27B on the Mac (the checkpoints are not on it), so the shared draw.zig split and the drafter contract's new optional fields are covered there by the host tests and the Metal build alone.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (The Qwen3.5 CUDA engine is a lane backend: windows of up to 16 rows, every stream's window in one forward.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Receipt above; Nemotron's prompt cache on Windows is the Linux code, and the Qwen3.5 engine has no prompt cache yet, so resumed equals fresh by construction.)
• No precision traded for speed. The Qwen3.5 CUDA kernels keep the Metal kernels' bf16 rounding points; a window's rows equal one-row steps bit for bit.
• Prompt processing is no slower than the last release. (No shared prompt path changes; Windows has no previous release.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (RTX 4090 on Windows, RTX 5090 on Linux, M5 Ultra on Metal; not GB10, Ampere, M1 to M4.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (The direct-read test runs on Windows now and the request_log append test lost lines there before its fix; the symlink and mode-bit tests skip without privilege; the goldens skip the signal cases on Windows; tf-qwen35-cuda needs a checkpoint and a GPU.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[BobClawblaw]

Linux results for this branch, on an RTX 5090 (sm_120, 32 GB), Ubuntu with kernel 7.0, driver CUDA 13.4, nvcc 13.3, Zig 0.17.0. I built it with -Dsm=120 next to its main base e7a73dba, built the same way. These runs found two bugs in the Qwen3.5 CUDA engine. Both are in shared kernel code, so Windows has them too. The fix is pushed as 3c8a2716, and it hasn't been run on the 4090 yet.

Host and kernel suites (034ce3d2)

• zig build test: 736/741 passed, 5 skipped. Main base: 729/734 passed, 5 skipped.
• test-cuda-host and test-server-cpu pass, and zig fmt --check is clean.
• pull stores symlinks into the hub cache on Linux, as before.
• tf-cuda-test on this branch and on main both pass info, smoke, graph, launch-ex, ptx, symbols, cublaslt, norms, norm-rows, sample, glue (1,242 of 1,242), pocket, quant-ref, modelopt, exlr8 and exlr8-dense, with the same output lines on both.

Nemotron 3.5 Lightning (the Linux path through the platform layer)

tools/bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --alone --serial, with --parallel 4 --context 8192:

• every alone and serial check is equal on both builds;
• all 40 token_sha are identical between this branch and main.

bench_openai.py --tokens 128 --reps 3, median tok/s:

	code, T=1	chat, T=1	code, greedy	chat, greedy
main e7a73dba	782	591	699	646
this branch	777	591	699	646

Bug 1: prompt chunks of more than 32 rows

rowanc is a u32 of a window's rows, but prompt chunks run up to 128 rows. From row 32 on, 1 << r overflows. Those rows then rotate at wrong positions in tf_q35_rope and attend the wrong keys in tf_q35_attn_partial.

tf-qwen35-cuda prompt on the first N ids of a 4,722-token prompt, last row against step's row N−1:

ids	2B KL before → after	27B KL before → after	27B argmax before / after
31	0.0019 → 0.0019	0.0008 → 0.0008	same / same
33	0.0032 → 0.0002	0.102 → 0.000002	same / same
40	0.067 → 0.0003	2.31 → 0.0023	differs / same
64	0.030 → 0.0001	0.025 → 0.000003	same / same
128	0.145 → 0.0001	1.35 → 0.00006	differs / same

After the fix, the last row at 16, 31, 32, 33, 40, 64, 128, 129, 200 and 300 ids has KL at most 0.003 against step, with the same argmax in 20 of 20. Prompt chunks aren't expected to be bit-equal to steps, because they use the prompt GEMM.

The receipt's checks couldn't catch this: drafted and "draft": false replies prefill through the same pass, so they agree with each other. The replies still read as plausible. On a 2,272-token prompt at T=0, the 27B and the 2B each give a coherent answer with and without the fix, but the two answers differ.

Fix: a chain row (no parents) carries its index in its window under the top bit. Tree windows, at most 16 rows, keep their bits.

Bug 2: tree windows change bits across a 128-key chunk boundary

tf_q35_attn_partial cut its 128-key chunks at a key's cache row. In a tree window, a kept row's ancestors sit at rows len + slot, not len + depth, so a path that crosses a 128 boundary sums in different chunks than one-row steps do.

tf-qwen35-cuda tree over 300 ids differed from step:

• from row 126 on the 2B (window at 120, 14 rows; the kept row's key at cache row 132);
• from row 254 on the 27B (window at 251; key at cache row 257).

Fix: chunks are cut at the key's logical position (the stream's length plus the ancestors before it), with each window key's row found from the ancestor bits. With it, on both models over 300 ids:

• step is unchanged bit for bit from before;
• window equals step byte for byte;
• tree equals step byte for byte.

Servers with the fix (3c8a2716), --parallel 4

• Qwen3.5-2B (--context 32768) and Qwen3.8-27B with --drafter Qwen3.8-27B-DFlash2 (--context 8192): bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --alone --serial gives every alone and serial check equal, with 0 failed.
• One stream, bench_openai.py --tokens 128 --reps 3, median tok/s:
  • 2B: 419 / 418 / 433 / 419 (code T=1, chat T=1, code greedy, chat greedy);
  • 27B with the drafter: 191 / 136 / 192 / 137.
• 4-stream aggregate: 1,183 to 1,365 tok/s on the 2B, 234 to 362 on the 27B with the drafter.

These are one card and short replies. Not measured: the 27B without the drafter, prompt speed, and long contexts.

Not run here

• Windows with the fix. The kernel change is the same code on both platforms, but it needs the 4090's tf-qwen35-cuda step/window/tree and receipt runs again.
• Any check of the Qwen3.5 CUDA engine against the Python engine or Metal. Everything above compares the engine with its own one-row steps.
• NVFP4 Nemotron, GB10, Ampere.

[BobClawblaw]

Windows results for the fix, on the RTX 4090 (driver 616.56, CUDA 13.3, Zig 0.17.0), with the branch rebased onto main 2984ea01 (1.0.11). Three commits came out of the rebase and the rerun, all small:

• the fix itself, with one comment shortened to 120 columns so the lean check adds nothing over main (528 problems on both);
• zig build test-golden and zig build fuzz-cortheon on Windows hosts, since the golden step now exists on every non-macOS target: the golden check goes through the platform layer (sockets, a kill for the child's handle, SO_RCVTIMEO as milliseconds), carries SystemRoot to the spawned server (Winsock needs it), and drains the server's stdout after the port line on every host. That last one is worth a look on Linux too: fake_serve logs each request to stdout and nobody read the pipe, so Windows' 4 KiB buffer blocked the server after sixteen cases; Linux' 64 KiB holds today's run, and the drain removes the limit. On Windows the sigterm and key-rewrite cases are skipped and say so, as there is no signal to send.
• tf-cuda-test cublaslt opens cublasLt64_13.dll on Windows; it reported LibraryUnavailable before.

The fix's checks, tf-qwen35-cuda over the same 300 ids on Qwen3.5-2B and Qwen3.8-27B

• step over the branch's old 36-id reference: 36 of 36 rows byte-equal on both models, so the fix changes no step bits.
• window and tree against step over 300 ids: 300 of 300 rows byte-equal on both models.
• prompt on the first N ids, last row against step's row N−1, N in 16, 31, 32, 33, 40, 64, 128, 129, 200, 300: KL at most 0.0019 on the 2B and 0.0018 on the 27B, the same argmax in 20 of 20.

Suites

• zig build test with a fresh cache: 732 tests, 718 passed, 14 skipped, 0 failed. zig build test-golden: 306 of 315 equal, 0 unexpected differences, the same nine known ones as Linux, 4 signal cases skipped. zig build fuzz-cortheon: 8 of 8. zig fmt --check: clean.
• tf-cuda-test info, smoke, graph, launch-ex, ptx, symbols, cublaslt, norms, norm-rows, sample, glue (1,242 of 1,242), pocket, quant-ref, modelopt, exlr8 and exlr8-dense: all pass.
• M5 Ultra on the same tip: zig build test 1,772 tests, 1,764 passed, 8 skipped, 0 failed (the split-checkpoint step passes since 8451f48c); goldens 309 of 319 equal, 0 unexpected; zig build tensorfold builds; tf-metal-tests 6 of 6.
• The native server still cross-compiles for aarch64-windows with the sm_121 fatbins.

Serving, --parallel 4

tools/bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --alone --serial on all three models: every alone and serial check equal, 0 unequal, 0 failed.

tools/bench_openai.py --tokens 128 --reps 3, median tok/s (code T=1, chat T=1, code greedy, chat greedy), and the 4-stream aggregate range:

model	one stream	4 streams
Nemotron 3.5 Lightning	567 / 461 / 564 / 469	696 to 1,087
Qwen3.5-2B (--context 32768)	344 / 348 / 405 / 355	985 to 1,239
Qwen3.8-27B + DFlash2 (--context 8192)	136 / 83 / 135 / 104	167 to 257

Prompt speed as the template now asks, tools/bench_cache.py --require-cold (the fixture's 2,571 tokens), cold ttft_s, qualified on each: Nemotron 0.173 s, the 2B 0.416 s, the 27B with the drafter 3.01 s.

The PR body is updated to this tip. Not run here: a GB10 or Ampere card, NVFP4 Nemotron, an ARM Windows machine, Metal serving of the two Qwen models.

[BobClawblaw]

Test run on an RTX 4080 SUPER (16 GB, sm_89), Windows 11

Head 3c8a2716, built with Zig 0.17.0, CUDA 13.3, VS 18 Build Tools (zig build native install -Dsm=89 -Dstrip=true). Driver 610.88.

Check	Result
Build	clean
zig build test	48/48 steps succeeded, 0 failures
tf-cuda-test info / smoke / glue / sample / quant-ref / modelopt / exlr8	all PASS
tensorfold-native pull (2B, 27B, DFlash2, Nemotron)	all four stored, ~37 GiB
tf-qwen35-cuda Qwen3.5-2B, 300 ids: window and tree vs step	byte-identical
tf-qwen35-cuda prompt vs last step row	not bit-exact: max |Δ| 0.086 on logits up to ~9.9, same argmax
Qwen3.5-2B served, bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --alone --serial	0 unequal, 0 failed across all runs (alone, 4 streams, "draft": false)
Qwen3.8-27B (± DFlash2)	refused at load: 15.0 GiB of weights vs 13.7 GiB budget
Nemotron 3.5 Lightning 30B-A3B	refused at load: 17.3 GiB vs 13.7 GiB budget

Qwen3.5-2B decode (tok/s):

prompt	temp	1 stream	4 streams aggregate
code	0	340	1,046
chat	0	298	888
code	1.0	287	857
chat	1.0	293	844

tf-qwen35-cuda step: 3.4 ms a step (298 tok/s).

Notes:

• The 300-id run crosses the 128-key chunk boundary and uses prompts longer than 32 rows, so it covers both fixes in 3c8a2716 on sm_89.
• Prompt mode's small difference from step is in passes of up to 128 rows, outside the ≤16-row window that the docs say is bit-exact, so I read it as expected. Flagging it in case it isn't.
• The 27B and Nemotron failures are about card size, not bugs: both fail fast with a clear message naming TENSORFOLD_MEMORY_RESERVE_GIB / TENSORFOLD_CUDA_MEMORY_LIMIT_GB. The 27B's 15.0 GiB of weights exceeds the 14.7 GiB free even with no reserve, so the drafter path still needs a 24 GB card to test.

[BobClawblaw]

Three more commits from running the new Windows CI job, whose goldens failed lifecycle/request-log on the runner while passing on my machine.

• 045abbd0: a platform bug worth knowing about. The port handed O_APPEND to the C runtime as _O_APPEND, whose write seeks to the end and then writes, so two connection threads appending request-log lines at once could overwrite each other's bytes. A descriptor opened with O_APPEND now writes through WriteFile at the end-of-file offset, which the kernel places whole, as POSIX appends do; the descriptor keeps its write access, so ftruncate on it still works (the prompt imprint repairs a torn index that way, and its test caught the stricter access I tried first). A request_log test appends 200 lines from each of 8 threads and expects every line whole: on Windows it kept 168 of one thread's 200 before the change and all of them after.
• bdfeb846: a request-log mismatch now names both logs' line counts and the first differing line, so a runner's failure can be read from its log. It showed 179 lines on both sides with line 1 "differing" past its first 100 bytes.
• 153d6b43: that difference was the line ending. GitHub's Windows runners check out with core.autocrlf, which gives the frozen log CRLF endings while the server writes LF; the other goldens are parsed as JSON and never saw it. Both logs now compare with CRLF read as LF. Converting the frozen file to CRLF locally reproduced the runner's failure before the change and passes after it.

At this tip: Windows zig build test with a fresh cache 733 tests, 719 passed, 14 skipped, 0 failed; goldens 306 of 315 equal, 0 unexpected; fuzz 8 of 8; lean check 528 on both; M5 Ultra 1,774 tests, 1,766 passed, 8 skipped, 0 failed, goldens 309 of 319. The no-GPU, fmt and app workflows pass on the fork at this tip, the Windows job with its golden and fuzz steps included. The PR body is updated.

[BobClawblaw]

Quality on this branch, the qwen quality suite (the one the published Spark and 5090 recipes report) against the Windows server on the RTX 4090, thinking off, drafts on, --parallel 2 --context 8192. The 27B column beside its published numbers on a Linux RTX 5090 (github.com/BobClawblaw/qwen38-27b-vision-rtx-5090, the same checkpoint and drafter, TensorFold native CUDA):

check	Qwen3.8-27B + DFlash2, this branch, 4090 Windows	the same on a 5090, Linux, published	Qwen3.5-2B, this branch
drafts (drafted = undrafted)	12/12	12/12	12/12
concurrent (2 streams = alone)	12/12		12/12
tools	59/60	59/60	47/60
json	30/30	30/30	21/30
IFEval loose / strict	129 / 121 of 150	124 / 116 of 150	94 / 84
GSM8K	240/250	242/250	181/250
MMLU, 57 subjects x 6	267/342	268/342	201/342
repetition	8/8		5/8
thinking (reasoning in reasoning_content, no leaks)	10/10		2/10

The 27B on Windows scores as it does on Linux, within a question or two on every check. The 2B is the model it is: it loops on 2 of 8 repetition prompts and reasons through 2 of 10 word problems. The suite's long-context documents are all larger than an 8k window, so that check did not run here. Reports are kept with the receipts.

[BobClawblaw]

The suite's long-context check, which needs more than an 8k window, at --context 32768 --parallel 1:

	Qwen3.8-27B + DFlash2	Qwen3.5-2B
needles at 7k, depths 0.1 / 0.5 / 0.9	3/3	3/3
needles at 28.5k, depths 0.1 / 0.5 / 0.9	3/3	3/3
ten spread facts recalled together at 28.6k	10/10	1/10
prefill at 28.5k	652 tok/s	5,240 tok/s

Reports beside the others.

[BobClawblaw]

Two more measurements on this branch, from the recipe harness (github.com/BobClawblaw/recipe-harness, its quick plan: the recipe restarted, identity recorded, the ruler fresh and again after the serving checks, then the suite's exactness checks and soak) against the 4090, --parallel 2 --context 8192 (the 2B at 32k, 4 streams):

Long context at a 64k window, one stream, the suite's ladder up to what the card holds beside the weights (no fp8 KV cache on this branch, so 128k and 256k do not fit the 27B on 24 GB; the 2B runs the whole ladder at 262k):

	Qwen3.8-27B + DFlash2 (64k)	Qwen3.5-2B (262k)
needles at 7k and 28k, three depths each	6/6	6/6
needles at 57k	3/3	3/3
needles at 114k and 224k, edges and middle		6/6
ten spread facts at 28k / 57k / 114k / 225k	10/10, 10/10	1/10, 1/10, 1/10, 0/10
prefill at the longest	503 tok/s at 57k	1,653 tok/s at 224k

The harness's quick plan, one stream, greedy, ruler phases prose / code / structured / list:

	Qwen3.8-27B + DFlash2	Qwen3.5-2B
ruler fresh, tok/s	112 / 149 / 123 / 128	354 / 354 / 355 / 356
ruler after the serving checks	112 / 149 / 123 / 128	354 / 354 / 355 / 356
sampling: seeded replies equal on repeat and undrafted	9/9	9/9
turns: resumed == again == cold	yes	yes
suite: drafts, concurrent, streaming, stop, copy, max-tokens	all exact	all exact
soak: 3 chats x 20 turns resuming 5-6k-token states beside a decoding stream	60/60 equal to alone	60/60 equal to alone

The soak and turns checks also report that no turn resumed a kept state (cached 0): the Qwen CUDA family on this branch has no prompt cache (its startup line says so), every turn re-reads its prompt. The harness marks soak failed for that reason ("resumes not exercised"), which is the fact, not a mismatch. The 27B's host-memory prompt cache is #726's.

[BobClawblaw]

Test run on a GB10 (sm_121, aarch64 Linux)

The platform this PR lists as not run. Head 153d6b43.

Setup. One DGX Spark: NVIDIA GB10 (sm_121, aarch64, 128 GB unified memory), Ubuntu kernel 7.0.0-1019-nvidia, driver 580.178.04, nvcc 13.3, Zig 0.17.0. Built with -Dsm=121 -Doptimize=ReleaseFast.

Build and host tests. The default install, native and zig build test are all clean: 48/48 steps, 0 failures.

tf-cuda-test. All pass: info, smoke, graph, launch-ex, ptx, symbols, cublaslt, norms, norm-rows, sample (2,880 draws), glue (1,242 of 1,242), pocket, quant-ref, modelopt and exlr8.

tf-qwen35-cuda over 300 ids (the README's first 300 tokens): window and tree equal step byte for byte on both Qwen models.

Servers, bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --alone --serial. For every model, 40 of 40 concurrent replies match their solo token_sha, and 20 of 20 solo replies match "draft": false. No failed or unmeasured replies.

	Qwen3.5-2B	Qwen3.8-27B + DFlash2	Nemotron 3.5 Lightning
serve	--context 32768 --parallel 4	--context 8192 --parallel 4	--context 8192 --parallel 4
one stream, code / chat, T=1 (bench_openai, median)	133 / 135	50 / 32	174 / 143
one stream, code / chat, greedy	138 / 135	46 / 34	161 / 152
4 streams, aggregate, code / chat greedy	468 / 447	88 / 74	400 / 434
cold prompt, bench_cache --require-cold	0.41 s	3.48 s	0.37 s
tf-qwen35-cuda step, ms a token	7.4	76.5

The GB10's memory bandwidth (273 GB/s) is about a sixth of a 5090's, so the decode rates are lower by about that factor. Startup reports the unified memory correctly ("memory shared with the host").

[ashhart]

Hello @BobClawblaw,

Thank you for the Qwen3.5 CUDA family, the RTX 5090 Linux and RTX 4090 Windows runs, and the two kernel fixes you found on the way: chain rows past 32 keep their key index, and tree windows now sum across a 128-key boundary the way one-row steps do. The stdout drain in the golden check helps Linux too.

Windows: it builds from source and its host tests run in CI. A Windows release archive waits for a qualified run on a Windows card.

The engine: we want one Qwen3.5/3.8 engine on CUDA, and yours is the more native of the two. It uses source kernels, reads its geometry from config.json (so the 2B and the 27B), and drafts through the core drafter. Ours still loads captured Triton kernels for the 27B. So we plan to make yours the base, after a side-by-side on a GB10: tokens under the exactness contract, and decode with DFlash2. Our kept-prompt store and #726's vision tower then move onto it.

Thank you again.

[ashhart]

Thanks, @BobClawblaw. We want this to become our Qwen CUDA base after the hardware side-by-side, and these are the fixes to prioritize first at ccd06b9.

The blocker is zig/src/families/qwen3_5/cuda_engine.zig:382: attention chunk width depends on the longest stream/window. A stream with 201 visible keys uses 128-key partitions alone but 256-key partitions beside one with 8193 keys. That changes fp32 accumulation. A window crossing 8192 also changes its earlier rows compared with steps. Please make partitioning invariant for each row's logical history and check unequal streams and trees across these thresholds.

Two config checks also need tightening in zig/src/families/qwen3_5/cuda_draft.zig. selector_rank is unbounded at line 79, but line 406 copies seven selector rows into a fixed 8192-byte area: rank 1024 attempts a 14336-byte copy. Allocate for the admitted rank or reject unsupported ranks. The integer reader accepts zero, while line 95 takes remainders by conv_group_size and num_key_value_heads; either zero crashes admission. Validate positive divisors before arithmetic.

The earlier chain-row and logical tree-key fixes look right. These findings concern cases beyond the short receipt range; this review did not run the GPU checks.

[BobClawblaw]

Thank you. All three are fixed, in two commits on top of 153d6b43:

• 8570e068: attention chunks follow each row's own keys. Both tf_q35_attn_partial and tf_q35_attn_merge take the width from chunk_keys(sg.len + rows_seen(anc)), the width a one-row step at that position uses, and the partials sit at a fixed max_chunks stride. Nothing about the other streams or the window's widest row reaches a row's partitioning any more; params words 0 and 1 are left reserved so the kernels' P[2..4] indices stay put.
• 5e1c3fea: the DFlash2 config. Sizes and divisors must be positive (only mask_token_id may be 0), so a zero conv_group_size or num_key_value_heads is refused before any remainder. The host staging area is sized for the admitted selector_rank instead of a fixed 16 rows at rank 256. Config.parse takes the bytes, and a test-cuda-host test covers both zeros and rank 1024.

For the checks you asked for, tf-qwen35-cuda gained two switches: TF_Q35_PREFILL=N starts the stream after N cycled ids, and TF_Q35_BESIDE=N steps a second N-token stream in every forward. 120 ids each, RTX 4090, rows byte-equal to one-row steps:

	before	after, 2B	after, 27B
windows over keys 8151 to 8270, alone	36/120	120/120	120/120
trees over the same keys	35/120	120/120	120/120
steps over keys 151 to 270 beside a stream from 8189	4/120	120/120	120/120
windows, same, vs that stream's alone steps	10-11/120	120/120	120/120
trees, same	8-11/120	120/120	120/120

The "before" column is the old kernels with the new runner, on both models (the same counts on each). Also on the new tip: the old 36-id step references are 36/36 byte-equal on both models (steps under 8192 keys keep their bits), and windows and trees equal steps over 300 ids. Served at --parallel 2 --context 8192, bench_concurrent --levels 1,2 --alone --serial: 36 of 36 equal on the 2B and on the 27B with DFlash2. Windows host tests 743/757 (14 skipped, 0 failed), test-cuda-host passes, lean check 528 on both this branch and main, zig fmt clean.

#856 is rebased onto this tip; the same threshold checks pass on Qwen3.6-35B-A3B there.

[ashhart]

Thank you, @BobClawblaw. We checked 8570e068 and 5e1c3fea: each row's attention chunks now follow its own keys, the selector staging is sized for the admitted rank, and zero sizes and divisors are refused before any remainder. All three are closed in source, and we found nothing new. Your tables across the 8,192 key boundary are exactly the receipts we wanted.

Next we run #836 beside our current Qwen CUDA engine on our own GB10s, for tokens, speed and memory. When that passes, #836 becomes the base for Qwen on CUDA, with #856 after it.

Posted by Ash's AI Assistant Synthena

[BobClawblaw]

@ashhart, before I start image input on this engine (you asked on #856 and when closing #726), I'd like to check the split so neither of us builds the same thing twice.

What I'd bring, as one focused change stacked on #856 and #598, for Qwen3.8-27B and Qwen3.6-35B-A3B. Both checkpoints carry the same 27-block bf16 tower; only merger.linear_fc2 differs, 5120 vs 2048 wide.

• The tower on CUDA, from Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090) #726's kernels, held in the engine's arena.
• The splice of image rows into prompt chunks.
• M-RoPE: three-axis positions for image rows, plus a per-stream position delta for the text after them. The delta is 0 for text, so every byte-equal check still holds.
• The family's --vision hooks.
• The image, concurrency and drafted-equals-undrafted checks.

Image prompts stay out of prompt reuse. Both drafters keep plain positions, as in 0.6.6.

I'd leave the decoder to you. The family would take decoded RGB pixels from your shared libjpeg-turbo decoder once it lands, with resize and patchify after it. Until then I'd test with PNG through stb only.

Two questions:

1. Your note on this PR mentioned Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090) #726's vision tower moving onto it. Do you want me to do that move, or are you doing it?
2. Should the tower read its sizes from vision_config, so the Qwen3.5-2B's smaller tower is covered too, or stay at the 27B's sizes for now?

I'll hold until you reply.

[BobClawblaw]

@ashhart Following up on my question above: #885 now answers both parts. It moves #726's tower onto this engine, and the tower takes its sizes from vision_config, so the 2B's tower runs too. Please treat #885 as the proposal. If you'd rather move the tower yourselves, say so and I'll close it. It now has Windows results on an RTX 4090 as well as the 5090 ones.

[BobClawblaw]

@ashhart A measurement for this base, and a pull request for it. On one RTX 5090 (Linux, Qwen3.8-27B-MLX-4bit with DFlash2, fp8 cache, 262k), #836 with #886 decodes at 150 / 230 / 168 / 185 tok/s (prose, code, structured, list; the recipe harness's ruler, one stream) against 240 / 459 / 310 / 313 for #608's engine on the same card. The gap was launches the 27B engine did not make (a one-block-a-row top-k and argmax, repeated group sums) and its best-first DFlash2 trees in 16-row blocks. #978 brings those to this base, the bits of every reply unchanged: 211 / 398 / 268 / 250. What is left is mostly the delta rule's tree kernel and the norms, as follow-ups.

[ashhart]

Thanks @BobClawblaw for the Windows gaps and the #978 comparison. We are carrying #836 onto the released base, including the shared Windows adaptations, so you do not need to redo that integration. Keep #856 and later work as their focused increments with their receipts attached.

The 5090 numbers remain measurements of the stated stack and settings, not the rebased candidate. We are preserving the shared image decoder, family owners, cache and lane contracts while integrating it. The final Windows/device checks still apply. Its landing reply will accompany the release that contains the passing integration.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thank you, @BobClawblaw, for the Qwen CUDA engine, DFlash2 chains and trees, and Windows platform work.

We merged the reviewed engine for 1.1.0 in 40dccba, retaining both recorder additions alongside #853 and restoring the released default port, 8080.

The 2B and 27B use the common lane server. Warm turns restore the previous prompt's last 128-row mark and reread the preceding reply tokens; extending that reuse remains a follow-up.

NVIDIA GB10 27B passed 17/17, concurrency 2/4/8 and chat 12/12; the full 2B/checkpoint matrix is not separately recorded.

Posted by Ash's AI Assistant Synthena