[Bizuayeu]

What this changes

While GLM-5.3-Flash's CUDA ranks have no stream, the ranks other than 0 sleep instead of spinning, the direction in #1033 (comment). Rank 0 sends an idle step when its last stream is released, and a wake record on the ranks' liveness lines (cuda/tp_watch.zig) before its next step; a waiting rank that reads the idle step sleeps on a futex until that wake and only then posts its receive. The ranks start asleep, so the first request wakes them. Rounds are unchanged. A rank that fails or dies while the others sleep still stops them through the same lines, and a stop while they sleep wakes them for the end step.

The idle step goes out from the GLM lanes' release when no lane is left, so a request already queued behind the last one still costs one idle step and one wake (a TCP write and one NCCL step). A hook before the lane host's wait would avoid that, but it touches the lane core; the step can move there if you prefer.

Receipt

• Environment: 7bfd9798 on d1018b31 (main: 1.1.1 and five commits that change no Zig file), Zig 0.17.0, CUDA 13.3 (nvcc 13.3.73), NCCL 2.30.7, driver 580.178.04; two DGX Spark (GB10, sm_121) hosts over a direct ConnectX-7 link, one GPU each; checkpoint nvidia/GLM-5.3-Flash-NVFP4 at 423acf37; TP=2, serve per rank with TF_GLM_RANK, TF_GLM_WORLD and NCCL_COMM_ID, drafts on, TF_HEAT_HIGH=92 and TF_HEAT_LOW=88. Before is 1.1.1 (b10231b9), built the same way. Two settings:
  • the idle windows, exactness, decode and the forty requests: --context 245760 --parallel 1, every core (no pinning);
  • the template's tools: --context 131072 --parallel 2, every core, except tools/bench_concurrent.py, which ran on the efficiency cores (0-4, 10-14) for both builds, as in the first receipt.
• Idle, the engine's threads above 5% CPU and the hottest ACPI zone's mean over the last 60 s of each window:

	Before	After
120 s after the start, no request yet: rank 1's host	two threads at about 100% (one user time, one about three quarters system time); 67.8 °C	no thread above 5%; 50.9 °C
120 s after the start: rank 0's host	no thread above 5%; 51.9 °C	no thread above 5%; 51.8 °C
600 s after requests: rank 1's host	the same two threads at about 100%; 66.8 °C	no thread above 5%; 48.9 °C
600 s after requests: rank 0's host	no thread above 5%; 50.0 °C	no thread above 5%; 49.8 °C

• Exactness: a counting, a prose and a code prompt, three greedy replies each, give one token_sha per prompt, the same before and after; the second and third replies resume from the prompt cache, so resumed equals fresh. Drafted equals "draft": false on three prompts (whole messages), before and after. With --parallel 2: two requests at once (256 and 96 tokens), the shorter one released while the other runs, each equal to itself alone after the ranks slept; tools/bench_cache.py's five rows, two of them concurrent, give one token_sha before and after; tools/bench_concurrent.py --levels 2 --alone --serial (efficiency cores) finds every pair equal to alone (24 of 24) and to serial (6 of 6) in its four cells, none failed, before and after.
• Decode speed, tools/bench_openai.py --tokens 512 --reps 5 (--parallel 2), median tok/s before → after (at temperature 1.0 the five reps spread widely, 32.8 to 39.4 tok/s on fibonacci-raw, and follow each other within 0.2 tok/s from before to after):

	Temperature 1.0	Temperature 0
fibonacci-raw	38.39 → 38.44	64.28 → 64.23
gpu-chat-no-think	29.66 → 29.57	32.95 → 32.88

Two streams at once, bench_concurrent.py's aggregate tok/s before → after: code 44.2 → 44.3 at temperature 1.0 and 62.3 → 62.5 at 0, chat 39.4 → 39.6 and 43.1 → 43.1. At --parallel 1, 512 tokens after a short prompt, median of three: 43.67 → 43.78.

• Prompt speed: the cold first request's ttft_s from tools/bench_cache.py --require-cold (2,570 prompt tokens; its first step after idle carries the wake) is 1.820 s before and 1.785 s after, and bench_openai's ttft_s medians differ by under 1 ms. At --parallel 1, the forty requests below have a median time to first token of 4.700 s before and 4.675 s after (about 5,900 prompt tokens, no heat wait in either).
• Heat, forty requests back to back (about 6,000 prompt tokens each, replies of up to 1,024 tokens, temperature 0) with a watch that stops the engine at two readings at or above 94 °C: 40 of 40 before and after, in 1,540 s and 1,533 s, rank 0's host peaking at 92.6 °C before and 92.8 °C after. While a request runs nothing changes, so rank 0's host heats as before; this change is about the time between requests.
• Tests run: zig build test-cuda-host 2178/2184 with 6 skipped (the new line test passes), zig build and zig build native, zig fmt --check and zig ast-check on the changed files, tools/zig/lean_check.py 449 problems, the same lines as on the base.
• Not run, and why: Metal (the change is the CUDA side's), TP=3 (our third GB10 is on per-host /32 links that NCCL_COMM_ID does not bootstrap, fabric: tensor parallelism across CUDA processes (bootstrap, NCCL, mailbox and RoCE collectives, plan link, fail-fast) (DeepSeek-V4.1 stack 3/14) #636), other CUDA GPUs, and bench_concurrent.py on every core for both builds.

The first receipt, on 1.1.0 (7a359838 on b7ce8121), was this description's earlier version; the rebase onto 1.1.1 kept the change to the same four files (+83/−4) and only met 1.1.1's new follow kinds in cuda_follow.zig.

Checklist

• ✓ Every token still goes through the lane rounds (the idle step is sent between them; no serial path; drafts stay on).
• ✓ Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (above).
• ✓ No precision traded for speed: no arithmetic changes.
• ✓ No new or changed GPU kernel.
• ✓ Prompt processing is no slower than the last release (the forty prompts' median time to first token 4.675 s against 4.700 s; the cold bench_cache.py request 1.785 s against 1.820 s; above).
• ✓ Platforms: CUDA GB10 at TP=2; Metal, TP=3 and other CUDA GPUs not run.
• ✓ New tests: the line test (a wake that came before the sleep is kept, a later one ends it) cannot build before the change and passes after; it skips on Windows like the file's other line tests.
• ✓ Comments and docstrings are one line; no measurements or history in the source.
• ✓ No personal data, machine names, internal hosts or local paths; no AI attribution lines.

[ashhart]

Thanks @Bizuayeu, this follows the idle-wait direction we asked for, and the A/B results help. Source review found no production blocker: early wakes are counted, idle stays between rounds, and shutdown wakes the follower for its end step. We'll carry this onto 1.1.2, preserving stepped admission, then gate decode equality/speed, idle CPU and TTFT on the combined tree. In our carry we'll also bound the new tp_watch test's waits and stop its watcher on every failure path, so a broken wake fails the test instead of hanging it. We'll include cancellation and idle shutdown in that gate.

Posted by Ash's AI Assistant Synthena

[Bizuayeu]

We'll carry this onto 1.1.2, preserving stepped admission, then gate decode equality/speed, idle CPU and TTFT on the combined tree.

(comment)

Thanks. In case it helps the carry, the branch is now on main (d1018b31, 1.1.1) as 7bfd9798: the same four files, with 1.1.1's many and redraw follow kinds kept beside idle in cuda_follow.zig. The description's receipt now has the same A/B on 1.1.1: between requests rank 1's host stays at 48.9 to 50.9 °C with no thread above 5%, against two spinning threads at 66.8 to 67.8 °C before; the token hashes are the same, and decode and time to first token differ by no more than the runs do. We left the tp_watch test's waits as they were, for your carry to bound.

[ashhart]

The GLM CUDA waiting-rank change is included in 1.1.2 (https://github.com/ashhart/TensorFold/releases/tag/v1.1.2), with your original authorship retained. Waiting ranks sleep on their watch lines while rank 0 has no active stream and wake for new work. Our follow-up, 3589cd0, sends the prompt cache's setup step through rank 0, so ranks that start asleep wake for it at TP=2 and above. Closing this as carried, and thank you.

It landed as 86a34f3.

- Synthema, Ash's AI Assistant.

[Bizuayeu]

Waiting ranks sleep on their watch lines while rank 0 has no active stream and wake for new work. Our follow-up, 3589cd0, sends the prompt cache's setup step through rank 0, so ranks that start asleep wake for it at TP=2 and above.

#1053 (comment)

Thank you for carrying it, and for 3589cd0. We retested 1.1.2 at TP=2 and TP=3 in #1033 (comment): idle ranks stay quiet, and requests that resume from the prompt cache while the other ranks sleep are answered, at TP=2 with v1.1.1's tokens.