Repository navigation
glm: TF_GLM_PREFILL_REDUCE=ring, for fabrics where a rank reaches only its neighbours - #902
Conversation
|
Thank you, @Bizuayeu, for relaying the original rank blocks into their existing slots, so the ring keeps split's reduction order. At 691cfd7, the routing and room offsets look consistent, including unequal row shares. One error path remains: Please add shared group cleanup and rank-wide failure handling, with a failure injected at each posting. We can fold that small fix in at landing. The relay planner should also move into shared core when DeepSeek uses it, preserving GLM's existing glue arithmetic. We'd compare ring against split on real two-rank and larger advertised configurations, with uneven pieces and prompt cuts, then check resumed/fresh and concurrent/solo tokens and a peer failing during an exchange. Posted by Ash's AI Assistant Synthena |
|
@ashhart, thank you. Pushed
Re-run at |
…ng, for a fabric where a rank reaches only its neighbours
…ay out of it; the ring relay posts through it, and a fake NCCL fails each posting in turn
ea26c0b to
98a5dc6
Compare
|
Rebased onto main as |
…ing start on a neighbours-only fabric loads Settings.read sets split and ring for TF_GLM_PREFILL_REDUCE=ring, and the engine's start guarded #899's probe with split alone, so a ring start still exchanged a value with every rank before loading: a ring of four without a switch, whose ranks reach only their neighbours, could not start in the mode #902 built for it (two ranks cannot tell the two apart). probeStart probes for split without ring, and the refusal now names ring or gather. Host tests over a fake fabric: a direct split start is refused at every rank of a four-rank ring and starts behind a switch and on two ranks; ring and gather starts on the four-rank ring post nothing before loading. Co-authored-by: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
…ollow-ups named as ours After review of the fourteen lines that overstated: #710's fresh-quantization gain is one measured case; #902's stop and group cleanup hold where the watch lines are open and through the shared helper; #887 says its chunks change the attention bits below 8,192, on M5 too, with no larger error against float64; #899 records the ring start and the wider refusal; #802 keeps the author's B70 receipts apart from our host-checked lifetime fixes; #836's equalities are the GB10's and its Windows build the author's; #888 says a failed restore's pinned memory is kept, never freed; #734's tree rounds are checked against the host's, not float64; the Living Weights lines (#876, #877, #878, #891) say which runs their numbers come from, the ten-fact costs, and the M3 Ultra's four-fact result; #946 and #910 read plainer.
|
Thank you, @Bizuayeu, for the ring relay and per-posting failure controls. We merged the work for 1.1.0 in 08c2e69. Our follow-up shares the relay plan, routes split/scatter/broadcast postings through one group helper and connects NCCL errors to the rank watch. Active watch lines notify peers before a local fatal exit; returning failures still end their group. The relay retains the rank-slot sum order, and ring startup skips direct split's all-peer probe. Rank-watch controls and two-GB10 EXL3 serving passed; four-rank relay and live killed-peer comparisons are not recorded. Posted by Ash's AI Assistant Synthena |
The live killed-peer comparison with |
…s along the ring, each step's postings in one group that a failed post still ends
…ing start on a neighbours-only fabric loads Settings.read sets split and ring for TF_GLM_PREFILL_REDUCE=ring, and the engine's start guarded ashhart#899's probe with split alone, so a ring start still exchanged a value with every rank before loading: a ring of four without a switch, whose ranks reach only their neighbours, could not start in the mode ashhart#902 built for it (two ranks cannot tell the two apart). probeStart probes for split without ring, and the refusal now names ring or gather. Host tests over a fake fabric: a direct split start is refused at every rank of a four-rank ring and starts behind a switch and on two ranks; ring and gather starts on the four-rank ring post nothing before loading. Co-authored-by: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
…ollow-ups named as ours After review of the fourteen lines that overstated: ashhart#710's fresh-quantization gain is one measured case; ashhart#902's stop and group cleanup hold where the watch lines are open and through the shared helper; ashhart#887 says its chunks change the attention bits below 8,192, on M5 too, with no larger error against float64; ashhart#899 records the ring start and the wider refusal; ashhart#802 keeps the author's B70 receipts apart from our host-checked lifetime fixes; ashhart#836's equalities are the GB10's and its Windows build the author's; ashhart#888 says a failed restore's pinned memory is kept, never freed; ashhart#734's tree rounds are checked against the host's, not float64; the Living Weights lines (ashhart#876, ashhart#877, ashhart#878, ashhart#891) say which runs their numbers come from, the ten-fact costs, and the M3 Ultra's four-fact result; ashhart#946 and ashhart#910 read plainer.
The second follow-up promised on #804 (comment), as two commits on main:
3b0093a3, and98a5dc65from review (a failed post still ends its NCCL group).@SvangenStudios offered to run it on the ring of four (comment).
What this changes
TF_GLM_PREFILL_REDUCE=ringis split with its exchanges passed along the ring, for a fabric where a rank reachesonly its two neighbours. A piece's partials take world - 1 steps: at each, a rank sends its next neighbour the blocks
still on their way and keeps the one meant for it in the slot of its room that split's direct receive fills, so the
owner sums every rank's block in rank order, as with split: the same bytes in the same order, and the same bits. The
glued rows go back as a ring all-gather, each step's rows straight into place. Blocks in transit sit in a piece's room
past split's slots (split uses about 1/world of it), so ring takes no new memory.
cuda_relay.zigholds the scheduleand the two exchanges; the fingerprint's
splitfield is 2 for ring, so every rank must pick it. Each step's postingsgo through
Library.group(cuda/nccl.zig), which ends the group when a post fails, so a failed post never leaveslater NCCL calls inside it.
With #899 also in, ring should skip that probe, which tries every pair; whichever of the two lands second makes the
probe run for split only and names ring in its refusal.
Receipt
46a08c18(TensorFold 1.0.12) plus this change; three DGX Spark (GB10, sm_121, CUDA 13.3)over RoCE, where at TP=3 every pair of ranks is linked; nvidia/GLM-5.3-Flash-NVFP4 at revision
423acf37.splitand then withring(each rank's log naming its setting),and every solo run's token ids, greedy and sampled, hashed the same under both, byte for byte (a test-only print, not
in this branch), and the same as before this branch moved onto main. The shared check passes with ring. Split, the
default, is unchanged.
linked, so it only adds hops).
98a5dc65: on one hostzig build test-cuda-host855 passed (852 at main),zig build test1269passed and 7 skipped, with three new tests (the relay simulated for 2 to 6 ranks: every owner gets every rank's
block in split's slot with the transits inside the room, and the glued rows reach every rank; and both exchanges
against a fake NCCL failing each call in turn, every rank of worlds 2 to 6: the error returns with the groups
balanced and nothing posted after it),
zig fmt --checkclean,glm-refand the thin and synthetic lanes checks;at TP=3 the concurrent check with split and with ring (41 PASS each) and the shared check with ring; at TP=2 the
concurrent check with split and with ring (39 PASS each, the same ids: at two ranks ring posts what split posts);
each rank's log naming its setting; 0 failed.
lean_check: 516, as main.gatheron the same build); a peer lost during an exchange (ending the other ranks then is fabric: tensor parallelism across CUDA processes (bootstrap, NCCL, mailbox and RoCE collectives, plan link, fail-fast) (DeepSeek-V4.1 stack 3/14) #636's fate channel).Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (the lanes checks above, with ring at TP=3).