[jayleaton]

What this changes

fabric: tensor parallelism across CUDA processes (bootstrap, NCCL, mailbox and RoCE collectives, plan link, fail-fast). Depends on #635. This cumulative PR targets main; the incremental range is c58d83ca..989ca7e6.

#635 → #636 → later DeepSeek-V4.1 layers

HCA selection accepts the documented device forms and rejects mixed modes. The host runner is executable. The final 48-case receipt passed against a fresh binary; this is modeled host coverage, with no hardware transport run.

Commits in this layer:

• 25c02d38 Tensor parallelism across CUDA processes: bootstrap, NCCL, mailbox and RoCE collectives, plan link, fail-fast.
• 6b5404dc TP tests: a drain on rank 0 then its clean stop ends rank 1's loop at the stop word with no fail-fast exit; a rank killed mid-drain still fails the other fast.
• 06c4e75f TP RoCE: with no TF_TP_ROCE_HCA, each rank picks the ports on an IPv4 /24 the peer also has
• 062af8b9 TP README: when -Dtp-nvcc, TF_TP_ROCE_HCA, NCCL_IB_HCA, NCCL_SOCKET_IFNAME and TF_TP_MASTER are needed, with a two-host example
• d7510ef8 TP test runner: a binary built without the mailbox kernel refuses the kernel backends up front; run_hosts.sh passes HCA picks
• e16e331a TP RoCE: every RDMA post is bounds-checked before the verb, and peers with another region layout are refused
• 649e7880 TP RoCE: no RDMA write past 1 MiB, and boot refuses any setting that would ask for one
• 664a8e2d TP RoCE: TF_TP_ROCE_CHUNKED_KB gathers strides past the slot in slot-sized ops, at most 2 MiB in flight
• 4ad50299 TP RoCE: a failed transport halts its QPs, drains them and keeps its region; the region is DONTFORK and NOHUGEPAGE
• 9bbc21ac TP RoCE: tf-tp-test failfast V transport checks the failed-transport path on real HCAs, only on an idle link
• cf19042f Fix RoCE test smaps lookup for merged mappings
• 989ca7e6 TP RoCE: run_hosts.sh gives NCCL the RoCE picker's ports when neither TF_TP_ROCE_HCA nor NCCL_IB_HCA is set

The combined correction removes personal MIT headers while retaining third-party notices, relocates development Python tools, and preserves stored fixtures with native consumers. Expanded provenance covers the EXL3 kernel stack; no copied text was found by the performed scan, and maintainer copy scanning remains authoritative.

Overlap with main

Main has a two-rank CUDA RoCE all-gather in zig/src/cuda/roce.zig (from 6636e696, after b12x's RoCEnante): its own
ibv_* loader at lines 90–125, Error at :147 and Gather at :212. This PR's equivalents are
zig/src/fabric/tp/roce.zig, roce_verbs.zig, mailbox.cu and roce_pick.zig. Its bootstrap, fate channel, planlink,
NCCL and hybrid transport have no counterpart in main. The two are kept side by side, and THIRD_PARTY_NOTICES.md
carries one b12x credit for each, naming its own files. Consolidation onto one loader is left to the maintainers'
common transport integration.

Receipt

Before: public head 41be68af. After: exact corrected head 989ca7e6578309da169d94f55b9ffd17541b1bf1, gated on 2026-10-12, Linux x86_64 and Zig 0.17.0.

Executed gate output:

G636: commit 989ca7e6578309da169d94f55b9ffd17541b1bf1 tree 710e15dbe42d122142772fe4b1a1ea1d863d24bf dirty 0 2026-10-12T04:35:00Z
install rc=0 Build Summary: 12/12 steps succeeded
test rc=0 Build Summary: 42/42 steps succeeded; 527/528 tests passed (1 skipped)
test-tp rc=0 Build Summary: 4/4 steps succeeded
tf-tp-test rc=0 Build Summary: 5/5 steps succeeded
zig fmt --check rc=0
git diff --check rc=0
989ca7e6 lean: 1206 files, 534 problems | new: 0 | gone: 0
FAILED STEPS 0

Lean uses pinned main e7c523eb: 534 inherited findings, zero new or removed identities. The mode gate covers all 57 introducing pins with zero executable-mode losses. Family suites and CLI receipts run where introduced. Host checks do not qualify checkpoint inference, GPU execution, concurrent/resumed serving parity, prompt/decode speed, or macOS/Metal behavior on this rewritten tree.

The family documentation pins development tools at recipe commit c1cb9184.

Exactness and prompt/decode speed: not run on this host, which has no GPU or checkpoint weights. No new performance claim. MLX/PyTorch/checkpoint revision: not applicable to the host gate.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. If the bits change, the description says which sums change and why.
• Prompt processing is no slower than the last release.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

Merge Danger

Door: two-way. Reverting restores the parent without a data migration.

Blast Radius: engine. The incremental commits define the affected runtime, build and test surfaces. The production branch is unchanged.

[plotarmordev]

Two-host receipt for this layer, which hadn't had a GPU run: two DGX Sparks (GB10, sm_121, CUDA 13.3, driver 580.173.02, NCCL 2.30.7 from the NGC PyTorch 26.07 container), joined by direct ConnectX-7 links, tf-tp-test built from db3bf32f with zig build tf-tp-test -Dtp-nvcc=… -Dsm=121, run_hosts.sh on both hosts (rank 0 the master), in the container with --network host --device /dev/infiniband --ulimit memlock=-1. Host tests on Linux aarch64: zig build test-tp 34/35 (1 skipped).

step	rank 0	rank 1
roce-host	PASS, 1,000 gathers 1 B–256 KiB bit-exact over verbs	PASS
suite roce	378 passed, 0 failed	378 passed, 0 failed
suite nccl	189 passed, 0 failed	189 passed, 0 failed
fail-fast, roce and nccl, fatal and kill on either rank (8 runs)	the injected rank exits 70 (fatal) or 137 (kill), the other exits 70 within 1.3–1.5 s, every run	same

A few latencies from the suite (rank 0, µs, eager / graph): RoCE all-reduce 4 KiB 8.7 / 8.1, 64 KiB 14.5 / 14.0, 4 MiB 304 / 360; all-gather 4 KiB 8.6 / 8.1, 4 MiB 256 / 290. NCCL on the same link: all-reduce 4 KiB 18.3 / 20.7, 64 KiB 58.4 / 45.6.

Three things I hit getting there, all setup rather than the collectives:

1. HCA auto-pick. Without TF_TP_ROCE_HCA, each host took its first two active ports (rocep1s0f0, rocep1s0f1), and on these Sparks only one of those reaches the other host; the second rail is a roceP2p1s0f* port. Result: connect failed: VerbsFailed / RoceUnavailable. Setting TF_TP_ROCE_HCA to the cabled ports on each host (they differ by host here) fixed it. Picking ports whose GID subnet reaches TF_TP_MASTER, or a line in the docs that cabled-to-several-neighbours Sparks need TF_TP_ROCE_HCA, would save the next person the hunt.
2. NCCL needs the same. With the RoCE ports right, both suites still failed in NCCL (ncclAllGather: unhandled system error, then a 900 s timeout) until NCCL_IB_HCA named the same ports. That's NCCL's own pick, but run_hosts.sh could pass NCCL_IB_HCA through the way it does NCCL_SOCKET_IFNAME.
3. -Dtp-nvcc. I first built with only -Dnvcc, which embeds an empty mailbox image; the RoCE device setup then fails at run time with NoKernel. Refusing to build tf-tp-test without the kernel, or saying so in zig build --help, would catch that earlier.

[plotarmordev]

@jayleaton, a follow-up, since GitHub now shows this as conflicting with main. The only conflict is the export and test lists in zig/src/cuda/root.zig: 15fb61eb added modelopt next to your memory, graph_cache and arena lines, and keeping both sides resolves it.

On the three things my receipt ran into, all of them setup rather than the layer:

• The RoCE HCA auto-pick took the wrong ports here. The receipt runs needed TF_TP_ROCE_HCA set.
• NCCL needed NCCL_IB_HCA set to the same ports.
• The build needed -Dtp-nvcc.

Exported variables already reach run_hosts.sh's ranks. Would it help if it wrote TF_TP_ROCE_HCA and NCCL_IB_HCA into host.txt, warned when ibv_devices lists more than one device and they're unset, and stopped at the start when the binary was built without -Dtp-nvcc, instead of reaching NoKernel? I can send that as a commit for your branch if you'd like it. Either way, I'm happy to re-run the two-host receipt on the rebased head.

[jayleaton]

Thank you for the two-host receipt; it is the first GPU run this layer has had. Thanks too for pinning down the setup problems.

Rebased onto current main (e7c523e). The root.zig conflict is resolved by keeping both sides: modelopt sits beside memory, graph_cache and arena, in the exports and in the test list. The only other conflict was build.zig, where main's grammar_build import sits next to tp_build. Head: db3bf32f → 151972b5.

HCA auto-pick: a real bug, fixed in a763a41. The pick took the first N active RoCE ports in device-list order, and nothing checked that the peer could reach them.

• Every rank now sends its ports' IPv4 addresses over the bootstrap. Each rank keeps the ports on a /24 that the peer also lists, so both sides pick the same number of rails.
• I matched against the peer's ports rather than TF_TP_MASTER's subnet, so that rails on separate /24s all count.
• With no shared /24 it falls back to the old order and warns you to set TF_TP_ROCE_HCA.
• A host test uses your topology. The old rule picks rocep1s0f0,rocep1s0f1; the new one picks rocep1s0f1,roceP2p1s0f1.

Your other two points, taken as you suggested in 151972b:

• run_hosts.sh exports TF_TP_ROCE_HCA and NCCL_IB_HCA. When NCCL_IB_HCA is unset, it is derived from TF_TP_ROCE_HCA.
• It warns when a host has more than one RDMA device and neither variable is set, and records all of them in host.txt.
• It stops before any step when the binary has no mailbox kernel. tf-tp-test refuses the mailbox and RoCE backends up front with a message naming -Dtp-nvcc.

I left zig build tf-tp-test buildable without the kernel, because the nccl and roce-host steps don't need it.

Docs: there is a new zig/src/fabric/tp/README.md (0d77854). It covers when -Dtp-nvcc, TF_TP_ROCE_HCA, NCCL_IB_HCA, NCCL_SOCKET_IFNAME and TF_TP_MASTER are needed. It also notes that :N means a GID index in TF_TP_ROCE_HCA but a port number in NCCL_IB_HCA.

Gates on the new head, host only (Linux x86_64, Zig 0.17.0): lean_check.py 534 problems, the same as main (0 new); zig fmt --check 0 files worse than main; zig build 12/12 steps; zig build test 42/42 steps, 510/511 tests (1 skipped: the hardware-only verbs test).

A re-run of your two-host receipt on this head, with TF_TP_ROCE_HCA unset, would be very welcome. That is the case the fix is for.

[jayleaton]

Added the production RoCE hardening as five commits (d71cee4b..06af238e: bounds-checked posts, a 1 MiB write cap on the roce backend, chunked gathers, halt and drain on a failed transport, run_transport.sh), each with host tests that fail without it; head 06af238e, gates in the body.

[jayleaton]

Fixed the flaky smaps test lookup in 41be68a: it now finds the mapping containing the address; the old test fails and the fixed TP suite passes 20 consecutive runs.

[cwschroeder]

Independent receipt for 41be68a on two ThinkStation PGX (GB10), one GPU per machine, aarch64 Linux, sm_121, over a direct ConnectX-7 link. Zig 0.17.0, driver 580.178.04, CUDA 13.0.2 (nvcc 13.0.88), NCCL 2.28.9+cuda13.0; built with -Dtp-nvcc. NCCL reports NET/IB RoCE and GDR 0.

zig build test: 524 passed, one opt-in local verbs test skipped per machine; fresh test-cuda-host: 347/347. Unmodified run_hosts.sh passes with explicit HCA selection and again with TF_TP_ROCE_HCA unset. In each configuration, roce-host passes 1,000 varied-size bit-exact host gathers per rank; the GPU suites pass 378/378 for suite roce (including 189 NCCL checks) and 189/189 for suite nccl. Both suites exit 0.

TensorFold auto-pick selects rocep1s0f1 and roceP2p1s0f1, GID 3, on both machines. NCCL HCA selection was configured explicitly in both runs.

All 16 fatal/kill cases across the two configurations pass. Fatal victims exit 70, killed victims 137, and every survivor exits 70 after 1.158 to 1.191 s from injection, rounded to milliseconds. run_transport.sh passes T0/T1/T3: the tested QPs are idle before injection and drained afterward, sampled HCA error counters remain unchanged, and no new kernel messages match the runner's filter.

Rank-0 microbenchmark latency in microseconds per call, explicit single HCA, eager / graph, measured with CUDA events:

operation	bytes/rank	RoCE route	NCCL
all-reduce	4 KiB	12.00 / 9.37	19.67 / 23.75
all-reduce	64 KiB	20.57 / 18.14	59.66 / 47.80
all-gather	4 KiB	12.06 / 9.26	14.42 / 22.44
all-gather	64 KiB	20.48 / 18.23	28.04 / 31.67

The RoCE route uses NCCL for fixed-length collectives above its slot size. These microbenchmarks do not establish model throughput or GPUDirect RDMA performance. The variable-length chunked-gather GPU path remains untested; the same-host/two-GPU mailbox is not applicable to this topology. No PR code changes.

[plotarmordev]

Re-ran the two-host receipt on 151972b5: the same two GB10 Sparks, direct ConnectX-7 links, tf-tp-test built with zig build tf-tp-test -Doptimize=ReleaseSafe -Dtp-nvcc=… -Dsm=121, and the branch's own run_hosts.sh on both hosts. Host tests on Linux aarch64: zig build test 510/511 (1 skipped), zig build test-tp 36/37 (1 skipped).

environment	roce-host	suite roce	suite nccl	fail-fast (8 runs)
neither TF_TP_ROCE_HCA nor NCCL_IB_HCA	PASS, both ranks	fails in NCCL (ncclAllGather: unhandled system error), then the 900 s timeout	fails (ncclAllReduce: unhandled system error)	NCCL errors and timeouts
NCCL_IB_HCA only	PASS	378 passed, 0 failed, both ranks	189 passed, 0 failed	every run: the injected rank exits 70 (fatal) or 137 (kill), the other exits 70 within 1.3–1.6 s
TF_TP_ROCE_HCA only (NCCL_IB_HCA derived)	PASS	378 / 0	189 / 0	same

The auto-pick fix works here. With TF_TP_ROCE_HCA unset, rank 0 took rocep1s0f0 and roceP2p1s0f0, and rank 1 took rocep1s0f1 and roceP2p1s0f1. Those are the ports I had named by hand, one on each of the two /24s the links use. The whole gate passed once NCCL alone was given its ports, and the new warning fired when neither variable was set.

What is left in the both-unset case is NCCL's own pick. suite always opens NCCL as its first backend (gpu_main.zig), so suite roce fails the same way suite nccl does. Before the error, NCCL logged ibv_query_port_speed failed … Protocol not supported. run_hosts.sh derives NCCL_IB_HCA only from TF_TP_ROCE_HCA. One option: tf-tp-test prints its auto-pick (say, a hca step), and run_hosts.sh sets NCCL_IB_HCA from it when neither variable is set. Happy to test that if you add it.

[jayleaton]

The HCA selection correction is now in #636 at 989ca7e, including the executable host runner. The fresh 48-case host receipt and TP tests pass; we have not run the transport on hardware. Could you retest your two-host case at this head and share the startup receipt?

[cwschroeder]

Thanks, @jayleaton. I can contribute another two-host PGX/GB10 receipt against 989ca7e once the pair is idle.

My earlier 41be68af receipt configured NCCL's HCA selection explicitly in both runs, including the run with TF_TP_ROCE_HCA unset. The both-unset arm adds coverage beyond that receipt. For the next receipt I propose three arms: neither variable set, NCCL_IB_HCA only, and TF_TP_ROCE_HCA only. I will record the actual selected ports, startup, suite results and exit status separately for each, using the branch's runner unchanged.

The new head's hardware transport is still unqualified on this pair. Failure-injection checks need a separate idle-link window.

[cwschroeder]

@jayleaton: the direct tf-tp-test hca check passes on another two-GB10 pair with both TF_TP_ROCE_HCA and NCCL_IB_HCA unset. Both ranks select the two active ports on the two shared IPv4 /24s and exit 0:

rank 0: rocep1s0f0:5,roceP2p1s0f0:5
rank 1: rocep1s0f0:5,roceP2p1s0f0:5
both stdout: NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0

This was an integration build of your 989ca7e6578309da169d94f55b9ffd17541b1bf1 with current main 0efbd44755b07b1bffb7a51c010992fe11a6142f, local merge 1f05a38d1ba407201b6c751fdc304173e44ac0e8. The sole merge conflict was the build imports; EXL3, EXLR8 and TP were all retained. No TP source changes.

Linux aarch64, Zig 0.17.0, CUDA compiler 13.0.88, NVIDIA driver 595.91.07. zig build test-tp -j2 --summary all: 52 passed, 1 skipped. ReleaseSafe tf-tp-test build with -Dtp-nvcc=/usr/local/cuda/bin/nvcc: all six build steps passed. Binary SHA256: ae7e335675d24e5c17df4e56eb19578b2861c4bf266cfe694b262d2cdcb8d858, verified on both ranks.

The direct check used a separate free bootstrap port, a 10-second plan timeout and a 25-second outer limit. Both test processes exited and released the listener. This receipt covers hardware port enumeration and peer selection only. The GPU/RDMA transport suites, runner handoff to NCCL and failure injection still need an idle window on the connected pair.

[jayleaton]

Thanks for the precise receipt. This adds hardware evidence for the both-unset port enumeration and peer-selection path on the integration build, including matching binaries and clean process exits. I will keep its scope separate from GPU/RDMA transport, the runner handoff to NCCL, and failure injection.

The stack is being reconciled onto 0efbd44755b07b1bffb7a51c010992fe11a6142f. I will post the resulting #636 commit and checks once the per-head gates and diff review pass, so subsequent transport receipts can name that exact head.

[Bizuayeu]

@jayleaton, two overlaps with #642 (NCCL send/recv and an exact reduce-scatter in rank order), so neither PR surprises the other.

1. zig/src/cuda/nccl.zig: both add ncclSend and ncclRecv to the same Api struct, with the same signatures. cuda: a reduce-scatter in rank order over NCCL send and receive #642 is the smaller change, with receipts from three GB10 setups, so I'd suggest it land first; this PR then drops those two lines when it rebases. ncclCommAbort and ncclCommGetAsyncError stay yours.

2. Rank-order sums: Collective.all_reduce promises a sum added in rank order, and the NCCL backend passes ncclAllReduce straight through. At two ranks that holds, since a + b is the same either way. At three or more, NCCL adds in its ring's order, so the bits can differ from a rank-order sum. That is the problem cuda: a reduce-scatter in rank order over NCCL send and receive #642 solves: RankScatter sends each rank its rows of every partial and adds the terms rank 0 first, and its test holds every rank's rows byte-equal to an all-gather plus a rank-order sum at TP=2 and TP=3. We'd like to keep that part on our side: the rank-order reduction lives in cuda: a reduce-scatter in rank order over NCCL send and receive #642, and the NCCL backend here can call it for world > 2 once both are in, or say in its doc comment that NCCL's order applies until then.

For the GLM-5.3-Flash family (#548), we plan to move onto #635's graph cache and this PR's bootstrap and fail-fast once they land, rather than keep our own.

[ashhart]

Hello @jayleaton,

Thank you for the CUDA tensor-parallelism host, the RoCE bounds checks, and the two-host receipts.

zig build test on a Mac stops in this module. zig/build/tp.zig:46 attaches the host tests to the default test step on every host. On macOS, std.c.pipe2 is an empty value, so fate.zig:56 does not compile. sock.zig:29 names posix.TCP.KEEPIDLE, and Zig 0.17's Darwin TCP struct has KEEPALIVE and no KEEPIDLE.

Please keep those tests off the Mac build, or give them a path that compiles there. Nemotron Metal, GLM, Flash Next, and the 27B all run zig build test, and they stop in fate.zig before their own tests.

A push that compiles on macOS is the update we need. Then it goes through the gate described on #635.

Thank you again.

[Bizuayeu]

Three sessions opening is useful evidence, but does not qualify three-rank collectives. [...] Our common transport integration still needs the exact merged-head failure and collective gates, followed by the consuming model's token checks.

(#636 (comment))

When the merged head you integrate is up, we can run on three GB10 hosts in a switchless ring, where each rank's address is a /32 routed over the links:

1. that head's three-rank collective gates and the failure cases you name, with the exact commands you want;
2. then GLM-5.3-Flash at TP=3 bootstrapped by it, in place of our NCCL-id file: the decode check's token hashes against our TP=3 references, drafted equal to serial, and a 1M-token prompt.

Tell us the head and the gate commands, and we will post the receipt here.

[ashhart]

Thanks @jayleaton. 1.1.1 ships the shared CUDA fabric, bootstrap, NCCL/mailbox/RoCE collectives, settings agreement and peer-failure handling. The later 88418d8/4c69b0d4b command-preflight work and 4df28ce/0a80da001 verbs/HCA diagnostics remain to carry into 1.1.2 with your credit, followed by the integrated failure controls. Keeping this PR open for those follow-ups; no TP3/4 model-support claim follows from the infrastructure landing.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thank you for this. We compared #636, and #637 on top of it, against the 1.1.2 line by their net change against 1.1.0. Most of the transport, cache and arena code and the rank paths are already there, often byte for byte, and carrying the whole PR would duplicate build wiring and CUDA declarations the line already has. So we're closing this one as superseded.

Three pieces are worth having on their own, and we'd take them as small PRs on top of 1.1.2, credited to you:

• the command parser that parses before taking any resource, with its invalid-command controls (88418d8 and 4c69b0d);
• the verbs-library error retention and diagnostics (4df28ce and 0a80da0);
• the generic-fabric attribution paragraph in the docs.

- Synthema, Ash's AI Assistant.