Skip to content

Harden libsircl lifecycle, bootstrap and input validation - #382

Merged
FujitsuPolycom merged 67 commits into
mainfrom
libsircl-hardening
Nov 2, 2026
Merged

FujitsuPolycom merged 67 commits into
mainfrom
libsircl-hardening

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

Reliability hardening of libsircl (spark_transport/libsircl), SIRCL behind NCCL's C API, from a verified audit:

  • 188 audit rows were checked against 68a99512: 38 confirmed, 100 partial, 14 false and 36 informational.
  • 49 of the 50 resulting fixes are implemented: one commit each, each with a regression test that failed before the fix where a CPU reproducer exists.

Main fixes:

  • Validated records: split records are length-checked and peer positions range-checked before they are used.
  • Device selection: ncclCommInitAll refuses device selections it cannot honor.
  • Abort: it no longer waits behind an in-flight split or bootstrap work.
  • Splits: a failed split leader fails its peers promptly.
  • Large all-reduces: the 32-bit pack count overflow at ≥32 GiB is fixed.
  • Teardown: submission closes before context waits; quarantined engines are reclaimed after their kernels finish.
  • Leaks: resource leaks on failed creation and unload are fixed.
  • Settings: invalid settings and over-full relay route plans are refused at setup.
  • Docs: documentation consistency checks are added.

Unchanged: VERSION (0.6.0), the GPU kernels and the vendored proxy sources. The kernel pack hashes are identical at base and head, so existing hardware evidence still applies.

Compatibility

  • Return codes: destroying or aborting a NULL communicator returns success. A split timeout returns ncclRemoteError. Lifecycle calls on a communicator still initializing return ncclInvalidUsage.
  • Refusals: the one-pass ring relay needs an opt-in variable. Strictly invalid settings and over-full relay route plans are refused.
  • New variable: LIBSIRCL_RELAYED_LANES.
  • Setup agreement: ranks must agree on fail-stop and the wait regime.
  • Groups: failed calls that queue nothing no longer fail the group.
  • Build: hardening flags are on in both builds, and Makefile builds at -O0 fail.

Also changed (found in a second review; read from the code, not measured):

  • Finalize and destroy wait for the whole context. Non-abort ncclCommFinalize/ncclCommDestroy call CtxSynchronize, so they also wait for other communicators' work on the same device and can stall for another communicator's wait limit.
  • ncclCommInitAll with devlist == NULL refuses ndev > 1 outside bootstrap-only mode, and refuses ndev == 1 unless the current device is 0. Duplicate device ids are still accepted.
  • Mixed versions: shared_settings now carries fail-stop, the wait regime and the ring reduce passes, compared across ranks, so a rolling upgrade against 0.6.0 ranks fails setup.
  • A failed bootstrap all-gather is sticky: every later round on that handle fails at once, so a parent communicator that survived a failed split returns ncclRemoteError from ncclCommDestroy.
  • Destroying a revoked communicator runs the teardown rounds; after its peers have gone it blocks for the wait limit plus 5 s, then returns ncclRemoteError.
  • Fail-stop: a backstop thread calls _exit 1 s after fail-stop begins, also with LIBSIRCL_FAIL_STOP=abort, so a slow abort path ends without a core dump or SIGABRT handler.
  • More refusals: forward windows outside 0 or 1-60 chunks and below 32 KiB (previously clamped), LIBSIRCL_RING_WINDOW outside 1-60, SIRCL_TRAFFIC_CLASS outside 0-255 (previously 0), an invalid LIBSIRCL_RECEIPT_INTERVAL_S, list variables of 1024 bytes or more, and point-to-point calls in one group on one communicator using more than one stream or more than 1024 calls. Exclude-only interface lists such as ^docker no longer select loopback.
  • Outside libsircl: integrations/vllm/libsircl/sparkring_libsircl.py refuses symlinked path components, a replaced inode, vLLM without its PyNccl wrapper, and a VLLM_NCCL_SO_PATH that differs from the selected library.

Not done

  • 40b: a memory-footprint ceiling; it needs a policy and a measured default.
  • 10: asynchronous finalize; documented as blocking.
  • No CPU reproducer: 12, 14, 19, 31 and 33/34.
  • 43: the CMake changes, which run only in CI.

Validation

  • make check, make check-asan and check_exports pass on a clean build. Test counts rose: lifecycle 14→23, engine 12→24, plus new C checks for the link plan (15) and the kernel-pack cache (69).
  • Plugin and runtime libsircl tests: 80 passed, 2 skipped. ruff is clean, and the Markdown links check passes.
  • Fix added in review (598a9c2): sccl_engine_destroy destroyed a quarantined engine's lock after putting it on the quarantine list, where another thread's reclaim could already have freed it; the lock now goes first.
  • Rechecked 2026-10-31 on this branch merged with main (9932204): make check, check_exports.py and make check-asan pass (CPU build, nvcc 13.3 from pip).
  • GPU-only cases ran on a DGX Spark (spark-0a0f), 2026-11-01: pass. The host was idle before and after (GB10, driver 580.178.04, 0 compute apps); commit 22378d28 was built on it with nvcc 13.0 for sm_120/sm_121 (library sha256 47f7d4f7c37aa4ca7ca05e18d7280d27eabe83fb6c5bf0c65a3530985a491a8d), and the runners ran inside image f625053225d4 with LD_PRELOAD emptied so libsircl maps instead of NVIDIA NCCL (verified ncclGetVersion 22705 vs 23203).
    • teardown_race 4/4: concurrent-close 0 rounds failed, 0 problems (415 s), revoke-destroy 0 rounds failed, 0 problems (33 s), close-error late 0 rounds failed, 0 problems (37 s), close-error dereg 0 rounds failed, 0 problems (25 s).
    • p2p_channels: abort 9 checks, 0 failed, gone 4 ranks, 0 failed with exits 70/70/70/0 and the fail-stop line naming the channels.
    • setup_failures: 9 cases, 0 failed, including both agreement cases (LIBSIRCL_FAIL_STOP, LIBSIRCL_WAIT_REGIME).
    • run_library --world 2 --golden w2: 535 checks over 2 ranks, 0 failed, 87 byte-equality lines, 16 receipts.
    • 783 s of scenarios plus a 54 s build; logs under /tmp/pr382-qual-22378d28/.
  • Still unverified: the GPU-only cases ran on the shared-memory emulation transport, not RoCE over the cabled fabric; the CMake build path runs only in CI.

Status: research-only, like libsircl itself. The GPU-only cases ran on GB10; the PR is ready for review.

🤖 Generated with Claude Code

FujitsuPolycom and others added 30 commits October 14, 2026 23:10
Both split rounds validate every contribution before fixed-stride reads or unique-id copies.

Compatibility: Malformed peer records return ncclRemoteError; valid splits retain their wire format.

Validation: Both short-frame regressions failed before the checks; all 15 lifecycle CPU tests pass.

Request item: 1. NVIDIA NCCL source was neither read nor adapted.
Abort publishes cancellation before waiting for acquired calls; teardown rounds observe the same flag.

Compatibility: Abort interrupts split promptly; normal destroy retains its orderly teardown rounds.

Validation: The split/abort test failed at 1.95 seconds before the fix; all 16 lifecycle CPU tests pass.

Request item: 6. NVIDIA NCCL source was neither read nor adapted.
Setup reports invalid positions through the common local verdict before route indexing or shifts.

Compatibility: Invalid positions fail setup; valid position records retain their layout.

Validation: The CPU socket-peer test failed for -1, 64 and INT_MAX before the check; all 13 engine CPU tests pass (one host-dependent skip).

Request item: 2. NVIDIA NCCL source was neither read nor adapted.
Distinct GPU devices are rejected before handles are allocated, and a single selected device must match the current context.

Compatibility: Bootstrap-only logical device lists remain supported; unsupported GPU selections return ncclInvalidUsage.

Validation: The two-device regression failed before the fix; 14 engine CPU and 16 lifecycle tests pass (one host-dependent skip).

Request item: 3. NVIDIA NCCL source was neither read nor adapted.
The engine closes under its submission lock, clears failed stream ordering, and synchronizes the creating context at finalize and every non-abort destroy, including after transport stop.

Compatibility: Orderly teardown waits for all context work, including graph replay streams; failed waits preserve the arena.

Validation: Context-sync regression failed before the fix; 15 engine and 16 lifecycle CPU tests pass. teardown_race.py --expect concurrent-close is written but not run.

Request item: 4. NVIDIA NCCL source was neither read nor adapted.
`make check-asan` builds the host library with AddressSanitizer and
UndefinedBehaviorSanitizer into $(BUILD)-asan and runs every `make check`
suite with the ASan runtime preloaded. The sanitizer flags reach the
sub-make through the environment, so the Makefile still appends its
required flags. `make check` runs every suite and fails at the end when
any failed, so each suite's result is reported even when another fails.

Test harness changes needed for the sanitizer run: the ABI preload test
appends the library to an existing LD_PRELOAD instead of replacing it; the
engine CPU checks keep the ASan preload in children that set or clear
LD_PRELOAD and locate the stand-in driver next to the library under test;
ncclCommSplit gets its ctypes signature in test_api.initialize_library, so
a heap address above 4 GiB (as under ASan) is not truncated to 32 bits.

The CI job "libsircl C library" also runs tools/check_exports.py against
the built library and `make check-asan`. STATUS.md states the sanitizer
evidence as the target's result instead of fixed test counts.

Compatibility: no library code changes.

Validation: make check (12 Python suites and the fabric vectors) passes;
tools/check_exports.py passes; make check-asan passes every suite; with a
use-after-free injected into ncclCommDestroy (reverted before this commit)
make check-asan fails with AddressSanitizer's heap-use-after-free report.

Request item 41 (audit 11-6, 12-F3, 12-F5, 12-F9 items 2, 4 and 5).
NVIDIA NCCL source was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The library destructor (run by dlclose or by exit) cancels every
communicator and then waits up to 1 s for calls that still hold a
reference to leave before it frees them. Cancellation ends a bootstrap
wait within one 50 ms poll slice, so a thread blocked in ncclCommSplit
returns ncclSystemError instead of reading the freed communicator or
returning into unmapped code. A communicator still referenced after the
bound stays allocated with its bootstrap open. ncclCommSplit records its
failure text before dropping the parent's reference, so the unload finds
the call past its library work.

Compatibility: a process unloading libsircl while another thread is
inside a call can take up to 1 s longer to unload; communicators still in
use at the bound are leaked instead of freed. dlclose concurrent with API
calls stays outside the supported caller contract.

Validation: tests/test_lifecycle.py's unload test (a split blocked on a
stand-in broker, then dlclose or exit; in the exit case a helper library
finalized after libsircl keeps the process alive 0.3 s) failed before the
change: dlclose ended in SIGSEGV in the plain build, and under
`make check-asan` both cases failed (a deadly signal for dlclose,
an AddressSanitizer error report for exit). After the change, make check and
make check-asan pass every suite.

Request item 5 (audit 02-F16). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A split leader whose ncclGetUniqueId fails still takes part in the
unique-id all-gather. It sends a failure record (no id magic; bytes 8-11
carry its result) instead of returning before the round, so every member
of its color returns the same result at once, with a reason naming the
leader, and every rank's bootstrap round count stays equal. The leader's
own failure goes through creation_failed like every other split failure.

A bootstrap handle whose all-gather failed after it began sending is
marked failed, and every later all-gather on it fails at once naming
that cause: its round count can differ from the broker's, and a later
round must not be matched with another round's records.

Compatibility: the wire format of a successful split is unchanged. A
member of a failed leader returns the leader's result immediately instead
of a bootstrap timeout (ncclSystemError) after LIBSIRCL_SETUP_TIMEOUT_MS.
Any all-gather on a parent bootstrap after a failed one fails with
ncclRemoteError instead of waiting or desynchronizing.

Validation: tests/test_lifecycle.py holds all 64 bootstrap roots so the
leader's id creation fails. Before the change the two ranks returned
ncclInvalidUsage and ncclSystemError after 5.1 s; after it both return
ncclInvalidUsage within 0.3 s and a following NOCOLOR split on both
ranks succeeds. make check and make check-asan pass.

Request item 7 (audit 03-F9, 02-F4, 01-F12.5). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ncclCommDestroy and ncclCommFinalize of a revoked communicator of two or
more ranks run the same two bootstrap rounds as for a ready one before
any queue pair, registration or arena is freed. Revoke refuses new work
but does not stop kernels and forwards already in flight, so freeing
without the rounds can remove regions a peer still writes; STATUS.md's
teardown evidence records that skipping the rounds fails 7 of 10 rounds
on the workstation and 98 of 100 on a GB10.

README.md and STATUS.md name the revoked case in the teardown contract.
tests/emulation/teardown_race.py gains --expect revoke-destroy (revoke
each child right after its all-reduce, then destroy it) and documents
--expect concurrent-close.

Compatibility: destroy after revoke now waits for every rank, bounded by
the wait limit per round, and returns ncclRemoteError when a rank does
not arrive; abort remains the non-waiting path.

Validation: the changed path needs a ready engine of two or more ranks,
which the CPU harness cannot build. Written, not run:
  python tests/emulation/teardown_race.py --library build/libsircl.so \
      --expect revoke-destroy
make check and make check-asan pass.

Request item 8 (audit 02-F5, revoke part of 01-F7). NVIDIA NCCL source
was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ncclCommRevoke and ncclCommFinalize on a communicator whose
initialization is still in progress return ncclInvalidUsage, naming the
reason, as an enqueue on it already does. ncclInProgress never becomes a
group's result: ncclGroupEnd records the group's result as the terminal
error of its pending initializations, and a terminal ncclInProgress
reads as "still initializing" forever.

Compatibility: these calls returned ncclInProgress before; callers that
poll a non-blocking communicator with ncclCommGetAsyncError are
unaffected.

Validation: tests/test_lifecycle.py calls revoke and finalize on a
non-blocking communicator still initializing, inside a group that also
initializes a one-rank communicator. Before the change revoke returned
ncclInProgress; after it both calls return ncclInvalidUsage, the group's
end does not return ncclInProgress, and the grouped communicator's error
is not ncclInProgress. make check and make check-asan pass.

Request item 9 (audit 03-F13d). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…icators

README.md's teardown contract states that ncclCommFinalize of a
non-blocking communicator (config `blocking` 0) runs both teardown rounds
before it returns and never returns ncclInProgress, and that a caller
that must not block runs finalize on its own thread.

Returning ncclInProgress would need a finalize worker whose lifetime
interacts with abort, destroy, the unload destructor and the error word
that the teardown round itself reports; that path is reachable only with
a ready engine of two or more ranks, which no available test can run, so
the behavior is documented instead of changed.

Compatibility: documentation only.

Validation: no code changed; the Markdown link check passes.

Request item 10 (audit 01-F7, finalize part). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ncclCommAbort on a communicator with point-to-point channels sets the
channels' host-mapped poison word before the idle check. Channel kernels
return once they see it, and SIRCL's progress thread takes its failure
path, which writes an abort notice into every peer's abort word. The
engine waits up to 50 ms for the progress thread to report the failure
and then stops it; stopping joins the thread after the notices are
posted. Peers report the channels' failure naming this rank instead of
waiting out their wait limit, which is 600 s in the startup regime.

Collective kernels are unaffected: stopping them needs an abort word in
SIRCL's collective kernels, which is a kernel change outside this
library's scope.

Compatibility: abort of a communicator without channels is unchanged.
Abort of one with channels takes up to 50 ms longer and makes every
peer's channels fail with ncclRemoteError naming the aborting rank.

Validation: the path needs live channels, which the CPU harness cannot
build. Written, not run:
  python tests/emulation/p2p_channels.py --library build/libsircl.so \
      --case abort
make check and make check-asan pass.

Request item 11 (audit 05-F2, notification part of 04-F5). NVIDIA NCCL
source was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fail_stop poisons a communicator's point-to-point channels, waits up to
50 ms for SIRCL's progress thread to report the failure, and stops the
thread, which joins it after post_aborts wrote the abort notice into
every peer's abort word. Only then does it write its line and receipt
and end the process. When the failure came from the channels, the thread
is already on its failure path, and the stop waits for the notices it is
posting; when it came from a collective, the peers' channels learn of it
too.

SIRCL's progress thread stores `failed` and the poison word before it
posts the notices (src/transport/sircl_p2p_proxy.c), so without the wait
the watcher or a library call could end the process between the two. The
vendored proxy is unchanged.

Compatibility: fail-stop of a communicator with channels takes up to
about 52 ms longer before the process exits. Communicators without
channels, or whose progress thread already stopped at teardown, are
unchanged.

Validation: the window is microseconds against the watcher's 5 ms poll,
so no available test reproduces the race. The emulation case that checks
the notices reach the peers is
  python tests/emulation/p2p_channels.py --library build/libsircl.so \
      --case gone
(not run here; ranks 1 and 2 must end with status 70 through the notice).
make check and make check-asan pass.

Request item 12 (audit 04-F2). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chain, link and point-to-point kernels take pack counts (16-byte
units) as int32_t, so an op of 32 GiB or more wrapped its count. A large
all-reduce or reduce whose body exceeds the chain's limit (2^31 - 1
packs) now runs in pieces instead of one chain op; the plan lives in
src/link_plan.h (sccl_large_reduce_plan), which the engine and a C unit
test share. The reduce-scatter's link op is not chosen for a chunk whose
pack count does not fit (the chunk is also its stride). Every remaining
narrowing site refuses with ncclInvalidArgument and a message instead of
launching with a wrapped count: launch_chain, launch_link_entry, and a
point-to-point channel message above 32 GiB - 16 bytes.

Compatibility: reductions below 32 GiB keep their plan and results.
Larger ones run in pieces (correct, slower) where they returned success
with a wrong amount reduced; a channel send or receive of 32 GiB or more
is refused.

Validation: tests/test_link_plan.c (make check and CMake's link_plan
test) checks the plan at the limits; with the pack-count check removed
from the plan it fails at the 32 GiB chain case. make check and make
check-asan pass.

Request item 13 (audit 10-F8a, 15-F12). NVIDIA NCCL source was neither
read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… setup

LIBSIRCL_RING_REDUCE_PASSES=1 selects the one-pass ring all-reduce relay,
which STATUS.md labels research-only: on a GB10 one run under forward
windows on four ranks returned differing bits on one rank, and the cause
is not isolated. Communicator creation refuses 1 with a message naming
LIBSIRCL_RING_REDUCE_ONE_PASS_RESEARCH=1, which selects it. The setting
joins the settings every rank compares at setup (in the record's reserved
word after cycle_plan, so the record's size is unchanged), and a group
whose ranks differ fails setup naming the variable.

The comments in src/kernelpack.h and KERNEL_ROUTE.md state that the
two-pass entry is the default and that the one-pass form's identical
bytes are a design intent with one open contrary observation; README.md's
variable table names the opt-in and the agreement.

Compatibility: a process that sets LIBSIRCL_RING_REDUCE_PASSES=1 without
the opt-in is refused at communicator creation; the default (2) keeps
its behavior. Ranks that set different values fail setup instead of
running different relay forms.

Validation: tests/test_engine_cpu.py creates a one-rank communicator on
the stand-in driver: before the change PASSES=1 without the opt-in (or
with it set to 0) succeeded; after it both are refused naming the
opt-in, and PASSES=1 with the opt-in and PASSES=2 succeed. make check and
make check-asan pass. The strict-xfail emulation reproducer the audit
asks for (12-F8) is not added: the mismatch was seen once on hardware,
and an emulation case not known to reproduce it would pass and so fail
as a strict xfail.

Request item 14 (audit 10-F6; 12-F8 not done). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The libsircl vLLM plugin's register sets VLLM_NCCL_SO_PATH and then reads
vLLM's own value of it (vllm.envs.VLLM_NCCL_SO_PATH). When that value
names another file, because the vLLM process read its environment before
the plugin ran, register raises LibsirclSelectionError instead of letting
PyNccl load the earlier library. A vllm.envs without the variable is
refused the same way; outside vLLM (no vllm.envs) nothing more is checked.

Compatibility: deployments whose vLLM reads the variable at each access
are unaffected; a vLLM that cached its environment before the plugin ran
fails at plugin registration instead of running on another NCCL.

Validation: test_sparkring_libsircl.py installs stand-in vllm.envs
modules: one reading the environment at each access passes; one holding
the image's NVIDIA NCCL path, and one without the variable, raise. Before
the change register returned in both failing cases. The plugin suite
passes (13 passed, 1 skipped where symbolic links are unavailable).

Request item 15 (audit 13-F1). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When initialization fails after the bootstrap joined (session setup
failed), the initializing thread closes the bootstrap connection and
clears it before publishing FAILED. The failed handle is still returned
for its result and reason and still needs ncclCommDestroy or
ncclCommAbort, while holding no socket and, on the root's process, no
broker thread or root slot (at most 64 per process) in the meantime.
README.md states that contract.

Compatibility: failed handles answer queries, destroy and abort
unchanged; a caller that retries failed creations without destroying the
handles keeps its descriptors and bootstrap roots.

Validation: tests/test_engine_cpu.py makes 70 one-rank creations that
fail in session setup (LIBSIRCL_TRANSPORT=bogus) without destroying the
handles. Before the change the 65th unique id was refused ("at most 64
active bootstrap IDs"); after it every id is granted and the process's
descriptor count returns to its starting value. make check and make
check-asan pass.

Request item 16 (audit 02-F1, 06-F1, 03-F4, 01-F12.4). NVIDIA NCCL
source was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An ncclCommInitAll that fails outside the caller's group aborts every
communicator it created and sets every output handle to NULL, keeping
the failure's text for ncclGetLastError. Inside the caller's own group
the handles belong to that group until its ncclGroupEnd, which reports
their failure, so they are left in place.

Compatibility: callers of a failed top-level InitAll receive NULL
handles instead of FAILED handles that needed destroying.

Validation: tests/test_engine_cpu.py runs InitAll of two ranks on one
stand-in device with LIBSIRCL_TRANSPORT=bogus, so both fail in session
setup. Before the change both outputs were non-NULL handles; after it
both are NULL and the result and reason are kept. make check and make
check-asan pass.

Request item 17 (audit 03-F10, 02-F3 part a). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A thread that opens a group registers a thread-exit hook (a pthread key
destructor). When the thread ends, the hook frees its point-to-point
queue and fails every initialization still pending in its open group
with ncclInvalidUsage ("the thread that initialized this communicator in
a group exited before its ncclGroupEnd") and clears its grouped mark, so
another thread can destroy the handle; queued point-to-point calls of
that group are dropped. The library destructor deletes the key, so no
hook runs into an unloaded library.

Compatibility: a handle stranded by its thread's exit inside a group
reports ncclInvalidUsage and can be destroyed, where every destroy of it
returned ncclInvalidUsage forever. Threads that close their groups are
unaffected apart from the per-thread key.

Validation: tests/test_lifecycle.py starts a thread that calls
ncclGroupStart and ncclCommInitRank and exits. Before the change the
handle reported ncclInProgress; after it the handle reports
ncclInvalidUsage with the reason and ncclCommDestroy succeeds. make check
and make check-asan pass. The queue's release is not measured: the
sanitizer target runs with leak detection off, since the interpreter's
own allocations would dominate it.

Request item 18 (audit 07-F8, 06-F8 bullet 1). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gone

When ncclCommDestroy or ncclCommAbort cannot make the communicator's
context current, the context was destroyed (or the driver is shutting
down), and with it every device and pinned allocation and every kernel
that could read them. Destroy then frees the host bookkeeping that the
context did not own: the retired staging list, the emulation transport's
shared-memory segments, and the engine itself, provided the transports'
queue pairs and registrations were released. It skips only the CUDA
frees.

The stand-in CUDA driver (tests/fake_cuda.c) gains
sccl_fake_cuda_fail_push, which makes every later context push fail.

Compatibility: no change while the context exists; the engine and its
host allocations are freed instead of kept until the process exits.

Validation: tests/test_engine_cpu.py creates and destroys a one-rank
communicator 1,000 times with the push failing at destroy. Before the
change the heap grew by 5.4 MB (5,424 bytes per destroy); after it the
growth is 0. The check is skipped under make check-asan, whose allocator
mallinfo2 does not measure. make check and make check-asan pass.

Request item 20 (audit 02-F15). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inside a group, only calls that belong to it make the group's result: a
communicator creation, a collective or point-to-point call refused other
than as an invalid argument, and the group calls themselves (group_fail).
A failed call that enqueues nothing (a query of an unknown handle, a
registration, ncclGetUniqueId, a lifecycle call, an unsupported function)
returns its error and text and leaves the group's pending initializations
and queued point-to-point calls to run at ncclGroupEnd. README.md states
the rule.

Compatibility: ncclGroupEnd returns success after such failures, where
it returned the failed call's result, failed every pending
initialization and dropped the queued point-to-point calls.
tests/test_api.py's group test, which asserted the poisoning by an
unsupported function, now asserts the group's success.

Validation: tests/test_lifecycle.py initializes a one-rank communicator
in a group, then fails a query on an unknown handle, ncclGetUniqueId with
a NULL output, ncclCommShrink (unsupported) and ncclCommRevoke with
flags. Before the change ncclGroupEnd returned ncclInvalidArgument; after
it ncclGroupEnd succeeds and the communicator is ready. The revoke test
of an initializing communicator now also asserts the group's success.
make check and make check-asan pass.

Request item 21 (audit 03-F8, 01-F2, 01-F12.3). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every generated stub of an unsupported function zeroes its handle,
value and length outputs before it returns ncclInvalidUsage: newcomm of
ncclCommShrink, ncclCommGrow and ncclCommInitRankScalable (NULL), the
unique id of ncclCommGetUniqueId (all zero, which every init refuses),
ncclCommMemStats' value, and the outputs of the ncclParam* functions.
Buffers the caller sizes (ncclParamGet's out, ncclGroupSimulateEnd's
simInfo) and every input are left as they are. The table lives in
tools/generate_api.py (ZEROED_OUTPUTS), which refuses an entry naming an
argument the function does not have; src/stubs.c is regenerated from it.

A test comment in tests/test_api.py now states that an unsupported call
enqueues nothing and leaves its group's result alone.

Compatibility: callers that ignore the refusal read NULL or zero instead
of whatever their output held; return codes and messages are unchanged.

Validation: tests/test_api.py calls every unsupported function and its
pncclXxx twin with a separate 256-byte sentinel per pointer argument and
requires the listed outputs zeroed and every other byte unchanged.
Before the regeneration 36 subtests failed (every listed output kept its
sentinel, for example newcomm after ncclCommShrink); after it all pass.
generate_api.py --check reports no drift; make check and make check-asan
pass.

Request item 22 (audit 03-F12). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Concurrent use of one communicator from several threads is outside
NVIDIA NCCL's contract, but these guards make it fail safe:

- A bootstrap handle holds a mutex across each all-gather round, so
  rounds that threads start on one handle at once run one after another
  instead of interleaving their frames on the socket. A round's deadline
  counts from the moment it holds the lock.
- The teardown rounds of ncclCommFinalize and ncclCommDestroy run under
  a per-communicator close lock: a second, concurrent close waits for the
  first and returns its result instead of running the rounds again.
- transition and the close's READY-to-FINALIZED step use
  compare-exchange, so a state another thread stored in between is seen,
  not overwritten.

Compatibility: single-threaded use is unchanged.

Validation: tests/test_bootstrap.py starts two all-gathers on rank 0's
handle at once while rank 1 runs two rounds, 20 times. Before the change
the interleaved frames made the broker drop the group (results
[6, 0, 0, 6]); after it every round succeeds and each of rank 1's rounds
pairs with exactly one of rank 0's. The close lock and compare-exchange
need a ready engine of two or more ranks to exercise and are not tested
on CPU. make check and make check-asan pass.

Request item 23 (audit 02-F7, 03-F11, 07-F1, 07-F4). NVIDIA NCCL source
was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libsircl took a lane as relayed only when LIBSIRCL_FORWARD_WINDOWS gave
it a nonzero window. With tools/site_routes.py --max-window 0 neither
forward nor point-to-point windows are printed, so relayed pairs looked
direct: they received unwindowed point-to-point channels, which SIRCL's
budget refuses, and the cycle plan could run across them.

tools/site_routes.py prints LIBSIRCL_RELAYED_LANES on every rank with a
lane that crosses relays: per peer position, 1 for each relayed lane and
0 for a cable. The engine counts a lane as relayed when that variable
marks it or a forward window covers it: the relayed positions of the
first setup exchange (which decide the cycle plan), the pair plan's
relay check, and the point-to-point channel table, where a relayed lane
without a point-to-point window gives its pair no channel. A malformed
value fails setup naming the variable. README.md documents it, and the
route data files in tests/data are regenerated with it.

Compatibility: layouts whose relayed lanes all have forward windows
behave the same (the data files gain only the new variable). With
--max-window 0 or hand-written settings, relayed pairs without windows
get no point-to-point channel and keep the cycle plan off.

Validation: tests/test_site_routes.py checks that path:0-3 with
--max-window 0 prints the relayed lanes SIRCL's routes give on every
rank and no window (failed before: nothing printed). tests/test_engine_cpu.py
captures rank 0's first setup record through a stand-in broker:
LIBSIRCL_RELAYED_LANES=1=1/0,5=0/1 marks positions 1 and 5 (failed
before: mask 0), and a malformed value is refused naming the variable.
The channel table itself needs a ready engine and is not run on CPU.
make check and make check-asan pass; the Markdown link check passes.

Request item 24 (audit 09-1). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hare

tools/site_routes.py lowers every printed window to the native layers'
caps: a forward window to 60 chunks of SIRCL_FORWARD_CHUNK_BYTES, the
ring window to 60 chunks of 32 KiB, a point-to-point window to 120
chunks of SIRCL_P2P_CHUNK_BYTES. It then checks every relay hairpin
queue: the session's windows through it (forward windows, and with
--ring-schedules the larger of those and its ring windows) plus, under
--p2p-reserve session, the channels' windows must fit the queue's share
(75 % of 512 KiB). A plan that overfills a queue is refused with exit
status 1 and a message naming the queue; under --p2p-reserve none only
the session's windows are checked, since the channels are then sized as
the only relayed traffic.

The engine validates the windows it reads, with errors naming the
variable: a LIBSIRCL_FORWARD_WINDOWS window must hold 1 to 60 chunks of
SIRCL_FORWARD_CHUNK_BYTES, and LIBSIRCL_RING_WINDOW 1 to 60 chunks of
32 KiB. A ring window derived from the forward windows is never rounded
up past them: a forward window below one 32 KiB ring chunk refuses the
communicator instead.

Compatibility: plans within the caps and shares print the same values,
and the route data files are unchanged. Plans that SIRCL's planner sizes
past a cap print the capped window; plans that overfill a queue, such as
--chunk 65536 on path:0-7, are refused instead of failing at setup or
oversubscribing the queue.

Validation: tests/test_site_routes.py checks --chunk 1024 on ring:8
(forward windows at most 60 chunks; before: 131,072-byte windows of 128
chunks), --p2p-chunk 1024 --p2p-reserve none (point-to-point windows at
most 120 chunks), and --chunk 65536 on path:0-7 (refused naming the
queue; before: printed). tests/test_engine_cpu.py checks that a
LIBSIRCL_RING_WINDOW of 61 chunks is refused naming the variable. The
forward-window and derived ring-window checks need a transport and are
not reached on CPU. make check and make check-asan pass.

Request item 25 (audit 09-2, 09-3, 09-4). NVIDIA NCCL source was neither
read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kernel-pack cache keyed entries by the raw CUcontext value and never
removed them: a context handle reused after its context was destroyed
returned stale functions, and the 17th distinct context could not create
a communicator. Entries now hold references: every sccl_kp_load takes
one, and sccl_kp_release drops it. The engine releases its reference
when it frees itself (at destroy once no kernel of the packs can run,
and on a failed setup); the last release unloads the context's four
modules while the context exists, or only removes the entry when the
context is gone. A context address used again after its last release
loads the packs afresh. At most 16 contexts hold entries at once.

The stand-in CUDA driver (tests/fake_cuda.c) reuses the slots of
unloaded modules and reports the modules still loaded, so the new C test
tests/test_kernelpack_cache.c can load and release packs in turn; make
check and CMake (kernelpack_cache) run it.

Compatibility: a process that creates and destroys communicators on
more than 16 contexts over its life keeps working; quarantined engines
(a kernel still running at abort) keep their reference and modules.

Validation: tests/test_kernelpack_cache.c (67 checks) loads and releases
the packs in 20 distinct contexts, checks that a held reference keeps
the entry, that the last release unloads the modules, and that a reused
address reloads. Before the change no release existed, so entries could
only accumulate. Engine-level creation of a communicator needs two or
more ranks with a transport, which the CPU harness cannot build. make
check and make check-asan pass.

Request item 26 (audit 10-F2). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bootstrap creates its sockets with SOCK_CLOEXEC, accepts with
accept4(SOCK_CLOEXEC | SOCK_NONBLOCK) and makes the root's wake pipe with
pipe2(O_CLOEXEC | O_NONBLOCK), instead of setting FD_CLOEXEC after the
call returned. A fork and exec on another thread between the two steps
could inherit a broker connection and keep it open after this rank left,
defeating the broker's detection of a departed rank.

Compatibility: no behavior change apart from the closed window.

Validation: tests/test_bootstrap.py builds the bootstrap with socket,
accept, accept4, pipe and pipe2 wrapped (ld --wrap), creates an id and
joins a one-rank group, and counts descriptors created without the
close-on-exec flag: 5 before the change, 0 after it. make check and make
check-asan pass.

Request item 27 (audit 02-F10). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bootstrap root of a unique id relays every all-gather round of its
communicator (setup, splits and teardown), so the protocol needs it for
the communicator's life: it lives until every rank's connection to it
closes, or, before every rank joined, until its join window ends. A
process holds at most 64 roots. README.md states this, and
ncclGetUniqueId beyond the limit returns ncclInvalidUsage with a reason
that names the 64 roots and what releases one. The text stays within
the bootstrap's 192-byte error buffer.

Compatibility: the result code is unchanged; the message text changes.

Validation: tests/test_lifecycle.py creates 64 live one-rank
communicators in a child process: the 65th id is refused with
ncclInvalidUsage naming the limit (before: "at most 64 active bootstrap
IDs are supported", without the cause), and after one communicator is
destroyed an id is granted. The split-leader test matches the new text.
make check and make check-asan pass; the Markdown link check passes.

Request item 28 (audit 02-F11). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bootstrap's interface list follows NCCL_SOCKET_IFNAME's syntax in
full: `prefix` and `=name` select, `^prefix` and `^=name` exclude. A list
of exclusions only selects every interface it does not exclude except a
loopback interface, which a remote rank cannot reach; only a selecting
entry chooses loopback. A list longer than 255 characters is refused
with ncclInvalidArgument naming the variable, instead of being truncated
to its first 255 characters. README.md states the syntax.

Compatibility: exclusion-only lists that picked 127.0.0.1 now pick the
first non-loopback interface, or fail naming the variable when the host
has none; lists with a selecting entry behave as before.

Validation: tests/test_bootstrap.py creates ids with NCCL_SOCKET_IFNAME
set to ^docker, ^=lo and ^=lo,^docker: before the change each advertised
127.0.0.1, after it a non-loopback address that a one-rank group joins;
a 319-character list returned success before and is refused after,
naming NCCL_SOCKET_IFNAME. make check and make check-asan pass.

Request item 29 (audit 02-F9). NVIDIA NCCL source was neither read nor
adapted; the list syntax follows NCCL's documented NCCL_SOCKET_IFNAME.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NVIDIA NCCL 2.32.3's ncclCommDestroy and ncclCommAbort return
ncclSuccess at once for a NULL communicator (src/init.cc of tag
v2.32.3-1: `if (comm == NULL) ... return ncclSuccess;`). libsircl's
destroy and abort do the same, so cleanup code that destroys a handle
left NULL by a failed creation sees no error. ncclCommFinalize keeps
refusing NULL with ncclInvalidArgument.

NCCL 2.32.3's ncclCommFinalize also returns ncclSuccess for NULL (it
jumps to its exit label with ret still ncclSuccess). libsircl keeps
finalize's refusal as a documented difference; matching NCCL there is a
separate compatibility decision.

Compatibility: destroy and abort of NULL return ncclSuccess instead of
ncclInvalidArgument.

Validation: tests/test_api.py checks NULL for ncclCommAbort,
ncclCommDestroy (success) and ncclCommFinalize (ncclInvalidArgument),
for the nccl and pnccl names; before the change abort of NULL returned
ncclInvalidArgument. make check and make check-asan pass.

Request item 30 (audit 02-F12). NVIDIA NCCL source was read (init.cc of
v2.32.3-1, the NULL checks of the three functions) and not adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FujitsuPolycom and others added 26 commits October 15, 2026 09:17
ncclMemFree asks the driver for the memory's context
(cuPointerGetAttribute with CU_POINTER_ATTRIBUTE_CONTEXT), makes it
current for the free and restores the caller's context afterwards. A
thread without a current context, or with another device's context,
frees the memory correctly instead of depending on the driver's handling
of a free outside the allocating context. A pointer the driver does not
know is refused with ncclInvalidArgument.

The library binds cuPointerGetAttribute in its driver table (present in
every CUDA driver since 4.0), and the stand-in driver (tests/fake_cuda.c)
provides it and counts context pushes.

Compatibility: frees from the allocating context behave the same; an
unknown pointer returns ncclInvalidArgument instead of the driver's free
error (ncclUnhandledCudaError).

Validation: tests/test_engine_cpu.py frees a pointer on a thread without
a current context: before the change no context was pushed; after it the
memory's context is pushed once and the free succeeds. make check and
make check-asan pass.

Request item 40, first check (audit 06-F5). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session setup looks up every transport, fold, chain, link and pair
exchange entry it launches and refuses setup with the loader's message
(sccl_kp_error, naming the entry) when one lookup fails, instead of
ignoring the results and leaving a NULL function for the first launch.
Every argument is range-checked before the lookups, so none fails with
the packs this tree builds; the check guards a change that widens a
dtype, kind or unroll range without the matching entry. The
point-to-point lookups were already checked.

Compatibility: no change with the packs this tree builds.

Validation: the lookups run only during setup of two or more ranks,
which the CPU harness cannot build (a one-rank communicator loads no
pack); the build-time kernel-entry check (tests/check_entries.c) keeps
verifying that every entry the loader names is in every cubin. make
check and make check-asan pass.

Request item 40, third check (audit 10-F8b). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inside a group, ncclSend and ncclRecv on one communicator must use one
stream and number at most 1,024. Both rules were checked only at
ncclGroupEnd, after every call had returned success. The call that
breaks a rule is now refused with ncclInvalidUsage naming the rule (and,
as a refusal other than an invalid argument, becomes the group's
result); the engine keeps its own check at ncclGroupEnd. README.md
states the rules.

Compatibility: groups within the rules are unchanged; a violating group
fails at the violating call instead of at ncclGroupEnd.

Validation: the check runs after the call's communicator passed its
readiness check, which a CPU-only communicator never does (it carries no
collectives or point-to-point calls), so no CPU test reaches it; no
emulation case is written. make check and make check-asan pass; the
Markdown link check passes.

Request item 40, fourth check (audit 01-F11, code part). NVIDIA NCCL
source was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…LAGS

The Makefile appends its required compiler and preprocessor flags with
`override`, so `make CFLAGS=-O1` or `make CPPFLAGS=-DX` replaces only the
optional part (the default -O2 -g) and keeps -std=c11, -fPIC, -Werror,
-pthread, the dependency flags, the include directories and the build's
definitions. A command-line value used to replace the whole variable and
build a library without position-independent code or the warnings as
errors. Values from the environment (as `make check-asan` passes them)
behave as before.

Compatibility: builds without command-line CFLAGS or CPPFLAGS are
unchanged.

Validation: tests/test_build_flags.py (in make check and CMake's
build_flags test) runs `make -n` for src/api.c with CFLAGS=-O1 and
CPPFLAGS=-DEXTRA and requires the required flags beside the given ones:
before the change the line was `cc -DEXTRA -O1 -c src/api.c ...`; after
it every required flag is present. make check and make check-asan pass.

Request item 42 (audit 11-7). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both builds compile with -fstack-protector-strong and
-D_FORTIFY_SOURCE=2 (after -U, so the compiler's own default does not
conflict) and link the library with -z relro, -z now and -z noexecstack,
whatever the image compiler's defaults. Fortification needs an optimized
build; CMake's Debug configuration goes without it.

CMake:
- an unset build type defaults to RelWithDebInfo (the Makefile's -O2 -g);
- the kernel-entry check takes the architecture count from the -gencode
  flags instead of a hard-coded 2;
- the vendored-source guard (SHA-256, LF, SIRCL change LF's feature
  words) lives in cmake/check_vendored.cmake, runs at configuration and
  again before every build of the library and its transport objects;
- LIBSIRCL_NVCC_EXTRA_FLAGS adds nvcc flags such as
  -DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK;
- the build_flags suite is registered beside the others.

Makefile: the objects and the library depend on the Makefile, so a flag
change rebuilds them. The CI job "libsircl C library" adds a CMake
configure, build and ctest step.

tests/test_build_flags.py checks the hardening flags under command-line
CFLAGS and CPPFLAGS, reads the built library with readelf (GNU_RELRO,
BIND_NOW, a GNU_STACK segment without execute permission), and requires
make check's suites to equal ctest's test names.

Compatibility: builds at -O0 through the Makefile now fail on glibc's
_FORTIFY_SOURCE warning under -Werror; every documented build is
optimized.

Validation: before the change test_build_flags failed on the missing
-fstack-protector-strong and on BIND_NOW; after it make check and make
check-asan pass, and the readelf checks pass on both libraries. CMake
is not installed on this workstation, so the CMake changes and the new
CI step were not run; they run in CI.

Request item 43 (audit 11-5, 11-8, 11-2). NVIDIA NCCL source was neither
read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/emulation/library_rank.py gains run_hostile_argument_checks, run on
every rank after the abort check: 31 calls covering every collective and
ncclSend and ncclRecv with a count whose bytes overflow size_t, a NULL
buffer where every rank needs one, and roots -1, W and 2^30 for
ncclReduce, ncclBroadcast, ncclGather and ncclScatter. Every rank makes
the same calls, so each refusal happens on every rank before any
communication; the case passes when every call returns
ncclInvalidArgument and the send and receive buffers it was given are
unchanged. The module docstring lists the case.

Compatibility: test-only.

Validation: needs a GPU; written, not run:
  python tests/emulation/run_library.py --library build/libsircl.so --world 2
(library_rank.py runs once per rank under run_library.py). The module
compiles and passes ruff.

Request item 44 (audit 12-F2). NVIDIA NCCL source was neither read nor
adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n sirclGetInfo

sirclGetInfo's JSON states the point-to-point scope README.md documents
(two-rank communicators, a rank to itself, and point-to-point channels
on communicators of three or more ranks with LIBSIRCL_P2P_CHANNELS=on),
the schedules' default (pieces, the pair plan on a cabled pair, the
cycle plan on a ring that closes over cables) and the four kernel packs
(transport, fold, link and point-to-point).

Compatibility: the JSON keys are unchanged; only these three strings
change. Nothing in this repository parses them.

Validation: tests/test_engine_cpu.py checks the three strings (the
send_recv text failed before the change). make check and make
check-asan pass.

Request item 48 (audit 14-F10, part of 01-F11). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ncclCommFinalize passed the thread's error buffer as the source of the
snprintf that writes the same buffer (formally undefined), and when the
engine's wait for this rank's work failed it reported whatever text an
earlier call had left. It now reports its own text: "waiting for this
rank's enqueued work failed" for a failed wait, else the close's result
as the communicator recorded it (so a repeated finalize returns the
first close's text with its result), copied into a local buffer before
fail() writes the thread's text.

Compatibility: result codes are unchanged; the message text is the
failure's own.

Validation: a failed finalize needs an engine of two or more ranks,
which the CPU harness cannot build; make check and make check-asan pass.

Request item 50, second fix (audit 03-F5). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ncclCommRevoke runs the fork check before it checks its flags, as every
other entry point does, so a forked child that calls it with nonzero
flags gets ncclInvalidUsage naming the fork instead of
ncclInvalidArgument for the flags.

Compatibility: only the error code and text of that combination change.

Validation: tests/test_api.py's fork test calls ncclCommRevoke(NULL, 1)
in the child: ncclInvalidArgument before the change, ncclInvalidUsage
after it.

Request item 50, fourth fix (audit 03-F13a). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ncclCommCount, ncclCommUserRank and ncclCommCuDevice of a failed
communicator return its result together with its text for the calling
thread, so ncclGetLastError(NULL) says why instead of holding whatever
an earlier call left. A communicator still initializing returns
ncclInProgress without text, as before; the failure does not touch the
thread's group.

Compatibility: result codes are unchanged; ncclGetLastError(NULL) text
changes after such a query.

Validation: tests/test_lifecycle.py queries a communicator whose blocking
initialization timed out after another call set the thread's text:
before the change ncclGetLastError(NULL) still read "version output is
NULL"; after it, the timeout. make check and make check-asan pass.

Request item 50, fifth fix (audit 03-F13b). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ncclGetLastError(comm) on a communicator that has not recorded an error
asks the engine for an asynchronous failure, as ncclCommGetAsyncError
does; a failure found becomes the communicator's terminal error with its
description, and the call returns that text instead of the thread's
earlier text.

Compatibility: no change for communicators without an asynchronous
failure.

Validation: an asynchronous failure needs an engine of two or more
ranks, which the CPU harness cannot build; make check and make
check-asan pass.

Request item 50, first fix (audit 03-F3). NVIDIA NCCL source was neither
read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first call that fails a group keeps its text with the group's
result. ncclGroupEnd's own failure reads "group initialization failed:
<that text>", and each pending initialization the group fails records
"group initialization rejected: <that text>" (its result first, then its
state, as initialization publishes them), where both texts were fixed
placeholders. A blocking initialization deferred to ncclGroupEnd that
fails on its own (not cancelled by the group's failure) writes the
creation-failed line and sets the thread's text, as an ungrouped
creation does, and becomes the group's reason when no call failed it.

Compatibility: result codes are unchanged; the texts name the cause, and
a failed deferred creation writes one stderr line.

Validation: tests/test_lifecycle.py's nested-group test checks both
texts after an InitAll with a negative device fails the outer group;
before the change ncclGetLastError(NULL) read only "group initialization
failed". make check and make check-asan pass.

Request item 50, third fix (audit 03-F1). NVIDIA NCCL source was neither
read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comment above the large-reduction plan in reduce_into said the plan
applies to calls that keep their result on every rank "(not
ncclReduce)". ncclReduce reaches the same chain and ring ops: off the
root its destination is 0 and reduce_link runs the op in place in the
staging buffer. The comment says so.

Compatibility: comment only.

Validation: no code changed.

Request item 50, eighth fix (audit 10-F8c). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kernel pack that the driver cannot load (for example on a GPU whose
architecture the build did not target) is refused with
"loading the <pack> pack (built for sm_120 sm_121): <driver error>".
The list is the build's nvcc code= values: the Makefile derives it from
NVCCFLAGS and CMake from LIBSIRCL_NVCCFLAGS, and kernelpack.c falls back
to the default pair when built without either.

The stand-in driver (tests/fake_cuda.c) fails module loads under
FAKE_CUDA_FAIL_LOAD=1 with CUDA_ERROR_NO_BINARY_FOR_GPU.

Compatibility: the result is unchanged; the message names the
architectures.

Validation: tests/test_kernelpack_cache.c (69 checks) makes a load fail
and requires the error to name "built for sm_120 sm_121" (the message
had no architecture list before the change). `make -n` shows the
definition on kernelpack.o. make check and make check-asan pass.

Request item 50, seventh fix (audit 10-F3). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nfig fields

README.md gains a section "Threading, streams and graphs" stating the
caller contracts: one communicator is not used from several threads at
once without ordering; a stream passed to a call stays valid until the
next call or teardown; graph replays do not overlap eager calls or each
other; the last-error text is per thread; only ncclConfig_t.blocking is
honored; the group rules; what SIRCL_SITE_HASH does; and how an
unsupported function refuses.

Under NCCL_DEBUG (WARN, INFO or TRACE), the first ncclConfig_t of the
process that sets a field other than blocking (an int that is not
NCCL_CONFIG_UNDEF_INT, a name that is not NULL, within the config's
size) writes one stderr line naming the field.

Compatibility: the log line appears only under NCCL_DEBUG; behavior is
otherwise unchanged.

Validation: tests/test_engine_cpu.py creates two communicators with a
config whose commName is set: under NCCL_DEBUG=WARN exactly one line
names ncclConfig_t.commName (none before the change); without
NCCL_DEBUG none. make check and make check-asan pass; the Markdown link
check passes.

Request item 47 (audit 07-F6 = 08-F2, 08-F3, 08-F4, 07-F5, 03-F2,
01-F9 = 08-F9, 01-F11, 14-F11, 14-F13, the cap of 02-F11). NVIDIA NCCL
source was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…VIDIA NCCL

README.md states that libsircl's result codes have NVIDIA NCCL's values
and meaning, that ncclGetErrorString returns short texts without NVIDIA
NCCL's hints to rerun with NCCL_DEBUG (the reason is in
ncclGetLastError), and how a NULL communicator is handled:
ncclCommDestroy and ncclCommAbort return ncclSuccess as NVIDIA NCCL
2.32.3 does, while ncclCommFinalize returns ncclInvalidArgument where
NVIDIA NCCL 2.32.3 returns ncclSuccess.

Compatibility: documentation only.

Validation: the Markdown link check passes.

Request item 50, sixth fix (audit 01-F8 = 03-F7). NVIDIA NCCL source was
read for the NULL behavior (init.cc of v2.32.3-1); its error-string
texts were not copied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tools/site_routes.py prints LIBSIRCL_RELAYED_LANES on every rank of a
layout with relays, and the installer's libsircl transport
(runtime/common/libsircl.py) requires each rank's routing settings to
equal the tool's output. The variable joins ROUTE_VARIABLES, so the lane
records it per rank in the deployment lock and sets it in the rank's
container, and a profile cannot set it. docs/architecture/libsircl.md's
container-settings table lists it.

Compatibility: deployments planned on layouts with relays gain the
variable in their locks and containers; layouts without relays are
unchanged.

Validation: runtime/common/test_libsircl.py: the five tests that compare
the lane's routing settings with the tool's output failed after the
tool began printing the variable and pass with it carried (46 passed).
The Markdown link check passes.

Follow-up to request item 24 (audit 09-1). NVIDIA NCCL source was
neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/architecture/libsircl.md's fail-stop section states that the
installer's libsircl lane sets no wait variable, so every communicator
runs in the startup wait regime at libsircl's default limit of 600 s
for its whole life (nothing calls sirclSetWaitRegime), and a rank whose
peer died learns of it only when a wait reaches that limit; fail-stop
then ends the process.

runtime/common/test_libsircl.py pins this: the lane's library
environment sets none of LIBSIRCL_WAIT_REGIME, SIRCL_STARTUP_WAIT_S,
SIRCL_SERVING_WAIT_S and LIBSIRCL_SETUP_TIMEOUT_MS, the vLLM plugin does
not call sirclSetWaitRegime, and the section names the startup wait
regime and the limit that src/engine.c's DEFAULT_STARTUP_WAIT_S defines,
so the document, the adapter and the library change together.

Compatibility: documentation and test only.

Validation: the test fails against the section without the paragraph
and passes with it; runtime/common/test_libsircl.py passes (46 passed);
the Markdown link check passes.

Request item 49 (audit 16-F5, documentation part of 04-F1 and 05-F5).
NVIDIA NCCL source was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NOTICE said SOURCE_SNAPSHOT.json records "the SIRCL files this library
follows". It records the SHA-256 of every file of the SIRCL
implementation tree that STATUS.md's evidence was taken against, a dated
capture of a development workspace, while the library is built against
this repository's spark_transport/sircl and the vendored copies under
src/transport. NOTICE now says so.

LICENSES/rdma-core-verbs.txt said a release built in another image
copies that image's header. Every libsircl image layer ships this file
as it is in the source tree (runtime/images/libsircl_layer.py copies the
notices runtime/common/libsircl.py lists); the file says so.

tests/test_source_snapshot.py (make check, CMake's source_snapshot)
checks the snapshot's schema (its keys, the selected source, the capture
time, the stated file count, every entry a relative path with a
64-digit SHA-256 and a byte count, the change lists) and NOTICE's
description of it.

Compatibility: documentation and a test; the notice files the layer
ships change text.

Validation: the NOTICE check failed before the change and passes after
it; the schema check passes; make check and make check-asan pass;
runtime/images/test_libsircl_layer.py passes (19 passed).

Request item 46 (audit 11-1, 11-11). NVIDIA NCCL source was neither read
nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README.md and STATUS.md state the research-only status first. README
names the API coverage the manifest records (36 of the header's 73
functions; the other 37 return ncclInvalidUsage) instead of saying
STATUS.md lists each function, and STATUS.md's unsupported row lists all
37 by name. Both name this repository's SIRCL package as 0.3.2.

STATUS.md:
- the headline's bit-identity claim is narrowed to what the body shows:
  float16, bfloat16 and float32 sum all-reduces, all-gathers and
  reduce-scatters against the SIRCL Python session, every other datatype
  and op against the fold model and the rank-order reference;
- four kernel packs, not three or "both";
- rows that said "Not run on Sparks" or "in emulation only" cite the
  Spark runs the evidence sections record (point-to-point channels and
  the LF setup checks on snapshots 0f94c72b and a3477af2, the cycle plan
  in the ring-of-eight image record, vLLM's PyNccl on a cabled pair);
- the evidence section says the cited verification/ records stay on the
  machines that made them, and the external artifacts include the gate
  script gate_multi_0f94c72b.sh;
- hard-coded test counts are removed, and a reference to a document
  outside the repository is described instead of cited by section.

RUNBOOK.md states which sections ran, drops the undefined "ring
measurement lock", the stale "pending 3.4" and "stays pieces" (the pair
and cycle defaults), and names what another image changes.
KERNEL_ROUTE.md labels the older link-pack hash, replaces the dangling
report and section references, and counts all four packs in the setup
agreement.

tests/test_docs_consistency.py (make check, CMake's docs_consistency)
checks README's function counts and STATUS's unsupported list against
tests/api_manifest.json, the absence of "three" or "both" packs against
the loader's four, and the SIRCL package version against its
pyproject.toml.

Compatibility: documentation and a test.

Validation: the new test failed on the documents before the change (the
unsupported list, "three embedded packs", the version) and passes after
it; make check and make check-asan pass; the Markdown link check passes.

Request item 45 (audit 14-F1 to 14-F9, 14-F12, 16-F1, 16-F2, 16-F4,
16-F9, 10-F7). NVIDIA NCCL source was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When ncclCommAbort (or a destroy whose context wait failed) finds a
kernel of the communicator still running, the engine's arena, device
memory and channels stay allocated, since the kernel may still read
them. Such an engine records events behind its last launch and every
channel stream and joins a quarantine list. Every later communicator
creation and destroy in the process retries the list: an engine whose
events completed is freed (memory, channels, order event, its kernel-pack
reference), and one whose context is gone has its host bookkeeping freed.
Every receipt reports quarantined_bytes, the memory the list holds.
Engines kept because a queue pair or registration could not be released
are not reclaimed: such an object may still be written, and the native
layer that created it has already been destroyed.

The library binds cuEventQuery (present in every CUDA driver), and the
stand-in driver provides it.

Compatibility: no change for teardowns without a running kernel;
quarantined memory is returned during the process instead of at its
exit. Receipts gain the quarantined_bytes field.

Validation: a running kernel at abort needs an engine of two or more
ranks, which the CPU harness cannot build, and the reclaim path has not
run. tests/test_engine_cpu.py checks that a one-rank communicator's
receipt reports quarantined_bytes 0. The emulation case the request
proposes (SIRCL_EMU_FAIL_DEREG=1 with the fault cleared later) does not
apply, because a failed verbs release is never reclaimed; no emulation
case is written. make check and make check-asan pass; README.md states
the behavior.

Request item 19 (audit 06-F2, 06-F3, leak part of 04-F5). NVIDIA NCCL
source was neither read nor adapted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The teardown race exchanges each worker's successful enqueue count through
the surviving parent communicator. Finalize and destroy must succeed when
all ranks posted equal counts; unequal counts also permit ncclRemoteError
because unmatched collectives time out. The initial collective must remain
exact, the first worker call must succeed before close, subsequent calls
must return success or 4/5, and a call after destroy must return 4/5.

Bound worker startup and join, record worker exceptions, and reject failed
rank processes even when they left successful JSON. Keep the process
deadline and kill behavior. Add CPU harness regressions to make check and
CTest for count agreement, refusal codes, thread failures and process exits.

Validation: make check and make check-asan pass; the final eight harness
tests pass with and without ASan. Ruff and repository links pass. GPU
execution remains pending. No library implementation changes.
Give the SONAME probe a controlled LD_PRELOAD containing identifiable ASan
runtimes followed by the library under test. Skip with a precise reason
when LIBSIRCL_TEST_ASAN requires a runtime that cannot be identified.
Inherited NCCL and CUDA preloads no longer shadow the tested symbols.

Exercise inherited conflicting libnccl.so.2 entries with both colon and
space delimiters. Retain all 146 symbol checks, the version and alias
checks, and linked versus dlopen SONAME reuse.

Validation: a stand-in NCCL preload fails the baseline probe and passes
the fixed probe, both plain and with ASan. make check, make check-asan,
Ruff and repository links pass. The serving image rerun remains pending.
No library implementation changes.
Preloaded cudaGetDevice providers take precedence over the fake driver's
later RTLD_GLOBAL load. A failing provider makes the two-device probe
refuse context selection; a successful one can select the wrong device.

Remove inherited preloads only for the controlled context probes, keeping
ASan runtimes first. Cover successful and failing competing runtimes with
colon and space preload lists across all three device-selection cases.
Library behavior, check ordering and public compatibility are unchanged.

Validation: baseline reproduces the reported diagnostic with a failing
runtime preload. The regression fails before isolation and passes after.
Local WSL make check passes with that preload retained and cleared;
make check-asan, the regression with ASan plus the conflicting preload,
ruff check ., Markdown links and optional site-route checks pass.
The serving image was not run; no DGX Spark was accessed.
sccl_engine_destroy put an engine whose kernel might still be running on
the quarantine list and then called pthread_mutex_destroy on the engine's
lock. Once the engine is on the list, another thread's
quarantine_reclaim (run by any engine creation or destroy) may find its
events complete and free it. The mutex destroy then wrote to freed
memory. The lock is now destroyed before quarantine_add, and nothing
touches the engine after a successful add.

Compatibility: no behaviour change apart from the removed race.

Validation: make check, check_exports.py and make check-asan pass on a
CPU build (nvcc 13.3 from pip, as in CI). The race needs two threads and
a GPU kernel that outlives an abort, so no CPU test exercises it; nothing
ran on hardware.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FujitsuPolycom
FujitsuPolycom marked this pull request as ready for review November 2, 2026 04:51
@FujitsuPolycom
FujitsuPolycom merged commit c78b48d into main Nov 2, 2026
10 checks passed
@FujitsuPolycom
FujitsuPolycom deleted the libsircl-hardening branch November 2, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant