Skip to content

Add opt-in SIRCL subset mode for direct-chain flat all-gather - #383

Merged
FujitsuPolycom merged 9 commits into
mainfrom
codex/sircl-subset-ring
Nov 2, 2026
Merged

FujitsuPolycom merged 9 commits into
mainfrom
codex/sircl-subset-ring

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Behavior

SIRCL gains an opt-in subset mode for sessions whose ranks are a contiguous stretch of a larger directly cabled ring that leaves one or two adjacent Sparks outside the session (for example six ranks on positions 0-5 of an eight-Spark ring). Groups that leave more Sparks outside, such as four ranks on half of an eight-Spark ring, are refused at setup; they already run large gathers as ring operations over the cabled path. With SIRCL_SUBSET_RING=1 and SIRCL_ROUTE_SCOPE=fabric, flat all-gathers use the direct cable chain between the ranks instead of the pieces schedule. Every other collective keeps its existing schedule; subset ring all-reduce stays disabled. The default (SIRCL_SUBSET_RING unset) is unchanged.

  • Subset sessions run with isolated relay queues, bounded so a forwarding Spark's hairpin queue is never oversubscribed.
  • schedule_selection.py adds an advisory, offline placement model (schedule, lane paths, queue windows) with measurement overrides; it does not change runtime selection. docs/architecture/sircl-schedule-selection.md documents the model, its misses and a derived 192 KiB shared-QP window bound.
  • spark_transport/sircl/SUBSET_RING.md documents the mode, its limits and validation.

Status: experimental, research-only.

Hardware evidence (research-only)

Six ranks on positions 0-5 of the eight-Spark ring, branch revision dcd84439 (this PR rebased onto main plus a lint-only commit), arms A SIRCL_SUBSET_RING=0 and B =1, three ABBA blocks:

Case Result
Exactness 12 harness runs × 112 cases, every case exact
Flat all-gather, 4 MiB shards 1.20× the pieces baseline, eager and graph
Flat all-gather, 16 MiB shards 1.32×
Padded / 2-D gathers, large all-reduce within 0.3% of baseline (unchanged path)

Not measured: the 192 KiB subset ring window, because subset mode refuses ring all-reduce by design.

Compatibility

Opt-in only; no default, lock or profile changes. No wire format changes for sessions without subset mode.

The GLM DCP decode plugin's SIRCL pin moves with this change: oneshot/runtime.py is one of the files integrations/vllm/glm_dcp_decode_comm pins, so the plugin is re-pinned as glm_dcp_decode_comm 2.0.3 (from 2.0.2). Its FileCheck reason still holds: every member it names keeps its definition, signature and call order, and every subset statement runs only when SIRCL_SUBSET_RING=1, so a session that leaves the variable unset takes the ops and settings 2.0.2's build takes. The SIRCL package version stays 0.3.2, and the plugin README's Pins section records this tree as that build. The GPU checks required by the re-pin pass on the newest image. The audit-mode serving qualification has not been run: the only profiles that load this plugin are glm53-nvfp4-tp8 and glm53-nvfp4-csf-tp8 (eight nodes, DCP4), and the full GLM-5.3 checkpoint is currently present on only two of the eight Sparks. Running that qualification requires the TP8 profile deployed on all eight Sparks, with the 412.5 GiB checkpoint on each.

Validation

  • ruff check --select E,F,W --ignore E501 .
  • python -m pytest spark_transport/sircl -q: 1210 passed, 3 skipped (Linux with GCC and CPU torch; the skips need the native posting order and SIRCL_TEST_VLLM_ROOTS)
  • refresh_pins.py --check --sircl-only: every pin matches; python -m pytest integrations/vllm/glm_dcp_decode_comm -q: 235 passed, 16 skipped (the image's vLLM tree is absent)
  • gpu_checks.py --parts kernels,dcp,register on the newest image, sparkring:sparkcache-4a17f1f5198e-status038 (sha256:f625053225d4b57bfd7d0bb2e8348fea38e69472b10cef87c4701928abc7c4a8): 41 checks, 0 failed; both DCP layouts (path:0-3, path:0-1) and all four registration flag sets; 78 s. All 13 pinned vLLM/b12x files and the edited attention.py hashed exactly against their pins before the run. The branch's plugin 2.0.3 and SIRCL were used, not the image's older installed plugin 2.0.1. Log SHA256: 3e2388b3bc2e06d7626a7c21205e5be400680b044c31355827041e9d0ea060ce.

🤖 Generated with Claude Code

FujitsuPolycom and others added 9 commits October 15, 2026 11:52
Admit cycle arcs leaving at most two adjacent non-members, validate universal lanes, and budget payload plus bidirectional control traffic. Keep disabled plans and setup records byte-identical for fixed build provenance. Disable delivery-proof credit release and exclude additional relayed P2P traffic in subset mode.

Validate the subset matrix, native progress with concurrent pair traffic, baseline byte hashes, and six-rank GPU chain/ring capture and replay. Hardware packet accounting and concurrent decode performance remain unqualified.
Estimate schedules, lane paths and conditional queue windows offline, with placement-qualified measurement overrides. Derive a 192 KiB subset ring bound while retaining the direct-chain runtime policy and all existing defaults. Document model misses, proof assumptions and hardware validation plans. Validate with the SIRCL suite, native progress and placement tests.
Rename ambiguous lane variables in the subset arena budget and exempt the
compatibility driver's post-stand-in imports from E402, matching the
subset chain driver.

Validation: ruff E,F,W; pytest spark_transport/sircl (1087 passed, 126 skipped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sparkring_sircl/oneshot/runtime.py is one of the files
integrations/vllm/glm_dcp_decode_comm pins by SHA-256. The subset change
edits that file, so test_every_sircl_pin_matches_the_sircl_tree and
test_the_sircl_only_refresh_finds_no_drift_and_needs_no_image_tree failed.

Behavior: tools/refresh_pins.py --sircl-only re-pinned that one file
(631cc652... -> 308e278b...); the other 16 SIRCL pins are unchanged. The
FileCheck reason still holds: all_gather(out=), the arena view,
_counter_addresses, _order_stream, _mark_stream, _tuned_op, _lock,
check_health and the launch geometry keep their definition, signature and
call order, and every statement the subset change adds runs only when
SIRCL_SUBSET_RING=1, which defaults to 0. The plugin is version 2.0.3
(PLUGIN_VERSION, dist-info directory and METADATA, the GLM-5.3 plugin
layer's DCP_DIST_INFO, and the tests that expect the layer's versions). The
SIRCL package version stays 0.3.2; the README's Pins section records this
tree as the build 2.0.3 pins.

Compatibility: published images and their release locks keep the plugin
version they carry. With every item flag 0, registration patches nothing and
compares no pin, so a profile that loads the plugin with its items off runs
on any build; with an item flag 1, the plugin refuses a SIRCL tree other
than this one. An image derived from this tree carries 2.0.3.

Validation: refresh_pins --check reports every pin matches; the plugin suite
passes (235 passed, 16 skipped without the image's vLLM tree); the SIRCL
suite passes (1210 passed, 3 skipped); ruff E,F,W, markdown links and
repository layout pass; runtime/images, runtime/host/test_images.py and
runtime/common/test_image_lock.py give 5 failed, 1007 passed, 2 skipped, the
same five pinned-source-byte failures the unchanged tree gives in this
environment. GPU checks and audit-mode qualification are pending; both need
Spark hardware.
@FujitsuPolycom
FujitsuPolycom merged commit dcfc974 into main Nov 2, 2026
10 checks passed
@FujitsuPolycom
FujitsuPolycom deleted the codex/sircl-subset-ring branch November 2, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant