Repository navigation
Add opt-in SIRCL subset mode for direct-chain flat all-gather - #383
Merged
Merged
Conversation
Admit cycle arcs leaving at most two adjacent non-members, validate universal lanes, and budget payload plus bidirectional control traffic. Keep disabled plans and setup records byte-identical for fixed build provenance. Disable delivery-proof credit release and exclude additional relayed P2P traffic in subset mode. Validate the subset matrix, native progress with concurrent pair traffic, baseline byte hashes, and six-rank GPU chain/ring capture and replay. Hardware packet accounting and concurrent decode performance remain unqualified.
Estimate schedules, lane paths and conditional queue windows offline, with placement-qualified measurement overrides. Derive a 192 KiB subset ring bound while retaining the direct-chain runtime policy and all existing defaults. Document model misses, proof assumptions and hardware validation plans. Validate with the SIRCL suite, native progress and placement tests.
Rename ambiguous lane variables in the subset arena budget and exempt the compatibility driver's post-stand-in imports from E402, matching the subset chain driver. Validation: ruff E,F,W; pytest spark_transport/sircl (1087 passed, 126 skipped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sparkring_sircl/oneshot/runtime.py is one of the files integrations/vllm/glm_dcp_decode_comm pins by SHA-256. The subset change edits that file, so test_every_sircl_pin_matches_the_sircl_tree and test_the_sircl_only_refresh_finds_no_drift_and_needs_no_image_tree failed. Behavior: tools/refresh_pins.py --sircl-only re-pinned that one file (631cc652... -> 308e278b...); the other 16 SIRCL pins are unchanged. The FileCheck reason still holds: all_gather(out=), the arena view, _counter_addresses, _order_stream, _mark_stream, _tuned_op, _lock, check_health and the launch geometry keep their definition, signature and call order, and every statement the subset change adds runs only when SIRCL_SUBSET_RING=1, which defaults to 0. The plugin is version 2.0.3 (PLUGIN_VERSION, dist-info directory and METADATA, the GLM-5.3 plugin layer's DCP_DIST_INFO, and the tests that expect the layer's versions). The SIRCL package version stays 0.3.2; the README's Pins section records this tree as the build 2.0.3 pins. Compatibility: published images and their release locks keep the plugin version they carry. With every item flag 0, registration patches nothing and compares no pin, so a profile that loads the plugin with its items off runs on any build; with an item flag 1, the plugin refuses a SIRCL tree other than this one. An image derived from this tree carries 2.0.3. Validation: refresh_pins --check reports every pin matches; the plugin suite passes (235 passed, 16 skipped without the image's vLLM tree); the SIRCL suite passes (1210 passed, 3 skipped); ruff E,F,W, markdown links and repository layout pass; runtime/images, runtime/host/test_images.py and runtime/common/test_image_lock.py give 5 failed, 1007 passed, 2 skipped, the same five pinned-source-byte failures the unchanged tree gives in this environment. GPU checks and audit-mode qualification are pending; both need Spark hardware.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Behavior
SIRCL gains an opt-in subset mode for sessions whose ranks are a contiguous stretch of a larger directly cabled ring that leaves one or two adjacent Sparks outside the session (for example six ranks on positions 0-5 of an eight-Spark ring). Groups that leave more Sparks outside, such as four ranks on half of an eight-Spark ring, are refused at setup; they already run large gathers as ring operations over the cabled path. With
SIRCL_SUBSET_RING=1andSIRCL_ROUTE_SCOPE=fabric, flat all-gathers use the direct cable chain between the ranks instead of the pieces schedule. Every other collective keeps its existing schedule; subset ring all-reduce stays disabled. The default (SIRCL_SUBSET_RINGunset) is unchanged.schedule_selection.pyadds an advisory, offline placement model (schedule, lane paths, queue windows) with measurement overrides; it does not change runtime selection.docs/architecture/sircl-schedule-selection.mddocuments the model, its misses and a derived 192 KiB shared-QP window bound.spark_transport/sircl/SUBSET_RING.mddocuments the mode, its limits and validation.Status: experimental, research-only.
Hardware evidence (research-only)
Six ranks on positions 0-5 of the eight-Spark ring, branch revision
dcd84439(this PR rebased ontomainplus a lint-only commit), arms ASIRCL_SUBSET_RING=0and B=1, three ABBA blocks:Not measured: the 192 KiB subset ring window, because subset mode refuses ring all-reduce by design.
Compatibility
Opt-in only; no default, lock or profile changes. No wire format changes for sessions without subset mode.
The GLM DCP decode plugin's SIRCL pin moves with this change:
oneshot/runtime.pyis one of the filesintegrations/vllm/glm_dcp_decode_commpins, so the plugin is re-pinned asglm_dcp_decode_comm2.0.3 (from 2.0.2). ItsFileCheckreason still holds: every member it names keeps its definition, signature and call order, and every subset statement runs only whenSIRCL_SUBSET_RING=1, so a session that leaves the variable unset takes the ops and settings 2.0.2's build takes. The SIRCL package version stays 0.3.2, and the plugin README's Pins section records this tree as that build. The GPU checks required by the re-pin pass on the newest image. The audit-mode serving qualification has not been run: the only profiles that load this plugin areglm53-nvfp4-tp8andglm53-nvfp4-csf-tp8(eight nodes, DCP4), and the full GLM-5.3 checkpoint is currently present on only two of the eight Sparks. Running that qualification requires the TP8 profile deployed on all eight Sparks, with the 412.5 GiB checkpoint on each.Validation
ruff check --select E,F,W --ignore E501 .python -m pytest spark_transport/sircl -q: 1210 passed, 3 skipped (Linux with GCC and CPU torch; the skips need the native posting order andSIRCL_TEST_VLLM_ROOTS)refresh_pins.py --check --sircl-only: every pin matches;python -m pytest integrations/vllm/glm_dcp_decode_comm -q: 235 passed, 16 skipped (the image's vLLM tree is absent)gpu_checks.py --parts kernels,dcp,registeron the newest image,sparkring:sparkcache-4a17f1f5198e-status038(sha256:f625053225d4b57bfd7d0bb2e8348fea38e69472b10cef87c4701928abc7c4a8): 41 checks, 0 failed; both DCP layouts (path:0-3,path:0-1) and all four registration flag sets; 78 s. All 13 pinned vLLM/b12x files and the editedattention.pyhashed exactly against their pins before the run. The branch's plugin 2.0.3 and SIRCL were used, not the image's older installed plugin 2.0.1. Log SHA256:3e2388b3bc2e06d7626a7c21205e5be400680b044c31355827041e9d0ea060ce.🤖 Generated with Claude Code