Repository navigation
Reduce-scatter DeepSeek V4.1 prefill before row-local mHC - #282
FujitsuPolycom wants to merge 1 commit into
Conversation
|
Categorized as design review for the DeepSeek V4.1 prefill decomposition. The change moves mHC work onto rank-local rows while retaining full-row Engram and boundary layers; review row ownership, collective order and continuation/draft state at each transition. The reported GB10 TP4 result motivates the change, but CPU transport models do not validate a device collective. Please clarify the transport used in the measured control: the results paragraph says the vendored proxy was updated to ABI 15, while the merge requirements name ABI 14. Name the exact control and treatment revisions. Serving compatibility work in #279 is related, but should not be conflated with this optimization's gain. @FujitsuPolycom GitHub reports merge conflicts with
Please reconcile the branch with main and rerun the checks affected by the resolution. |
original-el8
left a comment
There was a problem hiding this comment.
Hold for qualification before merging. The design keeps model computation in Ignition and puts the collective mechanisms in shared components; I found no confirmed functional defect in this incremental change.
The reported position-zero logits on 50 prompts and isolated reduce-scatter/gather comparison do not establish complete model-state correctness. Please validate the submitted composition through compiled TP execution and an exported core loaded by flare run, covering partial chunks, inactive state, prefix continuation, and prefill-to-decode/speculative transitions. Compare complete cache state as well as outputs, and check repeated replay and scratch lifetimes. These are the affected contracts required by AGENTS.md and docs/performance-work.md; the default-off flag does not replace qualification.
The reported 5.1–13.0% throughput gains are promising. The comparison used a baseline with a proxy ABI update; #279 now supplies the pinned ABI 14 alignment on main, so preserve an equivalent baseline when validating the rebased change.
Reviewer validation: 195 focused CPU tests passed on the combined #282/#283/#284 stack at 749e75a with pinned FlashInfer 7b35510b. Those tests do not independently qualify this PR's GPU collectives; I did not run GPU collectives or native serving. The branch conflicts with inspected main a5b3b2b in ignition/preparation/runtime_core_pb2.py.
Full-row hyperconnections repeat the same prefill work on every tensor-parallel rank. Reduce-scatter their branch outputs into contiguous rank-owned rows, normalize there, and gather the branch inputs needed by full-row consumers. Engram layers and decoder boundaries retain complete rows and continuation state. Add the explicit preparation option, fixed-shape shared collective components, row-transition scheduling, and preparation-only rebuild metadata. Regenerate the Python bindings from the combined schema while retaining current main's fields. The intended benefit is less repeated hyperconnection work without changing the serving operation families or the disabled-option behavior. Three focused CLI/planning/metadata round-trip tests pass. The broader Windows CLI selection has 205 passes and 41 failures; the same 41 failures reproduce on unmodified main in this environment, which lacks Heliopause and assumes different configuration paths. Syntax and staged whitespace checks pass. Current-default CUDA preparation and native serving have not been run on this head. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
0ffb7ac to
e0187a0
Compare
|
The update at The historical 5.1-13.0% comparison used control Native qualification with reader The rebase's FlashInfer |
Why
Tensor-parallel prefill all-reduces attention and MLP partials, then repeats hyperconnection mixing (mHC) over every row on every rank. Each rank needs only its contiguous row partition for that work. The change removes that duplicate arithmetic while preserving the branch inputs required by the model.
What
The advanced, default-off
--row-shard-prefillexport option replaces eligible full-row reductions with reduce-scatter, local-row mHC and branch-input gather. At TP4, 4,096 rows become 1,024 per rank. Engram layers and the decoder boundary remain full-row; 17 packed-prefill layers use shards. Explicit row declarations and snapshot bindings preserve cache coverage without allocating shard state during serving. Unsupported geometry is rejected. Enabling this prepared layout requires export; existing packages and the default full-row path remain available. Separate tail-capacity/live-row-mask proposals are excluded.Benefit
The expected benefit is less mHC arithmetic and residual traffic. This does not claim less collective payload: an isolated 32 MiB-partial measurement was 2.84 ms for reduce-scatter/gather versus 2.82 ms for all-reduce. No isolated service gain is established for this implementation.
The generic snapshot-selector CPU fix should land with or before deployment. It is a runtime companion, not a model compilation dependency; its latency improvement is not credited to this PR.
Validation and limits
On four GB10 hosts, combined implementation
36c13197on main7413a949, FlashInfer4fd836dcand Transformers 5.23 passed real route/image tests and 54 compiled/uncompiled reduce-scatter/gather replays per rank against real all-reduce. Exact integer inputs cover partial rows and graph transitions. Nine existing compiler-boundary cases passed. CPU checks report 460 passes, two collective-ladder failures reproduced on unchanged main and one platform skip.Whole-model evidence uses immutable FlashInfer
7b35510b/ABI14 cores: full-row baselinegb10-tp4-202611140833and combined treatmentgb10-tp4-202611140129(preparation sources6c3f9d6bfor baseline and7522a7ecfor treatment). Native qualification with readerc71be653passed baseline replay, treatment replay and treatment resident-inactive preservation, including complete checkpoint/Restore and endpoint-state checks. The profile controls are concurrency 1, KV context 1,048,576 and reserve 4 GiB, with matching 8,192 pages, 105 state rows and 99 snapshots. Eleven observed root-rank execution intervals have zero allocation-management calls. Combined retained device-backing increases are 2,468,347,904 bytes on ranks 0/1 and 2,384,461,824 on ranks 2/3; registered-host spans remain 379,064,320 bytes per rank. Registered spans exclude other host storage; these are not peak or isolated-PR costs.Layer-1 floating-cache differences exceed observed baseline variation. Longer seeded-ID failures also reproduce on the full-row baseline. No fresh whole-model artifact prepared with FlashInfer
4fd836dcor new throughput result is claimed.Prepared with AI assistance.