Skip to content

Reduce-scatter DeepSeek V4.1 prefill before row-local mHC - #282

Open
FujitsuPolycom wants to merge 1 commit into
NVIDIA:mainfrom
FujitsuPolycom:rowshard-prefill-v010
Open

FujitsuPolycom wants to merge 1 commit into
NVIDIA:mainfrom
FujitsuPolycom:rowshard-prefill-v010

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

Tensor-parallel prefill all-reduces attention and MLP partials, then repeats hyperconnection mixing (mHC) over every row on every rank. Each rank needs only its contiguous row partition for that work. The change removes that duplicate arithmetic while preserving the branch inputs required by the model.

What

The advanced, default-off --row-shard-prefill export option replaces eligible full-row reductions with reduce-scatter, local-row mHC and branch-input gather. At TP4, 4,096 rows become 1,024 per rank. Engram layers and the decoder boundary remain full-row; 17 packed-prefill layers use shards. Explicit row declarations and snapshot bindings preserve cache coverage without allocating shard state during serving. Unsupported geometry is rejected. Enabling this prepared layout requires export; existing packages and the default full-row path remain available. Separate tail-capacity/live-row-mask proposals are excluded.

Benefit

The expected benefit is less mHC arithmetic and residual traffic. This does not claim less collective payload: an isolated 32 MiB-partial measurement was 2.84 ms for reduce-scatter/gather versus 2.82 ms for all-reduce. No isolated service gain is established for this implementation.

The generic snapshot-selector CPU fix should land with or before deployment. It is a runtime companion, not a model compilation dependency; its latency improvement is not credited to this PR.

Validation and limits

On four GB10 hosts, combined implementation 36c13197 on main 7413a949, FlashInfer 4fd836dc and Transformers 5.23 passed real route/image tests and 54 compiled/uncompiled reduce-scatter/gather replays per rank against real all-reduce. Exact integer inputs cover partial rows and graph transitions. Nine existing compiler-boundary cases passed. CPU checks report 460 passes, two collective-ladder failures reproduced on unchanged main and one platform skip.

Whole-model evidence uses immutable FlashInfer 7b35510b/ABI14 cores: full-row baseline gb10-tp4-202611140833 and combined treatment gb10-tp4-202611140129 (preparation sources 6c3f9d6b for baseline and 7522a7ec for treatment). Native qualification with reader c71be653 passed baseline replay, treatment replay and treatment resident-inactive preservation, including complete checkpoint/Restore and endpoint-state checks. The profile controls are concurrency 1, KV context 1,048,576 and reserve 4 GiB, with matching 8,192 pages, 105 state rows and 99 snapshots. Eleven observed root-rank execution intervals have zero allocation-management calls. Combined retained device-backing increases are 2,468,347,904 bytes on ranks 0/1 and 2,384,461,824 on ranks 2/3; registered-host spans remain 379,064,320 bytes per rank. Registered spans exclude other host storage; these are not peak or isolated-PR costs.

Layer-1 floating-cache differences exceed observed baseline variation. Longer seeded-ID failures also reproduce on the full-row baseline. No fresh whole-model artifact prepared with FlashInfer 4fd836dc or new throughput result is claimed.

Prepared with AI assistance.

@original-el8 original-el8 added area:kernels GPU kernels and numerical execution. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. enhancement New feature or request model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:design-review Needs a maintainer scope, architecture, or contract decision before implementation or integration. labels Nov 13, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

Categorized as design review for the DeepSeek V4.1 prefill decomposition. The change moves mHC work onto rank-local rows while retaining full-row Engram and boundary layers; review row ownership, collective order and continuation/draft state at each transition. The reported GB10 TP4 result motivates the change, but CPU transport models do not validate a device collective.

Please clarify the transport used in the measured control: the results paragraph says the vendored proxy was updated to ABI 15, while the merge requirements name ABI 14. Name the exact control and treatment revisions. Serving compatibility work in #279 is related, but should not be conflated with this optimization's gain.

@FujitsuPolycom GitHub reports merge conflicts with main, confirmed locally with git merge-tree for PR head 0ffb7ac1b49a against main 97cfeb0001f1. Conflicting paths:

  • ignition/preparation/runtime_core_pb2.py

Please reconcile the branch with main and rerun the checks affected by the resolution.

@original-el8 original-el8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold for qualification before merging. The design keeps model computation in Ignition and puts the collective mechanisms in shared components; I found no confirmed functional defect in this incremental change.

The reported position-zero logits on 50 prompts and isolated reduce-scatter/gather comparison do not establish complete model-state correctness. Please validate the submitted composition through compiled TP execution and an exported core loaded by flare run, covering partial chunks, inactive state, prefix continuation, and prefill-to-decode/speculative transitions. Compare complete cache state as well as outputs, and check repeated replay and scratch lifetimes. These are the affected contracts required by AGENTS.md and docs/performance-work.md; the default-off flag does not replace qualification.

The reported 5.1–13.0% throughput gains are promising. The comparison used a baseline with a proxy ABI update; #279 now supplies the pinned ABI 14 alignment on main, so preserve an equivalent baseline when validating the rebased change.

Reviewer validation: 195 focused CPU tests passed on the combined #282/#283/#284 stack at 749e75a with pinned FlashInfer 7b35510b. Those tests do not independently qualify this PR's GPU collectives; I did not run GPU collectives or native serving. The branch conflicts with inspected main a5b3b2b in ignition/preparation/runtime_core_pb2.py.

Full-row hyperconnections repeat the same prefill work on every tensor-parallel
rank. Reduce-scatter their branch outputs into contiguous rank-owned rows,
normalize there, and gather the branch inputs needed by full-row consumers.
Engram layers and decoder boundaries retain complete rows and continuation state.

Add the explicit preparation option, fixed-shape shared collective components,
row-transition scheduling, and preparation-only rebuild metadata. Regenerate the
Python bindings from the combined schema while retaining current main's fields.
The intended benefit is less repeated hyperconnection work without changing the
serving operation families or the disabled-option behavior.

Three focused CLI/planning/metadata round-trip tests pass. The broader Windows
CLI selection has 205 passes and 41 failures; the same 41 failures reproduce on
unmodified main in this environment, which lacks Heliopause and assumes different
configuration paths. Syntax and staged whitespace checks pass. Current-default
CUDA preparation and native serving have not been run on this head.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
@FujitsuPolycom FujitsuPolycom changed the title Reduce-scatter DeepSeek V4.1 prefill so each rank runs mHC on its own rows Reduce-scatter DeepSeek V4.1 prefill before row-local mHC Nov 15, 2026
@FujitsuPolycom

Copy link
Copy Markdown
Contributor Author

The update at e0187a04 resolves the Protobuf conflict against main 7413a949. Four-host reduce-scatter/gather passed 54 compiled/uncompiled replays per rank against real all-reduce, covering partial rows and graph transitions with bounded integer inputs. This path is opt-in at export through --row-shard-prefill, which is off by default.

The historical 5.1-13.0% comparison used control 7a3ea274, treatment 970b4a2d, and FlashInfer ffea9365/ABI 15 with matching vendored transport. It does not measure the rebased head. Subsequent native qualification uses FlashInfer 7b35510b/ABI 14 and an equivalent full-row baseline.

Native qualification with reader c71be653 passed baseline replay, treatment replay and treatment resident-inactive preservation. The replay cases covered partial prefill, checkpoint/prefix preservation, exact Restore and eight matching replay IDs; native, controller and profiler checks passed on all ranks. The resident check preserved 11,929,600 bytes per rank across another request's prefill. On diagnostic export bc7890f2 with native tool e5d9140a, control and compiler-pool-poison runs preserved the resident row and eight page planes exactly (11,929,600 bytes per rank) at three checked foreign-prefill boundaries, including a 456-row padded tail.

The rebase's FlashInfer 4fd836dc component checks and compiler-boundary tests passed separately; no fresh whole-model artifact with that dependency is claimed. Longer exact-ID failures also reproduce on unchanged baseline. Historical gains remain scoped to their measured revisions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:kernels GPU kernels and numerical execution. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. enhancement New feature or request model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:design-review Needs a maintainer scope, architecture, or contract decision before implementation or integration.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants