Skip to content

Gather selected routes for row-sharded DeepSeek V4.1 prefill - #284

Open
FujitsuPolycom wants to merge 3 commits into
NVIDIA:mainfrom
FujitsuPolycom:prefill-followups-v010
Open

FujitsuPolycom wants to merge 3 commits into
NVIDIA:mainfrom
FujitsuPolycom:prefill-followups-v010

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

With row-sharded attention, each rank produces router logits for its own rows, but gathering all 384 FP32 logits repeats top-k selection on every rank. Expert execution needs only the selected IDs and weights. The change gathers those operands directly.

What

Each rank selects routes locally, packs six int32 expert IDs and six FP32 weights into a 48-byte RouteRecord, then gathers a BF16 bit-view and unpacks contiguous expert operands. The transport preserves weight bits rather than numerically converting them. Temporary route declarations are released so compiled forwards own their storage.

Router preparation receives the rank-local image-mask slice, fixing the full-mask/sharded-logit shape failure on nonzero ranks. The reference uses actual sqrtsoftplus scoring and larger-ID ties, requires nonempty separated-margin coverage and rejects deliberately corrupted routes, including ambiguous rows.

This follows the attention change; it adds no independent flag. It applies when the default-off --row-shard-prefill and --row-shard-attention layouts are exported. Full-row/default execution remains available. Separate tail-capacity/live-row-mask proposals are excluded.

Benefit

At 8,192 rows, gathered route records occupy 384 KiB instead of 12 MiB of logits: 32 times less logical payload, and each row is selected once instead of on every rank. No isolated service gain is established; combined-stack measurements cannot attribute gains to this PR.

The generic snapshot-selector CPU fix should land with or before deployment. It is a runtime companion, not a model compilation dependency or performance credit for this change.

Validation and limits

Four-host GB10 tests at combined implementation 36c13197 on main 7413a949, FlashInfer 4fd836dc and Transformers 5.23 verify exact native IDs/FP32 weight bits through compiled/uncompiled capture, partial/padded rows, ties and graph transitions. The image case exercises the actual preparation callback with text/mixed/image masks on nonzero ranks; whole-model vision generation remains untested. The 54-replay-per-rank reduce-scatter/gather comparison and nine compiler-boundary cases also pass. CPU checks report 460 passes, two collective-ladder failures reproduced on unchanged main and one platform skip.

Whole-model execution checks use signed FlashInfer 7b35510b/ABI14 cores: full-row baseline gb10-tp4-202611140833 and combined treatment gb10-tp4-202611140129 (preparation sources 6c3f9d6b for baseline and 7522a7ec for treatment). Native qualification with reader c71be653 passed baseline replay, treatment replay and treatment resident-inactive preservation, including complete checkpoint/Restore, endpoint-state checks and matching eight-output replay. Profiling uses concurrency 1, KV context 1,048,576 and reserve 4 GiB; both cores retain 8,192 pages/105 state rows/99 snapshots; 11 observed root-rank execution intervals have zero allocation-management calls. Combined device-backing deltas are 2,468,347,904 bytes on ranks 0/1 and 2,384,461,824 on ranks 2/3; registered-host spans stay at 379,064,320 bytes per rank, not total host storage or peak memory.

The route-record comparisons require exact IDs and FP32 weight bits. Layer-1 cache changes exceed baseline variation, and longer seeded-ID failures also reproduce on the baseline. No fresh whole-model artifact prepared with FlashInfer 4fd836dc or new throughput result is claimed.

Prepared with AI assistance.

@original-el8 original-el8 added area:kernels GPU kernels and numerical execution. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. enhancement New feature or request model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:validation Proposed implementation needs acceptance evidence; no claim that a reported defect is fixed. labels Nov 13, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

Categorized as kernel performance work needing validation. This PR replaces the row-sharded router-logit gather with gathered expert IDs and weights; it depends on #283 and #282. The repeated stack files are dependencies, not evidence that the three PRs are redundant.

The reported 13.5–20.4% prefill gain is for the full stack. The exact record-gather path has CPU simulated-fabric coverage but no CUDA case; add real collective validation of IDs, FP32 weight bits, partial rows and graph transitions before attributing correctness to that simulation. Retarget or otherwise isolate the incremental diff after the prerequisite changes land.

@FujitsuPolycom GitHub reports merge conflicts with main, confirmed locally with git merge-tree for PR head 749e75a0254e against main 97cfeb0001f1. Conflicting paths:

  • ignition/preparation/runtime_core_pb2.py

Please reconcile the branch with main and rerun the checks affected by the resolution.

@original-el8 original-el8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes for the image-mask preparation failure and the ineffective routing-agreement assertion detailed below.

The route-record layout is a reasonable way to reduce communication, but the submitted validation does not qualify its GPU composition. The CPU decoder test replaces attention, routing, experts, and transport with CPU implementations; it cannot establish native routing, collective, cache-state, or exported replay correctness. Its router also substitutes softmax for DeepSeek V4.1's sqrtsoftplus and uses different tie ordering. Add focused validation of the real preparation contract and qualify compiled GPU route gathering and native replay with the #282/#283 prerequisites.

The reported 13.5–20.4% throughput gain belongs to the full stack. The cited whole-model parity run used an earlier build, so it does not establish numerical correctness of this submitted revision.

Reviewer validation: 195 focused CPU tests passed at 749e75a with pinned FlashInfer 7b35510b. The image-mask rejection was reproduced through the actual router preparation callback and b12x validation using CUDA-shaped fake tensors; no GPU kernel or complete export was executed. I did not run GPU collectives or native serving. The branch conflicts with inspected main a5b3b2b in ignition/preparation/runtime_core_pb2.py and ignition/recipes/deepseek41/recipe.py. Review of this PR was scoped to its two commits above #283.


[P1] Prepare the image mask with the router's shard row count

When attention_shard is enabled and the checkpoint has vision correction weights, router_logits and route_scratch have row_shard.rows rows, but self.image_mask was allocated from the full metadata.token_ids shape. PreparedRouter captures this full mask in its preparation callback. At TP4 with a 4,096-row prefill, the callback passes 1,024 logit rows and a 4,096-element mask to b12x, which rejects it with ValueError: image_mask must have shape (num_tokens,) before export can complete. The shard slicing in _run only fixes execution-time operands; preparation fails earlier.

I reproduced the rejection through PreparedRouter's preparation callback and the pinned b12x _RouteTopKState using CUDA-shaped fake tensors. Size the declaration mask to the rows this router prepares and add a regression that exercises the image-enabled preparation callback, including a nonzero rank.

Location: ignition/recipes/deepseek41/decoder.py:477.


[P2] Make the routing-margin assertion check reachable rows

ReferenceRouter records the difference between the second- and third-largest softmax probabilities because this test uses TOP_K = 2. The second-largest probability is at most 0.5, so this margin can never exceed 0.5. agree[margin > 0.5] is always empty and .all() always passes. The remaining >0.8 agreement check can therefore accept nearly 20% route disagreement without the advertised margin-based protection.

Use a meaningful margin/error criterion with a nonempty-coverage assertion and a regression showing that an incorrect route fails. For model qualification, preserve the actual sqrtsoftplus scoring and native tie ordering rather than treating this softmax substitute and its broad output bounds as numerical equivalence evidence.

Location: tests/recipes/test_deepseek41_row_shard_prefill_e2e.py:884.

Full-row hyperconnections repeat the same prefill work on every tensor-parallel
rank. Reduce-scatter their branch outputs into contiguous rank-owned rows,
normalize there, and gather the branch inputs needed by full-row consumers.
Engram layers and decoder boundaries retain complete rows and continuation state.

Add the explicit preparation option, fixed-shape shared collective components,
row-transition scheduling, and preparation-only rebuild metadata. Regenerate the
Python bindings from the combined schema while retaining current main's fields.
The intended benefit is less repeated hyperconnection work without changing the
serving operation families or the disabled-option behavior.

Three focused CLI/planning/metadata round-trip tests pass. The broader Windows
CLI selection has 205 passes and 41 failures; the same 41 failures reproduce on
unmodified main in this environment, which lacks Heliopause and assumes different
configuration paths. Syntax and staged whitespace checks pass. Current-default
CUDA preparation and native serving have not been run on this head.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Tensor-parallel prefill gathers normalized rows before attention and reduces its
output afterward. Compute all attention heads and output groups for each rank's
own rows, then gather the cache projections required by full-row consumers.
This removes those exchanges while preserving complete cache writes.

Load replicated query/output projections only for eligible prefill layers,
retain partitioned decode copies, and carry selection state across row
transitions. Account for values, both retained MXFP8 scale layouts and FP32
sinks: 17 layers add 2,121,273,600 numeric bytes per rank. Preserve main's
metadata and online-conversion options. Document the qualified hardware and
retained-core scope separately from unrun dependency/mode configurations.

The benefit is less attention communication at a retained-weight cost. Grouping
all eight output groups changes BF16 rounding association; comparisons use
operator-derived bounds rather than exact partitioned-output equality.

Validation includes accounting/layout tests and combined-stack GPU/native
checks. On 32 captured native rows, all 163,840 projection outputs satisfy the
unchanged operator budget and match native BF16 bits. Existing 7b35510b/ABI-14
cores pass checkpoint/Restore, inactive-state, replay and allocation checks;
4fd836dc component/compiler checks pass separately. Floating cache differences,
baseline-reproduced replay failures and measured dependency scopes remain
explicit; no fresh default-dependency whole-model export or isolated speedup
is claimed. Computational source is unchanged from the reviewed 8d38f298 tree.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Row-sharded attention owns each rank's query rows, but gathering all 384
router logits repeats communication and top-k work. Gather six selected
expert IDs and their FP32 weight bits in a 48-byte record, then unpack
contiguous expert operands without numeric conversion.

Prepare the router image mask at the rank-local row count, including
nonzero ranks, while retaining the full layer mask and runtime selection.
Use sqrtsoftplus and larger-ID ties in the reference, require nonempty
margin coverage, and reject incorrect routes on ambiguous rows. Preserve
main's optional activation-mode forwarding and explicit scratch operands.

The change reduces logical gathered route payload by 32 times at top-k
six. It introduces no independent runtime policy or isolated service-gain
claim; route gathering follows opt-in row-sharded attention.

Validation: four-host GB10 GPU route-record and image-router preparation
regressions pass with FlashInfer 4fd836dc, including exact IDs/FP32 bits,
partial/padded rows, ties and compiled graph transitions. Combined-stack
existing-core native cache/replay checks retain their 7b35510b/ABI-14
artifact scope. CPU reference tests reject wrong routes, empty coverage
and mask shape errors. Model numerical limits and older baseline replay
failures remain documented; no fresh default-dependency whole-model
export is claimed. The separate tail-capacity/masking proposal is excluded.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
@FujitsuPolycom FujitsuPolycom changed the title Gather routes instead of router logits for row-sharded DeepSeek V4.1 prefill Gather selected routes for row-sharded DeepSeek V4.1 prefill Nov 15, 2026
@FujitsuPolycom

Copy link
Copy Markdown
Contributor Author

The update at 885848f3 fixes router preparation to use rank-local image-mask rows; the full layer mask and runtime path remain unchanged. The reference now uses sqrtsoftplus and larger-ID ties, requires nonempty separated-margin coverage, and rejects deliberately wrong routes, including ambiguous rows. Route-record gathering is used only on the opt-in row-sharded-attention path; both export flags are off by default.

Four-host GB10 GPU tests with FlashInfer 4fd836dc passed for exact expert IDs and FP32 weight bits, partial/padded rows, ties, compiled graph transitions and actual image-enabled router preparation. This validates router preparation and transport, not whole-model vision generation. The separate tail-capacity/masking proposals remain excluded.

With native reader c71be653, the existing combined-stack core prepared with FlashInfer 7b35510b/ABI 14 passed checkpoint/prefix/Restore and speculative replay checks. The resident-inactive case preserved the complete row and eight page planes exactly: 11,929,600 bytes per rank across another request's prefill. On diagnostic export bc7890f2 with native tool e5d9140a, control and compiler-pool-poison runs preserved the resident row and eight page planes exactly (11,929,600 bytes per rank) at three checked foreign-prefill boundaries, including a 456-row padded tail. These bounded replay and resident-inactive checks returned zero.

The combined-stack numerical caveats remain: floating cache differences and longer exact-ID failures are documented, including baseline controls. Whole-model execution checks used the existing 7b35510b/ABI 14 core; 4fd836dc validation covered the affected component/compiler paths. No isolated #284 speedup is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:kernels GPU kernels and numerical execution. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. enhancement New feature or request model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:validation Proposed implementation needs acceptance evidence; no claim that a reported defect is fixed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants