Skip to content

Gather routes instead of router logits for row-sharded DeepSeek V4.1 prefill - #7

Closed
FujitsuPolycom wants to merge 3 commits into
review/rowshard-attention-v013from
review/prefill-followups-v013
Closed

FujitsuPolycom wants to merge 3 commits into
review/rowshard-attention-v013from
review/prefill-followups-v013

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Why

After NVIDIA#282 partitions prefill rows and NVIDIA#283 shards attention, ranks still gather all 384 FP32 router logits per row and repeat top-k. Full-row experts need only selected IDs and weights.

Change

Ranks now run local top-k, pack six int32 IDs and six FP32 weights into 48-byte RouteRecords, gather BF16 bit views, and unpack contiguous expert operands. An 8,192-row chunk transfers 384 KiB instead of 12 MiB. Releasing route declarations lets compilation plan temporaries.

Eligibility adds 768/1,280/1,792 rows per rank to multiples of 512: TP4 capacities 3,072/5,120/7,168. Other capacities remain full-row; graph selection is unchanged. DeepSeek V4.1 uses one unbounded mHC call per block; the optional 512-row bound also handles a 256-row remainder. Attention-sharded routing masks padding with ID -1/zero weight; other routing is unchanged.

A CPU decoder test with simulated collectives and PyTorch references for GPU-only components compares collective order, routes, hidden rows, logits and draft feedback against full-row execution at two and four ranks across nine layers, including Engram and row/decoder transitions.

Benefit and measurements

Combined NVIDIA#282/NVIDIA#283/NVIDIA#284 DeepSeek V4.1 results, four GB10 hosts/RoCE ring, 8,192-token chunks, prepared concurrency 16: client prompt tokens/TTFT for standalone cold-prefill after warmup, not aggregate concurrency-16 throughput.

Prompt v0.1.0, pinned ABI 14 Stack tok/s
16k 4,497 5,106 (+13.5%)
64k 4,713 5,674 (+20.4%)
128k 4,492 5,252 (+16.9%)

Baseline: two requests/cell; stack: one invocation/cell, median of four requests. Stack revisions: NVIDIA#282 0ffb7ac1, NVIDIA#283 335535a0, NVIDIA#284 749e75a0; neither isolates NVIDIA#284 nor measures this 97cfeb00-based head. KV tokens/rank: 24.9M versus 29.1M baseline, reflecting NVIDIA#283's replicated attention weights rather than the route-record change.

Separate buckets/masks export: 5,357/5,755/5,512 tok/s versus 5,106/5,674/5,252 (+4.9%/+1.4%/+5.0%) at 16k/64k/128k, same metric/hardware. New cells average four invocations, each median of 6–7 warmed requests; range/mean spreads 1.81%/0.66%/1.27%. Comparison cells use one invocation, median of four; no repeated-invocation variance or statistical significance established. Actual 64k prompts: 64,355–64,364 tokens, partial 8,192-row chunk. KV capacity: comparison 24.85M versus buckets/masks 25.58M tokens/rank. Different source tree: no proposed-branch GPU/service qualification or isolation of buckets versus masks.

Evidence and limits

Separate recipe-default all-40-layer attention plus extra buckets: +9.1–13.8% throughput versus NVIDIA#282 alone. On 50 seeded prompts, position-0 KL: 3.3e-6/2.7e-6 versus NVIDIA#282's 3.1e-6/3.0e-6 and baseline-repeat floor 2.4e-6; all top-1 agree. Unidentified revision; no route-gather GPU/head qualification.

At 749e75a0: 382 CPU passes with b12x, including row-local route references/four end-to-end cases; two planning cases deselected. Regression over CLI and component/recipe files changed by the stack, with installed b12x/CUTLASS, not pinned dependencies: 946 passed/27 skipped/9 CLI failures; route-gather/decoder-test-only tree: 913 passed. Same nine failing IDs on both and base 97cfeb00. Without b12x, both shared-file runs: 558 passed/167 failed/76 errors; tail adds nine failures, two new files cannot collect. GPU/checkpoint/service inputs absent.

Tail tests cover live counts/reference routing, capacities, offsets, bounded mHC and Dynamo eager graph reuse. End-to-end simulation uses fully live chunks, 512 rows/rank, not tails. CUDA transport, production tail mHC, complete compiled GPU decoder and native replay remain unqualified.

Dependencies

NVIDIA#282 supplies row sharding; NVIDIA#283 supplies attention on that partition; NVIDIA#284 adds route gathering, CPU decoder coverage and tail masks/buckets. Merge NVIDIA#282 → NVIDIA#283 → NVIDIA#284.

Generated with Claude Code

FujitsuPolycom and others added 3 commits November 13, 2026 10:26
Gathering 384 FP32 router logits per row makes every rank repeat
row-local top-k. Selected expert IDs and weights are sufficient for the
full-row experts.

Add RouteRecord with six int32 IDs and six FP32 weight words per row.
Route the local block, allocate packing records inside the forward,
gather their BF16 bit views, and unpack into forward-local contiguous
operands. Release route declarations instead of retaining them as
external forward destinations.

An 8,192-row chunk transports 384 KiB of records instead of 12 MiB of
logits. Compilation can plan route temporaries without preserving
declaration buffers.

CPU record tests check IDs and weight bits against a row-local
reference. Recorder and weak-reference tests check live outputs,
released examples, poisoned declarations and independent invocation
destinations. CUDA transport, native sqrtsoftplus tie behavior, retained
memory and replay stability remain unqualified.

Design-Signed-Off-By: FujitsuPolycom; design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Decoder row transitions and route-record gathers need a complete
forwarding check; isolated geometry tests cannot detect incorrect
operand handoffs.

Run each rank's prepared decoder through a simulated collective fabric
with PyTorch references for GPU-only components. A nine-layer schedule
includes Engram, both row transitions, the decoder boundary and draft
feedback.

The test checks collective ordering and row geometry, gathered routes,
hidden rows, logits and draft feedback against full-row execution at two
and four ranks, with 512 rows per rank.

All four CPU cases pass. The decoder comparison requires relative
tensor-norm error below 5e-2 and relative error below 1e-1 for every
sampled row; route records are compared with the routes produced by the
simulated ranks. These references do not qualify compiled kernels,
complete cache state, native replay or service performance.

Design-Signed-Off-By: FujitsuPolycom; design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
…rows

A DeepSeek V4.1 prefill graph executes a fixed row capacity. Restricting
row-shard eligibility to whole multiples of 512 rows per rank leaves
other prepared capacities on full-row execution. Padding also produces
expert routes that no live token needs.

Accept per-rank row counts of 768, 1,280 and 1,792 in addition to whole
multiples of 512, enabling row-sharded execution for the 3,072-, 5,120-
and 7,168-row TP4 graphs without changing graph selection.
The decoder leaves the mHC per-call bound unset and runs each block as
one call. The component's optional bounded-call path uses
`row_call_bounds` to enumerate complete spans, including a remainder;
CPU cases explicitly exercise it with a 512-row bound. The row tables
include every accepted count. A snapshot-backed `offset_table` gives
each graph its rank's block offset as tensor data, so the compiled
forward need not specialize on a rank-local Python integer.

An attention-sharded prefill router receives the number of live tokens
inside its block. Rows past that count receive expert ID -1 and zero
weight, which route packing skips. Full-row routing, prefill without
attention sharding, decode, verification and draft retain their existing
routing behavior.

On four GB10 hosts over a RoCE ring, a separate DeepSeek V4.1 export
configuration with these bucket and router-mask changes measured
5,357 / 5,755 / 5,512 tok/s at the 16k / 64k / 128k workload labels,
versus 5,106 / 5,674 / 5,252 without them: observed differences of
+4.9%, +1.4% and +5.0%. The metric is client prompt tokens divided by
time to first token. Each comparison cell is one benchmark invocation
reporting the median of four request samples. Each buckets-and-masks
cell is the mean of four invocations, each reporting the median of
6–7 request samples after warmup. Their within-cell range/mean spreads
are 1.81%, 0.66% and 1.27%; the comparison has no repeated-invocation
variance estimate, so these observations do not establish statistical
significance. Actual 64k prompts contain 64,355–64,364 tokens and leave
a partial 8,192-row chunk. Serving KV capacity is 24.85M versus 25.58M
tokens per rank for the comparison and buckets-and-masks configurations.
This measurement uses a different source tree from the proposed branch
based on `97cfeb00`; it does not establish this commit's compiled GPU
correctness or native-service performance, or separate the effects of
the buckets and router masks.

CPU references check the optional bounded mHC call decomposition against
a padded reference, accepted row counts and table segments, block-offset
values across ranks, and router masking against a full-row reference.
A CPU Dynamo eager-backend test checks reuse of the live-count helper's
graph across rank-specific tensor values. The complete decoder reference
cases use 512 rows per rank and fully live chunks; they do not exercise
the tail capacities end to end. These checks execute no GPU kernel,
compiled complete GPU decoder, exported-core replay, or native service.

Design-Signed-Off-By: FujitsuPolycom; design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant