Repository navigation
Gather routes instead of router logits for row-sharded DeepSeek V4.1 prefill - #7
Closed
FujitsuPolycom wants to merge 3 commits into
Closed
FujitsuPolycom wants to merge 3 commits into
FujitsuPolycom wants to merge 3 commits into
Conversation
Gathering 384 FP32 router logits per row makes every rank repeat row-local top-k. Selected expert IDs and weights are sufficient for the full-row experts. Add RouteRecord with six int32 IDs and six FP32 weight words per row. Route the local block, allocate packing records inside the forward, gather their BF16 bit views, and unpack into forward-local contiguous operands. Release route declarations instead of retaining them as external forward destinations. An 8,192-row chunk transports 384 KiB of records instead of 12 MiB of logits. Compilation can plan route temporaries without preserving declaration buffers. CPU record tests check IDs and weight bits against a row-local reference. Recorder and weak-reference tests check live outputs, released examples, poisoned declarations and independent invocation destinations. CUDA transport, native sqrtsoftplus tie behavior, retained memory and replay stability remain unqualified. Design-Signed-Off-By: FujitsuPolycom; design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Decoder row transitions and route-record gathers need a complete forwarding check; isolated geometry tests cannot detect incorrect operand handoffs. Run each rank's prepared decoder through a simulated collective fabric with PyTorch references for GPU-only components. A nine-layer schedule includes Engram, both row transitions, the decoder boundary and draft feedback. The test checks collective ordering and row geometry, gathered routes, hidden rows, logits and draft feedback against full-row execution at two and four ranks, with 512 rows per rank. All four CPU cases pass. The decoder comparison requires relative tensor-norm error below 5e-2 and relative error below 1e-1 for every sampled row; route records are compared with the routes produced by the simulated ranks. These references do not qualify compiled kernels, complete cache state, native replay or service performance. Design-Signed-Off-By: FujitsuPolycom; design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
…rows A DeepSeek V4.1 prefill graph executes a fixed row capacity. Restricting row-shard eligibility to whole multiples of 512 rows per rank leaves other prepared capacities on full-row execution. Padding also produces expert routes that no live token needs. Accept per-rank row counts of 768, 1,280 and 1,792 in addition to whole multiples of 512, enabling row-sharded execution for the 3,072-, 5,120- and 7,168-row TP4 graphs without changing graph selection. The decoder leaves the mHC per-call bound unset and runs each block as one call. The component's optional bounded-call path uses `row_call_bounds` to enumerate complete spans, including a remainder; CPU cases explicitly exercise it with a 512-row bound. The row tables include every accepted count. A snapshot-backed `offset_table` gives each graph its rank's block offset as tensor data, so the compiled forward need not specialize on a rank-local Python integer. An attention-sharded prefill router receives the number of live tokens inside its block. Rows past that count receive expert ID -1 and zero weight, which route packing skips. Full-row routing, prefill without attention sharding, decode, verification and draft retain their existing routing behavior. On four GB10 hosts over a RoCE ring, a separate DeepSeek V4.1 export configuration with these bucket and router-mask changes measured 5,357 / 5,755 / 5,512 tok/s at the 16k / 64k / 128k workload labels, versus 5,106 / 5,674 / 5,252 without them: observed differences of +4.9%, +1.4% and +5.0%. The metric is client prompt tokens divided by time to first token. Each comparison cell is one benchmark invocation reporting the median of four request samples. Each buckets-and-masks cell is the mean of four invocations, each reporting the median of 6–7 request samples after warmup. Their within-cell range/mean spreads are 1.81%, 0.66% and 1.27%; the comparison has no repeated-invocation variance estimate, so these observations do not establish statistical significance. Actual 64k prompts contain 64,355–64,364 tokens and leave a partial 8,192-row chunk. Serving KV capacity is 24.85M versus 25.58M tokens per rank for the comparison and buckets-and-masks configurations. This measurement uses a different source tree from the proposed branch based on `97cfeb00`; it does not establish this commit's compiled GPU correctness or native-service performance, or separate the effects of the buckets and router masks. CPU references check the optional bounded mHC call decomposition against a padded reference, accepted row counts and table segments, block-offset values across ranks, and router masking against a full-row reference. A CPU Dynamo eager-backend test checks reuse of the live-count helper's graph across rank-specific tensor values. The complete decoder reference cases use 512 rows per rank and fully live chunks; they do not exercise the tail capacities end to end. These checks execute no GPU kernel, compiled complete GPU decoder, exported-core replay, or native service. Design-Signed-Off-By: FujitsuPolycom; design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
After NVIDIA#282 partitions prefill rows and NVIDIA#283 shards attention, ranks still gather all 384 FP32 router logits per row and repeat top-k. Full-row experts need only selected IDs and weights.
Change
Ranks now run local top-k, pack six int32 IDs and six FP32 weights into 48-byte
RouteRecords, gather BF16 bit views, and unpack contiguous expert operands. An 8,192-row chunk transfers 384 KiB instead of 12 MiB. Releasing route declarations lets compilation plan temporaries.Eligibility adds 768/1,280/1,792 rows per rank to multiples of 512: TP4 capacities 3,072/5,120/7,168. Other capacities remain full-row; graph selection is unchanged. DeepSeek V4.1 uses one unbounded mHC call per block; the optional 512-row bound also handles a 256-row remainder. Attention-sharded routing masks padding with ID -1/zero weight; other routing is unchanged.
A CPU decoder test with simulated collectives and PyTorch references for GPU-only components compares collective order, routes, hidden rows, logits and draft feedback against full-row execution at two and four ranks across nine layers, including Engram and row/decoder transitions.
Benefit and measurements
Combined NVIDIA#282/NVIDIA#283/NVIDIA#284 DeepSeek V4.1 results, four GB10 hosts/RoCE ring, 8,192-token chunks, prepared concurrency 16: client prompt tokens/TTFT for standalone cold-prefill after warmup, not aggregate concurrency-16 throughput.
Baseline: two requests/cell; stack: one invocation/cell, median of four requests. Stack revisions: NVIDIA#282
0ffb7ac1, NVIDIA#283335535a0, NVIDIA#284749e75a0; neither isolates NVIDIA#284 nor measures this97cfeb00-based head. KV tokens/rank: 24.9M versus 29.1M baseline, reflecting NVIDIA#283's replicated attention weights rather than the route-record change.Separate buckets/masks export: 5,357/5,755/5,512 tok/s versus 5,106/5,674/5,252 (+4.9%/+1.4%/+5.0%) at 16k/64k/128k, same metric/hardware. New cells average four invocations, each median of 6–7 warmed requests; range/mean spreads 1.81%/0.66%/1.27%. Comparison cells use one invocation, median of four; no repeated-invocation variance or statistical significance established. Actual 64k prompts: 64,355–64,364 tokens, partial 8,192-row chunk. KV capacity: comparison 24.85M versus buckets/masks 25.58M tokens/rank. Different source tree: no proposed-branch GPU/service qualification or isolation of buckets versus masks.
Evidence and limits
Separate recipe-default all-40-layer attention plus extra buckets: +9.1–13.8% throughput versus NVIDIA#282 alone. On 50 seeded prompts, position-0 KL: 3.3e-6/2.7e-6 versus NVIDIA#282's 3.1e-6/3.0e-6 and baseline-repeat floor 2.4e-6; all top-1 agree. Unidentified revision; no route-gather GPU/head qualification.
At
749e75a0: 382 CPU passes withb12x, including row-local route references/four end-to-end cases; two planning cases deselected. Regression over CLI and component/recipe files changed by the stack, with installedb12x/CUTLASS, not pinned dependencies: 946 passed/27 skipped/9 CLI failures; route-gather/decoder-test-only tree: 913 passed. Same nine failing IDs on both and base97cfeb00. Withoutb12x, both shared-file runs: 558 passed/167 failed/76 errors; tail adds nine failures, two new files cannot collect. GPU/checkpoint/service inputs absent.Tail tests cover live counts/reference routing, capacities, offsets, bounded mHC and Dynamo eager graph reuse. End-to-end simulation uses fully live chunks, 512 rows/rank, not tails. CUDA transport, production tail mHC, complete compiled GPU decoder and native replay remain unqualified.
Dependencies
NVIDIA#282 supplies row sharding; NVIDIA#283 supplies attention on that partition; NVIDIA#284 adds route gathering, CPU decoder coverage and tail masks/buckets. Merge NVIDIA#282 → NVIDIA#283 → NVIDIA#284.
Generated with Claude Code