Repository navigation
Gather selected routes for row-sharded DeepSeek V4.1 prefill - #284
FujitsuPolycom wants to merge 3 commits into
Conversation
|
Categorized as kernel performance work needing validation. This PR replaces the row-sharded router-logit gather with gathered expert IDs and weights; it depends on #283 and #282. The repeated stack files are dependencies, not evidence that the three PRs are redundant. The reported 13.5–20.4% prefill gain is for the full stack. The exact record-gather path has CPU simulated-fabric coverage but no CUDA case; add real collective validation of IDs, FP32 weight bits, partial rows and graph transitions before attributing correctness to that simulation. Retarget or otherwise isolate the incremental diff after the prerequisite changes land. @FujitsuPolycom GitHub reports merge conflicts with
Please reconcile the branch with main and rerun the checks affected by the resolution. |
original-el8
left a comment
There was a problem hiding this comment.
Request changes for the image-mask preparation failure and the ineffective routing-agreement assertion detailed below.
The route-record layout is a reasonable way to reduce communication, but the submitted validation does not qualify its GPU composition. The CPU decoder test replaces attention, routing, experts, and transport with CPU implementations; it cannot establish native routing, collective, cache-state, or exported replay correctness. Its router also substitutes softmax for DeepSeek V4.1's sqrtsoftplus and uses different tie ordering. Add focused validation of the real preparation contract and qualify compiled GPU route gathering and native replay with the #282/#283 prerequisites.
The reported 13.5–20.4% throughput gain belongs to the full stack. The cited whole-model parity run used an earlier build, so it does not establish numerical correctness of this submitted revision.
Reviewer validation: 195 focused CPU tests passed at 749e75a with pinned FlashInfer 7b35510b. The image-mask rejection was reproduced through the actual router preparation callback and b12x validation using CUDA-shaped fake tensors; no GPU kernel or complete export was executed. I did not run GPU collectives or native serving. The branch conflicts with inspected main a5b3b2b in ignition/preparation/runtime_core_pb2.py and ignition/recipes/deepseek41/recipe.py. Review of this PR was scoped to its two commits above #283.
[P1] Prepare the image mask with the router's shard row count
When attention_shard is enabled and the checkpoint has vision correction weights, router_logits and route_scratch have row_shard.rows rows, but self.image_mask was allocated from the full metadata.token_ids shape. PreparedRouter captures this full mask in its preparation callback. At TP4 with a 4,096-row prefill, the callback passes 1,024 logit rows and a 4,096-element mask to b12x, which rejects it with ValueError: image_mask must have shape (num_tokens,) before export can complete. The shard slicing in _run only fixes execution-time operands; preparation fails earlier.
I reproduced the rejection through PreparedRouter's preparation callback and the pinned b12x _RouteTopKState using CUDA-shaped fake tensors. Size the declaration mask to the rows this router prepares and add a regression that exercises the image-enabled preparation callback, including a nonzero rank.
Location: ignition/recipes/deepseek41/decoder.py:477.
[P2] Make the routing-margin assertion check reachable rows
ReferenceRouter records the difference between the second- and third-largest softmax probabilities because this test uses TOP_K = 2. The second-largest probability is at most 0.5, so this margin can never exceed 0.5. agree[margin > 0.5] is always empty and .all() always passes. The remaining >0.8 agreement check can therefore accept nearly 20% route disagreement without the advertised margin-based protection.
Use a meaningful margin/error criterion with a nonempty-coverage assertion and a regression showing that an incorrect route fails. For model qualification, preserve the actual sqrtsoftplus scoring and native tie ordering rather than treating this softmax substitute and its broad output bounds as numerical equivalence evidence.
Location: tests/recipes/test_deepseek41_row_shard_prefill_e2e.py:884.
Full-row hyperconnections repeat the same prefill work on every tensor-parallel rank. Reduce-scatter their branch outputs into contiguous rank-owned rows, normalize there, and gather the branch inputs needed by full-row consumers. Engram layers and decoder boundaries retain complete rows and continuation state. Add the explicit preparation option, fixed-shape shared collective components, row-transition scheduling, and preparation-only rebuild metadata. Regenerate the Python bindings from the combined schema while retaining current main's fields. The intended benefit is less repeated hyperconnection work without changing the serving operation families or the disabled-option behavior. Three focused CLI/planning/metadata round-trip tests pass. The broader Windows CLI selection has 205 passes and 41 failures; the same 41 failures reproduce on unmodified main in this environment, which lacks Heliopause and assumes different configuration paths. Syntax and staged whitespace checks pass. Current-default CUDA preparation and native serving have not been run on this head. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Tensor-parallel prefill gathers normalized rows before attention and reduces its output afterward. Compute all attention heads and output groups for each rank's own rows, then gather the cache projections required by full-row consumers. This removes those exchanges while preserving complete cache writes. Load replicated query/output projections only for eligible prefill layers, retain partitioned decode copies, and carry selection state across row transitions. Account for values, both retained MXFP8 scale layouts and FP32 sinks: 17 layers add 2,121,273,600 numeric bytes per rank. Preserve main's metadata and online-conversion options. Document the qualified hardware and retained-core scope separately from unrun dependency/mode configurations. The benefit is less attention communication at a retained-weight cost. Grouping all eight output groups changes BF16 rounding association; comparisons use operator-derived bounds rather than exact partitioned-output equality. Validation includes accounting/layout tests and combined-stack GPU/native checks. On 32 captured native rows, all 163,840 projection outputs satisfy the unchanged operator budget and match native BF16 bits. Existing 7b35510b/ABI-14 cores pass checkpoint/Restore, inactive-state, replay and allocation checks; 4fd836dc component/compiler checks pass separately. Floating cache differences, baseline-reproduced replay failures and measured dependency scopes remain explicit; no fresh default-dependency whole-model export or isolated speedup is claimed. Computational source is unchanged from the reviewed 8d38f298 tree. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Row-sharded attention owns each rank's query rows, but gathering all 384 router logits repeats communication and top-k work. Gather six selected expert IDs and their FP32 weight bits in a 48-byte record, then unpack contiguous expert operands without numeric conversion. Prepare the router image mask at the rank-local row count, including nonzero ranks, while retaining the full layer mask and runtime selection. Use sqrtsoftplus and larger-ID ties in the reference, require nonempty margin coverage, and reject incorrect routes on ambiguous rows. Preserve main's optional activation-mode forwarding and explicit scratch operands. The change reduces logical gathered route payload by 32 times at top-k six. It introduces no independent runtime policy or isolated service-gain claim; route gathering follows opt-in row-sharded attention. Validation: four-host GB10 GPU route-record and image-router preparation regressions pass with FlashInfer 4fd836dc, including exact IDs/FP32 bits, partial/padded rows, ties and compiled graph transitions. Combined-stack existing-core native cache/replay checks retain their 7b35510b/ABI-14 artifact scope. CPU reference tests reject wrong routes, empty coverage and mask shape errors. Model numerical limits and older baseline replay failures remain documented; no fresh default-dependency whole-model export is claimed. The separate tail-capacity/masking proposal is excluded. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
749e75a to
885848f
Compare
|
The update at Four-host GB10 GPU tests with FlashInfer With native reader The combined-stack numerical caveats remain: floating cache differences and longer exact-ID failures are documented, including baseline controls. Whole-model execution checks used the existing |
Why
With row-sharded attention, each rank produces router logits for its own rows, but gathering all 384 FP32 logits repeats top-k selection on every rank. Expert execution needs only the selected IDs and weights. The change gathers those operands directly.
What
Each rank selects routes locally, packs six int32 expert IDs and six FP32 weights into a 48-byte
RouteRecord, then gathers a BF16 bit-view and unpacks contiguous expert operands. The transport preserves weight bits rather than numerically converting them. Temporary route declarations are released so compiled forwards own their storage.Router preparation receives the rank-local image-mask slice, fixing the full-mask/sharded-logit shape failure on nonzero ranks. The reference uses actual sqrtsoftplus scoring and larger-ID ties, requires nonempty separated-margin coverage and rejects deliberately corrupted routes, including ambiguous rows.
This follows the attention change; it adds no independent flag. It applies when the default-off
--row-shard-prefilland--row-shard-attentionlayouts are exported. Full-row/default execution remains available. Separate tail-capacity/live-row-mask proposals are excluded.Benefit
At 8,192 rows, gathered route records occupy 384 KiB instead of 12 MiB of logits: 32 times less logical payload, and each row is selected once instead of on every rank. No isolated service gain is established; combined-stack measurements cannot attribute gains to this PR.
The generic snapshot-selector CPU fix should land with or before deployment. It is a runtime companion, not a model compilation dependency or performance credit for this change.
Validation and limits
Four-host GB10 tests at combined implementation
36c13197on main7413a949, FlashInfer4fd836dcand Transformers 5.23 verify exact native IDs/FP32 weight bits through compiled/uncompiled capture, partial/padded rows, ties and graph transitions. The image case exercises the actual preparation callback with text/mixed/image masks on nonzero ranks; whole-model vision generation remains untested. The 54-replay-per-rank reduce-scatter/gather comparison and nine compiler-boundary cases also pass. CPU checks report 460 passes, two collective-ladder failures reproduced on unchanged main and one platform skip.Whole-model execution checks use signed FlashInfer
7b35510b/ABI14 cores: full-row baselinegb10-tp4-202611140833and combined treatmentgb10-tp4-202611140129(preparation sources6c3f9d6bfor baseline and7522a7ecfor treatment). Native qualification with readerc71be653passed baseline replay, treatment replay and treatment resident-inactive preservation, including complete checkpoint/Restore, endpoint-state checks and matching eight-output replay. Profiling uses concurrency 1, KV context 1,048,576 and reserve 4 GiB; both cores retain 8,192 pages/105 state rows/99 snapshots; 11 observed root-rank execution intervals have zero allocation-management calls. Combined device-backing deltas are 2,468,347,904 bytes on ranks 0/1 and 2,384,461,824 on ranks 2/3; registered-host spans stay at 379,064,320 bytes per rank, not total host storage or peak memory.The route-record comparisons require exact IDs and FP32 weight bits. Layer-1 cache changes exceed baseline variation, and longer seeded-ID failures also reproduce on the baseline. No fresh whole-model artifact prepared with FlashInfer
4fd836dcor new throughput result is claimed.Prepared with AI assistance.