Repository navigation
Compute DeepSeek V4.1 prefill attention on local query rows - #283
FujitsuPolycom wants to merge 2 commits into
Conversation
|
Categorized as design review for row-sharded DeepSeek V4.1 attention. Replicating the projections removes attention reductions but adds approximately 2.06 GB per rank in the described implementation; the combined run's KV capacity falls from 29.1M to 24.9M tokens. Review that memory/performance tradeoff and the numerical change from summing all output groups locally. #282 is the prerequisite, and #284 adds route gathering. The reported headline speedup belongs to that full stack. The 50-prompt parity result comes from an earlier build with different replication and bucket choices; retain that limitation and qualify the submitted composition across its eligible layers and boundaries. @FujitsuPolycom GitHub reports merge conflicts with
Please reconcile the branch with main and rerun the checks affected by the resolution. |
original-el8
left a comment
There was a problem hiding this comment.
Hold for numerical and complete-state qualification on the submitted revision before merging.
Replicating attention projections removes communication but changes reduction arithmetic and adds approximately 2.06 GB per rank at default settings. The combined stack's reported KV capacity falls from 29.1M to 24.9M tokens per rank. The 50-prompt whole-model parity result used a different build with different replication and bucket choices. The isolated output-projection test does not establish correctness of the submitted attention, cache-write, selection-transfer, and graph-transition composition.
Please validate compiled multi-rank prefill and exported-core native replay on the submitted/rebased composition, including partial chunks, full/sharded row transitions, complete cache state, prefix continuation, and decode/speculative transitions. Use numerical bounds justified by the operators and measured repeatability, and measure retained memory and allocation stability. AGENTS.md and docs/performance-work.md require validation of these affected contracts; an opt-in and a TODO do not complete that qualification.
Keep the performance attribution explicit: the reported 13.5–20.4% throughput improvement belongs to the combined #282/#283/#284 stack, not this PR alone.
Reviewer validation: 195 focused CPU tests passed on stack head 749e75a with pinned FlashInfer 7b35510b. I did not run GPU collectives or native serving. This branch conflicts with inspected main a5b3b2b in ignition/preparation/runtime_core_pb2.py and ignition/recipes/deepseek41/recipe.py. Review of this PR was scoped to its seven commits above #282.
Full-row hyperconnections repeat the same prefill work on every tensor-parallel rank. Reduce-scatter their branch outputs into contiguous rank-owned rows, normalize there, and gather the branch inputs needed by full-row consumers. Engram layers and decoder boundaries retain complete rows and continuation state. Add the explicit preparation option, fixed-shape shared collective components, row-transition scheduling, and preparation-only rebuild metadata. Regenerate the Python bindings from the combined schema while retaining current main's fields. The intended benefit is less repeated hyperconnection work without changing the serving operation families or the disabled-option behavior. Three focused CLI/planning/metadata round-trip tests pass. The broader Windows CLI selection has 205 passes and 41 failures; the same 41 failures reproduce on unmodified main in this environment, which lacks Heliopause and assumes different configuration paths. Syntax and staged whitespace checks pass. Current-default CUDA preparation and native serving have not been run on this head. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Tensor-parallel prefill gathers normalized rows before attention and reduces its output afterward. Compute all attention heads and output groups for each rank's own rows, then gather the cache projections required by full-row consumers. This removes those exchanges while preserving complete cache writes. Load replicated query/output projections only for eligible prefill layers, retain partitioned decode copies, and carry selection state across row transitions. Account for values, both retained MXFP8 scale layouts and FP32 sinks: 17 layers add 2,121,273,600 numeric bytes per rank. Preserve main's metadata and online-conversion options. Document the qualified hardware and retained-core scope separately from unrun dependency/mode configurations. The benefit is less attention communication at a retained-weight cost. Grouping all eight output groups changes BF16 rounding association; comparisons use operator-derived bounds rather than exact partitioned-output equality. Validation includes accounting/layout tests and combined-stack GPU/native checks. On 32 captured native rows, all 163,840 projection outputs satisfy the unchanged operator budget and match native BF16 bits. Existing 7b35510b/ABI-14 cores pass checkpoint/Restore, inactive-state, replay and allocation checks; 4fd836dc component/compiler checks pass separately. Floating cache differences, baseline-reproduced replay failures and measured dependency scopes remain explicit; no fresh default-dependency whole-model export or isolated speedup is claimed. Computational source is unchanged from the reviewed 8d38f298 tree. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
335535a to
da5ec45
Compare
|
The update at The existing baseline and combined-stack exports use FlashInfer On 32 actual native prefill rows, the eight-group projection matched native BF16 bits and all 163,840 outputs satisfied the unchanged operator rounding budget (maximum ratio 0.897331). It was closer to the shared-operand FP64-rounded reference than the partitioned BF16 composition. Layer-1 cache differences exceed measured baseline variation. Complete checkpoint/Restore and inactive-cache preservation were validated separately. Longer exact-ID failures also reproduce on baseline. Both existing-core checkpoint/Restore/replay runs and the resident-inactive case passed on all four ranks. Whole-model execution checks used the existing FlashInfer |
Why
After row-sharded prefill, attention still gathers a full input and computes only each rank's head partition. Keeping attention on the rank's existing query rows can remove that gather and the attention-output reduction.
What
The advanced, default-off
--row-shard-attentionexport option requires--row-shard-prefill. Eligible layers retain all attention heads/output groups beside the partitioned projections used by decode. Each rank computes complete attention for its query rows, then gathers latent/compression operands before full-row cache writes. The grouped output projection sums eight groups before BF16 rounding; the partitioned path rounds four partials before FP32 rank reduction, so bitwise numerical equality is not promised.The retained-weight estimate includes values, both MXFP8 scale layouts and FP32 sinks. This corrects weight planning without changing attention arithmetic; existing cores retain their recorded capacities. Changing the attention layout requires export. Separate tail-capacity/live-row-mask proposals are excluded.
Benefit
The expected benefit is removing two attention collectives, at an explicit weight cost: 17 layers add 2,121,273,600 numeric bytes per rank. No isolated service-throughput gain is established. The generic snapshot-selector CPU fix should land with or before deployment; it is neither a compilation dependency nor a model-PR gain.
Validation and limits
On four GB10 hosts, combined implementation
36c13197on main7413a949, FlashInfer4fd836dcand Transformers 5.23 passes real route/image tests and 54 compiled/uncompiled reduce-scatter/gather checks per rank. Nine compiler-boundary cases pass. CPU results are 460 passes, two collective-ladder failures reproduced on unchanged main and one platform skip.A FlashInfer
7b35510bdiagnostic-core projection check captures 32 actual native rows. All isolated eight-group outputs match native BF16 bits; quantized intermediates match the partitioned composition. All 163,840 sampled outputs satisfy the unchanged rounding budget, maximum ratio 0.897331. FP64-reference RMS error is lower for eight groups: 0.0001284 versus 0.0023506 for the partitioned composition. Unsampled filler rows are excluded. Layer-1 cache differences exceed baseline repeat variation; the projection check bounds that changed operator, while cache preservation is tested separately. Longer seeded-ID failures also reproduce on the unchanged baseline.Native evidence uses signed FlashInfer
7b35510b/ABI14 baselinegb10-tp4-202611140833and combined treatmentgb10-tp4-202611140129(preparation sources6c3f9d6bfor baseline and7522a7ecfor treatment). The weight-estimate correction changes planning, not their captured computation. Native qualification with readerc71be653passed baseline replay, treatment replay and treatment resident-inactive preservation, including complete checkpoint/Restore and endpoint-state checks. Profiling uses concurrency 1, KV context 1,048,576 and reserve 4 GiB; both cores retain 8,192 pages/105 state rows/99 snapshots; 11 observed root-rank execution intervals contain zero allocation-management calls. Measured combined device-backing deltas are 2,468,347,904 bytes on ranks 0/1 and 2,384,461,824 on ranks 2/3; registered-host spans remain 379,064,320 bytes per rank. Registered spans exclude other host storage. These deltas differ from the numeric weight cost and are not peak or isolated-PR measurements. Whole-model execution checks use the retained7b35510b/ABI-14 cores;4fd836dcvalidation covers the component/compiler boundaries. No new throughput result is claimed.Prepared with AI assistance.