Skip to content

Compute DeepSeek V4.1 prefill attention on local query rows - #283

Open
FujitsuPolycom wants to merge 2 commits into
NVIDIA:mainfrom
FujitsuPolycom:rowshard-attention-v010
Open

FujitsuPolycom wants to merge 2 commits into
NVIDIA:mainfrom
FujitsuPolycom:rowshard-attention-v010

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

After row-sharded prefill, attention still gathers a full input and computes only each rank's head partition. Keeping attention on the rank's existing query rows can remove that gather and the attention-output reduction.

What

The advanced, default-off --row-shard-attention export option requires --row-shard-prefill. Eligible layers retain all attention heads/output groups beside the partitioned projections used by decode. Each rank computes complete attention for its query rows, then gathers latent/compression operands before full-row cache writes. The grouped output projection sums eight groups before BF16 rounding; the partitioned path rounds four partials before FP32 rank reduction, so bitwise numerical equality is not promised.

The retained-weight estimate includes values, both MXFP8 scale layouts and FP32 sinks. This corrects weight planning without changing attention arithmetic; existing cores retain their recorded capacities. Changing the attention layout requires export. Separate tail-capacity/live-row-mask proposals are excluded.

Benefit

The expected benefit is removing two attention collectives, at an explicit weight cost: 17 layers add 2,121,273,600 numeric bytes per rank. No isolated service-throughput gain is established. The generic snapshot-selector CPU fix should land with or before deployment; it is neither a compilation dependency nor a model-PR gain.

Validation and limits

On four GB10 hosts, combined implementation 36c13197 on main 7413a949, FlashInfer 4fd836dc and Transformers 5.23 passes real route/image tests and 54 compiled/uncompiled reduce-scatter/gather checks per rank. Nine compiler-boundary cases pass. CPU results are 460 passes, two collective-ladder failures reproduced on unchanged main and one platform skip.

A FlashInfer 7b35510b diagnostic-core projection check captures 32 actual native rows. All isolated eight-group outputs match native BF16 bits; quantized intermediates match the partitioned composition. All 163,840 sampled outputs satisfy the unchanged rounding budget, maximum ratio 0.897331. FP64-reference RMS error is lower for eight groups: 0.0001284 versus 0.0023506 for the partitioned composition. Unsampled filler rows are excluded. Layer-1 cache differences exceed baseline repeat variation; the projection check bounds that changed operator, while cache preservation is tested separately. Longer seeded-ID failures also reproduce on the unchanged baseline.

Native evidence uses signed FlashInfer 7b35510b/ABI14 baseline gb10-tp4-202611140833 and combined treatment gb10-tp4-202611140129 (preparation sources 6c3f9d6b for baseline and 7522a7ec for treatment). The weight-estimate correction changes planning, not their captured computation. Native qualification with reader c71be653 passed baseline replay, treatment replay and treatment resident-inactive preservation, including complete checkpoint/Restore and endpoint-state checks. Profiling uses concurrency 1, KV context 1,048,576 and reserve 4 GiB; both cores retain 8,192 pages/105 state rows/99 snapshots; 11 observed root-rank execution intervals contain zero allocation-management calls. Measured combined device-backing deltas are 2,468,347,904 bytes on ranks 0/1 and 2,384,461,824 on ranks 2/3; registered-host spans remain 379,064,320 bytes per rank. Registered spans exclude other host storage. These deltas differ from the numeric weight cost and are not peak or isolated-PR measurements. Whole-model execution checks use the retained 7b35510b/ABI-14 cores; 4fd836dc validation covers the component/compiler boundaries. No new throughput result is claimed.

Prepared with AI assistance.

@original-el8 original-el8 added area:kernels GPU kernels and numerical execution. area:memory Memory budgets, allocation lifetimes, scratch sharing, and capacity. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. enhancement New feature or request model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:design-review Needs a maintainer scope, architecture, or contract decision before implementation or integration. labels Nov 13, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

Categorized as design review for row-sharded DeepSeek V4.1 attention. Replicating the projections removes attention reductions but adds approximately 2.06 GB per rank in the described implementation; the combined run's KV capacity falls from 29.1M to 24.9M tokens. Review that memory/performance tradeoff and the numerical change from summing all output groups locally.

#282 is the prerequisite, and #284 adds route gathering. The reported headline speedup belongs to that full stack. The 50-prompt parity result comes from an earlier build with different replication and bucket choices; retain that limitation and qualify the submitted composition across its eligible layers and boundaries.

@FujitsuPolycom GitHub reports merge conflicts with main, confirmed locally with git merge-tree for PR head 335535a086c8 against main 97cfeb0001f1. Conflicting paths:

  • ignition/preparation/runtime_core_pb2.py

Please reconcile the branch with main and rerun the checks affected by the resolution.

@original-el8 original-el8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold for numerical and complete-state qualification on the submitted revision before merging.

Replicating attention projections removes communication but changes reduction arithmetic and adds approximately 2.06 GB per rank at default settings. The combined stack's reported KV capacity falls from 29.1M to 24.9M tokens per rank. The 50-prompt whole-model parity result used a different build with different replication and bucket choices. The isolated output-projection test does not establish correctness of the submitted attention, cache-write, selection-transfer, and graph-transition composition.

Please validate compiled multi-rank prefill and exported-core native replay on the submitted/rebased composition, including partial chunks, full/sharded row transitions, complete cache state, prefix continuation, and decode/speculative transitions. Use numerical bounds justified by the operators and measured repeatability, and measure retained memory and allocation stability. AGENTS.md and docs/performance-work.md require validation of these affected contracts; an opt-in and a TODO do not complete that qualification.

Keep the performance attribution explicit: the reported 13.5–20.4% throughput improvement belongs to the combined #282/#283/#284 stack, not this PR alone.

Reviewer validation: 195 focused CPU tests passed on stack head 749e75a with pinned FlashInfer 7b35510b. I did not run GPU collectives or native serving. This branch conflicts with inspected main a5b3b2b in ignition/preparation/runtime_core_pb2.py and ignition/recipes/deepseek41/recipe.py. Review of this PR was scoped to its seven commits above #282.

Full-row hyperconnections repeat the same prefill work on every tensor-parallel
rank. Reduce-scatter their branch outputs into contiguous rank-owned rows,
normalize there, and gather the branch inputs needed by full-row consumers.
Engram layers and decoder boundaries retain complete rows and continuation state.

Add the explicit preparation option, fixed-shape shared collective components,
row-transition scheduling, and preparation-only rebuild metadata. Regenerate the
Python bindings from the combined schema while retaining current main's fields.
The intended benefit is less repeated hyperconnection work without changing the
serving operation families or the disabled-option behavior.

Three focused CLI/planning/metadata round-trip tests pass. The broader Windows
CLI selection has 205 passes and 41 failures; the same 41 failures reproduce on
unmodified main in this environment, which lacks Heliopause and assumes different
configuration paths. Syntax and staged whitespace checks pass. Current-default
CUDA preparation and native serving have not been run on this head.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Tensor-parallel prefill gathers normalized rows before attention and reduces its
output afterward. Compute all attention heads and output groups for each rank's
own rows, then gather the cache projections required by full-row consumers.
This removes those exchanges while preserving complete cache writes.

Load replicated query/output projections only for eligible prefill layers,
retain partitioned decode copies, and carry selection state across row
transitions. Account for values, both retained MXFP8 scale layouts and FP32
sinks: 17 layers add 2,121,273,600 numeric bytes per rank. Preserve main's
metadata and online-conversion options. Document the qualified hardware and
retained-core scope separately from unrun dependency/mode configurations.

The benefit is less attention communication at a retained-weight cost. Grouping
all eight output groups changes BF16 rounding association; comparisons use
operator-derived bounds rather than exact partitioned-output equality.

Validation includes accounting/layout tests and combined-stack GPU/native
checks. On 32 captured native rows, all 163,840 projection outputs satisfy the
unchanged operator budget and match native BF16 bits. Existing 7b35510b/ABI-14
cores pass checkpoint/Restore, inactive-state, replay and allocation checks;
4fd836dc component/compiler checks pass separately. Floating cache differences,
baseline-reproduced replay failures and measured dependency scopes remain
explicit; no fresh default-dependency whole-model export or isolated speedup
is claimed. Computational source is unchanged from the reviewed 8d38f298 tree.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
@FujitsuPolycom
FujitsuPolycom force-pushed the rowshard-attention-v010 branch from 335535a to da5ec45 Compare November 15, 2026 14:57
@FujitsuPolycom FujitsuPolycom changed the title Row-shard DeepSeek V4.1 prefill attention so each rank attends only its rows Compute DeepSeek V4.1 prefill attention on local query rows Nov 15, 2026
@FujitsuPolycom

Copy link
Copy Markdown
Contributor Author

The update at da5ec45c includes both retained MXFP8 scale layouts and FP32 sinks in the estimate: 2,121,273,600 additional numeric-storage bytes per rank across 17 layers. This corrects planning without changing captured computation. This path is opt-in at export through --row-shard-attention, which requires --row-shard-prefill; both are off by default.

The existing baseline and combined-stack exports use FlashInfer 7b35510b/ABI 14. Native loads with reader c71be653 measured an additional 2.47 GB on ranks 0/1 and 2.38 GB on ranks 2/3 for the combined stack. Both arms used concurrency 1, a 1,048,576-token KV context and 4 GiB reserve, with identical cache capacities and registered-host spans. Retained backings stayed unchanged through replay; traces showed no device-allocation activity between ready and drain. Fixed host startup buffers were freed at close. These are retained-backing costs, not peak memory or isolated #283 costs.

On 32 actual native prefill rows, the eight-group projection matched native BF16 bits and all 163,840 outputs satisfied the unchanged operator rounding budget (maximum ratio 0.897331). It was closer to the shared-operand FP64-rounded reference than the partitioned BF16 composition. Layer-1 cache differences exceed measured baseline variation. Complete checkpoint/Restore and inactive-cache preservation were validated separately. Longer exact-ID failures also reproduce on baseline.

Both existing-core checkpoint/Restore/replay runs and the resident-inactive case passed on all four ranks. Whole-model execution checks used the existing FlashInfer 7b35510b/ABI 14 cores; the 4fd836dc checks covered the dependency/compiler boundaries. Throughput measurements retain their original revision scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:kernels GPU kernels and numerical execution. area:memory Memory budgets, allocation lifetimes, scratch sharing, and capacity. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. enhancement New feature or request model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:design-review Needs a maintainer scope, architecture, or contract decision before implementation or integration.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants