Repository navigation
Shorten Inkling 2 prefill and drafting rounds on DGX Spark TP4 - #311
Closed
original-el8 wants to merge 4 commits into
Closed
original-el8 wants to merge 4 commits into
original-el8 wants to merge 4 commits into
Conversation
When `flare run` starts a multi-host core, rank zero reads every peer's rank
core through the page cache and sends it over TCP. On a host whose GPU
allocates from system memory, such as a DGX Spark (GB10), those cached pages
are not free memory for CUDA, so the pages of several rank cores can keep rank
zero from creating its CUDA context. Helios already releases the page cache of
a rank's own core files before it creates the context
(Files::release_page_cache in helios/src/runtime_core/model.rs), and a
receiving rank syncs the files it writes; the files rank zero reads on behalf
of other ranks were never released. Releasing them where the distribution
source stops closes that gap with the same eos operation and needs no change
to loading or to the transfer protocol.
Before this change the distribution source in flare-cli/src/distribution.rs
returned when its serving threads had joined and left the pages it had read in
the cache. Now release_served_pages calls flare_eos::release_page_cache
(POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one
and above when the source thread finishes, whether distribution succeeded or
failed. Rank zero's own files are left to the existing release in Helios.
flare-cli gains a path dependency on the in-tree eos crate, which the lock
file records.
The result is that a rank-zero host returns the memory used to read peer
cores before GPU initialization, without a manual page-cache drop.
Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host)
serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank
zero distributed three rank cores, its host showed 114 GiB of page cache and
2 GiB free, and rank zero failed at "Initialize GPU" with CudaError {
operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The
failure was intermittent: the first launch of a smaller core went through the
same distribution without it. After, with rebuilt binaries on all four hosts,
the peers' rank cores removed and no manual cache drop: three rank cores were
sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed
MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded.
The Linux test
source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a
4 MiB on-disk file to one peer and checks with mincore that the file has
resident pages after the transfer and none after the source finishes.
cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30
passed, 0 failed, 2 ignored (they need external fixtures). Not run: other
models, two- and three-rank cores, and hosts with discrete GPUs, where the
release changes only the page cache.
Design-Signed-Off-By: flare-02; design-review=ed22
…nd norm Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms 8,192-row chunk in the expert router's projection, which widened its BF16 source and weight to FP32 so that only b12x's FP32-operand kernels were eligible, and 376 ms in attention-residual mixtures that stored an FP32 mixture per stream for a separate RMSNorm launch. The router needs FP32 accumulation, not FP32 operands: the product of two BF16 values is exact in FP32. The mixture and its norm read and write the same rows, so one program can normalize without storing the mixture. The router projection takes its BF16 source and a BF16 router weight and declares accumulation="fp32", which UnquantizedProjection passes to b12x so that only plans carrying the whole sum in FP32 are raced. The weight loader delivers the FP32 checkpoint tensor as BF16 only after checking that every value is an exact BF16 upcast. PreparedResidualMix normalizes each stream in the mixture kernel with FP32 mean square, scale and weight and one rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention writes its partial sum in the cross-rank reduction's element type, BF16 over RoCEnante and FP32 on one host, in place of FP32 followed by a cast; decode and verification keep FP32. Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to 1.6. Router selection is unchanged in value class: operands are the same numbers and only summation order differs. The fused norm differs from the unfused one by at most one BF16 unit in at most ten elements per million. Evidence: one exported core per tree served by flare run at TP4 on GB10, 8,192-row chunks, warm and uncached, three repeats. Time to first token was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to 23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s). The measured core also contained FlashInfer's swapped-operand FC1 selection and the drafting changes, so the factor is not attributable to this commit alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the same top-8 experts on every row. tests/recipes/test_inkling2_target.py passes 87 tests on a GB10 with the checkpoint, including attention against the reference in prefill, decode and verification and the fused-norm bound. Limitations: the projection requires GemvQuery.accumulation from FlashInfer (local-inference-lab/flashinfer#110), which the pinned FlashInfer revision lacks. Completion length was compared between a core from this tree with the drafting changes and a core from the unmodified tree: over lookup prompts with seeds 1 to 30, median completion tokens were 363 against 420 and the share at or above 1,024 tokens differed by -1.7 points (95% interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses. tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and 5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not run. Design-Signed-Off-By: flare-02; design-review=ed22
… formats On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes. Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted. A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank. Evidence: one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch. Limitations: verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. Design-Signed-Off-By: flare-02; design-review=ed22
Between two cross-host reductions, an Inkling 2 decoder layer computes stages whose result for a row depends on that row or on the three rows before it: the attention-residual mixtures with their norms, the output convolutions, the expert router and the residual additions. In tensor- parallel prefill every rank computed them for every row of the chunk. On one GB10 they take 773 ms of an 8,192-row chunk, and four ranks produce four identical copies. RoCEnante's pipelined all-reduce is a reduce- scatter followed by a shard gather, so a rank can keep its quarter of the rows after the first half, compute these stages for those rows only, and gather where every rank needs every row, at the wire bytes of the all-reduce it replaces. Prefill graphs of at least 2,048 rows on ranks that reduce over RoCEnante shard their rows (SHARDED_PREFILL_ROWS in the Inkling 2 decoder). PreparedReduceScatter and PreparedShardGather in ignition/components/collectives.py prepare the two halves of the pipelined all-reduce as separate captured operations. ignition/components/ row_shards.py holds row ownership, each rank's view of the batch metadata, a column gather that exchanges the three rows before each rank's rows and each request's last three rows after every reduce-scatter (the output convolutions and their replicated histories read them), and PreparedRouteGather, which carries each row's int32 expert IDs and FP32 route weights as bytes in BF16 lanes. The decoder's layer loop and final mixture move into PreparedStack; in a sharded graph the residual state holds a rank's rows, and shard gathers supply the attention input streams, the expert input, the drafter taps and the final normalized rows. Row ownership follows graph capacity, so a rank whose rows are padding computes inactive rows. Smaller prefill graphs, decode, verification and ranks on one host are unchanged. Helios and the package contract are unchanged; a core gains sharded prefill when it is exported again. A sharded 8,192-row chunk computes the row-local stages once across the four ranks and runs 361 collectives where the replicated chunk runs 97. Every row-local value, route, gathered row and convolution history equals the replicated forward bit for bit. Evidence: on one GB10 with the checkpoint, the row-local stages of 48 layers take 175.7 ms for a rank's shard against 773.4 ms for 8,192 rows, and 37.5 ms against 173.3 ms at 2,048 rows, with no differing element. tests/components/test_row_shards.py (41 tests) covers ownership, halo and tail selection and capacity checks; tests/recipes/ test_inkling2_row_shards_cuda.py (6 tests) compares four ranks' sharded forwards of a five-layer stack with the replicated stack through an in-process exchange, eagerly and through a compiled, captured and replayed forward, on an RTX PRO 6000 and a GB10; tests/recipes/ test_inkling2_target.py passes 87 tests on a GB10. One core exported from this tree with FlashInfer 4fd836dc plus the FlashInfer changes this branch's parents require served at TP4 on four DGX Sparks with 8,192-row chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests complete. Over three service launches, time to first token was 2.07 to 2.34 s at 7,668 prompt tokens, 18.4 to 20.1 s at 67,145 and 40.7 to 43.1 s at 131,990, three repeats each; one launch of the core exported from the parent commit measured 2.59 to 2.69 s, 22.4 to 23.0 s and 48.8 to 49.8 s. In kernel traces of rank 0, a 7,168-row chunk spans 1,809 ms sharded against 2,034 ms replicated: collectives 586 ms against 508 ms, mixtures and norms 70 ms against about 216 ms. Limitations: a fourth launch of the sharded core ran every graph, decode included, about 10% slower (2.85 to 2.96 s and 48.8 to 53.8 s) for a cause that was not identified, and the replicated core was measured on one launch, so the serving comparison does not exclude a launch effect of that size. Capacities between 2,048 and 8,192 rows are not measured in serving; the threshold is an estimate. Design-Signed-Off-By: flare-02; design-review=ed22
original-el8
force-pushed
the
inkling2/token-sharded-row-stages
branch
from
November 15, 2026 17:38
1a492d2 to
c49ae59
Compare
This was referenced Nov 15, 2026
Collaborator
Author
|
Superseded by the consolidated Inkling 2 DGX Spark performance pull request opened from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inkling 2 Small NVFP4 at TP4 on four DGX Sparks spent most of a prefill chunk outside its attention and expert kernels and paid about 11 ms per drafting round for no tokens. This change shortens prefill and drafting rounds in three commits, each described below with its own reasoning, change, result and evidence, in commit order. The base branch is
inkling2/dgx-spark-tp4, which enables the recipe across hosts.The commits require three FlashInfer changes that Flare's pinned revision lacks: local-inference-lab/flashinfer#110 (FP32 accumulation for BF16 projections), #111 (swapped-operand NVFP4 FC1 for 32-aligned widths) and #112 (window staging lookback). This is a draft until the pin can move to a merged FlashInfer commit.
1. Shorten Inkling 2 prefill with BF16 router operands and a fused mix and norm
Why
Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms 8,192-row chunk in the expert router's projection, which widened its BF16 source and weight to FP32 so that only b12x's FP32-operand kernels were eligible, and 376 ms in attention-residual mixtures that stored an FP32 mixture per stream for a separate RMSNorm launch. The router needs FP32 accumulation, not FP32 operands: the product of two BF16 values is exact in FP32. The mixture and its norm read and write the same rows, so one program can normalize without storing the mixture.
Change
The router projection takes its BF16 source and a BF16 router weight and declares accumulation="fp32", which UnquantizedProjection passes to b12x so that only plans carrying the whole sum in FP32 are raced. The weight loader delivers the FP32 checkpoint tensor as BF16 only after checking that every value is an exact BF16 upcast. PreparedResidualMix normalizes each stream in the mixture kernel with FP32 mean square, scale and weight and one rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention writes its partial sum in the cross-rank reduction's element type, BF16 over RoCEnante and FP32 on one host, in place of FP32 followed by a cast; decode and verification keep FP32.
Result
Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to 1.6. Router selection is unchanged in value class: operands are the same numbers and only summation order differs. The fused norm differs from the unfused one by at most one BF16 unit in at most ten elements per million.
Evidence
one exported core per tree served by flare run at TP4 on GB10, 8,192-row chunks, warm and uncached, three repeats. Time to first token was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to 23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s). The measured core also contained FlashInfer's swapped-operand FC1 selection and the drafting changes, so the factor is not attributable to this commit alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the same top-8 experts on every row. tests/recipes/test_inkling2_target.py passes 87 tests on a GB10 with the checkpoint, including attention against the reference in prefill, decode and verification and the fused-norm bound.
Limitations
the projection requires GemvQuery.accumulation from FlashInfer (local-inference-lab/flashinfer#110), which the pinned FlashInfer revision lacks. Completion length was compared between a core from this tree with the drafting changes and a core from the unmodified tree: over lookup prompts with seeds 1 to 30, median completion tokens were 363 against 420 and the share at or above 1,024 tokens differed by -1.7 points (95% interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses. tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and 5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not run. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).
2. Cut the fixed cost of Inkling 2 drafting rounds and add proposal-head formats
Why
On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes.
Change
Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted.
Result
A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank.
Evidence
one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch.
Limitations
verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).
3. Shard the row-local stages of Inkling 2 prefill across hosts
Between two cross-host reductions, an Inkling 2 decoder layer computes stages whose result for a row depends on that row or on the three rows before it: the attention-residual mixtures with their norms, the output convolutions, the expert router and the residual additions. In tensor-parallel prefill every rank computed them for every row of the chunk. On one GB10 they take 773 ms of an 8,192-row chunk, and four ranks produce four identical copies. RoCEnante's pipelined all-reduce is a reduce-scatter followed by a shard gather, so a rank can keep its quarter of the rows after the first half, compute these stages for those rows only, and gather where every rank needs every row, at the wire bytes of the all-reduce it replaces.
Prefill graphs of at least 2,048 rows on ranks that reduce over RoCEnante shard their rows (SHARDED_PREFILL_ROWS in the Inkling 2 decoder). PreparedReduceScatter and PreparedShardGather in ignition/components/collectives.py prepare the two halves of the pipelined all-reduce as separate captured operations. ignition/components/row_shards.py holds row ownership, each rank's view of the batch metadata, a column gather that exchanges the three rows before each rank's rows and each request's last three rows after every reduce-scatter (the output convolutions and their replicated histories read them), and PreparedRouteGather, which carries each row's int32 expert IDs and FP32 route weights as bytes in BF16 lanes. The decoder's layer loop and final mixture move into PreparedStack; in a sharded graph the residual state holds a rank's rows, and shard gathers supply the attention input streams, the expert input, the drafter taps and the final normalized rows. Row ownership follows graph capacity, so a rank whose rows are padding computes inactive rows. Smaller prefill graphs, decode, verification and ranks on one host are unchanged. Helios and the package contract are unchanged; a core gains sharded prefill when it is exported again.
A sharded 8,192-row chunk computes the row-local stages once across the four ranks and runs 361 collectives where the replicated chunk runs 97. Every row-local value, route, gathered row and convolution history equals the replicated forward bit for bit.
Evidence: on one GB10 with the checkpoint, the row-local stages of 48 layers take 175.7 ms for a rank's shard against 773.4 ms for 8,192 rows, and 37.5 ms against 173.3 ms at 2,048 rows, with no differing element. tests/components/test_row_shards.py (41 tests) covers ownership, halo and tail selection and capacity checks; tests/recipes/test_inkling2_row_shards_cuda.py (6 tests) compares four ranks' sharded forwards of a five-layer stack with the replicated stack through an in-process exchange, eagerly and through a compiled, captured and replayed forward, on an RTX PRO 6000 and a GB10; tests/recipes/test_inkling2_target.py passes 87 tests on a GB10. One core exported from this tree with FlashInfer 4fd836dc plus the FlashInfer changes this branch's parents require served at TP4 on four DGX Sparks with 8,192-row chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests complete. Over three service launches, time to first token was 2.07 to 2.34 s at 7,668 prompt tokens, 18.4 to 20.1 s at 67,145 and 40.7 to 43.1 s at 131,990, three repeats each; one launch of the core exported from the parent commit measured 2.59 to 2.69 s, 22.4 to 23.0 s and 48.8 to 49.8 s. In kernel traces of rank 0, a 7,168-row chunk spans 1,809 ms sharded against 2,034 ms replicated: collectives 586 ms against 508 ms, mixtures and norms 70 ms against about 216 ms. Limitations: a fourth launch of the sharded core ran every graph, decode included, about 10% slower (2.85 to 2.96 s and 48.8 to 53.8 s) for a cause that was not identified, and the replicated core was measured on one launch, so the serving comparison does not exclude a launch effect of that size. Capacities between 2,048 and 8,192 rows are not measured in serving; the threshold is an estimate.