Skip to content

Shorten Inkling 2 prefill and drafting rounds on DGX Spark TP4 - #311

Closed
original-el8 wants to merge 4 commits into
inkling2/spark-draftingfrom
inkling2/token-sharded-row-stages
Closed

original-el8 wants to merge 4 commits into
inkling2/spark-draftingfrom
inkling2/token-sharded-row-stages

Conversation

@original-el8

@original-el8 original-el8 commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Inkling 2 Small NVFP4 at TP4 on four DGX Sparks spent most of a prefill chunk outside its attention and expert kernels and paid about 11 ms per drafting round for no tokens. This change shortens prefill and drafting rounds in three commits, each described below with its own reasoning, change, result and evidence, in commit order. The base branch is inkling2/dgx-spark-tp4, which enables the recipe across hosts.

The commits require three FlashInfer changes that Flare's pinned revision lacks: local-inference-lab/flashinfer#110 (FP32 accumulation for BF16 projections), #111 (swapped-operand NVFP4 FC1 for 32-aligned widths) and #112 (window staging lookback). This is a draft until the pin can move to a merged FlashInfer commit.

1. Shorten Inkling 2 prefill with BF16 router operands and a fused mix and norm

Why

Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms 8,192-row chunk in the expert router's projection, which widened its BF16 source and weight to FP32 so that only b12x's FP32-operand kernels were eligible, and 376 ms in attention-residual mixtures that stored an FP32 mixture per stream for a separate RMSNorm launch. The router needs FP32 accumulation, not FP32 operands: the product of two BF16 values is exact in FP32. The mixture and its norm read and write the same rows, so one program can normalize without storing the mixture.

Change

The router projection takes its BF16 source and a BF16 router weight and declares accumulation="fp32", which UnquantizedProjection passes to b12x so that only plans carrying the whole sum in FP32 are raced. The weight loader delivers the FP32 checkpoint tensor as BF16 only after checking that every value is an exact BF16 upcast. PreparedResidualMix normalizes each stream in the mixture kernel with FP32 mean square, scale and weight and one rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention writes its partial sum in the cross-rank reduction's element type, BF16 over RoCEnante and FP32 on one host, in place of FP32 followed by a cast; decode and verification keep FP32.

Result

Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to 1.6. Router selection is unchanged in value class: operands are the same numbers and only summation order differs. The fused norm differs from the unfused one by at most one BF16 unit in at most ten elements per million.

Evidence

one exported core per tree served by flare run at TP4 on GB10, 8,192-row chunks, warm and uncached, three repeats. Time to first token was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to 23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s). The measured core also contained FlashInfer's swapped-operand FC1 selection and the drafting changes, so the factor is not attributable to this commit alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the same top-8 experts on every row. tests/recipes/test_inkling2_target.py passes 87 tests on a GB10 with the checkpoint, including attention against the reference in prefill, decode and verification and the fused-norm bound.

Limitations

the projection requires GemvQuery.accumulation from FlashInfer (local-inference-lab/flashinfer#110), which the pinned FlashInfer revision lacks. Completion length was compared between a core from this tree with the drafting changes and a core from the unmodified tree: over lookup prompts with seeds 1 to 30, median completion tokens were 363 against 420 and the share at or above 1,024 tokens differed by -1.7 points (95% interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses. tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and 5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not run. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).

2. Cut the fixed cost of Inkling 2 drafting rounds and add proposal-head formats

Why

On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes.

Change

Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted.

Result

A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank.

Evidence

one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch.

Limitations

verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).

3. Shard the row-local stages of Inkling 2 prefill across hosts

Between two cross-host reductions, an Inkling 2 decoder layer computes stages whose result for a row depends on that row or on the three rows before it: the attention-residual mixtures with their norms, the output convolutions, the expert router and the residual additions. In tensor-parallel prefill every rank computed them for every row of the chunk. On one GB10 they take 773 ms of an 8,192-row chunk, and four ranks produce four identical copies. RoCEnante's pipelined all-reduce is a reduce-scatter followed by a shard gather, so a rank can keep its quarter of the rows after the first half, compute these stages for those rows only, and gather where every rank needs every row, at the wire bytes of the all-reduce it replaces.

Prefill graphs of at least 2,048 rows on ranks that reduce over RoCEnante shard their rows (SHARDED_PREFILL_ROWS in the Inkling 2 decoder). PreparedReduceScatter and PreparedShardGather in ignition/components/collectives.py prepare the two halves of the pipelined all-reduce as separate captured operations. ignition/components/row_shards.py holds row ownership, each rank's view of the batch metadata, a column gather that exchanges the three rows before each rank's rows and each request's last three rows after every reduce-scatter (the output convolutions and their replicated histories read them), and PreparedRouteGather, which carries each row's int32 expert IDs and FP32 route weights as bytes in BF16 lanes. The decoder's layer loop and final mixture move into PreparedStack; in a sharded graph the residual state holds a rank's rows, and shard gathers supply the attention input streams, the expert input, the drafter taps and the final normalized rows. Row ownership follows graph capacity, so a rank whose rows are padding computes inactive rows. Smaller prefill graphs, decode, verification and ranks on one host are unchanged. Helios and the package contract are unchanged; a core gains sharded prefill when it is exported again.

A sharded 8,192-row chunk computes the row-local stages once across the four ranks and runs 361 collectives where the replicated chunk runs 97. Every row-local value, route, gathered row and convolution history equals the replicated forward bit for bit.

Evidence: on one GB10 with the checkpoint, the row-local stages of 48 layers take 175.7 ms for a rank's shard against 773.4 ms for 8,192 rows, and 37.5 ms against 173.3 ms at 2,048 rows, with no differing element. tests/components/test_row_shards.py (41 tests) covers ownership, halo and tail selection and capacity checks; tests/recipes/test_inkling2_row_shards_cuda.py (6 tests) compares four ranks' sharded forwards of a five-layer stack with the replicated stack through an in-process exchange, eagerly and through a compiled, captured and replayed forward, on an RTX PRO 6000 and a GB10; tests/recipes/test_inkling2_target.py passes 87 tests on a GB10. One core exported from this tree with FlashInfer 4fd836dc plus the FlashInfer changes this branch's parents require served at TP4 on four DGX Sparks with 8,192-row chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests complete. Over three service launches, time to first token was 2.07 to 2.34 s at 7,668 prompt tokens, 18.4 to 20.1 s at 67,145 and 40.7 to 43.1 s at 131,990, three repeats each; one launch of the core exported from the parent commit measured 2.59 to 2.69 s, 22.4 to 23.0 s and 48.8 to 49.8 s. In kernel traces of rank 0, a 7,168-row chunk spans 1,809 ms sharded against 2,034 ms replicated: collectives 586 ms against 508 ms, mixtures and norms 70 ms against about 216 ms. Limitations: a fourth launch of the sharded core ran every graph, decode included, about 10% slower (2.85 to 2.96 s and 48.8 to 53.8 s) for a cause that was not identified, and the replicated core was measured on one launch, so the serving comparison does not exclude a launch effect of that size. Capacities between 2,048 and 8,192 rows are not measured in serving; the threshold is an estimate.

When `flare run` starts a multi-host core, rank zero reads every peer's rank
core through the page cache and sends it over TCP. On a host whose GPU
allocates from system memory, such as a DGX Spark (GB10), those cached pages
are not free memory for CUDA, so the pages of several rank cores can keep rank
zero from creating its CUDA context. Helios already releases the page cache of
a rank's own core files before it creates the context
(Files::release_page_cache in helios/src/runtime_core/model.rs), and a
receiving rank syncs the files it writes; the files rank zero reads on behalf
of other ranks were never released. Releasing them where the distribution
source stops closes that gap with the same eos operation and needs no change
to loading or to the transfer protocol.

Before this change the distribution source in flare-cli/src/distribution.rs
returned when its serving threads had joined and left the pages it had read in
the cache. Now release_served_pages calls flare_eos::release_page_cache
(POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one
and above when the source thread finishes, whether distribution succeeded or
failed. Rank zero's own files are left to the existing release in Helios.
flare-cli gains a path dependency on the in-tree eos crate, which the lock
file records.

The result is that a rank-zero host returns the memory used to read peer
cores before GPU initialization, without a manual page-cache drop.

Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host)
serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank
zero distributed three rank cores, its host showed 114 GiB of page cache and
2 GiB free, and rank zero failed at "Initialize GPU" with CudaError {
operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The
failure was intermittent: the first launch of a smaller core went through the
same distribution without it. After, with rebuilt binaries on all four hosts,
the peers' rank cores removed and no manual cache drop: three rank cores were
sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed
MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded.
The Linux test
source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a
4 MiB on-disk file to one peer and checks with mincore that the file has
resident pages after the transfer and none after the source finishes.
cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30
passed, 0 failed, 2 ignored (they need external fixtures). Not run: other
models, two- and three-rank cores, and hosts with discrete GPUs, where the
release changes only the page cache.

Design-Signed-Off-By: flare-02; design-review=ed22
…nd norm

Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms
8,192-row chunk in the expert router's projection, which widened its BF16
source and weight to FP32 so that only b12x's FP32-operand kernels were
eligible, and 376 ms in attention-residual mixtures that stored an FP32
mixture per stream for a separate RMSNorm launch. The router needs FP32
accumulation, not FP32 operands: the product of two BF16 values is exact
in FP32. The mixture and its norm read and write the same rows, so one
program can normalize without storing the mixture.

The router projection takes its BF16 source and a BF16 router weight and
declares accumulation="fp32", which UnquantizedProjection passes to b12x so
that only plans carrying the whole sum in FP32 are raced. The weight loader
delivers the FP32 checkpoint tensor as BF16 only after checking that every
value is an exact BF16 upcast. PreparedResidualMix normalizes each stream
in the mixture kernel with FP32 mean square, scale and weight and one
rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention
writes its partial sum in the cross-rank reduction's element type, BF16
over RoCEnante and FP32 on one host, in place of FP32 followed by a cast;
decode and verification keep FP32.

Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to
1.6. Router selection is unchanged in value class: operands are the same
numbers and only summation order differs. The fused norm differs from the
unfused one by at most one BF16 unit in at most ten elements per million.

Evidence: one exported core per tree served by flare run at TP4 on GB10,
8,192-row chunks, warm and uncached, three repeats. Time to first token
was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to
23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s).
The measured core also contained FlashInfer's swapped-operand FC1 selection
and the drafting changes, so the factor is not attributable to this commit
alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with
routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the
router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the
same top-8 experts on every row. tests/recipes/test_inkling2_target.py
passes 87 tests on a GB10 with the checkpoint, including attention against
the reference in prefill, decode and verification and the fused-norm bound.
Limitations: the projection requires GemvQuery.accumulation from FlashInfer
(local-inference-lab/flashinfer#110), which the pinned FlashInfer revision
lacks. Completion length was compared between a core from this tree with
the drafting changes and a core from the unmodified tree: over lookup
prompts with seeds 1 to 30, median completion tokens were 363 against 420
and the share at or above 1,024 tokens differed by -1.7 points (95%
interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses.
tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed
and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and
5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not
run.

Design-Signed-Off-By: flare-02; design-review=ed22
… formats

On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus
4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each
round bought no tokens and the balanced policy rarely chose to draft.
Verification staged each sliding-window layer's history with PyTorch
gathers over whole rings and committed accepted rows by rewriting whole
rings, the recurrent head projected its draw-independent context once per
draft position, and proposal heads read BF16 only. These costs do not
depend on the draws or on the number of drafts, so they can leave the
per-round path without changing what verification computes.

Verification attention stages each request's preceding window and its
candidate rows with one PreparedWindowStaging kernel per layer and commits
accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging
gains lookback, the count of preceding positions staged from a ring that
retains more than attention reads, and PreparedRingCommit gains
candidate_offset, the record index at which staged candidates start; both
default to their earlier meaning. The recurrent proposal head projects the
block's states and target context once per block, proposals read one
embedding row per request, and staging plans are shared between layers.
--draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8
prepare proposal-only copies computed from checkpoint values at load;
target verification keeps the BF16 head, so they change which tokens are
proposed, never the output distribution. Both flags default to omitted.

A drafting round at one request costs about 34.7 ms plus 4.7 ms per
verified draft: the fixed cost over a decode step falls from about 11 ms
to about 6 ms and the per-draft cost is unchanged. The preparation graph
pool falls from 2,840 MiB to 2,092 MiB per rank.

Evidence: one exported core per tree served by flare run at TP4 on GB10,
one request, ordinary prompts, a fresh launch per verification row budget.
Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9,
62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms.
Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and
279 ms at 32 (342 to 461 ms). The measured core used
--draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81,
63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the
BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The
measured core also contained the prefill changes of the preceding commit.
On a GB10 with the checkpoint, test_inkling2_target.py (87),
test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass,
covering verification attention against the reference and staged and
committed rings against PyTorch. Limitations: verification staging
requires WindowStageKernel's lookback from FlashInfer
(local-inference-lab/flashinfer#112), which the pinned revision lacks.
--markov-head-quantization fp8 and the MXFP8 draft head were not measured
in serving. Aggregate decode throughput is within a few percent of the
unmodified tree. lookback and candidate_offset extend two existing
component contracts with defaulted parameters; no graph kind is added.

Design-Signed-Off-By: flare-02; design-review=ed22
Between two cross-host reductions, an Inkling 2 decoder layer computes
stages whose result for a row depends on that row or on the three rows
before it: the attention-residual mixtures with their norms, the output
convolutions, the expert router and the residual additions. In tensor-
parallel prefill every rank computed them for every row of the chunk. On
one GB10 they take 773 ms of an 8,192-row chunk, and four ranks produce
four identical copies. RoCEnante's pipelined all-reduce is a reduce-
scatter followed by a shard gather, so a rank can keep its quarter of the
rows after the first half, compute these stages for those rows only, and
gather where every rank needs every row, at the wire bytes of the
all-reduce it replaces.

Prefill graphs of at least 2,048 rows on ranks that reduce over RoCEnante
shard their rows (SHARDED_PREFILL_ROWS in the Inkling 2 decoder).
PreparedReduceScatter and PreparedShardGather in
ignition/components/collectives.py prepare the two halves of the pipelined
all-reduce as separate captured operations. ignition/components/
row_shards.py holds row ownership, each rank's view of the batch
metadata, a column gather that exchanges the three rows before each
rank's rows and each request's last three rows after every reduce-scatter
(the output convolutions and their replicated histories read them), and
PreparedRouteGather, which carries each row's int32 expert IDs and FP32
route weights as bytes in BF16 lanes. The decoder's layer loop and final
mixture move into PreparedStack; in a sharded graph the residual state
holds a rank's rows, and shard gathers supply the attention input streams,
the expert input, the drafter taps and the final normalized rows. Row
ownership follows graph capacity, so a rank whose rows are padding
computes inactive rows. Smaller prefill graphs, decode, verification and
ranks on one host are unchanged. Helios and the package contract are
unchanged; a core gains sharded prefill when it is exported again.

A sharded 8,192-row chunk computes the row-local stages once across the
four ranks and runs 361 collectives where the replicated chunk runs 97.
Every row-local value, route, gathered row and convolution history equals
the replicated forward bit for bit.

Evidence: on one GB10 with the checkpoint, the row-local stages of 48
layers take 175.7 ms for a rank's shard against 773.4 ms for 8,192 rows,
and 37.5 ms against 173.3 ms at 2,048 rows, with no differing element.
tests/components/test_row_shards.py (41 tests) covers ownership, halo and
tail selection and capacity checks; tests/recipes/
test_inkling2_row_shards_cuda.py (6 tests) compares four ranks' sharded
forwards of a five-layer stack with the replicated stack through an
in-process exchange, eagerly and through a compiled, captured and replayed
forward, on an RTX PRO 6000 and a GB10; tests/recipes/
test_inkling2_target.py passes 87 tests on a GB10. One core exported from
this tree with FlashInfer 4fd836dc plus the FlashInfer changes this
branch's parents require served at TP4 on four DGX Sparks with 8,192-row
chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and
32 concurrent requests complete. Over three service launches, time to
first token was 2.07 to 2.34 s at 7,668 prompt tokens, 18.4 to 20.1 s at
67,145 and 40.7 to 43.1 s at 131,990, three repeats each; one launch of
the core exported from the parent commit measured 2.59 to 2.69 s, 22.4 to
23.0 s and 48.8 to 49.8 s. In kernel traces of rank 0, a 7,168-row chunk
spans 1,809 ms sharded against 2,034 ms replicated: collectives 586 ms
against 508 ms, mixtures and norms 70 ms against about 216 ms.
Limitations: a fourth launch of the sharded core ran every graph, decode
included, about 10% slower (2.85 to 2.96 s and 48.8 to 53.8 s) for a
cause that was not identified, and the replicated core was measured on
one launch, so the serving comparison does not exclude a launch effect of
that size. Capacities between 2,048 and 8,192 rows are not measured in
serving; the threshold is an estimate.

Design-Signed-Off-By: flare-02; design-review=ed22
@original-el8
original-el8 force-pushed the inkling2/token-sharded-row-stages branch from 1a492d2 to c49ae59 Compare November 15, 2026 17:38
@original-el8
original-el8 deleted the inkling2/token-sharded-row-stages branch November 15, 2026 17:39
@original-el8 original-el8 changed the title Shard the row-local stages of Inkling 2 prefill across hosts Shorten Inkling 2 prefill and drafting rounds on DGX Spark TP4 Nov 15, 2026
@original-el8

Copy link
Copy Markdown
Collaborator Author

Superseded by the consolidated Inkling 2 DGX Spark performance pull request opened from inkling2/spark-performance, which carries this commit unchanged alongside the prefill and drafting commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant