Repository navigation
Shorten Inkling 2 prefill and drafting rounds on DGX Spark TP4 - #312
Draft
original-el8 wants to merge 8 commits into
Draft
original-el8 wants to merge 8 commits into
original-el8 wants to merge 8 commits into
Conversation
…nd norm Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms 8,192-row chunk in the expert router's projection, which widened its BF16 source and weight to FP32 so that only b12x's FP32-operand kernels were eligible, and 376 ms in attention-residual mixtures that stored an FP32 mixture per stream for a separate RMSNorm launch. The router needs FP32 accumulation, not FP32 operands: the product of two BF16 values is exact in FP32. The mixture and its norm read and write the same rows, so one program can normalize without storing the mixture. The router projection takes its BF16 source and a BF16 router weight and declares accumulation="fp32", which UnquantizedProjection passes to b12x so that only plans carrying the whole sum in FP32 are raced. The weight loader delivers the FP32 checkpoint tensor as BF16 only after checking that every value is an exact BF16 upcast. PreparedResidualMix normalizes each stream in the mixture kernel with FP32 mean square, scale and weight and one rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention writes its partial sum in the cross-rank reduction's element type, BF16 over RoCEnante and FP32 on one host, in place of FP32 followed by a cast; decode and verification keep FP32. Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to 1.6. Router selection is unchanged in value class: operands are the same numbers and only summation order differs. The fused norm differs from the unfused one by at most one BF16 unit in at most ten elements per million. Evidence: one exported core per tree served by flare run at TP4 on GB10, 8,192-row chunks, warm and uncached, three repeats. Time to first token was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to 23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s). The measured core also contained FlashInfer's swapped-operand FC1 selection and the drafting changes, so the factor is not attributable to this commit alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the same top-8 experts on every row. tests/recipes/test_inkling2_target.py passes 87 tests on a GB10 with the checkpoint, including attention against the reference in prefill, decode and verification and the fused-norm bound. Limitations: the projection requires GemvQuery.accumulation from FlashInfer (local-inference-lab/flashinfer#110), which the pinned FlashInfer revision lacks. Completion length was compared between a core from this tree with the drafting changes and a core from the unmodified tree: over lookup prompts with seeds 1 to 30, median completion tokens were 363 against 420 and the share at or above 1,024 tokens differed by -1.7 points (95% interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses. tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and 5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not run. Design-Signed-Off-By: flare-02; design-review=ed22
… formats On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes. Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted. A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank. Evidence: one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch. Limitations: verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. Design-Signed-Off-By: flare-02; design-review=ed22
Between two cross-host reductions, an Inkling 2 decoder layer computes stages whose result for a row depends on that row or on the three rows before it: the attention-residual mixtures with their norms, the output convolutions, the expert router and the residual additions. In tensor- parallel prefill every rank computed them for every row of the chunk. On one GB10 they take 773 ms of an 8,192-row chunk, and four ranks produce four identical copies. RoCEnante's pipelined all-reduce is a reduce- scatter followed by a shard gather, so a rank can keep its quarter of the rows after the first half, compute these stages for those rows only, and gather where every rank needs every row, at the wire bytes of the all-reduce it replaces. Prefill graphs of at least 2,048 rows on ranks that reduce over RoCEnante shard their rows (SHARDED_PREFILL_ROWS in the Inkling 2 decoder). PreparedReduceScatter and PreparedShardGather in ignition/components/collectives.py prepare the two halves of the pipelined all-reduce as separate captured operations. ignition/components/ row_shards.py holds row ownership, each rank's view of the batch metadata, a column gather that exchanges the three rows before each rank's rows and each request's last three rows after every reduce-scatter (the output convolutions and their replicated histories read them), and PreparedRouteGather, which carries each row's int32 expert IDs and FP32 route weights as bytes in BF16 lanes. The decoder's layer loop and final mixture move into PreparedStack; in a sharded graph the residual state holds a rank's rows, and shard gathers supply the attention input streams, the expert input, the drafter taps and the final normalized rows. Row ownership follows graph capacity, so a rank whose rows are padding computes inactive rows. Smaller prefill graphs, decode, verification and ranks on one host are unchanged. Helios and the package contract are unchanged; a core gains sharded prefill when it is exported again. A sharded 8,192-row chunk computes the row-local stages once across the four ranks and runs 361 collectives where the replicated chunk runs 97. Every row-local value, route, gathered row and convolution history equals the replicated forward bit for bit. Evidence: on one GB10 with the checkpoint, the row-local stages of 48 layers take 175.7 ms for a rank's shard against 773.4 ms for 8,192 rows, and 37.5 ms against 173.3 ms at 2,048 rows, with no differing element. tests/components/test_row_shards.py (41 tests) covers ownership, halo and tail selection and capacity checks; tests/recipes/ test_inkling2_row_shards_cuda.py (6 tests) compares four ranks' sharded forwards of a five-layer stack with the replicated stack through an in-process exchange, eagerly and through a compiled, captured and replayed forward, on an RTX PRO 6000 and a GB10; tests/recipes/ test_inkling2_target.py passes 87 tests on a GB10. One core exported from this tree with FlashInfer 4fd836dc plus the FlashInfer changes this branch's parents require served at TP4 on four DGX Sparks with 8,192-row chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests complete. Over three service launches, time to first token was 2.07 to 2.34 s at 7,668 prompt tokens, 18.4 to 20.1 s at 67,145 and 40.7 to 43.1 s at 131,990, three repeats each; one launch of the core exported from the parent commit measured 2.59 to 2.69 s, 22.4 to 23.0 s and 48.8 to 49.8 s. In kernel traces of rank 0, a 7,168-row chunk spans 1,809 ms sharded against 2,034 ms replicated: collectives 586 ms against 508 ms, mixtures and norms 70 ms against about 216 ms. Limitations: a fourth launch of the sharded core ran every graph, decode included, about 10% slower (2.85 to 2.96 s and 48.8 to 53.8 s) for a cause that was not identified, and the replicated core was measured on one launch, so the serving comparison does not exclude a launch effect of that size. Capacities between 2,048 and 8,192 rows are not measured in serving; the threshold is an estimate. Design-Signed-Off-By: flare-02; design-review=ed22
An Inkling 2 decoder layer projects its input by a fused BF16 weight (query, output gate, key, value, relative-bias factors and sink logits) and its attention result by a BF16 output weight. At TP4 a rank reads 2.36 GB of these weights in every decode step whatever the row count, and on a GB10 the 48 layers' projections take 9.6 ms of a one-row step. Decode with one to four requests is bound by those reads. Kimi K3, Kimi K3.1 and Qwen3.8 already store such projections as MXFP8 under --online-mxfp8, which halves the bytes read. The Inkling 2 recipe declares online MXFP8 conversion. With the flag, attention_weights in ignition/recipes/inkling2/attention.py converts every fused input projection and the output projection of every decoder layer, key and value rows included, with quantize_bf16_linear_as_mxfp8, and PreparedAttention projects through QuantizedProjection with BF16 activations in prefill, decode and verification. The output projection then writes BF16 partial sums in every mode. The weight loader allocates the BF16 sources as temporaries, the flag enters the tuning identity, and PreparationConfig carries it from the plan. Routers, vocabulary shards, convolutions, norms, clip scales, the relative-logit projection, experts, DSpark and the vision encoder keep checkpoint precision. Without the flag preparation is unchanged. Helios and the package contract are unchanged. A rank reads 1.22 GB of attention projection weights per decode step and holds 1.14 GB less. The stored weights differ from the checkpoint's by 2.7% in relative norm, so generated text differs from a BF16-projection core's. An 8,192-row prefill projection is about 5% slower than with BF16 weights. Evidence: on one GB10 with the checkpoint, the 48 layers' projections of one rank take 5.62 ms against 9.57 ms at one row, 5.58 against 9.81 at four rows, and 221.9 ms against 211.4 ms at 8,192 rows. With the Transformers reference layers over 1,536 positions of prose and source code, next-token accuracy is 65.6% with MXFP8 projections and 65.8% without, and the most likely token agrees at 97.3% of positions. tests/recipes/test_inkling2_target.py checks that the decode, verification and prefill kernels of a window and a global layer multiply by the stored values, and tests/cli/test_inkling2.py covers the flag. One core exported with the flag served at TP4 on four DGX Sparks with 32 requests and 8,192-token chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7, lookups at 18,832, 67,151 and 131,996 prompt tokens return the expected value in 26 of 26 requests, and each rank holds 3,604,480 KV tokens against 3,416,064. Over three repeats on one launch of each core, a one-token round takes 31.5 to 32.7 ms of GPU time with two requests against 35.4 to 36.7 ms, and 49.8 to 50.0 ms with four against 52.0 to 53.2 ms; time to first token at 131,990 prompt tokens is 43.5 to 43.7 s against 40.7 to 43.1 s. Limitations: one-request round time and decode throughput at one to four requests do not separate from the BF16 core within three repeats. Completions of 60 seeded lookups had a median of 472 tokens against 356, one sample set per core. On a GB10, one of eight runs of test_inkling2_target.py -k checkpoint_attention reported one failing case of 20, which was not identified. Design-Signed-Off-By: flare-02; design-review=ed22
…n Sparks Row-sharded Inkling 2 prefill on four DGX Sparks loses 110 ms of a 2,072 ms chunk to a few collectives that take 7 to 72 ms against a 2 ms median. With the default routing a rank reaches one peer over a direct cable and two through the switch, and both ConnectX-7 ports share the host's PCIe links. The switch marks ECN on the congested flows and the senders' DCQCN rate control throttles them; flows over the cables are never throttled. The serving guide gave no routing advice for this configuration. docs/model-preparation.md states, in the Inkling 2 row-sharding section, that such a core is served with --roce-switch-only on every rank, with the mechanism and the measured effect. No code changes. A reader serving Inkling 2 at TP4 on cabled and switched Sparks is told the flag that removes the stalled collectives. Evidence: all four ranks of one core traced with node-level CUDA graph tracing over 67,000- and 132,000-token prompts. With the default routing, 90 collectives stalled in 32 chunks of 5,120 rows or more, hosts received 213,000 to 722,000 ECN-marked RoCE packets per 67,000-token prefill, and no CNP was counted on a cable port. With --roce-switch-only, no collective stalled in 36 such chunks and ECN, CNP and pause counters stayed at zero; chunks at the same position of the prompt were 92 ms (5.0%) shorter at 5,120 rows or more and 1.3% longer below, and a decode round was 0.08 to 0.09 ms longer. On untraced launches, time to first token at 131,990 tokens was 40.4 to 42.0 s against 43.5 to 43.7 s, three repeats each. Limitations: the two routings were measured on separate launches. The share of the stall caused by the shared PCIe links is inferred from pause and buffer-threshold counters; receive rate was not measured. Disabling ECN marking on the switch or DCQCN on the adapters was not tested. Design-Signed-Off-By: flare-02; design-review=ed22
…xfp8 An Inkling 2 drafting round reads the drafter's BF16 projections once whatever the number of drafts verified: the five draft layers' fused query/key/value and attention output projections and the recurrent head's context projection, 179 MB per TP4 rank. On a GB10 these kernels run at the memory rate (207 to 221 GB/s) and take about 0.85 ms of the 4.4 ms a round spends in the drafter and its proposal sampling. --online-mxfp8 already stores the target's attention projections as MXFP8; the drafter only proposes tokens, so its projections can take the same form without changing the output distribution. With --online-mxfp8, DraftWeights.load converts each draft layer's fused query/key/value projection and attention output projection and the recurrent head's context projection with quantize_bf16_linear_as_mxfp8, and the drafter declares them through prepare_projection with BF16 activations. ignition/components/recurrent_proposal.py accepts an MXFP8 context weight; its projection is then rounded to BF16 before the FP32 bias and the previous draw's part are added. Each draft layer's key/value projection of accepted target taps keeps BF16 because it also runs on every prefill row, where MXFP8 weights with BF16 activations are slower. The recurrent state projection, embeddings, norms, the checkpoint's MXFP8 feed-forward weights and both proposal heads keep their precision. Without the flag preparation is unchanged. Helios and the package contract are unchanged. A drafting round reads 92 MB of these weights per rank. Proposals and their acceptance rate may differ from a BF16 drafter's; target verification decides every emitted token. Evidence: on a GB10 with the checkpoint, tests/components/ test_inkling2_dspark_cuda.py and test_inkling2_dspark.py pass 45 of 45, including a check that the converted projections of a window and a global draft layer multiply by the stored values, eagerly and compiled; test_inkling2_target.py -k checkpoint_attention passes 20 of 20 and test_inkling2_row_shards_cuda.py 6 of 6. One core exported with --online-mxfp8, --draft-head-quantization nvfp4 and --markov-head-quantization fp8 served at TP4 on four DGX Sparks with 32 requests: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests return the expected values. Over three repeats each, acceptance of the first and second draft was 78% and 62% on ordinary prompts against 79% and 61% for the core with a BF16 drafter and BF16 correction head, and 58% and 34% against 58% and 33% on an article continuation; three decode benchmark passes completed without errors with single-request decode of 48.0 to 51.3 tokens/s at an 8,192-token context against 41.8 to 43.0. Limitations: the drafter's kernel time was not traced on this core, and its one launch also ran one-token rounds and prefill faster than the earlier core's launches (22.5 ms against 24.2 to 28.8 ms at one request), so the throughput difference is not attributed to this change. The served core also used the FP8 correction head, so the acceptance comparison covers both together. Design-Signed-Off-By: flare-02; design-review=ed22
…e with-vocabulary-head Every Inkling 2 decode and verification round multiplies the final hidden rows by the rank's BF16 unembedding shard, 463 MB per TP4 rank. On a GB10 that projection runs at the memory rate and takes 1.89 ms of a 23.2 ms one-token round with one request, and it is paid once per round whatever the number of requests. --online-mxfp8 stores the attention and DSpark projections as MXFP8 and leaves this shard in BF16. The Inkling 2 recipe declares two --online-mxfp8-scope groups. `all`, the default, is the existing conversion set, and a core exported with --online-mxfp8 alone records it. `with-vocabulary-head` also converts each rank's unembedding shard with mxfp8_draft_head and releases the BF16 rows after the drafter's proposal heads are derived from them. PreparedLogits and PreparedVocabularyProjection bind the packed shard through WidenedQuantizedProjection in ignition/components/vocabulary.py: b12x's A16 packed-weight kernel stores BF16, and the result is widened to FP32, so the softcap, the valid-vocabulary mask, the cross-rank gather, sampling, log-probabilities, resumed sampling and DSpark verification receive FP32 values in the layouts they have with a BF16 shard. Proposals read the drafter's quantized copy when it has one and otherwise the target's shard as stored; with --draft-head-quantization mxfp8 no duplicate is held. The scope enters the tuning identity and the rebuild manifest. No kernel, Helios code, Protobuf field or package contract changes. With the group selected, a round reads 239 MB of vocabulary weights per rank and each logit is an FP32 sum rounded to BF16 before the softcap. Unlike a quantized proposal head, this changes the distribution that sampling and verification use. Cores exported without the group are unchanged. Evidence: on a GB10, over the final hidden states of 1,536 positions of prose and source code, the MXFP8 shard's softcapped logits differ from the BF16 shard's by 0.084 on average and 0.93 at most at a softcap of 30; the most likely token agrees at 99.48% of positions, and at the eight others the BF16 shard's two largest logits are within 0.07; the KL divergence at temperature 1 has mean 3.6e-4 and maximum 7.7e-3; next-token accuracy is 60.33% against 60.39%. Rounding the BF16 shard's logits to BF16 alone gives 99.54% agreement and mean divergence 5.3e-5. With the checkpoint, tests/recipes/test_inkling2_target.py and test_inkling2_vocabulary_cuda.py -k checkpoint pass 30 of 30, the Inkling 2 CLI and planning tests pass 214 with the 2 failures the parent commit has, and the DSpark, sampling, commit, packed-speculation and draft-head component tests pass 178 with the parent's 11 failures, all of which need the qualification feature. One core exported with the group served at TP4 on four DGX Sparks with 32 requests and 8,192-token chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7, 32 concurrent requests return the expected values, and a 67,150-token lookup is correct at three seeds. A rank-0 kernel trace gives 0.94 ms for the vocabulary projection. One-token rounds take 22.2, 30.3 and 41.5 ms of GPU time at one, two and four requests against 22.4, 31.5 and 42.6 ms for the core exported from the parent commit. With a distinct sampling seed per stream, three decode benchmark passes against two give 67.1 to 67.6 tokens/s against 63.4 to 63.7 at two requests, 92.3 to 94.0 against 88.7 at four and 125.8 to 128.6 against 120.1 to 120.8 at eight, at an 8,192-token context, with the same order at 32,768 and 131,072 tokens; single-request decode is 44.9 to 48.0 against 42.5 to 51.6. Coding medians are 63.6, 47.1 and 67.7 tokens/s against 65.5 and 62.7. Limitations: the first export with the group failed when a compile worker could not create its CUDA context on rank 0 and an unchanged second export succeeded; host memory during both followed exports of the parent commit, and the cause is not established. Output quality beyond the logit comparison and the serving checks was not evaluated. Draft acceptance on article text was sampled on 46 offers. Design-Signed-Off-By: flare-02; design-review=ed22
…-alpha 0.5 With the default exponent of 1.0 on completed prefix-survival scores, the balanced scheduler serving Inkling 2 Small at TP4 on four DGX Sparks leaves drafting after one to three rejected first drafts: a request's estimates start at a weight of two and a rejection zeroes every deeper offered prefix. In one-token rounds nothing is observed, so the request returns only through an acceptance probe, and one request funds a probe about every 6 s. Of twelve sequential 25 s requests on task prompts, three spent 12 to 26 s in one-token rounds at 48 to 51 tokens/s although their first drafts were accepted 63% to 90% of the time once drafting, and single-request coding completions of 400 to 1,000 tokens ran either near 46 tokens/s or at 63 to 80. docs/model-preparation.md recommends --mtp-adaptive-verification-alpha 0.5 for Inkling 2 on DGX Sparks and states what it trades. The flag exists; no code, default, or package changes. With the exponent at 0.5 a request with high draft acceptance keeps drafting, and low-acceptance text is drafted more often than it repays. Evidence: one core exported from the parent commit with --online-mxfp8-scope with-vocabulary-head, served at TP4 on four DGX Sparks with the balanced policy, one launch per exponent. Ten sequential coding completions with 0.5 generated 70.6 to 81.9 tokens/s, with drafts in 99% of rounds; fifteen with 1.0 generated 46.5 to 76.7. On task prompts ending at EOS, three 20 s repeats gave 98.2 tokens/s (97.9 to 100.8) at four requests against 89.2 (88.3 to 90.4), 88.1 (80.4 to 92.4) at two against 80.4 (67.3 to 88.7), and 65.2 (63.2 to 69.2) at one against 71.3 (63.1 to 75.1). On an article continuation the exponent drafted in 100%, 84% and 49% of rounds at one, two and four requests for 46.7, 65.2 and 92.1 tokens/s against 46.3, 68.1 and 95.1. Two decode benchmark passes at an 8,192-token context with a distinct seed per stream gave 48.7 and 46.3 tokens/s at one request, 71.6 and 68.3 at two, 91.8 and 93.9 at four and 128.8 and 129.9 at eight, against 45.1 to 45.6, 67.1 to 67.6, 92.3 to 94.0 and 125.8 to 128.6 over three passes with 1.0. An exponent of 0.75 kept single requests drafting and did not change four requests (89.5). Limitations: launches differ by 3% in one-token round time; the exponent was not measured on other models or at more than eight requests; the probe cadence that makes the default slow to return is unchanged. Design-Signed-Off-By: flare-02; design-review=ed22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inkling 2 Small NVFP4 at TP4 on four DGX Sparks spent most of a prefill chunk outside its attention and expert kernels and paid about 11 ms per drafting round for no tokens. This change shortens prefill, drafting rounds and one-token decode rounds in eight commits, each described below with its own reasoning, change, result and evidence, in commit order. The base branch is
inkling2/dgx-spark-tp4, which enables the recipe across hosts.The commits require three FlashInfer changes that Flare's pinned revision lacks: local-inference-lab/flashinfer#110 (FP32 accumulation for BF16 projections), #111 (swapped-operand NVFP4 FC1 for 32-aligned widths) and #112 (window staging lookback). This is a draft until the pin can move to a merged FlashInfer commit.
1. Shorten Inkling 2 prefill with BF16 router operands and a fused mix and norm
Why
Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms 8,192-row chunk in the expert router's projection, which widened its BF16 source and weight to FP32 so that only b12x's FP32-operand kernels were eligible, and 376 ms in attention-residual mixtures that stored an FP32 mixture per stream for a separate RMSNorm launch. The router needs FP32 accumulation, not FP32 operands: the product of two BF16 values is exact in FP32. The mixture and its norm read and write the same rows, so one program can normalize without storing the mixture.
Change
The router projection takes its BF16 source and a BF16 router weight and declares accumulation="fp32", which UnquantizedProjection passes to b12x so that only plans carrying the whole sum in FP32 are raced. The weight loader delivers the FP32 checkpoint tensor as BF16 only after checking that every value is an exact BF16 upcast. PreparedResidualMix normalizes each stream in the mixture kernel with FP32 mean square, scale and weight and one rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention writes its partial sum in the cross-rank reduction's element type, BF16 over RoCEnante and FP32 on one host, in place of FP32 followed by a cast; decode and verification keep FP32.
Result
Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to 1.6. Router selection is unchanged in value class: operands are the same numbers and only summation order differs. The fused norm differs from the unfused one by at most one BF16 unit in at most ten elements per million.
Evidence
one exported core per tree served by flare run at TP4 on GB10, 8,192-row chunks, warm and uncached, three repeats. Time to first token was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to 23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s). The measured core also contained FlashInfer's swapped-operand FC1 selection and the drafting changes, so the factor is not attributable to this commit alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the same top-8 experts on every row. tests/recipes/test_inkling2_target.py passes 87 tests on a GB10 with the checkpoint, including attention against the reference in prefill, decode and verification and the fused-norm bound.
Limitations
the projection requires GemvQuery.accumulation from FlashInfer (local-inference-lab/flashinfer#110), which the pinned FlashInfer revision lacks. Completion length was compared between a core from this tree with the drafting changes and a core from the unmodified tree: over lookup prompts with seeds 1 to 30, median completion tokens were 363 against 420 and the share at or above 1,024 tokens differed by -1.7 points (95% interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses. tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and 5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not run. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).
2. Cut the fixed cost of Inkling 2 drafting rounds and add proposal-head formats
Why
On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes.
Change
Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted.
Result
A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank.
Evidence
one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch.
Limitations
verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).
3. Shard the row-local stages of Inkling 2 prefill across hosts
Between two cross-host reductions, an Inkling 2 decoder layer computes stages whose result for a row depends on that row or on the three rows before it: the attention-residual mixtures with their norms, the output convolutions, the expert router and the residual additions. In tensor-parallel prefill every rank computed them for every row of the chunk. On one GB10 they take 773 ms of an 8,192-row chunk, and four ranks produce four identical copies. RoCEnante's pipelined all-reduce is a reduce-scatter followed by a shard gather, so a rank can keep its quarter of the rows after the first half, compute these stages for those rows only, and gather where every rank needs every row, at the wire bytes of the all-reduce it replaces.
Prefill graphs of at least 2,048 rows on ranks that reduce over RoCEnante shard their rows (SHARDED_PREFILL_ROWS in the Inkling 2 decoder). PreparedReduceScatter and PreparedShardGather in ignition/components/collectives.py prepare the two halves of the pipelined all-reduce as separate captured operations. ignition/components/row_shards.py holds row ownership, each rank's view of the batch metadata, a column gather that exchanges the three rows before each rank's rows and each request's last three rows after every reduce-scatter (the output convolutions and their replicated histories read them), and PreparedRouteGather, which carries each row's int32 expert IDs and FP32 route weights as bytes in BF16 lanes. The decoder's layer loop and final mixture move into PreparedStack; in a sharded graph the residual state holds a rank's rows, and shard gathers supply the attention input streams, the expert input, the drafter taps and the final normalized rows. Row ownership follows graph capacity, so a rank whose rows are padding computes inactive rows. Smaller prefill graphs, decode, verification and ranks on one host are unchanged. Helios and the package contract are unchanged; a core gains sharded prefill when it is exported again.
A sharded 8,192-row chunk computes the row-local stages once across the four ranks and runs 361 collectives where the replicated chunk runs 97. Every row-local value, route, gathered row and convolution history equals the replicated forward bit for bit.
Evidence: on one GB10 with the checkpoint, the row-local stages of 48 layers take 175.7 ms for a rank's shard against 773.4 ms for 8,192 rows, and 37.5 ms against 173.3 ms at 2,048 rows, with no differing element. tests/components/test_row_shards.py (41 tests) covers ownership, halo and tail selection and capacity checks; tests/recipes/test_inkling2_row_shards_cuda.py (6 tests) compares four ranks' sharded forwards of a five-layer stack with the replicated stack through an in-process exchange, eagerly and through a compiled, captured and replayed forward, on an RTX PRO 6000 and a GB10; tests/recipes/test_inkling2_target.py passes 87 tests on a GB10. One core exported from this tree with FlashInfer 4fd836dc plus the FlashInfer changes this branch's parents require served at TP4 on four DGX Sparks with 8,192-row chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests complete. Over three service launches, time to first token was 2.07 to 2.34 s at 7,668 prompt tokens, 18.4 to 20.1 s at 67,145 and 40.7 to 43.1 s at 131,990, three repeats each; one launch of the core exported from the parent commit measured 2.59 to 2.69 s, 22.4 to 23.0 s and 48.8 to 49.8 s. In kernel traces of rank 0, a 7,168-row chunk spans 1,809 ms sharded against 2,034 ms replicated: collectives 586 ms against 508 ms, mixtures and norms 70 ms against about 216 ms. Limitations: a fourth launch of the sharded core ran every graph, decode included, about 10% slower (2.85 to 2.96 s and 48.8 to 53.8 s) for a cause that was not identified, and the replicated core was measured on one launch, so the serving comparison does not exclude a launch effect of that size. Capacities between 2,048 and 8,192 rows are not measured in serving; the threshold is an estimate.
4. Store Inkling 2 attention projections as MXFP8 with --online-mxfp8
An Inkling 2 decoder layer projects its input by a fused BF16 weight (query, output gate, key, value, relative-bias factors and sink logits) and its attention result by a BF16 output weight. At TP4 a rank reads 2.36 GB of these weights in every decode step whatever the row count, and on a GB10 the 48 layers' projections take 9.6 ms of a one-row step. Decode with one to four requests is bound by those reads. Kimi K3, Kimi K3.1 and Qwen3.8 already store such projections as MXFP8 under --online-mxfp8, which halves the bytes read.
The Inkling 2 recipe declares online MXFP8 conversion. With the flag, attention_weights in ignition/recipes/inkling2/attention.py converts every fused input projection and the output projection of every decoder layer, key and value rows included, with quantize_bf16_linear_as_mxfp8, and PreparedAttention projects through QuantizedProjection with BF16 activations in prefill, decode and verification. The output projection then writes BF16 partial sums in every mode. The weight loader allocates the BF16 sources as temporaries, the flag enters the tuning identity, and PreparationConfig carries it from the plan. Routers, vocabulary shards, convolutions, norms, clip scales, the relative-logit projection, experts, DSpark and the vision encoder keep checkpoint precision. Without the flag preparation is unchanged. Helios and the package contract are unchanged.
A rank reads 1.22 GB of attention projection weights per decode step and holds 1.14 GB less. The stored weights differ from the checkpoint's by 2.7% in relative norm, so generated text differs from a BF16-projection core's. An 8,192-row prefill projection is about 5% slower than with BF16 weights.
Evidence: on one GB10 with the checkpoint, the 48 layers' projections of one rank take 5.62 ms against 9.57 ms at one row, 5.58 against 9.81 at four rows, and 221.9 ms against 211.4 ms at 8,192 rows. With the Transformers reference layers over 1,536 positions of prose and source code, next-token accuracy is 65.6% with MXFP8 projections and 65.8% without, and the most likely token agrees at 97.3% of positions. tests/recipes/test_inkling2_target.py checks that the decode, verification and prefill kernels of a window and a global layer multiply by the stored values, and tests/cli/test_inkling2.py covers the flag. One core exported with the flag served at TP4 on four DGX Sparks with 32 requests and 8,192-token chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7, lookups at 18,832, 67,151 and 131,996 prompt tokens return the expected value in 26 of 26 requests, and each rank holds 3,604,480 KV tokens against 3,416,064. Over three repeats on one launch of each core, a one-token round takes 31.5 to 32.7 ms of GPU time with two requests against 35.4 to 36.7 ms, and 49.8 to 50.0 ms with four against 52.0 to 53.2 ms; time to first token at 131,990 prompt tokens is 43.5 to 43.7 s against 40.7 to 43.1 s. Limitations: one-request round time and decode throughput at one to four requests do not separate from the BF16 core within three repeats. Completions of 60 seeded lookups had a median of 472 tokens against 356, one sample set per core. On a GB10, one of eight runs of test_inkling2_target.py -k checkpoint_attention reported one failing case of 20, which was not identified.
5. Document switch-only RoCE routing for row-sharded Inkling 2 prefill on Sparks
Row-sharded Inkling 2 prefill on four DGX Sparks loses 110 ms of a 2,072 ms chunk to a few collectives that take 7 to 72 ms against a 2 ms median. With the default routing a rank reaches one peer over a direct cable and two through the switch, and both ConnectX-7 ports share the host's PCIe links. The switch marks ECN on the congested flows and the senders' DCQCN rate control throttles them; flows over the cables are never throttled. The serving guide gave no routing advice for this configuration.
docs/model-preparation.md states, in the Inkling 2 row-sharding section, that such a core is served with --roce-switch-only on every rank, with the mechanism and the measured effect. No code changes.
A reader serving Inkling 2 at TP4 on cabled and switched Sparks is told the flag that removes the stalled collectives.
Evidence: all four ranks of one core traced with node-level CUDA graph tracing over 67,000- and 132,000-token prompts. With the default routing, 90 collectives stalled in 32 chunks of 5,120 rows or more, hosts received 213,000 to 722,000 ECN-marked RoCE packets per 67,000-token prefill, and no CNP was counted on a cable port. With --roce-switch-only, no collective stalled in 36 such chunks and ECN, CNP and pause counters stayed at zero; chunks at the same position of the prompt were 92 ms (5.0%) shorter at 5,120 rows or more and 1.3% longer below, and a decode round was 0.08 to 0.09 ms longer. On untraced launches, time to first token at 131,990 tokens was 40.4 to 42.0 s against 43.5 to 43.7 s, three repeats each. Limitations: the two routings were measured on separate launches. The share of the stall caused by the shared PCIe links is inferred from pause and buffer-threshold counters; receive rate was not measured. Disabling ECN marking on the switch or DCQCN on the adapters was not tested.
6. Store Inkling 2 DSpark attention projections as MXFP8 with --online-mxfp8
An Inkling 2 drafting round reads the drafter's BF16 projections once whatever the number of drafts verified: the five draft layers' fused query/key/value and attention output projections and the recurrent head's context projection, 179 MB per TP4 rank. On a GB10 these kernels run at the memory rate (207 to 221 GB/s) and take about 0.85 ms of the 4.4 ms a round spends in the drafter and its proposal sampling. --online-mxfp8 already stores the target's attention projections as MXFP8; the drafter only proposes tokens, so its projections can take the same form without changing the output distribution.
With --online-mxfp8, DraftWeights.load converts each draft layer's fused query/key/value projection and attention output projection and the recurrent head's context projection with quantize_bf16_linear_as_mxfp8, and the drafter declares them through prepare_projection with BF16 activations. ignition/components/recurrent_proposal.py accepts an MXFP8 context weight; its projection is then rounded to BF16 before the FP32 bias and the previous draw's part are added. Each draft layer's key/value projection of accepted target taps keeps BF16 because it also runs on every prefill row, where MXFP8 weights with BF16 activations are slower. The recurrent state projection, embeddings, norms, the checkpoint's MXFP8 feed-forward weights and both proposal heads keep their precision. Without the flag preparation is unchanged. Helios and the package contract are unchanged.
A drafting round reads 92 MB of these weights per rank. Proposals and their acceptance rate may differ from a BF16 drafter's; target verification decides every emitted token.
Evidence: on a GB10 with the checkpoint, tests/components/ test_inkling2_dspark_cuda.py and test_inkling2_dspark.py pass 45 of 45, including a check that the converted projections of a window and a global draft layer multiply by the stored values, eagerly and compiled; test_inkling2_target.py -k checkpoint_attention passes 20 of 20 and test_inkling2_row_shards_cuda.py 6 of 6. One core exported with --online-mxfp8, --draft-head-quantization nvfp4 and --markov-head-quantization fp8 served at TP4 on four DGX Sparks with 32 requests: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests return the expected values. Over three repeats each, acceptance of the first and second draft was 78% and 62% on ordinary prompts against 79% and 61% for the core with a BF16 drafter and BF16 correction head, and 58% and 34% against 58% and 33% on an article continuation; three decode benchmark passes completed without errors with single-request decode of 48.0 to 51.3 tokens/s at an 8,192-token context against 41.8 to 43.0. Limitations: the drafter's kernel time was not traced on this core, and its one launch also ran one-token rounds and prefill faster than the earlier core's launches (22.5 ms against 24.2 to 28.8 ms at one request), so the throughput difference is not attributed to this change. The served core also used the FP8 correction head, so the acceptance comparison covers both together.
7. Store the Inkling 2 vocabulary head as MXFP8 with --online-mxfp8-scope with-vocabulary-head
Every Inkling 2 decode and verification round multiplies the final hidden rows by the rank's BF16 unembedding shard, 463 MB per TP4 rank. On a GB10 that projection runs at the memory rate and takes 1.89 ms of a 23.2 ms one-token round with one request, and it is paid once per round whatever the number of requests. --online-mxfp8 stores the attention and DSpark projections as MXFP8 and leaves this shard in BF16.
The Inkling 2 recipe declares two --online-mxfp8-scope groups.
all, the default, is the existing conversion set, and a core exported with --online-mxfp8 alone records it.with-vocabulary-headalso converts each rank's unembedding shard with mxfp8_draft_head and releases the BF16 rows after the drafter's proposal heads are derived from them. PreparedLogits and PreparedVocabularyProjection bind the packed shard through WidenedQuantizedProjection in ignition/components/vocabulary.py: b12x's A16 packed-weight kernel stores BF16, and the result is widened to FP32, so the softcap, the valid-vocabulary mask, the cross-rank gather, sampling, log-probabilities, resumed sampling and DSpark verification receive FP32 values in the layouts they have with a BF16 shard. Proposals read the drafter's quantized copy when it has one and otherwise the target's shard as stored; with --draft-head-quantization mxfp8 no duplicate is held. The scope enters the tuning identity and the rebuild manifest. No kernel, Helios code, Protobuf field or package contract changes.With the group selected, a round reads 239 MB of vocabulary weights per rank and each logit is an FP32 sum rounded to BF16 before the softcap. Unlike a quantized proposal head, this changes the distribution that sampling and verification use. Cores exported without the group are unchanged.
Evidence: on a GB10, over the final hidden states of 1,536 positions of prose and source code, the MXFP8 shard's softcapped logits differ from the BF16 shard's by 0.084 on average and 0.93 at most at a softcap of 30; the most likely token agrees at 99.48% of positions, and at the eight others the BF16 shard's two largest logits are within 0.07; the KL divergence at temperature 1 has mean 3.6e-4 and maximum 7.7e-3; next-token accuracy is 60.33% against 60.39%. Rounding the BF16 shard's logits to BF16 alone gives 99.54% agreement and mean divergence 5.3e-5. With the checkpoint, tests/recipes/test_inkling2_target.py and test_inkling2_vocabulary_cuda.py -k checkpoint pass 30 of 30, the Inkling 2 CLI and planning tests pass 214 with the 2 failures the parent commit has, and the DSpark, sampling, commit, packed-speculation and draft-head component tests pass 178 with the parent's 11 failures, all of which need the qualification feature. One core exported with the group served at TP4 on four DGX Sparks with 32 requests and 8,192-token chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7, 32 concurrent requests return the expected values, and a 67,150-token lookup is correct at three seeds. A rank-0 kernel trace gives 0.94 ms for the vocabulary projection. One-token rounds take 22.2, 30.3 and 41.5 ms of GPU time at one, two and four requests against 22.4, 31.5 and 42.6 ms for the core exported from the parent commit. With a distinct sampling seed per stream, three decode benchmark passes against two give 67.1 to 67.6 tokens/s against 63.4 to 63.7 at two requests, 92.3 to 94.0 against 88.7 at four and 125.8 to 128.6 against 120.1 to 120.8 at eight, at an 8,192-token context, with the same order at 32,768 and 131,072 tokens; single-request decode is 44.9 to 48.0 against 42.5 to 51.6. Coding medians are 63.6, 47.1 and 67.7 tokens/s against 65.5 and 62.7. Limitations: the first export with the group failed when a compile worker could not create its CUDA context on rank 0 and an unchanged second export succeeded; host memory during both followed exports of the parent commit, and the cause is not established. Output quality beyond the logit comparison and the serving checks was not evaluated. Draft acceptance on article text was sampled on 46 offers.
8. Document serving Inkling 2 on Sparks with --mtp-adaptive-verification-alpha 0.5
With the default exponent of 1.0 on completed prefix-survival scores, the balanced scheduler serving Inkling 2 Small at TP4 on four DGX Sparks leaves drafting after one to three rejected first drafts: a request's estimates start at a weight of two and a rejection zeroes every deeper offered prefix. In one-token rounds nothing is observed, so the request returns only through an acceptance probe, and one request funds a probe about every 6 s. Of twelve sequential 25 s requests on task prompts, three spent 12 to 26 s in one-token rounds at 48 to 51 tokens/s although their first drafts were accepted 63% to 90% of the time once drafting, and single-request coding completions of 400 to 1,000 tokens ran either near 46 tokens/s or at 63 to 80.
docs/model-preparation.md recommends --mtp-adaptive-verification-alpha 0.5 for Inkling 2 on DGX Sparks and states what it trades. The flag exists; no code, default, or package changes.
With the exponent at 0.5 a request with high draft acceptance keeps drafting, and low-acceptance text is drafted more often than it repays.
Evidence: one core exported from the parent commit with --online-mxfp8-scope with-vocabulary-head, served at TP4 on four DGX Sparks with the balanced policy, one launch per exponent. Ten sequential coding completions with 0.5 generated 70.6 to 81.9 tokens/s, with drafts in 99% of rounds; fifteen with 1.0 generated 46.5 to 76.7. On task prompts ending at EOS, three 20 s repeats gave 98.2 tokens/s (97.9 to 100.8) at four requests against 89.2 (88.3 to 90.4), 88.1 (80.4 to 92.4) at two against 80.4 (67.3 to 88.7), and 65.2 (63.2 to 69.2) at one against 71.3 (63.1 to 75.1). On an article continuation the exponent drafted in 100%, 84% and 49% of rounds at one, two and four requests for 46.7, 65.2 and 92.1 tokens/s against 46.3, 68.1 and 95.1. Two decode benchmark passes at an 8,192-token context with a distinct seed per stream gave 48.7 and 46.3 tokens/s at one request, 71.6 and 68.3 at two, 91.8 and 93.9 at four and 128.8 and 129.9 at eight, against 45.1 to 45.6, 67.1 to 67.6, 92.3 to 94.0 and 125.8 to 128.6 over three passes with 1.0. An exponent of 0.75 kept single requests drafting and did not change four requests (89.5). Limitations: launches differ by 3% in one-token round time; the exponent was not measured on other models or at more than eight requests; the probe cadence that makes the default slow to return is unchanged.