Skip to content

Shorten Inkling 2 prefill and drafting rounds on DGX Spark TP4 - #312

Draft
original-el8 wants to merge 8 commits into
inkling2/dgx-spark-tp4from
inkling2/spark-performance
Draft

original-el8 wants to merge 8 commits into
inkling2/dgx-spark-tp4from
inkling2/spark-performance

Conversation

@original-el8

@original-el8 original-el8 commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Inkling 2 Small NVFP4 at TP4 on four DGX Sparks spent most of a prefill chunk outside its attention and expert kernels and paid about 11 ms per drafting round for no tokens. This change shortens prefill, drafting rounds and one-token decode rounds in eight commits, each described below with its own reasoning, change, result and evidence, in commit order. The base branch is inkling2/dgx-spark-tp4, which enables the recipe across hosts.

The commits require three FlashInfer changes that Flare's pinned revision lacks: local-inference-lab/flashinfer#110 (FP32 accumulation for BF16 projections), #111 (swapped-operand NVFP4 FC1 for 32-aligned widths) and #112 (window staging lookback). This is a draft until the pin can move to a merged FlashInfer commit.

1. Shorten Inkling 2 prefill with BF16 router operands and a fused mix and norm

Why

Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms 8,192-row chunk in the expert router's projection, which widened its BF16 source and weight to FP32 so that only b12x's FP32-operand kernels were eligible, and 376 ms in attention-residual mixtures that stored an FP32 mixture per stream for a separate RMSNorm launch. The router needs FP32 accumulation, not FP32 operands: the product of two BF16 values is exact in FP32. The mixture and its norm read and write the same rows, so one program can normalize without storing the mixture.

Change

The router projection takes its BF16 source and a BF16 router weight and declares accumulation="fp32", which UnquantizedProjection passes to b12x so that only plans carrying the whole sum in FP32 are raced. The weight loader delivers the FP32 checkpoint tensor as BF16 only after checking that every value is an exact BF16 upcast. PreparedResidualMix normalizes each stream in the mixture kernel with FP32 mean square, scale and weight and one rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention writes its partial sum in the cross-rank reduction's element type, BF16 over RoCEnante and FP32 on one host, in place of FP32 followed by a cast; decode and verification keep FP32.

Result

Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to 1.6. Router selection is unchanged in value class: operands are the same numbers and only summation order differs. The fused norm differs from the unfused one by at most one BF16 unit in at most ten elements per million.

Evidence

one exported core per tree served by flare run at TP4 on GB10, 8,192-row chunks, warm and uncached, three repeats. Time to first token was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to 23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s). The measured core also contained FlashInfer's swapped-operand FC1 selection and the drafting changes, so the factor is not attributable to this commit alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the same top-8 experts on every row. tests/recipes/test_inkling2_target.py passes 87 tests on a GB10 with the checkpoint, including attention against the reference in prefill, decode and verification and the fused-norm bound.

Limitations

the projection requires GemvQuery.accumulation from FlashInfer (local-inference-lab/flashinfer#110), which the pinned FlashInfer revision lacks. Completion length was compared between a core from this tree with the drafting changes and a core from the unmodified tree: over lookup prompts with seeds 1 to 30, median completion tokens were 363 against 420 and the share at or above 1,024 tokens differed by -1.7 points (95% interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses. tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and 5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not run. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).

2. Cut the fixed cost of Inkling 2 drafting rounds and add proposal-head formats

Why

On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes.

Change

Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted.

Result

A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank.

Evidence

one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch.

Limitations

verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).

3. Shard the row-local stages of Inkling 2 prefill across hosts

Between two cross-host reductions, an Inkling 2 decoder layer computes stages whose result for a row depends on that row or on the three rows before it: the attention-residual mixtures with their norms, the output convolutions, the expert router and the residual additions. In tensor-parallel prefill every rank computed them for every row of the chunk. On one GB10 they take 773 ms of an 8,192-row chunk, and four ranks produce four identical copies. RoCEnante's pipelined all-reduce is a reduce-scatter followed by a shard gather, so a rank can keep its quarter of the rows after the first half, compute these stages for those rows only, and gather where every rank needs every row, at the wire bytes of the all-reduce it replaces.

Prefill graphs of at least 2,048 rows on ranks that reduce over RoCEnante shard their rows (SHARDED_PREFILL_ROWS in the Inkling 2 decoder). PreparedReduceScatter and PreparedShardGather in ignition/components/collectives.py prepare the two halves of the pipelined all-reduce as separate captured operations. ignition/components/row_shards.py holds row ownership, each rank's view of the batch metadata, a column gather that exchanges the three rows before each rank's rows and each request's last three rows after every reduce-scatter (the output convolutions and their replicated histories read them), and PreparedRouteGather, which carries each row's int32 expert IDs and FP32 route weights as bytes in BF16 lanes. The decoder's layer loop and final mixture move into PreparedStack; in a sharded graph the residual state holds a rank's rows, and shard gathers supply the attention input streams, the expert input, the drafter taps and the final normalized rows. Row ownership follows graph capacity, so a rank whose rows are padding computes inactive rows. Smaller prefill graphs, decode, verification and ranks on one host are unchanged. Helios and the package contract are unchanged; a core gains sharded prefill when it is exported again.

A sharded 8,192-row chunk computes the row-local stages once across the four ranks and runs 361 collectives where the replicated chunk runs 97. Every row-local value, route, gathered row and convolution history equals the replicated forward bit for bit.

Evidence: on one GB10 with the checkpoint, the row-local stages of 48 layers take 175.7 ms for a rank's shard against 773.4 ms for 8,192 rows, and 37.5 ms against 173.3 ms at 2,048 rows, with no differing element. tests/components/test_row_shards.py (41 tests) covers ownership, halo and tail selection and capacity checks; tests/recipes/test_inkling2_row_shards_cuda.py (6 tests) compares four ranks' sharded forwards of a five-layer stack with the replicated stack through an in-process exchange, eagerly and through a compiled, captured and replayed forward, on an RTX PRO 6000 and a GB10; tests/recipes/test_inkling2_target.py passes 87 tests on a GB10. One core exported from this tree with FlashInfer 4fd836dc plus the FlashInfer changes this branch's parents require served at TP4 on four DGX Sparks with 8,192-row chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests complete. Over three service launches, time to first token was 2.07 to 2.34 s at 7,668 prompt tokens, 18.4 to 20.1 s at 67,145 and 40.7 to 43.1 s at 131,990, three repeats each; one launch of the core exported from the parent commit measured 2.59 to 2.69 s, 22.4 to 23.0 s and 48.8 to 49.8 s. In kernel traces of rank 0, a 7,168-row chunk spans 1,809 ms sharded against 2,034 ms replicated: collectives 586 ms against 508 ms, mixtures and norms 70 ms against about 216 ms. Limitations: a fourth launch of the sharded core ran every graph, decode included, about 10% slower (2.85 to 2.96 s and 48.8 to 53.8 s) for a cause that was not identified, and the replicated core was measured on one launch, so the serving comparison does not exclude a launch effect of that size. Capacities between 2,048 and 8,192 rows are not measured in serving; the threshold is an estimate.

4. Store Inkling 2 attention projections as MXFP8 with --online-mxfp8

An Inkling 2 decoder layer projects its input by a fused BF16 weight (query, output gate, key, value, relative-bias factors and sink logits) and its attention result by a BF16 output weight. At TP4 a rank reads 2.36 GB of these weights in every decode step whatever the row count, and on a GB10 the 48 layers' projections take 9.6 ms of a one-row step. Decode with one to four requests is bound by those reads. Kimi K3, Kimi K3.1 and Qwen3.8 already store such projections as MXFP8 under --online-mxfp8, which halves the bytes read.

The Inkling 2 recipe declares online MXFP8 conversion. With the flag, attention_weights in ignition/recipes/inkling2/attention.py converts every fused input projection and the output projection of every decoder layer, key and value rows included, with quantize_bf16_linear_as_mxfp8, and PreparedAttention projects through QuantizedProjection with BF16 activations in prefill, decode and verification. The output projection then writes BF16 partial sums in every mode. The weight loader allocates the BF16 sources as temporaries, the flag enters the tuning identity, and PreparationConfig carries it from the plan. Routers, vocabulary shards, convolutions, norms, clip scales, the relative-logit projection, experts, DSpark and the vision encoder keep checkpoint precision. Without the flag preparation is unchanged. Helios and the package contract are unchanged.

A rank reads 1.22 GB of attention projection weights per decode step and holds 1.14 GB less. The stored weights differ from the checkpoint's by 2.7% in relative norm, so generated text differs from a BF16-projection core's. An 8,192-row prefill projection is about 5% slower than with BF16 weights.

Evidence: on one GB10 with the checkpoint, the 48 layers' projections of one rank take 5.62 ms against 9.57 ms at one row, 5.58 against 9.81 at four rows, and 221.9 ms against 211.4 ms at 8,192 rows. With the Transformers reference layers over 1,536 positions of prose and source code, next-token accuracy is 65.6% with MXFP8 projections and 65.8% without, and the most likely token agrees at 97.3% of positions. tests/recipes/test_inkling2_target.py checks that the decode, verification and prefill kernels of a window and a global layer multiply by the stored values, and tests/cli/test_inkling2.py covers the flag. One core exported with the flag served at TP4 on four DGX Sparks with 32 requests and 8,192-token chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7, lookups at 18,832, 67,151 and 131,996 prompt tokens return the expected value in 26 of 26 requests, and each rank holds 3,604,480 KV tokens against 3,416,064. Over three repeats on one launch of each core, a one-token round takes 31.5 to 32.7 ms of GPU time with two requests against 35.4 to 36.7 ms, and 49.8 to 50.0 ms with four against 52.0 to 53.2 ms; time to first token at 131,990 prompt tokens is 43.5 to 43.7 s against 40.7 to 43.1 s. Limitations: one-request round time and decode throughput at one to four requests do not separate from the BF16 core within three repeats. Completions of 60 seeded lookups had a median of 472 tokens against 356, one sample set per core. On a GB10, one of eight runs of test_inkling2_target.py -k checkpoint_attention reported one failing case of 20, which was not identified.

5. Document switch-only RoCE routing for row-sharded Inkling 2 prefill on Sparks

Row-sharded Inkling 2 prefill on four DGX Sparks loses 110 ms of a 2,072 ms chunk to a few collectives that take 7 to 72 ms against a 2 ms median. With the default routing a rank reaches one peer over a direct cable and two through the switch, and both ConnectX-7 ports share the host's PCIe links. The switch marks ECN on the congested flows and the senders' DCQCN rate control throttles them; flows over the cables are never throttled. The serving guide gave no routing advice for this configuration.

docs/model-preparation.md states, in the Inkling 2 row-sharding section, that such a core is served with --roce-switch-only on every rank, with the mechanism and the measured effect. No code changes.

A reader serving Inkling 2 at TP4 on cabled and switched Sparks is told the flag that removes the stalled collectives.

Evidence: all four ranks of one core traced with node-level CUDA graph tracing over 67,000- and 132,000-token prompts. With the default routing, 90 collectives stalled in 32 chunks of 5,120 rows or more, hosts received 213,000 to 722,000 ECN-marked RoCE packets per 67,000-token prefill, and no CNP was counted on a cable port. With --roce-switch-only, no collective stalled in 36 such chunks and ECN, CNP and pause counters stayed at zero; chunks at the same position of the prompt were 92 ms (5.0%) shorter at 5,120 rows or more and 1.3% longer below, and a decode round was 0.08 to 0.09 ms longer. On untraced launches, time to first token at 131,990 tokens was 40.4 to 42.0 s against 43.5 to 43.7 s, three repeats each. Limitations: the two routings were measured on separate launches. The share of the stall caused by the shared PCIe links is inferred from pause and buffer-threshold counters; receive rate was not measured. Disabling ECN marking on the switch or DCQCN on the adapters was not tested.

6. Store Inkling 2 DSpark attention projections as MXFP8 with --online-mxfp8

An Inkling 2 drafting round reads the drafter's BF16 projections once whatever the number of drafts verified: the five draft layers' fused query/key/value and attention output projections and the recurrent head's context projection, 179 MB per TP4 rank. On a GB10 these kernels run at the memory rate (207 to 221 GB/s) and take about 0.85 ms of the 4.4 ms a round spends in the drafter and its proposal sampling. --online-mxfp8 already stores the target's attention projections as MXFP8; the drafter only proposes tokens, so its projections can take the same form without changing the output distribution.

With --online-mxfp8, DraftWeights.load converts each draft layer's fused query/key/value projection and attention output projection and the recurrent head's context projection with quantize_bf16_linear_as_mxfp8, and the drafter declares them through prepare_projection with BF16 activations. ignition/components/recurrent_proposal.py accepts an MXFP8 context weight; its projection is then rounded to BF16 before the FP32 bias and the previous draw's part are added. Each draft layer's key/value projection of accepted target taps keeps BF16 because it also runs on every prefill row, where MXFP8 weights with BF16 activations are slower. The recurrent state projection, embeddings, norms, the checkpoint's MXFP8 feed-forward weights and both proposal heads keep their precision. Without the flag preparation is unchanged. Helios and the package contract are unchanged.

A drafting round reads 92 MB of these weights per rank. Proposals and their acceptance rate may differ from a BF16 drafter's; target verification decides every emitted token.

Evidence: on a GB10 with the checkpoint, tests/components/ test_inkling2_dspark_cuda.py and test_inkling2_dspark.py pass 45 of 45, including a check that the converted projections of a window and a global draft layer multiply by the stored values, eagerly and compiled; test_inkling2_target.py -k checkpoint_attention passes 20 of 20 and test_inkling2_row_shards_cuda.py 6 of 6. One core exported with --online-mxfp8, --draft-head-quantization nvfp4 and --markov-head-quantization fp8 served at TP4 on four DGX Sparks with 32 requests: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and 32 concurrent requests return the expected values. Over three repeats each, acceptance of the first and second draft was 78% and 62% on ordinary prompts against 79% and 61% for the core with a BF16 drafter and BF16 correction head, and 58% and 34% against 58% and 33% on an article continuation; three decode benchmark passes completed without errors with single-request decode of 48.0 to 51.3 tokens/s at an 8,192-token context against 41.8 to 43.0. Limitations: the drafter's kernel time was not traced on this core, and its one launch also ran one-token rounds and prefill faster than the earlier core's launches (22.5 ms against 24.2 to 28.8 ms at one request), so the throughput difference is not attributed to this change. The served core also used the FP8 correction head, so the acceptance comparison covers both together.

7. Store the Inkling 2 vocabulary head as MXFP8 with --online-mxfp8-scope with-vocabulary-head

Every Inkling 2 decode and verification round multiplies the final hidden rows by the rank's BF16 unembedding shard, 463 MB per TP4 rank. On a GB10 that projection runs at the memory rate and takes 1.89 ms of a 23.2 ms one-token round with one request, and it is paid once per round whatever the number of requests. --online-mxfp8 stores the attention and DSpark projections as MXFP8 and leaves this shard in BF16.

The Inkling 2 recipe declares two --online-mxfp8-scope groups. all, the default, is the existing conversion set, and a core exported with --online-mxfp8 alone records it. with-vocabulary-head also converts each rank's unembedding shard with mxfp8_draft_head and releases the BF16 rows after the drafter's proposal heads are derived from them. PreparedLogits and PreparedVocabularyProjection bind the packed shard through WidenedQuantizedProjection in ignition/components/vocabulary.py: b12x's A16 packed-weight kernel stores BF16, and the result is widened to FP32, so the softcap, the valid-vocabulary mask, the cross-rank gather, sampling, log-probabilities, resumed sampling and DSpark verification receive FP32 values in the layouts they have with a BF16 shard. Proposals read the drafter's quantized copy when it has one and otherwise the target's shard as stored; with --draft-head-quantization mxfp8 no duplicate is held. The scope enters the tuning identity and the rebuild manifest. No kernel, Helios code, Protobuf field or package contract changes.

With the group selected, a round reads 239 MB of vocabulary weights per rank and each logit is an FP32 sum rounded to BF16 before the softcap. Unlike a quantized proposal head, this changes the distribution that sampling and verification use. Cores exported without the group are unchanged.

Evidence: on a GB10, over the final hidden states of 1,536 positions of prose and source code, the MXFP8 shard's softcapped logits differ from the BF16 shard's by 0.084 on average and 0.93 at most at a softcap of 30; the most likely token agrees at 99.48% of positions, and at the eight others the BF16 shard's two largest logits are within 0.07; the KL divergence at temperature 1 has mean 3.6e-4 and maximum 7.7e-3; next-token accuracy is 60.33% against 60.39%. Rounding the BF16 shard's logits to BF16 alone gives 99.54% agreement and mean divergence 5.3e-5. With the checkpoint, tests/recipes/test_inkling2_target.py and test_inkling2_vocabulary_cuda.py -k checkpoint pass 30 of 30, the Inkling 2 CLI and planning tests pass 214 with the 2 failures the parent commit has, and the DSpark, sampling, commit, packed-speculation and draft-head component tests pass 178 with the parent's 11 failures, all of which need the qualification feature. One core exported with the group served at TP4 on four DGX Sparks with 32 requests and 8,192-token chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7, 32 concurrent requests return the expected values, and a 67,150-token lookup is correct at three seeds. A rank-0 kernel trace gives 0.94 ms for the vocabulary projection. One-token rounds take 22.2, 30.3 and 41.5 ms of GPU time at one, two and four requests against 22.4, 31.5 and 42.6 ms for the core exported from the parent commit. With a distinct sampling seed per stream, three decode benchmark passes against two give 67.1 to 67.6 tokens/s against 63.4 to 63.7 at two requests, 92.3 to 94.0 against 88.7 at four and 125.8 to 128.6 against 120.1 to 120.8 at eight, at an 8,192-token context, with the same order at 32,768 and 131,072 tokens; single-request decode is 44.9 to 48.0 against 42.5 to 51.6. Coding medians are 63.6, 47.1 and 67.7 tokens/s against 65.5 and 62.7. Limitations: the first export with the group failed when a compile worker could not create its CUDA context on rank 0 and an unchanged second export succeeded; host memory during both followed exports of the parent commit, and the cause is not established. Output quality beyond the logit comparison and the serving checks was not evaluated. Draft acceptance on article text was sampled on 46 offers.

8. Document serving Inkling 2 on Sparks with --mtp-adaptive-verification-alpha 0.5

With the default exponent of 1.0 on completed prefix-survival scores, the balanced scheduler serving Inkling 2 Small at TP4 on four DGX Sparks leaves drafting after one to three rejected first drafts: a request's estimates start at a weight of two and a rejection zeroes every deeper offered prefix. In one-token rounds nothing is observed, so the request returns only through an acceptance probe, and one request funds a probe about every 6 s. Of twelve sequential 25 s requests on task prompts, three spent 12 to 26 s in one-token rounds at 48 to 51 tokens/s although their first drafts were accepted 63% to 90% of the time once drafting, and single-request coding completions of 400 to 1,000 tokens ran either near 46 tokens/s or at 63 to 80.

docs/model-preparation.md recommends --mtp-adaptive-verification-alpha 0.5 for Inkling 2 on DGX Sparks and states what it trades. The flag exists; no code, default, or package changes.

With the exponent at 0.5 a request with high draft acceptance keeps drafting, and low-acceptance text is drafted more often than it repays.

Evidence: one core exported from the parent commit with --online-mxfp8-scope with-vocabulary-head, served at TP4 on four DGX Sparks with the balanced policy, one launch per exponent. Ten sequential coding completions with 0.5 generated 70.6 to 81.9 tokens/s, with drafts in 99% of rounds; fifteen with 1.0 generated 46.5 to 76.7. On task prompts ending at EOS, three 20 s repeats gave 98.2 tokens/s (97.9 to 100.8) at four requests against 89.2 (88.3 to 90.4), 88.1 (80.4 to 92.4) at two against 80.4 (67.3 to 88.7), and 65.2 (63.2 to 69.2) at one against 71.3 (63.1 to 75.1). On an article continuation the exponent drafted in 100%, 84% and 49% of rounds at one, two and four requests for 46.7, 65.2 and 92.1 tokens/s against 46.3, 68.1 and 95.1. Two decode benchmark passes at an 8,192-token context with a distinct seed per stream gave 48.7 and 46.3 tokens/s at one request, 71.6 and 68.3 at two, 91.8 and 93.9 at four and 128.8 and 129.9 at eight, against 45.1 to 45.6, 67.1 to 67.6, 92.3 to 94.0 and 125.8 to 128.6 over three passes with 1.0. An exponent of 0.75 kept single requests drafting and did not change four requests (89.5). Limitations: launches differ by 3% in one-token round time; the exponent was not measured on other models or at more than eight requests; the probe cadence that makes the default slow to return is unchanged.

…nd norm

Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms
8,192-row chunk in the expert router's projection, which widened its BF16
source and weight to FP32 so that only b12x's FP32-operand kernels were
eligible, and 376 ms in attention-residual mixtures that stored an FP32
mixture per stream for a separate RMSNorm launch. The router needs FP32
accumulation, not FP32 operands: the product of two BF16 values is exact
in FP32. The mixture and its norm read and write the same rows, so one
program can normalize without storing the mixture.

The router projection takes its BF16 source and a BF16 router weight and
declares accumulation="fp32", which UnquantizedProjection passes to b12x so
that only plans carrying the whole sum in FP32 are raced. The weight loader
delivers the FP32 checkpoint tensor as BF16 only after checking that every
value is an exact BF16 upcast. PreparedResidualMix normalizes each stream
in the mixture kernel with FP32 mean square, scale and weight and one
rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention
writes its partial sum in the cross-rank reduction's element type, BF16
over RoCEnante and FP32 on one host, in place of FP32 followed by a cast;
decode and verification keep FP32.

Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to
1.6. Router selection is unchanged in value class: operands are the same
numbers and only summation order differs. The fused norm differs from the
unfused one by at most one BF16 unit in at most ten elements per million.

Evidence: one exported core per tree served by flare run at TP4 on GB10,
8,192-row chunks, warm and uncached, three repeats. Time to first token
was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to
23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s).
The measured core also contained FlashInfer's swapped-operand FC1 selection
and the drafting changes, so the factor is not attributable to this commit
alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with
routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the
router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the
same top-8 experts on every row. tests/recipes/test_inkling2_target.py
passes 87 tests on a GB10 with the checkpoint, including attention against
the reference in prefill, decode and verification and the fused-norm bound.
Limitations: the projection requires GemvQuery.accumulation from FlashInfer
(local-inference-lab/flashinfer#110), which the pinned FlashInfer revision
lacks. Completion length was compared between a core from this tree with
the drafting changes and a core from the unmodified tree: over lookup
prompts with seeds 1 to 30, median completion tokens were 363 against 420
and the share at or above 1,024 tokens differed by -1.7 points (95%
interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses.
tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed
and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and
5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not
run.

Design-Signed-Off-By: flare-02; design-review=ed22
… formats

On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus
4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each
round bought no tokens and the balanced policy rarely chose to draft.
Verification staged each sliding-window layer's history with PyTorch
gathers over whole rings and committed accepted rows by rewriting whole
rings, the recurrent head projected its draw-independent context once per
draft position, and proposal heads read BF16 only. These costs do not
depend on the draws or on the number of drafts, so they can leave the
per-round path without changing what verification computes.

Verification attention stages each request's preceding window and its
candidate rows with one PreparedWindowStaging kernel per layer and commits
accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging
gains lookback, the count of preceding positions staged from a ring that
retains more than attention reads, and PreparedRingCommit gains
candidate_offset, the record index at which staged candidates start; both
default to their earlier meaning. The recurrent proposal head projects the
block's states and target context once per block, proposals read one
embedding row per request, and staging plans are shared between layers.
--draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8
prepare proposal-only copies computed from checkpoint values at load;
target verification keeps the BF16 head, so they change which tokens are
proposed, never the output distribution. Both flags default to omitted.

A drafting round at one request costs about 34.7 ms plus 4.7 ms per
verified draft: the fixed cost over a decode step falls from about 11 ms
to about 6 ms and the per-draft cost is unchanged. The preparation graph
pool falls from 2,840 MiB to 2,092 MiB per rank.

Evidence: one exported core per tree served by flare run at TP4 on GB10,
one request, ordinary prompts, a fresh launch per verification row budget.
Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9,
62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms.
Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and
279 ms at 32 (342 to 461 ms). The measured core used
--draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81,
63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the
BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The
measured core also contained the prefill changes of the preceding commit.
On a GB10 with the checkpoint, test_inkling2_target.py (87),
test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass,
covering verification attention against the reference and staged and
committed rings against PyTorch. Limitations: verification staging
requires WindowStageKernel's lookback from FlashInfer
(local-inference-lab/flashinfer#112), which the pinned revision lacks.
--markov-head-quantization fp8 and the MXFP8 draft head were not measured
in serving. Aggregate decode throughput is within a few percent of the
unmodified tree. lookback and candidate_offset extend two existing
component contracts with defaulted parameters; no graph kind is added.

Design-Signed-Off-By: flare-02; design-review=ed22
Between two cross-host reductions, an Inkling 2 decoder layer computes
stages whose result for a row depends on that row or on the three rows
before it: the attention-residual mixtures with their norms, the output
convolutions, the expert router and the residual additions. In tensor-
parallel prefill every rank computed them for every row of the chunk. On
one GB10 they take 773 ms of an 8,192-row chunk, and four ranks produce
four identical copies. RoCEnante's pipelined all-reduce is a reduce-
scatter followed by a shard gather, so a rank can keep its quarter of the
rows after the first half, compute these stages for those rows only, and
gather where every rank needs every row, at the wire bytes of the
all-reduce it replaces.

Prefill graphs of at least 2,048 rows on ranks that reduce over RoCEnante
shard their rows (SHARDED_PREFILL_ROWS in the Inkling 2 decoder).
PreparedReduceScatter and PreparedShardGather in
ignition/components/collectives.py prepare the two halves of the pipelined
all-reduce as separate captured operations. ignition/components/
row_shards.py holds row ownership, each rank's view of the batch
metadata, a column gather that exchanges the three rows before each
rank's rows and each request's last three rows after every reduce-scatter
(the output convolutions and their replicated histories read them), and
PreparedRouteGather, which carries each row's int32 expert IDs and FP32
route weights as bytes in BF16 lanes. The decoder's layer loop and final
mixture move into PreparedStack; in a sharded graph the residual state
holds a rank's rows, and shard gathers supply the attention input streams,
the expert input, the drafter taps and the final normalized rows. Row
ownership follows graph capacity, so a rank whose rows are padding
computes inactive rows. Smaller prefill graphs, decode, verification and
ranks on one host are unchanged. Helios and the package contract are
unchanged; a core gains sharded prefill when it is exported again.

A sharded 8,192-row chunk computes the row-local stages once across the
four ranks and runs 361 collectives where the replicated chunk runs 97.
Every row-local value, route, gathered row and convolution history equals
the replicated forward bit for bit.

Evidence: on one GB10 with the checkpoint, the row-local stages of 48
layers take 175.7 ms for a rank's shard against 773.4 ms for 8,192 rows,
and 37.5 ms against 173.3 ms at 2,048 rows, with no differing element.
tests/components/test_row_shards.py (41 tests) covers ownership, halo and
tail selection and capacity checks; tests/recipes/
test_inkling2_row_shards_cuda.py (6 tests) compares four ranks' sharded
forwards of a five-layer stack with the replicated stack through an
in-process exchange, eagerly and through a compiled, captured and replayed
forward, on an RTX PRO 6000 and a GB10; tests/recipes/
test_inkling2_target.py passes 87 tests on a GB10. One core exported from
this tree with FlashInfer 4fd836dc plus the FlashInfer changes this
branch's parents require served at TP4 on four DGX Sparks with 8,192-row
chunks: tests/server/test_inkling2.py -k default-effort passes 7 of 7 and
32 concurrent requests complete. Over three service launches, time to
first token was 2.07 to 2.34 s at 7,668 prompt tokens, 18.4 to 20.1 s at
67,145 and 40.7 to 43.1 s at 131,990, three repeats each; one launch of
the core exported from the parent commit measured 2.59 to 2.69 s, 22.4 to
23.0 s and 48.8 to 49.8 s. In kernel traces of rank 0, a 7,168-row chunk
spans 1,809 ms sharded against 2,034 ms replicated: collectives 586 ms
against 508 ms, mixtures and norms 70 ms against about 216 ms.
Limitations: a fourth launch of the sharded core ran every graph, decode
included, about 10% slower (2.85 to 2.96 s and 48.8 to 53.8 s) for a
cause that was not identified, and the replicated core was measured on
one launch, so the serving comparison does not exclude a launch effect of
that size. Capacities between 2,048 and 8,192 rows are not measured in
serving; the threshold is an estimate.

Design-Signed-Off-By: flare-02; design-review=ed22
An Inkling 2 decoder layer projects its input by a fused BF16 weight (query,
output gate, key, value, relative-bias factors and sink logits) and its
attention result by a BF16 output weight. At TP4 a rank reads 2.36 GB of
these weights in every decode step whatever the row count, and on a GB10
the 48 layers' projections take 9.6 ms of a one-row step. Decode with one
to four requests is bound by those reads. Kimi K3, Kimi K3.1 and Qwen3.8
already store such projections as MXFP8 under --online-mxfp8, which halves
the bytes read.

The Inkling 2 recipe declares online MXFP8 conversion. With the flag,
attention_weights in ignition/recipes/inkling2/attention.py converts every
fused input projection and the output projection of every decoder layer,
key and value rows included, with quantize_bf16_linear_as_mxfp8, and
PreparedAttention projects through QuantizedProjection with BF16
activations in prefill, decode and verification. The output projection
then writes BF16 partial sums in every mode. The weight loader allocates
the BF16 sources as temporaries, the flag enters the tuning identity, and
PreparationConfig carries it from the plan. Routers, vocabulary shards,
convolutions, norms, clip scales, the relative-logit projection, experts,
DSpark and the vision encoder keep checkpoint precision. Without the flag
preparation is unchanged. Helios and the package contract are unchanged.

A rank reads 1.22 GB of attention projection weights per decode step and
holds 1.14 GB less. The stored weights differ from the checkpoint's by
2.7% in relative norm, so generated text differs from a BF16-projection
core's. An 8,192-row prefill projection is about 5% slower than with BF16
weights.

Evidence: on one GB10 with the checkpoint, the 48 layers' projections of
one rank take 5.62 ms against 9.57 ms at one row, 5.58 against 9.81 at
four rows, and 221.9 ms against 211.4 ms at 8,192 rows. With the
Transformers reference layers over 1,536 positions of prose and source
code, next-token accuracy is 65.6% with MXFP8 projections and 65.8%
without, and the most likely token agrees at 97.3% of positions.
tests/recipes/test_inkling2_target.py checks that the decode, verification
and prefill kernels of a window and a global layer multiply by the stored
values, and tests/cli/test_inkling2.py covers the flag. One core exported
with the flag served at TP4 on four DGX Sparks with 32 requests and
8,192-token chunks: tests/server/test_inkling2.py -k default-effort passes
7 of 7, lookups at 18,832, 67,151 and 131,996 prompt tokens return the
expected value in 26 of 26 requests, and each rank holds 3,604,480 KV
tokens against 3,416,064. Over three repeats on one launch of each core, a
one-token round takes 31.5 to 32.7 ms of GPU time with two requests
against 35.4 to 36.7 ms, and 49.8 to 50.0 ms with four against 52.0 to
53.2 ms; time to first token at 131,990 prompt tokens is 43.5 to 43.7 s
against 40.7 to 43.1 s. Limitations: one-request round time and decode
throughput at one to four requests do not separate from the BF16 core
within three repeats. Completions of 60 seeded lookups had a median of 472
tokens against 356, one sample set per core. On a GB10, one of eight runs
of test_inkling2_target.py -k checkpoint_attention reported one failing
case of 20, which was not identified.

Design-Signed-Off-By: flare-02; design-review=ed22
…n Sparks

Row-sharded Inkling 2 prefill on four DGX Sparks loses 110 ms of a 2,072 ms
chunk to a few collectives that take 7 to 72 ms against a 2 ms median. With
the default routing a rank reaches one peer over a direct cable and two
through the switch, and both ConnectX-7 ports share the host's PCIe links.
The switch marks ECN on the congested flows and the senders' DCQCN rate
control throttles them; flows over the cables are never throttled. The
serving guide gave no routing advice for this configuration.

docs/model-preparation.md states, in the Inkling 2 row-sharding section,
that such a core is served with --roce-switch-only on every rank, with the
mechanism and the measured effect. No code changes.

A reader serving Inkling 2 at TP4 on cabled and switched Sparks is told the
flag that removes the stalled collectives.

Evidence: all four ranks of one core traced with node-level CUDA graph
tracing over 67,000- and 132,000-token prompts. With the default routing,
90 collectives stalled in 32 chunks of 5,120 rows or more, hosts received
213,000 to 722,000 ECN-marked RoCE packets per 67,000-token prefill, and no
CNP was counted on a cable port. With --roce-switch-only, no collective
stalled in 36 such chunks and ECN, CNP and pause counters stayed at zero;
chunks at the same position of the prompt were 92 ms (5.0%) shorter at
5,120 rows or more and 1.3% longer below, and a decode round was 0.08 to
0.09 ms longer. On untraced launches, time to first token at 131,990
tokens was 40.4 to 42.0 s against 43.5 to 43.7 s, three repeats each.
Limitations: the two routings were measured on separate launches. The
share of the stall caused by the shared PCIe links is inferred from pause
and buffer-threshold counters; receive rate was not measured. Disabling
ECN marking on the switch or DCQCN on the adapters was not tested.

Design-Signed-Off-By: flare-02; design-review=ed22
…xfp8

An Inkling 2 drafting round reads the drafter's BF16 projections once
whatever the number of drafts verified: the five draft layers' fused
query/key/value and attention output projections and the recurrent head's
context projection, 179 MB per TP4 rank. On a GB10 these kernels run at
the memory rate (207 to 221 GB/s) and take about 0.85 ms of the 4.4 ms a
round spends in the drafter and its proposal sampling. --online-mxfp8
already stores the target's attention projections as MXFP8; the drafter
only proposes tokens, so its projections can take the same form without
changing the output distribution.

With --online-mxfp8, DraftWeights.load converts each draft layer's fused
query/key/value projection and attention output projection and the
recurrent head's context projection with quantize_bf16_linear_as_mxfp8,
and the drafter declares them through prepare_projection with BF16
activations. ignition/components/recurrent_proposal.py accepts an MXFP8
context weight; its projection is then rounded to BF16 before the FP32
bias and the previous draw's part are added. Each draft layer's key/value
projection of accepted target taps keeps BF16 because it also runs on
every prefill row, where MXFP8 weights with BF16 activations are slower.
The recurrent state projection, embeddings, norms, the checkpoint's MXFP8
feed-forward weights and both proposal heads keep their precision. Without
the flag preparation is unchanged. Helios and the package contract are
unchanged.

A drafting round reads 92 MB of these weights per rank. Proposals and
their acceptance rate may differ from a BF16 drafter's; target
verification decides every emitted token.

Evidence: on a GB10 with the checkpoint, tests/components/
test_inkling2_dspark_cuda.py and test_inkling2_dspark.py pass 45 of 45,
including a check that the converted projections of a window and a global
draft layer multiply by the stored values, eagerly and compiled;
test_inkling2_target.py -k checkpoint_attention passes 20 of 20 and
test_inkling2_row_shards_cuda.py 6 of 6. One core exported with
--online-mxfp8, --draft-head-quantization nvfp4 and
--markov-head-quantization fp8 served at TP4 on four DGX Sparks with 32
requests: tests/server/test_inkling2.py -k default-effort passes 7 of 7
and 32 concurrent requests return the expected values. Over three repeats
each, acceptance of the first and second draft was 78% and 62% on
ordinary prompts against 79% and 61% for the core with a BF16 drafter and
BF16 correction head, and 58% and 34% against 58% and 33% on an article
continuation; three decode benchmark passes completed without errors with
single-request decode of 48.0 to 51.3 tokens/s at an 8,192-token context
against 41.8 to 43.0. Limitations: the drafter's kernel time was not
traced on this core, and its one launch also ran one-token rounds and
prefill faster than the earlier core's launches (22.5 ms against 24.2 to
28.8 ms at one request), so the throughput difference is not attributed
to this change. The served core also used the FP8 correction head, so the
acceptance comparison covers both together.

Design-Signed-Off-By: flare-02; design-review=ed22
…e with-vocabulary-head

Every Inkling 2 decode and verification round multiplies the final hidden
rows by the rank's BF16 unembedding shard, 463 MB per TP4 rank. On a GB10
that projection runs at the memory rate and takes 1.89 ms of a 23.2 ms
one-token round with one request, and it is paid once per round whatever
the number of requests. --online-mxfp8 stores the attention and DSpark
projections as MXFP8 and leaves this shard in BF16.

The Inkling 2 recipe declares two --online-mxfp8-scope groups. `all`, the
default, is the existing conversion set, and a core exported with
--online-mxfp8 alone records it. `with-vocabulary-head` also converts each
rank's unembedding shard with mxfp8_draft_head and releases the BF16 rows
after the drafter's proposal heads are derived from them. PreparedLogits
and PreparedVocabularyProjection bind the packed shard through
WidenedQuantizedProjection in ignition/components/vocabulary.py: b12x's
A16 packed-weight kernel stores BF16, and the result is widened to FP32,
so the softcap, the valid-vocabulary mask, the cross-rank gather, sampling,
log-probabilities, resumed sampling and DSpark verification receive FP32
values in the layouts they have with a BF16 shard. Proposals read the
drafter's quantized copy when it has one and otherwise the target's shard
as stored; with --draft-head-quantization mxfp8 no duplicate is held. The
scope enters the tuning identity and the rebuild manifest. No kernel,
Helios code, Protobuf field or package contract changes.

With the group selected, a round reads 239 MB of vocabulary weights per
rank and each logit is an FP32 sum rounded to BF16 before the softcap.
Unlike a quantized proposal head, this changes the distribution that
sampling and verification use. Cores exported without the group are
unchanged.

Evidence: on a GB10, over the final hidden states of 1,536 positions of
prose and source code, the MXFP8 shard's softcapped logits differ from the
BF16 shard's by 0.084 on average and 0.93 at most at a softcap of 30; the
most likely token agrees at 99.48% of positions, and at the eight others
the BF16 shard's two largest logits are within 0.07; the KL divergence at
temperature 1 has mean 3.6e-4 and maximum 7.7e-3; next-token accuracy is
60.33% against 60.39%. Rounding the BF16 shard's logits to BF16 alone
gives 99.54% agreement and mean divergence 5.3e-5. With the checkpoint,
tests/recipes/test_inkling2_target.py and test_inkling2_vocabulary_cuda.py
-k checkpoint pass 30 of 30, the Inkling 2 CLI and planning tests pass 214
with the 2 failures the parent commit has, and the DSpark, sampling,
commit, packed-speculation and draft-head component tests pass 178 with
the parent's 11 failures, all of which need the qualification feature. One
core exported with the group served at TP4 on four DGX Sparks with 32
requests and 8,192-token chunks: tests/server/test_inkling2.py -k
default-effort passes 7 of 7, 32 concurrent requests return the expected
values, and a 67,150-token lookup is correct at three seeds. A rank-0
kernel trace gives 0.94 ms for the vocabulary projection. One-token rounds
take 22.2, 30.3 and 41.5 ms of GPU time at one, two and four requests
against 22.4, 31.5 and 42.6 ms for the core exported from the parent
commit. With a distinct sampling seed per stream, three decode benchmark
passes against two give 67.1 to 67.6 tokens/s against 63.4 to 63.7 at two
requests, 92.3 to 94.0 against 88.7 at four and 125.8 to 128.6 against
120.1 to 120.8 at eight, at an 8,192-token context, with the same order at
32,768 and 131,072 tokens; single-request decode is 44.9 to 48.0 against
42.5 to 51.6. Coding medians are 63.6, 47.1 and 67.7 tokens/s against 65.5
and 62.7. Limitations: the first export with the group failed when a
compile worker could not create its CUDA context on rank 0 and an
unchanged second export succeeded; host memory during both followed
exports of the parent commit, and the cause is not established. Output
quality beyond the logit comparison and the serving checks was not
evaluated. Draft acceptance on article text was sampled on 46 offers.

Design-Signed-Off-By: flare-02; design-review=ed22
…-alpha 0.5

With the default exponent of 1.0 on completed prefix-survival scores, the
balanced scheduler serving Inkling 2 Small at TP4 on four DGX Sparks
leaves drafting after one to three rejected first drafts: a request's
estimates start at a weight of two and a rejection zeroes every deeper
offered prefix. In one-token rounds nothing is observed, so the request
returns only through an acceptance probe, and one request funds a probe
about every 6 s. Of twelve sequential 25 s requests on task prompts, three
spent 12 to 26 s in one-token rounds at 48 to 51 tokens/s although their
first drafts were accepted 63% to 90% of the time once drafting, and
single-request coding completions of 400 to 1,000 tokens ran either near
46 tokens/s or at 63 to 80.

docs/model-preparation.md recommends --mtp-adaptive-verification-alpha 0.5
for Inkling 2 on DGX Sparks and states what it trades. The flag exists;
no code, default, or package changes.

With the exponent at 0.5 a request with high draft acceptance keeps
drafting, and low-acceptance text is drafted more often than it repays.

Evidence: one core exported from the parent commit with
--online-mxfp8-scope with-vocabulary-head, served at TP4 on four DGX
Sparks with the balanced policy, one launch per exponent. Ten sequential
coding completions with 0.5 generated 70.6 to 81.9 tokens/s, with drafts
in 99% of rounds; fifteen with 1.0 generated 46.5 to 76.7. On task prompts
ending at EOS, three 20 s repeats gave 98.2 tokens/s (97.9 to 100.8) at
four requests against 89.2 (88.3 to 90.4), 88.1 (80.4 to 92.4) at two
against 80.4 (67.3 to 88.7), and 65.2 (63.2 to 69.2) at one against 71.3
(63.1 to 75.1). On an article continuation the exponent drafted in 100%,
84% and 49% of rounds at one, two and four requests for 46.7, 65.2 and
92.1 tokens/s against 46.3, 68.1 and 95.1. Two decode benchmark passes at
an 8,192-token context with a distinct seed per stream gave 48.7 and 46.3
tokens/s at one request, 71.6 and 68.3 at two, 91.8 and 93.9 at four and
128.8 and 129.9 at eight, against 45.1 to 45.6, 67.1 to 67.6, 92.3 to
94.0 and 125.8 to 128.6 over three passes with 1.0. An exponent of 0.75
kept single requests drafting and did not change four requests (89.5).
Limitations: launches differ by 3% in one-token round time; the exponent
was not measured on other models or at more than eight requests; the
probe cadence that makes the default slow to return is unchanged.

Design-Signed-Off-By: flare-02; design-review=ed22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant