Skip to content

Shorten Inkling 2 prefill with BF16 router operands and a fused mix and norm - #307

Closed
original-el8 wants to merge 1 commit into
inkling2/request-limit-32from
inkling2/spark-prefill
Closed

original-el8 wants to merge 1 commit into
inkling2/request-limit-32from
inkling2/spark-prefill

Conversation

@original-el8

Copy link
Copy Markdown
Collaborator

Why

Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms 8,192-row chunk in the expert router's projection, which widened its BF16 source and weight to FP32 so that only b12x's FP32-operand kernels were eligible, and 376 ms in attention-residual mixtures that stored an FP32 mixture per stream for a separate RMSNorm launch. The router needs FP32 accumulation, not FP32 operands: the product of two BF16 values is exact in FP32. The mixture and its norm read and write the same rows, so one program can normalize without storing the mixture.

Change

The router projection takes its BF16 source and a BF16 router weight and declares accumulation="fp32", which UnquantizedProjection passes to b12x so that only plans carrying the whole sum in FP32 are raced. The weight loader delivers the FP32 checkpoint tensor as BF16 only after checking that every value is an exact BF16 upcast. PreparedResidualMix normalizes each stream in the mixture kernel with FP32 mean square, scale and weight and one rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention writes its partial sum in the cross-rank reduction's element type, BF16 over RoCEnante and FP32 on one host, in place of FP32 followed by a cast; decode and verification keep FP32.

Result

Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to 1.6. Router selection is unchanged in value class: operands are the same numbers and only summation order differs. The fused norm differs from the unfused one by at most one BF16 unit in at most ten elements per million.

Evidence

one exported core per tree served by flare run at TP4 on GB10, 8,192-row chunks, warm and uncached, three repeats. Time to first token was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to 23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s). The measured core also contained FlashInfer's swapped-operand FC1 selection and the drafting changes, so the factor is not attributable to this commit alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the same top-8 experts on every row. tests/recipes/test_inkling2_target.py passes 87 tests on a GB10 with the checkpoint, including attention against the reference in prefill, decode and verification and the fused-norm bound.

Limitations

the projection requires GemvQuery.accumulation from FlashInfer (local-inference-lab/flashinfer#110), which the pinned FlashInfer revision lacks. Completion length was compared between a core from this tree with the drafting changes and a core from the unmodified tree: over lookup prompts with seeds 1 to 30, median completion tokens were 363 against 420 and the share at or above 1,024 tokens differed by -1.7 points (95% interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses. tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and 5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not run. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).

…nd norm

Inkling 2 Small prefill on DGX Spark TP4 spent 994 ms of a 3,808 ms
8,192-row chunk in the expert router's projection, which widened its BF16
source and weight to FP32 so that only b12x's FP32-operand kernels were
eligible, and 376 ms in attention-residual mixtures that stored an FP32
mixture per stream for a separate RMSNorm launch. The router needs FP32
accumulation, not FP32 operands: the product of two BF16 values is exact
in FP32. The mixture and its norm read and write the same rows, so one
program can normalize without storing the mixture.

The router projection takes its BF16 source and a BF16 router weight and
declares accumulation="fp32", which UnquantizedProjection passes to b12x so
that only plans carrying the whole sum in FP32 are raced. The weight loader
delivers the FP32 checkpoint tensor as BF16 only after checking that every
value is an exact BF16 upcast. PreparedResidualMix normalizes each stream
in the mixture kernel with FP32 mean square, scale and weight and one
rounding to BF16; PreparedRMSNorm leaves this path. Prefill attention
writes its partial sum in the cross-rank reduction's element type, BF16
over RoCEnante and FP32 on one host, in place of FP32 followed by a cast;
decode and verification keep FP32.

Prefill time to first token on four DGX Sparks falls by a factor of 1.5 to
1.6. Router selection is unchanged in value class: operands are the same
numbers and only summation order differs. The fused norm differs from the
unfused one by at most one BF16 unit in at most ten elements per million.

Evidence: one exported core per tree served by flare run at TP4 on GB10,
8,192-row chunks, warm and uncached, three repeats. Time to first token
was 2.59 to 2.69 s at 7,668 prompt tokens (4.18 to 4.51 s before), 22.4 to
23.0 s at 67,145 (35.2 to 35.4 s) and 48.8 to 49.8 s at 131,990 (73.5 s).
The measured core also contained FlashInfer's swapped-operand FC1 selection
and the drafting changes, so the factor is not attributable to this commit
alone. In a kernel trace of rank 0, an 8,192-row chunk spans 2,509 ms with
routing at 159 ms and mixtures with norms at 247 ms. On one GB10 the
router projection for 8,192 rows takes 2.14 ms against 19.5 ms, with the
same top-8 experts on every row. tests/recipes/test_inkling2_target.py
passes 87 tests on a GB10 with the checkpoint, including attention against
the reference in prefill, decode and verification and the fused-norm bound.
Limitations: the projection requires GemvQuery.accumulation from FlashInfer
(local-inference-lab/flashinfer#110), which the pinned FlashInfer revision
lacks. Completion length was compared between a core from this tree with
the drafting changes and a core from the unmodified tree: over lookup
prompts with seeds 1 to 30, median completion tokens were 363 against 420
and the share at or above 1,024 tokens differed by -1.7 points (95%
interval -7.2 to +3.8), with no wrong answer in 540 and 480 responses.
tests/server/test_inkling2.py's prefix-reuse case, which fixes one seed
and a 1,024-token limit, fails in 9 of 20 repeats on the changed core and
5 of 20 on the unmodified one. Single-host RTX PRO 6000 serving was not
run.

Design-Signed-Off-By: flare-02; design-review=ed22
@original-el8

Copy link
Copy Markdown
Collaborator Author

Folded into #311 as its own commit, so the Inkling 2 DGX Spark performance changes are one review. The commit and its description are unchanged there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant