Repository navigation
Cut the fixed cost of Inkling 2 drafting rounds and add proposal-head formats - #308
Closed
original-el8 wants to merge 1 commit into
Closed
original-el8 wants to merge 1 commit into
original-el8 wants to merge 1 commit into
Conversation
… formats On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes. Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted. A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank. Evidence: one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch. Limitations: verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. Design-Signed-Off-By: flare-02; design-review=ed22
Collaborator
Author
|
Folded into #311 as its own commit, so the Inkling 2 DGX Spark performance changes are one review. The commit and its description are unchanged there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes.
Change
Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted.
Result
A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank.
Evidence
one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch.
Limitations
verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).