Skip to content

Cut the fixed cost of Inkling 2 drafting rounds and add proposal-head formats - #308

Closed
original-el8 wants to merge 1 commit into
inkling2/spark-prefillfrom
inkling2/spark-drafting
Closed

original-el8 wants to merge 1 commit into
inkling2/spark-prefillfrom
inkling2/spark-drafting

Conversation

@original-el8

Copy link
Copy Markdown
Collaborator

Why

On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes.

Change

Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted.

Result

A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank.

Evidence

one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch.

Limitations

verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. This PR is a draft until Flare's FlashInfer pin can move to a main revision containing the FlashInfer changes named above; it is stacked on the Inkling 2 DGX Spark TP4 PRs (#304, #305).

… formats

On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus
4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each
round bought no tokens and the balanced policy rarely chose to draft.
Verification staged each sliding-window layer's history with PyTorch
gathers over whole rings and committed accepted rows by rewriting whole
rings, the recurrent head projected its draw-independent context once per
draft position, and proposal heads read BF16 only. These costs do not
depend on the draws or on the number of drafts, so they can leave the
per-round path without changing what verification computes.

Verification attention stages each request's preceding window and its
candidate rows with one PreparedWindowStaging kernel per layer and commits
accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging
gains lookback, the count of preceding positions staged from a ring that
retains more than attention reads, and PreparedRingCommit gains
candidate_offset, the record index at which staged candidates start; both
default to their earlier meaning. The recurrent proposal head projects the
block's states and target context once per block, proposals read one
embedding row per request, and staging plans are shared between layers.
--draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8
prepare proposal-only copies computed from checkpoint values at load;
target verification keeps the BF16 head, so they change which tokens are
proposed, never the output distribution. Both flags default to omitted.

A drafting round at one request costs about 34.7 ms plus 4.7 ms per
verified draft: the fixed cost over a decode step falls from about 11 ms
to about 6 ms and the per-draft cost is unchanged. The preparation graph
pool falls from 2,840 MiB to 2,092 MiB per rank.

Evidence: one exported core per tree served by flare run at TP4 on GB10,
one request, ordinary prompts, a fresh launch per verification row budget.
Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9,
62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms.
Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and
279 ms at 32 (342 to 461 ms). The measured core used
--draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81,
63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the
BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The
measured core also contained the prefill changes of the preceding commit.
On a GB10 with the checkpoint, test_inkling2_target.py (87),
test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass,
covering verification attention against the reference and staged and
committed rings against PyTorch. Limitations: verification staging
requires WindowStageKernel's lookback from FlashInfer
(local-inference-lab/flashinfer#112), which the pinned revision lacks.
--markov-head-quantization fp8 and the MXFP8 draft head were not measured
in serving. Aggregate decode throughput is within a few percent of the
unmodified tree. lookback and candidate_offset extend two existing
component contracts with defaulted parameters; no graph kind is added.

Design-Signed-Off-By: flare-02; design-review=ed22
@original-el8

Copy link
Copy Markdown
Collaborator Author

Folded into #311 as its own commit, so the Inkling 2 DGX Spark performance changes are one review. The commit and its description are unchanged there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant