Repository navigation
[Docs] MTP is shared: say what the fork lets you tune instead - #10
Merged
Merged
Conversation
The bullet read as if guess-then-check were the fork's, and the
clarifying paragraph named only two of the fork's three drafting
settings. Both are fixed, and one of the three turned out to be
upstream's already.
Both arms in the 0.2.13 comparison started with --spec 6 --spec-min-p 0.5
and sent no per-request override, so drafting contributes nothing to the
measured gain. That is now stated with the settings, and a small table
gives the real difference - control, not a different drafter:
draft depth stock: --spec at startup fork: also spec=N per request, 2..startup
prompt lookup stock: --suffix-draft at startup fork: also lookup=0|1 per request
draft confidence stock: spec_min_p per request fork: the same, plus live global retune
window policy stock: none fork: spec_window (off in the measurement)
spec_min_p per request is stock behaviour: upstream's server forwards
strata_tune{pcie_frac, spec_min_p, prefill_pipe, aux_cpus} in v0.2.13.
The fork's additions are `spec`, `lookup`, the window policy and the
TUNE/POST /v1/tune channel that changes settings globally without a
restart - checked in both serve/server.py files, not assumed.
The glossary's MTP entry now matches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result
The MTP bullet read as if guess-then-check were the fork's advantage, and the paragraph that was meant to correct that
named only two of the fork's three drafting settings. Checking the two servers line by line also turned up one item I
had implied was ours and isn't.
--spec 6 --spec-min-p 0.5and sent no per-request override, sodrafting contributes nothing to the measured gain. That is now on the page, with the settings.
spec_min_pper request is stock behaviour: upstream v0.2.13's server already forwardsstrata_tune{pcie_frac, spec_min_p, prefill_pipe, aux_cpus}. The fork's additions arespec(depth, clamped to2…startup),
lookup(prompt lookup on/off), thespec_windowpolicy, and the TUNE /POST /v1/tunechannel thatchanges settings globally without a restart.
Related discussion
none — documentation only.
Approach
Replaced the paragraph under the three engine decisions with: the shared-behaviour statement including the exact
drafting settings both arms used, a four-row table of stock vs fork around drafting, and one line saying none of it was
switched on for the numbers below (
spec_window0, GuideLLM sends nostrata_tune). The glossary's MTP entry nowcarries the same three additions and the note that
spec_min_pper request is on both sides.Validation
Passed:
grep -n 'strata_tune' serve/server.pyin the two 0.2.13 trees: the stock treeforwards
pcie_frac,spec_min_p,prefill_pipe,aux_cpus; the fork addsspec(with the comment "clamps it to2..--spec … a deeper window than the engine was started with is refused there") and
lookup.spec_windowexistsonly in the fork (
set_spec_window, 0 = off).--spec 6 --spec-min-p 0.5 --mtp … --mtp-window 8192.lookup=0|1cell stays one cell (escaped pipe).No broken relative links; all anchors resolve.
Failed:
none observed.
Not run:
Limits
additions, and says that
spec_windowwas off for the comparison.