Repository navigation
Start acceptance estimates from the acceptance of earlier requests - #306
Open
original-el8 wants to merge 1 commit into
Open
original-el8 wants to merge 1 commit into
original-el8 wants to merge 1 commit into
Conversation
Catalogs that price speculation from verified acceptance start every request's prefix-survival estimates at 0.5^d. Under the balanced policy a drafting round that costs more than a decode step is not worth choosing at that estimate, so each request runs width-one rounds until funded acceptance probes raise its estimates, whatever the acceptance of the requests before it. Confidence-forecast catalogs already start requests from a population; acceptance-priced catalogs had none. A population of verified acceptance gives a new request the best available estimate of its first rounds without changing how its own observations then govern it. The scheduler keeps an acceptance population per proposal source and draft depth while the model is loaded. Each mean starts at 0.5^d with the weight of eight observations. A validated completion contributes an observation when it is among the reporting request's first eight at that depth; the mean averages until its weight reaches 128 and then moves by 1/128 per observation. A new request's estimates take the population means at the initial weight of two. Width-one rounds contribute nothing, and catalogs that declare device prefix allocation for learned proposals keep the fixed 0.5^d start. docs/scheduling-runtime-design.md specifies the population. Requests that follow high-acceptance requests reach ordinary drafting within their first rounds; requests that follow low-acceptance requests, and low-acceptance requests after a high-acceptance population, return to width one from their own observations. One request supplies at most half of a mean that holds only the fixed start and about 6% at the weight cap. Evidence: Inkling 2 Small NVFP4 on four DGX Sparks (GB10, TP4), one exported core served by flare run under the balanced policy with seven draft tokens, each binary on a fresh launch. With the fixed start, the second to fifth requests of a coding prompt ran 17, 483, 197 and 70 width-one rounds before their first ordinary drafting round and produced 37 to 59 tokens/s over their first 10 s. With the population they ran 5, 3, 2 and 2 and produced 54 to 62 tokens/s. A chat prompt after those requests began drafting at once and returned to width one within 10 s (38.8 against 37.7 tokens/s over its first 10 s). Mixed prompts produced 102.4 against 98.0 tokens/s at 8 concurrent requests and 192.6 against 185.8 at 32; output-gap p50/p99 (78/223 ms and 166/376 ms) and time to first token were unchanged. The Helios scheduler tests pass (312), including seven in acceptance_population_tests.rs for the cases above. Other models and decoded-history proposals were not measured in serving. Design-Signed-Off-By: flare-02; design-review=ed22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Catalogs that price speculation from verified acceptance start every request's prefix-survival estimates at 0.5^d. Under the balanced policy a drafting round that costs more than a decode step is not worth choosing at that estimate, so each request runs width-one rounds until funded acceptance probes raise its estimates, whatever the acceptance of the requests before it. Confidence-forecast catalogs already start requests from a population; acceptance-priced catalogs had none. A population of verified acceptance gives a new request the best available estimate of its first rounds without changing how its own observations then govern it.
Change
The scheduler keeps an acceptance population per proposal source and draft depth while the model is loaded. Each mean starts at 0.5^d with the weight of eight observations. A validated completion contributes an observation when it is among the reporting request's first eight at that depth; the mean averages until its weight reaches 128 and then moves by 1/128 per observation. A new request's estimates take the population means at the initial weight of two. Width-one rounds contribute nothing, and catalogs that declare device prefix allocation for learned proposals keep the fixed 0.5^d start. docs/scheduling-runtime-design.md specifies the population.
Result
Requests that follow high-acceptance requests reach ordinary drafting within their first rounds; requests that follow low-acceptance requests, and low-acceptance requests after a high-acceptance population, return to width one from their own observations. One request supplies at most half of a mean that holds only the fixed start and about 6% at the weight cap.
Evidence
Inkling 2 Small NVFP4 on four DGX Sparks (GB10, TP4), one exported core served by flare run under the balanced policy with seven draft tokens, each binary on a fresh launch. With the fixed start, the second to fifth requests of a coding prompt ran 17, 483, 197 and 70 width-one rounds before their first ordinary drafting round and produced 37 to 59 tokens/s over their first 10 s. With the population they ran 5, 3, 2 and 2 and produced 54 to 62 tokens/s. A chat prompt after those requests began drafting at once and returned to width one within 10 s (38.8 against 37.7 tokens/s over its first 10 s). Mixed prompts produced 102.4 against 98.0 tokens/s at 8 concurrent requests and 192.6 against 185.8 at 32; output-gap p50/p99 (78/223 ms and 166/376 ms) and time to first token were unchanged. The Helios scheduler tests pass (312), including seven in acceptance_population_tests.rs for the cases above. Other models and decoded-history proposals were not measured in serving.
Limitations
The population applies to every catalog priced from verified acceptance, not only the measured model; DeepSeek V4/V4.1, GLM and decoded-history proposals were not run in serving with it. The population is process state and is not persisted across launches.