Skip to content

Start acceptance estimates from the acceptance of earlier requests - #306

Open
original-el8 wants to merge 1 commit into
mainfrom
scheduler/population-acceptance-prior
Open

original-el8 wants to merge 1 commit into
mainfrom
scheduler/population-acceptance-prior

Conversation

@original-el8

Copy link
Copy Markdown
Collaborator

Why

Catalogs that price speculation from verified acceptance start every request's prefix-survival estimates at 0.5^d. Under the balanced policy a drafting round that costs more than a decode step is not worth choosing at that estimate, so each request runs width-one rounds until funded acceptance probes raise its estimates, whatever the acceptance of the requests before it. Confidence-forecast catalogs already start requests from a population; acceptance-priced catalogs had none. A population of verified acceptance gives a new request the best available estimate of its first rounds without changing how its own observations then govern it.

Change

The scheduler keeps an acceptance population per proposal source and draft depth while the model is loaded. Each mean starts at 0.5^d with the weight of eight observations. A validated completion contributes an observation when it is among the reporting request's first eight at that depth; the mean averages until its weight reaches 128 and then moves by 1/128 per observation. A new request's estimates take the population means at the initial weight of two. Width-one rounds contribute nothing, and catalogs that declare device prefix allocation for learned proposals keep the fixed 0.5^d start. docs/scheduling-runtime-design.md specifies the population.

Result

Requests that follow high-acceptance requests reach ordinary drafting within their first rounds; requests that follow low-acceptance requests, and low-acceptance requests after a high-acceptance population, return to width one from their own observations. One request supplies at most half of a mean that holds only the fixed start and about 6% at the weight cap.

Evidence

Inkling 2 Small NVFP4 on four DGX Sparks (GB10, TP4), one exported core served by flare run under the balanced policy with seven draft tokens, each binary on a fresh launch. With the fixed start, the second to fifth requests of a coding prompt ran 17, 483, 197 and 70 width-one rounds before their first ordinary drafting round and produced 37 to 59 tokens/s over their first 10 s. With the population they ran 5, 3, 2 and 2 and produced 54 to 62 tokens/s. A chat prompt after those requests began drafting at once and returned to width one within 10 s (38.8 against 37.7 tokens/s over its first 10 s). Mixed prompts produced 102.4 against 98.0 tokens/s at 8 concurrent requests and 192.6 against 185.8 at 32; output-gap p50/p99 (78/223 ms and 166/376 ms) and time to first token were unchanged. The Helios scheduler tests pass (312), including seven in acceptance_population_tests.rs for the cases above. Other models and decoded-history proposals were not measured in serving.

Limitations

The population applies to every catalog priced from verified acceptance, not only the measured model; DeepSeek V4/V4.1, GLM and decoded-history proposals were not run in serving with it. The population is process state and is not persisted across launches.

Catalogs that price speculation from verified acceptance start every
request's prefix-survival estimates at 0.5^d. Under the balanced policy a
drafting round that costs more than a decode step is not worth choosing at
that estimate, so each request runs width-one rounds until funded
acceptance probes raise its estimates, whatever the acceptance of the
requests before it. Confidence-forecast catalogs already start requests
from a population; acceptance-priced catalogs had none. A population of
verified acceptance gives a new request the best available estimate of its
first rounds without changing how its own observations then govern it.

The scheduler keeps an acceptance population per proposal source and draft
depth while the model is loaded. Each mean starts at 0.5^d with the weight
of eight observations. A validated completion contributes an observation
when it is among the reporting request's first eight at that depth; the
mean averages until its weight reaches 128 and then moves by 1/128 per
observation. A new request's estimates take the population means at the
initial weight of two. Width-one rounds contribute nothing, and catalogs
that declare device prefix allocation for learned proposals keep the fixed
0.5^d start. docs/scheduling-runtime-design.md specifies the population.

Requests that follow high-acceptance requests reach ordinary drafting
within their first rounds; requests that follow low-acceptance requests,
and low-acceptance requests after a high-acceptance population, return to
width one from their own observations. One request supplies at most half
of a mean that holds only the fixed start and about 6% at the weight cap.

Evidence: Inkling 2 Small NVFP4 on four DGX Sparks (GB10, TP4), one
exported core served by flare run under the balanced policy with seven
draft tokens, each binary on a fresh launch. With the fixed start, the
second to fifth requests of a coding prompt ran 17, 483, 197 and 70
width-one rounds before their first ordinary drafting round and produced
37 to 59 tokens/s over their first 10 s. With the population they ran 5,
3, 2 and 2 and produced 54 to 62 tokens/s. A chat prompt after those
requests began drafting at once and returned to width one within 10 s
(38.8 against 37.7 tokens/s over its first 10 s). Mixed prompts produced
102.4 against 98.0 tokens/s at 8 concurrent requests and 192.6 against
185.8 at 32; output-gap p50/p99 (78/223 ms and 166/376 ms) and time to
first token were unchanged. The Helios scheduler tests pass (312),
including seven in acceptance_population_tests.rs for the cases above.
Other models and decoded-history proposals were not measured in serving.

Design-Signed-Off-By: flare-02; design-review=ed22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant