Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/mtp-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,10 @@ capture.
| Grammar advancement, stop sequences, output presentation | Rust with ordered token-vector completions |

Rust adapts among prepared widths from request-local prefix survival and measured
full-round costs. Every speculative graph is a complete proposal, verification,
full-round costs. A request's survival estimates start from the acceptance of
earlier requests' first rounds, as
[completed confidence and acceptance](scheduling-runtime-design.md#completed-confidence-and-acceptance)
specifies. Every speculative graph is a complete proposal, verification,
and commit program. The controller learns from its emitted/offered/accepted
records, or optional verification probabilities supplied by prepared graphs;
graphs that declare `confidence_width` also return proposal-confidence
Expand Down
26 changes: 24 additions & 2 deletions docs/scheduling-runtime-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ and selection), `utility.rs` (value arithmetic and cohort selection),
`service.rs` (service prices), `cost.rs` (duration estimates), `refresh.rs`
(timing-refresh and acceptance-learning credit), `learning.rs` (acceptance
probes), `rounds.rs` (speculative round candidates and confidence forecasts),
and `speculation.rs` (acceptance estimates).
and `speculation.rs` (acceptance estimates and their population).
[`scheduling-utility.md`](scheduling-utility.md) defines the utility and price
flow. Python experiments are in `scripts/scheduling_sim/`; they do not run in
Helios. The diagnostic `ignition debug-serve --direct` path has its own Python
Expand Down Expand Up @@ -92,6 +92,24 @@ Unobserved suffixes retain their means. Ordinary selection uses the historical
means; funded acceptance learning uses the bounded optimistic estimate
`min(1, mean + sqrt(ln(20) / (2 * aged_weight)))`.

A request's acceptance estimates start from an acceptance population that the
scheduler keeps per proposal source and draft depth while the model is loaded.
Each population mean begins at `0.5^d` with the weight of eight observations.
A validated completion adds an acceptance observation, binary or from
`completion_speculative_survival`, when it is among the reporting request's
first eight at that depth. The first observations of a request describe the
rounds that an initial estimate governs. A population mean averages its
observations until its weight reaches 128 and then moves toward each
observation with weight 1/128. One request therefore supplies at most half of a
mean that held only the fixed start, and at most `1 - (127/128)^8`, about 6%,
of a mean at the weight cap. Ordinary and funded-probe rounds contribute alike;
width-one rounds offer no draft and contribute nothing, so requests admitted
before a drafting round completes start from `0.5^d` exactly. A request's
estimate takes the population mean at the initial weight of two, and two of
its own observations at a depth weigh as much as that start. When the catalog
declares device prefix allocation for learned proposals, selection reads
confidence forecasts and acceptance estimates keep the fixed `0.5^d` start.

A prepared program may also write `completion_speculative_survival`, cumulative
products of `min(1, p(token)/q(token))` for evaluated prefixes. These replace
binary acceptance observations at those depths. They affect future selection,
Expand Down Expand Up @@ -594,7 +612,11 @@ includes its maximum packed token extent in capture headroom.
measured synthetic catalogs: one, two, and sixteen long prefills; fitting and
padded tails; high- and low-acceptance speculation; heterogeneous cohorts;
padded batch capacities; long decode pairs; prefill/decode contention; graph
expiry and recovery; and output gaps. Mixed tests compare every legal cohort
expiry and recovery; and output gaps.
`helios/src/scheduler/acceptance_population_tests.rs` covers the acceptance
population: the first drafting round of a request after high- and
low-acceptance requests, the fixed start before any drafting round, and one
request's bounded contribution. Mixed tests compare every legal cohort
against exhaustive useful-work/time enumeration, exercise fitting prompt tails,
ordinary and speculative decode rows, malformed completions, independent shape
expiry, and concurrent prompt/decode workloads. Native input tests check packed
Expand Down
6 changes: 5 additions & 1 deletion helios/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -392,7 +392,11 @@ requires confirmed stream drain before `release_after_shutdown` releases
resources associated with the attempted submission.

Each request owns conditional draft-acceptance statistics by depth and proposal
source. Validated speculative completions update only that request. The emitted
source. Validated speculative completions update that request. A request's
first eight observations at a depth also update the acceptance population from
which later requests' statistics start; see
[completed confidence and acceptance](../docs/scheduling-runtime-design.md#completed-confidence-and-acceptance).
The emitted
accepted prefix supplies positive observations. When a correction token was
emitted and an offered draft remained, the next draft supplies a rejection.
Termination within an accepted prefix censors the remaining depths. Expected
Expand Down
16 changes: 14 additions & 2 deletions helios/src/scheduler.rs
Original file line number Diff line number Diff line change
Expand Up @@ -665,6 +665,7 @@ pub struct RequestScheduler {
proposal_source: speculation::ProposalSource,
decoded_history: Option<decoded_history::DecodedHistory>,
forecasts: rounds::Forecasts,
acceptance_population: speculation::AcceptancePopulation,
costs: cost::ActionCosts,
refresh: refresh::RefreshState,
tickets: HashMap<PrefixKey, Ticket>,
Expand Down Expand Up @@ -720,6 +721,7 @@ impl RequestScheduler {
capabilities.page_tokens,
)?;
let forecasts = rounds::Forecasts::new(capabilities.completion_token_capacity);
let acceptance_population = speculation::AcceptancePopulation::new(capabilities.completion_token_capacity);
Ok(Self {
plan_owner: Arc::new(()),
limits,
Expand Down Expand Up @@ -756,6 +758,7 @@ impl RequestScheduler {
proposal_source: speculation::ProposalSource::Mtp,
decoded_history: None,
forecasts,
acceptance_population,
costs: cost::ActionCosts::default(),
refresh: refresh::RefreshState::default(),
tickets: HashMap::new(),
Expand Down Expand Up @@ -1627,7 +1630,11 @@ impl RequestScheduler {
window_snapshot_ready: false,
window_restore: false,
epoch: 0,
acceptance: speculation::AcceptanceProfile::new(self.capabilities.completion_token_capacity),
acceptance: if self.confidence_forecasts() {
speculation::AcceptanceProfile::new(self.capabilities.completion_token_capacity)
} else {
speculation::AcceptanceProfile::seeded(&self.acceptance_population)
},
accounting: self.accounting(),
handle,
state_slot: StateSlotId((1 + slot * execution_rows) as u32),
Expand Down Expand Up @@ -2491,6 +2498,7 @@ impl RequestScheduler {
let mut boundaries = Vec::new();
let mut captures = Vec::new();
let mut window_snapshots = Vec::new();
let mut population_observations = Vec::new();
for (row, lane) in step.lanes.iter().enumerate() {
let kind = step.lane_kind(row);
let Ok(request) = self.request_mut(lane.request) else {
Expand Down Expand Up @@ -2541,13 +2549,14 @@ impl RequestScheduler {
{
let vector = record.vector.as_ref().unwrap();
let offered = vector.offered_drafts as usize;
request.acceptance.observe(
request.acceptance.observe_shared(
proposal_source,
record.generated_tokens as u32,
offered,
vector.accepted_drafts as usize,
vector.emitted_count > vector.accepted_drafts,
survival.map(|rows| rows[lane.request.slot as usize].as_slice()),
Some(&mut |depth, value| population_observations.push((depth, value))),
);
}
if record.status == 3 {
Expand Down Expand Up @@ -2631,6 +2640,9 @@ impl RequestScheduler {
request.scheduled_consumed = request.consumed_tokens;
}
}
for (depth, value) in population_observations {
self.acceptance_population.observe(proposal_source, depth, value);
}
for lane in &step.lanes {
let pending = self.pending_token_ceiling(lane.request) as u64;
if let Ok(request) = self.request_mut(lane.request) {
Expand Down
249 changes: 249 additions & 0 deletions helios/src/scheduler/acceptance_population_tests.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*/

//! Seeding of a request's acceptance profile from the acceptance population.
//!
//! The fixtures are one request slot with host-allocated rounds of one to four
//! target rows lasting 28, 43.7, 48.5, and 53.2 ms, and two prefix-survival
//! curves: one at which the default policy drafts and one at which it does not.

use super::*;
use speculation::{AcceptanceProfile, ProposalSource::{Copy, Mtp}};

const ROUND_NS: [u64; 4] = [28_000_000, 43_700_000, 48_500_000, 53_200_000];
/// Expected output of 1.76, 2.32, and 2.74 tokens at two, three, and four rows.
const HIGH: [f32; 7] = [0.76, 0.56, 0.42, 0.30, 0.21, 0.15, 0.11];
/// Expected output of 1.35, 1.47, and 1.51 tokens at two, three, and four rows.
const LOW: [f32; 7] = [0.35, 0.12, 0.04, 0.01, 0.0, 0.0, 0.0];

fn measured_engine() -> RequestScheduler {
let mut s = interactive_engine(1, true);
s.enable_speculation(&[1, 2, 3, 4]).unwrap();
for (index, nanos) in ROUND_NS.into_iter().enumerate() {
set_cost(&mut s, GraphKind::Speculative, 1, index + 1, nanos);
}
s
}

fn decoding(s: &mut RequestScheduler, id: u32, output: u32) -> RequestHandle {
let handle = add(s, id, 64, output);
ready(s, &[handle]);
handle
}

/// Executes the selected plan. Each offered depth reports `curve` as its
/// survival and `accepted` drafts are emitted after the pending token.
fn round(s: &mut RequestScheduler, curve: &[f32; 7], accepted: usize) -> ScheduledPlan {
let plan = s.plan().expect("a decoding request must make progress");
let fence = Fence(s.submitted.0 + 1);
s.commit(&plan, fence).unwrap();
let duration = plan.estimated_duration_ns();
s.advance_to(s.now_ns + duration).unwrap();
let mut records = completion_records(s, &plan, fence, accepted);
let slot = plan.lanes[0].request.slot as usize;
let mut offered = 0;
if let Some(vector) = records[slot].vector.as_mut() {
offered = plan.lanes[0].tokens - 1;
vector.offered_drafts = offered as i64;
}
let mut survival = vec![-1.0; s.capabilities.completion_token_capacity - 1];
survival[..offered].copy_from_slice(&curve[..offered]);
let output = s.complete_with_survival(fence, &records, gpu_time(duration), Some(&[survival])).unwrap();
for token in output {
s.published_output(token.request, fence).unwrap();
}
let request = plan.lanes[0].request;
if s.request(request).is_ok_and(|r| r.phase == RequestPhase::Finishing && r.pending == 0) {
s.finish(request).unwrap();
}
plan
}

/// Completes `requests` requests whose own estimates keep them drafting while
/// their rounds report `curve`.
fn drafting_requests(s: &mut RequestScheduler, curve: &[f32; 7], requests: u32, rounds: usize) {
for id in 0..requests {
let handle = decoding(s, 100 + id, rounds as u32 + 16);
for _ in 0..128 {
s.request_mut(handle).unwrap().acceptance.observe(Mtp, 0, 7, 7, true, None);
}
for _ in 0..rounds {
round(s, curve, 0);
}
s.cancel(handle).unwrap();
}
}

fn seed(s: &mut RequestScheduler) -> Vec<f64> {
let handle = decoding(s, 1, 4096);
let seed = (0..7).map(|depth| s.request(handle).unwrap().acceptance
.prefix_probability(Mtp, depth).unwrap()).collect();
s.cancel(handle).unwrap();
seed
}

/// Index of the first drafting round among a request's first 64 decode
/// rounds, without acceptance probes.
fn first_drafting_round(s: &mut RequestScheduler) -> Option<usize> {
s.refresh.acceptance_credit_ns = f64::NEG_INFINITY;
(0..64).position(|_| round(s, &HIGH, 0).token_capacity > 1)
}

#[test]
fn a_request_after_high_acceptance_requests_drafts_within_its_first_rounds() {
let mut fixed = measured_engine();
decoding(&mut fixed, 1, 4096);
assert_eq!(first_drafting_round(&mut fixed), None);

// A request whose own estimate already is the reported curve still spends
// its first two decode rounds at width one.
let mut known = measured_engine();
let handle = decoding(&mut known, 1, 4096);
for _ in 0..128 {
known.request_mut(handle).unwrap().acceptance.observe(Mtp, 0, 7, 0, true, Some(&HIGH));
}
assert_eq!(first_drafting_round(&mut known), Some(2));

for (requests, expected) in [(1, Some(14)), (3, Some(5)), (16, Some(3))] {
let mut seeded = measured_engine();
drafting_requests(&mut seeded, &HIGH, requests, 8);
decoding(&mut seeded, 1, 4096);
assert_eq!(first_drafting_round(&mut seeded), expected, "{requests} requests");
}
}

#[test]
fn a_request_stays_at_width_one_after_low_acceptance_requests() {
let mut s = measured_engine();
drafting_requests(&mut s, &LOW, 3, 8);
let handle = decoding(&mut s, 1, 4096);
let profile = s.request(handle).unwrap().acceptance();
for depth in 0..3 {
assert!(profile.prefix_probability(Mtp, depth).unwrap() < 0.5f64.powi(depth as i32 + 1));
}
for _ in 0..64 {
let plan = round(&mut s, &LOW, 0);
assert!(plan.token_capacity == 1 || plan.origin.is_acceptance());
}
}

#[test]
fn requests_start_from_the_fixed_prior_until_a_drafting_round_completes() {
let fixed = AcceptanceProfile::new(16);
let mut s = measured_engine();
let handle = decoding(&mut s, 1, 4096);
s.refresh.acceptance_credit_ns = f64::NEG_INFINITY;
for _ in 0..64 {
assert_eq!(round(&mut s, &HIGH, 0).token_capacity, 1);
}
s.cancel(handle).unwrap();
let handle = decoding(&mut s, 2, 4096);
let profile = s.request(handle).unwrap().acceptance();
for source in [Mtp, Copy] {
for optimistic in [false, true] {
assert_eq!(profile.curve(source, 0, optimistic), fixed.curve(source, 0, optimistic));
}
assert_eq!(profile.information(source, 0, 0), fixed.information(source, 0, 0));
}
}

#[test]
fn one_request_moves_the_population_by_at_most_its_share() {
let mut short = measured_engine();
drafting_requests(&mut short, &[1.0; 7], 1, 8);
let mut long = measured_engine();
drafting_requests(&mut long, &[1.0; 7], 1, 2000);
let expected = seed(&mut short);
// Eight observations of 1.0 against the fixed prior's weight of eight.
for depth in 0..3 {
assert_eq!(expected[depth], (0.5f64.powi(depth as i32 + 1) + 1.0) / 2.0);
}
for depth in 3..7 {
assert_eq!(expected[depth], 0.5f64.powi(depth as i32 + 1));
}
assert_eq!(seed(&mut long), expected);

// At the weight cap, a ninth observation moves a mean by 1/128 of its error.
let mut population = speculation::AcceptancePopulation::new(16);
for _ in 0..4096 { population.observe(Mtp, 0, 0.0); }
let before = population.mean(Mtp, 0);
let mut request = AcceptanceProfile::seeded(&population);
for _ in 0..2000 {
request.observe_shared(Mtp, 0, 1, 1, true, None, Some(&mut |depth, value| population.observe(Mtp, depth, value)));
}
let share = 1.0 - (1.0f64 - 1.0 / 128.0).powi(8);
assert!((population.mean(Mtp, 0) - (before + share * (1.0 - before))).abs() < 1e-12);
assert_eq!(population.mean(Copy, 0), 0.5);
}

#[test]
fn a_low_acceptance_request_returns_to_width_one_after_a_high_acceptance_population() {
let mut s = measured_engine();
drafting_requests(&mut s, &HIGH, 16, 8);
decoding(&mut s, 1, 4096);
s.refresh.acceptance_credit_ns = f64::NEG_INFINITY;
let widths: Vec<_> = (0..64).map(|_| round(&mut s, &LOW, 0).token_capacity).collect();
let drafting = widths.iter().filter(|&&width| width > 1).count();
assert!((1..=4).contains(&drafting), "{widths:?}");
assert!(widths[32..].iter().all(|&width| width == 1), "{widths:?}");
}

#[test]
fn confidence_forecast_catalogs_keep_the_fixed_acceptance_prior() {
let fixture = measured_engine();
let mut capabilities = fixture.capabilities.clone();
capabilities.speculative_allocations = vec![(1, 7, 8, 8, crate::execution::PrefixAllocation::Gpu)];
let mut s = RequestScheduler::new([0; 32], fixture.limits.clone(), capabilities).unwrap();
for depth in 0..7 {
for _ in 0..64 { s.acceptance_population.observe(Mtp, depth, 1.0); }
}
let handle = add(&mut s, 1, 64, 64);
let fixed = AcceptanceProfile::new(16);
assert_eq!(s.request(handle).unwrap().acceptance().curve(Mtp, 0, false), fixed.curve(Mtp, 0, false));
}

/// Deterministic uniform draws for the simulated acceptance outcomes.
struct Draws(u64);

impl Draws {
fn next(&mut self) -> f32 {
self.0 = self.0.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
(self.0 >> 40) as f32 / (1u64 << 24) as f32
}
}

/// Five consecutive 1,024-token requests on text with the `HIGH` curve. Each
/// round's accepted prefix is drawn from that curve. Prints, per request, the
/// width-one rounds before its first ordinary drafting round, its width-one
/// and total rounds, and its simulated decode time.
#[test]
fn simulated_requests_reach_drafting_sooner_as_the_population_fills() {
let mut s = measured_engine();
let mut draws = Draws(1);
let mut before_drafting = Vec::new();
let mut seconds = Vec::new();
for id in 0..5 {
let handle = decoding(&mut s, id, 1024);
let start = s.now_ns;
let (mut first, mut narrow, mut rounds) = (None, 0, 0);
while s.request(handle).is_ok() {
let draw = draws.next();
let width = s.plan().unwrap().token_capacity;
let accepted = HIGH[..width - 1].iter().take_while(|&&survival| draw < survival).count();
let plan = round(&mut s, &HIGH, accepted);
if plan.token_capacity == 1 { narrow += 1; }
else if plan.origin == PlanOrigin::Ordinary { first.get_or_insert(rounds); }
rounds += 1;
}
let elapsed = (s.now_ns - start) as f64 / 1e9;
println!("request {id}: {} width-one rounds before ordinary drafting, {narrow} of {rounds} rounds at width one, {elapsed:.2} s, {:.1} tokens/s",
first.unwrap_or(rounds), 1024.0 / elapsed);
before_drafting.push(first.unwrap_or(rounds));
seconds.push(elapsed);
}
assert!(before_drafting[0] > 100, "{before_drafting:?}");
assert!(before_drafting[1..].iter().all(|&rounds| rounds <= 8), "{before_drafting:?}");
assert!(seconds[1..].iter().all(|&elapsed| elapsed < seconds[0]), "{seconds:?}");
}
Loading
Loading