From 2fa59c5a8108db01a3df6408e02eec34b48baabf Mon Sep 17 00:00:00 2001 From: Iein Valdez Date: Sun, 15 Nov 2026 10:40:46 -0800 Subject: [PATCH] Compute checkpoint selections once per service-rate pass When the scheduler commits a step, service_participants computes token and delivery service rates for every active request. service_rates calls prefill_work for each prefill capacity, and each prefill_work call computes the request's checkpoint selection: where optional prefix checkpoints would be captured and what their snapshot storage would cost. checkpoint_planning_scope memoizes selections by request handle, and select_plan already plans inside one, but service_participants runs outside any scope, so it recomputes the same selection for every capacity of every request. A selection depends only on the request and scheduler state, and service_participants changes no scheduler state until its rates and reference intervals are computed, so one selection per request is exact for the pass. service_participants now holds a checkpoint_planning_scope while it computes service rates and delivery reference intervals, and releases it before it writes initial service prices. Each request's checkpoint selection is computed once per commit instead of once per prefill capacity. Selections, rates and prices are unchanged. Measured result: with optional checkpoints enabled at the default extra work fraction, a TP4 GLM-5.3 Flash service completes a group of 32 concurrent requests in 59.5 s and 60.0 s instead of 67.9 s and 66.3 s. GPU replay time is unchanged, so the reduction is in time outside GPU replay; it is not a GPU throughput change. Evidence. Four DGX Spark (GB10) hosts, TP4 over --roce-ring, a GLM-5.3 Flash FP4-Local core exported at e774a468, --scheduler-policy throughput and the default --scheduler-checkpoint-extra-work-fraction. Each request contains the same 400-record table and asks, at temperature 0, for the value of a different record; 8 requests run concurrently and each run uses a fresh server. Baseline is 6b68b9b4, the parent of this commit. | Runtime | Wall | GPU replay | Endpoint captures | Correct | |---|---|---|---|---| | 6b68b9b4, run 1 | 67.9 s | 57.5 s | 64 | 32/32 | | 6b68b9b4, run 2 | 66.3 s | 57.3 s | 64 | 32/32 | | this change, run 1 | 59.5 s | 57.1 s | 64 | 32/32 | | this change, run 2 | 60.0 s | 57.4 s | 64 | 32/32 | GPU replay is the change in flare_replay_gpu_nanoseconds_total across the group; endpoint captures is the change in flare_checkpoint_captures_total{role="endpoint"}. Time outside replay falls from 9.0-10.4 s to 2.4-2.6 s. The Helios scheduler tests pass (315 passed, 4 ignored). No test checks the number of selection computations; the existing tests cover the unchanged selections and prices. Not run: other models, TP1, other workloads, more than two runs per runtime, and Helios tests outside the scheduler module. Design-Signed-Off-By: Claude Code (Claude Opus 5.5); design-review=ed22 Co-Authored-By: Claude Opus 5.5 Signed-off-by: Iein Valdez --- helios/src/scheduler/service.rs | 43 +++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/helios/src/scheduler/service.rs b/helios/src/scheduler/service.rs index ad76940a1..635facd0b 100644 --- a/helios/src/scheduler/service.rs +++ b/helios/src/scheduler/service.rs @@ -283,24 +283,31 @@ impl RequestScheduler { pub(super) fn service_participants(&mut self) -> [Option; REQUEST_LANES] { let mut participants = [None; REQUEST_LANES]; let mut count = 0; - for (slot, entry) in self.slots.iter().enumerate() { - let Some(request) = entry - .request - .as_ref() - .filter(|r| !matches!(r.phase, RequestPhase::Inbound | RequestPhase::Finishing)) - else { - continue; - }; - count += 1; - let (token_rate, delivery_rate) = self.service_rates(request); - participants[slot] = Some(ServiceParticipant { - handle: request.handle, - token_rate, - delivery_rate, - }); - } - let references = self - .delivery_reference_intervals(&participants.map(|p| p.and_then(|p| p.delivery_rate))); + let references = { + // Scheduler state is unchanged until the prices below are written, so each + // request's checkpoint selection is computed once rather than once per + // prefill capacity that `service_rates` evaluates. + let _checkpoint_selections = self.checkpoint_planning_scope(); + for (slot, entry) in self.slots.iter().enumerate() { + let Some(request) = entry + .request + .as_ref() + .filter(|r| !matches!(r.phase, RequestPhase::Inbound | RequestPhase::Finishing)) + else { + continue; + }; + count += 1; + let (token_rate, delivery_rate) = self.service_rates(request); + participants[slot] = Some(ServiceParticipant { + handle: request.handle, + token_rate, + delivery_rate, + }); + } + self.delivery_reference_intervals( + &participants.map(|p| p.and_then(|p| p.delivery_rate)), + ) + }; for (slot, participant) in participants .iter() .enumerate()