Repository navigation
Conversation
iein
force-pushed
the
fix-292-snapshot-selection
branch
2 times, most recently
from
November 15, 2026 01:08
593d860 to
ea6e728
Compare
When the scheduler commits a step, service_participants computes token and delivery service rates for every active request. service_rates calls prefill_work for each prefill capacity, and each prefill_work call computes the request's checkpoint selection: where optional prefix checkpoints would be captured and what their snapshot storage would cost. checkpoint_planning_scope memoizes selections by request handle, and select_plan already plans inside one, but service_participants runs outside any scope, so it recomputes the same selection for every capacity of every request. A selection depends only on the request and scheduler state, and service_participants changes no scheduler state until its rates and reference intervals are computed, so one selection per request is exact for the pass. service_participants now holds a checkpoint_planning_scope while it computes service rates and delivery reference intervals, and releases it before it writes initial service prices. Each request's checkpoint selection is computed once per commit instead of once per prefill capacity. Selections, rates and prices are unchanged. Measured result: with optional checkpoints enabled at the default extra work fraction, a TP4 GLM-5.3 Flash service completes a group of 32 concurrent requests in 59.5 s and 60.0 s instead of 67.9 s and 66.3 s. GPU replay time is unchanged, so the reduction is in time outside GPU replay; it is not a GPU throughput change. Evidence. Four DGX Spark (GB10) hosts, TP4 over --roce-ring, a GLM-5.3 Flash FP4-Local core exported at e774a46, --scheduler-policy throughput and the default --scheduler-checkpoint-extra-work-fraction. Each request contains the same 400-record table and asks, at temperature 0, for the value of a different record; 8 requests run concurrently and each run uses a fresh server. Baseline is 6b68b9b, the parent of this commit. | Runtime | Wall | GPU replay | Endpoint captures | Correct | |---|---|---|---|---| | 6b68b9b, run 1 | 67.9 s | 57.5 s | 64 | 32/32 | | 6b68b9b, run 2 | 66.3 s | 57.3 s | 64 | 32/32 | | this change, run 1 | 59.5 s | 57.1 s | 64 | 32/32 | | this change, run 2 | 60.0 s | 57.4 s | 64 | 32/32 | GPU replay is the change in flare_replay_gpu_nanoseconds_total across the group; endpoint captures is the change in flare_checkpoint_captures_total{role="endpoint"}. Time outside replay falls from 9.0-10.4 s to 2.4-2.6 s. The Helios scheduler tests pass (315 passed, 4 ignored). No test checks the number of selection computations; the existing tests cover the unchanged selections and prices. Not run: other models, TP1, other workloads, more than two runs per runtime, and Helios tests outside the scheduler module. Design-Signed-Off-By: Claude Code (Claude Opus 5.5); design-review=ed22 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Iein Valdez <ivaldez@gmail.com>
iein
force-pushed
the
fix-292-snapshot-selection
branch
from
November 15, 2026 18:43
ea6e728 to
2fa59c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On every step commit,
service_participantscallsservice_ratesfor each active request, which callsprefill_workonce per prefill capacity. Each call recomputes the request's checkpoint selection, becauseservice_participantsruns outsidecheckpoint_planning_scope. The selection depends only on the request and scheduler state, which does not change during the pass.This PR opens
checkpoint_planning_scopeinservice_participants, so each selection is computed once per commit. Selections, rates and prices are unchanged.This addresses the scheduler host cost in #292 that remains after #309. On a 4×GB10 TP4 GLM-5.3 Flash service, 32 table-lookup requests run 8 at a time at the default checkpoint extra-work fraction, with a fresh server per run:
GPU time is unchanged; the reduction is host time outside GPU replay. Helios scheduler tests pass (315 passed, 4 ignored). Not run: other models, TP1, other workloads.