Skip to content

Compute checkpoint selections once per service-rate pass - #296

Open
iein wants to merge 1 commit into
NVIDIA:mainfrom
iein:fix-292-snapshot-selection
Open

iein wants to merge 1 commit into
NVIDIA:mainfrom
iein:fix-292-snapshot-selection

Conversation

@iein

@iein iein commented Nov 14, 2026 •

Copy link
Copy Markdown

On every step commit, service_participants calls service_rates for each active request, which calls prefill_work once per prefill capacity. Each call recomputes the request's checkpoint selection, because service_participants runs outside checkpoint_planning_scope. The selection depends only on the request and scheduler state, which does not change during the pass.

This PR opens checkpoint_planning_scope in service_participants, so each selection is computed once per commit. Selections, rates and prices are unchanged.

This addresses the scheduler host cost in #292 that remains after #309. On a 4×GB10 TP4 GLM-5.3 Flash service, 32 table-lookup requests run 8 at a time at the default checkpoint extra-work fraction, with a fresh server per run:

Runtime Wall GPU replay Correct
6b68b9b (parent) 67.9 s, 66.3 s 57.5 s, 57.3 s 32/32
this PR 59.5 s, 60.0 s 57.1 s, 57.4 s 32/32

GPU time is unchanged; the reduction is host time outside GPU replay. Helios scheduler tests pass (315 passed, 4 ignored). Not run: other models, TP1, other workloads.

When the scheduler commits a step, service_participants computes token
and delivery service rates for every active request. service_rates calls
prefill_work for each prefill capacity, and each prefill_work call
computes the request's checkpoint selection: where optional prefix
checkpoints would be captured and what their snapshot storage would cost.
checkpoint_planning_scope memoizes selections by request handle, and
select_plan already plans inside one, but service_participants runs
outside any scope, so it recomputes the same selection for every
capacity of every request. A selection depends only on the request and
scheduler state, and service_participants changes no scheduler state
until its rates and reference intervals are computed, so one selection
per request is exact for the pass.

service_participants now holds a checkpoint_planning_scope while it
computes service rates and delivery reference intervals, and releases it
before it writes initial service prices. Each request's checkpoint
selection is computed once per commit instead of once per prefill
capacity. Selections, rates and prices are unchanged.

Measured result: with optional checkpoints enabled at the default extra
work fraction, a TP4 GLM-5.3 Flash service completes a group of 32
concurrent requests in 59.5 s and 60.0 s instead of 67.9 s and 66.3 s.
GPU replay time is unchanged, so the reduction is in time outside GPU
replay; it is not a GPU throughput change.

Evidence. Four DGX Spark (GB10) hosts, TP4 over --roce-ring, a GLM-5.3
Flash FP4-Local core exported at e774a46, --scheduler-policy throughput
and the default --scheduler-checkpoint-extra-work-fraction. Each request
contains the same 400-record table and asks, at temperature 0, for the
value of a different record; 8 requests run concurrently and each run
uses a fresh server. Baseline is 6b68b9b, the parent of this commit.

| Runtime | Wall | GPU replay | Endpoint captures | Correct |
|---|---|---|---|---|
| 6b68b9b, run 1 | 67.9 s | 57.5 s | 64 | 32/32 |
| 6b68b9b, run 2 | 66.3 s | 57.3 s | 64 | 32/32 |
| this change, run 1 | 59.5 s | 57.1 s | 64 | 32/32 |
| this change, run 2 | 60.0 s | 57.4 s | 64 | 32/32 |

GPU replay is the change in flare_replay_gpu_nanoseconds_total across
the group; endpoint captures is the change in
flare_checkpoint_captures_total{role="endpoint"}. Time outside replay
falls from 9.0-10.4 s to 2.4-2.6 s. The Helios scheduler tests pass
(315 passed, 4 ignored). No test checks the number of selection
computations; the existing tests cover the unchanged selections and
prices.

Not run: other models, TP1, other workloads, more than two runs per
runtime, and Helios tests outside the scheduler module.

Design-Signed-Off-By: Claude Code (Claude Opus 5.5); design-review=ed22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Iein Valdez <ivaldez@gmail.com>
@iein
iein force-pushed the fix-292-snapshot-selection branch from ea6e728 to 2fa59c5 Compare November 15, 2026 18:43
@iein iein changed the title Select one or two snapshot destinations without copying page sets Compute checkpoint selections once per service-rate pass Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant