Skip to content

Reduce CPU cost of snapshot selection and eviction quotes - #309

Merged
lukealonso merged 1 commit into
NVIDIA:mainfrom
FujitsuPolycom:codex/review-snapshot-quote
Nov 15, 2026
Merged

lukealonso merged 1 commit into
NVIDIA:mainfrom
FujitsuPolycom:codex/review-snapshot-quote

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Why

Snapshot planning can spend seconds on CPU before dispatching native graphs when thousands of snapshot slots share KV-page backing. Two-row quotes build an ordered page set for every candidate pair; repeated quote/revalidation calls can leave the GPU idle. This is shared runtime work, independent of model identity.

What

Replace per-pair set construction with an inverted page index that computes exact union costs. Single-row selection scans for minimum cost; both paths preserve the remaining-page budget and lexicographic ID ties. Pointer-sized posting offsets support aggregate indices above 32 bits. Quote simulation fills free-page backing once and updates it as eviction returns pages, replacing free-set clones and repeated membership sweeps. Pair enumeration remains quadratic; the per-pair allocations are removed. The conservative selection for more than two rows, eviction policy, reservation and stale-quote validation remain unchanged. No model computation, graph, precision or capacity changes.

This companion runtime fix can land before the row-sharded model PRs; it does not depend on their model changes and requires no re-export.

Benefit

A WSL release CPU reproducer using signed-core cache geometry and a 2,457-snapshot pool measured two-row selection at 4.578 s before versus 7.884 ms after, with identical selected IDs. In one four-GB10 TP4 flare run observation with matched request history, 16,210-token TTFT was 96.281 s before versus 3.344 s after, without the observed CPU-only pause.

Validation and limits

Validation on upstream main 5a4e8679: 315 scheduler tests passed, four ignored; the ignored large-pool quote/reference-parity test was also run explicitly and passed. Checks cover exact selected IDs/whole quotes, ties, protected aliases, budgets, eviction and lifecycle conformance.

The service observation used runtime 36c13197 before and fdeba766 after, the same signed gb10-tp4-202611140129 core (FlashInfer 7b35510b/ABI 14), and byte-identical smoke/128k-warmup/16k request history. GPU time was 2.740/3.019 s; cache capacity was 202,752/202,865 pages and 2,455/2,457 slots, with three/five cached checkpoints. Automatically sized pools and selected graph shapes differed. This is supporting host-latency evidence, not a GPU speedup or a serving measurement of the 5a4e8679-based head.

Prepared with AI assistance.

Snapshot quotes rebuild ordered page sets for every candidate pair and
repeat free-page tree scans. With thousands of snapshot slots, this CPU
work can delay native graph dispatch even when GPU execution is short.

Compute exact two-row union costs through a shared-page inverted index,
select one row linearly, and fill free backing once while simulation
updates returned pages. Preserve minimum cost, lexicographic ties,
protected aliases, page budgets and quote revalidation. The conservative
selection for more than two rows and reservation/eviction policy remain
unchanged; no model computation, graph or capacity changes.

The signed-geometry WSL CPU reproducer reduces two-row selection from
4.578 s to 7.884 ms with identical IDs. A controlled four-GB10 TP4 native
comparison with identical core/request bodies reduces 16,210-token TTFT
from 96.281 s to 3.344 s. GPU time increases from 2.740 s to 3.019 s;
this is a host-planning result, not a GPU throughput claim.

Validation on upstream base 5a4e867: 315 focused scheduler tests pass,
four are ignored, and the large-pool reference-parity test passes when
run explicitly. Tests compare complete quotes/choices, ties, protected
references, budgets, eviction and lifecycle conformance. Native timings
use runtimes 36c13197/fdeba766, slightly different auto-sized cache pools
and one matched history; no GPU run of this main-based head is claimed.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants