Repository navigation
Reclaim host prefix rows in least-recently-used order - #302
Merged
Merged
Conversation
The DRAM prefix store treated the row of every non-root record with exactly one child as redundant. Once free retention space fell to two continuation rows, storing a record dropped the rows of all such parents at once. A single child does not show that the parent's row is unused. A request that branches from a checkpoint, such as a client side request or history the client renders differently from the generated tokens, stores a child that the original conversation never extends. The parent then keeps its pages but no longer matches lookups, so a conversation evicted from the GPU cannot be restored from host memory and prefills its whole prompt again. Ordering reclamation by use keeps the rows that requests reused most recently, which are the ones a returning conversation resumes from, without adding a retention rule or a cap. Core::thin_rows_for now reclaims queued redundant rows in ascending access order and stops once the free retention space exceeds the requested headroom. Rows it did not need stay queued. The redundancy test, pin handling, the headroom threshold, and leaf eviction are unchanged, and the disk store is not affected. docs/prefix-cache-offload.md states the order and its reason. The expected result is that a recently used parent keeps its row while older redundant rows exist, so a host lookup for the conversation it belongs to still finds a resume point. This is an expected benefit; it is not measured on a GPU service in this commit. Evidence. A Kimi K3.1 server with a 128 GiB DRAM store under agent traffic stored 4,369 records and dropped the rows of 4,070, and about 1,300 host lookups produced 16 restores; two requests of one conversation that had been idle for two and three minutes each prefilled more than 300,000 tokens again. Those figures were recorded before this change. The regression test pressure_keeps_the_row_of_a_recently_used_parent_whose_child_branched_away stores a chain, reuses its last checkpoint, and stores a branching child under pressure: it fails before the change and passes after it. The Selene library suite passes with 402 tests on Linux x86_64 with Rust 1.95.0. No GPU execution or service benchmark was run, and the effect on restore rates is unqualified. Signed-off-by: Martin Vit <martin@voipmonitor.org> Design-Signed-Off-By: Claude (Fable 5.2); design-review=ed22 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The DRAM prefix store dropped the continuation row of every parent record as soon as it had one child and retention space was short. A conversation evicted from the GPU then had pages in host memory but no row to resume from, and prefilled its whole prompt again.
Cause
Core::thin_rows_fortreats the row of a non-root record with exactly one child as redundant, and once free retention space falls to two rows it drops all queued ones at once. One child does not show that the parent's row is unused: a request that branches from a checkpoint (a client side request, or history the client renders differently from the generated tokens) stores a child that the original conversation never extends. The parent keeps its pages, stops matching lookups, and the conversation loses its only host resume point.Change
Redundant rows are reclaimed in least-recently-used order, and only until the headroom is restored. Rows that were not needed stay queued. The redundancy test, pins, the headroom threshold, leaf eviction and the disk store are unchanged.
Result and evidence
Expected: a recently used parent keeps its row while older redundant rows exist, so a returning conversation can be restored from host memory. This is not measured on a GPU service in this PR.
pressure_keeps_the_row_of_a_recently_used_parent_whose_child_branched_awayIndependent of #298, which changes GPU-side retention and restore admission. No GPU execution or service benchmark was run; no measurement files are included.
🤖 Generated with Claude Code