Skip to content

Reclaim host prefix rows in least-recently-used order - #302

Merged
lukealonso merged 1 commit into
mainfrom
martin/host-row-thinning-recency
Nov 15, 2026
Merged

lukealonso merged 1 commit into
mainfrom
martin/host-row-thinning-recency

Conversation

@voipmonitor

Copy link
Copy Markdown
Collaborator

The DRAM prefix store dropped the continuation row of every parent record as soon as it had one child and retention space was short. A conversation evicted from the GPU then had pages in host memory but no row to resume from, and prefilled its whole prompt again.

Cause

Core::thin_rows_for treats the row of a non-root record with exactly one child as redundant, and once free retention space falls to two rows it drops all queued ones at once. One child does not show that the parent's row is unused: a request that branches from a checkpoint (a client side request, or history the client renders differently from the generated tokens) stores a child that the original conversation never extends. The parent keeps its pages, stops matching lookups, and the conversation loses its only host resume point.

Change

Redundant rows are reclaimed in least-recently-used order, and only until the headroom is restored. Rows that were not needed stay queued. The redundancy test, pins, the headroom threshold, leaf eviction and the disk store are unchanged.

Result and evidence

Expected: a recently used parent keeps its row while older redundant rows exist, so a returning conversation can be restored from host memory. This is not measured on a GPU service in this PR.

Check Result
Kimi K3.1 server, 128 GiB DRAM store, agent traffic, before the change 4,369 records stored, 4,070 rows dropped; about 1,300 host lookups, 16 restores; two requests of a conversation idle for 2 and 3 minutes each prefilled more than 300,000 tokens again
pressure_keeps_the_row_of_a_recently_used_parent_whose_child_branched_away fails before, passes after
Selene library suite, Linux x86_64, Rust 1.95.0 402 passed, 0 failed

Independent of #298, which changes GPU-side retention and restore admission. No GPU execution or service benchmark was run; no measurement files are included.

🤖 Generated with Claude Code

The DRAM prefix store treated the row of every non-root record with exactly
one child as redundant. Once free retention space fell to two continuation
rows, storing a record dropped the rows of all such parents at once. A single
child does not show that the parent's row is unused. A request that branches
from a checkpoint, such as a client side request or history the client renders
differently from the generated tokens, stores a child that the original
conversation never extends. The parent then keeps its pages but no longer
matches lookups, so a conversation evicted from the GPU cannot be restored from
host memory and prefills its whole prompt again. Ordering reclamation by use
keeps the rows that requests reused most recently, which are the ones a
returning conversation resumes from, without adding a retention rule or a cap.

Core::thin_rows_for now reclaims queued redundant rows in ascending access
order and stops once the free retention space exceeds the requested headroom.
Rows it did not need stay queued. The redundancy test, pin handling, the
headroom threshold, and leaf eviction are unchanged, and the disk store is not
affected. docs/prefix-cache-offload.md states the order and its reason.

The expected result is that a recently used parent keeps its row while older
redundant rows exist, so a host lookup for the conversation it belongs to
still finds a resume point. This is an expected benefit; it is not measured
on a GPU service in this commit.

Evidence. A Kimi K3.1 server with a 128 GiB DRAM store under agent traffic
stored 4,369 records and dropped the rows of 4,070, and about 1,300 host
lookups produced 16 restores; two requests of one conversation that had been
idle for two and three minutes each prefilled more than 300,000 tokens again.
Those figures were recorded before this change. The regression test
pressure_keeps_the_row_of_a_recently_used_parent_whose_child_branched_away
stores a chain, reuses its last checkpoint, and stores a branching child under
pressure: it fails before the change and passes after it. The Selene library
suite passes with 402 tests on Linux x86_64 with Rust 1.95.0. No GPU execution
or service benchmark was run, and the effect on restore rates is unqualified.

Signed-off-by: Martin Vit <martin@voipmonitor.org>
Design-Signed-Off-By: Claude (Fable 5.2); design-review=ed22
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants