perf(ArcInfer/ArcKV): bound the alloc cache against real headroom — a 1 GiB cap cannot bind a card with 262 MiB free

[heydryft]

Rebased onto f709872ab. Source claims are read at that ref; measurements are from the H200 legs run on this lane, attributed inline.

⚠️ This is not the crash fix — #209 is

The V4 OOM was a headroom failure. ARC_QTIP_TCFRAG's repack retains ~63 GB against a ~79.5 GB model (ARC_QTIP_TCFRAG=0 frees 64 262 MiB vs 262 MiB), and #209 made it opt-in. This PR bounds the retention rate that rode on top of that. Both rescue the run independently, which is the honest way to state it:

leg	change	result
ARC_QTIP_TCFRAG=0, alloc cache still ON	#209	256/256 tokens, finish=length, five runs
ARC_CANDLE_ALLOC_CACHE=0, TCFRAG still ON	this PR's mechanism	1 000 tokens, finish=length, 0.19 MiB/token
both default (before either)	—	OOM at 19 tokens, ~12.8 MiB/step, held 236.9 MiB / cap 1024 MiB

The cache-off leg reported held 0.0 MiB / sizes 0 on every step, so it was genuinely out of circuit rather than merely quiet. That confirms retention is a mechanism, and that the owner is AllocCache.free on CudaDevice (candle-core/src/cuda_backend/device.rs), not a leaked Tensor — which is what the original bug report assumed. This change is defence in depth on the rate.

The defect: the bound is denominated above available VRAM

The retention bound already exists — #182's candle side (89ab14ef) has been on master since 65dc0fb4a, its mistralrs-core side since 7d91c1127. Both are ancestors of master; git merge-base --is-ancestor 7d91c1127 origin/master passes. #182 is not a missing fix and can be closed as landed-by-another-route rather than rebased.

What is wrong is the number it is set from. DEFAULT_CAPACITY_BYTES is a constant 1 GiB, picked to be "far larger than a decode step's working set and far smaller than any card this runs on". The second half is false for the configuration Arc ships: V4 leaves 262 MiB free at decode start on the measured box (844 MiB on an earlier leg), so the cap sits ~4× above everything the card has left. Crossing it is the only trigger for the LRU eviction behind it (AllocCache::put → evict_if_over_capacity), so eviction never ran once — free/step was 0.0 on every step after step 3, and the run died holding 236.9 MiB against a 1 024 MiB budget. A bound that cannot be reached is arithmetically identical to no bound. Same defect class as every other one-data-point constant in this tree.

The cross-request half of the original report is not a second leak: cuMemFreeAsync returns bytes to the CUDA async pool, and cuMemGetInfo/nvidia-smi count pool-held memory as used (utils/memory_usage.rs). trim_cuda_memory_pools() runs once, post-ISQ, never between requests.

The fix

• plan_alloc_cache_capacity(raw, free_bytes) = min(1 GiB, min(free/2, max(64 MiB, free/8))), from MemoryUsage::get_memory_available. The free/2 term stops the 64 MiB floor becoming the loosest term on exactly the hardware the policy protects — at 262 MiB free, free/8 is 32.75 MiB, under the floor, so the floor alone would have set the cap at 24% of everything left. ARC_ALLOC_CACHE_MAX_MB still wins outright, 0 (unbounded, for A/B) included; a typo still does not fall back to unbounded; no reading = previous behaviour exactly.
• arm_alloc_cache_capacity on the enable edge and every 64 decode steps, because free VRAM shrinks as the KV cache grows. set_alloc_cache_capacity evicts to the low-water mark before returning, so it also hands back what is already held above the new bound.

On the leg that died (262 MiB free) that is a 64 MiB cap against the 236.9 MiB it actually held. On the 844 MiB leg it is 105 MiB. Above 8 GiB free the 1 GiB ceiling binds instead, so every card that is not tight keeps byte-identical allocator behaviour. Only the configuration that OOMs is re-tuned.

🔑 A bound on an unreusable-by-construction growth — the counters say why

sizes (distinct retained size classes) climbed ~10 per step, and free/step was 0.0 on every step after step 3. The reuse rate of the growing buffers is zero by construction: each decode step's KV-shaped tensors are one token longer than the last, so every step files a byte size that will never be requested again. A cache entry that can never be hit is not caching anything — it is a deferred free.

So this PR makes the deferred frees bounded and prompt. The fix is to stop filing a new size every step: preallocate the KV buffers to the sequence's maximum length once and write each step in place, removing the per-step allocation rather than recycling it. That is the named follow-up — larger than this change, since it touches SingleCache's growth policy and the graph path's pinned-capacity contract, and it wants a GPU leg.

Two proposed mechanisms for the growth were killed by arithmetic rather than by argument. Both are recorded in the source doc so they are not re-proposed:

1. repeat_kv's 64× MQA expansion — unreachable. V4-Flash is 64 Q heads : 1 KV head (deepseek4.rs, pinned by v4_flash_real_config_parses), but dsv4_attention.rs gates on t_q == 1 && k_cat.dim(1)? == 1 && sdpa_params.sinks.is_some() && !absorbed_decode_disabled() and returns absorbed_mqa_decode, which folds the 64 heads into the GEMM M-dimension. Counterfactually it would allocate ~1.6 GiB per decode step at a 2 k context — it would fail on token 1, not token 19. The only other call sites (arcflash.rs) are inside #[cfg(test)] mod tests.
2. KV-length-dependent size churn (wave65-CQ) — off by ~20×. I proposed this and it is wrong. Its rate is ∝ t_c ∝ prompt length: at a 2 k prompt the arithmetic gives ~7.7 MiB/token, which fits. But the failing run's prompt was one message under 100 tokens (TTFT 0.207 s corroborates independently), where the same formula gives ~0.55 MiB/token against ~12.8 measured. Dead.

The growing term is therefore per generated token, not per-context — which is exactly what sizes += ~10/step at a short prompt says directly.

The 6.04 MiB/token in our record, reconciled

normal.rs recorded "+6.04 MiB per decoded token" for the unbounded cache, from #182's V4 leg over 2 600 tokens. Tonight's leg measured ~12.8 MiB/step and died at 19 tokens. The obvious reading — that a run surviving 2 600 tokens and a run dying at 19 cannot be the same defect — is wrong, and the arithmetic says why.

6.04 MiB/token × 2 600 tokens is 15.3 GiB retained, so #182's leg had at least that much free at decode start. Tonight's box had 844 MiB. TCFRAG landed as #203 on 08-20, after #182's measurement, and retains ~63 GB. Same per-token retention in both runs, two orders of magnitude difference in headroom. That closes it without a second mechanism.

Residual caveat, stated rather than buried: 6.04 against ~12.8 is still a 2× gap in the rate, and that is not explained. It is no longer load-bearing — what needed explaining was the survival difference — but neither number should be quoted as "the V4 rate" without its configuration. The likeliest candidate is TCFRAG itself (tonight's leg had it on; #182's predates it), and that is directly testable: the base256 arm already runs ARC_QTIP_TCFRAG=0 with the cache on, so its held/step is the discriminator. Flagged on #182.

Frame 4 of the original OOM backtrace, resolved as far as source can take it

<Tensor>::cat_contiguous::<&Tensor> is a checkable constraint. Tensor::cat reaches cat_contiguous only when every argument is contiguous, and the &Tensor monomorphization only when the call site literally passes &[&Tensor, …]. Tensor::stack collects Vec<Tensor> and calls Self::cat(&args, dim), so stack can never produce cat_contiguous::<&Tensor> — which removes every packed/fused expert-weight path in one step.

Under that constraint, an exhaustive sweep finds no Tensor::cat of F8E4M3 operands anywhere on a V4 decode forward: blockwise_fp8/ has no cat/stack at all and dequantizes to bf16 first; every DType::F8E4M3 in distributed/layers.rs is inside PackedExperts::new or FusedExperts::new, load-time, and concatenates with stack; moe/experts.rs and deepseek4.rs's compressor fuse are load-time; mxfp4/ops.rs passes &Vec<Tensor>; nvfp4's is #[cfg(test)]; and on the FP8-KV path dsv4_kv_fp8.rs casts off FP8 before the cat.

So one of three things is true, and only the box can say which: the pasted frames were not contiguous (frame 3 was elided); the symbolizer mis-attributed the inlined arm (all ten dtype arms of alloc_uninit inline into one function); or the run carried a configuration not in this tree. It does not change this PR — the OOM was a card-full condition, and the victim says what to shrink next, not what leaked.

Interaction with #211 (merged) — sequencing

A cap that is reachable means LRU eviction actually runs, and eviction cannot express "this address is baked into an instantiated CUgraphExec" — the hazard #211 registers. That hazard is masked twice over today: the cap has never been crossed on a V4 serve, so evict_if_over_capacity has never executed; and the process died at token 19–30, so no graph stayed alive long enough for its baked buffers to age to the front of the LRU queue. Any probe of it before this PR returns a green the instrument never had a chance to fail.

Not reachable in the shipping configuration — capture is behind ARC_V4_CAPTURE_PROBE, unset by default, and arc-cuda-graph drains the cache before every cuMemPoolDestroy. #211's fix must land before capture is defaulted on.

Surfaced, not shipped

• Polarity-class instance for the sweeping lane: ARC_V4_STANDARD_DENSE is read as std::env::var_os(..).is_some() in mistralrs-core/src/models/dsv4_attention.rs, so ARC_V4_STANDARD_DENSE=0 turns dense-causal attention ON for V4's two Standard layers — the spelling that reads as "off". Same class as ARC_NO_DEDICATED_DECODE. For contrast, ARC_V4_NO_ABSORBED_DECODE twenty lines earlier in the same file is correctly value-read (Ok("1") | Ok("true")), so both conventions sit side by side. Not fixed here — different subsystem, behaviour change, another lane owns the class.
• Prefix-cache view pinning. prefix_cacher.rs stores seq.normal_cache().to_vec(), and after clone_out_cache those all_data tensors are chunk() views of the cohort's batched buffer (candle narrow shares storage). One entry pins the whole B-row buffer, up to n_on_device = 16 entries, each inserted twice per request. Degenerate (free) at B=1, so it is not the measured per-token growth. The honest fix is a force_contiguous in store() — one device copy per request, which wants a GPU leg before it lands on a tight card.

Tests

10 host-runnable tests carrying the measured numbers, two of them mutation-checked against executable copies of the policy they replace rather than descriptions of it:

• the_cap_must_be_reachable_before_the_card_is_exhausted against old_policy_constant_cap → cap 1073741824 B exceeds the 884998144 B this card has free. Unmutated: 110 624 768 B planned, 774 373 376 B left free.
• the_floor_never_exceeds_half_of_what_is_free with the free/2 term dropped → planned 67108864 B against 0 B free — the floor became the loosest term on the tightest card.
• the_measured_oom_leg_is_bounded_below_what_it_actually_held pins 262 MiB free / 236.9 MiB held against the planned 64 MiB.

cargo check --workspace --all-targets green; cargo test -p mistralrs-core 689 passed, 0 failed, including doc_citations_do_not_rot after the rebase. No new rustfmt diffs (3 before, 3 after, the same 3 pre-existing).

What a GPU leg should falsify

ARC_ALLOC_CACHE_STATS=32 prints held against cap. The claim is that held plateaus at the planned cap, and free/step goes non-zero within the first few steps (it was 0.0 forever before). That is the whole claim — the crash is #209's to have fixed.

Note what this will not fix: sizes will keep climbing ~10/step, because the growth is unreusable by construction. Watching sizes go flat is the acceptance test for the preallocation follow-up, not for this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     81        29464        20161         6264         3039
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     75         4896         4893            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  147        15371        12677          824         1870
 Shell                    42        10274         6849         2750          675
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1498         1294           54          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                212        47873            0        37220        10653
 |- BASH                  72         1655         1203          331          121
 |- C                      3           17           17            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  19          779          779            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  67         2058         1723           77          258
 |- TOML                   6          207          164            0           43
 |- YAML                   5           41           36            5            0
 (Total)                            53723         4766        37762        11195
─────────────────────────────────────────────────────────────────────────────────
 Rust                    680       339917       292580        17753        29584
 |- Markdown             503        31934          471        27528         3935
 (Total)                           371851       293051        45281        33519
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1351       513965       362501        97307        54157
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━