docs(ArcGraph): LRU eviction cannot see which addresses a live graph baked (wave65-CR) — issues are disabled, so this is the issue by heydryft · Pull Request #211 · aeonmindai/arc

heydryft · 2026-08-21T12:45:09Z

Docs only. Registers two silent hazards in the forked candle CUDA caching allocator that belong to neither of the two diverged candle lines, and therefore have no owner unless written down. Issues are disabled on this repo, so this is the issue.

Found while merging master's candle 89ab14ef ("eviction must not run inside the capture window") with the capture branch's 9211966e into 9586979d on aeonmindai/candle@arcgraph/bounded-alloc-plus-prewarm.

No hardware leg was taken. Every claim is derived from source — arc refs at bb30559, candle claims at 9586979d. Borrowed measurements (the 1026.1 MiB peak, the 6.04 MiB/token leak, the ~315 MiB prewarm pool, tonight's OOM figures) are attributed inline to whoever recorded them and marked as theirs.

Hazard 1 — LRU eviction cannot see which addresses a live graph baked

AllocCache stores (bytes, ptr) and nothing about whether an address is baked into a CUgraphExec's memory nodes. After a successful capture the graph's buffers are dropped by the eager tensors that held them and land back in free, indistinguishable from anything else.

This is the steady state rather than a corner case for a structural reason: a cuGraphLaunch executes baked addresses and calls no cache_take, so replay never refreshes a tick — the graph's buffers age monotonically from the moment capture ends. Meanwhile default_verify_replays() is 3, so eager forwards keep stamping fresh ticks on every other size, and drain_alloc_cache_and_free runs only on pool-destroying paths, never after a successful instantiate.

Raising the cap, draining after instantiate, and suspending eviction are each ruled out in the doc with the reason. The proposed fix is a pin the allocator can express. Probe: alloc_cache_stats().evicted_at_demand_size == 0 for the life of the graph — a counter 9586979d adds for exactly this.

Hazard 2 — `ARC_NO_DEFERRED_FREE` silently disables the capture instruments

capturing never becomes true, so the demand profile stays empty, prewarm_alloc_cache returns (0, 0), and capture_miss_count() returns 0 — the refuse-to-instantiate-on-a-miss gate passes unconditionally. A clean capture and a silently disabled capture report identical numbers. Predates both candle lines.

Interaction with #182 and the KV `cat` growth (new section)

#182 is not a neighbour of hazard 1 — it is its precondition. perf/alloc-cache-bounded is the arc-side landing of the same allocator line; its candle side is b2a4dd80 → 89ab14ef, one of the two lines merged here. Before the bound there was no eviction at all, so hazard 1 had nothing to fire with. #182 converts a bounded memory leak into a correctness risk that is live exactly when a graph is live. Still the right trade, but a trade, and now on the record as one.

The KV cat_contiguous growth sets hazard 1's rate. Retention only reaches the cap because something presents byte sizes nothing asks for again, and a buffer whose width tracks KV length is exactly the shape b2a4dd80 named. More churn per token → cap crossed sooner → the sweep runs more often, and the sweep is what can reach a graph's baked buffers.

This does not settle whether #182 and the KV growth are one bug or two — that is being settled elsewhere and the doc must not pre-empt it. Both branches are written out and hazard 1's mechanism is unchanged under either, so the pin is needed either way.

Consequence: hazard 1 is currently masked. A process that OOMs at token 22–30 never keeps a graph alive long enough for its baked buffers to age to the front of the queue. The probe cannot return a meaningful zero or non-zero until the OOM is fixed — a green probe today would be a green light from an instrument that never had the chance to go red, the same failure shape as hazard 2 by a different road.

CI

Typos — was red on arange_step ×2. .typos.toml allow-lists arange as a whole identifier for markdown, but arange_step is a different identifier, so it gets split and the arange word is flagged. Fixed by rewording, not by widening the config: every other md file in the repo already uses the bare arange, and the call site is now cited as deepseek4.rs:3033, which is more useful than the symbol name.
doc_citations — would have hard-failed and had not run yet. candle is not vendored, so device.rs:N resolves by component-suffix match to arc-profiler/src/device.rs (268 lines): all 11 candle citations were LINE-OUT-OF-RANGE, i.e. SYMBOL-ROT, which has no waiver path. Candle is now cited by file+symbol with no :N, and a preamble note says why so nobody adds them back. Verified by running the gate standalone against this tree (it is std-only): 12/12 green, and an engagement check re-introducing device.rs:1054 reproduces the exact failure first, so the green is not vacuous.
arc citations re-verified against current master. normal.rs:248 → 299 (moved by fix(ArcGraph): the dense-layer inventory accepts a model whose layers are in the wrong order #208). deepseek4.rs:3009/3013/6031-6032 and graph.rs:56/214/278/517 unchanged. Added deepseek4.rs:3033, and made the :6033-6041 range a real citation rather than decoration.

Sequencing

Against 9586979d, held-out commits 492b7baa7, 1c371b2cc and 8968c1fa3 resolve. f7c65cb72 and 4475f9e71 must never be cherry-picked — their pins to 5bdddcf / 9211966 would revert master's eviction-inside-capture fix. They collapse into one rev bump plus a lock regen, which is not part of this PR.

Also flagged for #207: wave65-CQ item 2 would inherit the same single-slot alternation defect if it memoizes the cos/sin gathers on the existing (t_c, ratio, device) pattern verbatim. Key on the pair, or use a small map, not one slot.

Related: #205 (capture lane), #182 (allocator bound), #207 / wave65-CQ.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7

github-actions · 2026-08-21T12:46:54Z

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     81        29145        19998         6130         3017
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     75         4896         4893            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  147        15371        12677          824         1870
 Shell                    42        10274         6849         2750          675
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1498         1294           54          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                211        47578            0        36992        10586
 |- BASH                  72         1655         1203          331          121
 |- C                      3           17           17            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  19          779          779            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  67         2058         1723           77          258
 |- TOML                   6          207          164            0           43
 |- YAML                   5           41           36            5            0
 (Total)                            53428         4766        37534        11128
─────────────────────────────────────────────────────────────────────────────────
 Rust                    679       339224       291995        17706        29523
 |- Markdown             502        31672          471        27302         3899
 (Total)                           370896       292466        45008        33422
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1349       512396       361753        96672        53971
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

…zard (wave65-CR) Two silent hazards found while merging the two diverged candle lines (89ab14ef + 9211966e -> 9586979d on aeonmindai/candle). Neither belongs to either line, so neither has an owner unless it is written down. Issues are disabled on this repo, so this is the issue. 1. LRU eviction cannot express "this address is baked into a CUgraphExec". After a successful capture the graph's buffers land back in `free`, indistinguishable from any other cached buffer, and replay never refreshes their tick — a cuGraphLaunch executes baked addresses and calls no cache_take. Meanwhile verify-replay eager forwards keep stamping every other size. The graph's buffers therefore go from newest to oldest, and the first eager put that crosses the 1 GiB cap frees memory the next launch writes into. Proposed fix is a pin the allocator can express, held out of the merge because it is a new mechanism on both sides of the candle boundary. Probe: `evicted_at_demand_size == 0`, a counter the merge adds for exactly this. 2. ARC_NO_DEFERRED_FREE silently disables the capture instruments. `capturing` never becomes true, so the demand profile stays empty, prewarm returns (0, 0), and the refuse-to-instantiate-on-a-miss gate passes unconditionally. A clean capture and a silently disabled capture report identical numbers. Predates both candle lines. Cross-references the compressed_row_positions single-slot thrash found independently by the capture-lane rebase — V4's layers strictly alternate CSA(4)/HCA(128) (deepseek4.rs:6031-6032), so a cache keyed on `ratio` misses on all 41 compressed layers — because the two are distinct root causes behind one observed crash and neither closes the other. Also flags that wave65-CQ item 2 would inherit the same alternation defect if it copies the existing memo pattern verbatim. Docs only. No hardware leg; every claim is derived from source, with candle refs at 9586979d and arc refs at cc5487a, and borrowed measurements attributed inline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… record the #182 interaction Three fixes to wave65-CR. Typos: `arange_step` twice. `.typos.toml` allow-lists `arange` as a whole identifier under `[type.md.extend-identifiers]`, but `arange_step` is a different identifier, so typos splits it and flags the `arange` word. Fixed by rewording, not by widening the config — every other md file in the repo (CLAUDE.md, wave64-CP, BUDGET_V4_B1, 00_RESUME_HERE) already refers to the bare `arange`, and the call site is now cited by line instead, which is more useful than the symbol name anyway. doc_citations: this would have hard-failed, and the coordinator's warning is what caught it. candle is not vendored here, so `device.rs:N` resolves by component-suffix match to arc-profiler/src/device.rs (268 lines) and every one of the 11 candle citations was LINE-OUT-OF-RANGE — SYMBOL-ROT, which has no waiver path. All candle references now name file and symbol with no `:N`, and a note in the preamble says why so nobody adds them back. Verified by running the gate standalone (it is std-only) against this tree: 12/12 green, and an engagement check re-introducing `device.rs:1054` reproduces the exact failure first, so the green is not vacuous. Re-verified every arc citation against current master. normal.rs:248 -> 299 (moved by #208). deepseek4.rs:3009/3013/6031-6032 and graph.rs:56/214/278/517 unchanged. Added deepseek4.rs:3033 for the arange call site and made the :6033-6041 range a real citation rather than decoration. New section on the interaction the coordinator asked about. #182 (perf/alloc-cache-bounded) is not a neighbour of hazard 1 — it is the arc-side landing of the same allocator line, and therefore its precondition: before the bound there was no eviction at all, so hazard 1 had nothing to fire with. The KV cat_contiguous growth sets hazard 1's rate, because retention only reaches the cap when something churns distinct byte sizes per token. Both branches of the open one-bug-or-two question are written out and leave hazard 1's mechanism unchanged, so this does not pre-empt the agent settling it. Records that hazard 1 is currently masked: a process that OOMs at token 22-30 never keeps a graph alive long enough to observe the eviction, so a green probe today would come from an instrument that never had a chance to go red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… V4 — 1 GiB of budget on a card with 844 MiB free ## The defect V4 dies with `CUDA_ERROR_OUT_OF_MEMORY` after 22-30 generated tokens, on the only KV path it has (eager; PagedAttention does not support MLA). Measured on an H200: 142 927 -> 143 151 MiB over 20 tokens, ~11 MiB/token in ~32 MiB granules, and the next request starts at the previous one's high-water mark. The retention bound on candle's caching allocator already exists (#182's candle side, `89ab14ef`, on master since `65dc0fb4a`). It is set from a constant — 1 GiB — chosen to be "far smaller than any card this runs on". That is false for the configuration Arc ships: DeepSeek-V4 loads to 142 927 MiB of 143 771, so decode starts with **~844 MiB free**. A 1 024 MiB cap cannot be crossed before the card is exhausted, and crossing it is the *only* thing that triggers the LRU eviction behind it. The bound was present and out of reach, so retention was unbounded in practice — arithmetically identical to having no cap at all. ## The fix Plan the cap against the headroom the card actually reports, and re-arm it during decode because free VRAM shrinks as the KV cache grows: * `plan_alloc_cache_capacity(raw, free_bytes)` — `min(1 GiB, max(64 MiB, free/8))`. An explicit `ARC_ALLOC_CACHE_MAX_MB` still wins outright, `0` (unbounded, for A/B) included. `None` free reading = previous behaviour. * `arm_alloc_cache_capacity` on the enable edge and every 64 decode steps. `set_alloc_cache_capacity` evicts to the new low-water mark before returning, so this also hands back whatever is already held above the new bound. At V4's headroom the cap becomes 105 MiB, less than half the 224 MiB the failing run consumed, leaving ~739 MiB to the forward. Above 8 GiB free the 1 GiB ceiling binds instead, so **every card that is not tight keeps byte-identical allocator behaviour** and only the configuration that OOMs is re-tuned. ## What this does NOT do, and what it un-masks It bounds retention; it does not remove the per-token size churn that drives retention up to the cap. That churn is the `t_c`-sized `Tensor::cat` in `compressed_kv_from_rows` -> `forward_at_positions` (~22 MB/token, wave65-CQ), plus every other KV-length-dependent transient. Removing it is a separate change with a wrong-answer hazard of its own. A reachable cap means eviction actually runs, and eviction cannot express "this address is baked into an instantiated CUgraphExec" (PR #211). That hazard is currently masked by this very defect — a process that dies at token 22 never keeps a graph alive long enough for its baked buffers to age to the front of the queue — so landing this makes it reachable for the first time. It is not reachable in the shipping config (capture is behind `ARC_V4_CAPTURE_PROBE`, unset by default; `arc-cuda-graph` drains before every `cuMemPoolDestroy`), but #211 must land before capture is defaulted on. ## Tests 8 new host-runnable tests carrying the measured H200 numbers. `the_cap_must_be_reachable_before_the_card_is_exhausted` is mutation-checked against `old_policy_constant_cap`, an executable copy of the previous policy: with it, `cap 1073741824 B exceeds the 884998144 B this card has free`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7

… V4 — 1 GiB of budget on a card with 844 MiB free ## The defect V4 dies with `CUDA_ERROR_OUT_OF_MEMORY` after 22-30 generated tokens, on the only KV path it has (eager; PagedAttention does not support MLA). Measured on an H200: 142 927 -> 143 151 MiB over 20 tokens, ~11 MiB/token in ~32 MiB granules, and the next request starts at the previous one's high-water mark. The retention bound on candle's caching allocator already exists (#182's candle side, `89ab14ef`, on master since `65dc0fb4a`). It is set from a constant — 1 GiB — chosen to be "far smaller than any card this runs on". That is false for the configuration Arc ships: DeepSeek-V4 loads to 142 927 MiB of 143 771, so decode starts with **~844 MiB free**. A 1 024 MiB cap cannot be crossed before the card is exhausted, and crossing it is the *only* thing that triggers the LRU eviction behind it. The bound was present and out of reach, so retention was unbounded in practice — arithmetically identical to having no cap at all. ## The fix Plan the cap against the headroom the card actually reports, and re-arm it during decode because free VRAM shrinks as the KV cache grows: * `plan_alloc_cache_capacity(raw, free_bytes)` — `min(1 GiB, max(64 MiB, free/8))`. An explicit `ARC_ALLOC_CACHE_MAX_MB` still wins outright, `0` (unbounded, for A/B) included. `None` free reading = previous behaviour. * `arm_alloc_cache_capacity` on the enable edge and every 64 decode steps. `set_alloc_cache_capacity` evicts to the new low-water mark before returning, so this also hands back whatever is already held above the new bound. At V4's headroom the cap becomes 105 MiB, less than half the 224 MiB the failing run consumed, leaving ~739 MiB to the forward. Above 8 GiB free the 1 GiB ceiling binds instead, so **every card that is not tight keeps byte-identical allocator behaviour** and only the configuration that OOMs is re-tuned. ## What fills the cache is still open — and two candidates are dead This bounds retention; it does not identify what drives retention to the bound. Both mechanisms proposed tonight were killed by arithmetic, and both are recorded in the source doc so they are not re-proposed: * `repeat_kv`'s 64x MQA expansion is **unreachable** — V4 decode returns through `absorbed_mqa_decode`. Counterfactually it would allocate ~1.6 GiB per decode step at 2 k context: it would fail on token 1, not token 22. * KV-length-dependent size churn (wave65-CQ) is **off by ~20x** — its rate is proportional to `t_c`, hence to prompt length, and the failing run's prompt was one message under 100 tokens (TTFT 0.207 s corroborates). The arithmetic gives ~0.55 MiB/token against 11 measured. The survivor must be per-generated-token rather than per-context, and files a new byte size each step (the ~32 MiB granule is size-class quantisation). `ARC_CACHE_DEBUG=1` names it directly. ## What this un-masks A reachable cap means eviction actually runs, and eviction cannot express "this address is baked into an instantiated CUgraphExec" (PR #211). That hazard is currently masked by this very defect — a process that dies at token 22 never keeps a graph alive long enough for its baked buffers to age to the front of the queue. Not reachable in the shipping config (capture is behind `ARC_V4_CAPTURE_PROBE`, unset; `arc-cuda-graph` drains before every `cuMemPoolDestroy`), but #211 must land before capture is defaulted on. ## Tests 8 new host-runnable tests carrying the measured H200 numbers. `the_cap_must_be_reachable_before_the_card_is_exhausted` is mutation-checked against `old_policy_constant_cap`, an executable copy of the previous policy: with it, `cap 1073741824 B exceeds the 884998144 B this card has free`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7

heydryft · 2026-08-21T13:32:28Z

Cross-reference from #213 (headroom-aware allocator cap).

#213 makes this hazard reachable for the first time, and the sequencing matters.

The hazard is currently masked by a different defect: candle's alloc-cache retention cap is a constant 1 GiB, and DeepSeek-V4 leaves only ~844 MiB free on an H200, so evict_if_over_capacity (candle-core/src/cuda_backend/device.rs:293) has never run once in a V4 serve — the cap cannot be crossed before the card is exhausted. On top of that, the process dies at token 22–30, so no graph stays alive long enough for its baked buffers to age to the front of the LRU queue. Any probe of this today returns a green the instrument never had a chance to fail.

#213 plans the cap from measured free VRAM (105 MiB at V4's headroom), which makes eviction run continuously from the first decode step — more reachable than the constant, deliberately.

It is not reachable in the shipping config: capture is behind ARC_V4_CAPTURE_PROBE, unset by default (normal.rs:2055), and arc-cuda-graph drains the cache before every cuMemPoolDestroy (graph.rs:56, :214). So the order is #213 first, this second, and this must be in before capture is defaulted on.

The box measured the missing ~60 GB after this file was drafted: ARC_QTIP_TCFRAG=0 frees 64,262 MiB against 262 MiB with it on. tcfrag_words is a per-weight OnceLock caching a repacked copy of every weight for the process lifetime — ~63 GB against a ~79.5 GB model. #182's leg predates TCFRAG, so it had ~15.7 GB of headroom and survived 2,600 tokens at the same per-token rate. Same leak, two orders of magnitude less room. The concurrency-budget hypothesis is WITHDRAWN. It was a fair reading of the evidence available; the measurement removed the need for it. Attribution corrected: two independent single-variable rescues each work alone (ARC_QTIP_TCFRAG=0 -> 256/256 tokens x5; ARC_CANDLE_ALLOC_CACHE=0 -> 1,000 tokens), so #209 restored the headroom and #213 is defence in depth on the rate. Also splits an over-claim of my own. "Every single-user number was measured inside a crashing run" is two claims and only one is universal: A: every V4 single-user tok/s figure is a <=30-token short window. Stands. B: "inside a crashing run" holds only at or after cc5487a (#203, today). The b=1 trio is from 08-17, before TCFRAG existed - its 24 was a harness choice, not a truncation. Claiming otherwise would be a fabrication. And the 33.4 -> 34.2 pair is a third case: its source CAPTURE_LANE.md exists nowhere in this repo, on any branch. Unsourced, like the 27.2% and the 16.0%. Drops the unsourced "declined 4 of 4 loads" claim, records that #210 merged with a projection multiplied out of the unsourced 16.0% share, and notes #211 merged (registers the hazard; the pin is still unwritten) - and that #209 un-masks it, since 256-token runs now live long enough for the probe to mean something. cuda_ops.rs: fix the mis-quote #209 shipped. The header phrase "the whole of the b=1 decode path" describes qtip2b_gemv_tuned_kernel, not TCFRAG. The conclusion rests on the dispatch sites instead. Adds the 63 GB retention to the gate's doc comment, where the next reader of #203 will look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7

…held 63 GB (wave66-CS) (#214) * docs(ArcInfer/ArcKV): V4 cannot serve past ~22 tokens, so every single-user number is void (wave66-CS) The durable record of session 9. Issues are disabled on this repo, so this is the issue. THE BLOCKER. At cc5487a on an H200, batch=1 decode dies with CUDA_ERROR_OUT_OF_MEMORY after 22-30 generated tokens, 3/3 runs, two independent arms. Growth is ~11 MiB/token and is never released across requests: one run ended at 142,959 MiB used and the next started there. The card is genuinely full at failure, 143,151 of 143,771 MiB. Consequence, stated as strongly as the evidence supports: every single-user V4 throughput number this project has recorded was measured over <=30 tokens before a crash. The three-way b=1 agreement (15.11 / 15.39 / 15.51 tok/s) has "24 tokens" written in its own FACTS row. The best clean fragment tonight was 37.35 tok/s over 20 tokens - a fragment, not a rate. Six more findings, each with its ref: TCFRAG was default-ON with "UNVERIFIED ON HARDWARE - NEVER RUN" in its own header (fixed, #209); two host round-trips per decode step, not one; the dedicated decode tier cannot accept V4 architecturally; the b=1 FP8 GEMV is instruction-bound; four instrument failures; and #182 is a precondition whose hazard is masked by the OOM. Retractions added to FACTS: the 5,300-cast / 1,571-MHC premise (stale, profiled at 05af600), DecodeKvGeometry (never existed in Rust), the 27.2% TAIL (sole source is a docs file not on master), the latency-bound b=1 story, and "op count is retired as a lever". Corrections made while verifying, all against 9c127a2 rather than against the report: normal.rs:2515 -> :2619; has_cached_kv_info is 3 prose references and 0 definitions, not zero occurrences; ba026e9 is a root/import commit and introduced nothing; the TCFRAG OnceLock does not panic, it caches None permanently; tcfrag2b.rs:151-157 -> :151-158; the 2.97 TB/s issue ceiling used the FP32 warp rate, and the INT32 pipe gives 1.49, so the defensible claim is a 1.5-3.0 TB/s range; ARC_NO_DEFERRED_FREE was overstated; and mark_unreachable being dark was already on the record from session 8. Also surfaced, unresolved: PR #182's H200 leg ran 2,600 tokens with no OOM, which needs >=15.7 GB free at decode start against tonight's 844 MiB. Nobody has explained the ~60 GB gap, and if it is the serve config then this is a concurrency-budget defect as much as a per-token leak. Gates: doc_citations 12/12 green, run standalone against this tree, and the gate reproduced a real failure on an earlier revision of this file first, so the green is not vacuous. typos clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: put wave66-CS in the bounded read order, and point FACTS at §2026-08-21 The blocker block at the top is no use if the read order still sends people to the session-8 FACTS section first. Adds wave66-CS as item 2 (before the census, because a GPU-spend decision made on void single-user numbers is worse than one made late) and renumbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: make the op-count retraction exact — the captured arm kept both round-trips The first wording said the 1,137x launch cut "left two host syncs in both arms", which is wrong: the eager arm never replays a graph, so it never reaches graph.rs:362. The correct and stronger statement is about the CAPTURED arm — it still pays the replay sync and the sampler's 4-byte D2H, so the CPU never left the loop and the ~8% cannot be read as the value of removing it. Also adds the guard against over-claiming in the other direction: this does not show the two syncs ARE the limiter, only that the experiment could not see them. The number that would settle it has not been taken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: two wording fixes — "three corrections" not two, and the =0 polarity cuts both ways The env-flag consequence was stated as "both arms ran with the feature on". That holds for the 13 .is_some() flags; for the 5 .is_none() ones, =0 counts as set and therefore SUPPRESSES, so the legs are inverted rather than identical. Either way the control leg was not a control, but the two failure shapes are different and a reader debugging an old A/B needs to know which one they have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: name where PR #210's 16.0% kernel share actually comes from It was cross-referenced loosely as "the same provenance problem as the 27.2%". Both numbers live on the same line of docs/engineering/OPENROUTER_READY.md, which is not on master — it exists only on agent/tail-sinkhorn-warp-v2 and agent/decode-share-probe. One of the two is already being multiplied into a projected win, so the pointer should be explicit rather than a cross-reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: flag the mis-quote #209 shipped into cuda_ops.rs #209's merged doc comment argues TCFRAG "owns the whole of the b=1 decode path" and cites the .cu header for it. The header says that about qtip2b_gemv_tuned_kernel — the shipped kernel TCFRAG was written to replace — at qtip2b_tcfrag.cu:19, not about TCFRAG. The conclusion is still right, and the two dispatch sites in bitshift.rs establish it without the quote, so this changes nothing about #209 being the correct fix. But the quoted sentence does not say what it was cited as saying, and it is now on master saying it. Recorded rather than left for someone to re-derive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: TCFRAG's 63 GB closes the blocker — #209 is the fix, not #213 The box measured the missing ~60 GB after this file was drafted: ARC_QTIP_TCFRAG=0 frees 64,262 MiB against 262 MiB with it on. tcfrag_words is a per-weight OnceLock caching a repacked copy of every weight for the process lifetime — ~63 GB against a ~79.5 GB model. #182's leg predates TCFRAG, so it had ~15.7 GB of headroom and survived 2,600 tokens at the same per-token rate. Same leak, two orders of magnitude less room. The concurrency-budget hypothesis is WITHDRAWN. It was a fair reading of the evidence available; the measurement removed the need for it. Attribution corrected: two independent single-variable rescues each work alone (ARC_QTIP_TCFRAG=0 -> 256/256 tokens x5; ARC_CANDLE_ALLOC_CACHE=0 -> 1,000 tokens), so #209 restored the headroom and #213 is defence in depth on the rate. Also splits an over-claim of my own. "Every single-user number was measured inside a crashing run" is two claims and only one is universal: A: every V4 single-user tok/s figure is a <=30-token short window. Stands. B: "inside a crashing run" holds only at or after cc5487a (#203, today). The b=1 trio is from 08-17, before TCFRAG existed - its 24 was a harness choice, not a truncation. Claiming otherwise would be a fabrication. And the 33.4 -> 34.2 pair is a third case: its source CAPTURE_LANE.md exists nowhere in this repo, on any branch. Unsourced, like the 27.2% and the 16.0%. Drops the unsourced "declined 4 of 4 loads" claim, records that #210 merged with a projection multiplied out of the unsourced 16.0% share, and notes #211 merged (registers the hazard; the pin is still unwritten) - and that #209 un-masks it, since 256-token runs now live long enough for the probe to mean something. cuda_ops.rs: fix the mis-quote #209 shipped. The header phrase "the whole of the b=1 decode path" describes qtip2b_gemv_tuned_kernel, not TCFRAG. The conclusion rests on the dispatch sites instead. Adds the 63 GB retention to the gate's doc comment, where the next reader of #203 will look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: re-verify every citation against fdf0b6c — #205 moved fifteen of them Master moved twice more while this was being written (#210 -> f709872, #205 -> fdf0b6c). #205 added +292 lines to normal.rs, +275 to layers.rs and +144 to deepseek4.rs, moving fifteen citations in this file including every mark_unreachable site and the return Ok(None) that section 4 is built on. doc_citations stayed GREEN through both. That is not a gate failure, it is the gate's contract: it checks a line is in RANGE, not that it is the RIGHT line, and a citation with no backticked symbol gets no symbol check at all. Recorded in the provenance block, because it is the most useful thing on the page for anyone maintaining these docs. Everything re-derived by grep, twice. Corrections: normal.rs 1432->1465, 2379->2665, 2370->2656, 2575->2861, 2604->2890, 2619->2905, 2977->3263, and the six mark_unreachable sites 1556/1635/1921/2404/2474/2602 -> 1589/1668/1954/2690/2760/2888; deepseek4.rs 2968->3030, 3149->3190, 3165->3206, 3283->3324, 3334->3375, 2219->2281; cuda_ops.rs 1601->1614, 1612->1627; graph.rs 214->212; blockwise_fp8/ops.rs 1139->1179, 1140->1180; and three self-citations into 00_RESUME_HERE.md that my own edits had shifted. Also notes that master's own code carries the same rot: the site marker inside mark_unreachable("cuda_graph.autonomous_decode.capture") still reads "normal.rs:2619". It is a string literal, so the gate cannot see it by design. Title no longer says "taken inside a crashing run" — that was the half of the claim the TCFRAG measurement showed is not universal. Verified capture is still default-off after #205 (ARC_V4_CAPTURE_PROBE, unset, normal.rs:1940), so section 7's sequencing statement stands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: reorder section 1 so the resolution precedes the remaining work The "Owner: PR #213" block still sat above the TCFRAG resolution and read as if #213 owned the blocker. Moved below it and retitled "the rate half", with the part of #213's diagnosis that survives stated plainly: the 1 GiB constant really was unreachable at 844 MiB free, and that stays true at any headroom small enough to matter. It is worth landing; it is not why serving was broken. Adds the sequencing point that follows: #213's falsification criterion wants a 256-token run, which only exists because #209 landed. Testing it against a 22-token crash would have measured nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: fix the FACTS header claim, the master SHA, and a date I got wrong Three corrections to my own text: - The FACTS session-9 header still asserted "every single-user V4 tok/s figure was measured inside a crashing run", which the Claim A/B split immediately below it contradicts. Header now states the universal half (<=30-token fragment) and points at section 2 for what it does and does not void. - Provenance SHA f709872 -> fdf0b6c (#205 landed after the rebase). - TCFRAG (#203, cc5487a) landed 2026-08-21, not 08-20. git log -1 --date=short is the check; the date is load-bearing because it is what establishes that #182's leg predates TCFRAG. Also 11 -> ~11.8 MiB/token where the retained figure is contrasted with wave65-CQ's ~22 MB/token moved, so the two are not confusable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

heydryft and others added 2 commits August 21, 2026 14:04

heydryft force-pushed the docs/alloc-cache-eviction-vs-baked-addresses branch from b65d3e8 to e89efd1 Compare August 21, 2026 13:12

heydryft mentioned this pull request Aug 21, 2026

perf(ArcInfer/ArcKV): bound the alloc cache against real headroom — a 1 GiB cap cannot bind a card with 262 MiB free #213

Merged

heydryft mentioned this pull request Aug 21, 2026

docs(ArcQuant/ArcInfer): V4 could not serve past ~22 tokens — TCFRAG held 63 GB (wave66-CS) #214

Merged

heydryft merged commit b91e9b6 into master Aug 21, 2026
15 checks passed

Navigation Menu

Repository navigation

docs(ArcGraph): LRU eviction cannot see which addresses a live graph baked (wave65-CR) — issues are disabled, so this is the issue - #211

docs(ArcGraph): LRU eviction cannot see which addresses a live graph baked (wave65-CR) — issues are disabled, so this is the issue#211
heydryft merged 2 commits into
masterfrom
docs/alloc-cache-eviction-vs-baked-addresses

heydryft commented Aug 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 21, 2026 •

edited

Loading

Uh oh!

heydryft commented Aug 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

heydryft commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Hazard 1 — LRU eviction cannot see which addresses a live graph baked

Hazard 2 — ARC_NO_DEFERRED_FREE silently disables the capture instruments

Interaction with #182 and the KV cat growth (new section)

CI

Sequencing

Uh oh!

github-actions Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

heydryft commented Aug 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

heydryft commented Aug 21, 2026 •

edited

Loading

Hazard 2 — `ARC_NO_DEFERRED_FREE` silently disables the capture instruments

Interaction with #182 and the KV `cat` growth (new section)

github-actions Bot commented Aug 21, 2026 •

edited

Loading