docs(ArcQuant/ArcInfer): V4 could not serve past ~22 tokens — TCFRAG held 63 GB (wave66-CS)

[heydryft]

The durable record of session 9. Issues are disabled on this repo, so this is the issue.

Rebased onto fdf0b6c19. Every citation re-verified there — see "the citations rotted twice" below, which is the most reusable thing in this PR.

What lands

• memory/mission/wave66-CS-session9-the-22-token-wall.md (new) — seven findings, each with its ref and how it was established.
• FACTS.md — a 2026-08-21 block with the measured numbers and seven retraction rows, plus an in-situ marker on the b=1 table.
• 00_RESUME_HERE.md — the blocker prepended above everything.
• mistralrs-quant/src/qtip/cuda_ops.rs — comment-only. Fixes a mis-quote fix(ArcQuant): make ARC_QTIP_TCFRAG opt-in — an unverified kernel is default-on for b=1 decode #209 shipped, and records the 63 GB where the next reader of perf(ArcQuant/ArcKernels): TCFRAG-2B — put the qtip2b trellis GEMV on the tensor cores #203 will look. The only non-memory/ edit.

🔴 The blocker, and it is CLOSED — but not by the PR everyone expected

V4 died with CUDA_ERROR_OUT_OF_MEMORY after 22–30 generated tokens, 3/3 runs, card full at 143,151 of 143,771 MiB. It looked like an allocator leak. The leak was never what changed — the headroom was:

probe	frees
ARC_QTIP_TCFRAG=0	64,262 MiB
TCFRAG on (the shipped default at cc5487ad3)	262 MiB

tcfrag_words is a per-weight OnceLock caching a repacked copy of every weight for the process lifetime — ~63 GB against a ~79.5 GB model. That left decode 844 MiB free, so a ~12 MiB/token leak that PR #182's leg survived for 2,600 tokens killed this one in ~20. #182's leg predates TCFRAG entirely.

Two independent single-variable rescues, each sufficient alone — which is what makes this an attribution rather than a story: ARC_QTIP_TCFRAG=0 → 256/256 tokens, five runs, finish=length (cache still on); ARC_CANDLE_ALLOC_CACHE=0 → 1,000 tokens (TCFRAG still on).

🔑 PR #209 (merged) is the fix that restored serving. PR #213 is defence in depth on the leak rate and must not be recorded as "the OOM fix".

#203 costed instructions and never costed memory. The .cu's performance section still only costs instructions.

The tok/s consequence — two claims, and only one is universal

The brief said "every single-user number was measured inside a crashing run". The TCFRAG result forces that apart, and I did not keep the half that isn't supported:

• A (universal, stands): every V4 single-user tok/s figure is a ≤30-token short window. The b=1 trio (15.11 / 15.39 / 15.51) has "24 tokens" in its own FACTS row. Warm-up dominated; none is a sustained rate. This is the claim that matters for the mission.
• B (NOT universal): "inside a crashing run" holds only at or after cc5487ad3 (perf(ArcQuant/ArcKernels): TCFRAG-2B — put the qtip2b trellis GEMV on the tensor cores #203, today). The b=1 trio is from 2026-08-17, before TCFRAG existed — its 24 was a harness choice, not a truncation. Claiming otherwise would be a fabrication.
• The capture lane's 33.4 → 34.2 is a third case: its source CAPTURE_LANE.md does not exist in this repo, on any branch. Unsourced, like the 27.2% TAIL and perf(ArcQuant/ArcKernels): fp8_gemv_wide — the b=1 decode GEMV is instruction-bound, not latency-bound #210's 16.0%.

✅ A real steady-state single-user number is now obtainable for the first time — 256/256 runs clean. Nobody has taken it.

🔴 The citations in this file rotted twice while it was being written, and the gate caught neither

• perf(ArcQuant/ArcKernels): fp8_gemv_wide — the b=1 decode GEMV is instruction-bound, not latency-bound #210 moved the two GEMV dispatch citations in blockwise_fp8/ops.rs ~40 lines. The kernel's own lines survived only because perf(ArcQuant/ArcKernels): fp8_gemv_wide — the b=1 decode GEMV is instruction-bound, not latency-bound #210 appended rather than inserted.
• ArcGraph: land the CUDA-graph capture lane — 3,961 -> 3.8 launch APIs/token, 10,114 -> 131 allocs/step #205 added +292 lines to normal.rs, +275 to layers.rs, +144 to deepseek4.rs — moving fifteen citations, including every mark_unreachable site and the return Ok(None) §4 is built on.

doc_citations stayed green through both, and that is its contract, not a bug: it checks a line is in range, not that it is the right line, and a citation with no backticked symbol gets no symbol check at all. A green gate is not a verified citation. Everything was re-derived by grep against fdf0b6c19.

Master carries the same rot in code: the site marker inside mark_unreachable("cuda_graph.autonomous_decode.capture", …) still reads "normal.rs:2619", which #205 moved to 2905. It is a string literal, so the gate cannot see it by design.

The other six

§	finding	state
2	TCFRAG was default-ON while its own header reads "UNVERIFIED ON HARDWARE — NEVER RUN"	✅ #209, merged
3	Two host round-trips per decode step, not one. Retracts "op count is retired as a lever" — the captured arm kept both, so the CPU never left the loop	open; zero-sync arm unmeasured
4	The dedicated decode tier cannot accept V4 architecturally — seven non-optional projection slots, no MLA slot, no expert slot	open, design change
5	The b=1 FP8 GEMV is instruction-bound. Retracts the "4% memory-controller ⇒ latency-bound" story	#210 merged
6	Four instrument failures, incl. ci_cuda.yaml gating nothing and 21 presence-tested env flags where =0 turns a feature ON	#212 open
7	#182 is a precondition, not a fix — and its hazard is now UN-masked, because 256-token runs finally live long enough for the probe to mean something	#211 merged (registers it; the pin is unwritten)

Corrections to the brief, kept as corrections

normal.rs:2515 → :2619 → now :2905. tcfrag2b.rs:151-157 → :151-158. has_cached_kv_info is 3 prose occurrences, 0 definitions, and the misleading comment is already fixed in-tree. ba026e9d1 is a root/import commit (1,430 files, 0 deletions) and introduced nothing — the analysis is 6e58c5dbf. The OnceLock caches None permanently rather than panicking — same blast radius, different mechanism. "Declined 4 of 4" is removed as unsourced, and the 63 GB measurement points the other way: a repack that declined every time would retain nothing. The 2.97 TB/s ceiling used the FP32 warp rate; the INT32 pipe gives 1.49, so the honest form is the range 1.5–3.0 TB/s. "45 inst / 4 B" is derived from #210's own prose, not measured — probe is cuobjdump -sass. ARC_NO_DEFERRED_FREE was overstated. mark_unreachable being dark was already on the record from session 8; only the consequence is new.

One correction goes the other way: the 08-19 row narrowing "the default config silently disables prefix caching" is confirmed, not contradicted.

Gates

doc_citations 12/12, run standalone (std-only) against fdf0b6c19; zero drift from the new content. The green is not vacuous — the gate failed on an earlier revision of this file first, on a device.rs citation inside the very sentence explaining why candle must not be cited that way. typos clean (two SHAs reworded rather than widening .typos.toml, per #211's precedent). cargo check -p mistralrs-quant green; cargo fmt -p mistralrs-quant --check shows no diff in cuda_ops.rs.

Related: #209, #213, #211, #210, #212, #205, #206, #182, #203.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     81        29464        20161         6264         3039
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     75         4896         4893            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  147        15371        12677          824         1870
 Shell                    42        10274         6849         2750          675
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1498         1294           54          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                212        47877            0        37224        10653
 |- BASH                  72         1655         1203          331          121
 |- C                      3           17           17            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  19          779          779            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  67         2058         1723           77          258
 |- TOML                   6          207          164            0           43
 |- YAML                   5           41           36            5            0
 (Total)                            53727         4766        37766        11195
─────────────────────────────────────────────────────────────────────────────────
 Rust                    680       340521       292882        18037        29602
 |- Markdown             503        32080          471        27656         3953
 (Total)                           372601       293353        45693        33555
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1351       514719       362803        97723        54193
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━