Repository navigation
ArcGraph: land the CUDA-graph capture lane — 3,961 -> 3.8 launch APIs/token, 10,114 -> 131 allocs/step - #205
Merged
Merged
Conversation
Code Metrics Report━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Language Files Lines Code Comments Blanks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ C Header 5 305 210 52 43 CSS 2 1181 1036 34 111 CUDA 81 29145 19998 6130 3017 Dockerfile 1 39 22 8 9 JavaScript 16 3546 2676 482 388 Jinja2 7 694 656 5 33 JSON 75 4896 4893 0 3 Makefile 1 6 5 0 1 Metal Shading Lan| 33 12224 9431 1142 1651 PowerShell 1 300 227 30 43 Python 147 15371 12677 824 1870 Shell 42 10274 6849 2750 675 Plain Text 4 3801 0 2479 1322 TOML 33 1498 1294 54 150 YAML 3 25 23 2 0 ───────────────────────────────────────────────────────────────────────────────── HTML 4 2687 2604 43 40 |- CSS 2 543 479 37 27 |- JavaScript 1 1233 1215 12 6 (Total) 4463 4298 92 73 ───────────────────────────────────────────────────────────────────────────────── Jupyter Notebooks 4 122 83 23 16 |- Markdown 1 60 30 22 8 |- Python 1 122 113 1 8 (Total) 304 226 46 32 ───────────────────────────────────────────────────────────────────────────────── Markdown 211 47578 0 36992 10586 |- BASH 72 1655 1203 331 121 |- C 3 17 17 0 0 |- CUDA 2 84 56 16 12 |- JSON 19 779 779 0 0 |- PowerShell 1 1 1 0 0 |- Python 23 1008 787 113 108 |- Rust 67 2058 1723 77 258 |- TOML 6 207 164 0 43 |- YAML 5 41 36 5 0 (Total) 53428 4766 37534 11128 ───────────────────────────────────────────────────────────────────────────────── Rust 679 339224 291995 17706 29523 |- Markdown 502 31672 471 27302 3899 (Total) 370896 292466 45008 33422 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Total 1349 512396 361753 96672 53971 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
heydryft
force-pushed
the
arcgraph/capture-lane-to-master
branch
from
August 21, 2026 12:15
dcdffc2 to
478a514
Compare
This was referenced Aug 21, 2026
…OST pointer
V4 decode capture reached "forward RECORDED" and then SIGSEGV'd on its very
first cuGraphLaunch, with a device-side assert from candle's index_select:
ids[id_i] < src_dim_size (T = __nv_bfloat16, I = unsigned int)
That dtype pair is the bf16 cos table indexed by U32 positions, i.e.
`self.cos.index_select(&positions, 0)` in
`DeepSeekV2RotaryEmbedding::forward_at_positions` — NOT the TD-MoE
`tid2eid.index_select` (tid2eid is i64, so it cannot produce T=bf16), and not
the embedding (its ids come from the already-address-stable graph-mode buffer).
`positions` was built one line earlier by `compressed_kv_from_rows` with
`Tensor::arange(0u32, t_c, dev)`. `arange` materialises a transient host `Vec`
and uploads it via `CudaDevice::clone_htod` -> an ASYNC `cuMemcpyHtoDAsync`.
Under `cuStreamBeginCapture` that copy is not executed, it is RECORDED: the
graph stores the HOST POINTER and re-reads it on the first launch and on every
replay. The `Vec` is freed when the expression returns, so the graph copied
freed host memory into `positions` and handed garbage indices to index_select.
This is not an address-staleness bug — it fires on the first launch, before any
replay — which is why the static input_ids buffer did not prevent it. candle
already documents this exact mechanism and works around it in
`CudaDevice::htod_info` (it leaks the host source while capturing); nothing
protects `clone_htod`, which is the path `arange` takes.
Fix: serve the strided compressed positions from a per-(ratio, device) table
built once, outside capture, and take a zero-copy `narrow` view each step. The
values are a pure function of (ratio, capacity) and never change. This also
removes a host round trip from every compressor layer of every decode step.
Keyed per ratio because V4 interleaves CSA and HCA compression ratios; a
single-slot cache would thrash and reintroduce the copy on every layer.
A sweep of every host->device constructor reachable from the B=1 decode forward
found three more `Tensor::arange` calls with the same defect as the compressor
positions, all in `dsv4_attention`, all unconditional on every layer of every
step:
:610 kp = arange(raw_base, raw_base + t_k) absolute key positions
:613 qp = arange(q0, q0 + t_q) query positions (ONE element
at decode -- a whole clone_htod
for 4 bytes, 43x per token)
:628 bp = arange(0, t_c) compressed-block indices
Each is a contiguous integer ramp cast to F32, so all three are views of a
single cached device ramp: `layers::positions_f32(start, len, dev)` narrows
`[0.0, 1.0, 2.0, …]`. Capacity grows by powers of two from 8192, so a typical
run allocates the ramp once during warmup and never again -- rebuilding inside
capture is precisely the hazard being removed.
Fixing only the compressor site would not have been enough: any one of these
would have re-armed the same dangling-host-pointer crash on the first launch.
Also removes three H2D copies per layer per token from the decode hot path
(129 per token at 43 layers), independent of graph capture.
`append_graph` read a fixed `0..read_capacity` window but wrote at the ABSOLUTE position, so from token `read_capacity` on, the new row landed outside the window that is read: the freshest key was invisible to attention and a stale row was attended in its place. The arm was correct only below `sliding_window`. `read_capacity` IS `sliding_window`, so a ring of that many slots holds exactly the key set the raw branch is defined to attend. Permuting the keys is invariant here: softmax over keys is permutation-invariant, K and V go to the same slot so pairing is preserved, and V4 rotates K before caching so each row carries its own absolute RoPE. The validity mask needs no change — `slot <= position` is already ring-correct in both regimes (filling: written iff slot <= position; full: all slots valid, predicate universally true). The slot is derived on-device (`layers::graph_ring_slot`); only the constant `capacity` is baked into kernel arguments. A host-resolved `position % capacity` would repeat blocker 1's defect one buffer over.
…s hitting an unwarmed alloc MEASURED root cause of the first-launch CUDA_ERROR_ILLEGAL_ADDRESS (700): exactly ONE capture-time allocator miss, 327680 bytes. At hidden 4096 in BF16 that is 8192 B/token, so the miss is a 40-token buffer — the retained raw tail — one token wider than the 319488 B (39-token) miss on the preceding warmup step. A capture-time miss is an unstable graph memory node, i.e. an INVALID graph, which is why it surfaces as an illegal address on the first cuGraphLaunch rather than as a slow one. The width moves because `advance_uniform` retains only what a future compressed row can consume, which is a function of `tokens % ratio`. That gives `ratio` distinct widths on a period of `ratio`: 4 for CSA (the "four unwarmed allocation sizes" an earlier session recorded) but 128 for HCA, growing one token per step — which is why no practical warmup count ever covered it. `pin_tail_width` holds the tail at `span_groups * ratio + margin`, which is an upper bound on what the unpinned policy would keep (width is maximised at `r = ratio - 1` as `span_groups*ratio + margin - 1`), so it only ever retains more than the compressor needs, never less. Takes effect once the sequence is that long; capture must land after that point — 144 tokens for V4's HCA layers.
…o capacities `1.0 / capacity` is exact in binary only when capacity is a power of two. V4's sliding_window is 128 so the original form happened to be right, but at any other capacity `p * (1/capacity)` can round just under an integer when p is an exact multiple, the floor loses a block, and the remainder comes out as `capacity` — one slot past the read window, i.e. an out-of-bounds KV write. The error is well under one ulp of the quotient so the remainder is in [0, capacity]; one correction step is exact for every input.
Bench mode, not server mode (the alloc cache that capture requires leaks in server mode). Pins git SHA + binary sha256 into every log, refuses to start until the GPU is verifiably empty, and scopes its process reaping to this checkout — an unscoped `pkill -x mistralrs` from a concurrent agent's harness killed two of these legs (SIGTERM, BENCH_EXIT=143), so the binary also runs under a distinct process name.
…_cache With the pre-warm converged, a V4 capture records with ZERO allocator misses and then aborts the process inside this function with `malloc_consolidate(): invalid chunk size`. That retracts the standing assumption that the glibc corruption is downstream of a poisoned CUDA context — there is no capture-time miss left to poison it, and it still aborts. A glibc abort leaves no CUDA error to read, and these five driver calls had no logging between them, so the log could only say "somewhere in EndCapture .. Instantiate .. Destroy .. Launch .. Sync". One marker per call turns that into a located blocker. Also un-assumes AUTO_FREE_ON_LAUNCH. The comment claimed it was "harmless if the graph has no alloc nodes", and no-alloc-nodes is now the NORMAL case because the capture-miss gate refuses to reach this function otherwise. ARC_GRAPH_AUTO_FREE=0 drops the flag so it can be ablated instead of trusted.
… the context is poisoned before the graph runs The first cuGraphLaunch returns 700 SYNCHRONOUSLY on a capture that recorded with zero allocator misses. A launch call returning illegal-address synchronously is characteristic of a sticky context error raised by earlier work — CUDA reports a real graph fault asynchronously, at the following sync. Two changes make that testable instead of arguable: - The deferred warm pass did `let _ = self.device().synchronize()`. That was the ONLY place an asynchronous fault in a warm-pass forward could ever be reported: the forward returns Ok because kernels are launched, not awaited, and the sync is what collects the fault. Discarding it leaves a sticky context error whose first reporter is the graph launch — so the graph gets blamed for a fault it inherited. Now it is logged and it disables capture. - A sync between cuStreamEndCapture and instantiate. Nothing executes between begin_capture and there, so non-zero puts the fault in warmup and zero puts it in the graph. Those two were previously indistinguishable.
…t replays
The replay-vs-eager gate failed at max|Δ|=6.348e0 and there was no way to
tell a mis-wired graph from a correctly-wired one reading stale per-step
state. Three env-gated probes settle it, and one counter stops the lane
from ever having to infer whether a graph served.
ARC_GRAPH_SAMESTEP=1
* compares the FIRST cuGraphLaunch against an eager forward on the
CAPTURE step itself, where the position buffer, the token ids and the
KV ring are exactly what capture recorded against. Measured on H200 /
V4: max|Δ| = 0.000e0, and 0.000e0 again on a second launch. The graph
is wired correctly; every later divergence is stale per-step state.
* at a verification step, launches the graph AGAIN after the eager
forward has run. Measured: 6.348e0 before vs 6.316e0 after — so the
fault is not host work missing from the recording, it is a genuinely
stale input.
* 🔴 the verdict is now taken from `out` BEFORE that extra launch:
`replay` hands back a CLONE of the captured output tensor, so a second
launch overwrites the storage the gate was about to grade.
ARC_GRAPH_VERIFY_ALWAYS=1 keeps comparing after the first failure, so Δ
can be read against distance from the capture step. Measured: the replay
is no closer to the capture step's logits than the truth is, i.e. it
advances — it is not frozen — which excludes a wholly unread static input.
ARCGRAPH REPLAY #n / ARCGRAPH FINAL (D18): the replay count was printed
only at init and at capture, so "replays served" had to be inferred, and
this lane has been bitten four times by a graph that records and serves
nothing. It is now logged on the first ten replays, every hundredth, and
once more to stderr on drop — stderr because a tracing subscriber may be
gone by then and a tally that vanishes at exit is the silent success the
line exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The +1-step divergence was known to be a stale per-step input in the raw path, but not WHERE. This bisects it. Method: a captured forward's per-layer outputs are ordinary warm-pool buffers, and the recorded kernels write to THOSE addresses on every replay. Holding a clone of each therefore gives a live window onto the graph's own intermediates — after a replay they hold the replay's values — and holding them also keeps them out of the allocator's free list, so the eager forward that follows cannot land on top of them. Handles only: tracing is an Arc bump per layer, no device work, safe inside capture. Measured on H200 / V4 @ ARC_GRAPH_WARMUP=170, first verification step: slot 0 graph length mask 0.00e0 slot 1 embedding 0.00e0 slot 2 mHC lift 0.00e0 slot 3 LAYER 0 output 1.95e-1 <<FIRST slot 4 layer 1 1.72e-1 So every model-level input to layer 0 is BIT-IDENTICAL under replay: the device length mask is rebuilt correctly from the position buffer, and the token id reaches the graph. The stale state is INSIDE layer 0 — its attention/KV path — not in anything `forward` hands it. That also rules the model-level static input set (positions, ids, mask) fully in the clear, which four previous theories had suspected. Also: `replay()` returns a CLONE of the captured output tensor, i.e. a handle to storage the next launch overwrites, not a snapshot. Named at the return site — it nearly voided the probe that found it, and a verification path that silently grades the wrong bytes is the worst version of this lane's recurring fault. Also: the clock sampler slept 150 s against an ~80 s run, so it always wrote an EMPTY .clocks file while looking green. Now samples from t+60 s every 2 s; measured 1980 MHz at 92-100% GPU / 261-321 W during the request. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Splits layer 0's interior into the three tensors that separate the two
candidate causes. Measured on H200 / V4, first verification step
(slots = 3 + 43*4; per graph-arm layer: k, k_full, attn, layer_out):
0 mask 0.00e0 3 L0.k 4.05e0 <<FIRST
1 embed 0.00e0 4 L0.k_full 0.00e0 (SEE CAVEAT)
2 lift 0.00e0 5 L0.attn 3.73e0
6 L0.out 1.95e-1
7 L1.k 2.84e0
So the NEW KEY for the current token is already wrong before it is
written. The fault is upstream of the KV cache, in the K computation --
not the ring slot, not the window geometry, not the mask.
The magnitudes corroborate: k is off by 4.05 but the layer output only by
1.95e-1, which is what one wrong key out of a 128-wide window does to a
softmax. It also explains the growth with distance from capture: under a
pure replay run each step writes another wrong key into the ring, so the
window degrades cumulatively.
🔴 CAVEAT on slot 4, and it is the same class of trap as `replay()`:
`append_graph` returns a NARROW OF THE RING BUFFER, not a copy, so the
graph's `k_full` handle and the eager one VIEW THE SAME STORAGE. Its
0.00e0 is an aliasing artifact and is NOT evidence the window is read
correctly. Only slot 3 is load-bearing, because a projection output is a
fresh allocation per forward.
RoPE is already cleared by reading, not guessing: `cos_sin_for`
(layers.rs:1650) resolves seq_len==1 graph-mode positions with
`index_select` off the device buffer, so it is replay-safe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…per #206 orphaned Rebased onto 56c9a99 (post-#206, post-#208). Four defects this lane introduced against master's gates, plus one it retires for free, plus the consequence of #206 landing first. 1. `arc_layer_bisect` / `max_abs_diff` (pipeline/normal.rs) are called only from inside the `#[cfg(feature = "cuda")]` capture block, so on a CPU/Metal build they are genuinely dead. Gated with the same cfg rather than `#[allow(dead_code)]`: an allow would blind the dead-symbol detector on a CUDA build too, where these MUST stay reachable — the opposite of what is wanted. 2. Two `///` doc comments sat on `thread_local!` macro invocations (`ARC_LAYER_TRACE`, `GRAPH_MODE_SLOT_IDS`), which rustc reports as `unused doc comment` — the text attaches to a macro expansion and reaches no rustdoc page. Demoted to `//`, matching the convention the neighbouring `COMPRESS_POSITIONS` block already uses. The `GRAPH_MODE_SLOT_IDS` one is PRE-EXISTING on master, so the net warning count lands one BELOW master. 3. The Typos lane rejected two test names. `.typos.toml` allow-lists `arange` under `[type.rust.extend-identifiers]`, which matches a WHOLE identifier — so `Tensor::arange` in a body and `arange` in a doc comment both pass, but `matches_the_arange_expression_it_replaced` does not: typos falls through to word-splitting on the snake_case name. Renamed rather than widening the allow-list, which is the precedent set in 745c871 and e4eb59d. 4. `layers::positions_f32` is REMOVED, with its `IOTA_F32` cache. #206 (02edd31) deleted two of its three call sites outright — at `t_q == 1` the raw-window mask is provably all-ones, so the kp/qp chain that fed it is gone — and this rebase drops the third swap in favour of master's spelling. That left a `pub fn` with ZERO callers workspace-wide, and two tests exercising nothing but itself, which is a vacuous test by the repo's own definition. Its two tests are replaced by one that pins a claim still LIVE: `compress_positions(t_c, 1, dev)` is bit-identical to `Tensor::arange(0u32, t_c as u32, dev)`, which is what `dsv4_attention`'s compressed-branch mask now relies on to serve block indices from the cache instead of a per-layer host->device upload. Net test count is -1, and coverage moves from a dead helper onto a live one. Also re-indents the same-step-probe block in `normal.rs`: it was written against a nesting level introduced by the alloc-prewarm commit, which is not in this PR (it needs candle APIs master's pin does not carry). rustfmt on the touched region only — mistralrs-core is deliberately outside the CI fmt lane per the fork's no-mass-reformat policy, and the file's other rustfmt findings are master's and are left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7
…replaces the one this PR removes #207 landed `wave65-CQ-v4-compressed-rope-recompute.md` while this branch was rebasing. It cites `compressed_row_positions` (`deepseek4.rs:3013-3037`) twice, as "the established in-tree pattern to copy" for memoizing a `(t_c, ratio, device)`-keyed tensor. This PR DELETES that function, so both citations became SYMBOL-ROT and `doc_citations` failed — correctly: memory/mission/wave65-CQ-v4-compressed-rope-recompute.md:46 [SYMBOL-ROT] memory/mission/wave65-CQ-v4-compressed-rope-recompute.md:83 [SYMBOL-ROT] Repointed at `layers::compress_positions` (`layers.rs:2391-2459`), which is what replaces it, and — because wave65-CQ is explicitly telling a future reader to COPY this pattern — both sites now also carry the reason the old one was the wrong thing to copy: it was keyed on `(t_c, ratio, device)` in a SINGLE slot, and V4 interleaves CSA (ratio 4) with HCA (ratio 128) layers, so consecutive layers of the same step evict each other and the memo never hits. The replacement is keyed per `(ratio, device)`. Left alone: the `xs_rolling.rs:465` / `:1065` and `GRAPH_BLOCK_IDS` citations in the same document, which this PR does not touch and which still resolve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7
heydryft
force-pushed
the
arcgraph/capture-lane-to-master
branch
from
August 21, 2026 13:08
478a514 to
2aadcee
Compare
heydryft
added a commit
that referenced
this pull request
Aug 21, 2026
…f them Master moved twice more while this was being written (#210 -> f709872, #205 -> fdf0b6c). #205 added +292 lines to normal.rs, +275 to layers.rs and +144 to deepseek4.rs, moving fifteen citations in this file including every mark_unreachable site and the return Ok(None) that section 4 is built on. doc_citations stayed GREEN through both. That is not a gate failure, it is the gate's contract: it checks a line is in RANGE, not that it is the RIGHT line, and a citation with no backticked symbol gets no symbol check at all. Recorded in the provenance block, because it is the most useful thing on the page for anyone maintaining these docs. Everything re-derived by grep, twice. Corrections: normal.rs 1432->1465, 2379->2665, 2370->2656, 2575->2861, 2604->2890, 2619->2905, 2977->3263, and the six mark_unreachable sites 1556/1635/1921/2404/2474/2602 -> 1589/1668/1954/2690/2760/2888; deepseek4.rs 2968->3030, 3149->3190, 3165->3206, 3283->3324, 3334->3375, 2219->2281; cuda_ops.rs 1601->1614, 1612->1627; graph.rs 214->212; blockwise_fp8/ops.rs 1139->1179, 1140->1180; and three self-citations into 00_RESUME_HERE.md that my own edits had shifted. Also notes that master's own code carries the same rot: the site marker inside mark_unreachable("cuda_graph.autonomous_decode.capture") still reads "normal.rs:2619". It is a string literal, so the gate cannot see it by design. Title no longer says "taken inside a crashing run" — that was the half of the claim the TCFRAG measurement showed is not universal. Verified capture is still default-off after #205 (ARC_V4_CAPTURE_PROBE, unset, normal.rs:1940), so section 7's sequencing statement stands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7
heydryft
added a commit
that referenced
this pull request
Aug 21, 2026
Three corrections to my own text: - The FACTS session-9 header still asserted "every single-user V4 tok/s figure was measured inside a crashing run", which the Claim A/B split immediately below it contradicts. Header now states the universal half (<=30-token fragment) and points at section 2 for what it does and does not void. - Provenance SHA f709872 -> fdf0b6c (#205 landed after the rebase). - TCFRAG (#203, cc5487a) landed 2026-08-21, not 08-20. git log -1 --date=short is the check; the date is load-bearing because it is what establishes that #182's leg predates TCFRAG. Also 11 -> ~11.8 MiB/token where the retained figure is contrasted with wave65-CQ's ~22 MB/token moved, so the two are not confusable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7
heydryft
added a commit
that referenced
this pull request
Aug 21, 2026
…held 63 GB (wave66-CS) (#214) * docs(ArcInfer/ArcKV): V4 cannot serve past ~22 tokens, so every single-user number is void (wave66-CS) The durable record of session 9. Issues are disabled on this repo, so this is the issue. THE BLOCKER. At cc5487a on an H200, batch=1 decode dies with CUDA_ERROR_OUT_OF_MEMORY after 22-30 generated tokens, 3/3 runs, two independent arms. Growth is ~11 MiB/token and is never released across requests: one run ended at 142,959 MiB used and the next started there. The card is genuinely full at failure, 143,151 of 143,771 MiB. Consequence, stated as strongly as the evidence supports: every single-user V4 throughput number this project has recorded was measured over <=30 tokens before a crash. The three-way b=1 agreement (15.11 / 15.39 / 15.51 tok/s) has "24 tokens" written in its own FACTS row. The best clean fragment tonight was 37.35 tok/s over 20 tokens - a fragment, not a rate. Six more findings, each with its ref: TCFRAG was default-ON with "UNVERIFIED ON HARDWARE - NEVER RUN" in its own header (fixed, #209); two host round-trips per decode step, not one; the dedicated decode tier cannot accept V4 architecturally; the b=1 FP8 GEMV is instruction-bound; four instrument failures; and #182 is a precondition whose hazard is masked by the OOM. Retractions added to FACTS: the 5,300-cast / 1,571-MHC premise (stale, profiled at 05af600), DecodeKvGeometry (never existed in Rust), the 27.2% TAIL (sole source is a docs file not on master), the latency-bound b=1 story, and "op count is retired as a lever". Corrections made while verifying, all against 9c127a2 rather than against the report: normal.rs:2515 -> :2619; has_cached_kv_info is 3 prose references and 0 definitions, not zero occurrences; ba026e9 is a root/import commit and introduced nothing; the TCFRAG OnceLock does not panic, it caches None permanently; tcfrag2b.rs:151-157 -> :151-158; the 2.97 TB/s issue ceiling used the FP32 warp rate, and the INT32 pipe gives 1.49, so the defensible claim is a 1.5-3.0 TB/s range; ARC_NO_DEFERRED_FREE was overstated; and mark_unreachable being dark was already on the record from session 8. Also surfaced, unresolved: PR #182's H200 leg ran 2,600 tokens with no OOM, which needs >=15.7 GB free at decode start against tonight's 844 MiB. Nobody has explained the ~60 GB gap, and if it is the serve config then this is a concurrency-budget defect as much as a per-token leak. Gates: doc_citations 12/12 green, run standalone against this tree, and the gate reproduced a real failure on an earlier revision of this file first, so the green is not vacuous. typos clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: put wave66-CS in the bounded read order, and point FACTS at §2026-08-21 The blocker block at the top is no use if the read order still sends people to the session-8 FACTS section first. Adds wave66-CS as item 2 (before the census, because a GPU-spend decision made on void single-user numbers is worse than one made late) and renumbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: make the op-count retraction exact — the captured arm kept both round-trips The first wording said the 1,137x launch cut "left two host syncs in both arms", which is wrong: the eager arm never replays a graph, so it never reaches graph.rs:362. The correct and stronger statement is about the CAPTURED arm — it still pays the replay sync and the sampler's 4-byte D2H, so the CPU never left the loop and the ~8% cannot be read as the value of removing it. Also adds the guard against over-claiming in the other direction: this does not show the two syncs ARE the limiter, only that the experiment could not see them. The number that would settle it has not been taken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: two wording fixes — "three corrections" not two, and the =0 polarity cuts both ways The env-flag consequence was stated as "both arms ran with the feature on". That holds for the 13 .is_some() flags; for the 5 .is_none() ones, =0 counts as set and therefore SUPPRESSES, so the legs are inverted rather than identical. Either way the control leg was not a control, but the two failure shapes are different and a reader debugging an old A/B needs to know which one they have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: name where PR #210's 16.0% kernel share actually comes from It was cross-referenced loosely as "the same provenance problem as the 27.2%". Both numbers live on the same line of docs/engineering/OPENROUTER_READY.md, which is not on master — it exists only on agent/tail-sinkhorn-warp-v2 and agent/decode-share-probe. One of the two is already being multiplied into a projected win, so the pointer should be explicit rather than a cross-reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: flag the mis-quote #209 shipped into cuda_ops.rs #209's merged doc comment argues TCFRAG "owns the whole of the b=1 decode path" and cites the .cu header for it. The header says that about qtip2b_gemv_tuned_kernel — the shipped kernel TCFRAG was written to replace — at qtip2b_tcfrag.cu:19, not about TCFRAG. The conclusion is still right, and the two dispatch sites in bitshift.rs establish it without the quote, so this changes nothing about #209 being the correct fix. But the quoted sentence does not say what it was cited as saying, and it is now on master saying it. Recorded rather than left for someone to re-derive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: TCFRAG's 63 GB closes the blocker — #209 is the fix, not #213 The box measured the missing ~60 GB after this file was drafted: ARC_QTIP_TCFRAG=0 frees 64,262 MiB against 262 MiB with it on. tcfrag_words is a per-weight OnceLock caching a repacked copy of every weight for the process lifetime — ~63 GB against a ~79.5 GB model. #182's leg predates TCFRAG, so it had ~15.7 GB of headroom and survived 2,600 tokens at the same per-token rate. Same leak, two orders of magnitude less room. The concurrency-budget hypothesis is WITHDRAWN. It was a fair reading of the evidence available; the measurement removed the need for it. Attribution corrected: two independent single-variable rescues each work alone (ARC_QTIP_TCFRAG=0 -> 256/256 tokens x5; ARC_CANDLE_ALLOC_CACHE=0 -> 1,000 tokens), so #209 restored the headroom and #213 is defence in depth on the rate. Also splits an over-claim of my own. "Every single-user number was measured inside a crashing run" is two claims and only one is universal: A: every V4 single-user tok/s figure is a <=30-token short window. Stands. B: "inside a crashing run" holds only at or after cc5487a (#203, today). The b=1 trio is from 08-17, before TCFRAG existed - its 24 was a harness choice, not a truncation. Claiming otherwise would be a fabrication. And the 33.4 -> 34.2 pair is a third case: its source CAPTURE_LANE.md exists nowhere in this repo, on any branch. Unsourced, like the 27.2% and the 16.0%. Drops the unsourced "declined 4 of 4 loads" claim, records that #210 merged with a projection multiplied out of the unsourced 16.0% share, and notes #211 merged (registers the hazard; the pin is still unwritten) - and that #209 un-masks it, since 256-token runs now live long enough for the probe to mean something. cuda_ops.rs: fix the mis-quote #209 shipped. The header phrase "the whole of the b=1 decode path" describes qtip2b_gemv_tuned_kernel, not TCFRAG. The conclusion rests on the dispatch sites instead. Adds the 63 GB retention to the gate's doc comment, where the next reader of #203 will look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: re-verify every citation against fdf0b6c — #205 moved fifteen of them Master moved twice more while this was being written (#210 -> f709872, #205 -> fdf0b6c). #205 added +292 lines to normal.rs, +275 to layers.rs and +144 to deepseek4.rs, moving fifteen citations in this file including every mark_unreachable site and the return Ok(None) that section 4 is built on. doc_citations stayed GREEN through both. That is not a gate failure, it is the gate's contract: it checks a line is in RANGE, not that it is the RIGHT line, and a citation with no backticked symbol gets no symbol check at all. Recorded in the provenance block, because it is the most useful thing on the page for anyone maintaining these docs. Everything re-derived by grep, twice. Corrections: normal.rs 1432->1465, 2379->2665, 2370->2656, 2575->2861, 2604->2890, 2619->2905, 2977->3263, and the six mark_unreachable sites 1556/1635/1921/2404/2474/2602 -> 1589/1668/1954/2690/2760/2888; deepseek4.rs 2968->3030, 3149->3190, 3165->3206, 3283->3324, 3334->3375, 2219->2281; cuda_ops.rs 1601->1614, 1612->1627; graph.rs 214->212; blockwise_fp8/ops.rs 1139->1179, 1140->1180; and three self-citations into 00_RESUME_HERE.md that my own edits had shifted. Also notes that master's own code carries the same rot: the site marker inside mark_unreachable("cuda_graph.autonomous_decode.capture") still reads "normal.rs:2619". It is a string literal, so the gate cannot see it by design. Title no longer says "taken inside a crashing run" — that was the half of the claim the TCFRAG measurement showed is not universal. Verified capture is still default-off after #205 (ARC_V4_CAPTURE_PROBE, unset, normal.rs:1940), so section 7's sequencing statement stands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: reorder section 1 so the resolution precedes the remaining work The "Owner: PR #213" block still sat above the TCFRAG resolution and read as if #213 owned the blocker. Moved below it and retitled "the rate half", with the part of #213's diagnosis that survives stated plainly: the 1 GiB constant really was unreachable at 844 MiB free, and that stays true at any headroom small enough to matter. It is worth landing; it is not why serving was broken. Adds the sequencing point that follows: #213's falsification criterion wants a 256-token run, which only exists because #209 landed. Testing it against a 22-token crash would have measured nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs: fix the FACTS header claim, the master SHA, and a date I got wrong Three corrections to my own text: - The FACTS session-9 header still asserted "every single-user V4 tok/s figure was measured inside a crashing run", which the Claim A/B split immediately below it contradicts. Header now states the universal half (<=30-token fragment) and points at section 2 for what it does and does not void. - Provenance SHA f709872 -> fdf0b6c (#205 landed after the rebase). - TCFRAG (#203, cc5487a) landed 2026-08-21, not 08-20. git log -1 --date=short is the check; the date is load-bearing because it is what establishes that #182's leg predates TCFRAG. Also 11 -> ~11.8 MiB/token where the retained figure is contrasted with wave65-CQ's ~22 MB/token moved, so the two are not confusable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 21, 2026
Draft
heydryft
added a commit
that referenced
this pull request
Aug 21, 2026
…to 0, opt-in (#217) * perf(ArcInfer/ArcGraph): device decode loop — N steps, N cuGraphLaunch, zero host syncs CUDA-graph capture already cut launch APIs per token 3,961 -> 3.8 and bought only ~8%. The reason is not that op count stopped mattering: replay() launches the graph and then blocks on cudaStreamSynchronize (graph.rs:362), the caller samples on the host, and the host-side argmax D2H (sampler.rs:1479) re-synchronizes anyway. Capture removed the cost of *issuing* the work and left the serialized host round trip between steps completely intact. Removing the sync alone buys nothing, so this removes the sync and the host sample together, behind an opt-in. Per step, all on one stream: cuGraphLaunch -> CudaSampler (on device, token to an i32 device buffer, never copied down) -> arc_graph_step_commit (scatters the token into the PINNED U32 buffer the captured graph reads, advances the device position, publishes into a pinned+mapped host ring). Nothing blocks. Fault detection replaces the sync rather than dropping it, with three signals that need no blocking call: non-blocking cudaStreamQuery (which NVIDIA documents as also returning errors from previous asynchronous launches), an idle-but-short ring (every step retired yet fewer tokens landed — a fault with no timeout heuristic in it), and a sticky device fault word the commit kernel latches before it writes anything. A spin backstop covers a kernel that hangs without ever erroring. The driver is expressed against a DeviceOps trait that deliberately has no synchronize method, so it is structurally incapable of blocking the stream, and it compiles without CUDA — 20 unit tests run against a scripted fake GPU on any host. Two mutations were checked: dropping the device-fault read and skipping one launch per burst each fail exactly the tests that name them. Aliasing: the logits tensor aliases storage the next launch overwrites. Safe here only because every access is stream-ordered and none is a host read; the ring the host drains is a separate pinned allocation no launch touches. The invariant that lets the aliasing tensor be returned at all is enforced at runtime, not just documented. Default OFF (ARC_GRAPH_DEVICE_LOOP). Greedy-only: the device sampler runs Splitmix64 against the host's Isaac64, so any stochastic mode would draw a different, equally valid token and silently break seeded reproducibility. Every refusal falls back to the existing path; one failure latches it off for the process. Kept as a pure insertion, away from replay(), to merge cleanly with PR #205. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * fix(ArcGraph): device loop stands aside for ARC_GRAPH_VERIFY_ALWAYS The device-loop branch sits earlier in the chain than the runner.replay(bs) arm that PR #205 extended with ARC_GRAPH_VERIFY_ALWAYS. Without this it would take every step and that switch would silently verify nothing — a clean diagnostic that never ran. Also cross-references #205's BOOBY TRAP note from replay_handles, since the same aliasing applies to the handles it returns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * docs(ArcGraph): record the burst's scheduler-admission limitation A burst admits no new request for its whole length (default 4 decode steps). Harmless at batch 1 — the only shape admit() accepts — and not harmless above it, which is why admit() refuses batch_size != 1 outright rather than treating it as a tuning question. Stated in the module docs so it survives past the return it was first noted in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 * fix(ArcGraph): the device-loop sampling hook takes &mut Sequence The CUDA lane's whole purpose, on its first run against this branch: three jobs, one error, in the one block a macOS `cargo check` structurally cannot reach. error[E0596]: cannot borrow `*seq` as mutable, as it is behind a `&` reference --> mistralrs-core/src/pipeline/sampling.rs:557:12 557 | && seq.sampler().is_greedy_trivial() `Sequence::sampler()` takes `&mut self` (sequence.rs:885) even though it only clones an Arc, so the `&Sequence` parameter could never call it. `sample_sequence` already holds `&mut Sequence` and was reborrowing it down to `&`, which is why the call site compiled and the body did not. This is rustc's own suggested fix. Nothing here mutates the sequence, and the `&&` chain still borrow-checks: `sampler()` returns an owned `Arc<Sampler>`, so the mutable borrow ends before `seq.recognizer` is read. Verified by compiling the cuda body on a host with no CUDA — cfg removed, the three `arc_cuda_graph` calls stubbed — which puts the real `Sequence` through borrowck and passes. The `#[cfg(not(feature = "cuda"))]` twin moves with it so the two signatures cannot drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the ArcGraph CUDA-graph capture lane, which until now existed only on
arcgraph/capture-finalandarcgraph/rope-device-posand reached nobody.What this measured (on the branch, before rebase)
The graph served 87 real decode steps (captured=1, replayed=87).
compressed_row_positions(master,models/deepseek4.rs) is a single-slotthread-local keyed
(t_c, ratio, device). Its own comment claims it "hits fromthe second decode step onward, and the copy leaves the capture region for good."
That claim is false on V4.
V4 interleaves compression ratios across layers — CSA layers use ratio 4,
HCA layers use ratio 128. A single slot holds one
(t_c, ratio)pair, so layern (ratio 4) evicts what layer n-1 (ratio 128) just stored, and layer n+1
evicts it right back. The cache therefore misses on essentially every layer of
every step, and the
Tensor::arange_stepit was introduced to eliminate runs~43 times per decode forward instead of once per process.
Outside capture that is a host round trip per layer. Inside capture it is the
SIGSEGV.
arangeuploads a transient hostVecthroughclone_htod→ an asynccuMemcpyHtoDAsync; undercuStreamBeginCapturethat copy is recorded, notexecuted, so the graph stores the host pointer and re-reads it on the first
launch and on every replay — long after the
Vecwas freed. The graph then handsgarbage indices to
index_select, which trips candle's device-side bounds assert(
ids[id_i] < src_dim_size,T = __nv_bfloat16for the bf16 cos table,I = unsigned intfor the U32 positions) and the process takes SIGSEGV on thevery first
cuGraphLaunch.The fix is
layers::compress_positions: one table per(ratio, device), sothe interleave cannot thrash it; capacity rounded up to 1024 so growth does not
rebuild it; served as a zero-copy
narrowview. The singlearange_stepthenhappens once, during warmup, outside any capture region.
Capture stays default-OFF
ARC_V4_CAPTURE_PROBEis unset by default —mistralrs-core/src/pipeline/normal.rs:1886.With it unset the forward runs eagerly and
arc_profiler::mark_unreachablerecords why the graph node reports 0 ns, so an unexecuted node cannot read
like an instant one.
ARC_V4_KV_WINDOW(models/deepseek4.rs:2994) is likewisedefault-off. This PR does not flip either.
The substance
Five bugs that made capture impossible, plus the instruments that found them:
Tensor::arangein the captured forward is a dangling HOST pointer.arangeuploads a transient hostVecviaclone_htod→ an asynccuMemcpyHtoDAsync. UndercuStreamBeginCapturethat copy is recorded,not executed: the graph stores the HOST POINTER and re-reads it on the first
launch and every replay. The
Vecis freed when the expression returns, sothe graph copied freed memory into
positionsand handed garbage indices toindex_select→ device-side bounds assert → SIGSEGV on the very firstcuGraphLaunch. Four call sites, now served from cached device tables builtonce outside capture (
layers::compress_positions,layers::positions_f32).for non-power-of-two capacities.
allocation.
end_capture_and_cacheis marked, and the warm-pass sync is no longerdiscarded, so a poisoned context is visible before the graph runs.
graph is wired correctly, max|Δ| = 0.000e0 on the capture step) and the
per-layer trace that localises the surviving divergence to layer 0's interior
and identifies it as a wrong KEY write rather than a wrong window read.
Rebase notes
origin/arcgraph/rope-device-poswas 45 commits ahead of master, but 26 ofthem had already landed under different SHAs (#177, #185, #189, #199–#204 and
friends).
git rebase origin/masterdropped those 26 by patch-id, leaving 19.Of those 19: 11 land here (+1 hygiene commit = the 12 in this PR), 5 are
blocked on the candle pin and 1 is held back — both explained below — and 2 were
dropped as superseded:
472851585(EXPECTED_KERNEL_COUNT 39→40) — superseded. Recounted with theexact glob
mistralrs-quant/build.rsuses (kernels/*/*.cu): 44, which iswhat master already has. Master's "6 dummy stubs" note is also correct (the
7th match is
dummy_marlin_kernel.cuhitting both patterns).afb9c26f6(close the F32 seam) — already on master as1f6ef9da9; the twodiffer only in comment reflow and one local's name.
The junk that
7f5f3955chad committed into a merge commit (14.playwright-mcp/captures, two PNGs, two
.tgzlog bundles, and an unrelatedCLAUDE.mdedit —present in neither parent) is gone, because a rebase drops merge commits.
Conflicts resolved
deepseek4.rs— compressed-row positions. Master hadcompressed_row_positions, a SINGLE-slot thread-local keyed(t_c, ratio, device);the branch had
layers::compress_positions, a per-(ratio, device)table,chunk-rounded to 1024 and served as a zero-copy
narrow. Kept the branch's.A single slot thrashes on V4, which interleaves CSA (ratio 4) and HCA
(ratio 128) layers — consecutive layers of the same step evict each other, so
master's "hits from the second decode step onward" does not hold and the
arange_stepis paid per layer, including inside the capture region where itis a SIGSEGV rather than a cost. Master's version carried no engagement
counters, so nothing telemetric is lost; its arithmetic argument for
arange_step(the old float chain'sfmaf(j, ratio, 0.0)is exact) is keptverbatim at the old site, and the now-dead helper (one caller, replaced) is
removed.
dsv4_attention.rs— compressed-branch mask. Rebased over perf(ArcInfer/ArcAttention): the decode raw-window mask is all ones — 467 launches/token, and the 1,571-cast MHC premise retracted #206(
02edd31cc), which restructured all threearangesites here. perf(ArcInfer/ArcAttention): the decode raw-window mask is all ones — 467 launches/token, and the 1,571-cast MHC premise retracted #206 provedthe raw-window mask is all-ones at
t_q == 1and deleted thekp/qpchainoutright on the decode path, so those two swaps are dropped — deletion
beats substitution, and master's spelling is kept verbatim. One swap survives:
the compressed-branch
bpindex range, which perf(ArcInfer/ArcAttention): the decode raw-window mask is all ones — 467 launches/token, and the 1,571-cast MHC premise retracted #206 keeps as aTensor::arangeeven at
t_q == 1. It is now served from the same cache in its U32 form,compress_positions(t_c, 1, dev), which is bit-identical toTensor::arange(0u32, t_c as u32, dev)and is pinned by a new test. Net deltain this file is one line of code.
layers::positions_f32had no callersleft afterwards and is removed with it.
normal.rs— same-step probe. Indentation only; the block was writtenagainst a nesting level introduced by the alloc-prewarm commit that is not in
this PR. Re-indented, no logic change.
EXPECTED_KERNEL_COUNT— recomputed, master's 44 kept (above).The candle pin
Unchanged:
89ab14ef1216331a782539f533b99f6816708604, byte-identical to master.Cargo.tomlandCargo.lockare not in this diff at all.This is deliberate and it is the reason this PR is not the whole lane. The two
candle lines have diverged, they are not fast-forward:
Master's three commits include
89ab14ef1 fix(cuda): eviction must not run inside the capture window— itself a capture fix. Pinning back to9211966would silently revert it. So the five arc commits that call the branch-side
candle APIs (
arc_capture_retain_host,arc_capture_htod_retained,arc_capture_dtoh_count,prewarm_alloc_cache,reset_capture_misses— master's89ab14ehas onlyset_capture_mode) are held out of this PR:492b7baa7,1c371b2cc,f7c65cb72,8968c1fa3,4475f9e71. They need a candle-side mergeof the two lines first, and both rewrite the same ~400 lines of
cuda_backend/device.rs(LRU eviction + capacity vs. prewarm demand-profile +capture-miss ledger). That merge is real work and it is not GPU-validatable from
CI, so it should not ride in on a rebase.
Verified
cargo check --workspaceunused doc commentin a file this PR touches is fixed)-D warnings)cargo test -p mistralrs-core -p mistralrs-quant -p mistralrs-visionNo test was weakened, ignored, or deleted.
Not in this PR
ab161876c"UNVERIFIED — device-resolve the fused-QK RoPE position" is held backon
arcgraph/rope-device-pos-rebased. It changes the default eager path(
ARC_QK_FUSEDis on unless set to 0), it has never been compiled by nvccanywhere, and its own commit message says no PR should open until it has been
measured on a box.