fix(ArcGraph): correct cuGraphAddNode ABI; make GPU-autonomous decode reachable

[heydryft]

The ABI bug, first — it closes an open mystery

arc-cuda-graph/src/ffi.rs declared cudaGraphAddNode with five arguments and the first two transposed. CUDA 13.1 cuda.h:21829 declares six, and the first parameter is an out-pointer:

CUresult cuGraphAddNode(CUgraphNode *phGraphNode, CUgraph hGraph,
                        const CUgraphNode *dependencies,
                        const CUgraphEdgeData *dependencyData,
                        size_t numDependencies,
                        CUgraphNodeParams *nodeParams);

The driver therefore received a CUgraph handle where it expected CUgraphNode * and wrote the new node handle through it, while numDependencies got a pointer and nodeParams an uninitialised register.

This gives a cause to the "host heap corruption" CUDA-graph capture failure already on our record, which had been logged with no explanation.

Why graph replay had never once launched

Four further defects, each individually fatal to the autonomous loop:

Defect	Evidence
Conditional handles created with flags = 0	cuda.h:21919 applies defaultLaunchValue only with CU_GRAPH_COND_ASSIGN_DEFAULT; the condition read 0 and the WHILE body ran zero times while every CUDA call returned success
WHILE body captured into a throwaway graph, then destroyed	body left empty; now populated via cuStreamBeginCaptureToGraph (cuda.h:1976 names it as the supported way) with a node-count assert
CUDA_CONDITIONAL_NODE_PARAMS missing its 5th field ctx	cuda.h:1958
DecodeState allocated I64 against int32_t* kernels; reset() reallocated buffers	host/device disagreed on element offsets; reset silently invalidated pointers a captured graph had baked in

And tensor_device_ptr had no I32 arm, so CudaSampler::sample returned unsupported dtype I32 on every call — the correct top-k/top-p sampler had never executed on a GPU. Its tests are a CPU simulator, which by construction cannot observe that.

Measured on an H200 (CUDA 13.1, sm_90)

GPU-autonomous decode works. WHILE conditional node, one launch for N steps: 0.0156 host calls/step at N=128 vs 2.0 for launch-and-sync-per-step. Device-side loop overhead is constant in body size (+2.90/+2.39/+1.98 µs at 1/64/256 kernels) — 0.004% of a 66.68 ms V4 step. The control that makes it mean something: one instantiated graph run with device-side target=1 then target=N returned counter 1 then N, so the device chose the trip count.

Replay safety. The old in-graph sampler took rng_offset as a by-value kernel argument, which a captured graph bakes: 1 distinct token over 64 replays, vs 64 distinct with a device-resident rng_state. It also was not nucleus sampling — it walked the vocabulary in token-id order.

Sampler cost (host + GPU verified exclusive before and after):

support	legacy µs	hybrid µs
1	134.9	166.6	enumerate
8	503.0	624.2	enumerate
64	3127.8	4178.5	fallback
512	24320.7	4185.0	fallback
4096	205125.2	4187.3	fallback
12928	664200.0	4188.5	fallback

Neither component wins everywhere, so this ships a hybrid: exact enumeration while the nucleus is small, threshold bisection past a fixed budget, chosen block-uniformly on device because a captured graph cannot ask the host which branch to take. Costs +24% on peaked distributions (32 µs, 0.05% of a decode step) and removes a 664 ms cliff that was ~10× an entire decode step for one token.

Distribution-verified against the enumerating sampler, with branch counters proving which path each case exercised — TV=0.0000 on peaked, scrambled-order (15.6% per-draw agreement, so the orders genuinely diverged), diffuse (A=0 B=3000), and tie-boundary cases; the deliberately-narrowed control correctly disagreed at TV=0.5027.

Known, and deliberately left

• On tied diffuse supports the fallback keeps 116 tokens where enumeration keeps 111 (TV=0.1103) — it keeps every token sharing a boundary bit pattern, erring wider, never narrower.
• The budget is derived from bisection's pass count (32) when it should follow the fallback's (~70), costing a 0.75× window near support 64. Correcting it needs a more diffuse fixture so the fallback stays covered.
• No V4 end-to-end number. Autonomous decode is unreachable on V4 by construction — no PagedAttention means cache_config is None and the runner is never built (normal.rs:1907, pinned by normal_loaders.rs:5687) — and cuda.h:1971 bars alloc/free nodes inside a conditional body against 11,436 allocations per token.

🤖 Generated with Claude Code

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     81        29475        20161         6275         3039
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     75         4896         4893            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  147        15371        12677          824         1870
 Shell                    42        10304         6858         2769          677
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1498         1294           54          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                213        48932            0        38081        10851
 |- BASH                  72         1655         1203          331          121
 |- C                      4           19           19            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  19          779          779            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  68         2063         1727           78          258
 |- TOML                   6          207          164            0           43
 |- YAML                   5           41           36            5            0
 (Total)                            54789         4772        38624        11393
─────────────────────────────────────────────────────────────────────────────────
 Rust                    681       340988       293252        18096        29640
 |- Markdown             504        32562          471        28063         4028
 (Total)                           373550       293723        46159        33668
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1353       516771       363188        99077        54506
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[heydryft]

Triage verdict: REBASE, not close — LIVE and valuable. Both signals confirmed on current master, with one important correction.

Confirmed unique against master:

• arc-cuda-graph/src/ffi.rs:191 still declares cudaGraphAddNode(graph, node_out, deps, num_deps, params) — five params with the first two transposed against CUDA's six-param cuGraphAddNode(phGraphNode, hGraph, deps, depData, numDeps, nodeParams). That is a real ABI defect on master right now.
• CudaConditionalNodeParams (ffi.rs:75-80) is still missing its 5th ctx field.
• autonomous.rs:426 still only mentions cudaStreamBeginCaptureToGraph in a comment rather than calling it.
• sampling_cuda.rs has no tensor_device_ptr at all.

Correction to the premise this was triaged under. The blocking return Ok(None) is real but has MOVED — master's pipeline/normal.rs is now 2530 lines and the unconditional return sits at line 2324, right after the autonomous_decode: runner allocated ... Graph capture is deferred log, with !runner.is_captured() at 2337 as a second gate. And this PR does not remove it: its entire normal.rs delta is +10 lines adding a top_k field to the sampler config.

So the honest statement is: this PR is genuinely live and unique at the arc-cuda-graph layer (ABI fix, conditional-node flags, +456-line sampling_kernel.cu, device-resident rng_state), but landing it does not by itself make the GPU-autonomous path reachable. The normal.rs:2324 wiring is a separate follow-up and should be its own PR so the ABI fix is not held hostage to it.

Conflict: 1 file, weights.rs.

Not merging now: the new sampling_kernel.cu puts sampling on a capture path, and the bug class currently blocking replay is exactly a host scalar passed as a CUDA kernel parameter during capture (pos_offset; see mistralrs-core/src/layers.rs:1633-1641, measured max|delta| ~ 30). A 456-line sampling kernel entering the capture region needs that specific audit on hardware before it lands. Leaving open with the decision recorded: rebase, split the ABI fix from the sampling kernel, land the ABI fix first.

[heydryft]

Retargeted at the integration branch

Base changed: master → release/openrouter-ready.

The queue is being restructured to the shape the owner asked for: one PR open against master (#194), with everything else merging into a single integration branch. Agents branch off release/openrouter-ready and PR back into it; #194 is the single gate from there to master.

Two things had to land on master first for this to be workable, and both have:

1. The base-branch CI lane used to hard-fail any PR not targeting master (ci(ArcGate): let the base-branch lane permit the single integration branch — 7 PRs are red for addressing, not quality #216). Seven PRs were red for their addressing, not their quality. The lane now accepts master or release/openrouter-ready — an exact-match allowlist, nothing else — and its intent is intact: release/openrouter-ready reaches master only through Arc → OpenRouter-ready: the single integration PR (everything else merges into this branch) #194, whose own base is master, so nothing enters master without a full run whose base is master.
2. release/openrouter-ready was re-cut from current master. It had diverged badly — merging it as-is would have reverted TCFRAG (perf(ArcQuant/ArcKernels): TCFRAG-2B — put the qtip2b trellis GEMV on the tensor cores #203) and re-broken the flag-polarity work (fix(ArcGate): read boolean ARC_* flags by value, so =0 means off #212). It is now master plus only the work it uniquely carried.

This PR was not closed and is not considered stale. An audit of the queue found the overwhelming majority of it to be real work that was never merged, not noise.

What you need to do: rebase onto release/openrouter-ready and resolve conflicts against it rather than against master. CI lanes will now actually run and report on this PR instead of failing at the base check.

[heydryft]

Rebased onto release/openrouter-ready

One conflict, in arc-cuda-graph/src/weights.rs, both halves benign:

• DType::I32 — the integration branch had already added it, with a
  different comment. Kept its wording and folded in this PR's specifics, which
  are the more useful half: CudaSampler::sample requires I32 token_ids
  (sampling_cuda.rs:347) and allocates an I32 keep_idx_scratch (:298), so
  before I32 was listed both fell to the catch-all bail! and sample()
  returned unsupported dtype I32 on every call.
• DType::I16 — present on the integration branch, absent here. Kept;
  taking this PR's side would have deleted it.

Typos gate

The Typos lane failed on 14 hits in the new sampling_kernel.cu: thr and
thr_m, which the dictionary reads as the. Renamed to threshold /
threshold_m rather than widening the allow-list — the precedent set by
a4a083a9f (rename typo-flagged locals (thr->threshold, acount->a_count)).
threshold occurred only in comments beforehand, so there is no shadowing, and
the substitution was word-boundary-anchored: 14 identifiers, no other text
touched. typos now exits 0.

Verification

• cargo check --workspace and cargo check -p arc-cuda-graph --all-targets green.
• cargo clippy --no-deps -p arc-cuda-graph --tests --examples -- -D warnings clean.
• 707 tests pass on the merged #198 + #181 + #130 + #121 tree.
• TCFRAG still opt-in; env_flag_is_set per-file counts identical to the integration branch.

Not verified: the cuGraphAddNode ABI correction and the sampling kernel
are CUDA-only and were not run. The nvcc lanes compile them; nothing here
executes them, and "GPU-autonomous decode is reachable" remains this branch's
claim, not a measurement on this tree.

[heydryft]

ABI independently verified against a machine-generated binding

#[repr(C)] mismatches are invisible to the nvcc lanes — those compile the
.cu, not the Rust struct — so the corrected layout was checked against
cudarc's bindgen output (cudarc-0.19.9/src/driver/sys/mod.rs:6078), which is
generated from the real cuda.h:

pub struct CUDA_CONDITIONAL_NODE_PARAMS {
    pub handle: CUgraphConditionalHandle,
    pub type_: CUgraphConditionalNodeType,
    pub size: ::core::ffi::c_uint,
    pub phGraph_out: *mut CUgraph,
    pub ctx: CUcontext,
}

Field-for-field identical to this PR's corrected CudaConditionalNodeParams,
ctx fifth included. The pre-fix struct omitted ctx entirely, so it was one
pointer short of what the driver writes — the driver's store to ctx landed
past the end of the caller's struct. That is a genuine memory-corruption bug,
not a cosmetic one.

Two more, same source:

• CU_GRAPH_COND_ASSIGN_DEFAULT — mod.rs:240, = 1. Matches this PR's 0x1.
• cuGraphGetNodes(CUgraph, *mut CUgraphNode, *mut usize) -> CUresult —
  mod.rs:10576. Matches this PR's declaration exactly.

(Note the NVIDIA doc page lists the struct's fields alphabetically, not in
declaration order — ctx, handle, phGraph_out, size, type — so it is not usable
for an ABI check. The generated binding is.)