perf(ArcInfer/ArcAttention): fuse the V4 Q/K pre-attention block — 16 launches per layer become 1 - #187
Merged
Conversation
Contributor
Author
|
🚫 UNVERIFIED — NOT MERGEABLE. Do not read the check list on this PR as a pass. Every CI run on this PR is in The standing rule is that This PR has never had a real CI verdict. Treat it as red until it does. I am re-triggering CI staggered (not all six at once — simultaneous dispatch is what exhausted the runners). Once this PR shows a genuine To re-trigger by hand: |
Contributor
Author
|
Correction to my earlier UNVERIFIED note — the cause is now identified, and this PR structurally cannot go green while it is stacked. What the single So it is not a code defect and not infrastructure — it is 🔑 But re-triggering will not make this PR green, and that is by design.
The repo deliberately chose red over blank for stacked PRs. So this PR's red is honest and expected, not a defect, and it will remain red until its parent merges and it is retargeted to What that means practically: do not try to make this green in place, and do not read its red as a code problem. The verification order is forced — parent lands, retarget to master, then CI can return a real verdict on this rung. The blast-radius objection above (default-ON Same structural constraint currently applies to #98, #109, #172 and #184. |
…path
The V4 B=1 decode step pays 43 blocking `cuMemcpyDtoHAsync_v2` per token
(~109us each, 4.81 ms/token) inside `dsv4_kv_fp8::e4m3_codes_cpu`, which
round-trips every layer's scaled K block through the HOST because candle
has no CUDA F8E4M3 cast. Those copies are invisible to the obvious grep
(`*Synchronize*` = 0.0 calls/step) and they make CUDA graph capture
impossible: a graph cannot record a blocking D2H.
The measured trap: the existing sync-free path (`ARC_GPU_ACT_QUANT=1`,
`GpuApprox`) is SLOWER on H200 - interleaved A1 66.99 / B1 67.71 / A2
68.05 / B2 70.30 ms/token, with `kv_fp8_quant` 73.49 -> 134.18 us/call
(+83%). It swaps one blocking copy for ~19 extra elementwise launches per
layer, and this machine is bottlenecked on op count (9,131 launches/token,
median kernel 1.18us, memory controller 4% utilized), not kernel speed. So
`GpuApprox` is not the fix and is documented here as not being one.
This adds `mistralrs_quant::arc_kvquant`: one fused kernel per direction,
replacing ~11 candle ops on the quantize side and ~13 on the dequantize
side with a single launch each, and removing the D2H entirely. The byte
format is unchanged.
Bit-parity with the CPU path is the bar, and it is obtained by
construction rather than by hope:
* the E4M3 rounding is a transcription of NVIDIA's
`__nv_cvt_double_to_fp8(x, SATFINITE, E4M3)`, which is what the Rust
`float8` crate ports and therefore what `F8E4M3::from_f32` computes;
* `scale` reproduces `(amax / 448.0)?.affine(1.0, 1e-12)` exactly,
including that candle lowers `Tensor / f64` to a MULTIPLY by the
f32-rounded reciprocal;
* mistralrs-quant compiles with `--use_fast_math`, so every float op is
an explicit `__f*_rn` intrinsic (IEEE, unaffected by -prec-div/-ftz)
and the amax reduction runs on absolute-value bit patterns as
unsigned integers rather than through `fabsf`/`fmaxf`;
* dequant indexes the SAME 256-entry `F8E4M3::from_bits(i).to_f32()`
table the candle path fed to `index_select`.
D33: the kernel ships a deliberate mutant (RNE replaced by truncation,
everything else identical) reachable only from the parity test, so the
comparison is shown to fail on a wrong kernel. The test also asserts the
fused call counters moved - a parity check on this exact subsystem has
already passed vacuously by comparing an implementation to itself.
D14: the GPU parity test exits 2 (environment failure) rather than
passing when no CUDA device is present.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…z) + box paths The emitted PTX showed nvcc 13.1 turning __fmul_rn/__fadd_rn/__fdiv_rn into mul.rn.ftz.f32 / div.rn.ftz.f32 / add.rn.ftz.f32 under --use_fast_math, which candle-kernels (no fast math) does not do. Replaced with inline PTX, which no optimisation flag rewrites, so 'grep -c .ftz.f32' over the PTX is the audit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Runs on hardware with nvcc alone (no cargo build). Compares the kernel's transcription against NVIDIA's software reference (what the Rust float8 crate ports, hence what candle's CPU cast computes) and against the sm_90 hardware path, over ALL 2^32 f32 bit patterns. Measured on H200 / CUDA 13.1: visited 4,294,967,296 of 4,294,967,296 mismatch vs NVIDIA sw 0 mismatch sw vs hw 0 negative control 123,731,850 inputs caught (2.88%) The visited counter and the -DMUTANT=1 control exist because '0 mismatches' from a sweep that ran zero iterations is indistinguishable from a pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 43 blocking cuMemcpyDtoHAsync_v2 per V4 decode step are invisible to the obvious instrument (*Synchronize* is 0.0 calls/step), so this counts the copies themselves and pins the step count from the trace with two independent anchors rather than assuming it. Known-answer test on the recorded baseline trace (/root/budget-chain/nsys): D2H PER STEP 44.09 (BUDGET_V4_B1.md records 44) LAUNCHES PER STEP 9131.5 (records 9,131) 1,792 B x 8,144 = 43.09/step; 517,120 B x 189 = 1.00/step -- the exact two DtoH sizes the budget names. Instrument validated before being pointed at the new traces. Drops the earlier measure_kv_fp8_fused.sh: it was written before the box paths were known and its exclusivity check had a defect (it cleared the pid it was meant to compare against). Replaced by the scripts actually run on the box. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Counts survive contention (nsys traces only this process), but V4 is ~79 GB of the H200's 143 GB, so holding the bench lock is not enough — the previous holder's process can still be resident. Gates on nvidia-smi free memory and exits 2 on OOM or a missing report rather than reporting a partial trace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sync Adds cuMemcpyDtoHAsync_v2 / cuLaunchKernel / cuMemAllocAsync / cuMemFreeAsync per step, and prints ALL *Synchronize* alongside — it reads 0.00/step, which is exactly why counting syncs on this workload finds nothing and concludes wrongly. Known-answer re-test on the recorded baseline trace now reproduces every headline in BUDGET_V4_B1.md: 44.09/step cuMemcpyDtoHAsync_v2 (recorded 44) 11436.33 + 11436.23 alloc/free/step (recorded 11,436 each) 2818.31/step cuMemcpyHtoDAsync_v2 (recorded 2,818) 0.00/step ALL *Synchronize* (recorded 0.0) 9131.5 launches/step (recorded 9,131) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured on the box: a chain wrapping its whole pipeline in one flock held /root/locks/bench.lock with 13-17 waiters queued while the GPU read 0 %, 0 MiB, 78.28 W. nsys report export and the per-step counting are pure CPU on files already written, and must not hold the card. The script now takes the lock itself, per leg, around the VRAM wait and the traced run, and releases it the instant the bench exits — and says so in the output so the release time is auditable. Do not wrap it in flock. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lock covers exactly one leg (model load, which allocates, plus the timed run) and is released before parsing. Interleaves A-B-A-B and prints the within-arm drift beside the delta, because this box once drifted 3.3% monotonically — more than the arm difference. On a bench failure it dumps 'dmesg | grep -i xid' first: this box carries ~1,485 Xid GPU faults (ECC uncorrected 0, so not memory corruption), and a process killed by one dies with no error line. Distinguishing the box's fault from the code's is the difference between fixing a bug and chasing one that does not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…URED]
Box arc-v4-stack (H200), binary md5 e15259dc9ce935fa8782ba832cac1992 in a
PRIVATE target dir (/root/arc-wt/fp8-target, not the shared /root/arc-wt/target
that a neighbour's build overwrote tonight), 118 arc_kv_fp8_quantize symbol
hits, --features 'cuda flash-attn'. nsys, 20 s steady-state tail, step count
pinned from the trace by two independent anchors.
before (cpu) after (fused)
cuMemcpyDtoHAsync_v2 43.89/step 1.00/step
... of which 1,792 B 42.89/step size ABSENT from the trace
... of which 517,120 B 1.00/step 1.00/step (logits, not ours)
kernel launches 9,081.8/step 8,404.5/step (-677.3)
cuLaunchKernel 7,892.4/step 7,119.0/step (-773.5)
cuMemAllocAsync 11,376.7/step 10,618.0/step (-758.6)
ALL *Synchronize* 0.00/step 0.00/step (why grep lies here)
Per-call, which is the evidence that survives a contended box:
before cuMemcpyDtoHAsync_v2 48.56 us/call HOST time -> 2.131 ms/step blocking
after arc_kv_fp8_quantize_kernel 1.98 us/call, 42.93 calls/step
arc_kv_fp8_dequantize_kernel 2.34 us/call, 42.93 calls/step
fused total 0.1853 ms/step device time
42.93 rather than 43.00 is one step straddling the window edge (99.84%); the
before arm's 1,792 B D2H shows the same 0.26% at 42.89/43. Engagement is
therefore proven per layer, not assumed.
NO end-to-end tok/s delta is reported. A naive A-B-A-B delta on this box is
biased by exactly one slot of drift, and four end-to-end numbers here turned
out to be pure environment in one night. The A/B driver is dropped rather than
shipped with a result it cannot support; the case rests on per-call cost and
launch counts, which do not depend on how long the step took.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… typos selecting GpuApprox The name says "mode", which reads as an on/off for FP8 KV. It is not one, and it never was: it selects WHICH ARITHMETIC produces the E4M3 code, and all three variants quantize. There is no "off" — V4 is FP8-QAT, so the quantize/dequantize round trip at `deepseek4.rs:1689` is the model's numerics, not an optimisation, and it is correctly unconditional. The flag that gates FP8 KV *storage* is a different variable, `ARC_V4_FP8_KV`. Three defects, in descending severity: 1. A TYPO COULD SILENTLY SELECT NON-BIT-EXACT ARITHMETIC. The old table was `_ if ARC_GPU_ACT_QUANT is set => GpuApprox` followed by `_ => FusedDevice`. Both arms were reachable ONLY for an unset or MISSPELLED value. So on any box that still had `ARC_GPU_ACT_QUANT` exported from an earlier experiment, `ARC_KV_FP8_MODE=fusd` selected `GpuApprox` — the one variant this module documents as round-half-away-from-zero rather than round-half-to-even, and labels "NOT the fix and must not be shipped as one". Resolved once into a `OnceLock`, so it stuck for the process lifetime with no trace. An unparsable value now lands on the bit-exact default and SAYS SO, on both stderr and `tracing::error!` — never on `GpuApprox`. 2. The docs stated the opposite of the code. `PROFILING.md` annotated the `kv_fp8_quant` span "opt-in, ARC_V4_FP8_KV=1". That span opens at `deepseek4.rs:1688` and runs on every forward regardless of the flag; the adjacent `kv_fp8_dequant` line carried no such note, so the doc was internally inconsistent too. The annotation moves to `kv_cache_append`, which is where `ARC_V4_FP8_KV` actually acts. 3. `unwrap_or_default()` collapsed unset and empty-string, and nothing trimmed. Renamed to `ARC_KV_FP8_IMPL`. The old spelling still works and prints a deprecation on both channels — renaming it silently would convert every operator's muscle memory into a fresh silent failure, which is the disease being cured, not the cure. The table is now a pure `parse_impl(Option<&str>, bool)` with a test that pins every arm INCLUDING the typo-must-not-reach-GpuApprox regression. It runs on the free CPU lane; no GPU is required to keep this honest.
…uant.cu The ArcGate kernel-count tripwire caught a real regression introduced by rebasing this branch onto current master, and it is worth recording how, because the failure mode is subtle. This branch originally carried `EXPECTED_KERNEL_COUNT 39 -> 40 — arc_kvquant.cu was never counted`. Meanwhile master independently went 39 -> 40 for a DIFFERENT kernel. On rebase, git compared patch texts, saw an identical `-39 / +41`-shaped hunk already upstream, and dropped the commit as "patch contents already upstream". The number was right; the REASON was not. Net effect: the glob discovers 41 sources while the file still claims 40, so `arc_kvquant.cu` would once again be the kernel that goes missing quietly — which is the exact failure this file was created to make impossible. Caught by `cuda_kernel_build_guard::expected_kernel_count_matches_disk` on the free CPU lane, before any GPU time was spent. That is the tripwire earning its keep, not a nuisance — do not "fix" a future occurrence by relaxing the guard.
`set_alloc_cache_enabled(true)` sat behind three stacked default-off gates:
`probe && seq_len == 1 && env("ARC_CANDLE_ALLOC_CACHE")`. The first conjunct
tied a general-purpose allocator to the V4 capture probe, so the only way to
recycle a decode step's frees was to also be capturing. The ~11k allocations
per token were a disabled feature, not a missing one.
Replaced with a pure policy, `alloc_cache_action(seq_len, enabled, killed)`:
* decode (`seq_len == 1`) -> Enable
* prefill (`seq_len != 1`) -> DrainAndDisable
* already in that state -> Leave
Prefill draining is not incidental. The cache is keyed on exact byte count
(`free: HashMap<usize, Vec<CUdeviceptr>>`, no bucketing, no smallest-fit) and
has no capacity bound and no eviction, so a prefill's large one-shot buffers
would be parked for the process lifetime under a key nothing asks for again.
`ARC_CANDLE_ALLOC_CACHE=0` is the kill switch. Any other value, and unset,
leave the policy in force, so the `=1` the ops scripts pass still means what
it always meant.
The capture probe keeps its own gate for the graph-mode positions; only the
allocator moved out from under it.
Six host-runnable tests for the policy. The allocator has no tests at all in
either repo — candle's `cuda_backend/{device,mod}.rs` carry no `#[cfg(test)]`
and every arc-side exercise is `#[cfg(feature = "cuda")]` — so the decision of
when it is on is now the part that CI can see.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
… prove it Picks up candle b2a4dd80, which gives `AllocCache` a capacity and LRU eviction. The allocator had neither, and arc only drains it on a prefill, so a single long generation grew forever. Measured on an H200 over 2 600 tokens, `memory.used` at 2 Hz: **+6.04 MiB per decoded token with no plateau**, against +0.057 MiB/token for the same run with the cache off — so the growth is the cache's and nothing else's. `ARC_ALLOC_CACHE_MAX_MB` sets the cap; unset leaves candle's 1 GiB default; `0` restores the old unbounded behaviour for A/B. A typo deliberately does *not* fall back to unbounded. `ARC_ALLOC_CACHE_STATS=N` prints the allocator's counters every N decode steps: allocations per step, **frees per step**, hit rate, bytes held against the cap. Those are the numbers this has to be judged on. A green log is not evidence — an earlier arena here reported "accounting OK" and bit-identical output over 52 steps while silently bypassing itself for every buffer under 128 bytes (KERNEL_RULES.md:977-984). Allocations staying low *and* frees being non-zero is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
Test-only change on the candle side; `git diff b2a4dd80..859c49c8` touches nothing outside `#[cfg(test)] mod alloc_cache_tests`. The H200 numbers in the branch description were measured at b2a4dd80 and stand. Correction to that commit message: the allocator has **ten** tests, not eleven. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
Closes a free path this change introduced: leaving capture mode re-filed parked buffers through the evicting put, which could hand a private-pool pointer back to the driver at a moment arc-cuda-graph does not control. Decode is unaffected — `set_capture_mode` is only called during a capture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
Two sites in pipeline/normal.rs — a doc comment and a test name. No behaviour change; the Typos job was the only red check on this PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`chore(deps): candle 89ab14ef` bumped all five candle crates in
`Cargo.toml` but left `Cargo.lock` pinning 88d86a2. That is the stale-lock
trap: a `--locked` CI build resolves the LOCK, so the lane would have
compiled the OLD, UNBOUNDED allocator while reporting green on a PR whose
entire subject is bounding it. `cargo metadata --locked` now succeeds; it
failed before this commit.
Regenerated with `cargo update -p candle-{core,nn,flash-attn,flash-attn-v3,metal-kernels}`.
The only non-candle churn is a `windows-core` 0.61.2/0.62.2 dedup that fell
out of the re-resolution; nothing arc builds on Linux or macOS reads it.
… fix the sinkhorn local-memory spill At b=1 the V4 decode step is launch-bound: ~7,924 cuLaunchKernel calls, of which the MoE router region is 3,259 (41.8%) across 86 contiguous spans — exactly two per layer (mhc_attn_pre, mhc_ffn_pre; 43 layers). Mean kernel in that region is 2.79 us and 89% are under 5 us, on [1,24] / [1,4,4] / [1,256] tensors. That is launch overhead wearing a kernel costume. Three changes, all bit-identical to what they replace: 1. cuda/hc_fused.cu `hc_pre_fused_f32` collapses hc_pre's 18-launch middle — a hand-decomposed RMS statistic in SEVEN launches (sqr, fast_sum, affine, affine, recip, sqrt, bmul) plus ELEVEN for the pre/post/comb scoring — into one kernel. `mixes` is never materialised. 2. cuda/hc_fused.cu `sqrt_softplus_f32` collapses MoeGate's sqrt(softplus(x)) from nine launches (zeros_like, bmaximum, uabs, uneg, uexp, affine, ulog, badd, usqrt) into one. 3. cuda/sinkhorn.cu is templated on `hc`. It took `hc` at runtime, so nvcc could not keep r[16]/buf[16]/col[16] in registers and demoted all three to local memory — `ptxas -v` reported 192 bytes stack frame. With hc=4 the kernel runs ONE block of FOUR threads, so nothing hides that latency: measured 30.0 us/call on a [1,4,4] tensor, 2.549 ms/step over 86 calls, 28% of the router's GPU time. Templating makes every index compile-time; ptxas now reports 0 bytes stack frame, 29 registers. Arithmetic unchanged. Bit-identity is by construction, not by tolerance: this region decides WHICH EXPERTS RUN, so a reassociated sum or a contracted FMA can change the emitted token. hc_fused.cu joins sinkhorn.cu in build.rs's dedicated no-fast-math, --fmad=false builder and carries the same #error guard, and it transcribes candle's ops exactly — including that `recipg` is `1.0 / a` with a *double* literal, that `mean_keepdim` is fast_sum followed by a separate affine, and that candle's FastReduce block_dim is min(1024, len).next_power_of_two(), which fixes the reduction tree's shape and therefore the f32 rounding. cuda/hc_fused.rs carries scalar replicas of both sides asserted bit-identical without a GPU, plus two anti-vacuity guards. The softplus one earned its place: its first version sampled [-30, -12.5, -1, 0.5, 3.25, 17] and passed against the naive log(1+exp(x)) form, because the two agree bit for bit until exp overflows near 88.7. It is now a three-way mutation probe. ARC_HC_FUSED=0 restores the eager chains so both can be A/B'd from one binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
…phic wrap closure Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
The sinkhorn guard asserted build.rs contained the literal `.exclude(&["sinkhorn.cu"])`, so adding a second bit-identity-critical kernel to that list failed it for the one reason that is not a regression — the kind of failure that gets 'fixed' by deleting the assertion. It now matches on the exclude list's CONTENTS, and hc_fused.rs carries the mirror guard: fast-math #error present, IEEE intrinsics present, __expf/__logf/ __fdividef/rsqrtf absent, and the file wired into the --fmad=false builder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
Measured on H200, git 6f4cd0d, bin f6ec4e85, toggled with ARC_HC_FUSED on one binary so the comparison has no second variable: ARC_HC_FUSED=0 7,494.3 launches/step 10,892.4 allocs/step 17.3 tok/s 57.70 ms/T ARC_HC_FUSED=1 5,691.0 launches/step 8,918.5 allocs/step 20.1 tok/s 49.66 ms/T The off leg reproduces origin/master (7,477.9 launches, 17.0 tok/s) to within 0.2%, so the switch is the only thing that moved and nothing else regressed. Router region 76 -> 38 kernels/layer, 0.207 -> 0.100 ms GPU. Sinkhorn 30.03 -> 9.12 us/call (2.583 -> 0.785 ms/step). Output bit-identical across the toggle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SpVNMpb13HkUXqSqbN1o9H
…hes per mixing point become 7 At b=1 the V4 decode step is launch-bound, not bandwidth-bound. After the router-region fusion (8b79311) one mHC mixing point was still FOURTEEN kernel launches, and there are 86 of them per token (43 layers x {attn, ffn}). Two blocks are collapsed here, both operating on [1, 4, 4096] tensors: hc_pre tail `pre.unsqueeze(-1).broadcast_mul(x_f32).sum(1)` + the narrowing cast — 3 launches to compute a 4-term weighted average. hc_post whole — 6 launches whose real work is 4 multiply-adds per output. MEASURED, H200, clocks locked at 1980 MHz, ONE binary toggled with ARC_HC_FUSE2 (a two-binary before/after would confound the change with the rebuild): launches/step 6,192.5 -> 5,566.4 (-626.1, -10.1%) ms/token 40.50 -> 38.18 (24.7 -> 26.2 tok/s, +6.1%) GPU time/step -1.48 ms Every kernel that moved is one of the nine this touches, each at 86/step: cast_bf16_f32 -172.2, cast_f32_bf16 -172.1, bmul_f32 -172.0, fast_sum_f32 -86.1, badd_f32 -86.0, sm80_xmma_gemm -86.0, against hc_post_fused +85.9 and hc_y_combine +85.9. Nothing else moved: arc_kv_fp8_quantize 42.98 -> 42.95, qtip_gather_gemv 128.93 -> 128.86, dot_kernel 86.95 -> 86.90. The gate GEMV is deliberately left alone. cuBLAS splits the [1,16384]x[16384,24] product into dot_kernel + reduce_1Block, and its K=16384 reduction order is not reproducible by construction. Fusing it is the only way to merge hc_post with the next hc_pre into one cross-layer kernel; that trade — one launch per seam against a tolerance-based result — is not taken. BIT-IDENTICAL, and proven on live data rather than argued. ARC_HC_AB=1 compared 4,000 tensors against the eager chain with zero mismatching bits. ARC_HC_AB_POISON=1 perturbs one element by a single ULP and the same comparison flags all 4,000 (fused=0xbe32 eager=0xbe31), so the check is live rather than vacuous. hc_fused.cu has cited `ARC_HC_AB=1` as "the final proof" since it was written; the string appeared nowhere else in the tree. This adds it. Also records a measured defect rather than fixing it silently: the `#if defined(__USE_FAST_MATH__)` guard in BOTH hc_fused.cu and sinkhorn.cu is dead. nvcc 12.4 defines neither __USE_FAST_MATH__ nor __FAST_MATH__ in either pass — nvcc --use_fast_math -E -dM x.cu | grep -i fast -> no output nvcc --use_fast_math -arch=sm_90 -c guard.cu -> compiles clean — while --use_fast_math does reach the device pass (a float divide drops from 3 MUFU/RCP instructions to 1). Two bit-identity-critical files were relying on a macro that cannot exist. The real protection is the build.rs wiring already asserted in tests plus the runtime A/B added here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…6 launches per layer become 1
The b=1 decode step is launch-bound, and after the mHC/router fusions 34.3% of
what is left is PURE DATA MOVEMENT: ucopy_bf16 514/step, copy2d_bf16 418,
cast_bf16_f32 356, cast_f32_bf16 221, cast_u32_f32 203. This takes the largest
ucopy/copy2d contributor in that list.
`Attention::forward` spelled the head transpose, the per-head Q RMS-norm and the
RoPE as SIXTEEN candle launches per layer, TEN of them pure copies materialising
intermediates that exist only because the chain is written as separate ops: a
transpose `.contiguous()`, two `.contiguous()` narrows inside `rope_i`, and two
two-input `cat`s. Their entire data footprint at decode is 64 KB of Q and 1 KB
of K.
`cuda/qk_norm_rope.cu` does all of it in one launch of n_heads+1 blocks. The
transpose becomes an output ADDRESS, the NoPE/PE split and re-`cat` become two
write ranges of the same row, and the RMS statistic stays in a register. The kv
projection is hoisted above the Q RMS-norm so both rows fit one launch; the two
projections read only `xs`, so no arithmetic moves.
BIT-IDENTICAL, proven on the live model rather than asserted: ARC_QK_VERIFY=1
runs both paths at every layer of every step and compares bf16 BIT PATTERNS, and
every comparison first runs a negative control -- a 1-ULP poison that MUST be
detected, else the leg declares itself VOID rather than passed. 16 forwards x 43
layers x 2 tensors, zero mismatches, on an H200 with clocks locked at 1980 MHz.
Getting there cost two wrong theories, both recorded in the kernel header so the
next reader does not re-derive them:
* bf16 arithmetic is NOT immune to --fmad. nvcc contracts one of the two
products of `a*c - b*s` into an fma.rn.bf16, rounding it once with the
subtraction instead of twice. Measured for head 0 / pair 0 at position 1
(a=-5.65625, b=-3.546875, cos=0x3f0a, sin=0x3f57, with the diagnostic
confirming both sides read the SAME table row): three roundings give
-0.0625, full f32 gives -0.069824, and candle gives -0.064453125 -- an
exact tie resolved by round-half-to-even. So the file gets its OWN builder
carrying candle-kernels' flags, and the contraction is written out with
__hfma rather than left to the compiler.
* The first attempt at that fix changed NOTHING -- byte-identical output.
`ar t` showed the stale --fmad=false object still inside
libmistralrssinkhornieee.a while the new archive was empty, so the linker
resolved the old definition and a green build proved nothing. The symbol is
renamed arc_qk_norm_rope_bf16_v2 so that failure mode becomes a link error,
never a silent wrong answer.
Toggle with ARC_QK_FUSED=0 (restores the eager chain, so the A/B runs from ONE
binary and cannot be confounded by a rebuild). The shape gate refuses anything
outside the specialised set rather than approximating it, and logs the refusal
once, so a permanently disengaged fast path cannot masquerade as a working one.
An engagement counter backs that up: "the benchmark got faster" and "the kernel
ran" are different claims.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MEASURED, H200, ONE binary toggled with ARC_QK_FUSED, clocks stable at 1830 MHz
in every leg (that is this card's actual load clock; `-lgc 1980` is applied but
nothing throttles -- 122 W of 700 W, 28 C, no active clock-event reasons).
ops/token 5,347.8 -> 4,744.9 launches/forward (-602.9, -11.3%)
memcpy 1,840.3 -> 1,580.9 (-259.4, -14.1%)
allocs 212.7 -> 205.7 ( -7.1, -3.3%)
Both legs pinned by TWO INDEPENDENT ANCHORS that had to agree before the leg
counted: the API's completion_tokens, and a per-layer kernel count factoring as
43 x k x N (bminimum_f32 = 27,520 = 43 x 4 x 160). Counts read from a full
GROUP BY over the exported sqlite, never a top-N table.
Every kernel that moved is one of the sixteen this change removes, and the
per-kernel deltas reconcile to the launch:
ucopy_bf16 474.1 -> 300.8 -173.3 (~4 x 43)
rope_i_bf16 155.5 -> 69.5 -86.0 ( 2 x 43)
affine_bf16 128.7 -> 42.7 -86.0 ( 2 x 43)
copy2d_bf16 376.4 -> 290.9 -85.5 (~2 x 43)
bmul_bf16 43.0 -> 0.0 -43.0
fast_sum_bf16 43.0 -> 0.0 -43.0
urecip_bf16 43.0 -> 0.0 -43.0
usqr_bf16 43.0 -> 0.0 -43.0
usqrt_bf16 43.0 -> 0.0 -43.0
qk_norm_rope_kernel 0.0 -> 43.0 +43.0
-------
-602.8 vs -602.9 measured
Decode, INTERLEAVED A-B-A-B (a single before/after on a shared box is not
evidence), ms/token by the difference method so TTFT cancels:
OFF r1 37.967 ms 26.34 tok/s ON r1 35.732 ms 27.99 tok/s
OFF r2 37.960 ms 26.34 tok/s ON r2 34.964 ms 28.60 tok/s
----------------------------------------------------------------
OFF 37.964 ms 26.34 tok/s ON 35.348 ms 28.29 tok/s
-6.9% ms/token, +7.4% tok/s. The two OFF legs agree to 0.02%, and neither ON leg
overlaps either OFF leg. The OFF baseline reproduces the previous session's
38.18 ms/token independently.
All four legs returned the SAME 160 greedy tokens (text sha 6d6456512d885a6f),
so 640 tokens are byte-identical across the toggle -- an end-to-end check on top
of the per-tensor bit comparison.
Engagement asserted in both directions: engaged_lines=1 / declined_lines=0 with
the kernel on, and 0/0 with it off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
heydryft
force-pushed
the
agent/mhc-fuse-post
branch
from
August 20, 2026 17:48
df466d6 to
179e405
Compare
heydryft
force-pushed
the
agent/datamove-fuse
branch
from
August 20, 2026 17:48
b424f5c to
cd647ce
Compare
Code Metrics Report━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Language Files Lines Code Comments Blanks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ C Header 5 305 210 52 43 CSS 2 1181 1036 34 111 CUDA 74 25734 18232 4703 2799 Dockerfile 1 39 22 8 9 JavaScript 16 3546 2676 482 388 Jinja2 7 694 656 5 33 JSON 74 4600 4597 0 3 Makefile 1 6 5 0 1 Metal Shading Lan| 33 12224 9431 1142 1651 PowerShell 1 300 227 30 43 Python 145 15139 12482 811 1846 Shell 40 9948 6637 2655 656 Plain Text 4 3801 0 2479 1322 TOML 33 1498 1294 54 150 YAML 3 25 23 2 0 ───────────────────────────────────────────────────────────────────────────────── HTML 4 2687 2604 43 40 |- CSS 2 543 479 37 27 |- JavaScript 1 1233 1215 12 6 (Total) 4463 4298 92 73 ───────────────────────────────────────────────────────────────────────────────── Jupyter Notebooks 4 122 83 23 16 |- Markdown 1 60 30 22 8 |- Python 1 122 113 1 8 (Total) 304 226 46 32 ───────────────────────────────────────────────────────────────────────────────── Markdown 204 45330 0 35183 10147 |- BASH 72 1655 1203 331 121 |- C 3 17 17 0 0 |- CUDA 2 84 56 16 12 |- JSON 18 708 708 0 0 |- PowerShell 1 1 1 0 0 |- Python 23 1008 787 113 108 |- Rust 66 2051 1716 77 258 |- TOML 6 207 164 0 43 |- YAML 5 41 36 5 0 (Total) 51102 4688 35725 10689 ───────────────────────────────────────────────────────────────────────────────── Rust 673 331228 285444 16819 28965 |- Markdown 491 29538 471 25464 3603 (Total) 360766 285915 42283 32568 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Total 1324 495675 352655 90603 52417 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third rung of the session-8 decode ladder. Fuses the V4 Q/K pre-attention block: 16 launches per layer become 1 (QK-norm + RoPE).
Blast radius — read before merging
mistralrs-core/src/cuda/qk_norm_rope.rs:49reads*ENABLED.get_or_init(|| !matches!(std::env::var("ARC_QK_FUSED").as_deref(), Ok("0")))— fused path ON when unset. Adds
ARC_QK_DIAGandARC_QK_VERIFYalongside (the verify path runs the eager chain next to the fused kernel and compares raw output bytes — that is the right kind of guard). House rule is default-OFF; flip the polarity or merge deliberately.pos_offsethost-scalar-as-kernel-parameter bug class that currently blocks graph replay (mistralrs-core/src/layers.rs:1633-1641, measuredmax|delta| ~ 30). Before merging, confirm the fused kernel takes position from a device tensor, not a host scalar, on any path that may be captured.ARC_QK_VERIFY=1will not catch this outside capture. There is a sibling brancharcgraph/rope-device-posbeing authored specifically for device-resident RoPE positions — check whether it supersedes or completes this.88d86a2, identical to master.Cargo.lockagrees.On the numbers
Measured on this branch's base, not today's master. Re-measure before quoting.
Ordering
Rung 3 of 5. Under it: #175, #177, #185, #186. Merge base-first (D20).
🤖 Generated with Claude Code