feat(ep): expert parallelism stage 0 + stage 1 (EP=2) for V4's MoE

[heydryft]

Splits V4's 256 routed experts across ranks so each card reads only its slice, with attention left data-parallel. Design and arithmetic: memory/mission/wave44-BV-expert-parallelism.md (PR #70). Full log: memory/mission/wave60-CK-expert-parallel.md.

🔴 NO GPU RAN. $0.00 spent. Nothing here is a multi-GPU measurement — Arc has still never run a forward pass on more than one GPU. The per-user tok/s number is not in this PR; §6 of the wave log is the exact script and its cost.

Stage 0 — three ways stage 1 could have failed for reasons unrelated to EP

• The QTIP CUDA wrappers validated device kind, not ordinal. Activations on cuda:0 with expert weights on cuda:1 passed matches!(.., Device::Cuda(_)) and the kernel launched on the wrong device's stream — silent corruption, not an error. Unreachable while mapper.map co-locates everything; exactly the invariant EP breaks. Fixed in all 10 multi-operand wrappers via qtip/device_guard.rs, whose core predicate is a pure function over DeviceLocation so the ordinal comparison is testable without a GPU.
• No NVLink peer access. Neither candle nor cudarc ever calls cuCtxEnablePeerAccess, so every cross-GPU copy is staged through host RAM — a cost the existing pipeline-parallel map already pays. Adds cuda_peer.rs and, crucially, a call site in DeviceMapSetting::into_mapper that logs at WARN when any pair is not peered. cudarc symbols verified against the real 0.19.4 registry source, not from memory.
• The offline tid2eid distribution, which wave44-BV flagged as "free to check and nobody has": V4's first 3 layers route by a fixed token-id table, so their expert load is a closed-form property of the tokenizer. Plus a per-rank balancedness counter recording both tokens (compute) and distinct experts (bytes), off unless ARC_EP_BALANCE=1 — it needs a D2H read, which is incompatible with CUDA-graph capture.

Stage 1 — EP=2

ExpertPlacement (contiguous / explicit / LPT-balanced) and ExpertParallelPlan remap global expert ids to rank-local ones and zero the weights of unowned slots, from device-resident tables built once at load. Rank r's partial is Σ_{j owned by r} w·E_g(x); every slot is owned by exactly one rank, so the combine is a plain sum.

ExpertSubset slices the expert axis at load and refuses by name on formats whose leading dimension cannot be sliced, rather than quietly putting all 256 experts on every rank — "EP appeared to work and bought nothing" is a capacity bug that looks exactly like a healthy run. A UQFF artifact holds every expert, so QuantMethod::select_experts (implemented for QtipLayer and Qtip2bLayer, V4's serving rung) narrows the stacks after deserialize, and forward refuses until it has.

ep_size now does something. It was deserialized at arc-engine/src/deepseek_v4.rs:48 and never read — the twelfth "wired but never invoked" case in this repo. It is now read from the model config, overridable with ARC_EP_SIZE, and it errors when it disagrees with the communicator's world size instead of silently degrading.

The tests are the teeth

41 new tests, all green, each proved discriminating.

Collective correctness — ring_algebra asserts the all-reduce result equals the true sum, not merely that it ran, and asserts the construction guard agrees with the algebra at every world size 2..16. That disagreement is precisely how the original single-exchange ring bug shipped: it returned self + left neighbour at N≥4 with no error. It lives outside the ring feature so CI actually runs it.

EP=2 / EP=4 reproduce EP=1, including under a permuted placement where local 0 ≠ global 0. Fixture discrimination: every expert computes a different function, so routing to the wrong one cannot hide. Three mutations were applied, run, and reverted:

Break	Caught by
drop the global→local remap	localize_masks_..., ep2_reproduces_ep1_output, ep2_..._permuted_placement
never mask unowned slots	those three + masks_partition_the_routing_slots_across_ranks, ep4_reproduces_ep1_output
combine keeps unmasked weights	the three EP-equivalence tests

ep4 survived the first mutation — with one expert per rank the local index is always 0, so the remap is right by coincidence. Recorded rather than hidden.

A correction to wave44-BV §3.2

The note's per-N imbalance spread (+4.8/4.6/4.2% at B=128) does not follow from its own definition. Above E/N coverage the N cancels: (E/N)/(distinct/N) = E/distinct. The real bound is a single +5.05% at B=128 (+28.1% at B=64, +0.23% at B=256) — slightly worse than claimed, with the conclusion unchanged: skip EPLB in stage 1, instrument for it. The test asserts the corrected values.

What is not done

• NCCL is still not compiled into the production build. With the shipped --features 'cuda flash-attn', Comm is Dummy and sum_all_reduce is Ok(xs.clone()). The ep_size != world_size check is the only thing standing between a mis-built binary and a fluent, wrong model.
• Two wave44-BV §1.5 landmines are live and would confound a measurement: normal.rs:374-380 skips auto device mapping under NCCL, and attention/mod.rs:232 silently forces naive_sdpa — turning NCCL on turns flash-attention off.
• The UQFF slice is post-load, so each card reads the full 74 GB then narrows to ~37 GB.
• cargo check --features cuda has never run on this branch (cudarc's build.rs needs nvcc).

Verification

cargo check --workspace; cargo test -p mistralrs-core -p mistralrs-quant -p arc-engine (383 / 262 / 149, 0 failures); the scoped clippy lane clean. Upstream mistralrs-* files were not mass-reformatted (fork policy, fab114fe3) — incidental rustfmt churn in deepseek4.rs (341 lines) and distributed/layers.rs (854 lines) was reverted, leaving only semantic hunks.

🤖 Generated with Claude Code

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     72        24645        17702         4215         2728
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     74         4600         4597            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  143        14830        12217          797         1816
 Shell                    24         5924         4071         1457          396
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1485         1292           43          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                194        38699            0        29715         8984
 |- BASH                  71         1632         1192          324          116
 |- C                      3           17           17            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  18          708          708            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  65         2048         1713           77          258
 |- TOML                   6          207          164            0           43
 |- YAML                   4           38           33            5            0
 (Total)                            44442         4671        30250         9521
─────────────────────────────────────────────────────────────────────────────────
 Rust                    663       313742       271458        14550        27734
 |- Markdown             478        24544          471        21171         2902
 (Total)                           338286       271929        35721        30636
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1284       461100       335289        76855        48956
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[heydryft]

BLOCKED — triage pass 2026-08-17.

Disposition: BLOCKED, not closed. Stage 0 is real and independently valuable (the QTIP device-ordinal guard is a genuine latent-corruption fix; ep_size going from deserialized-and-never-read to enforced is the 12th "wired but dead" case closed). None of that is in question.

What blocks it: this PR claims EP=2 works. Arc has never executed a forward pass on more than one GPU — not once. 41 green CPU tests establish that the algebra is right (ring all-reduce equals the true sum at every world size 2..16; EP=2/EP=4 reproduce EP=1 under a permuted placement). They establish nothing about NCCL, peer access, or the actual expert-weight slicing on device. Merging this to master would put "expert parallelism works" in the tree on the strength of a simulation.

Exact unblocking condition — all three, on a 2-GPU box:

1. ARC_EP_SIZE=2 on V4 + qtip2b, greedy/t=0, ≥6 prompts: output is TOKEN_IDENTICAL to the same build at ARC_EP_SIZE=1 on the same box. This is the only check that proves select_experts narrowed the right slice — an EP that silently keeps all 256 experts per rank passes every other check and buys nothing.
2. nvidia-smi per-rank memory shows the expert stacks actually halved. Name the two numbers.
3. The into_mapper peer-access WARN either does not fire, or fires and is recorded — cross-GPU staged through host RAM is a real cost and it should be on record which one happened.

This PR ships no script. That is part of the blocker: add scripts/validate_ep2.sh doing the three checks above with a machine-readable verdict, so the GPU run is one command and cannot be fake-passed. Rebase needed regardless — #87 landed and also touches deepseek4.rs.

Per D15 I do not call runcrate; Jish owns hardware.

[heydryft]

Two updates from the PR sweep.

The Typos gate is fixed (unparseable → unparsable, deepseek4.rs:366) and this branch was separately dirty against master — #101 and #97 had landed under it. Resolved the moe/experts.rs conflict additively: master's expert-backend comment and this PR's EP-localize block are independent, and the topk_weights shadow is preserved so the match &self.backend below still reads the localized weights. cargo check -p mistralrs-core clean.

The body says cargo check --features cuda has never run on this branch. That is no longer true — nvcc compile (sm_90) has now passed, with nvcc compile (sm_80) and cargo check (cuda, workspace) still queued. Worth updating that line, since it's the kind of caveat that outlives its accuracy. Zero real failures across the 15 non-comment checks so far.

To be explicit about what that does and does not buy: it proves the cuda-gated arms compile. EP=2 remains entirely unmeasured — no forward pass has run on more than one GPU, and it cannot be tested on the current single-H200 box, so the honest framing in the header stays exactly as written. Merging this is merging inert, default-off code (default_ep_size = 1), not a supported feature.

One thing I checked while resolving and want on the record rather than in a review queue: report_peer_access is now called unconditionally from DeviceMapSetting::into_mapper, so every multi-GPU load calls cuCtxEnablePeerAccess. It early-returns below 2 CUDA devices and is non-fatal by construction, so it is inert on the single-GPU target — but it is a behavior change on a path nothing has exercised.