fix(ArcKernels): FP8 threshold was 512 because 512 was the only point measured; cuBLASLt branch never flattened to 2-D

[heydryft]

Measured on an H200 against V4-Flash, on top of the MoE crossover fix (#197).

1. The threshold was 512 because 512 was the only point measured

arc_fp8_cublas_min_m defaulted to 512, and the doc said why outright: "the
default is set to that measured point rather than to an interpolated crossover."
fp8_gemv_warp owns M <= 4, so M = 5..511 fell through to fp8_matmul_tiled —
the kernel whose own comment says there is "no tensor-core instruction anywhere in
it." A profile of the shipped default caught it at 7,525 launches for 17.6% of GPU
time in one decode window.

Default is now 5, the first row fp8_gemv_warp does not own. M = 5..7 is
unmeasured. Fast by default; ARC_FP8_CUBLAS_MIN_M remains the kill switch.

2. The cuBLASLt branch never flattened rank-3 to 2-D — and that invalidated the A/B

The native branch flattens [B,T,hidden] to [B*T,hidden]; the dequantize+cuBLASLt
branch did not. V4's activation is rank-3 and bias is None on every V4 linear, so
UnquantLinear::forward took w.broadcast_left(B) into cublaslt.batch_matmul with
stride_b = 0. At decode T=1 that is a B-way batched GEMM with m=1 — a batched
GEMV wearing a GEMM's name, the one shape cuBLASLt has no advantage in. The documented
27× was prefill, where [1,512,hidden] collapses to batch=1 and the call is a real GEMM.

Worth +10.4% on its own at B=256 (156.85 → 173.15).

3. Measured

Post-flatten, one binary, one toggle (ARC_FP8_CUBLAS_MIN_M=999999 = kill switch):

B	cuBLASLt (min_m=5)	tiled (kill switch)	ratio
1	34.98 CLEAN	34.86 CLEAN	1.00×
64	101.17 dirty	88.04 CLEAN	1.15×
256	173.15 dirty	130.75 CLEAN	1.32×

vs the old 512 default at B=256: 139.33 → 173.15 = 1.24×.

b=1 is unchanged (34.98 vs 34.86). Both arms take fp8_gemv_warp at M=1, so the
floor is untouched by this constant — which is the check that mattered.

⚠️ What is NOT established

The clean confirmation for the constant did not survive. A re-run at NTOK=160
(which previously gave 96–97% fill) completed both arms — 8 rows recorded — and the box
went down (port 22 closed, 100% ICMP loss) before the log could be read. It did not
come back over ~10 minutes of retries.

So the shipped 5 currently rests on the dirty B=64/B=256 rows above plus the
pre-flatten clean rows, and those pre-flatten rows compared against the degenerate
batched-GEVM shape described in §2. Every arm measured, clean or dirty, favours
cuBLASLt above the floor, and none is a clean post-flatten row. Re-run
arc-tools/batch_sweep.py at NTOK=160 before quoting a ratio from this.

4. Two follow-ups this work turned up

The real fix is a descriptor, not a kernel. CUDA 13.0's cuBLASLt already exposes
exactly what blockwise FP8 needs:

936:  CUBLASLT_MATMUL_MATRIX_SCALE_BLK128x128_32F = 5
1640: CUBLASLT_MATMUL_DESC_A_SCALE_MODE = 31
1645: CUBLASLT_MATMUL_DESC_B_SCALE_MODE = 32

BLK128x128_32F is precisely the [N/128, K/128] grid weight_scale_inv already
carries. That means native block-scaled FP8 on tensor cores with no dequantize at
all — one descriptor attribute plus the existing scale tensor, not a kernel rewrite.

512 was never a kernel property. It is the amortisation point of dequantize_w(),
which runs uncached on every forward (~12.7 GB/step). Cache that and the correct
threshold is 5 regardless.

5. A recurring failure, recorded in the doc

This is the second gate found this way. qtip::gather_policy's tile-fill predicate
required 6n/256 >= 16, i.e. n >= 683, keeping the amortising grouped GEMM unreachable
in every decode step — worth 1.45× at B=128 when removed. Both gates were derived from a
single working point and neither was ever swept. The doc now says: if you are about to
set a dispatch threshold from one measurement, sweep it first and write down which rows
were clean.

The last commit (b5de0f1d4, bank-conflict padding +4 → +1, bit-identical by
construction) is labelled UNVERIFIED — never compiled, for the same box outage.

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     78        27867        19381         5568         2918
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     74         4600         4597            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  147        15371        12677          824         1870
 Shell                    42        10258         6849         2734          675
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1498         1294           54          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                204        45330            0        35183        10147
 |- BASH                  72         1655         1203          331          121
 |- C                      3           17           17            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  18          708          708            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  66         2051         1716           77          258
 |- TOML                   6          207          164            0           43
 |- YAML                   5           41           36            5            0
 (Total)                            51102         4688        35725        10689
─────────────────────────────────────────────────────────────────────────────────
 Rust                    678       335044       288629        17172        29243
 |- Markdown             497        30361          471        26183         3707
 (Total)                           365405       289100        43355        32950
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1337       502989       357396        92632        52961
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[heydryft]

Superseded by #215, which is the same work cherry-picked onto origin/master
(6ffdac7ae) instead of release/openrouter-ready.

This PR was red for exactly one reason — the Base branch lane refuses any base
that is not master. Every other lane here was green, so nothing about the code
was in question; the base was.

Cherry-picked rather than retargeted, deliberately: this branch carries
145bbbc0a (docs/engineering/OPENROUTER_READY.md), which exists only because
gh pr create refused an empty branch, and is the sole source of a fabricated
"27.2%" figure that has already contaminated one merged PR. #215 does not
contain that file.

What changed in the move:

• b5de0f1d4 (bank conflict) lands unchanged, UNVERIFIED label intact.
• 52a8750f5 lands as documentation only. The 512 -> 5 default does not: perf(ArcKernels): blockwise-FP8 GEMM on tensor cores, scale promoted per 128-K #200
  (9ee459191) merged a tensor-core blockwise-FP8 GEMM at 07:04 on 08-21, six
  hours after this sweep was taken, and it now owns M > 4 by default. Lowering
  the threshold to 5 would make that kernel unreachable at every batch size that
  matters. The sweep table and the M=1 -> 0.72x GEMV floor are preserved in the
  doc so the three-arm re-run starts from data.
• 445b92063 (rank-3 flatten) is dropped as already-fixed: The 794 ms decode step, itemised: lm_head's 244x, the duplicate K/V, and the 91 ms named #202 (611d236c2)
  landed the general form in UnquantLinear::forward
  (mistralrs-quant/src/unquantized/mod.rs:155-161), and its comment names this
  commit and says it subsumes it. Master's version is gated !is_cpu(); this
  one is not, so applying it now would reintroduce the CPU flatten for
  blockwise-FP8 layers only.

Closing so the fixes are not orphaned a second time.