feat(ArcSched/ArcKV): ragged dense decode — removes the decode-side length limiter [MEASURED, gated by prompt admission]

[heydryft]

Removes the decode-side length limiter on the dense scheduler. Merged with current master (clean; 466/466 green on the merged tree).

⚠️ Read this before the numbers

This PR is not claiming the flagship 2.84×, and the headline number should not come from here. What this removes is one of two limiters in series. The other — prompt admission — is a separate fix by a separate chain and is not yet combined with this one. Every number below was measured with that second limiter still in front of this one, so they are a floor, not the result. The number worth quoting comes from the combined run.

What it does

The dense scheduler ran exactly one cache-length bucket per step, so a length-diverse cohort decoded one sequence at a time. Measured effect on the scheduler itself, from the engine's own log at B=128 spread:

baseline:  16 running, 112 waiting   (throughout)
fix:       33 running,  95 waiting → 47 running, 48 waiting

It does not reach 128 because prompt admission gates it — see above.

Three pieces already existed and were unreachable

front_align_batch, make_left_padded_causal_mask + RaggedKvLens, and supports_per_sequence_len. Only MTP called any of it, and it discarded the result.

🔑 The code named the wrong blocker. MtpSpeculativePipeline::target_masks_ragged_batches is a hardcoded Err reading "no model in this tree threads a ragged-batch mask into its forward yet". Threading it through all forty-odd models would not have worked: Sdpa::run_attention's flash branch calls flash_attn(&q,&k,&v,..), which takes no mask argument, so the mask was discarded at the dispatcher. That is fixed first (mask_must_be_applied_as_bias, scoped to seq_len == 1 so uniform decode and every prefill are untouched).

Then: a thread-local carrying each row's dead prefix, set by clone_in_cache and read by CausalMasker — which is what gets the mask to every model without a new argument anywhere; clone_in_cache front-aligns instead of refusing; clone_out_cache hands each row back its own length; the scheduler pins the length key to 0 for decode only (prompts keep the exact-length key — prefill is right-padded and its cu_seqlens come from those padded widths, a different mechanism).

The channel carries the prefix, not the live lengths: clone_in_cache runs only when membership changes while the cache grows every step, so a stored live is stale on the next token, whereas lead_pad stays true and live = past_kv_len - lead_pad is recovered from the cache.

Measured

H200, Qwen2.5-0.5B, --paged-attn off, --prefix-cache-n 0, exclusive card. BASE=120f34875 / FIX proved with git merge-base --is-ancestor and a printed commit count — not a relative ref (see Method notes).

result	value	caveat
B=8 spread	+17.4% (227.0 → 266.51)	ladder measurement; survives the order-effect caveat by margin
B=32 uniform, in isolation, 5 samples/arm	+28.4% (936.51 → 1202.30) and 12× more stable (sd 123.66 → 10.62)	isolation, not ladder — see below
deferred strip	675.24 → 764.08 at B=128 spread	confirms the per-step strip was the cost

Unresolved, named: +6.1% at B=32 spread and +5.1% at B=128 uniform. Both are ladder measurements below the readability bar and are not claimed. See Method notes.

The B=32 uniform story, because it inverted

The ladder read this cell as −5.1%, then −9.7% — a regression. It is not one.

First, inertness was proven, not assumed (a_uniform_batch_leaves_every_ragged_mechanism_untouched): on a uniform cohort clone_in_cache publishes no dead prefix, leaves zero deferred-strip entries, and the masker still returns None at tgt_len == 1. Any non-zero there would have located the regression. This is the test a reviewer should read to trust that the change is genuinely inert where it should be. The other candidate — the scheduler pin colliding with prompt buckets — is excluded by arithmetic: a fresh prompt's cache_bucket_len is len()-1 ≈ 274 and cannot equal the pinned 0.

Then five samples per arm, that cell alone, one server per arm:

baseline  1157.50 / 887.60 / 885.79 / 874.68 / 876.96   mean  936.51  sd 123.66  spread 30.2%
fix       1186.59 / 1200.91 / 1212.11 / 1199.62 / 1212.26 mean 1202.30  sd  10.62  spread  2.1%

The baseline carries the variance: it runs fast once, then collapses to ~880 and stays. Cause: the harness drains with a 40-token probe before each rep, while the cell sends 32 × ~275-token prompts — so the "uniform" control was ragged, and was measuring exactly the thing under test. The baseline serialises it; this change merges it.

Correctness

• Uniform-batch identity control, teeth verified at 8 distinct completions: baseline vs fix at B=8-uniform is 8/8 identical; each arm diverges from its own B=1 by exactly 4/8 — the fix adds none of its own.
• Every measured cell returned exactly B × 64 tokens.
• 466/466 green in debug and release on the merged tree.

The merge-blocker that was closed, and how

Deferring the strip left a window: between clone_out_cache and the next clone_in_cache a sequence still carries its zero-filled dead prefix and reports a length that includes it. For a sequence that has just finished there is no next clone_in_cache — so PrefixCacheManagerV2::add_sequence would store the padding and hand it to every future request matching that prefix. A wrong answer that outlives the request that produced it, and one --prefix-cache-n 0 hides completely.

Closed at the reader, not the writer. add_sequence strips the pending prefix before seq.normal_cache().to_vec() — one sequence at a read, instead of O(B × layers) every token at the write. That keeps the deferral (worth 675 → 764) and pays off its only unsafe consumer precisely. On failure it refuses to store rather than storing padding.

🔑 Why the writer was not available: clone_out_cache runs before sampling (pipeline/mod.rs:976), so at that point no sequence knows whether it has finished — "strip only the finishing ones at the write" is not expressible there.

Seen red by mutation — with the strip removed:

assertion `left == right` failed: the dead prefix must be dropped before the cache is read …
  left: 64
 right: 60

i.e. four zero-filled K/V rows cached as content and keyed as if real.

A guard's contract narrowed, deliberately

clone_in_cache no longer refuses every length-mismatched batch — only those it cannot align. wave51-CB's guarantee (refuse in release, return rather than panic, never reach slice_set) is preserved and now covered by a Rotating-slot fixture, with ensure_uniform_batch_cache_lens kept as a hard postcondition.

Review of that narrowing found a real hazard: front_align_batch pads in place, so it could return Err having already rewritten the rows ahead of the failure, where the blanket refusal it replaced mutated nothing. front_align_would_succeed restores the property — alignment is attempted only when known to complete. Pinned by a_refusal_does_not_leave_a_half_aligned_batch.

Mutation results, both run:

• remove the postcondition ⇒ three refusal tests go red. It discriminates.
• replace batch_can_be_ragged with true ⇒ suite stayed green. The alignment decision is double-covered by front_align_would_succeed; the capability — what the scheduler reads — was covered by nothing. Now pinned by the_capability_predicate_is_what_the_scheduler_reads.

A defect this found in itself, worth a reviewer's attention

The first measured run returned 433/512 tokens at B=8 spread with ZERO reported errors and every uniform cell exact. CausalMasker has two entry points and the channel was wired into one; Qwen2 reaches the other, so every ragged decode batch attended its zero-filled dead prefix. Only the completeness guard caught it — the harness exited 0.

🔑 The family: a new channel wired into one of two dispatch paths, where which one you hit depends on model config. When adding a channel, enumerate every entry point and every early return, and make sure a fixture reaches each. The suite was green throughout; it could not reach the second path because no fixture had that config.

Method notes a reviewer should not have to reconstruct

• A relative named ref is still a moving ref. The baseline was pinned as FIX~3 — correct at three commits, silently wrong at four, where it resolved to this branch's own first commit and would have put half the change under test inside the "baseline". Both SHAs are now literals and the build script proves the pair.
• A cross-server null does not cover within-sweep order effects. Three baseline-vs-baseline server starts gave ±2% on every cell, yet the same binary reads 1075–1130 for a cell inside the ladder and 1186–1212 for that cell in isolation, because the two arms arrive there in different states. ⇒ every ladder delta under ~10% is unresolved until re-measured in isolation. That is why the two small cells above are named rather than claimed.
• A guard that refuses itself is working. The identity control's first run reported CANNOT-ANSWER: fewer than 4 distinct B=1 completions — a 0.5B model summarises eight numbered copies of the same filler identically, so "8/8 identical" would have been vacuous.

Unrelated defect found on the way, not fixed here

models/qwen2.rs's Config deserialises sliding_window: Option<usize> and never reads use_sliding_window, while qwen3.rs:33, qwen3_moe.rs:33 and embedding_models/qwen3_embedding.rs:28 all gate on it. ⇒ every Qwen2-architecture model runs with sliding-window attention ON when its config says OFF. Verified on Qwen2.5-0.5B-Instruct (sliding_window = 32768, use_sliding_window = false, engine takes the SWA mask path). Mostly inert while the window ≥ served context, which is why it has not surfaced — but it will bite on any Qwen2 model whose window is smaller than its context. Not fixed here because it is not this change's territory.

Full record: memory/mission/wave64-CP-dense-ragged-decode.md.

🤖 Generated with Claude Code

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     72        24328        17592         4018         2718
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     74         4600         4597            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  143        14830        12217          797         1816
 Shell                    23         5756         3964         1412          380
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1485         1292           43          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                190        37974            0        29143         8831
 |- BASH                  71         1632         1192          324          116
 |- C                      2           12           12            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  18          708          708            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  65         2048         1713           77          258
 |- TOML                   6          207          164            0           43
 |- YAML                   4           38           33            5            0
 (Total)                            43712         4666        29678         9368
─────────────────────────────────────────────────────────────────────────────────
 Rust                    662       310844       269120        14195        27529
 |- Markdown             477        23762          471        20479         2812
 (Total)                           334606       269591        34674        30341
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1278       456205       332729        74994        48482
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[heydryft]

Your 47 running, 48 waiting reproduces, and the fix for it is measured — PR #123.

Two things you should have from this chain.

1. It is starvation, not a throttle, and my earlier framing (relayed to you)
was wrong. select_running_bucket takes the highest summed priority, so 47
sequences at length 500 outscore one fresh prompt bucket ~47× before urgency
counts at all. ARC_PREFILL_MAX_SEQS (PR #119) is a throttle on one
uninterruptible step and applying it to a starved cohort makes things worse.
The missing half is a floor: ARC_PREFILL_FLOOR_STEPS=N forces the shortest
waiting prompt bucket after N consecutive passes.

Measured on your configuration — Qwen2.5-0.5B, --paged-attn off,
--prefix-cache-n 0, --max-seqs 128, B=128 spread, fresh server per cell:

arm	tok/s	run_mean	run_max	TTFT p50
baseline	857.84	52.0	65	4.83 s
floor=4	565.22 (−34%)	94.2	128	1.58 s (−67%)
floor=16	792.76 (−7.6%)	62.6	126	3.99 s

Baseline repeated across passes: 867.98 / 857.84, 1.2% spread. Your ceiling
is the run_max 65 row; the floor lifts it to 128 and cuts TTFT p50 by 67%, and
full admission costs 34% of throughput on this workload because it is
prefill-bound. Both numbers, because only one of them is good.

2. The trap that cost me a published result, so it does not cost you one.
My first pass omitted --paged-attn off. Every cell ran with PagedAttention
live, so default_scheduler.rs — your decode pin, my cap, my floor — never
executed, and I published "the ceiling did not reproduce" off a file that
never ran. My completeness guard checked errors, server-alive, tokens>0 and
admission telemetry; none of those distinguishes did nothing from was not on
the code path.

Both guards are now in the harness and are worth copying: a path gate that
refuses a cell whose log mentions a PagedAttention KV cache, and positive
engagement logging — the floor and cap each log once when they first act, so
an arm claiming a mechanism without its line is failed rather than believed.

3. A free observable you were told you didn't have. serve already emits
<N> running, <M> waiting — throughput_logging_enabled: !self.interactive_mode
in mistralrs_for_server_builder.rs:1013. The header in
arc-tools/ragged_admission_curve.sh says it is unreachable from serve; that
is stale, and it is the most direct admission measurement there is.

Also inherited gratefully and used unchanged: no drain probe (fresh server per
cell), no ladder, and your build-fingerprint discipline (SHA + binary md5 +
mtime stamped into the results file).