feat(sched): bound the uninterruptible prefill step — 41.9 s to 23.2 s TTFT p50, and 4 users to 101 at K=128 [MEASURED]

[heydryft]

Half the machine's time is decode stalled behind a prompt, and a user experiences
it as the server hanging. This is the first, cheap half of the fix, measured.

The mechanism, confirmed

A prompt step cannot yield, at three levels:

1. scheduler/default_scheduler.rs::schedule runs exactly one bucket per
   engine iteration. A fresh prompt buckets at cache_bucket_len() == len()-1
   (no cache yet), so when the prompt bucket wins, completion is empty and
   no decode happens in that iteration at all.
2. engine/mod.rs prompt branch calls pipeline.step(.., is_prompt = true, ..)
   once, holding the pipeline mutex for the whole prefill, then moves every
   scheduled sequence straight from RunningPrompt to RunningCompletion.
3. There is no partially-prefilled state. Sequence::set_token_offset
   (sequence.rs:639) has zero callers, so token_offset() is always 0 and
   make_prompt_chunk's chunk_offset_toks / is_first_prompt_chunk are dead
   plumbing.

That plumbing is a vestige of the intra-step chunk loop upstream deleted in
863f33de7 (mistral.rs #1591, "Remove prompt chunksize apis"). It would not
have helped: every chunk ran inside ONE pipeline.step, so the scheduler never
regained control between them. Nothing in this tree, present or historical,
splits a prompt across engine iterations.

What this PR does

ARC_PREFILL_MAX_SEQS caps how many waiting sequences may be admitted to
prefill in one iteration. Unset or 0 reproduces the previous admission
expression for expression, in both arms.

This is the request-level half of chunked prefill, and it is the shape
SGLang runs by default — prefill and decode batches alternate, with mixing
behind an opt-in --enable-mixed-chunk. vLLM V1 goes further (decode first,
prefill fills the remaining max_num_batched_tokens, 8192 on an H200 server),
which needs machinery we do not have yet — see below.

Measured (H200, qtip2b, MTP d=3, PR #113's profiler, WARMUP_S=0 so the opening burst is inside the window)

K=32, 32 × 256-word prompts

	cap off	ARC_PREFILL_MAX_SEQS=4
TTFT p50	41.93 s	23.22 s (−45%)
TTFT p90	42.18 s	39.59 s (−6%)
TTFT max	42.19 s	43.80 s (+3.8%)
TTFT min	2.77 s	1.49 s
aggregate over the window	36.14 tok/s	47.95 tok/s
prompt steps / mean duration	4 / 21,680 ms	10 / 4,381 ms
decode steps / mean duration	56 / 709 ms	50 / 702 ms

K=128 — cap off, 4 tokens reached a client in 140 s, and only 4 of 128
users ever saw a first token:

	cap off	ARC_PREFILL_MAX_SEQS=8
tokens delivered in 140 s	4	101 (25×)
users with a first token	4 / 128	101 / 128
prompt steps / mean duration	3 / 58,825 ms	18 / 9,517 ms
decode steps / mean duration	57 / 2,335 ms	42 / 2,320 ms
total prefill wall	176.5 s	171.3 s (same work)

Engagement is structural, not inferred: prompt steps 4→10 and 3→18, mean
prompt-step duration −80% and −84%, with decode step duration unchanged
(709→702 ms, 2,335→2,320 ms) — the cap chops the same prefill work into more,
smaller uninterruptible steps and costs the decode path nothing.

All four profiles: selftest PASS, unresolved_device_spans: 0,
violations: [], device÷wall 92–96%.

⚠️ The K=32 aggregate windows differ in length (130.1 s vs 80.1 s) because
the profile-write stop fires at different times. The shorter window contains
proportionally more prefill, so the comparison is biased against the
capped arm and it still wins — but this is an end-to-end number over a finite
window, not a steady-state throughput claim.

What this does NOT fix, and what would

The cap bounds when prefill blocks; it does not make prefill cheaper and it
does not let decode ride along on a prefill's GEMMs. At K=128 the capped arm
still delivers only ~1 token per user in 140 s, because prefill itself runs at
~281–405 tok/s — 12,160 prompt tokens in 43.2 s. That is the real scandal, and
it is consistent with attention being 57% of the marginal decode cost (PR #113).

True token-level chunked prefill needs, in order:

• a per-sequence prefill cursor — sequence.rs (set_token_offset already
  exists, unused) and inputs_processor.rs::get_prompt_input (slice toks to
  [offset, offset+C); chunk_offset_toks already threads into positions,
  RoPE offsets, seqlens_k and slot mappings);
• suppression of sampling on non-final chunks — pipeline/mod.rs, the
  DefaultInstructions arm;
• and for mixed prefill+decode in one forward, the blocking piece:
  ragged cu_seqlens_q decoupled from per-row seqused_k, plus bottom-right
  causal alignment (FlashAttention ≥2.1 semantics — a top-left-aligned mask
  silently produces wrong results for every chunk after the first). That is
  PR feat(v4): per-row q0 in dsv4_attention — the ragged-mask blocker #92 named #100's per-row q0 territory in models/dsv4_attention.rs, on top of
  PR feat(scheduler): admit ragged cohorts — the admission layer #103 named, and a general throughput bug #104's ragged cache lengths.

Chunk-size guidance from Sarathi-Serve worth carrying into that work: 512 is the
practical floor (≤25% extra prefill runtime; prefill throughput saturates around
length 512), 2048 is near-free, and chunk size must be a multiple of the GEMM
tile — 257 costs 32% more than 256.

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     72        24328        17592         4018         2718
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     74         4600         4597            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  143        14830        12217          797         1816
 Shell                    23         5756         3964         1412          380
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1485         1292           43          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                190        37932            0        29111         8821
 |- BASH                  71         1623         1192          315          116
 |- C                      2           12           12            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  18          708          708            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  65         2048         1713           77          258
 |- TOML                   6          207          164            0           43
 |- YAML                   4           38           33            5            0
 (Total)                            43661         4666        29637         9358
─────────────────────────────────────────────────────────────────────────────────
 Rust                    662       309221       267775        14013        27433
 |- Markdown             477        23236          471        20016         2749
 (Total)                           332457       268246        34029        30182
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1278       454005       331384        74308        48313
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[heydryft]

Closing as superseded. Naming what carries each piece, because a close without that is a discard.

The cap — ARC_PREFILL_MAX_SEQS — is reproduced in #123, scheduler/default_scheduler.rs::prefill_admission_cap(), same env var, same unset or 0 reproduces the previous admission exactly semantics. #123 additionally pairs it with the floor (ARC_PREFILL_FLOOR_STEPS) and with the engagement logging this branch does not have — the logging that exists because five published cells turned out to have been served by a scheduler file that never executed.

Both profiler commits are already on master, byte-identical. git diff origin/master origin/perf/prefill-admission-cap -- arc-profiler/src/device.rs is empty — 66a61259d ("the device and sync channels were dead on every default CUDA device", the NULL-CUstream-is-the-legacy-default-stream fix) landed independently. 30cdb6c3b (V4 decode split into attention and MoE) likewise.

The remaining unique content is other PRs, still open on their own. This branch is ab42c4508 (#104, ragged cohort admission) plus 583fda6ac (#113, the profiler port onto that branch). Both PRs are open. Nothing in this branch is lost by closing it.

State that made this unrecoverable as-is anyway: mergeable: DIRTY, 67 commits behind master, and 1 check — the comment bot. Zero CI lanes ever ran here.

Reopen if any of the above is wrong; the branch is not deleted.