Repository navigation
perf: Prefill-path kernels for GDN hybrid deciders (PPLX/Clef) — align with vLLM/SGLang #173
Description
Goal
Optimize the prefill path for GDN hybrid deciders (classification C1 has no decode). Owner direction: mirror proven vLLM / SGLang kernel choices for the same model family; do not replace our serving runtime wholesale.
Assignee: @psv666
Why prefill
Public231 C1 on L20X is dominated by backbone forward (prefill-like). PPLX profile @ 1ba07d15 (in-process ~136 tok):
- total ~89 ms; forward CUDA ~87.7 ms (~98%)
- missing
causal_conv1d→ HF fallback to generic depthwiseF.conv1d - HF-SDPA + GDN; no CUDA graphs; short-seq floor ~88–91 ms
Baselines (L20X, c=1, 231/231, no-thinking where needed)
| PR | Model | SHA | p50 | p95 |
|---|---|---|---|---|
| #166 | PPLX-27B | 1ba07d15 |
98.777 ms | 341.202 ms |
| #165 | Clef | 9030065 |
247.518 ms | 413.025 ms |
| #164 | StartLux 4B (contrast) | 5bb25be |
6.785 ms | 51.432 ms |
Scope (prefill first)
- Required:
causal_conv1don sm_89 — no “falling back to reference PyTorch” warning on ready; worker may fail closed if unbound. - GDN prefill kernels: follow vLLM/SGLang (Triton default / FlashInfer GDN prefill where applicable on our arch).
- Full-attn prefill: FlashInfer / SDPA as appropriate; document what actually binds on L20X.
- Optional later: short-seq CUDA graphs — only if profile still shows launch overhead after (1)–(3).
Out of scope for this issue: decode-only kernels; merging draft PRs without separate owner yes.
Evidence rule
Profile → tip on the draft PR branch → L20X Public231 C1 remeasure → keep only verified p50/p95 gains. Tips with 0 samples are not wins.
Related tips (unverified until remeasured): #166 7233620 (causal_conv1d gate); #165 033685b7 (FLA + causal_conv1d).
Related
- Drafts: Add StartLux Instant 4B native worker (#160) #164 / Add pinned Clef-flash and Clef text reference worker #165 / pplx-decider: reference worker and Public231 C1 procedure #166
- Skills draft: skills: add Public231 C1, remeasure gate, and new-model integration #172 (
optimize-remeasure-gate,public231-c1) - Cross-lab RSI track: InferRSI#141
Metadata
Metadata
Assignees
Labels
No labels
Activity
hsliuustc0106 commentedon Oct 25, 2026
Kernel mapping from Clef CloudAgent (no new tip commit; tip still
033685b7, 0 L20X samples)Serve path: eager text prefill (
use_cachefalse, c=1) on L20X sm_89. Decode schedulers / CUDA graphs from vLLM/SGLang do not fit this C1 path.flash-linear-attention==0.5.2+causal-conv1d>=1.5, import beforemodeling_qwen3_5. Do not vendor vLLM/SGLang kernels.chunk_gated_delta_rule). FlashInfer GDN prefill is SM90+ (or SM100 constraints); CuteDSL SM100-only — not for sm_89.causal_conv1d_updatebranch./healthlinear_attention=fla+causal_conv1d. After remeasure, refuse startup if stilltorch_chunk/F.conv1d.torch.compile.causal-conv1dwheel that loads on 8.9; GDN prefill is Triton JIT (not gencode-selected).mamba_ssm_dtypefloat32.Baseline still
9030065p50 247.518 / p95 413.025. Remeasure tip before counting any win.Cross-ref: InferRSI#141.
hsliuustc0106 commentedon Oct 25, 2026
@psv666 Owner (system1): go — wire prefill ops first (
causal_conv1d+ FLA chunk on sm_89 / L20X). Mapping note above. Keep draft PRs; verified Public231 C1 deltas only.hsliuustc0106 commentedon Oct 25, 2026
Owner correction: target GPU is H200 (sm_90 / Hopper), not L20X sm_89.
@psv666 Please retarget prefill ops for Hopper:
causal_conv1d/ FLA still required; gencode/cubin for sm_90 as needed.Evidence still: Public231 C1 on H200, keep only verified p50/p95 deltas.
hsliuustc0106 commentedon Oct 25, 2026
Progress: #165 tip
4e430e18(H200 / sm_90)CloudAgent wired Clef prefill ops on draft #165:
causal-conv1d≥1.5,flash-linear-attention==0.5.2,flashinfer-python==0.6.16.post3before Transformers builds Qwen3.5chunk_gated_delta_rule(use_cp=false); conv viacausal_conv1d_fn; FLA forFusedRMSNormGated+ Q/Kl2norm_fwd/healthlinear_attentionmust beflashinfer+fla+causal_conv1d; refuse torch_chunk / F.conv1d / non-9.0torch.compileNo H200 Public231 C1 yet — prior L20X numbers at
9030065are not same-device baseline. @psv666 next: measure on H200.hsliuustc0106 commentedon Oct 25, 2026
#166 tip
0b20933(ops in, device target wrong)PPLX draft now has causal-conv1d 1.7.0 + FLA 0.5.2 Triton chunk;
/health→fla-chunk. 0 samples.Still oriented to sm_89 cubin. Owner target is H200 sm_90 — CloudAgent queued to retarget like #165
4e430e18(FlashInfer GDN prefill + FLA + causal_conv1d). @psv666 FYI.hsliuustc0106 commentedon Oct 25, 2026
#165 tip now
594528a9: L20X reported 8.9 can start (FLA+causal_conv1d); FlashInfer only if major 9/10/12. Remeasure in progress vs9030065.hsliuustc0106 commentedon Oct 25, 2026
Verified Public231 C1 (L20X-labeled target, 231/231)
594528a99030065247.518/413.025 → −33% / −48%fla+causal_conv1df968e5761ba07d1598.777/341.202 → −0.7% / −11%fla-chunkKeep tips; drafts not merged. @psv666 Clef op path is a clear win; PPLX mainly p95.