FluxServe is a lighweight and high-performance serving engine for diffusion langauge models. It is designed and implemented to deliver low-latency and high-throughput inference for autoregressive (AR) diffusion models across different setups, ranging from single GPU batched inference to multi-GPU distributed serving.
Its core features include:
- Native Block-Causal Attention: Provides efficient attention runtime with a block-casual attention mechanism suitable for AR diffusion in real-world scenarios, including varlen prefill and varlen block-deocde with CUDA graph support.
- Dynamic Request Scheduler: Provides scheduler with low-overhead C++ control plane and Python execution plane with fine-grained block-level request management suitable for block diffusion models.
- Open Model Support: Provides native support for a wide range of open diffusion language models, such as LLaDA2.X and Diffusion-Gemma.
- Efficient Multi-GPU Serving: Provides tensor paralllel (TP), data parallel (DP) and expert parallel (EP) support for large-scale models.
We learned the system design and reused code from the following projects: vllm, SGLang, TokenSpeed, dInfer, FlashInfer, and Flash-Attention.