MiniCPM-o 4.5 · Single GPU

A dedicated streaming speech engine for MiniCPM-o 4.5 on one GPU. The Thinker (Qwen3-8B) and the Talker (20-layer Llama TTS) share one process, a paged FlashInfer KV runtime and CUDA graphs. The Talker is conditioned in place on the Thinker's hidden states while the reply is still being generated. Token2Wav (flow-matching DiT and HiFT) runs in a worker process beside it under CUDA MPS. The server speaks the OpenAI-compatible /v1/chat/completions API with streamed text and 24 kHz audio.

Architecture

Module details and configurations · Implemented optimizations and experiment results · Why the target is not met yet · Test results

Quickstart

Requires Linux, Python 3.12, uv, a CUDA 13 driver and one GPU. The measured profiles use about 95 GB of GPU memory (measured on an L20X with 143 GB). The lock pins torch 2.13 (cu130), transformers 5.14 and flashinfer 0.7. minicpmo-utils provides the reference Token2Wav modules. It is installed without its dependencies, which the lock already lists.

git clone https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine.git
cd minicpm-o-4.5-single-gpu-engine

uv venv --python 3.12 .venv-engine
uv pip install --python .venv-engine/bin/python --no-deps \
  --index-strategy unsafe-best-match --extra-index-url https://download.pytorch.org/whl/cu130 \
  -r engine/env/engine.lock

hf download openbmb/MiniCPM-o-4_5 --revision 503e754207c94da6bb26850b4469f367c9ea3582

CUDA_VISIBLE_DEVICES=0 ENGINE_MPS=1 scripts/serve_engine.sh track_s_fast_r16 18080

scripts/serve_engine.sh <config> <port> loads configs/engine/<config>.json. With ENGINE_MPS=1 it starts a private CUDA MPS daemon for the engine and the worker, addresses the GPU by UUID, and removes the daemon when the server exits. The checkpoint is found in the Hugging Face cache (HF_HUB_CACHE) or at MODEL_PATH. Every machine-specific location is listed in scripts/env.sh and can be set in an untracked scripts/env.local.sh. The server is ready after model loading, CUDA graph capture and the feature-process warmup; the first start also compiles the Token2Wav kernels.

From another terminal:

curl --fail http://127.0.0.1:18080/health
REF="${HF_HUB_CACHE:-$HOME/.cache/huggingface/hub}"/models--openbmb--MiniCPM-o-4_5/snapshots/503e754207c94da6bb26850b4469f367c9ea3582/assets/HT_ref_audio.wav
printf '{"model": "openbmb/MiniCPM-o-4_5", "stream": true, "modalities": ["text", "audio"],
 "messages": [{"role": "user", "content": "Please read the following text out loud in English: Hello from MiniCPM-o."}],
 "chat_template_kwargs": {"use_tts_template": true, "enable_thinking": false},
 "max_tokens": 256, "temperature": 0, "seed": 1, "ref_audio": "data:audio/wav;base64,%s"}' "$(base64 -w0 "$REF")" > body.json
curl --fail-with-body -N http://127.0.0.1:18080/v1/chat/completions \
  -H 'Content-Type: application/json' --data-binary @body.json

Text arrives as {"modality": "text", "choices": [{"delta": {"content": ...}}]} and audio as {"modality": "audio", "audio_offset_samples": o, "num_samples": n, "choices": [{"delta": {"content": <base64 WAV>}}]} (24 kHz, mono, 16-bit PCM). The default Talker sampling is the model card's (temperature 0.8, top-p 0.85, top-k 25, repetition penalty 1.05 over 16 frames); talker_sampling overrides it per request. Image, audio and video inputs use the usual image_url, input_audio/audio_url and video_url content parts. See API and request flow.

Performance

Recorded on one L20X (143 GB, driver 570.133.20) with the same Seed-TTS English voice-cloning requests at each concurrency: 320 evaluation requests per run (640 at C=32), a warmup, a discarded first run, then K=5 measured runs on a freshly started server each. Rows are means over the K runs. Versions: MiniCPM-o 4.5 503e754, vLLM-Omni 2ec5f26, SGLang-Omni 22d4ace.

C	Framework	Config	Throughput (frames/s)	Throughput / SGLang	TTFA p50 (ms)	TTFA p95 (ms)	RTF (mean)
1	minicpm-o-4.5-single-gpu-engine	track_s_fast_r16	406.01	1.46×	125.8	137.6	0.0623
1	vllm-omni	minicpmo_4_5_turn_mps-mps-on	187.52	0.67×	295.3	339.9	0.1354
1	sglang-omni	speech-mm32k	278.72	1.00×	381.2	502.9	0.0931
2	minicpm-o-4.5-single-gpu-engine	track_s_fast_r16	562.93	1.25×	177.3	212.2	0.0906
2	vllm-omni	minicpmo_4_5_turn_mps-mps-on	318.61	0.71×	380.0	465.1	0.1596
2	sglang-omni	speech-mps-on	450.37	1.00×	471.0	637.7	0.1152
4	minicpm-o-4.5-single-gpu-engine	track_s_fast_r16	810.38	1.14×	272.7	326.7	0.1261
4	vllm-omni	not evaluated at this C	—	—	—	—	—
4	sglang-omni	speech-mps-on	709.73	1.00×	597.6	825.6	0.1463
8	minicpm-o-4.5-single-gpu-engine	track_s_fast_r16	1180.65	1.14×	367.2	439.6	0.1734
8	vllm-omni	not evaluated at this C	—	—	—	—	—
8	sglang-omni	speech-mps-on	1033.40	1.00×	853.9	1150.4	0.2001
16	minicpm-o-4.5-single-gpu-engine	track_s_fast_r16	1682.74	1.16×	484.6	597.5	0.2426
16	vllm-omni	minicpmo_4_5_turn_mrv2_h200_mps-mps-on	411.03	0.28×	1057.2	1982.3	1.0148
16	sglang-omni	speech-mps-on	1455.39	1.00×	1179.2	1686.0	0.2829
32	minicpm-o-4.5-single-gpu-engine	track_s_fast_r16	2419.94	1.21×	654.1	804.2	0.3366
32	vllm-omni	minicpmo_4_5_turn_mrv2_h200_mps-mps-on	407.68	0.20×	1041.1	1965.6	2.0344
32	sglang-omni	speech-mps-off	1996.67	1.00×	1613.0	2374.3	0.4137

C is the number of concurrent requests. Throughput counts returned audio as 25 Hz codec frames (25 × delivered seconds / wall time); TTFA is the time to the first audio sample; RTF is each request's time to its last audio sample divided by its audio duration. Baseline configurations were chosen per concurrency on separate tuning requests and then measured on the evaluation requests. Where two SGLang-Omni configurations were measured at one C, the faster one is shown; the target ratios use the configuration selected on the tuning requests for each metric.

• Missing vLLM-Omni rows. Only tuning winners were carried into the evaluation. At C=4 and C=8 SGLang-Omni won every metric, so vLLM-Omni has no evaluation cell there. It was measured there on the tuning requests: 331–405 frames/s, TTFA p50 0.63–1.19 s (all 42 tuning cells).
• vLLM-Omni stops at ~400 frames/s. Every configuration plateaus from C=4 upward (about 16 s of audio per second), so RTF reaches 1 at C=16 and 2 at C=32 and streams stall. Its stage trace puts the limit in the eager, launch-bound Code2Wav stage (analysis).
• SGLang-Omni does not stream audio. It returns one waveform per request, so its TTFA is its end-to-end latency (details).

Target status: not met. The goal is 2× the better baseline on every metric at every C, with the one-sided 95% bootstrap lower bound (LCB) at or above 2.0. TTFA reaches it in 11 of 12 cells (not TTFA p50 at C=32). Throughput and RTF reach 1.14–1.49× (LCB 1.13–1.49). The track_s_fast_r16 profile passes the Token2Wav numerical bounds and the speech-quality absolute floors over three seeds at C=1 and C=32 (UTMOS 4.166 / 4.167 against 4.163). Its UTMOS stays 0.13 below SGLang-Omni's, which does not stream audio; that comparison is reported, not required. Ratios, streaming metrics, quality, run ranges and raw evidence · Baseline selection and analysis · Test results.

References

• MiniCPM-o 4.5: checkpoint, processor, encoders and the streaming_generate reference ported here.
• minicpmo-utils (stepaudio2): the reference Token2Wav, CosyVoice2 flow and HiFT modules.
• FlashInfer, Triton and PyTorch: GPU operators and CUDA graphs.
• vLLM-Omni and SGLang-Omni: serving baselines and the speech evaluation tools (Seed-TTS WER, speaker similarity).
• Seed-TTS eval: the voice-cloning workload.
• OmniRSI: recorded optimization knowledge and evidence.

See NOTICE and the Apache-2.0 license text of the reused code.