This file provides guidance to WARP (warp.dev) when working with code in this repository.
This is a production-ready FastAPI-based router that enables seamless switching between multiple LLM inference backends (SGLang, TabbyAPI, llama.cpp). It acts as an OpenAI-compatible API gateway on port 8002, automatically managing backend services, streaming status updates during model switches, and appending real-time performance metrics (tok/s) to responses.
Key Value Proposition: Solves the limitation that backends like SGLang/vLLM/TabbyAPI can only serve one model at a time, enabling Open-WebUI users to switch between models from a dropdown without manual service restarts.
Version 3.3.0 Changes: Simplified configuration - no startup scripts needed! Just point to model files. Added interactive model management script for non-technical users. Changed from YAML to JSON config.
router.py - Main application with three key responsibilities:
- Model Switching Logic: Detects when a different backend is needed, stops the current service, updates systemd service configs (for SGLang/llama.cpp) or config.yml (for TabbyAPI), restarts services with new model
- Streaming Status Updates: Yields SSE messages during model loading (🔄 Switching → ⏳ Loading... Xs/180s → ✅ Ready!)
- Request Forwarding: Proxies chat completion requests to the active backend with performance tracking
Backend Support:
- SGLang (port 30000): AWQ quantized models, switched via systemd ExecStart script updates
- TabbyAPI (port 5000): EXL2 quantized models, switched via config.yml model_name updates with 3s shutdown wait + 5s warmup
- llama.cpp (port 8085): GGUF quantized models, switched via systemd ExecStart script updates
Model Configuration (loaded from /opt/llm-router/config.json):
{
"models": {
"model-id": {
"backend": "sglang|tabbyapi|llamacpp",
"model_path": "/full/path/to/model/file/or/directory"
}
}
}Note: Version 3.3.0 removed the need for startup scripts. The router now only needs the model path and automatically handles backend startup with the correct model.
The router uses systemd service restarts (not API-based model loading) because:
- Provides clean state on every switch
- Prevents stuck/orphaned processes
- TabbyAPI's API-based model loading proved unreliable (truncated responses)
- SGLang doesn't support runtime model switching
TabbyAPI Switching Sequence:
- Stop all backend services
- Wait 3s for clean shutdown (critical to prevent file locks)
- Update TabbyAPI
config.ymlwith new model_name (searches ~/tabbyAPI/, /opt/tabbyAPI/, /etc/tabbyapi/) - Start TabbyAPI service
- Wait for health check + 5s warmup (prevents truncated responses)
# Run router directly (for development)
python3 router.py
# Install dependencies
pip install fastapi uvicorn httpx pyyaml# List available models
curl http://localhost:8002/v1/models
# Health check
curl http://localhost:8002/health
# Test streaming chat
curl -X POST http://localhost:8002/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mistral-large-2411-awq",
"messages": [{"role": "user", "content": "Hello!"}],
"stream": true
}'# Install with interactive prompts
sudo ./install.sh
# Service management
sudo systemctl start llm-router
sudo systemctl stop llm-router
sudo systemctl status llm-router
sudo systemctl enable llm-router # auto-start on boot
# View logs
sudo journalctl -u llm-router -f
# Check backend services
sudo systemctl status sglang
sudo systemctl status tabbyapi
sudo systemctl status llamacpp# Easy model management (recommended)
/opt/llm-router/manage-models.sh add
# Or edit configuration directly
sudo nano /opt/llm-router/config.json
# Reload after config changes
sudo systemctl restart llm-router
# List current models
/opt/llm-router/manage-models.sh list- Uses
asyncio.Lock()(switching_lock) to prevent concurrent model switches - Global
current_modelvariable tracks active model to avoid unnecessary switches
- Tracks token count during streaming by parsing SSE chunks (rough estimate: ~4 chars per token)
- Calculates tok/s = tokens / elapsed_time
- Appends "⚡ X.X tok/s (~N tokens in X.Xs)" to responses
- Non-streaming responses use backend's usage.completion_tokens if available
- Each backend has a health endpoint (
/health) wait_for_backend_ready_streaming()polls every 2s with 180s timeout- Yields status updates every 10s for UX feedback
- For TabbyAPI: health check pass + 5s warmup before declaring ready
- Model switch failures yield error SSE messages but don't crash the router
- Timeouts after 180s with clear error messaging
- HTTP exceptions for unknown models (400) and switch failures (503)
- Must be compiled with CUDA support for GPU acceleration
- For Blackwell GPUs (sm_90): Requires open-source kernel modules and CUDA 12.8+
cmake -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=90 \ -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \ -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler" - Use
-ngl 999to offload all layers to GPU - Port 8085 by default, health endpoint at
/health
- Config file locations checked:
~/tabbyAPI/config.yml,/opt/tabbyAPI/config.yml,/etc/tabbyapi/config.yml - Updates the model.model_name field in config.yml
- Requires YAML library:
pip install pyyaml - Service restart method is MORE reliable than API-based model loading
- 5s warmup critical: prevents first response truncation
- Uses per-model startup scripts that specify --model-path
- Service config updated via sed:
sed -i 's|ExecStart=.*|ExecStart=/path/to/script.sh|' - Requires
systemctl daemon-reloadafter service file changes
Setup:
- In Open-WebUI Settings → Connections
- Set API URL:
http://localhost:8002/v1 - API Key: (any value or leave empty)
- All configured models appear in model dropdown
User Experience:
- Select different model from dropdown → router automatically switches
- Chat window shows: "🔄 Switching to model-name..." → loading progress → "✅ Model ready!"
- Normal chat response streams with "⚡ tok/s" appended at end
The install.sh script:
- Detects Python 3.10+ (tries python3.12, python3.11, python3.10, python3)
- Prompts interactively for install dir, ports, backend details
- Creates Python venv at specified location
- Installs dependencies: fastapi, uvicorn, httpx, pyyaml
- Generates config files in
/etc/llm-router/ - Creates systemd service at
/etc/systemd/system/llm-router.service - User must manually edit
/etc/llm-router/models.ymlto add models
Use the management script (easiest):
/opt/llm-router/manage-models.sh addOr edit /opt/llm-router/config.json:
- All backends: Just provide
backendtype andmodel_path - No scripts, service files, or complex configuration needed
- Add backend URL constant (e.g., NEWBACKEND_URL)
- Add health check case in
get_backend_health() - Create
switch_to_newbackend_model_streaming()generator function - Add backend case in
streaming_chat_with_model_switch() - Update backend_url routing logic
- Update health endpoint to include new backend
- Current bottleneck on Blackwell GPUs: immature sm_120 kernels
- FlashAttention-4 for sm_120 not yet available (when released: expect 50-65 tok/s)
- Current performance: AWQ (SGLang) 22-25 tok/s, EXL2 (TabbyAPI) 17-18 tok/s
Operating System: Ubuntu/Debian Linux (systemd-based) Python: 3.10+ Dependencies: FastAPI, httpx, uvicorn, pyyaml Privileges: Root/sudo required for systemd service management Network: Backends must be accessible on localhost
- v3.2.0: Added llama.cpp support, improved Blackwell GPU compatibility
- v3.1.1: Added tok/s metrics, improved TabbyAPI reliability with warmup
- v3.0.0: Initial multi-backend support (SGLang + TabbyAPI)
Current production version uses service restart approach after determining API-based switching was unreliable for TabbyAPI.