An open toolkit and public dataset hub for collecting, sanitizing, analyzing, and visualizing coding agent traces.
-
Updated
Jul 24, 2026 - Python
#
serving-infrastructure
Here are 11 public repositories matching this topic...
A high-performance, universal serving framework for any-to-any models.
distributed-systems robotics inference high-performance-computing diffusion multimodal world-models vision-language-models serving-infrastructure speech-language-model agentic-infrastructure vision-language-action-model unified-multimodal-models
-
Updated
Aug 1, 2026 - Python
Learn the ins and outs of efficiently serving Large Language Models (LLMs). Dive into optimization techniques, including KV caching and Low Rank Adapters (LoRA), and gain hands-on experience with Predibase’s LoRAX framework inference server.
text-generation batch-processing server-optimization model-serving model-acceleration inference-optimization optimization-techniques machine-learning-operations deep-learning-techniques model-inference-service performance-enhancement scalability-strategies serving-infrastructure large-scale-deployment
-
Updated
Apr 12, 2024 - Jupyter Notebook
Discrete-event simulation of LLM request routing across multiple serving instances: round-robin, least-load, prefix-aware, and hybrid cache-aware routing with queue-depth threshold sweep.
python simulation latency inference load-balancing multi-instance request-routing mooncake kv-cache llm-serving serving-infrastructure prefix-caching s-lora mlsystems
-
Updated
Jul 19, 2026 - Python
Discrete-event simulation of SLO-aware autoscaling for multi-instance LLM serving — comparing reactive, predictive, and conservative policies across workload patterns and startup delays.
python simulation latency capacity-planning inference slo autoscaling llm-serving serving-infrastructure startup-delay mlsystems workload-patterns
-
Updated
Jul 19, 2026 - Python
Simulating KV cache reuse across multi-turn LLM conversations using real ShareGPT traces. Compares sticky, least-load, hybrid, and cost-aware routing strategies for cache locality vs load balance.
python simulation routing load-balancing conversational-ai multi-turn kv-cache sharegpt llm-serving serving-infrastructure prefix-caching mlsystems
-
Updated
Jul 19, 2026 - Python
Discrete-event simulator for studying throughput, TTFT, and KV-cache transfer tradeoffs in disaggregated LLM serving.
python gpu latency scheduler decoder inference systems throughput discrete-event-simulation performance-modeling prefill kv-cache llm-serving serving-infrastructure
-
Updated
Jul 19, 2026 - Python
Comparing KV-cache-aware scheduling policies for LLM serving: greedy preempts 49% of requests under memory pressure while memory-first achieves 43% higher effective throughput with zero preemptions.
python simulation scheduling inference preemption kv-cache memory-pressure admission-control llm-serving serving-infrastructure mlsystems
-
Updated
Jul 19, 2026 - Python
Benchmarking output length prediction quality and calibration for KV-cache-aware LLM serving scheduling. A noisy predictor with 1.5x safety margin captures 99.7% of oracle gains — calibration matters more than accuracy.
python simulation scheduling inference kv-cache admission-control llm-serving serving-infrastructure mlsystems output-length-prediction prediction-calibration
-
Updated
Jul 19, 2026 - Python
Comparing LLM serving under real ShareGPT traces vs synthetic Poisson workloads. Real distributions produce 42x worse P99 TTFT and earlier SLO violations — scheduling policies only differentiate under heavy-tailed real traffic.
python benchmarking simulation scheduling slo workload-characterization sharegpt llm-serving serving-infrastructure p99-latency heavy-tail mlsystems
-
Updated
Jul 19, 2026 - Python
Step-level simulation of chunked prefill under KV cache pressure. Block policy wastes 89.6% of prefill chunks while achieving the same throughput as upfront rejection. Chunk waste rate is the metric standard throughput metrics miss.
python simulation inference kv-cache memory-pressure llm-serving serving-infrastructure continuous-batching chunked-prefill mlsystems
-
Updated
Jul 19, 2026 - Python
Improve this page
Add a description, image, and links to the serving-infrastructure topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the serving-infrastructure topic, visit your repo's landing page and select "manage topics."