Note
Initial release (experimental). KVCR supports cross-node DRAM sharing, local disk caching, and resiliency for vLLM. Future releases will add SGLang and TRT-LLM integrations and object storage support.
Feedback and contributions are welcome - see CONTRIBUTING.md.
Modern AI workloads require KV cache infrastructure that delivers speed, scale, and resilience.
KV Cache Runner (KVCR) reimagines KV cache as a system-wide distributed resource: available across memory and storage tiers, resilient by design, and managed holistically to unlock system-level optimizations and maximize end-to-end performance.
KVCR is agentic-native and built to accelerate AI workloads of every kind.
KVCR is at its most powerful when working in tandem with a KV-aware request router. The router knows where the KV cache resides and can provide the selected worker with hints about where to retrieve it. The cache can be sourced locally or from remote peers, within or across memory and storage tiers, or from any combination of these sources.
KVCR leaves local KV cache offloading to host memory under the engine's control. It focuses on system-level optimizations enabled by this architecture, including cross-node KV cache sharing, KV-aware request load balancing, and KV cache prefetching — all guided by the router’s system-wide view. By maximizing KV cache reuse and overlapping cache onboarding with computation, KVCR reduces redundant work and improves prefill efficiency.
KVCR is resilient by design. KVCR-Guard is a sidecar KVCR process operating in active-passive mode and can outlive an engine failure. In such an event, KVCR-Guard remains available as a remote source of KV cache, minimizing disruption to serving in the face of failures.
KVCR provides a flexible policy interface that allows its behavior to be customized for different workloads.
For more details, see the design document.
kvcr.api: Contains the user-facing API for configuring KVCR and interacting with its core.kvcr.policy: Contains the policy interface for customizing KVCR's behavior for different workloads.
KVCR is vendor and framework agnostic, with no dependency on a specific request router. The list below summarizes framework integrations that are complete or in progress. We plan to extend support to additional frameworks and routers in the future.
-
TRT-LLM
-
vLLM
-
SGLang
-
Dynamo KV Router — supported
-
TRT-LLM
-
vLLM — supported
-
SGLang
The quick start is a public preview that builds and runs KVCR with vLLM, Dynamo, and NIXL using pinned source revisions and a compatible base image. See the developer guide for source installation and verification.
For local development, API lifecycle guidance, validation, integration, and KVCR guard service usage, see the developer guide.
KVCR is released under the Apache License 2.0. The full license text is in LICENSE.
KVCR package source files carry an SPDX Apache-2.0 identifier and the NVIDIA copyright notice. The vLLM source files used in the public quick-start build retain their Apache-2.0 contributor headers.
vLLM and Dynamo are installed separately in the quick-start image and are not
bundled into the kvcr wheel. The build uses the public sources and
pinned revisions described in the quick start.
KVCR declares runtime dependencies on msgspec, pyzmq, and nixl. Each is
installed from its own distribution under its own license. None of them are
redistributed by this repository or bundled into the kvcr wheel, which
packages only src/kvcr.
This project accepts external contributions. See CONTRIBUTING.md for the workflow.
Contributions require a Developer Certificate of Origin sign-off
(git commit -s), whose full text is reproduced in that file and which is
enforced on every pull request by the dco CI workflow. Participation is
governed by CODE_OF_CONDUCT.md.
To report a security vulnerability, do not open a public issue — follow SECURITY.md.