diff --git a/docs/index.md b/docs/index.md index 999d9d4d..528decc9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -94,8 +94,10 @@ Maru (/mɑːruː/) — named after the *maru* (마루), the central open floor i ## Getting Started +- {doc}`System Setup ` - {doc}`Installation ` -- {doc}`Quick Start ` +- {doc}`Quick Start ` — Single-host and multi-host KV sharing +- {doc}`Examples ` - {doc}`Architecture Overview ` ```{toctree} @@ -103,9 +105,10 @@ Maru (/mɑːruː/) — named after the *maru* (마루), the central open floor i :caption: Getting Started :hidden: +System Setup Installation -Quickstart -More Examples +Quick Start +Examples ``` ```{toctree} @@ -127,6 +130,7 @@ CPU/CXL L1 and SSD L2 (Proposal) LMCache vLLM +NVIDIA Dynamo SGLang HiCache ``` diff --git a/docs/source/design_doc/consistency_and_safety.md b/docs/source/design_doc/consistency_and_safety.md index 6e24e9b1..e00f9cad 100644 --- a/docs/source/design_doc/consistency_and_safety.md +++ b/docs/source/design_doc/consistency_and_safety.md @@ -4,19 +4,16 @@ ## 1. Data Visibility -Maru provides **strong consistency** for store operations: once `store()` returns -successfully, the stored data is immediately retrievable by any other instance. +Maru publishes an object's location after the writer finishes writing its +payload. Once `store()` returns successfully, other instances can look up that +location through the metadata server. -This relies on **write-then-register** ordering. The handler always -completes these steps in sequence: +This relies on **write-then-register** ordering. Every store completes these steps in sequence: -1. **Write** — copy data into the CXL memory region. -2. **Register** — notify the metadata registry that the key now maps to that - location. +1. **Write** — the caller writes the payload into a page from `alloc()`, by CPU through `handle.buf` or by GPU DMA. +2. **Register** — `store()` records in the metadata registry that the key now maps to that page. -Because registration only occurs after the write is fully committed to shared -memory, no reader can ever observe a partial or in-progress write. The key simply -does not exist in the registry until the data is complete. +The caller must finish its write before calling `store()`. Publishing only after the write completes prevents readers from discovering an in-progress object through the registry, provided the written data is visible to the reader (see {ref}`cross-host-visibility`). ```mermaid sequenceDiagram @@ -25,21 +22,37 @@ sequenceDiagram participant Meta as Metadata Registry participant R as Reader (Instance B) - W->>CXL: 1. Write data to region - Note over CXL: Data fully committed + W->>CXL: 1. Write data to page (flush CPU writes) + Note over CXL: Payload reached the device W->>Meta: 2. Register key -> location Note over Meta: Key now globally visible R->>Meta: 3. Lookup key Meta-->>R: location (region, offset) - R->>CXL: 4. Read data (zero-copy) - Note over R: Complete, consistent data + R->>CXL: 4. Read data (zero-copy, invalidate before CPU reads) + Note over R: Complete data ``` The visibility point is when `register_kv` RPC completes. The server holds the key in an in-memory registry protected by a lock, ensuring that concurrent lookups always see a fully committed entry or no entry at all. +(cross-host-visibility)= +### Cross-Host Visibility + +The ordering above assumes that writes reach the shared memory device and readers cannot consume stale host-cache copies. On a non-coherent multi-host CXL platform, metadata ordering alone does not establish those conditions. See {doc}`../getting_started/bios_setup` for the platform settings that GPU DMA writes depend on. + +On Intel GNR, the BIOS option `Allocating Write Flows` controls where device writes to CXL memory, such as GPU DMA writes, land. With the default `Allocating` policy, a write can stay in the writing host's cache, so other hosts read stale data. `Non-Allocating` stops these writes from allocating new lines in the host cache. It does not change lines the host already caches, for example after CPU access to the same memory, so CPU access still needs the explicit write-back and invalidation below. + +CPU access to a shared payload can leave cached copies that other hosts do not invalidate, so CPU writers and readers must write back and invalidate explicitly. `flush_range` from `maru_shm._cxl_flush` does both: it runs `clflush` over each cache line of the buffer, then `mfence`. + +1. **Writer:** write the payload to `handle.buf`, call `flush_range(handle.buf)`, then `store()`. +2. **Reader:** `retrieve()`, call `flush_range(result.view)` to drop local cached copies, then read `result.view`. + +The {doc}`../getting_started/quick_start` producer and consumer follow this pattern. The CPU-based `examples/basic/producer.py` and `consumer.py` are single-host examples and do not flush. + +For multi-node use, validate the deployed transfer path across repeated updates and memory reuse, including any CPU access performed by the runtime or driver. + > **See also:** [MaruHandler Architecture](maru_handler.md) --- @@ -144,4 +157,3 @@ mismatch triggers reclamation. | Resource Manager crash | New region allocation blocked | WAL + checkpoint replay on restart | None | | Network partition (client-server) | Affected client cannot store/retrieve | Client reconnects when network recovers | None | | CXL device failure | All data on the device is lost | Not supported -- no cross-device replication | **Total** | - diff --git a/docs/source/getting_started/bios_setup.md b/docs/source/getting_started/bios_setup.md new file mode 100644 index 00000000..3ceac525 --- /dev/null +++ b/docs/source/getting_started/bios_setup.md @@ -0,0 +1,44 @@ +# System Setup + +Maru requires CXL memory exposed as **DEV_DAX** (`/dev/dax*`). Follow your +server vendor's CXL setup instructions on each host. + +## Intel (GNR) + +For multi-host KV sharing on Intel Granite Rapids (GNR), configure the BIOS as follows: + +- If a **DDIO** control is available, use it to **disable DDIO**. +- Otherwise, set **`Allocating Write Flows`** to **`Non-Allocating`**. + +Option names and menu locations vary by server vendor and BIOS version. +If neither option is available, consult your server vendor's documentation. + +Apply the setting on every participating host, save the configuration, and reboot. + +## AMD (Turin) + +Complete your platform's CXL setup, then verify DEV_DAX access below. + +## Verify after reboot + +On every participating host, inspect the CXL/DAX devices: + +```bash +# Install the inspection utilities if needed. +sudo apt-get install -y cxl daxctl +sudo cxl list -R -D +sudo daxctl list +ls -l /dev/dax* +``` + +Confirm that the selected memory is in `devdax` mode and the account running +Maru has read/write access to its local device. If it appears as `system-ram`, +follow your platform's instructions to provision it as DEV_DAX before continuing. + +For multi-node sharing, also configure the CXL fabric to expose the same memory +to every host. + +BIOS settings alone do not guarantee cross-host cache coherence. See +{doc}`../design_doc/consistency_and_safety` for the data-visibility requirements. + +Once device access is verified, continue to {doc}`installation`. diff --git a/docs/source/getting_started/examples/dynamo/index.md b/docs/source/getting_started/examples/dynamo/index.md new file mode 100644 index 00000000..927f3627 --- /dev/null +++ b/docs/source/getting_started/examples/dynamo/index.md @@ -0,0 +1,134 @@ +# Dynamo Examples + +Run Dynamo's vLLM backend with Maru to share KV cache between workers. +The scripts live in `examples/dynamo/single_node/`. + +## Prerequisites + +- A local CXL DEV_DAX pool and a running `maru-resource-manager` (see {ref}`single-host quick start `). The example starts its own MaruServer, so you do not need to start one. +- Two GPUs by default, or enough memory on one GPU for two small-model workers. +- Maru installed with its KV placement kernels (see {doc}`../../installation`). +- Dynamo's vLLM backend installed in the **same Python environment** as Maru + (see [Dynamo's vLLM installation guide](https://docs.nvidia.com/dynamo/backends/v-llm)). + +## P2P KV Cache Sharing + +One Dynamo frontend routes requests to two `dynamo.vllm` workers on a single +host. The first worker stores KV cache in Maru; the second reuses it for the +same prompt. The launcher disables vLLM's local prefix cache for this check. + +This example runs on a single host only. A multi-host Dynamo example is coming soon. + +### Automated + +```bash +cd examples/dynamo/single_node +./single_node_example.sh # Qwen/Qwen2.5-0.5B; GPUs 0 and 1 + +# Both small-model workers on GPU 0 +W0_GPU=0 W1_GPU=0 ./single_node_example.sh + +# Another model on the default two GPUs +./single_node_example.sh meta-llama/Llama-3.1-8B-Instruct +``` + +The script starts MaruServer, the frontend, and both workers, runs the sharing +check, and stops the processes it started. Logs are saved in the example +directory as `maru_server.log`, `frontend.log`, `w0.log`, and `w1.log`. + +**Success:** both workers return the same answer, and the second worker reports +a nonzero external prefix-cache hit rate. The script exits nonzero if either +check fails. Latency may not improve for the small default model. + +### Step-by-step + +In **every terminal**, activate the same environment and run the setup below. +Choose a new, unused discovery directory for each run and use the same path +in every terminal: + +```bash +cd examples/dynamo/single_node +export MODEL=Qwen/Qwen2.5-0.5B +export DYN_FILE_KV=/tmp/maru-dynamo-manual-run-1 +source env.sh +``` + +Run each service in a separate terminal, in this order: + +1. Start MaruServer: + + ```bash + maru-server --port "$MARU_SERVER_PORT" + ``` + +2. Start the frontend and wait for its health endpoint to respond: + + ```bash + ./dynamo_launcher.sh frontend + ``` + + From another terminal with the same environment: + + ```bash + curl --fail "http://localhost:${DYN_HTTP_PORT}/health" + ``` + +3. Start each worker in its own terminal. Wait for `w0` to initialize before + starting `w1`. Keep the logs as shown; the verification script reads them: + + ```bash + set -o pipefail + ./dynamo_launcher.sh worker w0 2>&1 | tee w0.log + ``` + + In the other worker terminal: + + ```bash + set -o pipefail + ./dynamo_launcher.sh worker w1 2>&1 | tee w1.log + ``` + +4. Check that the frontend lists both workers. The response should contain two `instances` entries whose `endpoint` is `generate`: + + ```bash + curl -s "http://localhost:${DYN_HTTP_PORT}/health" + ``` + + Then run: + + ```bash + ./run_simple_query.sh + ``` + + The frontend can return HTTP 503 for a short time after the workers register. If the query script fails with a 503, wait a few seconds and run it again. + +The query script sends the same prompt to each worker through the frontend +and checks both the answers and the second worker's external cache hit rate. +Stop the services with Ctrl+C when finished. + +### Configuration + +Settings live in `examples/dynamo/single_node/env.sh` and can be overridden +through the environment: + +| Variable | Default | Description | +|----------|---------|-------------| +| `DYN_HTTP_PORT` | `14000 + uid` | Frontend HTTP port | +| `DYN_W0_SYSTEM_PORT`, `DYN_W1_SYSTEM_PORT` | Frontend port + `81`, + `82` | Worker health/metrics ports | +| `MARU_SERVER_PORT` | `10000 + uid` | Local MaruServer port | +| `MARU_POOL_SIZE` | `4G` | Initial CXL allocation per handler | +| `MARU_KV_CHUNK_TOKENS` | `256` | Tokens in a complete stored chunk | +| `W0_GPU`, `W1_GPU` | `0`, `1` | Worker GPU assignment | +| `GPU_MEM_UTIL` | `0.3` | vLLM memory utilization per worker | +| `DYN_FILE_KV` | `/tmp/dynamo_kv_` | Shared local discovery directory | + +Send API requests to the frontend; worker system ports serve health and metrics. +The automated script clears `DYN_FILE_KV` for file discovery at startup, so use +a directory dedicated to this example. + +## Further reading + +- {doc}`../../../integration/dynamo` — Integration architecture +- {doc}`../../../integration/vllm` — Shared connector settings +- [Example README](https://github.com/xcena-dev/maru/blob/main/examples/dynamo/single_node/README.md) + — Discovery backends, etcd setup, and troubleshooting diff --git a/docs/source/getting_started/examples/index.md b/docs/source/getting_started/examples/index.md index 74c38376..592265f4 100644 --- a/docs/source/getting_started/examples/index.md +++ b/docs/source/getting_started/examples/index.md @@ -1,8 +1,12 @@ -# More Examples +# Examples + +Start with {doc}`../quick_start` for single-host or multi-host KV sharing. +The examples below cover vLLM, LMCache, and Dynamo serving configurations. ```{toctree} :maxdepth: 2 vllm/index lmcache/index +dynamo/index ``` diff --git a/docs/source/getting_started/installation.md b/docs/source/getting_started/installation.md index 4cfac876..5035936d 100644 --- a/docs/source/getting_started/installation.md +++ b/docs/source/getting_started/installation.md @@ -25,6 +25,8 @@ In a single-node setup, all components run on the same machine. In a multi-node - CXL DAX device (`/dev/dax*`) or emulation environment - **Multi-node:** All participating nodes must be connected to a shared CXL memory pool (e.g., via CXL switch). +See {doc}`bios_setup` for platform BIOS settings and DEV_DAX verification, and {doc}`../design_doc/consistency_and_safety` for the cache-visibility requirements of cross-host sharing. + ```bash sudo apt-get update sudo apt-get install -y python3 python3-venv python3-pip git \ @@ -139,6 +141,7 @@ pre-commit run --all-files
+(installation-verify)= ## 2. Verify Installation Verify that the Maru Python package is installed: @@ -147,13 +150,22 @@ Verify that the Maru Python package is installed: python3 -c "import maru_shm; print('ok')" ``` +Verify the x86 cache-flush extension used by the {doc}`quick_start` producer and consumer. It is an optional build product, so the install succeeds without it when no C compiler is available: + +```bash +python3 -c "from maru_shm._cxl_flush import HAVE_CLFLUSH; print(HAVE_CLFLUSH)" +``` + +`1` means the extension is available. An `ImportError` means it was not built: install a C compiler (`build-essential`) and the headers for your Python interpreter (`python3-dev` for the system Python), then rerun `install.sh`. + If you installed with the Resource Manager, verify the binary: ```bash which maru-resource-manager ``` -Once installation is verified, proceed to the {doc}`quick_start` guide to start services and run your first store/retrieve. +Once installation is verified, choose the single-host or multi-host setup in +{doc}`quick_start` to start services and verify KV sharing.
@@ -176,6 +188,7 @@ In a multi-node deployment, the Resource Manager, Metadata Server, and MaruHandl > `maru-server` and `maru-resource-manager` do not need to run on the same node. The diagram above shows the simplest configuration; each can be deployed independently as long as MaruHandler can reach both over the network. +(installation-multi-node-config)= ### Configuration Multi-node requires changing default bind addresses from `127.0.0.1` to a network-accessible address: @@ -184,6 +197,23 @@ Multi-node requires changing default bind addresses from `127.0.0.1` to a networ - **Metadata Server**: change `--host` and set `--rm-address` to the Resource Manager's externally reachable address - **MaruHandler**: set `server_url` to the Metadata Server's address. The Resource Manager address is received automatically via handshake. +To run the Resource Manager as the systemd service on a network address, override its command line: + +```bash +sudo systemctl edit maru-resource-manager +``` + +```ini +[Service] +ExecStart= +ExecStart=/usr/local/bin/maru-resource-manager --host --port 9850 +``` + +```bash +sudo systemctl restart maru-resource-manager +``` + > **Security:** When binding to a non-loopback address, auth tokens and device paths are transmitted in plaintext. Use an encrypted tunnel (WireGuard, SSH tunnel, IPsec) in production. -> Multi-node end-to-end examples and deployment guide are coming soon. +Follow the {ref}`multi-host quickstart ` to start the +services and share data between a Python producer and consumer on two hosts. diff --git a/docs/source/getting_started/quick_start.md b/docs/source/getting_started/quick_start.md index 16ed0fee..2034a0eb 100644 --- a/docs/source/getting_started/quick_start.md +++ b/docs/source/getting_started/quick_start.md @@ -1,160 +1,189 @@ -# Quickstart +# Quick Start -## 1. Start the Resource Manager +Start the services for your setup, then run the same Python producer and +consumer below. No GPU or inference engine is required. -The resource manager must be running before any other Maru service. +| Setup | Required environment | Where the clients run | +|-------|----------------------|-----------------------| +| {ref}`Single-host ` | One x86 host with a CXL DEV_DAX device | Two terminals on the same host | +| {ref}`Multi-host ` | Two x86 hosts connected to shared CXL memory through a CXL switch/fabric | Producer on Node A, consumer on Node B | -**Production** — start as a systemd daemon: +Complete {doc}`bios_setup` and {doc}`installation` first. Activate the Maru +Python environment in each terminal. Both setups use port `9850` for the +Resource Manager and `5555` for MaruServer. -```bash -# Default (127.0.0.1:9850) -sudo systemctl start maru-resource-manager +## 1. Start the services -# With custom host/port -sudo systemctl edit maru-resource-manager +Choose one setup below. Keep the services running while using the clients. -[Service] -ExecStart= -ExecStart=/usr/local/bin/maru-resource-manager --host --port +(quickstart-single-host)= +### Single-host -sudo systemctl restart maru-resource-manager +Start the Resource Manager as a systemd service: +```bash +sudo systemctl start maru-resource-manager ``` -**Development/debugging** — run directly with custom options: +Alternatively, with the service stopped, run it in a dedicated terminal: ```bash -# Default (127.0.0.1:9850) -sudo maru-resource-manager --log-level debug - -# With custom host/port -sudo maru-resource-manager --host --port --log-level debug +sudo maru-resource-manager --host 127.0.0.1 --port 9850 ``` -> If you change the RM port, pass the same address to `maru-server`: `maru-server --rm-address 127.0.0.1:9851` -> -> See {doc}`../design_doc/maru_resource_manager` for the full list of CLI options. - -## 2. Start the Metadata Server +Wait until the Resource Manager logs `ready — listening on 127.0.0.1:9850` (for the systemd service, check `journalctl -u maru-resource-manager`), then start MaruServer in another terminal: ```bash -# Default (127.0.0.1:5555, connects to resource manager at 127.0.0.1:9850) maru-server +``` -# With custom host/port -maru-server --host --port - -# With custom resource manager address (default: 127.0.0.1:9850) -maru-server --rm-address : +In **both client terminals**, set the server address: -# With debug logging -maru-server --log-level DEBUG +```bash +export MARU_SERVER_URL=tcp://127.0.0.1:5555 ``` -## 3. Run a Client Example +Continue to {ref}`the producer and consumer `. -> Both `maru-resource-manager` and `maru-server` must be running before proceeding. +(quickstart-multi-host)= +### Multi-host -### Zero-Copy Store & Retrieve +This example runs both services and the producer on **Node A**, with the +consumer on **Node B**: -```python -from maru import MaruConfig, MaruHandler - -config = MaruConfig( - server_url="tcp://localhost:5555", - pool_size=1024 * 1024 * 100, # Request 100MB from the CXL memory pool -) +```mermaid +flowchart TB + subgraph A["Node A · 192.0.2.10"] + P["Producer
MaruHandler"] + Services["MaruServer: 5555
Resource Manager: 9850"] + P -.-> Services + end + subgraph B["Node B"] + C["Consumer
MaruHandler"] + end + + C -.-> Services + P --- Switch["CXL switch / fabric"] + C --- Switch + Switch --- Memory[("Shared CXL memory")] +``` -with MaruHandler(config) as handler: - data = b"A" * (1024 * 1024) # 1MB KV chunk +Dashed lines are TCP connections to the services on Node A. Solid lines show +each host's direct access to the shared memory through its local DEV_DAX device +and the CXL fabric. - # 1. Allocate a page in CXL shared memory - handle = handler.alloc(size=len(data)) +Both hosts must have read/write access to the **same physical CXL memory at +matching device offsets**. Install Maru with `./install.sh` on Node A and +`./install.sh --no-rm` on Node B. - # 2. Write directly to CXL memory (mmap — no intermediate buffer) - handle.buf[:] = data +Replace `192.0.2.10` with Node A's reachable IP. Node B must be able to reach +Node A on TCP ports `5555` and `9850`. - # 3. Register the key — only metadata (key → region, offset) is sent - handler.store(key=42, handle=handle) +Run one Resource Manager for the shared pool on **Node A**. Stop the systemd service first: a service still bound to `127.0.0.1:9850` does not conflict with a second Resource Manager on `192.0.2.10:9850`, and the two would manage the same devices and state directory. Then start it in a dedicated terminal: - # Retrieve: returns a memoryview pointing into CXL memory - result = handler.retrieve(key=42) - assert result is not None - assert bytes(result.view[:5]) == b"AAAAA" +```bash +sudo systemctl stop maru-resource-manager +sudo maru-resource-manager --host 192.0.2.10 --port 9850 ``` -The three steps — `alloc()`, write to `handle.buf`, `store(handle=)` — make Maru's control/data plane separation explicit: +To run it as the systemd service instead, see the {ref}`multi-node configuration `. -- **Control plane**: `alloc()` requests a memory region from the Resource Manager; `store()` registers the key's location metadata (~tens of bytes) with MaruServer. -- **Data plane**: `handle.buf[:]` reads/writes directly on mmap'd CXL memory. No server involved. +Wait until it logs `ready — listening on 192.0.2.10:9850`, then start MaruServer in another terminal on **Node A**: -> Full runnable example: `examples/basic/single_instance.py` +```bash +maru-server --host 192.0.2.10 --port 5555 \ + --rm-address 192.0.2.10:9850 \ + --dax-path /dev/dax0.0 +``` -### Cross-Instance Sharing +Replace `/dev/dax0.0` with the shared device's path on Node A. Local device +names may differ on Node B; Maru resolves them by UUID. The advertised +`--rm-address` must be reachable from Node B, so use Node A's IP here. -This is the core of Maru — two independent processes sharing KV cache through CXL shared memory with zero data copy. **Metadata travels, data doesn't.** +In **both client terminals** (one on each host), set the same server address: -```mermaid -flowchart LR - P[Producer] -->|"alloc + write"| CXL[(CXL Memory)] - P -->|"store(key)"| S[MaruServer] - C[Consumer] -->|"retrieve(key)"| S - S -.->|"location"| C - C -->|"read"| CXL +```bash +export MARU_SERVER_URL=tcp://192.0.2.10:5555 ``` -**Terminal 1** — Producer: allocate, write, register metadata: +Continue to {ref}`the producer and consumer `. + +(quickstart-clients)= +## 2. Run the producer and consumer + +Use these same snippets for either setup. Keep `MARU_SERVER_URL` set as shown +above in each client terminal. For single-host, run both clients on the same +host; for multi-host, run the producer on Node A and the consumer on Node B. + +### Producer: write and store + +Run this first and keep it running until the consumer finishes: ```python +import os + from maru import MaruConfig, MaruHandler +from maru_shm._cxl_flush import HAVE_CLFLUSH, flush_range + +assert HAVE_CLFLUSH, "This CPU example requires x86 cache-flush support" config = MaruConfig( - server_url="tcp://localhost:5555", + server_url=os.environ["MARU_SERVER_URL"], instance_id="producer", - pool_size=1024 * 1024 * 100, + pool_size=100 * 1024 * 1024, ) with MaruHandler(config) as handler: - for i, key in enumerate([1001, 1002, 1003]): - data = bytes([ord("A") + i]) * (1024 * 1024) - - handle = handler.alloc(size=len(data)) - handle.buf[:] = data - handler.store(key=key, handle=handle) + data = b"A" * (1024 * 1024) + handle = handler.alloc(size=len(data)) + handle.buf[:] = data + flush_range(handle.buf) # Make the CPU write visible before publishing. + handler.store(key="quickstart-demo", handle=handle) - input("Press Enter to exit...") + print("Stored 1 MiB. Run the consumer.") + input("Press Enter after the consumer finishes...") ``` -**Terminal 2** — Consumer: retrieve from CXL (zero copy): +### Consumer: retrieve and verify + +Run this in the other client terminal: ```python +import os + from maru import MaruConfig, MaruHandler +from maru_shm._cxl_flush import HAVE_CLFLUSH, flush_range + +assert HAVE_CLFLUSH, "This CPU example requires x86 cache-flush support" config = MaruConfig( - server_url="tcp://localhost:5555", + server_url=os.environ["MARU_SERVER_URL"], instance_id="consumer", - pool_size=1024 * 1024 * 100, + pool_size=100 * 1024 * 1024, ) with MaruHandler(config) as handler: - for key in [1001, 1002, 1003]: - result = handler.retrieve(key=key) - - # result.view points directly into Producer's CXL region (mapped read-only). - # No data was copied — consumer reads the same physical memory. - assert result is not None - print(f"key={key}: {len(result.view)} bytes, char={chr(result.view[0])!r}") + result = handler.retrieve(key="quickstart-demo") + assert result is not None, "Run the producer first" + flush_range(result.view) # Discard local cached copies before reading. + assert bytes(result.view) == b"A" * (1024 * 1024) + print("Success: read the producer's data from shared CXL memory") ``` -> Full runnable examples: `examples/basic/producer.py` + `examples/basic/consumer.py` -> -> Single-script version: `examples/basic/cross_instance.py` +The consumer should print `Success: read the producer's data from shared CXL memory`. +The `flush_range` calls make CPU writes and reads visible across hosts and are used in both setups. They require Maru's x86 cache-flush extension; if the import fails, see {ref}`verifying the installation `. See {doc}`../design_doc/consistency_and_safety` for the write-back and invalidation pattern. + +## 3. Stop the example +Press Enter in the producer terminal to exit. Stop MaruServer, then stop the +Resource Manager if you started it for this example. Use Ctrl+C for foreground +processes, or `sudo systemctl stop maru-resource-manager` for the systemd service. -## Next Steps +## Next steps -- {doc}`installation` — Multi-node configuration -- [LMCache Examples](examples/lmcache/index.md) — Use Maru as a shared KV cache backend for LMCache/vLLM -- [Architecture Overview](../design_doc/architecture_overview.md) — System architecture and component interactions -- [API Reference](../api_reference/api.md) — Python API documentation -- [Configuration](../api_reference/config.md) — Configuration options +- {doc}`examples/vllm/index` — More vLLM serving examples +- {doc}`examples/dynamo/index` — Run Dynamo with Maru-backed vLLM workers +- {doc}`examples/lmcache/index` — Use Maru as a shared KV cache backend for LMCache +- {doc}`../design_doc/architecture_overview` — System architecture and component interactions +- {doc}`../api_reference/api` — Python API +- {doc}`../api_reference/config` — Configuration options diff --git a/docs/source/integration/dynamo.md b/docs/source/integration/dynamo.md new file mode 100644 index 00000000..7321586c --- /dev/null +++ b/docs/source/integration/dynamo.md @@ -0,0 +1,24 @@ +# NVIDIA Dynamo + +Dynamo connects to Maru through its **vLLM backend**. Configure each +`dynamo.vllm` worker with the same `MaruKVConnector` and `--kv-transfer-config` +used for standalone vLLM. Dynamo routes requests to the workers, and the +connector handles KV cache sharing through Maru. + +```mermaid +flowchart TB + Client["OpenAI API client"] --> FE["Dynamo frontend"] + subgraph W0["Dynamo worker A"] + V0["vLLM"] --> C0["MaruKVConnector"] + end + subgraph W1["Dynamo worker B"] + V1["vLLM"] --> C1["MaruKVConnector"] + end + FE --> V0 & V1 + C0 & C1 -.->|"metadata"| MS["MaruServer"] + C0 & C1 <-->|"KV payload"| CXL[("CXL shared memory")] +``` + +See {doc}`vllm` for connector settings and the +{doc}`Dynamo example <../getting_started/examples/dynamo/index>` for setup and +execution instructions.