docs: add multi-host quick start, system setup, and Dynamo pages

docs/index.md

@@ -94,18 +94,21 @@ Maru (/mɑːruː/) — named after the *maru* (마루), the central open floor i
 
 ## Getting Started
 
+- {doc}`System Setup <source/getting_started/bios_setup>`
 - {doc}`Installation <source/getting_started/installation>`
-- {doc}`Quick Start <source/getting_started/quick_start>`
+- {doc}`Quick Start <source/getting_started/quick_start>` — Single-host and multi-host KV sharing
+- {doc}`Examples <source/getting_started/examples/index>`
 - {doc}`Architecture Overview <source/design_doc/architecture_overview>`
 
 ```{toctree}
 :maxdepth: 2
 :caption: Getting Started
 :hidden:
 
+System Setup <source/getting_started/bios_setup>
 Installation <source/getting_started/installation>
-Quickstart <source/getting_started/quick_start>
-More Examples <source/getting_started/examples/index>
+Quick Start <source/getting_started/quick_start>
+Examples <source/getting_started/examples/index>
 ```
 
 ```{toctree}
@@ -127,6 +130,7 @@ CPU/CXL L1 and SSD L2 (Proposal) <source/design_doc/tiered_kv_cache>
 
 LMCache <source/integration/lmcache>
 vLLM <source/integration/vllm>
+NVIDIA Dynamo <source/integration/dynamo>
 SGLang HiCache <source/integration/sglang>
 ```
 

docs/source/design_doc/consistency_and_safety.md

@@ -4,19 +4,16 @@
 
 ## 1. Data Visibility
 
-Maru provides **strong consistency** for store operations: once `store()` returns
-successfully, the stored data is immediately retrievable by any other instance.
+Maru publishes an object's location after the writer finishes writing its
+payload. Once `store()` returns successfully, other instances can look up that
+location through the metadata server.
 
-This relies on **write-then-register** ordering. The handler always
-completes these steps in sequence:
+This relies on **write-then-register** ordering. Every store completes these steps in sequence:
 
-1. **Write** — copy data into the CXL memory region.
-2. **Register** — notify the metadata registry that the key now maps to that
-   location.
+1. **Write** — the caller writes the payload into a page from `alloc()`, by CPU through `handle.buf` or by GPU DMA.
+2. **Register** — `store()` records in the metadata registry that the key now maps to that page.
 
-Because registration only occurs after the write is fully committed to shared
-memory, no reader can ever observe a partial or in-progress write. The key simply
-does not exist in the registry until the data is complete.
+The caller must finish its write before calling `store()`. Publishing only after the write completes prevents readers from discovering an in-progress object through the registry, provided the written data is visible to the reader (see {ref}`cross-host-visibility`).
 
 ```mermaid
 sequenceDiagram
@@ -25,21 +22,37 @@ sequenceDiagram
     participant Meta as Metadata Registry
     participant R as Reader (Instance B)
 
-    W->>CXL: 1. Write data to region
-    Note over CXL: Data fully committed
+    W->>CXL: 1. Write data to page (flush CPU writes)
+    Note over CXL: Payload reached the device
     W->>Meta: 2. Register key -> location
     Note over Meta: Key now globally visible
 
     R->>Meta: 3. Lookup key
     Meta-->>R: location (region, offset)
-    R->>CXL: 4. Read data (zero-copy)
-    Note over R: Complete, consistent data
+    R->>CXL: 4. Read data (zero-copy, invalidate before CPU reads)
+    Note over R: Complete data
 ```
 
 The visibility point is when `register_kv` RPC completes. The server holds the
 key in an in-memory registry protected by a lock, ensuring that concurrent
 lookups always see a fully committed entry or no entry at all.
 
+(cross-host-visibility)=
+### Cross-Host Visibility
+
+The ordering above assumes that writes reach the shared memory device and readers cannot consume stale host-cache copies. On a non-coherent multi-host CXL platform, metadata ordering alone does not establish those conditions. See {doc}`../getting_started/bios_setup` for the platform settings that GPU DMA writes depend on.
+
+On Intel GNR, the BIOS option `Allocating Write Flows` controls where device writes to CXL memory, such as GPU DMA writes, land. With the default `Allocating` policy, a write can stay in the writing host's cache, so other hosts read stale data. `Non-Allocating` stops these writes from allocating new lines in the host cache. It does not change lines the host already caches, for example after CPU access to the same memory, so CPU access still needs the explicit write-back and invalidation below.
+
+CPU access to a shared payload can leave cached copies that other hosts do not invalidate, so CPU writers and readers must write back and invalidate explicitly. `flush_range` from `maru_shm._cxl_flush` does both: it runs `clflush` over each cache line of the buffer, then `mfence`.
+
+1. **Writer:** write the payload to `handle.buf`, call `flush_range(handle.buf)`, then `store()`.
+2. **Reader:** `retrieve()`, call `flush_range(result.view)` to drop local cached copies, then read `result.view`.
+
+The {doc}`../getting_started/quick_start` producer and consumer follow this pattern. The CPU-based `examples/basic/producer.py` and `consumer.py` are single-host examples and do not flush.
+
+For multi-node use, validate the deployed transfer path across repeated updates and memory reuse, including any CPU access performed by the runtime or driver.
+
 > **See also:** [MaruHandler Architecture](maru_handler.md)
 
 ---
@@ -144,4 +157,3 @@ mismatch triggers reclamation.
 | Resource Manager crash | New region allocation blocked | WAL + checkpoint replay on restart | None |
 | Network partition (client-server) | Affected client cannot store/retrieve | Client reconnects when network recovers | None |
 | CXL device failure | All data on the device is lost | Not supported -- no cross-device replication | **Total** |
-

docs/source/getting_started/bios_setup.md

@@ -0,0 +1,44 @@
+# System Setup
+
+Maru requires CXL memory exposed as **DEV_DAX** (`/dev/dax*`). Follow your
+server vendor's CXL setup instructions on each host.
+
+## Intel (GNR)
+
+For multi-host KV sharing on Intel Granite Rapids (GNR), configure the BIOS as follows:
+
+- If a **DDIO** control is available, use it to **disable DDIO**.
+- Otherwise, set **`Allocating Write Flows`** to **`Non-Allocating`**.
+
+Option names and menu locations vary by server vendor and BIOS version.
+If neither option is available, consult your server vendor's documentation.
+
+Apply the setting on every participating host, save the configuration, and reboot.
+
+## AMD (Turin)
+
+Complete your platform's CXL setup, then verify DEV_DAX access below.
+
+## Verify after reboot
+
+On every participating host, inspect the CXL/DAX devices:
+
+```bash
+# Install the inspection utilities if needed.
+sudo apt-get install -y cxl daxctl
+sudo cxl list -R -D
+sudo daxctl list
+ls -l /dev/dax*
+```
+
+Confirm that the selected memory is in `devdax` mode and the account running
+Maru has read/write access to its local device. If it appears as `system-ram`,
+follow your platform's instructions to provision it as DEV_DAX before continuing.
+
+For multi-node sharing, also configure the CXL fabric to expose the same memory
+to every host.
+
+BIOS settings alone do not guarantee cross-host cache coherence. See
+{doc}`../design_doc/consistency_and_safety` for the data-visibility requirements.
+
+Once device access is verified, continue to {doc}`installation`.

docs/source/getting_started/examples/dynamo/index.md

@@ -0,0 +1,134 @@
+# Dynamo Examples
+
+Run Dynamo's vLLM backend with Maru to share KV cache between workers.
+The scripts live in `examples/dynamo/single_node/`.
+
+## Prerequisites
+
+- A local CXL DEV_DAX pool and a running `maru-resource-manager` (see {ref}`single-host quick start <quickstart-single-host>`). The example starts its own MaruServer, so you do not need to start one.
+- Two GPUs by default, or enough memory on one GPU for two small-model workers.
+- Maru installed with its KV placement kernels (see {doc}`../../installation`).
+- Dynamo's vLLM backend installed in the **same Python environment** as Maru
+  (see [Dynamo's vLLM installation guide](https://docs.nvidia.com/dynamo/backends/v-llm)).
+
+## P2P KV Cache Sharing
+
+One Dynamo frontend routes requests to two `dynamo.vllm` workers on a single
+host. The first worker stores KV cache in Maru; the second reuses it for the
+same prompt. The launcher disables vLLM's local prefix cache for this check.
+
+This example runs on a single host only. A multi-host Dynamo example is coming soon.
+
+### Automated
+
+```bash
+cd examples/dynamo/single_node
+./single_node_example.sh                     # Qwen/Qwen2.5-0.5B; GPUs 0 and 1
+
+# Both small-model workers on GPU 0
+W0_GPU=0 W1_GPU=0 ./single_node_example.sh
+
+# Another model on the default two GPUs
+./single_node_example.sh meta-llama/Llama-3.1-8B-Instruct
+```
+
+The script starts MaruServer, the frontend, and both workers, runs the sharing
+check, and stops the processes it started. Logs are saved in the example
+directory as `maru_server.log`, `frontend.log`, `w0.log`, and `w1.log`.
+
+**Success:** both workers return the same answer, and the second worker reports
+a nonzero external prefix-cache hit rate. The script exits nonzero if either
+check fails. Latency may not improve for the small default model.
+
+### Step-by-step
+
+In **every terminal**, activate the same environment and run the setup below.
+Choose a new, unused discovery directory for each run and use the same path
+in every terminal:
+
+```bash
+cd examples/dynamo/single_node
+export MODEL=Qwen/Qwen2.5-0.5B
+export DYN_FILE_KV=/tmp/maru-dynamo-manual-run-1
+source env.sh
+```
+
+Run each service in a separate terminal, in this order:
+
+1. Start MaruServer:
+
+   ```bash
+   maru-server --port "$MARU_SERVER_PORT"
+   ```
+
+2. Start the frontend and wait for its health endpoint to respond:
+
+   ```bash
+   ./dynamo_launcher.sh frontend
+   ```
+
+   From another terminal with the same environment:
+
+   ```bash
+   curl --fail "http://localhost:${DYN_HTTP_PORT}/health"
+   ```
+
+3. Start each worker in its own terminal. Wait for `w0` to initialize before
+   starting `w1`. Keep the logs as shown; the verification script reads them:
+
+   ```bash
+   set -o pipefail
+   ./dynamo_launcher.sh worker w0 2>&1 | tee w0.log
+   ```
+
+   In the other worker terminal:
+
+   ```bash
+   set -o pipefail
+   ./dynamo_launcher.sh worker w1 2>&1 | tee w1.log
+   ```
+
+4. Check that the frontend lists both workers. The response should contain two `instances` entries whose `endpoint` is `generate`:
+
+   ```bash
+   curl -s "http://localhost:${DYN_HTTP_PORT}/health"
+   ```
+
+   Then run:
+
+   ```bash
+   ./run_simple_query.sh
+   ```
+
+   The frontend can return HTTP 503 for a short time after the workers register. If the query script fails with a 503, wait a few seconds and run it again.
+
+The query script sends the same prompt to each worker through the frontend
+and checks both the answers and the second worker's external cache hit rate.
+Stop the services with Ctrl+C when finished.
+
+### Configuration
+
+Settings live in `examples/dynamo/single_node/env.sh` and can be overridden
+through the environment:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `DYN_HTTP_PORT` | `14000 + uid` | Frontend HTTP port |
+| `DYN_W0_SYSTEM_PORT`, `DYN_W1_SYSTEM_PORT` | Frontend port + `81`, + `82` | Worker health/metrics ports |
+| `MARU_SERVER_PORT` | `10000 + uid` | Local MaruServer port |
+| `MARU_POOL_SIZE` | `4G` | Initial CXL allocation per handler |
+| `MARU_KV_CHUNK_TOKENS` | `256` | Tokens in a complete stored chunk |
+| `W0_GPU`, `W1_GPU` | `0`, `1` | Worker GPU assignment |
+| `GPU_MEM_UTIL` | `0.3` | vLLM memory utilization per worker |
+| `DYN_FILE_KV` | `/tmp/dynamo_kv_<uid>` | Shared local discovery directory |
+
+Send API requests to the frontend; worker system ports serve health and metrics.
+The automated script clears `DYN_FILE_KV` for file discovery at startup, so use
+a directory dedicated to this example.
+
+## Further reading
+
+- {doc}`../../../integration/dynamo` — Integration architecture
+- {doc}`../../../integration/vllm` — Shared connector settings
+- [Example README](https://github.com/xcena-dev/maru/blob/main/examples/dynamo/single_node/README.md)
+  — Discovery backends, etcd setup, and troubleshooting

docs/source/getting_started/examples/index.md

@@ -1,8 +1,12 @@
-# More Examples
+# Examples
+
+Start with {doc}`../quick_start` for single-host or multi-host KV sharing.
+The examples below cover vLLM, LMCache, and Dynamo serving configurations.
 
 ```{toctree}
 :maxdepth: 2
 
 vllm/index
 lmcache/index
+dynamo/index
 ```

docs/source/getting_started/installation.md

@@ -25,6 +25,8 @@ In a single-node setup, all components run on the same machine. In a multi-node
 - CXL DAX device (`/dev/dax*`) or emulation environment
   - **Multi-node:** All participating nodes must be connected to a shared CXL memory pool (e.g., via CXL switch).
 
+See {doc}`bios_setup` for platform BIOS settings and DEV_DAX verification, and {doc}`../design_doc/consistency_and_safety` for the cache-visibility requirements of cross-host sharing.
+
 ```bash
 sudo apt-get update
 sudo apt-get install -y python3 python3-venv python3-pip git \
@@ -139,6 +141,7 @@ pre-commit run --all-files
 
 <br/>
 
+(installation-verify)=
 ## 2. Verify Installation
 
 Verify that the Maru Python package is installed:
@@ -147,13 +150,22 @@ Verify that the Maru Python package is installed:
 python3 -c "import maru_shm; print('ok')"
 ```
 
+Verify the x86 cache-flush extension used by the {doc}`quick_start` producer and consumer. It is an optional build product, so the install succeeds without it when no C compiler is available:
+
+```bash
+python3 -c "from maru_shm._cxl_flush import HAVE_CLFLUSH; print(HAVE_CLFLUSH)"
+```
+
+`1` means the extension is available. An `ImportError` means it was not built: install a C compiler (`build-essential`) and the headers for your Python interpreter (`python3-dev` for the system Python), then rerun `install.sh`.
+
 If you installed with the Resource Manager, verify the binary:
 
 ```bash
 which maru-resource-manager
 ```
 
-Once installation is verified, proceed to the {doc}`quick_start` guide to start services and run your first store/retrieve.
+Once installation is verified, choose the single-host or multi-host setup in
+{doc}`quick_start` to start services and verify KV sharing.
 
 <br/>
 
@@ -176,6 +188,7 @@ In a multi-node deployment, the Resource Manager, Metadata Server, and MaruHandl
 
 > `maru-server` and `maru-resource-manager` do not need to run on the same node. The diagram above shows the simplest configuration; each can be deployed independently as long as MaruHandler can reach both over the network.
 
+(installation-multi-node-config)=
 ### Configuration
 
 Multi-node requires changing default bind addresses from `127.0.0.1` to a network-accessible address:
@@ -184,6 +197,23 @@ Multi-node requires changing default bind addresses from `127.0.0.1` to a networ
 - **Metadata Server**: change `--host` and set `--rm-address` to the Resource Manager's externally reachable address
 - **MaruHandler**: set `server_url` to the Metadata Server's address. The Resource Manager address is received automatically via handshake.
 
+To run the Resource Manager as the systemd service on a network address, override its command line:
+
+```bash
+sudo systemctl edit maru-resource-manager
+```
+
+```ini
+[Service]
+ExecStart=
+ExecStart=/usr/local/bin/maru-resource-manager --host <node-a-ip> --port 9850
+```
+
+```bash
+sudo systemctl restart maru-resource-manager
+```
+
 > **Security:** When binding to a non-loopback address, auth tokens and device paths are transmitted in plaintext. Use an encrypted tunnel (WireGuard, SSH tunnel, IPsec) in production.
 
-> Multi-node end-to-end examples and deployment guide are coming soon.
+Follow the {ref}`multi-host quickstart <quickstart-multi-host>` to start the
+services and share data between a Python producer and consumer on two hosts.