From ed1d02c2754a22d3da5939e3a20919b95f541a06 Mon Sep 17 00:00:00 2001 From: seohui-XCENA Date: Fri, 2 Oct 2026 18:29:19 +0900 Subject: [PATCH 1/4] docs: add multi-host quick start, system setup, and Dynamo pages Rewrite the quick start around one producer/consumer pair that runs on either a single host or two hosts sharing CXL memory, with explicit flush_range calls for cross-host CPU visibility. Add a System Setup page for DEV_DAX verification and the GNR BIOS setting, Dynamo integration and example pages, and qualify the data-visibility section of the consistency doc for non-coherent multi-host platforms. --- docs/index.md | 10 +- .../design_doc/consistency_and_safety.md | 25 +- docs/source/getting_started/bios_setup.md | 47 ++++ .../getting_started/examples/dynamo/index.md | 128 ++++++++++ docs/source/getting_started/examples/index.md | 6 +- docs/source/getting_started/installation.md | 9 +- docs/source/getting_started/quick_start.md | 222 ++++++++++-------- docs/source/integration/dynamo.md | 24 ++ 8 files changed, 363 insertions(+), 108 deletions(-) create mode 100644 docs/source/getting_started/bios_setup.md create mode 100644 docs/source/getting_started/examples/dynamo/index.md create mode 100644 docs/source/integration/dynamo.md diff --git a/docs/index.md b/docs/index.md index 999d9d4d..528decc9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -94,8 +94,10 @@ Maru (/mɑːruː/) — named after the *maru* (마루), the central open floor i ## Getting Started +- {doc}`System Setup ` - {doc}`Installation ` -- {doc}`Quick Start ` +- {doc}`Quick Start ` — Single-host and multi-host KV sharing +- {doc}`Examples ` - {doc}`Architecture Overview ` ```{toctree} @@ -103,9 +105,10 @@ Maru (/mɑːruː/) — named after the *maru* (마루), the central open floor i :caption: Getting Started :hidden: +System Setup Installation -Quickstart -More Examples +Quick Start +Examples ``` ```{toctree} @@ -127,6 +130,7 @@ CPU/CXL L1 and SSD L2 (Proposal) LMCache vLLM +NVIDIA Dynamo SGLang HiCache ``` diff --git a/docs/source/design_doc/consistency_and_safety.md b/docs/source/design_doc/consistency_and_safety.md index 6e24e9b1..29f203fb 100644 --- a/docs/source/design_doc/consistency_and_safety.md +++ b/docs/source/design_doc/consistency_and_safety.md @@ -4,8 +4,21 @@ ## 1. Data Visibility -Maru provides **strong consistency** for store operations: once `store()` returns -successfully, the stored data is immediately retrievable by any other instance. +Maru publishes an object's location after the writer finishes writing its +payload. Once `store()` returns successfully, other instances can look up that +location through the metadata server. + +The payload-visibility guarantees below assume that writes reach the shared +memory device and readers cannot consume stale host-cache copies. On a +non-coherent multi-host CXL platform, metadata ordering alone does not establish +those conditions. See {doc}`../getting_started/bios_setup` for platform setup. + +CPU access to a shared payload can leave cached copies that other hosts do not +invalidate. Applications using that path need to handle write-back and +invalidation explicitly. The CPU-based `examples/basic/producer.py` and +`consumer.py` are single-host examples and do not implement that protocol. +For multi-node use, validate the deployed transfer path across repeated updates +and memory reuse, including any CPU access performed by the runtime or driver. This relies on **write-then-register** ordering. The handler always completes these steps in sequence: @@ -14,9 +27,10 @@ completes these steps in sequence: 2. **Register** — notify the metadata registry that the key now maps to that location. -Because registration only occurs after the write is fully committed to shared -memory, no reader can ever observe a partial or in-progress write. The key simply -does not exist in the registry until the data is complete. +For a handle-based store, the caller must finish its CPU/GPU write before +registering the handle. Under the visibility conditions above, publishing only +after the write completes prevents readers from discovering an in-progress +object through the registry. ```mermaid sequenceDiagram @@ -144,4 +158,3 @@ mismatch triggers reclamation. | Resource Manager crash | New region allocation blocked | WAL + checkpoint replay on restart | None | | Network partition (client-server) | Affected client cannot store/retrieve | Client reconnects when network recovers | None | | CXL device failure | All data on the device is lost | Not supported -- no cross-device replication | **Total** | - diff --git a/docs/source/getting_started/bios_setup.md b/docs/source/getting_started/bios_setup.md new file mode 100644 index 00000000..71bfac23 --- /dev/null +++ b/docs/source/getting_started/bios_setup.md @@ -0,0 +1,47 @@ +# System Setup + +Maru requires CXL memory exposed as **DEV_DAX** (`/dev/dax*`). Follow your +server vendor's CXL setup instructions on each host. + +If you use **InfiniteMemory**, first follow its +[BIOS configuration guide](https://xcena-dev.github.io/InfiniteMemory_docs/getting-started.html#bios-configuration). + +## Intel (GNR) + +For multi-node KV sharing, apply this setting on each participating GNR host: + +| Setting | Value | +|---------|-------| +| `Allocating Write Flows` | `Non-Allocating` | + +Save the settings and reboot. The menu location depends on your server vendor +and BIOS version. + +## AMD (Turin) + +Complete your platform's CXL setup, then verify DEV_DAX access below. For +InfiniteMemory, use the BIOS configuration guide linked above. + +## Verify after reboot + +On every participating host, inspect the CXL/DAX devices: + +```bash +# Install the inspection utilities if needed. +sudo apt-get install -y cxl-cli daxctl +sudo cxl list -R -D +sudo daxctl list +ls -l /dev/dax* +``` + +Confirm that the selected memory is in `devdax` mode and the account running +Maru has read/write access to its local device. If it appears as `system-ram`, +follow your platform's instructions to provision it as DEV_DAX before continuing. + +For multi-node sharing, also configure the CXL fabric to expose the same memory +to every host. + +BIOS settings alone do not guarantee cross-host cache coherence. See +{doc}`../design_doc/consistency_and_safety` for the data-visibility requirements. + +Once device access is verified, continue to {doc}`installation`. diff --git a/docs/source/getting_started/examples/dynamo/index.md b/docs/source/getting_started/examples/dynamo/index.md new file mode 100644 index 00000000..76c3795b --- /dev/null +++ b/docs/source/getting_started/examples/dynamo/index.md @@ -0,0 +1,128 @@ +# Dynamo Examples + +Run Dynamo's vLLM backend with Maru to share KV cache between workers. +The scripts live in `examples/dynamo/single_node/`. + +## Prerequisites + +- A local CXL DEV_DAX pool and a running `maru-resource-manager` + (see {ref}`single-host quick start `). +- Two GPUs by default, or enough memory on one GPU for two small-model workers. +- Maru installed with its KV placement kernels (see {doc}`../../installation`). +- Dynamo's vLLM backend installed in the **same Python environment** as Maru + (see [Dynamo's vLLM installation guide](https://docs.nvidia.com/dynamo/backends/v-llm)). + +## P2P KV Cache Sharing + +One Dynamo frontend routes requests to two `dynamo.vllm` workers on a single +host. The first worker stores KV cache in Maru; the second reuses it for the +same prompt. The launcher disables vLLM's local prefix cache for this check. + +For Maru setup across hosts, see the +{ref}`multi-host quick start `. + +### Automated + +```bash +cd examples/dynamo/single_node +./single_node_example.sh # Qwen/Qwen2.5-0.5B; GPUs 0 and 1 + +# Both small-model workers on GPU 0 +W0_GPU=0 W1_GPU=0 ./single_node_example.sh + +# Another model on the default two GPUs +./single_node_example.sh meta-llama/Llama-3.1-8B-Instruct +``` + +The script starts MaruServer, the frontend, and both workers, runs the sharing +check, and stops the processes it started. Logs are saved in the example +directory as `maru_server.log`, `frontend.log`, `w0.log`, and `w1.log`. + +**Success:** both workers return the same answer, and the second worker reports +a nonzero external prefix-cache hit rate. The script exits nonzero if either +check fails. Latency may not improve for the small default model. + +### Step-by-step + +In **every terminal**, activate the same environment and run the setup below. +Choose a new, unused discovery directory for each run and use the same path +in every terminal: + +```bash +cd examples/dynamo/single_node +export MODEL=Qwen/Qwen2.5-0.5B +export DYN_FILE_KV=/tmp/maru-dynamo-manual-run-1 +source env.sh +``` + +Run each service in a separate terminal, in this order: + +1. Start MaruServer: + + ```bash + maru-server --port "$MARU_SERVER_PORT" + ``` + +2. Start the frontend and wait for its health endpoint to respond: + + ```bash + ./dynamo_launcher.sh frontend + ``` + + From another terminal with the same environment: + + ```bash + curl --fail "http://localhost:${DYN_HTTP_PORT}/health" + ``` + +3. Start each worker in its own terminal. Wait for `w0` to initialize before + starting `w1`. Keep the logs as shown; the verification script reads them: + + ```bash + set -o pipefail + ./dynamo_launcher.sh worker w0 2>&1 | tee w0.log + ``` + + In the other worker terminal: + + ```bash + set -o pipefail + ./dynamo_launcher.sh worker w1 2>&1 | tee w1.log + ``` + +4. After both workers register and become ready, run: + + ```bash + ./run_simple_query.sh + ``` + +The query script sends the same prompt to each worker through the frontend +and checks both the answers and the second worker's external cache hit rate. +Stop the services with Ctrl+C when finished. + +### Configuration + +Settings live in `examples/dynamo/single_node/env.sh` and can be overridden +through the environment: + +| Variable | Default | Description | +|----------|---------|-------------| +| `DYN_HTTP_PORT` | `14000 + uid` | Frontend HTTP port | +| `DYN_W0_SYSTEM_PORT`, `DYN_W1_SYSTEM_PORT` | Frontend port + `81`, + `82` | Worker health/metrics ports | +| `MARU_SERVER_PORT` | `10000 + uid` | Local MaruServer port | +| `MARU_POOL_SIZE` | `4G` | Initial CXL allocation per handler | +| `MARU_KV_CHUNK_TOKENS` | `256` | Tokens in a complete stored chunk | +| `W0_GPU`, `W1_GPU` | `0`, `1` | Worker GPU assignment | +| `GPU_MEM_UTIL` | `0.3` | vLLM memory utilization per worker | +| `DYN_FILE_KV` | `/tmp/dynamo_kv_` | Shared local discovery directory | + +Send API requests to the frontend; worker system ports serve health and metrics. +The automated script clears `DYN_FILE_KV` for file discovery at startup, so use +a directory dedicated to this example. + +## Further reading + +- {doc}`../../../integration/dynamo` — Integration architecture +- {doc}`../../../integration/vllm` — Shared connector settings +- [Example README](https://github.com/xcena-dev/maru/blob/main/examples/dynamo/single_node/README.md) + — Discovery backends, etcd setup, and troubleshooting diff --git a/docs/source/getting_started/examples/index.md b/docs/source/getting_started/examples/index.md index 74c38376..592265f4 100644 --- a/docs/source/getting_started/examples/index.md +++ b/docs/source/getting_started/examples/index.md @@ -1,8 +1,12 @@ -# More Examples +# Examples + +Start with {doc}`../quick_start` for single-host or multi-host KV sharing. +The examples below cover vLLM, LMCache, and Dynamo serving configurations. ```{toctree} :maxdepth: 2 vllm/index lmcache/index +dynamo/index ``` diff --git a/docs/source/getting_started/installation.md b/docs/source/getting_started/installation.md index 4cfac876..3c0d7f25 100644 --- a/docs/source/getting_started/installation.md +++ b/docs/source/getting_started/installation.md @@ -25,6 +25,9 @@ In a single-node setup, all components run on the same machine. In a multi-node - CXL DAX device (`/dev/dax*`) or emulation environment - **Multi-node:** All participating nodes must be connected to a shared CXL memory pool (e.g., via CXL switch). +See {doc}`bios_setup` for Intel GNR and AMD Turin firmware settings, DAX +verification, and the cache-visibility requirements for cross-host sharing. + ```bash sudo apt-get update sudo apt-get install -y python3 python3-venv python3-pip git \ @@ -153,7 +156,8 @@ If you installed with the Resource Manager, verify the binary: which maru-resource-manager ``` -Once installation is verified, proceed to the {doc}`quick_start` guide to start services and run your first store/retrieve. +Once installation is verified, choose the single-host or multi-host setup in +{doc}`quick_start` to start services and verify KV sharing.
@@ -186,4 +190,5 @@ Multi-node requires changing default bind addresses from `127.0.0.1` to a networ > **Security:** When binding to a non-loopback address, auth tokens and device paths are transmitted in plaintext. Use an encrypted tunnel (WireGuard, SSH tunnel, IPsec) in production. -> Multi-node end-to-end examples and deployment guide are coming soon. +Follow the {ref}`multi-host quickstart ` to start the +services and share data between a Python producer and consumer on two hosts. diff --git a/docs/source/getting_started/quick_start.md b/docs/source/getting_started/quick_start.md index 16ed0fee..fdcbee9c 100644 --- a/docs/source/getting_started/quick_start.md +++ b/docs/source/getting_started/quick_start.md @@ -1,160 +1,190 @@ -# Quickstart +# Quick Start -## 1. Start the Resource Manager +Start the services for your setup, then run the same Python producer and +consumer below. No GPU or inference engine is required. -The resource manager must be running before any other Maru service. +| Setup | Required environment | Where the clients run | +|-------|----------------------|-----------------------| +| {ref}`Single-host ` | One x86 host with a CXL DEV_DAX device | Two terminals on the same host | +| {ref}`Multi-host ` | Two x86 hosts connected to shared CXL memory through a CXL switch/fabric | Producer on Node A, consumer on Node B | -**Production** — start as a systemd daemon: +Complete {doc}`bios_setup` and {doc}`installation` first. Activate the Maru +Python environment in each terminal. Both setups use port `9850` for the +Resource Manager and `5555` for MaruServer. -```bash -# Default (127.0.0.1:9850) -sudo systemctl start maru-resource-manager +## 1. Start the services -# With custom host/port -sudo systemctl edit maru-resource-manager +Choose one setup below. Keep the services running while using the clients. -[Service] -ExecStart= -ExecStart=/usr/local/bin/maru-resource-manager --host --port +(quickstart-single-host)= +### Single-host -sudo systemctl restart maru-resource-manager +Start the Resource Manager as a systemd service: +```bash +sudo systemctl start maru-resource-manager ``` -**Development/debugging** — run directly with custom options: +Alternatively, with the service stopped, run it in a dedicated terminal: ```bash -# Default (127.0.0.1:9850) -sudo maru-resource-manager --log-level debug - -# With custom host/port -sudo maru-resource-manager --host --port --log-level debug +sudo maru-resource-manager --host 127.0.0.1 --port 9850 ``` -> If you change the RM port, pass the same address to `maru-server`: `maru-server --rm-address 127.0.0.1:9851` -> -> See {doc}`../design_doc/maru_resource_manager` for the full list of CLI options. - -## 2. Start the Metadata Server +Wait for device discovery to finish, then start MaruServer in another terminal: ```bash -# Default (127.0.0.1:5555, connects to resource manager at 127.0.0.1:9850) maru-server +``` -# With custom host/port -maru-server --host --port - -# With custom resource manager address (default: 127.0.0.1:9850) -maru-server --rm-address : +In **both client terminals**, set the server address: -# With debug logging -maru-server --log-level DEBUG +```bash +export MARU_SERVER_URL=tcp://127.0.0.1:5555 ``` -## 3. Run a Client Example +Continue to {ref}`the producer and consumer `. -> Both `maru-resource-manager` and `maru-server` must be running before proceeding. +(quickstart-multi-host)= +### Multi-host -### Zero-Copy Store & Retrieve +This example runs both services and the producer on **Node A**, with the +consumer on **Node B**: -```python -from maru import MaruConfig, MaruHandler +```mermaid +flowchart TB + subgraph A["Node A · 192.0.2.10"] + P["Producer
MaruHandler"] + Services["MaruServer: 5555
Resource Manager: 9850"] + P -.-> Services + end + subgraph B["Node B"] + C["Consumer
MaruHandler"] + end + + C -.-> Services + P --- Switch["CXL switch / fabric"] + C --- Switch + Switch --- Memory[("Shared CXL memory")] +``` -config = MaruConfig( - server_url="tcp://localhost:5555", - pool_size=1024 * 1024 * 100, # Request 100MB from the CXL memory pool -) +Dashed lines are TCP connections to the services on Node A. Solid lines show +each host's direct access to the shared memory through its local DEV_DAX device +and the CXL fabric. -with MaruHandler(config) as handler: - data = b"A" * (1024 * 1024) # 1MB KV chunk +Both hosts must have read/write access to the **same physical CXL memory at +matching device offsets**. Install Maru with `./install.sh` on Node A and +`./install.sh --no-rm` on Node B. - # 1. Allocate a page in CXL shared memory - handle = handler.alloc(size=len(data)) +Replace `192.0.2.10` with Node A's reachable IP. Node B must be able to reach +Node A on TCP ports `5555` and `9850`. - # 2. Write directly to CXL memory (mmap — no intermediate buffer) - handle.buf[:] = data +Run one Resource Manager for the shared pool on **Node A**. With the systemd +service stopped, start it in a dedicated terminal: - # 3. Register the key — only metadata (key → region, offset) is sent - handler.store(key=42, handle=handle) +```bash +sudo maru-resource-manager --host 192.0.2.10 --port 9850 +``` + +Wait for device discovery to finish, then start MaruServer in another terminal +on **Node A**: - # Retrieve: returns a memoryview pointing into CXL memory - result = handler.retrieve(key=42) - assert result is not None - assert bytes(result.view[:5]) == b"AAAAA" +```bash +maru-server --host 192.0.2.10 --port 5555 \ + --rm-address 192.0.2.10:9850 \ + --dax-path /dev/dax0.0 ``` -The three steps — `alloc()`, write to `handle.buf`, `store(handle=)` — make Maru's control/data plane separation explicit: +Replace `/dev/dax0.0` with the shared device's path on Node A. Local device +names may differ on Node B; Maru resolves them by UUID. The advertised +`--rm-address` must be reachable from Node B, so use Node A's IP here. -- **Control plane**: `alloc()` requests a memory region from the Resource Manager; `store()` registers the key's location metadata (~tens of bytes) with MaruServer. -- **Data plane**: `handle.buf[:]` reads/writes directly on mmap'd CXL memory. No server involved. +In **both client terminals** (one on each host), set the same server address: -> Full runnable example: `examples/basic/single_instance.py` +```bash +export MARU_SERVER_URL=tcp://192.0.2.10:5555 +``` -### Cross-Instance Sharing +Continue to {ref}`the producer and consumer `. -This is the core of Maru — two independent processes sharing KV cache through CXL shared memory with zero data copy. **Metadata travels, data doesn't.** +(quickstart-clients)= +## 2. Run the producer and consumer -```mermaid -flowchart LR - P[Producer] -->|"alloc + write"| CXL[(CXL Memory)] - P -->|"store(key)"| S[MaruServer] - C[Consumer] -->|"retrieve(key)"| S - S -.->|"location"| C - C -->|"read"| CXL -``` +Use these same snippets for either setup. Keep `MARU_SERVER_URL` set as shown +above in each client terminal. For single-host, run both clients on the same +host; for multi-host, run the producer on Node A and the consumer on Node B. -**Terminal 1** — Producer: allocate, write, register metadata: +### Producer: write and store + +Run this first and keep it running until the consumer finishes: ```python +import os + from maru import MaruConfig, MaruHandler +from maru_shm._cxl_flush import HAVE_CLFLUSH, flush_range + +assert HAVE_CLFLUSH, "This CPU example requires x86 cache-flush support" config = MaruConfig( - server_url="tcp://localhost:5555", + server_url=os.environ["MARU_SERVER_URL"], instance_id="producer", - pool_size=1024 * 1024 * 100, + pool_size=100 * 1024 * 1024, ) with MaruHandler(config) as handler: - for i, key in enumerate([1001, 1002, 1003]): - data = bytes([ord("A") + i]) * (1024 * 1024) - - handle = handler.alloc(size=len(data)) - handle.buf[:] = data - handler.store(key=key, handle=handle) + data = b"A" * (1024 * 1024) + handle = handler.alloc(size=len(data)) + handle.buf[:] = data + flush_range(handle.buf) # Make the CPU write visible before publishing. + handler.store(key="quickstart-demo", handle=handle) - input("Press Enter to exit...") + print("Stored 1 MiB. Run the consumer.") + input("Press Enter after the consumer finishes...") ``` -**Terminal 2** — Consumer: retrieve from CXL (zero copy): +### Consumer: retrieve and verify + +Run this in the other client terminal: ```python +import os + from maru import MaruConfig, MaruHandler +from maru_shm._cxl_flush import HAVE_CLFLUSH, flush_range + +assert HAVE_CLFLUSH, "This CPU example requires x86 cache-flush support" config = MaruConfig( - server_url="tcp://localhost:5555", + server_url=os.environ["MARU_SERVER_URL"], instance_id="consumer", - pool_size=1024 * 1024 * 100, + pool_size=100 * 1024 * 1024, ) with MaruHandler(config) as handler: - for key in [1001, 1002, 1003]: - result = handler.retrieve(key=key) - - # result.view points directly into Producer's CXL region (mapped read-only). - # No data was copied — consumer reads the same physical memory. - assert result is not None - print(f"key={key}: {len(result.view)} bytes, char={chr(result.view[0])!r}") + result = handler.retrieve(key="quickstart-demo") + assert result is not None, "Run the producer first" + flush_range(result.view) # Discard local cached copies before reading. + assert bytes(result.view) == b"A" * (1024 * 1024) + print("Success: read the producer's data from shared CXL memory") ``` -> Full runnable examples: `examples/basic/producer.py` + `examples/basic/consumer.py` -> -> Single-script version: `examples/basic/cross_instance.py` +The consumer should print `Success: read the producer's data from shared CXL memory`. +The `flush_range` calls support CPU data visibility across hosts and are used +in both setups. They require Maru's x86 cache-flush extension, built during +{doc}`installation`. See {doc}`../design_doc/consistency_and_safety` for details. + +## 3. Stop the example +Press Enter in the producer terminal to exit. Stop MaruServer, then stop the +Resource Manager if you started it for this example. Use Ctrl+C for foreground +processes, or `sudo systemctl stop maru-resource-manager` for the systemd service. -## Next Steps +## Next steps -- {doc}`installation` — Multi-node configuration -- [LMCache Examples](examples/lmcache/index.md) — Use Maru as a shared KV cache backend for LMCache/vLLM -- [Architecture Overview](../design_doc/architecture_overview.md) — System architecture and component interactions -- [API Reference](../api_reference/api.md) — Python API documentation -- [Configuration](../api_reference/config.md) — Configuration options +- {doc}`examples/vllm/index` — More vLLM serving examples +- {doc}`examples/dynamo/index` — Run Dynamo with Maru-backed vLLM workers +- {doc}`examples/lmcache/index` — Use Maru as a shared KV cache backend for LMCache +- {doc}`../design_doc/architecture_overview` — System architecture and component interactions +- {doc}`../api_reference/api` — Python API +- {doc}`../api_reference/config` — Configuration options diff --git a/docs/source/integration/dynamo.md b/docs/source/integration/dynamo.md new file mode 100644 index 00000000..7321586c --- /dev/null +++ b/docs/source/integration/dynamo.md @@ -0,0 +1,24 @@ +# NVIDIA Dynamo + +Dynamo connects to Maru through its **vLLM backend**. Configure each +`dynamo.vllm` worker with the same `MaruKVConnector` and `--kv-transfer-config` +used for standalone vLLM. Dynamo routes requests to the workers, and the +connector handles KV cache sharing through Maru. + +```mermaid +flowchart TB + Client["OpenAI API client"] --> FE["Dynamo frontend"] + subgraph W0["Dynamo worker A"] + V0["vLLM"] --> C0["MaruKVConnector"] + end + subgraph W1["Dynamo worker B"] + V1["vLLM"] --> C1["MaruKVConnector"] + end + FE --> V0 & V1 + C0 & C1 -.->|"metadata"| MS["MaruServer"] + C0 & C1 <-->|"KV payload"| CXL[("CXL shared memory")] +``` + +See {doc}`vllm` for connector settings and the +{doc}`Dynamo example <../getting_started/examples/dynamo/index>` for setup and +execution instructions. From 025da3b97e733167fd542d80115bcf153e592f00 Mon Sep 17 00:00:00 2001 From: seohui-XCENA Date: Fri, 2 Oct 2026 18:49:52 +0900 Subject: [PATCH 2/4] docs: tighten setup, visibility, and Dynamo instructions - System Setup: install the cxl package; Ubuntu 24.04 has no cxl-cli. - Consistency: put write-then-register ahead of the caveats, describe every store as a caller write followed by store(), and add a Cross-Host Visibility section with the flush_range writer/reader pattern and what Allocating Write Flows does on Intel GNR. - Installation: check the optional cache-flush extension, and show a systemd override for running the Resource Manager on a network address. - Quick start: wait for the Resource Manager's ready log line, and stop the systemd service before starting a second Resource Manager on a different address for multi-host. - Dynamo: mark the example as single-host with a multi-host example to follow, note that it starts its own MaruServer, and check worker registration before running the query. --- .../design_doc/consistency_and_safety.md | 53 +++++++++---------- docs/source/getting_started/bios_setup.md | 2 +- .../getting_started/examples/dynamo/index.md | 16 ++++-- docs/source/getting_started/installation.md | 29 +++++++++- docs/source/getting_started/quick_start.md | 15 +++--- 5 files changed, 72 insertions(+), 43 deletions(-) diff --git a/docs/source/design_doc/consistency_and_safety.md b/docs/source/design_doc/consistency_and_safety.md index 29f203fb..e00f9cad 100644 --- a/docs/source/design_doc/consistency_and_safety.md +++ b/docs/source/design_doc/consistency_and_safety.md @@ -8,29 +8,12 @@ Maru publishes an object's location after the writer finishes writing its payload. Once `store()` returns successfully, other instances can look up that location through the metadata server. -The payload-visibility guarantees below assume that writes reach the shared -memory device and readers cannot consume stale host-cache copies. On a -non-coherent multi-host CXL platform, metadata ordering alone does not establish -those conditions. See {doc}`../getting_started/bios_setup` for platform setup. - -CPU access to a shared payload can leave cached copies that other hosts do not -invalidate. Applications using that path need to handle write-back and -invalidation explicitly. The CPU-based `examples/basic/producer.py` and -`consumer.py` are single-host examples and do not implement that protocol. -For multi-node use, validate the deployed transfer path across repeated updates -and memory reuse, including any CPU access performed by the runtime or driver. - -This relies on **write-then-register** ordering. The handler always -completes these steps in sequence: - -1. **Write** — copy data into the CXL memory region. -2. **Register** — notify the metadata registry that the key now maps to that - location. - -For a handle-based store, the caller must finish its CPU/GPU write before -registering the handle. Under the visibility conditions above, publishing only -after the write completes prevents readers from discovering an in-progress -object through the registry. +This relies on **write-then-register** ordering. Every store completes these steps in sequence: + +1. **Write** — the caller writes the payload into a page from `alloc()`, by CPU through `handle.buf` or by GPU DMA. +2. **Register** — `store()` records in the metadata registry that the key now maps to that page. + +The caller must finish its write before calling `store()`. Publishing only after the write completes prevents readers from discovering an in-progress object through the registry, provided the written data is visible to the reader (see {ref}`cross-host-visibility`). ```mermaid sequenceDiagram @@ -39,21 +22,37 @@ sequenceDiagram participant Meta as Metadata Registry participant R as Reader (Instance B) - W->>CXL: 1. Write data to region - Note over CXL: Data fully committed + W->>CXL: 1. Write data to page (flush CPU writes) + Note over CXL: Payload reached the device W->>Meta: 2. Register key -> location Note over Meta: Key now globally visible R->>Meta: 3. Lookup key Meta-->>R: location (region, offset) - R->>CXL: 4. Read data (zero-copy) - Note over R: Complete, consistent data + R->>CXL: 4. Read data (zero-copy, invalidate before CPU reads) + Note over R: Complete data ``` The visibility point is when `register_kv` RPC completes. The server holds the key in an in-memory registry protected by a lock, ensuring that concurrent lookups always see a fully committed entry or no entry at all. +(cross-host-visibility)= +### Cross-Host Visibility + +The ordering above assumes that writes reach the shared memory device and readers cannot consume stale host-cache copies. On a non-coherent multi-host CXL platform, metadata ordering alone does not establish those conditions. See {doc}`../getting_started/bios_setup` for the platform settings that GPU DMA writes depend on. + +On Intel GNR, the BIOS option `Allocating Write Flows` controls where device writes to CXL memory, such as GPU DMA writes, land. With the default `Allocating` policy, a write can stay in the writing host's cache, so other hosts read stale data. `Non-Allocating` stops these writes from allocating new lines in the host cache. It does not change lines the host already caches, for example after CPU access to the same memory, so CPU access still needs the explicit write-back and invalidation below. + +CPU access to a shared payload can leave cached copies that other hosts do not invalidate, so CPU writers and readers must write back and invalidate explicitly. `flush_range` from `maru_shm._cxl_flush` does both: it runs `clflush` over each cache line of the buffer, then `mfence`. + +1. **Writer:** write the payload to `handle.buf`, call `flush_range(handle.buf)`, then `store()`. +2. **Reader:** `retrieve()`, call `flush_range(result.view)` to drop local cached copies, then read `result.view`. + +The {doc}`../getting_started/quick_start` producer and consumer follow this pattern. The CPU-based `examples/basic/producer.py` and `consumer.py` are single-host examples and do not flush. + +For multi-node use, validate the deployed transfer path across repeated updates and memory reuse, including any CPU access performed by the runtime or driver. + > **See also:** [MaruHandler Architecture](maru_handler.md) --- diff --git a/docs/source/getting_started/bios_setup.md b/docs/source/getting_started/bios_setup.md index 71bfac23..f3c1ed48 100644 --- a/docs/source/getting_started/bios_setup.md +++ b/docs/source/getting_started/bios_setup.md @@ -28,7 +28,7 @@ On every participating host, inspect the CXL/DAX devices: ```bash # Install the inspection utilities if needed. -sudo apt-get install -y cxl-cli daxctl +sudo apt-get install -y cxl daxctl sudo cxl list -R -D sudo daxctl list ls -l /dev/dax* diff --git a/docs/source/getting_started/examples/dynamo/index.md b/docs/source/getting_started/examples/dynamo/index.md index 76c3795b..927f3627 100644 --- a/docs/source/getting_started/examples/dynamo/index.md +++ b/docs/source/getting_started/examples/dynamo/index.md @@ -5,8 +5,7 @@ The scripts live in `examples/dynamo/single_node/`. ## Prerequisites -- A local CXL DEV_DAX pool and a running `maru-resource-manager` - (see {ref}`single-host quick start `). +- A local CXL DEV_DAX pool and a running `maru-resource-manager` (see {ref}`single-host quick start `). The example starts its own MaruServer, so you do not need to start one. - Two GPUs by default, or enough memory on one GPU for two small-model workers. - Maru installed with its KV placement kernels (see {doc}`../../installation`). - Dynamo's vLLM backend installed in the **same Python environment** as Maru @@ -18,8 +17,7 @@ One Dynamo frontend routes requests to two `dynamo.vllm` workers on a single host. The first worker stores KV cache in Maru; the second reuses it for the same prompt. The launcher disables vLLM's local prefix cache for this check. -For Maru setup across hosts, see the -{ref}`multi-host quick start `. +This example runs on a single host only. A multi-host Dynamo example is coming soon. ### Automated @@ -90,12 +88,20 @@ Run each service in a separate terminal, in this order: ./dynamo_launcher.sh worker w1 2>&1 | tee w1.log ``` -4. After both workers register and become ready, run: +4. Check that the frontend lists both workers. The response should contain two `instances` entries whose `endpoint` is `generate`: + + ```bash + curl -s "http://localhost:${DYN_HTTP_PORT}/health" + ``` + + Then run: ```bash ./run_simple_query.sh ``` + The frontend can return HTTP 503 for a short time after the workers register. If the query script fails with a 503, wait a few seconds and run it again. + The query script sends the same prompt to each worker through the frontend and checks both the answers and the second worker's external cache hit rate. Stop the services with Ctrl+C when finished. diff --git a/docs/source/getting_started/installation.md b/docs/source/getting_started/installation.md index 3c0d7f25..5035936d 100644 --- a/docs/source/getting_started/installation.md +++ b/docs/source/getting_started/installation.md @@ -25,8 +25,7 @@ In a single-node setup, all components run on the same machine. In a multi-node - CXL DAX device (`/dev/dax*`) or emulation environment - **Multi-node:** All participating nodes must be connected to a shared CXL memory pool (e.g., via CXL switch). -See {doc}`bios_setup` for Intel GNR and AMD Turin firmware settings, DAX -verification, and the cache-visibility requirements for cross-host sharing. +See {doc}`bios_setup` for platform BIOS settings and DEV_DAX verification, and {doc}`../design_doc/consistency_and_safety` for the cache-visibility requirements of cross-host sharing. ```bash sudo apt-get update @@ -142,6 +141,7 @@ pre-commit run --all-files
+(installation-verify)= ## 2. Verify Installation Verify that the Maru Python package is installed: @@ -150,6 +150,14 @@ Verify that the Maru Python package is installed: python3 -c "import maru_shm; print('ok')" ``` +Verify the x86 cache-flush extension used by the {doc}`quick_start` producer and consumer. It is an optional build product, so the install succeeds without it when no C compiler is available: + +```bash +python3 -c "from maru_shm._cxl_flush import HAVE_CLFLUSH; print(HAVE_CLFLUSH)" +``` + +`1` means the extension is available. An `ImportError` means it was not built: install a C compiler (`build-essential`) and the headers for your Python interpreter (`python3-dev` for the system Python), then rerun `install.sh`. + If you installed with the Resource Manager, verify the binary: ```bash @@ -180,6 +188,7 @@ In a multi-node deployment, the Resource Manager, Metadata Server, and MaruHandl > `maru-server` and `maru-resource-manager` do not need to run on the same node. The diagram above shows the simplest configuration; each can be deployed independently as long as MaruHandler can reach both over the network. +(installation-multi-node-config)= ### Configuration Multi-node requires changing default bind addresses from `127.0.0.1` to a network-accessible address: @@ -188,6 +197,22 @@ Multi-node requires changing default bind addresses from `127.0.0.1` to a networ - **Metadata Server**: change `--host` and set `--rm-address` to the Resource Manager's externally reachable address - **MaruHandler**: set `server_url` to the Metadata Server's address. The Resource Manager address is received automatically via handshake. +To run the Resource Manager as the systemd service on a network address, override its command line: + +```bash +sudo systemctl edit maru-resource-manager +``` + +```ini +[Service] +ExecStart= +ExecStart=/usr/local/bin/maru-resource-manager --host --port 9850 +``` + +```bash +sudo systemctl restart maru-resource-manager +``` + > **Security:** When binding to a non-loopback address, auth tokens and device paths are transmitted in plaintext. Use an encrypted tunnel (WireGuard, SSH tunnel, IPsec) in production. Follow the {ref}`multi-host quickstart ` to start the diff --git a/docs/source/getting_started/quick_start.md b/docs/source/getting_started/quick_start.md index fdcbee9c..2034a0eb 100644 --- a/docs/source/getting_started/quick_start.md +++ b/docs/source/getting_started/quick_start.md @@ -31,7 +31,7 @@ Alternatively, with the service stopped, run it in a dedicated terminal: sudo maru-resource-manager --host 127.0.0.1 --port 9850 ``` -Wait for device discovery to finish, then start MaruServer in another terminal: +Wait until the Resource Manager logs `ready — listening on 127.0.0.1:9850` (for the systemd service, check `journalctl -u maru-resource-manager`), then start MaruServer in another terminal: ```bash maru-server @@ -79,15 +79,16 @@ matching device offsets**. Install Maru with `./install.sh` on Node A and Replace `192.0.2.10` with Node A's reachable IP. Node B must be able to reach Node A on TCP ports `5555` and `9850`. -Run one Resource Manager for the shared pool on **Node A**. With the systemd -service stopped, start it in a dedicated terminal: +Run one Resource Manager for the shared pool on **Node A**. Stop the systemd service first: a service still bound to `127.0.0.1:9850` does not conflict with a second Resource Manager on `192.0.2.10:9850`, and the two would manage the same devices and state directory. Then start it in a dedicated terminal: ```bash +sudo systemctl stop maru-resource-manager sudo maru-resource-manager --host 192.0.2.10 --port 9850 ``` -Wait for device discovery to finish, then start MaruServer in another terminal -on **Node A**: +To run it as the systemd service instead, see the {ref}`multi-node configuration `. + +Wait until it logs `ready — listening on 192.0.2.10:9850`, then start MaruServer in another terminal on **Node A**: ```bash maru-server --host 192.0.2.10 --port 5555 \ @@ -170,9 +171,7 @@ with MaruHandler(config) as handler: ``` The consumer should print `Success: read the producer's data from shared CXL memory`. -The `flush_range` calls support CPU data visibility across hosts and are used -in both setups. They require Maru's x86 cache-flush extension, built during -{doc}`installation`. See {doc}`../design_doc/consistency_and_safety` for details. +The `flush_range` calls make CPU writes and reads visible across hosts and are used in both setups. They require Maru's x86 cache-flush extension; if the import fails, see {ref}`verifying the installation `. See {doc}`../design_doc/consistency_and_safety` for the write-back and invalidation pattern. ## 3. Stop the example From 6c0ef87921c143ada56fb638734e9083d127e964 Mon Sep 17 00:00:00 2001 From: seohui-XCENA Date: Wed, 7 Oct 2026 11:29:30 +0900 Subject: [PATCH 3/4] docs: simplify system setup guide --- docs/source/getting_started/bios_setup.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/source/getting_started/bios_setup.md b/docs/source/getting_started/bios_setup.md index f3c1ed48..307fcc16 100644 --- a/docs/source/getting_started/bios_setup.md +++ b/docs/source/getting_started/bios_setup.md @@ -3,9 +3,6 @@ Maru requires CXL memory exposed as **DEV_DAX** (`/dev/dax*`). Follow your server vendor's CXL setup instructions on each host. -If you use **InfiniteMemory**, first follow its -[BIOS configuration guide](https://xcena-dev.github.io/InfiniteMemory_docs/getting-started.html#bios-configuration). - ## Intel (GNR) For multi-node KV sharing, apply this setting on each participating GNR host: @@ -19,8 +16,7 @@ and BIOS version. ## AMD (Turin) -Complete your platform's CXL setup, then verify DEV_DAX access below. For -InfiniteMemory, use the BIOS configuration guide linked above. +Complete your platform's CXL setup, then verify DEV_DAX access below. ## Verify after reboot From 83d6d59edfe977f9c7092b2daad130e72b8703d6 Mon Sep 17 00:00:00 2001 From: seohui-XCENA Date: Wed, 7 Oct 2026 11:56:27 +0900 Subject: [PATCH 4/4] docs: clarify Intel BIOS options for multi-host sharing --- docs/source/getting_started/bios_setup.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/source/getting_started/bios_setup.md b/docs/source/getting_started/bios_setup.md index 307fcc16..3ceac525 100644 --- a/docs/source/getting_started/bios_setup.md +++ b/docs/source/getting_started/bios_setup.md @@ -5,14 +5,15 @@ server vendor's CXL setup instructions on each host. ## Intel (GNR) -For multi-node KV sharing, apply this setting on each participating GNR host: +For multi-host KV sharing on Intel Granite Rapids (GNR), configure the BIOS as follows: -| Setting | Value | -|---------|-------| -| `Allocating Write Flows` | `Non-Allocating` | +- If a **DDIO** control is available, use it to **disable DDIO**. +- Otherwise, set **`Allocating Write Flows`** to **`Non-Allocating`**. -Save the settings and reboot. The menu location depends on your server vendor -and BIOS version. +Option names and menu locations vary by server vendor and BIOS version. +If neither option is available, consult your server vendor's documentation. + +Apply the setting on every participating host, save the configuration, and reboot. ## AMD (Turin)