[youngrok-XCENA]

🤔 Background & Motivation (Why)

With this PR, Maru sends the memory node's device a read hint: where a request's KV pages are, looked up before the load starts. The device can prefetch those pages (on a CXL-SSD, from the SSD into DRAM), and the load waits until each page is ready before its RDMA READ. Writes use the same hooks to have pages held in DRAM and written back.

• A CXL-SSD keeps a DRAM cache in front of an SSD. An RDMA READ runs at DRAM speed only on a page already prefetched from the SSD into DRAM. An RDMA WRITE needs its page held in DRAM, and written back to the SSD before the page holds new data.
• Only software next to the device can prefetch, hold or write back a page, and it must know early which pages are coming. The GPU node reaches the memory only through RDMA.
• Maru knows early: vLLM and Maru know which pages a request will load while the request waits for the running engine step. Yet Maru tells the device nothing, and it transfers each page as soon as maru-server grants the keys.

This PR lets a device package learn about Maru's transfers in advance and hold each one until its pages are ready. It adds two vendor-neutral extension points: an access policy on the GPU node that decides when each transfer runs, and a region extension in the memory node's agent that drives the device. Any memory that needs pages prepared before access can use them; CXL-SSD is the case that motivated them, and the measurements below cover its read side.

sequenceDiagram
  participant V as vLLM + Maru (GPU node)
  participant D as CXL-SSD (memory node)
  rect rgb(245,235,235)
  Note over V,D: Today
  Note over V: request waits for the running step
  V->>D: RDMA READ pages 1..n
  Note over D: pages still on the SSD, READs wait
  end
  rect rgb(235,245,235)
  Note over V,D: With this PR (hint on arrival, opt-in)
  V->>D: read hint on arrival: pages 1..n
  Note over V: request waits for the running step
  Note over D: prefetch pages 1..n from the SSD into DRAM
  V->>D: RDMA READ pages 1..n as they are ready
  Note over D: served from DRAM
  end

Loading

With an out-of-tree package for one CXL-SSD, the extra time to first token of a reused prefix, compared with the same device holding everything in its cache, was:

When the memory learns which pages a load reads	extra time to first token
Never (Maru today)	about +7% with the device's own read-ahead, several times slower without it
When the worker starts the load (no hint)	+3.7% (+8.1 ms)
When vLLM receives the request (hint on arrival)	+0.5% (+1.1 ms)

The rows come from different stages of a separate prototype, not this branch, so they are not one controlled comparison.

🏗️ Design Changes

Grey boxes are Maru; orange boxes are the installed package, selected by name. Every message between the package's two halves passes through Maru.

flowchart TB
  subgraph gpu["GPU node"]
    direction LR
    pol["access policy<br/>(maru.access_policies)"]
    mc["Maru in vLLM<br/>(scheduler, workers)"]
  end
  subgraph mem["memory node"]
    direction LR
    ms["maru-server"]
    ag["remote CXL agent"]
    ext["region extension<br/>(maru.region_extensions)"]
    dev[("CXL-SSD<br/>DRAM cache + SSD")]
  end
  mc -->|"① pages to be read or written<br/>(read hint, then transfer)"| pol
  pol -->|"② when to transfer,<br/>messages for the extension"| mc
  mc -->|"lookup, lease"| ms
  mc -->|"carries messages"| ag
  ag -->|"③ after checking the pages"| ext
  ext -.->|"prefetch, hold,<br/>write back"| dev
  mc ==>|"RDMA READ / WRITE"| dev
  classDef maru fill:#e8e8e8,stroke:#555,color:#111
  classDef pkg fill:#fbe3cc,stroke:#c06000,color:#111
  class mc,ms,ag maru
  class pol,ext pkg

Loading

• Access policy (GPU node). Maru tells it which pages each transfer touches (①). The policy decides when each transfer runs (②): it can delay a READ until its page is prefetched and a WRITE until its pages are held, but cannot skip or reorder a READ. It also decides when a page given back may hold new data, for example after its write-back. It sees pages, never keys.
• Region extension (memory node). The remote CXL agent passes it the policy's messages after checking that every page lies in a region the agent maps (③). The extension drives the device: it prefetches, holds and writes back pages.
• Read hints. vLLM's scheduler tells the policy which pages a request will read before the worker loads them. By default it does so at the scheduler's first lookup of the request. An opt-in setting does it when vLLM receives the request, which adds the time the request waits for the running engine step. Both need vLLM's asynchronous loading.
• Scope. The need is not specific to remote_cxl: a CXL-SSD attached to the GPU node's own host has the same problem, and engines other than vLLM also know their requests early. This PR covers remote_cxl first because there nothing else can help: the GPU node cannot reach the device software at all, and Maru's storage client itself issues every RDMA transfer, so it can hold one back. Read hints are sent only by the vLLM connector; LMCache and SGLang on remote_cxl get the policy's timing but no hints. On the local path the engine's connector, not Maru, copies the bytes, and the existing handler plugins see a batch only after its lookup; bringing that path under the same policy is left for a follow-up.
• Why the existing handler plugins do not fit. They observe a batch after its lookup in the engine's process. They cannot hold back a transfer, they have no counterpart next to the memory, and remote_cxl loads do not reach them.
• Safe to ignore. A hint is advice: it never permits a write, and the load still leases and reads every page. When a policy step fails, Maru takes an outcome it already has: the hits load as misses, or the keys are not stored. Without a package, nothing changes in the storage paths.

📝 Implementation Details

Hook contracts, locks and failure outcomes are documented in the module docstrings and docs/source/design_doc/access_policies.md.

• Access policy: maru_handler/storage/access_policy.py, entry-point group maru.access_policies, selected by maru_access_policy / maru_access_policy_config (remote_cxl only). Hooks: read(channel, ranges, transfer), write_batch_limit, prepare_write, written, completed_write, reusable, optional read_hint / read_hint_end. A policy calls transfer(start, end) in order until every range is read. It sends messages with AccessChannel.request(action, ranges, ...); action is opaque to Maru, and a message may be delivered twice. With a policy, READs are grouped per region, and WRITEs are batched by write_batch_limit.
• Region extension: maru_remote_cxl_agent/extension.py, entry-point group maru.region_extensions, selected by --region-extension. Hooks: validate_access and mapped when a region is first mapped, request, retire, close. request and retire run under the agent's lock. A region the extension still holds stays out of use until retire answers True. The agent exits if the named extension cannot load.
• Agent checks (service.py): every range must match its region's handle and fit in it, and a read handle cannot prepare a write. hint: true is accepted from any connection for a mapped region that some connection holds, and never with write, so the extension must treat every hint as read-only.
• Read hints (remote_cxl/hints.py, client.py, router.py, handler.py): MaruHandler.hint_read, end_read_hint, set_read_request. A maru-read-hints thread per storage client probes maru-server and calls the policy. It opens its own authenticated connections to the agents, so the scheduler needs the agent key file. Above 1024 queued hints, new hints are dropped; ends are never dropped.
• maru-server (replica_directory.py): probe with locations: true also returns each remote CXL hit's location and its region's read handle, without a lease.
• vLLM connector (maru_vllm/connector.py): hints at the first asynchronous hit and ends the hint in request_finished. Hints need maru_async_load. maru_read_hint_on_arrival monkeypatches EngineCore.preprocess_add_request (vLLM 0.22.1 and 0.32.0); even when storage takes no hints, it installs the wrapper and starts one idle maru-arrival-hint thread. The worker brackets batch_retrieve with set_read_request.

✅ Tests

• Unit tests: tests/unit/test_remote_cxl_access_policy.py, tests/unit/test_remote_cxl_read_hints.py (real directory, agent and clients with a fake policy and extension: leading hits per region, once per request, end routing, agent checks, probe locations), tests/unit/test_vllm_read_hints.py (scheduler hint points, arrival hook, no hint after the end). Also: ends kept and dropped hints warned when too many wait, hints after close ignored, a hint for a region nobody holds refused, a three-argument policy read, no hint work when storage takes none, the arrival setting without asynchronous loading. Full unit suite: 1770 passed, 4 skipped.
• Integration tests
• Manual tests — not done for this branch; the numbers above come from a separate prototype.
• No tests needed (reason: )

🔗 Related Issues (optional)

🌿 Related PRs (optional)

• Builds on the remote_cxl backend (feat: add the remote_cxl storage backend (CXL memory of memory nodes over RDMA) #107).

🌿 Related Branches (optional)

📦 Release Note (for auto-generation / write in English)

NEW

CHANGED

FIXED

IMPORTANT NOTES

[youngrok-XCENA]

Superseded by #130. Gating every RDMA transfer from the GPU node turned out unnecessary for a transparent device cache and added a round trip per READ; #130 keeps only read hints and the region extension hooks, on top of the current main.