[feat] introduce payload transfer abstraction and NIXL-UCX backend

[Ginray]

Motivation

SimpleStorage currently couples payload transfer with its ZMQ request path, which makes it difficult to add alternative high-throughput transports without changing storage routing and public APIs.

This PR introduces a small payload-transfer abstraction so that SimpleStorage can keep ZMQ as its control plane while allowing Host-to-Host payloads to use alternative transports such as NIXL-UCX.

The goals are to:

• keep existing SimpleStorage APIs and default ZMQ behavior compatible;
• make the payload transfer backend independently replaceable;
• improve large-payload transfer performance without redesigning the storage layer.

Summary

• Add PayloadTransfer as the payload backend interface for SimpleStorage.
• Keep the existing ZMQ payload path as the default backend.
• Add NIXL-UCX as an optional Host-to-Host payload backend.
• Keep storage routing independent from the payload data path while retaining ZMQ as the control plane.

Design

AsyncSimpleStorageManager selects the configured payload backend and delegates logical payload PUT/GET operations to PayloadTransfer.

AsyncSimpleStorageManager
        |
        | logical put/get
        v
PayloadTransfer
      /            \
ZmqPayloadTransfer  NixlPayloadTransfer

On the StorageUnit side, the selected PayloadTransfer handles backend-specific PUT/GET requests, while SimpleStorageUnit continues to own storage semantics, data access, CLEAR, metrics, and checkpoint operations.

For NIXL-UCX:

• ZMQ remains the control plane for prepare/commit/error coordination.
• Non-empty payload data is transferred through NIXL-UCX.
• Backend-specific endpoint and transfer metadata remain internal to the NIXL implementation.

Usage

The default remains ZMQ.

To enable NIXL-UCX:

backend:
  storage_backend: SimpleStorage
  SimpleStorage:
    payload_transfer:
      backend: nixl-ucx
      ucx_env_vars: {}

See docs/nixl_ucx_payload.md for installation, UCX configuration, and RDMA verification.

Compatibility

• ZMQ remains the default backend.
• Existing SimpleStorage PUT/GET/CLEAR callers and public APIs remain unchanged.
• The existing ZMQ wire protocol and external behavior remain unchanged.
• NIXL-UCX is opt-in.
• NIXL transfer failures are reported directly and do not silently fall back to ZMQ.

Validation

Two-node A2 SimpleStorage tests covered 33 payload sizes from 128 KiB to 1 GiB. All PUT/GET/CLEAR operations and data checks passed.

Native UCX is shown only as a benchmark reference and is not part of this PR.

Payload	ZMQ	Native UCX	NIXL-UCX
128 KiB	10.23 / 10.45	26.04 / 27.63	27.95 / 26.06
1 MiB	51.73 / 51.88	72.23 / 72.35	79.22 / 77.70
16 MiB	104.47 / 104.33	106.75 / 106.65	106.16 / 106.04
1 GiB	112.09 / 111.92	110.30 / 110.29	109.17 / 109.18

Median MiB/s, PUT / GET.

The fixed real DAPO payload produced identical digests and training inputs through ZMQ and NIXL-UCX. Both paths completed 20/20 VIME steps on 80 real DAPO problems without OOM or malformed output.

Follow-up

• Perform broader correctness, stability, and performance validation across larger-scale and sustained multi-node workloads.
• Add user-facing tutorials and complete the related feature documentation for the NIXL-UCX payload path.
• Prepare the feature for integration into main once it is considered stable.

[ascend-robot]

CLA Signature Guide

@Ginray , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
d18010f3 feat: add NIXL-UCX payload trans...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d18010f305

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Expire abandoned NIXL preparations

When a client process is killed after receiving PUT_DATA_READY, no commit or cancel request arrives, so this entry and its registered receive buffer remain retained for the storage unit's lifetime; the analogous GET preparation also retains its encoded frames. A single abandoned large transfer can therefore permanently consume substantial memory, and repeated client failures can OOM the storage unit, so pending transfers need deadline-based cleanup independent of client cancellation.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Measure NIXL operations as logical PUTs and GETs

When payload_transfer.backend is nixl-ucx, requests use the new *_PREPARE and *_COMMIT types, so they bypass this only monitor.measure branch. Consequently the existing PUT_DATA/GET_DATA operation statistics stay at zero even while NIXL traffic is active; aggregate the strategy requests into the corresponding logical PUT/GET measurements so this supported backend remains traceable through observability.

AGENTS.md reference: AGENTS.md:L7-L9

Useful? React with 👍 / 👎.

[ascend-robot]

CLA Signature Pass

Ginray, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

Ginray, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

Ginray, thanks for your pull request. All authors of the commits have signed the CLA. 👍