RelateAnything

Real-time open-vocabulary relation prediction from any inputs.

Paper · Project page · Models · Training data · Documentation

Give RelateAnything an image, object regions, and the relations you want to look for. It returns scored (subject, relation, object) triplets, such as person → riding → horse.

• Use your own regions. Boxes can come from a detector, a segmenter, or a person. Masks are optional.
• Choose the relation vocabulary. Supply phrases at inference time without retraining. A small text encoder embeds them once; no language model runs per frame.
• Get one graph or two. Return a single ranked list, or separate spatial and semantic graphs from the same forward pass.

hero_30s_five_clips.mp4

_{The same relation model across five scenes, with tracked regions and relations. Clip credits · Reproduce the video}

Try the browser demo — no installation; inference runs on your device using WebGPU or WASM.

Quickstart · Models · Deployment · Performance · Results · How it works

Quickstart

Use Python 3.12+. Install a CUDA-enabled PyTorch build for NVIDIA GPU inference, or use device="cpu" in the example below.

git clone https://github.com/Maelic/RelateAnything.git
cd RelateAnything
pip install -e ".[hub,detector]"   # detector: ultralytics YOLO, used below

Run this from the repository root. A detector finds the objects in the included sample photo; RelateAnything predicts the relations between them.

from PIL import Image
from ultralytics import YOLO
from relsgg import RelateAnything

image = Image.open("assets/reel/images/horse.jpg").convert("RGB")

# 1. Find the objects. Any detector or segmenter works; this one is YOLO26.
detections = YOLO("yolo26m.pt")(image, conf=0.4, verbose=False)[0]
boxes = detections.boxes.xyxy.cpu().numpy()  # [N, 4]: x1, y1, x2, y2 in pixels
labels = [detections.names[int(c)] for c in detections.boxes.cls]  # for display only

# 2. Predict the relations between them.
model = RelateAnything.from_pretrained("maelic/relsgg-vits16plus", device="cuda")
for triplet in model.predict(image, boxes, box_labels=labels, topk=3):
    print(triplet)
# (person) --riding [0.76]--> (horse)
# (horse) --carrying [0.51]--> (person)
# (horse) --walking past [0.43]--> (car)

# 3. Change the relations to look for, without retraining.
model.set_vocabulary(["riding", "carrying a rider", "beside", "in front of"])
graphs = model.predict(image, boxes, box_labels=labels, decompose=True, topk=3)
print(graphs["spatial"])   # where things are, e.g. (horse) --in front of--> (car)
print(graphs["semantic"])  # what they do, e.g. (person) --riding--> (horse)

The relation model reads the image and the boxes; the detector's labels are never an input. Boxes can come from any detector, a segmenter (pass masks= as well) or a person: an [N, 4] array in original-image pixels is all it needs. Released checkpoints include the backbone configuration and text encoder; no gated DINOv3 login is needed for inference. The detector extra installs ultralytics, which is AGPL-3.0 and downloads the YOLO26 weights on first use.

API guide: inputs, vocabularies, masks, and scores · Installation and offline use

Models

Start with relsgg-vits16plus, the recommended balance of quality and speed. For more accuracy, use relsgg-convnext-base: within 0.3 OVS of ConvNeXt-L at half its parameters. All seven checkpoints share the training recipe and differ in their vision backbone; the ConvNeXt towers also train their backbone at a lower learning rate.

Checkpoint	Backbone	Parameters
relsgg-vits16	DINOv3 ViT-S/16	46.1 M
relsgg-vits16plus	DINOv3 ViT-S/16+	53.2 M
relsgg-vitb16	DINOv3 ViT-B/16	113.8 M
relsgg-convnext-tiny	DINOv3 ConvNeXt-T	58.6 M
relsgg-convnext-small	DINOv3 ConvNeXt-S	80.2 M
relsgg-convnext-base	DINOv3 ConvNeXt-B	123.5 M
relsgg-convnext-large	DINOv3 ConvNeXt-L	245.7 M

_{Accuracy (OVS-F1, four benchmark axes) against latency of the relation model on a laptop RTX 3080 with TensorRT. The shaded line is the Pareto frontier.}

Compare model quality and speed. Each model card includes evaluation results and calibration details.

Deployment

• PyTorch: use the Python API above on CPU or CUDA.
• NVIDIA GPU / TensorRT 10: build a local FP32 engine from the released ONNX graph. Vocabulary selection, calibration and two-graph decoding stay available. Setup and examples.
• Laptop / ONNX or OpenVINO: run the local image or webcam demo. Setup · OpenVINO.
• Browser: try the demo without installing Python.

After downloading the relation graph and building the local detector as explained in the deployment guide:

# CPU image demo; omit --image to use a webcam.
python deploy/demo_webcam.py --dist deploy/dist/relsgg-vits16plus --image photo.jpg

# NVIDIA GPU, after building both TensorRT engines.
python deploy/demo_webcam.py --dist deploy/dist/relsgg-vits16plus \
  --backend tensorrt --device cuda --image photo.jpg --decompose

The relation model consumes regions; it does not detect objects itself. Demo detectors are rebuilt locally and have separate licenses. The exported ONNX/TensorRT graph scores boxes; native mask inputs are supported by the PyTorch API. Deployment contracts and limitations.

Performance

On an RTX 3080 Laptop GPU with TensorRT, the relation model takes 14.7 ms per image with ConvNeXt-T, the fastest checkpoint, and 16.8 ms with the recommended ViT-S+. The full 19,103-predicate vocabulary (Python API) adds at most 1.5 ms.

With YOLO26m detection included, a frame takes 33.5 ms with ConvNeXt-T and 38.7 ms with ViT-S+ (TensorRT relations).

On one A100, the compiled ViT-S+ pipeline with detection takes 25.6 ms per image, against 232 ms for OvSGTR, 498 ms for ACC and about 40 s for ROBIN-3B.

All checkpoints and runtimes · With a detector · Across scene-graph systems

Results

The paper evaluates relation prediction across datasets, open vocabularies and spatial reasoning tasks. Selected transfer results for ViT-S+, using ground-truth boxes and one relation per ordered pair:

Test set	OvSGTR F1@50	RelateAnything F1@50
VG150	16.5	36.9
PSG	13.5	34.7
IndoorVG	20.2	37.8

Both models use the same evaluator and vocabulary. OvSGTR receives object labels; RelateAnything does not. The full results cover stronger baselines including ROBIN-3B, zero-shot controls and the complete evaluation.

Evaluation protocol · Scoring guide

How it works

Figure 2 from the paper. Click to enlarge.

A DINOv3 backbone reads the image once. Regions become visual tokens; a pair sampler selects candidate relations; a transformer refines them using the scene. A vocabulary head scores each pair against embeddings of your predicate strings, mixing spatial and semantic features. Swapping that embedding bank changes the vocabulary without retraining the vision model.

Architecture guide

Data and evaluation packs

• RA-4M: training annotations for open-vocabulary relations. Images are referenced by ID and are not redistributed. Dataset details and generation.
• OV-SGG-Bench: evaluation packs, explicit negatives and calibration files for six complementary axes. Protocol · run it in two commands.

Reproduce and contribute

• Train: python training/setup_data.py && ./train.sh downloads the data and retrains the released mixture. Training guide, released configs.
• Evaluate: evaluation guide and benchmark/.
• Generate annotations: datagen/. Explore the paper's probes: research/.
• Contribute: contributor guide. Bug reports, deployment feedback and examples of new uses are welcome in GitHub issues. Include the model, runtime, hardware and a small reproducer when reporting a problem.

The documentation index covers the API, architecture, training, data and deployment. Read evaluation pitfalls before comparing results or changing the scoring path.

License

• Code: AGPL-3.0-only, with the additional attribution terms in NOTICE. If you run a modified version as a network service, you must offer its users the complete source of that version.
• Commercial license: to use RelateAnything in a product or service without the AGPL obligations, contact the author (Maëlic Neau) through GitHub issues or the email in the paper.
• Model weights: derivatives of Meta DINOv3; see the model cards and DINOv3 license.
• RA-4M annotations: carry the Gemma Terms of Use notice. Source images are not redistributed.
• Demo detectors: ultralytics-derived components use AGPL-3.0 and are excluded from the relation model's release artifacts.

Full third-party notices and credits

Citation

@article{neau2026relateanything,
  title   = {RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs},
  author  = {Neau, Ma\"elic},
  journal = {arXiv preprint arXiv:2609.12552},
  eprint  = {2609.12552},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url     = {https://arxiv.org/abs/2609.12552},
  year    = {2026}
}