OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments

Abstract

3D scene graphs (3DSGs) provide a compact and structured abstraction of 3D environments. Although advances in foundation models have enabled open-vocabulary 3DSG generation, existing approaches remain object-centric and encode limited relational information---restricting their applicability in real-world scenarios that require fine-grained understanding. We propose OP3DSG, an open-vocabulary part-aware 3DSG generation framework that constructs unified graphs that jointly model objects, interactive parts, spatial relations, functional relations, and affordances. OP3DSG integrates object-part knowledge-guided detection with part-aware 3D fusion to preserve small and interaction-relevant components, and employs a geometry-initialized prior graph with LLM-based refinement to reduce spurious relational predictions while enabling efficient graph construction. To systematically evaluate unified 3D scene graph construction, we introduce UniGraph3D, a benchmark designed for part-aware perception and multi-level relational reasoning. Experimental results show that OP3DSG achieves state-of-the-art performance and demonstrates its effectiveness as a perception backbone in diverse real-world robotics tasks.

_{▲ Conceptual Comparison of 3D Scene Graphs (3DSGs). (a) Given a 3D scene, (b) prior work falls into two scopes: Spatial 3DSGs, which recognize objects and encode spatial relations, and Functional 3DSGs, which mainly recognize interactive parts and functional relations. (c) Our scope unifies these notions into Unified 3DSGs.}

1. Setup

1.1. Create Conda Environment

# Create conda environment
conda create -n op3dsg python=3.10
conda activate op3dsg

# Install Pytorch
conda install -y -c pytorch -c nvidia -c defaults pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 cuda-toolkit=11.8 cuda-nvcc=11.8 faiss-cpu=1.7.4 mkl=2021 "blas=1.0=mkl" pyyaml

# Install Pytorch3D
conda install -y https://anaconda.org/pytorch3d/pytorch3d/0.7.4/download/linux-64/pytorch3d-0.7.4-py310_cu118_pyt201.tar.bz2

# Install the required libraries 
cd OP3DSG
export PIP_CONSTRAINT="constraints.txt"
pip install --upgrade-strategy only-if-needed tyro timm==1.0.17 open_clip_torch wandb h5py openai hydra-core distinctipy ultralytics dill supervision==0.21.0 open3d imageio natsort kornia rerun-sdk pyliblzfse pypng git+https://github.com/ultralytics/CLIP.git transformers==4.25.1 fairscale

# Check path by 'conda env list' and set the env
export CUDA_HOME=/path/to/.conda/envs/op3dsg

# Install OP3DSG
pip install -e .

1.2. Set Project Path

Before running OP3DSG, update the project paths in configs/base.yaml to match your local environment.

root: /path/to/parent_dir
project_root: ${root}/OP3DSG

1.3. Install Third-Party Repositories

OP3DSG relies on several third-party repositories, including VLPart. Since VLPart depends on detectron2, please install a version that matches your PyTorch, CUDA, and Python environment. Compatible pre-built Detectron2 wheels are available here.

# used 'detectron2-0.6+2a420edpt2.0.1cu118-cp310-cp310-linux_x86_64.whl'
pip install --extra-index-url https://miropsota.github.io/torch_packages_builder 'detectron2==0.6+2a420edpt2.0.1cu118'

You may install SAM, RAM, and VLPart either by following the official instructions for each repository or by running the provided setup script.

bash scripts/setup_thirdparty.sh

1.4. UniGraph3D Dataset

Download the customized SceneFun3D dataset and the FunGraph3D dataset. Both datasets were originally used in OpenFunGraph. However, due to errors in the 3DSG annotations and ambiguity in the labels, we newly annotate and integrate the two datasets into a unified dataset, UniGraph3D.

Please organize the datasets under the dataset/ directory.

Dataset structure

dataset/
├── FunGraph3D/
│   ├── 0kitchen/
│   │   ├── video0/
│   │   ├── video1/
│   │   └── 0kitchen.ply
│   └── ...
├── SceneFun3D/
│   ├── dev/
│   │   ├── 420683/
│   │   ├── 421013/
│   │   ├── ...
│   │   └── metadata.csv
│   └── test/
│       ├── 421380/
│       ├── 422391/
│       ├── ...
│       └── metadata.csv
└── UniGraph3D/

Integrated Functional Relations

FunGraph3D → UniGraph3D

_{* Cells with the same color indicate labels that are merged into a single unified label in UniGraph3D.}

SceneFun3D → UniGraph3D

_{* Cells with the same color indicate labels that are merged into a single unified label in UniGraph3D.}

1.5. Checkpoints

cd checkpoints
# SAM
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
# RAM
wget https://huggingface.co/spaces/xinyu1205/Tag2Text/resolve/main/ram_swin_large_14m.pth
# VLPart
wget https://github.com/PeizeSun/VLPart/releases/download/v0.1/swinbase_cascade_lvis_paco.pth

2. Run

Stage 1. Generate Geometry-Initialized Prior Graph

This stage performs open-vocabulary 2D object/part detection followed by multi-view fusion to construct a geometry-initialized 3D scene graph, referred to as the prior graph.

python scripts/prior_graph.py \
  scene_id=0kitchen/video0 \
  dataset=FunGraph3D \
  save_folder_name=<RUN_NAME>

• scene_id : scene/sequence id, e.g. 0kitchen/video0, 420683/42445135
• dataset : FunGraph3D | SceneFun3Ddev | SceneFun3Dtest
• save_folder_name : output subfolder name used to distinguish runs

Outputs under output/<scene_id>/<save_folder_name>/(use output_root=<path> to change the output base directory):

• object/, part/ — per-frame detections and fused point clouds (pcd_saves/full_pcd_ram_update.pkl.gz)
• scene_graph/initial_3d_scene_graph.json

Stage 2. Generate Unified Graph

This stage uses LLM-based reasoning to refine and verify the prior graph, producing the final unified 3D scene graph with object/part nodes, spatial relations, functional relations, and affordances. It requires an OPENAI_API_KEY.

export OPENAI_API_KEY=<OPENAI_API_KEY>

python scripts/uni_graph.py \
  scene_id=0kitchen/video0 \
  dataset=FunGraph3D \
  save_folder_name=<RUN_NAME>

Use the same scene_id / dataset / save_folder_name as Stage 1 so it picks up the matching prior-graph outputs.

Outputs under output/<scene_id>/<save_folder_name>/scene_graph/(use output_root=<path> to change the output base directory):

• unified_3d_scene_graph.json

3. Experiments

3.1. Run the Full Benchmark

This script runs the two-stage OP3DSG pipeline from Section 2 — Stage 1 scripts/prior_graph.py (2D detection → object/part 3D fusion → initial graph) followed by Stage 2 scripts/uni_graph.py (LLM-based refinement) — over every benchmark scene of FunGraph3D, SceneFun3Ddev, and SceneFun3Dtest. Stage 2 uses the OpenAI API, so an OPENAI_API_KEY is required.

export OPENAI_API_KEY=<OPENAI_API_KEY>

SAVE=<RUN_NAME> bash scripts/run_OP3DSG.sh

All settings are overridable via environment variables:

• SAVE : save_folder_name used to distinguish runs
• PRIOR_OPTS / UNI_OPTS : extra Hydra overrides passed to Stage 1 / Stage 2, e.g. PRIOR_OPTS="obj_fusion.mask_conf_threshold=0.25". The per-stage object/part 3D-fusion parameters live in configs/prior_graph.yaml (obj_fusion: / part_fusion:).
• FAIL_FAST=1 : stop the whole run on the first failing stage (default: continue)
• OMP_NUM_THREADS, MKL_NUM_THREADS, ... : thread limits (default 8)

Per-scene logs are written to logs/<timestamp>/<dataset>/<scene>.log (the / in the scene id is replaced with _, e.g. 0kitchen_video0.log), with an overall pass/fail summary in logs/<timestamp>/summary.log. Pipeline artifacts (detections, fused point clouds, and scene graphs) are stored under output/<scene_id>/<SAVE>/, exactly as in Section 2.

3.2. Evaluate on UniGraph3D

This script evaluates the generated unified 3D scene graphs (unified_3d_scene_graph.json) against the UniGraph3D ground truth (dataset/UniGraph3D/) for the same benchmark scene lists, by calling src/eval/eval_uni.py per scene. The evaluation scripts require Sentence Transformers. To avoid PyTorch version conflicts with the main environment, please create a separate Conda environment, install sentence-transformers, and run the evaluation scripts within this environment.

Before running, set the following variables — either as environment variables or by editing the top of the script (they are left empty on purpose):

• SAVE : the save_folder_name of the run to evaluate (the SAVE value used in Section 3.1)
• ROOT_FG : FunGraph3D dataset root (used for GT geometry such as scene point clouds), e.g. <repo>/dataset/FunGraph3D
• ROOT_SF : SceneFun3D dataset root containing the dev/ and test/ splits, e.g. <repo>/dataset/SceneFun3D
• RESULT_ROOT (optional) : root of the generated results, <repo>/output by default. Override it only if you generated the graphs with a custom output_root=<path>.

SAVE=<RUN_NAME> \
ROOT_FG=dataset/FunGraph3D \
ROOT_SF=dataset/SceneFun3D \
bash scripts/run_eval_UniGraph3D.sh

For each scene it locates <RESULT_ROOT>/<scene>/<video>/<SAVE>/ and evaluates the fused object/part point clouds together with the unified graph (bbox IoU matching) against the UniGraph3D GT. Scenes with missing outputs are skipped with a [SKIP] message, so partially finished runs can still be evaluated.

4. For New Datasets

The scripts under scripts/ also support running OP3DSG on your own data that is not registered as one of the built-in datasets (FunGraph3D, SceneFun3D, ...). Two entry points are provided depending on what data you have. We used the code in this section for experiments with real-world robotics applications.

4.1. 2D Detection Only

Use this when you only have RGB images (no depth or camera poses) and just want to inspect open-vocabulary 2D object/part detection results. It runs RAM tagging, VLPart detection, and SAM segmentation on every image, then saves the annotated visualizations — it does not perform 3D fusion or scene-graph generation.

This script can take a plain image folder directly through a lightweight CLI (no Hydra command syntax required):

python scripts/2D_detection_only.py \
  --image_dir /path/to/image_folder \
  --scene_id <SCENE_NAME> \
  --detector vlp \
  --tagger ram \
  --override save_folder_name=detect_only \
  --override vlp.score_threshold=0.4 \
  --override vlp.nms_threshold=0.4 \
  --override vlp.part_nms_threshold=0.4

• --image_dir : folder containing .jpg / .jpeg / .png images (required)
• --scene_id : name used for the output subfolder (defaults to the image-folder name)
• --detector : detector to use, currently vlp (VLPart) is supported
• --tagger : tagging model, ram for open-vocabulary tags or none for SAM dense mode
• --camera_axis : optional, e.g. Left if the images need a 90° rotation
• --output_root : output base directory (default: OP3DSG/output)

Outputs under output/<scene_id>/<save_folder_name>/:

• object/gsa_vis_<tagger>/, part/gsa_vis_<tagger>/ — annotated detection visualizations
• gsa_classes_<tagger>.json, object/gsa_classes_<tagger>_obj.json, part/gsa_classes_<tagger>_part.json — detected object/part class lists

If your data is already registered as a dataset, you can also run it in Hydra mode with scene_id=<id> dataset=<name> instead of --image_dir.

4.2. Prior Graph from Self-Captured RGB-D

Use this when you have posed RGB-D captures (RGB + depth + camera poses) and want the full Stage 1 pipeline (2D detection → 3D fusion → geometry-initialized prior graph) on data that is not a built-in dataset. It reuses the exact pipeline of prior_graph.py and produces the identical output format.

Prepare your capture folder with the following layout:

<input_dir>/
├── rgb/             RGB frames        000000.jpg, 000001.jpg, ...
├── depth/           depth maps        000000.png, 000001.png, ...
├── poses/           camera poses      000000.txt, 000001.txt, ...
└── intrinsics.txt   pinhole intrinsics (shared by all frames)

• Frames are ordered lexicographically by RGB filename — use zero-padded names.
• depth/ and poses/ files are matched to RGB frames by file stem; every RGB frame must have a matching depth and pose file.
• depth/ : 16-bit .png (or .npy). Metric depth = raw_value / depth_scale (depth_scale defaults to 1000.0, i.e. millimetre PNG).
• poses/ : a 4×4 camera-to-world matrix (if the fused point cloud looks wrong, your poses are likely world-to-camera — invert them first).
• intrinsics.txt : RGB intrinsics at the native RGB resolution, given as a 3×3 K matrix or a single line fx fy cx cy.

python scripts/prior_graph_inference.py \
  input_dir=/path/to/capture \
  save_folder_name=<RUN_NAME> \
  depth_scale=1000.0

• input_dir : path to the capture folder described above (required)
• save_folder_name : output subfolder name used to distinguish runs
• depth_scale : raw-depth-to-metres divisor (default 1000.0)
• output_root=<path> : optional, change the output base directory

Outputs under output/<input_dir name>/<save_folder_name>/ — identical in format to Stage 1 of prior_graph.py:

• object/, part/ — per-frame detections and fused point clouds
• scene_graph/initial_3d_scene_graph.json

You can then run Stage 2 (scripts/uni_graph.py) on these outputs using the same scene_id (the input_dir folder name) and save_folder_name.

5. Information

5.1. Citation

If you find our work useful for your research, please consider citing as:

@InProceedings{Kim2026OP3DSG,
    title     = {OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments},
    author    = {Yirum Kim and Uehwan Kim},
    booktitle = {The European Conference on Computer Vision (ECCV)},
    month     = {September},
    year      = {2026},
}