Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs

Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs
Yung-Hsu Yang, Luigi Piccinelli, Samuel Rota Bulò, Sunghwan Hong, Denis Rozumny, Johannes Schönberger, Zuria Bauer, Hermann Blum, Peter Kontschieder, and Marc Pollefeys
ECCV 2026, Paper at arXiv 2608.12179

News and ToDo

• 17.08.2026: Release code and models.
• 18.06.2026: Map-Det3D is accepted at ECCV 2026!

Getting Started

Try our HuggingFace Demo with your own data directly!

Installation

We support Python 3.11+ and PyTorch 2.8.0+. Please install the correct PyTorch version according to your own hardware settings.

conda create -n mapdet3d python=3.11 -y

conda activate mapdet3d

# Install PyTorch
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu126

# Install CUDA ops
pip install git+https://github.com/SysCV/vis4d_cuda_ops.git --no-build-isolation --no-cache-dir

# Install Map-Det3D
pip install -v -e .

Model

We host our model on HuggingFace and provide the demo.py as the example.

import torch

from mapdet3d.model.mapdet3d import MapDet3D
from mapdet3d.op.mapdet3d.head import RoI2Det

device = "cuda" if torch.cuda.is_available() else "cpu"

# TF32
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

torch.set_float32_matmul_precision("highest")

# Init model
model = MapDet3D.from_pretrained("RoyYang0714/Map-Det3D").to(device)

# (Optional) Enable tracking -> the output 3D bounding boxes will be in world coordinate if enable tracking
model.track_whole_scene = True
model.roi2det = RoI2Det(nms=True, score_threshold=0.25, iou_threshold=0.5)

# Inference
model.eval()

with torch.no_grad():

with torch.autocast("cuda", enabled=True, dtype=torch.bfloat16):
    predictions: MapDet3DOut = model(
        images=[image],
        intrinsics=[intrinsics],
        extrinsics=[extrinsics],
        frame_ids=[frame_id],
    )

The model weight can also be download as map-det3d-ca1m.pt.

Data

CA-1M

We use CA-1M as the training and in-domain testing sets.

1. Download the train.txt and val.txt from here, and put them under data/CA1M

2. Use the provided script to download the data and unzip them.

python scripts/ca1m/download.py --split train
python scripts/ca1m/download.py --split val

It will download the full CA1M train and val data under data/CA1M.

3. Convert the dataset for training and testing.

python scripts/ca1m/convert.py --split train
python scripts/ca1m/convert.py --split val

It will parse the dataset and save the cached files and HDF5 under data/ca1m.

4. (Optional) Mesh file.

The mesh is for the visualization purpose. You can download CA-1M mesh from BoxFusion or just generate them with open3d.

The final data structure should be like this:

REPO_ROOT
├── data
│   ├── CA1M
│   │   ├── train.txt
│   │   ├── val.txt
│   │   ├── train
│   │   │   ├── $SEQ_NAME
│   │   │   ├── ...
│   │   └── val
│   └── ca1m
│       ├── cache
│       ├── mesh
│       │   ├── $SEQ_NAME
│       │   │   └── mesh.ply
│       │   ├── ...
│       ├── train
│       │   ├── $SEQ_NAME.hdf5
│       │   ├── ...
│       └── val
├── ...

ScanNet

We follow BoxFusion and use ScanNet as the out-of-domain testing sets.

1. Download ScanNet and use the script to extract the images, depth, intrinsics, and poses according to the val.txt:

REPO_ROOT
├── data
│   ├── scannet
│   │   ├── data
│   │   │   └── $SEQ_NAME
│   │   │   │   ├── frames
│   │   │   │   │   ├── colors
│   │   │   │   │   ├── ...
│   │   │   │   ├── ...
│   │   └── meta_data
├── ...

2. Extract annotations.

python scripts/scannet/batch_load_scannet_data.py
python scripts/scannet/batch_load_scannet_data.py --scannet200

3. Convert the data for testing.

python scripts/scannet/convert.py
python scripts/scannet/convert.py --scannet200

The final data structure should be like this:

REPO_ROOT
├── data
│   ├── scannet
│   │   ├── cache
│   │   ├── data
│   │   ├── meta_data
│   │   ├── scannet_instance_data
│   │   └── scannet200_instance_data
├── ...

Training

# 2 nodes and 8 gpus each node 
mapdet3d fit --config mapdet3d/zoo/mapdet3d/mapdet3d_ca1m.py --gpus 8 --nodes 2

The output will be dumped under ./work_dir/${experiment_name}/${version}.

You can also enable wandb logging by adding --wandb.

Testing

# CA1M
mapdet3d test --config mapdet3d/zoo/mapdet3d/mapdet3d_ca1m.py --gpus 1 --ckpt https://huggingface.co/RoyYang0714/map-det3d-ca1m/resolve/main/map-det3d-ca1m.pt

# ScanNet200
mapdet3d test --config mapdet3d/zoo/mapdet3d/mapdet3d_scannet200.py --gpus 1 --ckpt https://huggingface.co/RoyYang0714/map-det3d-ca1m/resolve/main/map-det3d-ca1m.pt

# Tracking
mapdet3d test --config mapdet3d/zoo/mapdet3d/mapdet3d_track_scannet.py --gpus 1 --ckpt https://huggingface.co/RoyYang0714/map-det3d-ca1m/resolve/main/map-det3d-ca1m.pt

We provide the rerun visualization. Enable it with --vis flag, and the .rrd file will be saved under ./work_dir/${experiment_name}/${version}/rerun_vis folder.

Citation

If you find our work useful in your research, please consider citing our publications:

@misc{yang2026mapdet3dmetricfeedforward3d,
      title={Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs}, 
      author={Yung-Hsu Yang and Luigi Piccinelli and Samuel Rota Bulò and Sunghwan Hong and Denis Rozumny and Johannes Schönberger and Zuria Bauer and Hermann Blum and Peter Kontschieder and Marc Pollefeys},
      year={2026},
      eprint={2608.12179},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.12179}, 
}

Acknowledgements

This project builds upon Vis4D, 3D-MOOD, BoxFusion, and MapAnything. We thank the authors of these projects for making their code available.