Skip to content
YumengHePublic

About

SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge

Resources

Stars

42 stars

Watchers

2 watching

Forks

Repository files navigation

SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge

Official implementation of SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge. SPARK combines VLM-guided part-level and global image guidance with diffusion transformers to produce high-quality articulated object reconstructions from a single image.

📰 News

  • [2026-11] Image generation moved from Gemini to OpenAI's gpt-image-2.6-sunburst, and the VLM steps now use gpt-6.1-sol, so only an OpenAI API key is required. We compared the latest GPT image models here.

🔧 Installation

1. Clone and create the env

git clone https://github.com/YumengHe/SPARK.git
cd SPARK

conda create -n spark python=3.10 -y
conda activate spark

2. Install PyTorch + torch-cluster + project dependencies

# PyTorch (CUDA 12.4 wheels)
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124

# torch-cluster (must match torch + CUDA)
pip install torch-cluster==1.6.3 -f https://data.pyg.org/whl/torch-2.5.1+cu124.html

# Project dependencies (pins numpy etc. before PyTorch3D is compiled)
pip install -r requirements.txt

3. Point the build at a CUDA 12.4 toolchain

PyTorch3D compiles CUDA code, so nvcc must match the 12.4 wheel above. Pick one of the two options below.

Option A — use an existing system CUDA 12.4 (lightweight, recommended if available)
export CUDA_HOME=/usr/local/cuda-12.4
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
nvcc --version   # sanity-check: must report release 12.4
Option B — install CUDA 12.4 toolkit into the conda env (~3 GB, use if no system CUDA 12.4)
conda install -c "nvidia/label/cuda-12.4.0" cuda-toolkit -y
export CUDA_HOME=$CONDA_PREFIX
nvcc --version   # sanity-check: must report release 12.4

4. Build PyTorch3D

pip install ninja
pip install --no-build-isolation "git+https://github.com/facebookresearch/pytorch3d.git@stable"

--no-build-isolation lets the build see the torch we just installed (otherwise pip spins up an isolated env without it).

Pretrained weights

All checkpoints (PartCrafter, TripoSG, RMBG-1.4) live in 2bidoubi/SPARK and are fetched into pretrained_weights/<name>/ automatically on first run. To pre-download them:

python -m src.utils.download_weights

💡 Quick Start

Create an .env file and add API key inside:

# OpenAI: metadata.json prediction, joint-axis correction, prompt generation, part/open-state image generation
OPENAI_API_KEY=

# Meshy: texturing
MESHY_API_KEY=

Place input images in ./image/ and run the full pipeline:

python run.py

run.py runs nine stages end-to-end (metadata → URDF → per-part image generation → PartCrafter inference → mesh cleanup → texture). Toggle individual steps with the step1 … step9 flags at the top of the file.

📚 Data Preprocessing

See datasets/README.md for the full pipeline. A typical PartNet-Mobility flow:

# Convert PartNet-Mobility (obj+mtl) to GLB
python datasets/merge_to_glb.py mesh/partnet_test mesh/partnet_glb

# Voxelize meshes
python datasets/voxel_surface.py mesh/partnet_glb --subfolder -r 200

# Build PartCrafter training data
CUDA_VISIBLE_DEVICES=0 python datasets/preprocess/preprocess_partnet.py \
    --input mesh/partnet_glb --output preprocessed_data

🏋️ Training

bash scripts/train/train_partcrafter.sh --config configs/mp16_nt1024.yaml

Set WANDB_API_KEY in the script if you want experiment tracking.

🚀 Common Script Entrypoints

Single-image object inference:

python scripts/inference/object.py \
    --image_path image/example.png \
    --num_parts 4 \
    --output_dir results \
    --render

Per-part image inference:

python scripts/inference/part_images.py \
    --image_folder output/example_case \
    --output_dir results \
    --render

Render a generated mesh:

python scripts/render/glb.py \
    --input results/example/object.glb \
    --output_dir render_output \
    --gif

🗂️ Script Layout

  • scripts/inference/: inference entrypoints
  • scripts/render/: mesh and URDF rendering helpers
  • scripts/train/: training launch scripts
  • scripts/batch/: batch inference runners
  • scripts/tools/: maintenance utilities
  • scripts/legacy/: archived old scripts

🤗 Pushing Pretrained Weights to HuggingFace

For maintainers: see docs/upload_weights.md.

🙏 Acknowledgement

Thanks to Yongfei She for organizing the code for open source release.

📝 Citation

If you find SPARK useful for your research, please cite:

@inproceedings{he2026spark,
  title={Spark: Sim-ready part-level articulated reconstruction with vlm knowledge},
  author={He, Yumeng and Jiang, Ying and Lu, Jiayin and Yang, Yin and Jiang, Chenfanfu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={7233--7243},
  year={2026}
}

About

SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge

Resources

Stars

42 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages