Mingyang Bao1 Fangda Ye1 Xihao Chen1 Bobo Li1 Jason Li2 Shengqiong Wu3,✉
1National University of Singapore 2Nanyang Technological University 3University of Oxford
NeurIPS 2026
Seeing the Unseen generates physically consistent videos from a single image. It predicts how the scene moves with a learned particle simulator, and passes that motion to a video diffusion model as guidance. The guidance covers motion that is visible in the input frame and motion that only appears later, for example a candle face that rotates into view after a collision.
- PGDS (Particle-based Graph Dynamics Simulator) is a lightweight graph network trained on Physion. It predicts 3D particle trajectories for collisions, domino chains, free fall and their combinations.
- VIMF (Visible–Invisible Motion Field) encodes the simulated motion as two tracking videos. The reference-visible video covers points seen in the first frame. The future-emergent video covers points that become visible later.
- The two motion videos condition a fine-tuned Wan2.2-TI2V-5B through zero-initialized channel injection. On seven physical scenarios our method obtains the best Semantic Adherence and Physical Commonsense scores against open-source, physics-aware and closed-source models, including Veo 3.1 and Wan-2.7.
- 2026.10 Code for all five modules is released. Checkpoints will follow on Hugging Face.
- Seeing the Unseen is accepted to NeurIPS 2026.
The input image is lifted into a 3D particle scene, using SAM3 segmentation, SAM-3D-Objects reconstruction and voxelization. A vision-language model estimates the initial velocities from the text prompt. PGDS then rolls out the particle dynamics. The resulting 3D trajectories are rendered into the reference-visible and future-emergent motion videos, which guide the video generation backbone.
Both motion videos are encoded by the frozen 3D VAE and concatenated with the noisy video latent along the channel axis. The new input channels are zero-initialized, so training starts from the unmodified pretrained model and adds the motion guidance gradually.
Semantic Adherence (SA) measures whether the video follows the prompt, including force direction and contact location. Physical Commonsense (PC) measures whether the motion is physically plausible. Both use a 1–5 scale, higher is better. "–" marks a scenario the method does not support.
| Method | Domino | Collision | Free Fall | Coll.+Dom. | F.F.+Dom. | F.F.+Coll. | Translation | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SA | PC | SA | PC | SA | PC | SA | PC | SA | PC | SA | PC | SA | PC | |
| Open-source models | ||||||||||||||
| Wan2.2-I2V-14B | 2.483 | 3.017 | 2.750 | 3.083 | 2.317 | 3.033 | 2.217 | 3.050 | 2.533 | 2.950 | 2.517 | 3.000 | 3.000 | 4.000 |
| Wan2.2-TI2V-5B | 2.650 | 3.033 | 2.917 | 3.067 | 2.183 | 3.083 | 2.200 | 3.033 | 2.467 | 3.033 | 2.183 | 3.083 | 3.000 | 4.100 |
| HunyuanVideo-1.5 | 2.550 | 3.017 | 2.867 | 3.067 | 2.317 | 3.017 | 2.233 | 3.000 | 2.617 | 2.950 | 2.467 | 3.000 | 3.050 | 3.933 |
| CogVideoX1.5-5B | 2.450 | 3.200 | 2.833 | 3.017 | 2.333 | 3.000 | 2.300 | 2.983 | 2.650 | 3.000 | 2.333 | 3.000 | 3.050 | 3.933 |
| Physics-aware models | ||||||||||||||
| PhysT2V | 2.267 | 3.867 | 2.533 | 3.400 | 2.050 | 4.283 | 2.283 | 3.267 | 2.383 | 3.333 | 2.117 | 3.933 | 3.033 | 4.400 |
| NewtonGen | – | – | – | – | – | – | – | – | – | – | – | – | 3.033 | 3.983 |
| PhysCtrl | – | – | – | – | – | – | – | – | – | – | – | – | 2.867 | 4.567 |
| Closed-source models | ||||||||||||||
| Wan-2.7 | 2.200 | 2.800 | 2.800 | 3.000 | 2.400 | 3.000 | 2.200 | 3.000 | 2.400 | 2.800 | 2.200 | 3.000 | 3.000 | 4.600 |
| Veo 3.1 | 2.800 | 3.000 | 2.800 | 3.000 | 2.400 | 3.000 | 2.400 | 3.000 | 2.800 | 3.000 | 2.200 | 3.000 | 3.200 | 4.000 |
| Ours | 3.976 | 4.013 | 4.160 | 4.394 | 4.347 | 4.328 | 3.854 | 4.016 | 3.985 | 4.017 | 4.327 | 4.589 | 4.013 | 4.578 |
On the compatible VideoPhy2 subset:
| Model | SA ↑ | PC ↑ |
|---|---|---|
| CogVideoX-5B | 3.21 | 3.65 |
| Wan2.2-5B | 3.14 | 3.46 |
| Ours (visible-only) | 3.47 | 3.78 |
| Ours | 4.01 | 4.13 |
Pairwise human evaluation against baselines (win / tie / lose):
The code is split into five modules. They exchange data only through files on disk, and each module has its own environment and README.
seeing_the_unseen_code/
├── scene_reconstruction/ # image → 3D particle scene → PGDS rollout (inference entry point)
├── gns_training/ # PGDS training / evaluation (PyG, multi-GPU DDP)
├── scene_to_tracking/ # rollout + meshes → reference-visible / future-emergent motion videos
├── tracking_video/ # the same two motion videos from real videos (training data, SpaTrackerV2)
├── video_generation/ # dual-motion fine-tuning and inference of Wan2.2-TI2V-5B (VideoX-Fun)
└── assets/ # figures used in this README
| Module | Role | Upstream code |
|---|---|---|
scene_reconstruction |
Image + prompt → 3D scene → particles → PGDS rollout | SAM-3D-Objects @ 81a8237 + patch |
gns_training |
Train and evaluate PGDS on Physion | Physion-particles @ 5ac002a |
scene_to_tracking |
Render the two motion videos from a rollout (inference time) | – |
tracking_video |
Extract the two motion videos from real videos (training data) | SpaTrackerV2 @ 7e12274 + patch |
video_generation |
Train and run the dual-motion video model | VideoX-Fun @ 54bf97a + patch |
Clone the repository:
git clone https://github.com/vid-gen/seeing_the_unseen_code.git
cd seeing_the_unseen_codeThe modules depend on different upstream projects, so we recommend one environment per module. All of them were tested with Python 3.10 on Linux with NVIDIA GPUs.
| Module | Main dependencies | GPU |
|---|---|---|
scene_reconstruction |
SAM-3D-Objects, SAM3 (transformers>=5), PyTorch 2.5.1 + CUDA 12.1, binvox |
≥ 32 GB |
gns_training |
PyTorch 2.6.0 + CUDA 12.4, PyG 2.7.0, torch_scatter |
1–4 GPUs |
scene_to_tracking |
numpy, scipy, trimesh, OpenCV, imageio-ffmpeg | CPU only |
tracking_video |
SpaTrackerV2 requirements, PyTorch 2.4.1 + CUDA 12.1, ffmpeg |
1+ GPUs |
video_generation |
VideoX-Fun requirements, PyTorch 2.5.1 + CUDA 12.4, diffusers 0.37 | 1–2 GPUs for inference, 2 × H200 for training |
Step-by-step instructions, including the upstream checkout and git apply of our patches, are in each
module README.
| Model | Used by | Size | Download |
|---|---|---|---|
PGDS (without_Drape/best_model.pth) |
scene_reconstruction, gns_training |
6 MB | Hugging Face (coming soon) |
Dual-motion Wan2.2-TI2V-5B transformer (checkpoint-10500) |
video_generation |
10 GB | Hugging Face (coming soon) |
| Wan2.2-TI2V-5B base model | video_generation |
– | Wan-AI/Wan2.2-TI2V-5B |
Place the downloaded weights under checkpoints/:
checkpoints/
├── gns/without_Drape/best_model.pth
└── video_generation/checkpoint-10500/transformer/
Step 1. Reconstruct the scene and simulate it with PGDS.
cd scene_reconstruction
export OPENAI_API_KEY=... # VLM object detection and velocity estimation (or use --no-gpt)
python pipeline_3dgs_to_physics.py \
--image input.png -o out/scene \
--motion-prompt "The wooden ball rolls forward and knocks the candle over."Step 2. Render the reference-visible and future-emergent motion videos.
cd ../scene_to_tracking
python generate_tracking_videos.py --data_dir ../scene_reconstruction/out/sceneStep 3. Generate the video. Run from the VideoX-Fun checkout prepared as described in
video_generation/README.md.
cd /path/to/VideoX-Fun
S=/path/to/seeing_the_unseen_code/scene_reconstruction/out/scene
python examples/wan2.2_fun/predict_dual_control.py \
--base_model_path /path/to/Wan2.2-TI2V-5B \
--config_path config/wan2.2/wan_dual_control_5b.yaml \
--transformer_path /path/to/seeing_the_unseen_code/checkpoints/video_generation/checkpoint-10500 \
--tracking_video_1 $S/visible_tracking_das.mp4 \
--tracking_video_2 $S/future_visible_tracking_das.mp4 \
--start_image $S/first_frame_1280x704.png \
--prompt "The wooden ball rolls forward and knocks the candle over." \
--guidance_scale 3.0 --num_inference_steps 30 --seed 2048 \
--output_path outputs/scene.mp4--guidance_scale 1.0 disables classifier-free guidance, so each step needs one forward pass instead of two.
PGDS. Preprocess the Physion training data and train on seven rigid-body scenarios with 4 GPUs.
See gns_training/README.md.
cd gns_training
bash scripts/train_4gpu.sh <DATA_ROOT>Motion videos for training data. Extract the reference-visible and future-emergent videos from
MiraData clips with SpaTrackerV2. See tracking_video/README.md.
SPATRACKER_ROOT=/path/to/SpaTrackerV2 DATA_DIR=/path/to/videos OUTPUT_DIR=/path/to/tracking \
bash tracking_video/scripts/run_batch.shDual-motion video model. Fine-tune Wan2.2-TI2V-5B on the MiraData videos and their motion videos.
Run from the VideoX-Fun checkout, see video_generation/README.md.
cd /path/to/VideoX-Fun
bash scripts/wan2.2_fun/train_dual_control.sh \
--model_dir /path/to/Wan2.2-TI2V-5B \
--data_root /path/to/miradata \
--train_meta /path/to/miradata/metadata_dual_control_4999.jsonThis codebase builds on SAM-3D-Objects, SAM3, Physion and Physion-particles, SpaTrackerV2, Diffusion as Shader, VideoX-Fun, Wan2.2 and MiraData. We thank the authors for making their work publicly available.
If you find this work useful, please cite:
@inproceedings{bao2026seeing,
title = {Seeing the Unseen: Unified Visible--Invisible Motion for Physically Consistent Video Generation},
author = {Bao, Mingyang and Ye, Fangda and Chen, Xihao and Li, Bobo and Li, Jason and Wu, Shengqiong},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}The license is to be announced. binvox_rw.py, used by scene_reconstruction and gns_training, is
distributed under GPL-3.0. Upstream projects keep their own licenses.





