Efficient End-to-end Diffusion Model for Onestep SAR-to-Optical Translation

🚀 E3Diff won the 1st in CVPR PBVS2025 Multi-modal Aerial View Image Challenge - T (Translation) 🎉

Brief

This is an official implementation of Efficient End-to-end Diffusion Model for Onestep SAR-to-Optical Translation (E3Diff) by PyTorch.

• [√] released dataset
• [√] log / logger
• [√] metrics evaluation
• [√] multi-gpu support
• [√] resume training / pretrained model
• [√] [Weights and Biases Logging]
• [√] 1/multi steps training and sampling
• [√] SEN12 results of baseline methods are released
• [√] Pretrained weights of both SAR2EO and SEN12 are released

Pipeline

Result of SEN12 Dataset

Result of SAR2EO Dataset

Usage

Environment

• create a new environment:

$ conda env create -f environment.yml

• install softpool.

$ cd SoftPool/pytorch
$ make install
--- (optional) ---
$ make test

Dataset

SAR-EO dataset: baiduyun code gn72

SEN12 dataset: google drive

Training:

Download the dataset, and train your model using the following commands (about 1 week using 2 A6000 48GB GPU):

SAR2EO DATA

Filtered SAR data and canny maps are use in SAR2EO task.

# 1. Prepare filtered SAR data using SAR2EO_filter.m
# 2. Prepare canny map data using canny_dataset.py
# 3. stage 1 training for SAR2EO dataset
python main.py --config 'config/SAR2EO_256_s1.json'

# 4. stage 2 training for SAR2EO dataset
python main.py --config 'config/SAR2EO_256_s2_1step.json'

You might be willing to download the well-trained model of SAR2EO from here(code is 0615), and test the model.

SEN12 DATA

# stage 1 training for sen12 dataset (PPB filtering is not used for SEN12)
python main.py --config 'config/SEN12_256_s1.json'

# stage 2 training for sen12 dataset (PPB filtering is not used for SEN12)
python main.py --config 'config/SEN12_256_s2_1step.json'

Also, you might be willing to download the well-trained model of SEN12 from here, and test the model. If needed, results of some baseline methods can also be downloaded here(code is tf95).

# stage 2 validation for SAR2EO dataset
# If you want to reproduce results of SAR2EO, please use the matlab code 'SAR2EO_filter.m' to filter speckles and use canny_dataset.py to extract canny maps of SAR images before feeding into E3Diff.
python main.py --config 'config/SAR2EO_256_s2_test.json' --phase 'val'  --seed 1

# stage 2 validation for sen12 dataset
python main.py --config 'config/SEN12_256_s2_test.json' --phase 'val'  --seed 1

🚀 Weights and Biases 🎉

The library now supports experiment tracking, model checkpointing and model prediction visualization with Weights and Biases. You will need to install W&B and login by using your access token.

pip install wandb

# get your access token from wandb.ai/authorize
wandb login

Acknowledgements

Our work is mainly based on the following projects:

• https://github.com/Janspiry/Image-Super-Resolution-via-Iterative-Refinement
• https://github.com/GaParmar/img2img-turbo
• https://github.com/alexandrosstergiou/SoftPool

Citation

If you find the project useful, please cite the papers:

@ARTICLE{10767752,
  author={Qin, Jiang and Zou, Bin and Li, Haolin and Zhang, Lamei},
  journal={IEEE Geoscience and Remote Sensing Letters}, 
  title={Efficient End-to-End Diffusion Model for One-step SAR-to-Optical Translation}, 
  year={2024},
  volume={},
  number={},
  pages={1-1},
  doi={10.1109/LGRS.2024.3506566}}

@article{qin2024conditional,
  title={Conditional Diffusion Model with Spatial-Frequency Refinement for SAR-to-Optical Image Translation},
  author={Qin, Jiang and Wang, Kai and Zou, Bin and Zhang, Lamei and van de Weijer, Joost},
  journal={IEEE Transactions on Geoscience and Remote Sensing},
  year={2024},
  publisher={IEEE}
}