Skip to content
yeung-labPublic

About

SFG: inference-time guidance for SD3/SD3.5, FLUX and HunyuanVideo. Research inference source release.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

SFG

Training-free guidance for multimodal diffusion transformers

SFG provides inference-time guidance for image and video generation, without fine-tuning or additional learned weights.

Paper: Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

Supported models: SD3 Medium · SD3.5 Medium · FLUX.1-dev · FLUX-de-distill · HunyuanVideo-1.5 (T2V / I2V)

Installation

Use Python 3.10+ and a CUDA GPU.

git clone https://github.com/yyc1215lll-creator/SFG.git
cd SFG

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements-image.txt

For video generation, use a separate environment:

pip install -r video/third_party/HunyuanVideo-1.5/requirements.txt
pip install flash-attn==2.8.3 --no-build-isolation

Install a PyTorch build compatible with your CUDA environment before installing FlashAttention.

Prepare model weights

Download the model and pass its local directory with --weights.

Model Weights
SD3 Medium stabilityai/stable-diffusion-3-medium-diffusers
SD3.5 Medium stabilityai/stable-diffusion-3.5-medium
FLUX.1-dev black-forest-labs/FLUX.1-dev
FLUX-de-distill transformer InstantX/flux-dev-de-distill-diffusers
HunyuanVideo-1.5 Upstream model setup

Use complete Diffusers model directories for image models. FLUX-de-distill additionally requires a de-distilled transformer directory with guidance_embeds=false; pass it with --transformer. For HunyuanVideo, use the native Base 480p_t2v or 480p_i2v weights.

For example, after accepting any required model access terms:

hf auth login
hf download stabilityai/stable-diffusion-3.5-medium --local-dir models/sd35m
hf download black-forest-labs/FLUX.1-dev --local-dir models/flux-dev
hf download InstantX/flux-dev-de-distill-diffusers --local-dir models/flux-de-distill
hf download tencent/HunyuanVideo-1.5 --local-dir models/HunyuanVideo-1.5

The downloaded models/flux-de-distill directory itself is the transformer directory to pass to --transformer.

Image generation

Provide a text file with one prompt per line. Example prompts are included in examples/prompts.txt.

# SD3.5 Medium
python run_image.py --model sd35m --weights /path/to/SD3.5-medium \
  --prompt-file examples/prompts.txt --output outputs/sd35m

# SD3 Medium
python run_image.py --model sd3m --weights /path/to/SD3-medium \
  --prompt-file examples/prompts.txt --output outputs/sd3m

# FLUX.1-dev
python run_image.py --model flux-dev --weights /path/to/FLUX.1-dev \
  --prompt-file examples/prompts.txt --output outputs/flux-dev

# FLUX-de-distill
python run_image.py --model flux-de-distill --weights /path/to/FLUX.1-dev \
  --transformer /path/to/de-distilled/transformer \
  --prompt-file examples/prompts.txt --output outputs/flux-de-distill

Images are saved to OUTPUT/result/. Use a fresh output directory for each run. Use --batch-size 1 to reduce memory usage.

Video generation

Create a prompt manifest, then launch generation:

# Text-to-video
python prepare_video_manifest.py \
  --prompt 'A boat moves across a calm lake.' --output t2v.jsonl

torchrun --standalone --nproc_per_node=1 run_video.py --task t2v \
  --weights /path/to/HunyuanVideo-1.5 \
  --manifest t2v.jsonl --output outputs/t2v

# Image-to-video
python prepare_video_manifest.py \
  --prompt 'The subject slowly turns its head.' \
  --reference-image /path/to/reference.png --output i2v.jsonl

torchrun --standalone --nproc_per_node=1 run_video.py --task i2v \
  --weights /path/to/HunyuanVideo-1.5 \
  --manifest i2v.jsonl --output outputs/i2v

# Image-to-video with text-only guidance
torchrun --standalone --nproc_per_node=1 run_video.py --task i2v-text \
  --weights /path/to/HunyuanVideo-1.5 \
  --manifest i2v.jsonl --output outputs/i2v-text

Use --offloading to enable model offloading or --resume to resume a run.

Multi-GPU generation

For image generation, replace python with torchrun --standalone --nproc_per_node=8. For video generation, set --nproc_per_node=8 with a multi-row manifest. Each GPU loads a complete model and processes independent samples.

Citation

If you find this work useful, please cite our paper:

@article{yang2026imagestalkback,
  title={Images Talk Back: Guiding {MMDiTs} by Suppressing Image-to-Text Feedback},
  author={Yang, Yucheng and Sun, Xiaoxiao and Yang, Shusheng and Chen, Jiakang and Yeung-Levy, Serena},
  journal={arXiv preprint arXiv:2610.13941},
  year={2026},
  doi={10.48550/arXiv.2610.13941},
  url={https://arxiv.org/abs/2610.13941}
}

Acknowledgments

Built on Diffusers, Stable Diffusion, FLUX, and HunyuanVideo-1.5. Third-party code and model weights retain their respective licenses; see THIRD_PARTY_NOTICES.md.

About

SFG: inference-time guidance for SD3/SD3.5, FLUX and HunyuanVideo. Research inference source release.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages