Hi IVEBench authors,
Thank you for your excellent and comprehensive benchmark work!
I am currently reproducing the Wan2.1-VACE baseline results reported in your paper. However, my local reproductions using the Wan2.1-VACE 1.3B model have yielded very poor results, particularly regarding instruction following. To help align my local setup with your benchmark's evaluation, could you please clarify the following implementation details?
-
Codebase: diffusers vs. ali-vilab/VACE
Did you use the official ali-vilab/VACE repository or the diffusers pipeline for your evaluation? This heavily impacts hyperparameter settings. For example, the shift parameter behaves differently; diffusers utilizes a flow_shift=3.0 for 480P, whereas the native repository defaults to a sample_shift of 16.
-
Model Scale: 1.3B vs. 14B
Which model size was evaluated? The paper mentions a peak VRAM usage of 122–133GB on dual H20 GPUs, which leads me to suspect the 14B model was used. Could you confirm if the 1.3B model was tested at all, and if so, how it performed?
-
Video Ingestion Pipeline (Crucial)
How exactly was the source video introduced to VACE for the benchmark?
Since VACE is a multi-conditional model, I am trying to understand the exact conditioning pathway used:
Did you directly encode the source RGB via the VAE for semantic editing?
Or did you use an online preprocessing step to extract structural signals (e.g., running DepthAnything to get a depth map) and use that control(depth) + reference image as the condition? Understanding whether VACE directly "sees" the source RGB or only its structural abstraction is critical for my downstream experiments.
Thank you very much for your time and guidance!
Hi IVEBench authors,
Thank you for your excellent and comprehensive benchmark work!
I am currently reproducing the Wan2.1-VACE baseline results reported in your paper. However, my local reproductions using the Wan2.1-VACE 1.3B model have yielded very poor results, particularly regarding instruction following. To help align my local setup with your benchmark's evaluation, could you please clarify the following implementation details?
Codebase: diffusers vs. ali-vilab/VACE
Did you use the official ali-vilab/VACE repository or the diffusers pipeline for your evaluation? This heavily impacts hyperparameter settings. For example, the shift parameter behaves differently; diffusers utilizes a flow_shift=3.0 for 480P, whereas the native repository defaults to a sample_shift of 16.
Model Scale: 1.3B vs. 14B
Which model size was evaluated? The paper mentions a peak VRAM usage of 122–133GB on dual H20 GPUs, which leads me to suspect the 14B model was used. Could you confirm if the 1.3B model was tested at all, and if so, how it performed?
Video Ingestion Pipeline (Crucial)
How exactly was the source video introduced to VACE for the benchmark?
Since VACE is a multi-conditional model, I am trying to understand the exact conditioning pathway used:
Did you directly encode the source RGB via the VAE for semantic editing?
Or did you use an online preprocessing step to extract structural signals (e.g., running DepthAnything to get a depth map) and use that control(depth) + reference image as the condition? Understanding whether VACE directly "sees" the source RGB or only its structural abstraction is critical for my downstream experiments.
Thank you very much for your time and guidance!