Awesome

Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model

<a href='https://arxiv.org/abs/2406.15735'><img src='https://img.shields.io/badge/arXiv-2406.15735-b31b1b.svg'></a>   <a href='https://cond-image-leak.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a>   <a href='https://zhuanlan.zhihu.com/p/705300028?utm_campaign=shareopn&utm_medium=social&utm_psn=1788953432507416577&utm_source=wechat_session'><img src='https://img.shields.io/badge/Zhihu-Page-Blue'></a>   <a href='https://huggingface.co/spaces/Xiang-cd/DynamiCrafter-CIL'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20DynamiCrafter%20CIL%20Hugging%20Face%20I2V-Demo-blue'></a> 

Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, Jun Zhu

🔆 Overview

Diffusion models have obtained substantial progress in image-to-video (I2V) generation. However, such models are not fully understood. In this paper, we report a significant but previously overlooked issue in I2V diffusion models (I2V-DMs), namely, conditional image leakage. I2V-DMs tend to over-rely on the conditional image at large time steps, neglecting the crucial task of predicting the clean video from noisy inputs, which results in videos lacking dynamic and vivid motion. We further address this challenge from both inference and training aspects by presenting plug-and-play strategies accordingly. These strategies are validated on various I2V-DMs including DynamiCrafter, SVD and VideoCrafter1.

✅ To do list:

• [2024.06.25]: Release code, models and project page.
• [2024.08.11]: Add inference code for Animate-Anything and PIA
• Release the models without watermarks and in other resolutions.

⚙️ Setup Environment

Our plug-and-play strategies can be applied to various I2V-DMs, allowing the direct use of their original environments. For example, to set up DynamiCrafter:

cd examples/DynamiCrafter
conda create -n dynamicrafter python=3.8.5
conda activate dynamicrafter
pip install -r requirements.txt

To set up VideoCrafter1:

cd examples/VideoCrafter
conda create -n videocrafter python=3.8.5
conda activate videocrafter
pip install -r requirements.txt

To set up SVD:

cd examples/SVD
conda create -n svd python=3.9.18
conda activate svd
pip install -r requirements.txt

To set up Animate-Anything:

cd examples/animate-anything
conda create -n animation python=3.10
conda activate animation
pip install -r requirements.txt

To set up PIA:

cd examples/PIA
conda env create -f pia.yml
conda activate pia

☀️ Dataset

Download the WebVid dataset from here, where we use Webvid-2M subset. Put .csv file in examples/dataset/results_2M_train.csv and video data in examples/dataset/. We use the raw data without any filters.

🧊 Inference Strategy

We clone the repo of DynamiCrafter and VideoCrafter1 and implement SVD by ourselves. We apply our plug and plug-and-play strategies on them.

🎒 Initial Noise Distribution

😄 Example Results

DynamiCrafter

1. Download the original DynamiCrafter checkpoints from the repository and put it in examples/DynamiCrafter/ckpt/original ,or download our DynamiCrafter-CIL from here and put it in examples/DynamiCrafter/ckpt/finetuned. Download the initial noise in the above table and put it in examples/DynamiCrafter/ckpt/.
2. Run the following commands:

cd examples/DynamiCrafter

# for original DynamiCrafter with 320x512 resolution
sh inference_512.sh

# for our DynamiCrafter-CIL with 320x512 resolution
sh inference_CIL_512.sh

# for our DynamiCrafter-CIL with 576x1024 resolution
sh inference_CIL_1024.sh

The relevant parameters in inference.sh for our strategy are explained as follows:

• M: the start timestep M.
• whether_analytic_init: indicates whether to use Analytic-Init; 0 means it is not applied, while 1 means it is applied
• analytic_init_path: the path for initializing the mean and variance of the noise if Analytic-Init is applied

Note that M=1000, whether_analytic_init=0 is the baseline.

The effect of start time M is as follows:

SVD

1. Download the pretrained SVD model and put it in examples/SVD/ckpt/pretrained/stable-video-diffusion-img2vid . Download our SVD-CIL from here and put it in examples/SVD/ckpt/finetuned. Download the initial noise in the above table and put them in examples/SVD/ckpt/.
2. Run the following commands:

cd examples/SVD

# for original SVD
sh inference.sh

# for SVD-CIL with 320x512 resolution
sh inference_CIL_512.sh
 

The relevant parameters for inference are set in examples/SVD/config/inference.yaml,which are explained as follows:

• sigma_max: the start time M.
• analytic_init_path: the path for initializing the mean and variance of the noise if Analytic-Init is applied

VideoCrafter1

1. Download the original VideoCrafter checkpoints from the repository and put it in examples/VideoCrafter/ckpt/original,or download our VideoCrafter-CIL from here and put it in examples/VideoCrafter/ckpt/finetuned.Download the initial noise in the above table and put them in examples/VideoCrafter/ckpt.
2. Run the following commands:

cd examples/VideoCrafter

# for original VideoCrafter with 320x512 resolution
sh inference_512.sh

# for VideoCrafter-CIL with 320x512 resolution
sh inference_CIL_512.sh

The relevant parameters in inference.sh for our strategy are explained as follows:

• M: the start time M
• analytic_init_path: the path for initializing the mean and variance of the noise if Analytic-Init is applied

Animate-Anything

1. Download the original Animate-Anything checkpoints from the repository and put it in cond-image-leakage/examples/animate-anything/output/latent/animate_anything_512_v1.02
2. Run the following commands:

cd cond-image-leakage/examples/animate-anything
python inference.py --config example/config/concert.yaml

The relevant parameters in cond-image-leakage/examples/PIA/example/config/base.yaml for our strategy are explained as follows:

• noise_scheduler_kwargs.num_train_timesteps: the start time M

PIA

1. Download the original Animate-Anything checkpoints from the repository and put them following cond-image-leakage/examples/PIA/README.md.
2. Run the following commands:

cd cond-image-leakage/examples/PIA

The relevant parameters in inference.sh for our strategy are explained as follows:

• M: the start time M

🔥 Training Strategy

Similar to the inference strategy, we finetune the baselines based on the repository DynamiCrafter, VideoCrafter1 and SVD.

😄 Example Results

DynamiCrafter

1. Download the DynamiCrafter checkpoints from the repository and put them in examples/DynamiCrafter/ckpt/original.
2. Run the following commands:

cd examples/DynamiCrafter
sh train.sh

The relevant parameters in train.sh for our strategy are explained as follows:

• beta_m: the maximum noise level.
• a: the exponent of center of the distribution: $\mu(t)=2t^a-1$, where $a > 0$.

The effect of beta_mis as follows:

The effect of a is as follows:

Lower a correspond to more dynamic motion and lower temporal consistency and image alignment.

SVD

1. Download the SVD checkpoints from the repository and put them in examples/SVD/ckpt/pretrained/stable-video-diffusion-img2vid;
2. Run the following commands:

cd examples/SVD
sh train.sh

The relevant parameters in examples/SVD/config/train.yamlfor our strategy are explained as follows:

• beta_m: the maximum noise level. Higher beta_m correspond to more dynamic motion and lower temporal consistency and image alignment.
• a: the exponent of center of the distribution: $\mu(t)=2t^a-1$, where $a > 0$. Lower $a$ correspond to more dynamic motion and lower temporal consistency and image alignment.

Note that original SVD first add noise on conditional image and then feed it into VAE. Here we first feed the conditional image into VAE and then add noise on the conditional latents.

VideoCrafter1

1. Download the VideoCrafter checkpoints from the repository and put them in examples/VideoCrafter/original/ckpt/.
2. Run the following commands:

cd examples/VideoCrafter
sh train.sh

The relevant parameters in train.sh for our strategy are explained as follows:

• beta_m: the maximum noise level. Higher beta_m correspond to more dynamic motion and lower temporal consistency and image alignment.
• a: the exponent of center of the distribution: $\mu(t)=2t^a-1$, where $a > 0$. Lower $a$ correspond to more dynamic motion and lower temporal consistency and image alignment.

🎒 Checkpoints

Naive fine-tuning and ours were trained under the same settings for fair comparison. In the future, we will release the model without watermark.

😄 Citation

If you find this repository helpful, please cite as:

@article{zhao2024identifying,
  title={Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model},
  author={Zhao, Min and Zhu, Hongzhou and Xiang, Chendong and Zheng, Kaiwen and Li, Chongxuan and Zhu, Jun},
  journal={arXiv preprint arXiv:2406.15735},
  year={2024}
}

❤️ Acknowledgements

This implementation is based on the following work:

• DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
• VideoCrafter1: Open Diffusion Models for High-Quality Video Generation Thanks to the authors for sharing their code and models.