Awesome

CamI2V: Camera-Controlled Image-to-Video Diffusion Model

Official repo of paper for "CamI2V: Camera-Controlled Image-to-Video Diffusion Model".

Project page: https://zgctroy.github.io/CamI2V/

Abstract: Recent advancements have integrated camera pose as a user-friendly and physics-informed condition in video diffusion models, enabling precise camera control. In this paper, we identify one of the key challenges as effectively modeling noisy cross-frame interactions to enhance geometry consistency and camera controllability. We innovatively associate the quality of a condition with its ability to reduce uncertainty and interpret noisy cross-frame features as a form of noisy condition. Recognizing that noisy conditions provide deterministic information while also introducing randomness and potential misguidance due to added noise, we propose applying epipolar attention to only aggregate features along corresponding epipolar lines, thereby accessing an optimal amount of noisy conditions. Additionally, we address scenarios where epipolar lines disappear, commonly caused by rapid camera movements, dynamic objects, or occlusions, ensuring robust performance in diverse environments. Furthermore, we develop a more robust and reproducible evaluation pipeline to address the inaccuracies and instabilities of existing camera control metrics. Our method achieves a 25.64% improvement in camera controllability on the RealEstate10K dataset without compromising dynamics or generation quality and demonstrates strong generalization to out-of-domain images. Training and inference require only 24GB and 12GB of memory, respectively, for 16-frame sequences at 256×256 resolution. We will release all checkpoints, along with training and evaluation code. Dynamic videos are available for viewing on our project page.

:star2: News and Todo List

!!!!! Code is not complete and clean. Environment installer, training scripts, evaluation code and gradio demo are on the way. In addition, we implement camera control methods using code injected to lvdm, which is not easy for python beginners. We will reconstruct code in several weeks. !!!!

• :fire: 2024/12/23: Release 256x256 checkpoint of CamI2V. Higher resolution is on the way, please stay tuned!
• :fire: 2024/12/16: Release 256x256 checkpoint of MotionCtrl and CameraCtrl on DynamiCrafter.
• :fire: 2024/12/09: Release training configs and scripts.
• :fire: 2024/12/06: Release dataset pre-process code for RealEstate10K.
• :fire: 2024/12/02: Release evaluation code for RotErr, TransErr, CamMC and FVD.
• :fire: 2024/11/16: Update project page for more visualization.
• :fire: 2024/11/16: Release model code for training and inference of CamI2V, including implementation for MotionCtrl and CameraCtrl.
• :seedling: 2024/10/14: Release of checkpoints, training, and evaluation codes in months.

:chart_with_upwards_trend: Performance

• Our method demonstrates significant improvements over CameraCtrl, achieving a 32.96% reduction in Rotation Error, a 25.64% decrease in CamMC, and a 20.77% improvement in Translation Error, without decrease in FVD. These results were obtained using text and image CFG set to 7.5, 25 steps, and camera CFG set to 1.0 (no camera CFG).

• Compared with CamCo-like (arXiv in June) method, we improve 17.08%, 6.61%, 9.00% on RotErr, TransErr, and CamMC without FVD decrease, respectively.

Inference Speed and GPU Memory

Measured under 256x256 resolution, 16 frames, 25steps.

:gear: Environment

Quick Start

conda create -n cami2v python=3.10
conda activate cami2v

conda install -y pytorch==2.4.1 torchvision==0.19.1 pytorch-cuda=12.1 -c pytorch -c nvidia
conda install -y xformers -c xformers
pip install -r requirements.txt

:dizzy: Inference

Download Checkpoints

Currently we release checkpoints of DynamiCrafter-based CamI2v, CameraCtrl and MotionCtrl under 256x256 resolution on Huggingface. CamI2V with higher resolution is on the way, please stay tuned!

Download above checkpoints and put under ckpts folder. Please edit ckpt_path in models.json if you have a different model path.

Run Gradio Demo

python cami2v_gradio_app.py

Gradio may struggle to establish network connection, please re-try with --use_host_ip.

:rocket: Training

Prepare Dataset

Please follow instructions in datasets folder in this repo to download RealEstate10K dataset and pre-process necessary items like video_clips and valid_metadata.

Download Pretrained Models

Download pretrained weights of base model DynamiCrafter and put under pretrained_models folder:

─┬─ pretrained_models\
 └─┬─ DynamiCrafter\
   └─── model.ckpt

Launch

Start training by passing config yaml to --base argument of main/trainer.py. Example training configs are provided in configs folder.

torchrun --standalone --nproc_per_node 8 main/trainer.py --train \
    --logdir $(pwd)/logs \
    --base configs/<YOUR_CONFIG_NAME>.yaml \
    --name <YOUR_LOG_NAME>

:wrench: Evaluation

We calculate RotErr, TransErr, CamMC and FVD to evaluate camera controllability and visual quality. Code and installation guide for requirements are provided in evaluation folder, including COLMAP and GLOMAP. Support for VBench is planned in months as well.

:hugs: Related Repo

CameraCtrl: https://github.com/hehao13/CameraCtrl

MotionCtrl: https://github.com/TencentARC/MotionCtrl

DynamiCrafter: https://github.com/Doubiiu/DynamiCrafter

:spiral_notepad: Citation

@article{zheng2024cami2v,
  title={CamI2V: Camera-Controlled Image-to-Video Diffusion Model},
  author={Zheng, Guangcong and Li, Teng and Jiang, Rui and Lu, Yehao and Wu, Tao and Li, Xi},
  journal={arXiv preprint arXiv:2410.15957},
  year={2024}
}