Awesome

FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models

Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia*, and Ziwei Liu* <br> (* corresponding author)

🔆 Introduction

🤗🤗🤗 FreeTraj is a tuning-free method for trajectory-controllable video generation based on pre-trained video diffusion models.

Showcases (320x512)

<table class="center"> <td><img src=assets/demo/0047_1_0.4_0.3.gif width="256"></td> <td><img src=assets/demo/0026_0_0.4_0.4.gif width="256"></td> <td><img src=assets/demo/0035_1_0.35_0.35.gif width="256"></td> <tr> <td style="text-align:center;" width="256">"<b>A chihuahua</b> in an astronaut suit floating in the universe, cinematic lighting, glow effect."</td> <td style="text-align:center;" width="256">"<b>A swan</b> floating gracefully on a lake."</td> <td style="text-align:center;" width="256">"<b>A corgi</b> running on the grassland on the grassland."</td> <tr> </table > <table class="center"> <td><img src=assets/demo/0051_1_0.4_0.4.gif width="256"></td> <td><img src=assets/demo/0041_0_0.35_0.35.gif width="256"></td> <td><img src=assets/demo/0019_0_0.3_0.3.gif width="256"></td> <tr> <td style="text-align:center;" width="256">"<b>A barrel</b> floating in a river."</td> <td style="text-align:center;" width="256">"<b>A dog</b> running across the garden, photorealistic, 4k."</td> <td style="text-align:center;" width="256">"<b>A helicopter</b> hovering above a cityscape."</td> <tr> </table >

📝 Changelog

[2024.07.04]: 🔥🔥 Release the FreeTraj, trajectory controllable video generation!
[2024.07.09]: 🔥🔥 Release a user-friendly interface.
TODO: 1. a powerful mode for better control. <br>

🧰 Models

Model	Resolution	Checkpoint	Description
VideoCrafter2 (Text2Video)	320x512	Hugging Face

<br>

⚙️ Setup

Install Environment via Anaconda (Recommended)

conda create -n freetraj python=3.8.5
conda activate freetraj
pip install -r requirements.txt

<br>

🤗 Quick start with Gradio

  gradio app/app.py

<br>

💫 Inference with Command

1. Demo

Download pretrained T2V models via Hugging Face, and put the model.ckpt in checkpoints/base_512_v2/model.ckpt.
Input the following commands in terminal.

  sh scripts/run_text2video_freetraj_512.sh

2. Plan new trajectory

Write new trajectory files, the format should be frame index, h start, h end, w start, w end. In the current version, the bbox size should be the same. Please refer to prompts/freetraj/traj_l.txt.
Modify scripts/run_text2video_freetraj_512.sh and set $traj_file.
Slightly increase $ddim_edit to enhance the control ability, but may reduce the video quality.

<br>

😉 Citation

@misc{qiu2024freetraj,
      title={FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models}, 
      author={Haonan Qiu and Zhaoxi Chen and Zhouxia Wang and Yingqing He and Menghan Xia and Ziwei Liu},
      year={2024},
      eprint={2406.16863},
      archivePrefix={arXiv}
}

<br>

📢 Disclaimer

We develop this repository for RESEARCH purposes, so it can only be used for personal/research/non-commercial purposes. The success rate is not guaranteed due to the variety of generative video prior.