Awesome

FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling

🔥🔥🔥 The LongerCrafter for longer high-quality video generation are now released!

<div align="center"> <p style="font-weight: bold"> ✅ totally <span style="color: red; font-weight: bold">no</span> tuning      ✅ less than <span style="color: red; font-weight: bold">20%</span> extra time      ✅ support <span style="color: red; font-weight: bold">512</span> frames      </p>

Haonan Qiu, Menghan Xia*, Yong Zhang, Yingqing He, <br> Xintao Wang, Ying Shan, and Ziwei Liu* <br><br> (* corresponding author)

From Tencent AI Lab and Nanyang Technological University.

<p>Input: "A chihuahua in astronaut suit floating in space, cinematic lighting, glow effect"; <br> Resolution: 1024 x 576; Frames: 64.</p> <img src=assets/t2v/hd02.gif> <p>Input: "Campfire at night in a snowy forest with starry sky in the background"; <br> Resolution: 1024 x 576; Frames: 64.</p> </div>

🔆 Introduction

🤗🤗🤗 LongerCrafter (FreeNoise) is a tuning-free and time-efficient paradigm for longer video generation based on pretrained video diffusion models.

1. Longer Single-Prompt Text-to-video Generation

<div align="center"> <img src=assets/t2v/sp512.gif> <p>Longer single-prompt results. Resolution: 256 x 256; Frames: 512. (Compressed)</p> </div>

2. Longer Multi-Prompt Text-to-video Generation

<div align="center"> <img src=assets/t2v/mp256.gif> <p>Longer multi-prompt results. Resolution: 256 x 256; Frames: 256. (Compressed)</p> </div>

📝 Changelog

[2024.01.28]: 🔥🔥 Support FreeNoise on VideoCrafter2!
[2024.01.23]: 🔥🔥 Support FreeNoise on other two video frameworks AnimateDiff and LaVie!
[2023.10.25]: 🔥🔥 Release the 256x256 model and support multi-prompt generation!
[2023.10.24]: 🔥🔥 Release the LongerCrafter (FreeNoise), longer video generation! <br>

🧰 Models

Model	Resolution	Checkpoint	Description
VideoCrafter (Text2Video)	576x1024	Hugging Face	Support 64 frames on NVIDIA A100 (40GB)
VideoCrafter (Text2Video)	256x256	Hugging Face	Support 512 frames on NVIDIA A100 (40GB)
VideoCrafter2 (Text2Video)	320x512	Hugging Face	Support 128 frames on NVIDIA A100 (40GB)

(Reduce the number of frames when you have smaller GPUs, e.g. 256x256 resolutions with 64 frames.)

⚙️ Setup

Install Environment via Anaconda (Recommended)

conda create -n freenoise python=3.8.5
conda activate freenoise
pip install -r requirements.txt

💫 Inference

1. Longer Text-to-Video

Download pretrained T2V models via Hugging Face, and put the model.ckpt in checkpoints/base_1024_v1/model.ckpt.
Input the following commands in terminal.

  sh scripts/run_text2video_freenoise_1024.sh

2. Longer Multi-Prompt Text-to-Video

Download pretrained T2V models via Hugging Face, and put the model.ckpt in checkpoints/base_256_v1/model.ckpt.
Input the following commands in terminal.

  sh scripts/run_text2video_freenoise_mp_256.sh

🧲 Support For Other Models

FreeNoise is supposed to work on other similar frameworks. An easy way to test compatibility is by shuffling the noise to see whether a new similar video can be generated (set eta to 0). If your have any questions about applying FreeNoise to other frameworks, feel free to contact Haonan Qiu.

Current official implementation: FreeNoise-VideoCrafter, FreeNoise-AnimateDiff, FreeNoise-LaVie

👨‍👩‍👧‍👦 Crafter Family

VideoCrafter: Framework for high-quality video generation.

ScaleCrafter: Tuning-free method for high-resolution image/video generation.

TaleCrafter: An interactive story visualization tool that supports multiple characters.

😉 Citation

@misc{qiu2023freenoise,
      title={FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling}, 
      author={Haonan Qiu and Menghan Xia and Yong Zhang and Yingqing He and Xintao Wang and Ying Shan and Ziwei Liu},
      year={2023},
      eprint={2310.15169},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

📢 Disclaimer

We develop this repository for RESEARCH purposes, so it can only be used for personal/research/non-commercial purposes.