Awesome

VideoTuna

<a href='https://github.com/user-attachments/assets/a48d57a3-4d89-482c-8181-e0bce4f750fd'><img src='https://badges.aleen42.com/src/wechat.svg'></a>

🤗🤗🤗 Videotuna is a useful codebase for text-to-video applications.
🌟 VideoTuna is the first repo that integrates multiple AI video generation models for text-to-video, image-to-video, text-to-image generation (to the best of our knowledge).
🌟 VideoTuna is the first repo that provides comprehensive pipelines in video generation, including pre-training, continuous training, post-training (alignment), and fine-tuning (to the best of our knowledge).
🌟 The models of VideoTuna include both U-Net and DiT architectures for visual generation tasks.
🌟 A new 3D video VAE, and a controllable facial video generation model will be released soon.

Features

🌟 All-in-one framework: Inference and fine-tune up-to-date video generation models.
🌟 Pre-training: Build your own foundational text-to-video model.
🌟 Continuous training: Keep improving your model with new data.
🌟 Domain-specific fine-tuning: Adapt models to your specific scenario.
🌟 Concept-specific fine-tuning: Teach your models with unique concepts.
🌟 Enhanced language understanding: Improve model comprehension through continuous training.
🌟 Post-processing: Enhance the videos with video-to-video enhancement model.
🌟 Post-training/Human preference alignment: Post-training with RLHF for more attractive results.

🔆 Updates

• [2024-11-01] We make the VideoTuna V0.1.0 public!

Demo

Model Inference and Comparison

3D Video VAE

The 3D video VAE from VideoTuna can accurately compress and reconstruct the input videos with fine details.

Face domain

Storytelling

⏰ TODOs

• More demo and applications
• More functionalities such as control modules. (Suggestions are welcome!)

🔆 Information

Code Structure

VideoTuna/
    ├── assets       # put images for readme
    ├── checkpoints  # put model checkpoints here
    ├── configs      # model and experimental configs
    ├── data         # data processing scripts and dataset files
    ├── docs         # documentations
    ├── eval         # evaluation scripts
    ├── inputs       # input examples for testing 
    ├── scripts      # train and inference python scripts
    ├── shsripts     # train and inference shell scripts
    ├── src          # model-related source code
    ├── tests        # testing scripts
    ├── tools        # some tool scripts

Supported Models

• Note: H: height; W: width; L: length

Please check docs/CHECKPOINTS.md to download all the model checkpoints.

🔆 Get started

1.Prepare environment

conda create --name videotuna python=3.10 -y
conda activate videotuna
pip install -r requirements.txt
git clone https://github.com/JingyeChen/SwissArmyTransformer
pip install -e SwissArmyTransformer/
git clone https://github.com/tgxs002/HPSv2.git
cd ./HPSv2
pip install -e .
cd ..
conda config --add channels conda-forge
conda install ffmpeg

2.Prepare checkpoints

Please follow docs/CHECKPOINTS.md to download model checkpoints.
After downloading, the model checkpoints should be placed as Checkpoint Structure.

3.Inference state-of-the-art T2V/I2V/T2I models

• Inference a set of text-to-video models in one command: bash tools/video_comparison/compare.sh
  • The default mode is to run all models, e.g., inference_methods="videocrafter2;dynamicrafter;cogvideo—t2v;cogvideo—i2v;opensora"
  • If the users want to inference specific models, modify the inference_methods variable in compare.sh, and list the desired models separated by semicolons.
  • Also specify the input directory via the input_dir variable. This directory should contain a prompts.txt file, where each line corresponds to a prompt for the video generation. The default input_dir is inputs/t2v
• Inference a set of image-to-video models in one command: bash tools/video_comparison/compare_i2v.sh

• Inference a specific model, run the corresponding commands as follows:

4. Finetune T2V models

4.1 Prepare dataset

Please follow the docs/datasets.md to try provided toydataset or build your own datasets.

4.2 Fine-tune

1. VideoCrafter2 Full Fine-tuning

Before started, we assume you have finished the following two preliminary steps:

1. Install the environment
2. Prepare the dataset
3. Download the checkpoints and get these two checkpoints

  ll checkpoints/videocrafter/t2v_v2_512/model.ckpt
  ll checkpoints/stablediffusion/v2-1_512-ema/model.ckpt

First, run this command to convert the VC2 checkpoint as we make minor modifications on the keys of the state dict of the checkpoint. The converted checkpoint will be automatically save at checkpoints/videocrafter/t2v_v2_512/model_converted.ckpt.

python tools/convert_checkpoint.py --input_path checkpoints/videocrafter/t2v_v2_512/model.ckpt

Second, run this command to start training on the single GPU. The training results will be automatically saved at results/train/${CURRENT_TIME}_${EXPNAME}

bash shscripts/train_videocrafter_v2.sh

2. VideoCrafter2 Lora Fine-tuning

We support lora finetuning to make the model to learn new concepts/characters/styles.

• Example config file: configs/001_videocrafter2/vc2_t2v_lora.yaml
• Training lora based on VideoCrafter2: bash shscripts/train_videocrafter_lora.sh
• Inference the trained models: bash shscripts/inference_vc2_t2v_320x512_lora.sh

3. Open-Sora Fine-tuning

We support open-sora finetuning, you can simply run the following commands:

# finetune the Open-Sora v1.0
bash shscripts/train_opensorav10.sh

Finetuning for enhanced langugage understanding

5. Evaluation

We support VBench evaluation to evaluate the T2V generation performance. Please check eval/README.md for details.

Acknowledgement

We thank the following repos for sharing their awesome models and codes!

• Mochi: A new SOTA in open-source video generation models
• VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
• VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
• DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
• Open-Sora: Democratizing Efficient Video Production for All
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• VADER: Video Diffusion Alignment via Reward Gradients
• VBench: Comprehensive Benchmark Suite for Video Generative Models
• Flux: Text-to-image models from Black Forest Labs.
• SimpleTuner: A fine-tuning kit for text-to-image generation.

Some Resources

• LLMs-Meet-MM-Generation: A paper collection of utilizing LLMs for multimodal generation (image, video, 3D and audio).
• MMTrail: A multimodal trailer video dataset with language and music descriptions.
• Seeing-and-Hearing: A versatile framework for Joint VA generation, V2A, A2V, and I2A.
• Self-Cascade: A Self-Cascade model for higher-resolution image and video generation.
• ScaleCrafter and HiPrompt: Free method for higher-resolution image and video generation.
• FreeTraj and FreeNoise: Free method for video trajectory control and longer-video generation.
• Follow-Your-Emoji, Follow-Your-Click, and Follow-Your-Pose: Follow family for controllable video generation.
• Animate-A-Story: A framework for storytelling video generation.
• LVDM: Latent Video Diffusion Model for long video generation and text-to-video generation.

🍻 Contributors

📋 License

Please follow CC-BY-NC-ND. If you want a license authorization, please contact the project leads Yingqing He (yhebm@connect.ust.hk) and Yazhou Xing (yxingag@connect.ust.hk).

😊 Citation

@software{videotuna,
  author = {Yingqing He and Yazhou Xing and Zhefan Rao and Haoyu Wu and Zhaoyang Liu and Jingye Chen and Pengjun Fang and Jiajun Li and Liya Ji and Runtao Liu and Xiaowei Chi and Yang Fei and Guocheng Shao and Yue Ma and Qifeng Chen},
  title = {VideoTuna: A Powerful Toolkit for Video Generation with Model Fine-Tuning and Post-Training},
  month = {Nov},
  year = {2024},
  url = {https://github.com/VideoVerses/VideoTuna}
}

Star History