Awesome

VideoTuna

<a href='https://github.com/user-attachments/assets/a48d57a3-4d89-482c-8181-e0bce4f750fd'><img src='https://badges.aleen42.com/src/wechat.svg'></a>

🤗🤗🤗 Videotuna is a useful codebase for text-to-video applications.
🌟 VideoTuna is the first repo that integrates multiple AI video generation models for text-to-video, image-to-video, text-to-image generation (to the best of our knowledge).
🌟 VideoTuna is the first repo that provides comprehensive pipelines in video generation, including pre-training, continuous training, post-training (alignment), and fine-tuning (to the best of our knowledge).
🌟 The models of VideoTuna include both U-Net and DiT architectures for visual generation tasks.
🌟 A new 3D video VAE, and a controllable facial video generation model will be released soon.

Features

🌟 All-in-one framework: Inference and fine-tune up-to-date video generation models.
🌟 Pre-training: Build your own foundational text-to-video model.
🌟 Continuous training: Keep improving your model with new data.
🌟 Domain-specific fine-tuning: Adapt models to your specific scenario.
🌟 Concept-specific fine-tuning: Teach your models with unique concepts.
🌟 Enhanced language understanding: Improve model comprehension through continuous training.
🌟 Post-processing: Enhance the videos with video-to-video enhancement model.
🌟 Post-training/Human preference alignment: Post-training with RLHF for more attractive results.

🔆 Updates

• [2024-11-01] We make the VideoTuna V0.1.0 public!

Demo

Model Inference and Comparison

3D Video VAE

The 3D video VAE from VideoTuna can accurately compress and reconstruct the input videos with fine details.

Face domain

Storytelling

⏰ TODOs

• More demo and applications
• More functionalities such as control modules. (Suggestions are welcome!)

🔆 Information

Code Structure

VideoTuna/
    ├── assets       # put images for readme
    ├── checkpoints  # put model checkpoints here
    ├── configs      # model and experimental configs
    ├── data         # data processing scripts and dataset files
    ├── docs         # documentations
    ├── eval         # evaluation scripts
    ├── inputs       # input examples for testing 
    ├── scripts      # train and inference python scripts
    ├── shsripts     # train and inference shell scripts
    ├── src          # model-related source code
    ├── tests        # testing scripts
    ├── tools        # some tool scripts

Supported Models

• Note: H: height; W: width; L: length

Please check docs/CHECKPOINTS.md to download all the model checkpoints.

Title## 🔆 Get started

1.Prepare environment

conda create --name videotuna python=3.10 -y
conda activate videotuna
pip install -U poetry pip
poetry config virtualenvs.create false
poetry install
pip install optimum-quanto==0.2.1
pip install -r requirements.txt
git clone https://github.com/JingyeChen/SwissArmyTransformer
pip install -e SwissArmyTransformer/
rm -rf SwissArmyTransformer
git clone https://github.com/tgxs002/HPSv2.git
cd ./HPSv2
pip install -e .
cd ..

2.Prepare checkpoints

Please follow docs/CHECKPOINTS.md to download model checkpoints.
After downloading, the model checkpoints should be placed as Checkpoint Structure.

3.Inference state-of-the-art T2V/I2V/T2I models

• Inference a set of text-to-video models in one command: bash tools/video_comparison/compare.sh
  • The default mode is to run all models, e.g., inference_methods="videocrafter2;dynamicrafter;cogvideo—t2v;cogvideo—i2v;opensora"
  • If the users want to inference specific models, modify the inference_methods variable in compare.sh, and list the desired models separated by semicolons.
  • Also specify the input directory via the input_dir variable. This directory should contain a prompts.txt file, where each line corresponds to a prompt for the video generation. The default input_dir is inputs/t2v
• Inference a set of image-to-video models in one command: bash tools/video_comparison/compare_i2v.sh

• Inference a specific model, run the corresponding commands as follows:

4. Finetune T2V models

(1). Prepare Dataset

Please follow the docs/datasets.md to try provided toydataset or build your own datasets.

(2). Finetune

Open-Sora finetuning

We support open-sora finetuning, you can simply run the following commands:

# finetune the Open-Sora v1.0
bash shscripts/train_opensorav10.sh

Lora finetuning

We support lora finetuning to make the model to learn new concepts/characters/styles.

• Example config file: configs/001_videocrafter2/vc2_t2v_lora.yaml
• Training lora based on VideoCrafter2: bash shscripts/train_videocrafter_lora.sh
• Inference the trained models: bash shscripts/inference_vc2_t2v_320x512_lora.sh

Finetuning for enhanced langugage understanding

5. Evaluation

We support VBench evaluation to evaluate the T2V generation performance. Please check eval/README.md for details.

Acknowledgement

We thank the following repos for sharing their awesome models and codes!

• VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
• VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
• DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
• Open-Sora: Democratizing Efficient Video Production for All
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• VADER: Video Diffusion Alignment via Reward Gradients
• VBench: Comprehensive Benchmark Suite for Video Generative Models
• Flux: Text-to-image models from Black Forest Labs.
• SimpleTuner: A fine-tuning kit for text-to-image generation.

Some Resources

• LLMs-Meet-MM-Generation: A paper collection of utilizing LLMs for multimodal generation (image, video, 3D and audio).
• MMTrail: A multimodal trailer video dataset with language and music descriptions.
• Seeing-and-Hearing: A versatile framework for Joint VA generation, V2A, A2V, and I2A.
• Self-Cascade: A Self-Cascade model for higher-resolution image and video generation.
• ScaleCrafter and HiPrompt: Free method for higher-resolution image and video generation.
• FreeTraj and FreeNoise: Free method for video trajectory control and longer-video generation.
• Follow-Your-Emoji, Follow-Your-Click, and Follow-Your-Pose: Follow family for controllable video generation.
• Animate-A-Story: A framework for storytelling video generation.
• LVDM: Latent Video Diffusion Model for long video generation and text-to-video generation.

🍻 Contributors

📋 License

Please follow CC-BY-NC-ND. If you want a license authorization, please contact yhebm@connect.ust.hk and yxingag@connect.ust.hk.

😊 Citation

To be updated...

Star History