Awesome

<img width="55%" alt="VideoSys" src="./assets/figures/logo.png?raw=true"> <h3 align="center"> An easy and efficient system for video generation </h3> | <a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#installation">Quick Start</a> | <a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#usage">Supported Models</a> | <a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#acceleration-techniques">Accelerations</a> | <a href="https://discord.gg/WhPmYm9FeG">Discord</a> | <a href="https://oahzxl.notion.site/VideoSys-News-42391db7e0a44f96a1f0c341450ae472?pvs=4">Media</a> | <a href="https://huggingface.co/VideoSys">HuggingFace Space</a> |

Latest News 🔥

[2024/11] 🔥 Propose Data-Centric Parallel (DCP) [blog][doc], a simple and efficient method for variable sequences (e.g., videos) training.
[2024/09] Support CogVideoX, Vchitect-2.0 and Open-Sora-Plan v1.2.0.
[2024/08] 🔥 Evole from OpenDiT to VideoSys: An easy and efficient system for video generation.
[2024/08] 🔥 Release PAB paper: Real-Time Video Generation with Pyramid Attention Broadcast.
[2024/06] 🔥 Propose Pyramid Attention Broadcast (PAB) [paper][blog][doc], the first approach to achieve real-time DiT-based video generation, delivering negligible quality loss without requiring any training.
[2024/06] Support Open-Sora-Plan and Latte.
[2024/03] 🔥 Propose Dynamic Sequence Parallel (DSP)[paper][doc], achieves 3x speed for training and 2x speed for inference in Open-Sora compared with sota sequence parallelism.
[2024/03] Support Open-Sora.
[2024/02] 🎉 Release OpenDiT: An Easy, Fast and Memory-Efficent System for DiT Training and Inference.

About

VideoSys is an open-source project that provides a user-friendly and high-performance infrastructure for video generation. This comprehensive toolkit will support the entire pipeline from training and inference to serving and compression.

We are committed to continually integrating cutting-edge open-source video models and techniques. Stay tuned for exciting enhancements and new features on the horizon!

Installation

Prerequisites:

Python >= 3.10
PyTorch >= 1.13 (We recommend to use a >2.0 version)
CUDA >= 11.6

We strongly recommend using Anaconda to create a new environment (Python >= 3.10) to run our examples:

conda create -n videosys python=3.10 -y
conda activate videosys

Install VideoSys:

git clone https://github.com/NUS-HPC-AI-Lab/VideoSys
cd VideoSys
pip install -e .

Usage

VideoSys supports many diffusion models with our various acceleration techniques, enabling these models to run faster and consume less memory.

You can find all available models and their supported acceleration techniques in the following table. Click Code to see how to use them.

<table> <tr> <th rowspan="2">Model</th> <th rowspan="2">Train</th> <th rowspan="2">Infer</th> <th colspan="3">Acceleration Techniques</th> </tr> <tr> <th><a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#dyanmic-sequence-parallelism-dsp-paperdoc">DSP</a></th> <th><a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#pyramid-attention-broadcast-pab-blogdoc">PAB</a></th> <th><a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#pyramid-attention-broadcast-pab-blogdoc">DCP</a></th> </tr> <tr> <td>Vchitect [<a href="https://github.com/Vchitect/Vchitect-2.0">source</a>]</td> <td align="center">/</td> <td align="center"><a href="./examples/inference/vchitect/sample.py">Code</a></td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">/</td> </tr> <tr> <td>CogVideoX [<a href="https://github.com/THUDM/CogVideo">source</a>]</td> <td align="center">🟡</td> <td align="center"><a href="./examples/inference/cogvideox/sample.py">Code</a></td> <td align="center">/</td> <td align="center">✅</td> <td align="center">🟡</td> </tr> <tr> <td>Latte [<a href="https://github.com/Vchitect/Latte">source</a>]</td> <td align="center">/</td> <td align="center"><a href="./examples/inference/latte/sample.py">Code</a></td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">/</td> </tr> <tr> <td>Open-Sora-Plan [<a href="https://github.com/PKU-YuanGroup/Open-Sora-Plan">source</a>]</td> <td align="center">/</td> <td align="center"><a href="./examples/inference/open_sora_plan/sample.py">Code</a></td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">/</td> </tr> <tr> <td>Open-Sora [<a href="https://github.com/hpcaitech/Open-Sora">source</a>]</td> <td align="center"><a href="./examples/training/open_sora">Code</a></td> <td align="center"><a href="./examples/inference/open_sora/sample.py">Code</a></td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">✅</td> </tr> </table>

You can also find easy demo with HuggingFace Space <a href="https://huggingface.co/VideoSys">[link]</a> and Gradio <a href="./gradio">[link]</a>. 🟡 means work in progress.

Acceleration Techniques

Data-Centric Parallel (DCP) [blog][doc]

<img src="./assets/figures/dcp_overview.png" alt="method" height="300"> Data-Centric Parallel (DCP) is a simple but effective approach to accelerate distributed training of variable sequences. Unlike previous methods that fix training settings, DCP dyanmically adjusts parallelism and other configs driven by incoming data during runtime, achieving up to 2.1x speedup. As a ease-of-use method, DCP can enpower any video models and parallel methods with minimal code changes.

See its details here.

Pyramid Attention Broadcast (PAB) [paper][blog][doc]

method

PAB is the first approach to achieve real-time DiT-based video generation, delivering lossless quality without requiring any training. By mitigating redundant attention computation, PAB achieves up to 21.6 FPS with 10.6x acceleration, without sacrificing quality across popular DiT-based video generation models including Open-Sora, Latte and Open-Sora-Plan.

See its details here.

Dyanmic Sequence Parallelism (DSP) [paper][doc]

dsp_overview

DSP is a novel, elegant and super efficient sequence parallelism for Open-Sora, Latte and other multi-dimensional transformer architecture.

It achieves 3x speed for training and 2x speed for inference in Open-Sora compared with sota sequence parallelism (DeepSpeed Ulysses). For a 10s (80 frames) of 512x512 video, the inference latency of Open-Sora is:

Method	1xH800	8xH800 (DS Ulysses)	8xH800 (DSP)
Latency(s)	106	45	22

See its details here.

Contributing

We welcome and value any contributions and collaborations. Please check out CONTRIBUTING.md for how to get involved.

Contributors

Star History

Citation

@misc{videosys2024,
  author={VideoSys Team},
  title={VideoSys: An Easy and Efficient System for Video Generation},
  year={2024},
  publisher={GitHub},
  url = {https://github.com/NUS-HPC-AI-Lab/VideoSys},
}

@misc{zhao2024pab,
  title={Real-Time Video Generation with Pyramid Attention Broadcast},
  author={Xuanlei Zhao and Xiaolong Jin and Kai Wang and Yang You},
  year={2024},
  eprint={2408.12588},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2408.12588},
}

@misc{zhao2024dsp,
  title={DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers},
  author={Xuanlei Zhao and Shenggan Cheng and Chang Chen and Zangwei Zheng and Ziming Liu and Zheming Yang and Yang You},
  year={2024},
  eprint={2403.10266},
  archivePrefix={arXiv},
  primaryClass={cs.DC},
  url={https://arxiv.org/abs/2403.10266},
}

@misc{zhao2024opendit,
  author={Xuanlei Zhao, Zhongkai Zhao, Ziming Liu, Haotian Zhou, Qianli Ma, and Yang You},
  title={OpenDiT: An Easy, Fast and Memory-Efficient System for DiT Training and Inference},
  year={2024},
  publisher={GitHub},
  url={https://github.com/NUS-HPC-AI-Lab/VideoSys/tree/v1.0.0},
}