Awesome

<img width="55%" alt="VideoSys" src="./assets/figures/logo.png?raw=true"> <h3 align="center"> An easy and efficient system for video generation </h3>

Latest News 🔥

[2024/08] 🔥 Evole from OpenDiT to VideoSys: An easy and efficient system for video generation.
[2024/08] 🔥 Release PAB paper: Real-Time Video Generation with Pyramid Attention Broadcast.
[2024/06] Propose Pyramid Attention Broadcast (PAB)[paper][blog][doc], the first approach to achieve real-time DiT-based video generation, delivering negligible quality loss without requiring any training.
[2024/06] Support Open-Sora-Plan and Latte.
[2024/03] Propose Dynamic Sequence Parallel (DSP)[paper][doc], achieves 3x speed for training and 2x speed for inference in Open-Sora compared with sota sequence parallelism.
[2024/03] Support Open-Sora: Democratizing Efficient Video Production for All.
[2024/02] 🎉 Release OpenDiT: An Easy, Fast and Memory-Efficent System for DiT Training and Inference.

About

VideoSys is an open-source project that provides a user-friendly and high-performance infrastructure for video generation. This comprehensive toolkit will support the entire pipeline from training and inference to serving and compression.

We are committed to continually integrating cutting-edge open-source video models and techniques. Stay tuned for exciting enhancements and new features on the horizon!

Installation

Prerequisites:

Python >= 3.10
PyTorch >= 1.13 (We recommend to use a >2.0 version)
CUDA >= 11.6

We strongly recommend using Anaconda to create a new environment (Python >= 3.10) to run our examples:

conda create -n videosys python=3.10 -y
conda activate videosys

Install VideoSys:

git clone https://github.com/NUS-HPC-AI-Lab/VideoSys
cd VideoSys
pip install -e .

Usage

VideoSys supports many diffusion models with our various acceleration techniques, enabling these models to run faster and consume less memory.

You can find all available models and their supported acceleration techniques in the following table. Click Doc to see how to use them.

<table> <tr> <th rowspan="2">Model</th> <th rowspan="2">Train</th> <th rowspan="2">Infer</th> <th colspan="2">Acceleration Techniques</th> <th rowspan="2">Usage</th> </tr> <tr> <th><a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#dyanmic-sequence-parallelism-dsp-paperdoc">DSP</a></th> <th><a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#pyramid-attention-broadcast-pab-blogdoc">PAB</a></th> </tr> <tr> <td>Open-Sora [<a href="https://github.com/hpcaitech/Open-Sora">source</a>]</td> <td align="center">🟡</td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">✅</td> <td align="center"><a href="./examples/open_sora/sample.py">Code</a></td> </tr> <tr> <td>Open-Sora-Plan [<a href="https://github.com/PKU-YuanGroup/Open-Sora-Plan">source</a>]</td> <td align="center">/</td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">✅</td> <td align="center"><a href="./examples/open_sora_plan/sample.py">Code</a></td> </tr> <tr> <td>Latte [<a href="https://github.com/Vchitect/Latte">source</a>]</td> <td align="center">/</td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">✅</td> <td align="center"><a href="./examples/latte/sample.py">Code</a></td> </tr> <tr> <td>CogVideoX [<a href="https://github.com/THUDM/CogVideo">source</a>]</td> <td align="center">/</td> <td align="center">✅</td> <td align="center">/</td> <td align="center">✅</td> <td align="center"><a href="./examples/cogvideox/sample.py">Code</a></td> </tr> </table>

Acceleration Techniques

Pyramid Attention Broadcast (PAB) [paper][blog][doc]

Real-Time Video Generation with Pyramid Attention Broadcast

Authors: Xuanlei Zhao1*, Xiaolong Jin2*, Kai Wang1*, and Yang You1 (* indicates equal contribution)

1National University of Singapore, 2Purdue University

method

PAB is the first approach to achieve real-time DiT-based video generation, delivering lossless quality without requiring any training. By mitigating redundant attention computation, PAB achieves up to 21.6 FPS with 10.6x acceleration, without sacrificing quality across popular DiT-based video generation models including Open-Sora, Latte and Open-Sora-Plan.

See its details here.

Dyanmic Sequence Parallelism (DSP) [paper][doc]

dsp_overview

DSP is a novel, elegant and super efficient sequence parallelism for Open-Sora, Latte and other multi-dimensional transformer architecture.

It achieves 3x speed for training and 2x speed for inference in Open-Sora compared with sota sequence parallelism (DeepSpeed Ulysses). For a 10s (80 frames) of 512x512 video, the inference latency of Open-Sora is:

Method	1xH800	8xH800 (DS Ulysses)	8xH800 (DSP)
Latency(s)	106	45	22

See its details here.

Contributing

We welcome and value any contributions and collaborations. Please check out CONTRIBUTING.md for how to get involved.

Contributors

Star History

Citation

@misc{videosys2024,
  author={VideoSys Team},
  title={VideoSys: An Easy and Efficient System for Video Generation},
  year={2024},
  publisher={GitHub},
  url = {https://github.com/NUS-HPC-AI-Lab/VideoSys},
}

@misc{zhao2024pab,
  title={Real-Time Video Generation with Pyramid Attention Broadcast},
  author={Xuanlei Zhao and Xiaolong Jin and Kai Wang and Yang You},
  year={2024},
  eprint={2408.12588},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2408.12588},
}

@misc{zhao2024dsp,
  title={DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers},
  author={Xuanlei Zhao and Shenggan Cheng and Chang Chen and Zangwei Zheng and Ziming Liu and Zheming Yang and Yang You},
  year={2024},
  eprint={2403.10266},
  archivePrefix={arXiv},
  primaryClass={cs.DC},
  url={https://arxiv.org/abs/2403.10266},
}

@misc{zhao2024opendit,
  author={Xuanlei Zhao, Zhongkai Zhao, Ziming Liu, Haotian Zhou, Qianli Ma, and Yang You},
  title={OpenDiT: An Easy, Fast and Memory-Efficient System for DiT Training and Inference},
  year={2024},
  publisher={GitHub},
  url={https://github.com/NUS-HPC-AI-Lab/VideoSys/tree/v1.0.0},
}