Awesome

Latest News 🔥

• [2024/09] Support CogVideoX, Vchitect-2.0 and Open-Sora-Plan v1.2.0.
• [2024/08] 🔥 Evole from OpenDiT to <b>VideoSys: An easy and efficient system for video generation</b>.
• [2024/08] 🔥 Release PAB paper: <b>Real-Time Video Generation with Pyramid Attention Broadcast</b>.
• [2024/06] 🔥 Propose Pyramid Attention Broadcast (PAB)[paper][blog][doc], the first approach to achieve <b>real-time</b> DiT-based video generation, delivering <b>negligible quality loss</b> without <b>requiring any training</b>.
• [2024/06] Support Open-Sora-Plan and Latte.
• [2024/03] 🔥 Propose Dynamic Sequence Parallel (DSP)[paper][doc], achieves 3x speed for training and 2x speed for inference in Open-Sora compared with sota sequence parallelism.
• [2024/03] Support Open-Sora.
• [2024/02] 🎉 Release OpenDiT: An Easy, Fast and Memory-Efficent System for DiT Training and Inference.

About

VideoSys is an open-source project that provides a user-friendly and high-performance infrastructure for video generation. This comprehensive toolkit will support the entire pipeline from training and inference to serving and compression.

We are committed to continually integrating cutting-edge open-source video models and techniques. Stay tuned for exciting enhancements and new features on the horizon!

Installation

Prerequisites:

• Python >= 3.10
• PyTorch >= 1.13 (We recommend to use a >2.0 version)
• CUDA >= 11.6

We strongly recommend using Anaconda to create a new environment (Python >= 3.10) to run our examples:

conda create -n videosys python=3.10 -y
conda activate videosys

Install VideoSys:

git clone https://github.com/NUS-HPC-AI-Lab/VideoSys
cd VideoSys
pip install -e .

Usage

VideoSys supports many diffusion models with our various acceleration techniques, enabling these models to run faster and consume less memory.

<b>You can find all available models and their supported acceleration techniques in the following table. Click Code to see how to use them.</b>

You can also find easy demo with HuggingFace Space <a href="https://huggingface.co/VideoSys">[link]</a> and Gradio <a href="./gradio">[link]</a>.

Acceleration Techniques

Pyramid Attention Broadcast (PAB) [paper][blog][doc]

Real-Time Video Generation with Pyramid Attention Broadcast

Authors: Xuanlei Zhao^1*, Xiaolong Jin^2*, Kai Wang^1*, and Yang You¹ (* indicates equal contribution)

¹National University of Singapore, ²Purdue University

PAB is the first approach to achieve <b>real-time</b> DiT-based video generation, delivering <b>lossless quality</b> without <b>requiring any training</b>. By mitigating redundant attention computation, PAB achieves up to 21.6 FPS with 10.6x acceleration, without sacrificing quality across popular DiT-based video generation models including Open-Sora, Latte and Open-Sora-Plan.

See its details here.

Dyanmic Sequence Parallelism (DSP) [paper][doc]

DSP is a novel, elegant and super efficient sequence parallelism for Open-Sora, Latte and other multi-dimensional transformer architecture.

It achieves 3x speed for training and 2x speed for inference in Open-Sora compared with sota sequence parallelism (DeepSpeed Ulysses). For a 10s (80 frames) of 512x512 video, the inference latency of Open-Sora is:

See its details here.

Contributing

We welcome and value any contributions and collaborations. Please check out CONTRIBUTING.md for how to get involved.

Contributors

Star History

Citation

@misc{videosys2024,
  author={VideoSys Team},
  title={VideoSys: An Easy and Efficient System for Video Generation},
  year={2024},
  publisher={GitHub},
  url = {https://github.com/NUS-HPC-AI-Lab/VideoSys},
}

@misc{zhao2024pab,
  title={Real-Time Video Generation with Pyramid Attention Broadcast},
  author={Xuanlei Zhao and Xiaolong Jin and Kai Wang and Yang You},
  year={2024},
  eprint={2408.12588},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2408.12588},
}

@misc{zhao2024dsp,
  title={DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers},
  author={Xuanlei Zhao and Shenggan Cheng and Chang Chen and Zangwei Zheng and Ziming Liu and Zheming Yang and Yang You},
  year={2024},
  eprint={2403.10266},
  archivePrefix={arXiv},
  primaryClass={cs.DC},
  url={https://arxiv.org/abs/2403.10266},
}

@misc{zhao2024opendit,
  author={Xuanlei Zhao, Zhongkai Zhao, Ziming Liu, Haotian Zhou, Qianli Ma, and Yang You},
  title={OpenDiT: An Easy, Fast and Memory-Efficient System for DiT Training and Inference},
  year={2024},
  publisher={GitHub},
  url={https://github.com/NUS-HPC-AI-Lab/VideoSys/tree/v1.0.0},
}