Home

Awesome

๐Ÿ“ท EasyAnimate | An End-to-End Solution for High-Resolution and Long Video Generation

๐Ÿ˜Š EasyAnimate is an end-to-end solution for generating high-resolution and long videos. We can train transformer based diffusion generators, train VAEs for processing long videos, and preprocess metadata.

๐Ÿ˜Š We use DIT and transformer as a diffuser for video and image generation.

๐Ÿ˜Š Welcome!

Arxiv Page Project Page Modelscope Studio Hugging Face Spaces Discord Page

English | ็ฎ€ไฝ“ไธญๆ–‡ | ๆ—ฅๆœฌ่ชž

Table of Contents

Introduction

EasyAnimate is a pipeline based on the transformer architecture, designed for generating AI images and videos, and for training baseline models and Lora models for Diffusion Transformer. We support direct prediction from pre-trained EasyAnimate models, allowing for the generation of videos with various resolutions, approximately 6 seconds in length, at 8fps (EasyAnimateV5, 1 to 49 frames). Additionally, users can train their own baseline and Lora models for specific style transformations.

We will support quick pull-ups from different platforms, refer to Quick Start.

New Features:

Function๏ผš

Our UI interface is as follows: ui

Quick Start

1. Cloud usage: AliyunDSW/Docker

a. From AliyunDSW

DSW has free GPU time, which can be applied once by a user and is valid for 3 months after applying.

Aliyun provide free GPU time in Freetier, get it and use in Aliyun PAI-DSW to start EasyAnimate within 5min!

DSW Notebook

b. From ComfyUI

Our ComfyUI is as follows, please refer to ComfyUI README for details. workflow graph

c. From docker

If you are using docker, please make sure that the graphics card driver and CUDA environment have been installed correctly in your machine.

Then execute the following commands in this way:

# pull image
docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:easyanimate

# enter image
docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:easyanimate

# clone code
git clone https://github.com/aigc-apps/EasyAnimate.git

# enter EasyAnimate's dir
cd EasyAnimate

# download weights
mkdir models/Diffusion_Transformer
mkdir models/Motion_Module
mkdir models/Personalized_Model

# Please use the hugginface link or modelscope link to download the EasyAnimateV5 model.
# I2V models
# https://huggingface.co/alibaba-pai/EasyAnimateV5-12b-zh-InP
# https://modelscope.cn/models/PAI/EasyAnimateV5-12b-zh-InP
# T2V models
# https://huggingface.co/alibaba-pai/EasyAnimateV5-12b-zh
# https://modelscope.cn/models/PAI/EasyAnimateV5-12b-zh

2. Local install: Environment Check/Downloading/Installation

a. Environment Check

We have verified EasyAnimate execution on the following environment:

The detailed of Windows:

The detailed of Linux:

We need about 60GB available on disk (for saving weights), please check! The video size for EasyAnimateV5-12B can be generated by different GPU Memory, including:

GPU memory384x672x72384x672x49576x1008x25576x1008x49768x1344x25768x1344x49
16GB๐Ÿงก๐ŸงกโŒโŒโŒโŒ
24GB๐Ÿงก๐Ÿงก๐Ÿงก๐ŸงกโŒโŒ
40GBโœ…โœ…โœ…โœ…โŒโŒ
80GBโœ…โœ…โœ…โœ…โœ…โœ…

โœ… indicates it can run under "model_cpu_offload", ๐Ÿงก represents it can run under "model_cpu_offload_and_qfloat8", โญ•๏ธ indicates it can run under "sequential_cpu_offload", โŒ means it can't run. Please note that running with sequential_cpu_offload will be slower.

Some GPUs that do not support torch.bfloat16, such as 2080ti and V100, require changing the weight_dtype in app.py and predict files to torch.float16 in order to run.

The generation time for EasyAnimateV5-12B using different GPUs over 25 steps is as follows:

GPU384x672x72384x672x49576x1008x25576x1008x49768x1344x25768x1344x49
A10 24GB~120s (4.8s/it)~240s (9.6s/it)~320s (12.7s/it)~750s (29.8s/it)โŒโŒ
A100 80GB~45s (1.75s/it)~90s (3.7s/it)~120s (4.7s/it)~300s (11.4s/it)~265s (10.6s/it)~710s (28.3s/it)

(โญ•๏ธ) indicates it can run with low_gpu_memory_mode=True, but at a slower speed, and โŒ means it can't run.

<details> <summary>(Obsolete) EasyAnimateV3:</summary>

The video size for EasyAnimateV3 can be generated by different GPU Memory, including:

GPU memory384x672x72384x672x144576x1008x72576x1008x144720x1280x72720x1280x144
12GBโญ•๏ธโญ•๏ธโญ•๏ธโญ•๏ธโŒโŒ
16GBโœ…โœ…โญ•๏ธโญ•๏ธโญ•๏ธโŒ
24GBโœ…โœ…โœ…โœ…โœ…โŒ
40GBโœ…โœ…โœ…โœ…โœ…โœ…
80GBโœ…โœ…โœ…โœ…โœ…โœ…
</details>

b. Weights

We'd better place the weights along the specified path:

EasyAnimateV5:

๐Ÿ“ฆ models/
โ”œโ”€โ”€ ๐Ÿ“‚ Diffusion_Transformer/
โ”‚   โ”œโ”€โ”€ ๐Ÿ“‚ EasyAnimateV5-12b-zh-InP/
โ”‚   โ””โ”€โ”€ ๐Ÿ“‚ EasyAnimateV5-12b-zh/
โ”œโ”€โ”€ ๐Ÿ“‚ Personalized_Model/
โ”‚   โ””โ”€โ”€ your trained trainformer model / your trained lora model (for UI load)

่ง†้ข‘ไฝœๅ“

The results displayed are all based on image.

EasyAnimateV5-12b-zh-InP

I2V

<table border="0" style="width: 100%; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/bb393b7c-ba33-494c-ab06-b314adea9fc1" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/cb0d0253-919d-4dd6-9dc1-5cd94443c7f1" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/09ed361f-c0c5-4025-aad7-71fe1a1a52b1" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/9f42848d-34eb-473f-97ea-a5ebd0268106" width="100%" controls autoplay loop></video> </td> </tr> </table> <table border="0" style="width: 100%; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/903fda91-a0bd-48ee-bf64-fff4e4d96f17" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/407c6628-9688-44b6-b12d-77de10fbbe95" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/ccf30ec1-91d2-4d82-9ce0-fcc585fc2f21" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/5dfe0f92-7d0d-43e0-b7df-0ff7b325663c" width="100%" controls autoplay loop></video> </td> </tr> </table> <table border="0" style="width: 100%; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/2b542b85-be19-4537-9607-9d28ea7e932e" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/c1662745-752d-4ad2-92bc-fe53734347b2" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/8bec3d66-50a3-4af5-a381-be2c865825a0" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/bcec22f4-732c-446f-958c-2ebbfd8f94be" width="100%" controls autoplay loop></video> </td> </tr> </table>

T2V

<table border="0" style="width: 100%; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/eccb0797-4feb-48e9-91d3-5769ce30142b" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/76b3db64-9c7a-4d38-8854-dba940240ceb" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/0b8fab66-8de7-44ff-bd43-8f701bad6bb7" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/9fbddf5f-7fcd-4cc6-9d7c-3bdf1d4ce59e" width="100%" controls autoplay loop></video> </td> </tr> </table> <table border="0" style="width: 100%; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/19c1742b-e417-45ac-97d6-8bf3a80d8e13" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/641e56c8-a3d9-489d-a3a6-42c50a9aeca1" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/2b16be76-518b-44c6-a69b-5c49d76df365" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/e7d9c0fc-136f-405c-9fab-629389e196be" width="100%" controls autoplay loop></video> </td> </tr> </table>

EasyAnimateV5-12b-zh-Control

<table border="0" style="width: 100%; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/53002ce2-dd18-4d4f-8135-b6f68364cabd" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/fce43c0b-81fa-4ab2-9ca7-78d786f520e6" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/b208b92c-5add-4ece-a200-3dbbe47b93c3" width="100%" controls autoplay loop></video> </td> <tr> <td> <video src="https://github.com/user-attachments/assets/3aec95d5-d240-49fb-a9e9-914446c7a4cf" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/60fa063b-5c1f-485f-b663-09bd6669de3f" width="100%" controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/4adde728-8397-42f3-8a2a-23f7b39e9a1e" width="100%" controls autoplay loop></video> </td> </tr> </table>

How to use

<h3 id="video-gen">1. Inference </h3>

a. Using Python Code

b. Using webui

c. From ComfyUI

Please refer to ComfyUI README for details.

d. GPU Memory Saving Schemes

Due to the large parameters of EasyAnimateV5, we need to consider GPU memory saving schemes to conserve memory. We provide a GPU_memory_mode option for each prediction file, which can be selected from model_cpu_offload, model_cpu_offload_and_qfloat8, and sequential_cpu_offload.

2. Model Training

A complete EasyAnimate training pipeline should include data preprocessing, Video VAE training, and Video DiT training. Among these, Video VAE training is optional because we have already provided a pre-trained Video VAE.

<h4 id="data-preprocess">a. data preprocessing</h4>

We have provided a simple demo of training the Lora model through image data, which can be found in the wiki for details.

A complete data preprocessing link for long video segmentation, cleaning, and description can refer to README in the video captions section.

If you want to train a text to image and video generation model. You need to arrange the dataset in this format.

๐Ÿ“ฆ project/
โ”œโ”€โ”€ ๐Ÿ“‚ datasets/
โ”‚   โ”œโ”€โ”€ ๐Ÿ“‚ internal_datasets/
โ”‚       โ”œโ”€โ”€ ๐Ÿ“‚ train/
โ”‚       โ”‚   โ”œโ”€โ”€ ๐Ÿ“„ 00000001.mp4
โ”‚       โ”‚   โ”œโ”€โ”€ ๐Ÿ“„ 00000002.jpg
โ”‚       โ”‚   โ””โ”€โ”€ ๐Ÿ“„ .....
โ”‚       โ””โ”€โ”€ ๐Ÿ“„ json_of_internal_datasets.json

The json_of_internal_datasets.json is a standard JSON file. The file_path in the json can to be set as relative path, as shown in below:

[
    {
      "file_path": "train/00000001.mp4",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "video"
    },
    {
      "file_path": "train/00000002.jpg",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "image"
    },
    .....
]

You can also set the path as absolute path as follow:

[
    {
      "file_path": "/mnt/data/videos/00000001.mp4",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "video"
    },
    {
      "file_path": "/mnt/data/train/00000001.jpg",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "image"
    },
    .....
]
<h4 id="vae-train">b. Video VAE training (optional)</h4>

Video VAE training is an optional option as we have already provided pre trained Video VAEs. If you want to train video vae, you can refer to README in the video vae section.

<h4 id="dit-train">c. Video DiT training </h4>

If the data format is relative path during data preprocessing, please set scripts/train.sh as follow.

export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/json_of_internal_datasets.json"

If the data format is absolute path during data preprocessing, please set scripts/train.sh as follow.

export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/json_of_internal_datasets.json"

Then, we run scripts/train.sh.

sh scripts/train.sh

For details on setting some parameters, please refer to Readme Train and Readme Lora.

<details> <summary>(Obsolete) EasyAnimateV1:</summary> If you want to train EasyAnimateV1. Please switch to the git branch v1. </details>

Model zoo

EasyAnimateV5:

NameTypeStorage SpaceHugging FaceModel ScopeDescription
EasyAnimateV5-12b-zh-InPEasyAnimateV534 GB๐Ÿค—Link๐Ÿ˜„LinkOfficial image-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports bilingual prediction in Chinese and English.
EasyAnimateV5-12b-zh-ControlEasyAnimateV534 GB๐Ÿค—Link๐Ÿ˜„LinkOfficial video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc. Supports video prediction at multiple resolutions (512, 768, 1024) and is trained with 49 frames at 8 frames per second. Bilingual prediction in Chinese and English is supported.
EasyAnimateV5-12b-zhEasyAnimateV534 GB๐Ÿค—Link๐Ÿ˜„LinkOfficial text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports bilingual prediction in Chinese and English.
<details> <summary>(Obsolete) EasyAnimateV4:</summary>
NameTypeStorage SpaceUrlHugging FaceDescription
EasyAnimateV4-XL-2-InP.tar.gzEasyAnimateV4Before extraction: 8.9 GB / After extraction: 14.0 GBDownload๐Ÿค—LinkOur official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 144 frames at a rate of 24 frames per second.
</details> <details> <summary>(Obsolete) EasyAnimateV3:</summary>
NameTypeStorage SpaceUrlHugging FaceDescription
EasyAnimateV3-XL-2-InP-512x512.tarEasyAnimateV318.2GBDownload๐Ÿค—LinkEasyAnimateV3 official weights for 512x512 text and image to video resolution. Training with 144 frames and fps 24
EasyAnimateV3-XL-2-InP-768x768.tarEasyAnimateV318.2GBDownload๐Ÿค—LinkEasyAnimateV3 official weights for 768x768 text and image to video resolution. Training with 144 frames and fps 24
EasyAnimateV3-XL-2-InP-960x960.tarEasyAnimateV318.2GBDownload๐Ÿค—LinkEasyAnimateV3 official weights for 960x960 text and image to video resolution. Training with 144 frames and fps 24
</details> <details> <summary>(Obsolete) EasyAnimateV2:</summary> | Name | Type | Storage Space | Url | Hugging Face | Description | |--|--|--|--|--|--| | EasyAnimateV2-XL-2-512x512.tar | EasyAnimateV2 | 16.2GB | [Download](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/Diffusion_Transformer/EasyAnimateV2-XL-2-512x512.tar) | [๐Ÿค—Link](https://huggingface.co/alibaba-pai/EasyAnimateV2-XL-2-512x512) | EasyAnimateV2 official weights for 512x512 resolution. Training with 144 frames and fps 24 | | EasyAnimateV2-XL-2-768x768.tar | EasyAnimateV2 | 16.2GB | [Download](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/Diffusion_Transformer/EasyAnimateV2-XL-2-768x768.tar) | [๐Ÿค—Link](https://huggingface.co/alibaba-pai/EasyAnimateV2-XL-2-768x768) | EasyAnimateV2 official weights for 768x768 resolution. Training with 144 frames and fps 24 | | easyanimatev2_minimalism_lora.safetensors | Lora of Pixart | 485.1MB | [Download](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/Personalized_Model/easyanimatev2_minimalism_lora.safetensors) | - | A lora training with a specifial type images. Images can be downloaded from [Url](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v2/Minimalism.zip). | </details> <details> <summary>(Obsolete) EasyAnimateV1:</summary>

1ใ€Motion Weights

NameTypeStorage SpaceUrlDescription
easyanimate_v1_mm.safetensorsMotion Module4.1GBdownloadTraining with 80 frames and fps 12

2ใ€Other Weights

NameTypeStorage SpaceUrlDescription
PixArt-XL-2-512x512.tarPixart11.4GBdownloadPixart-Alpha official weights
easyanimate_portrait.safetensorsCheckpoint of Pixart2.3GBdownloadTraining with internal portrait datasets
easyanimate_portrait_lora.safetensorsLora of Pixart654.0MBdownloadTraining with internal portrait datasets
</details>

TODO List

Contact Us

  1. Use Dingding to search group 77450006752 or Scan to join
  2. You need to scan the image to join the WeChat group or if it is expired, add this student as a friend first to invite you.
<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/group/dd.png" alt="ding group" width="30%"/> <img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/group/wechat.jpg" alt="Wechat group" width="30%"/> <img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/group/person.jpg" alt="Person" width="30%"/>

Reference

License

This project is licensed under the Apache License (Version 2.0).