Awesome

This repository is the official implementation of paper "Animate-X: Universal Character Image Animation with Enhanced Motion Representation". Animate-X is a universal animation framework based on latent diffusion models for various character types (collectively named X), including anthropomorphic characters.

📌 Updates

• [2024.12.20] 🔥 We release our Animate-X inference codes.
• [2024.12.10] 🔥 We release our Animate-X CKPT checkpoints.
• [2024.10.14] 🔥 Our paper is in public on arxiv.

🌄 Gallery

Introduction

Animations produced by Animate-X

🚀 Installation

Install with conda:

conda env create -f environment.yaml
conda activate animate-x

🚀 Download Checkpoints

Download Animate-X checkpoints and put all files in checkpoints dir, which should be like:

./checkpoints/
|---- animate-x.pth 
|---- dw-ll_ucoco_384.onnx
|---- open_clip_pytorch_model.bin
|---- v2-1_512-ema-pruned.ckpt
└---- yolox_l.onnx

💡 Inference

The default inputs are a image (.jpg) and a dance video (.mp4). The default output is a 32-frame video (.mp4) with 768x512 resolution, which will be saved in ./results dir.

1. pre-process the video.

   python process_data.py \
       --source_video_paths data/videos \
       --saved_pose_dir data/saved_pkl \
       --saved_pose data/saved_pose \
       --saved_frame_dir data/saved_frames
   

2. run Animate-X.

   python inference.py --cfg configs/Animate_X_infer.yaml 
   

Some key parameters in the .yaml configuration file are described as follows. For example, users can adjust the max_frames or sampling interval of the dance video to generate videos of varying durations or speeds.

• max_frames: Number of frames (default as 32) in the generated video (fps: 8).
  • If you want to generage longer video with more frames, you should modify
    • max_frames as the number of frames
    • seq_len in UNet as the number of frames + 1
  • We take 96 frames as an example, and the config should be:

    {
        max_frames: 96   # 1. modify `max_frames` as the number of frames (e.g. 96)
        ......
        UNet: {
            ......
            'use_sim_mask': False,
            'seq_len': 97, # 2. modify `seq_len` in `UNet` as the number of frames + 1  (e.g. 97 = 96 + 1)
        }
    }
    

• round: The number of times each test case is generated.
• test_list_path: The input paths for all test cases, for example:

      [
          [2, "data/images/1.jpg", "data/saved_pose/dance_1","data/saved_frames/dance_1","data/saved_pkl/dance_1.pkl", 14],
          [2, "data/images/4.png", "data/saved_pose/dance_1","data/saved_frames/dance_1","data/saved_pkl/dance_1.pkl", 14],
          ......
      ]
  

  • 2 indicates that 1 frame is sampled from every 2 frames of the reference dance video to be used as input for the model.
  • "data/images/1.jpg" indicates the path to the reference image.
  • "data/saved_pose/dance_1" indicates the path to the saved pose images. (output by process_data.py, $I^p$, keypoints visualization)
  • "data/saved_frames/dance_1" indicates the path to the saved frames from the driven video. (output by process_data.py)
  • "data/saved_pkl/dance_1.pkl" indicates the path to the saved pose keypoints. (output by process_data.py, $p^d$, DWPose)
  • 14 indicates the random seed.
• log_dir: path to the generated animation videos, e.g., ./results.

✔ Some tips:

Although Animate-x does not rely on strict pose alignment and we did not perform any manual alignment operations for all the results in the paper, we cannot guarantee that all cases are perfect. Therefore, users can perform handmade pose alignment operations themselves, e.g, applying the overall x/y translation and scaling on the pose skeleton of each frame to align with the position of the subject in the reference image. (put in data/saved_pose)

📧 Acknowledgement

Our implementation is based on UniAnimate, MimicMotion, and MusePose. Thanks for their remarkable contribution and released code! If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.

⚖ License

This repository is released under the Apache-2.0 license as found in the LICENSE file.

📚 Citation

If you find this codebase useful for your research, please use the following entry.

@article{AnimateX2025,
  title={Animate-X: Universal Character Image Animation with Enhanced Motion Representation},
  author={Tan, Shuai and Gong, Biao and Wang, Xiang and Zhang, Shiwei and Zheng, Dandan and Zheng, Ruobing and Zheng, Kecheng and Chen, Jingdong and Yang, Ming},
  journal={arXiv preprint arXiv:2410.10306},
  year={2025}
}

@article{Mimir2025,
  title={Mimir: Improving Video Diffusion Models for Precise Text Understanding},
  author={Tan, Shuai and Gong, Biao and Feng, Yutong and Zheng, Kecheng and Zheng, Dandan and Shi, Shuwei and Shen, Yujun and Chen, Jingdong and Yang, Ming},
  journal={arXiv preprint arXiv:2412.03085},
  year={2025}
}