Awesome

Yunxin Li, Haoyuan Shi, Baotian Hu*, Longyue Wang*,<br>Jiashun Zhu, Jinyi Xu, Zhen Zhao, and Min Zhang

(* Corresponding Authors)

Harbin Institute of Technology, Shenzhen

🚀 Welcome to the repo of Anim-Director.

If you appreciate our project, please consider giving us a star ⭐ on GitHub to stay updated with the latest developments. </h2>

🎏 Abstract

<b>TL; DR: <font color="red">Anim-Director</font> is an autonomous animation-making agent where LMM interacts seamlessly with generative tools to create detailed animated videos from simple narratives.</b>

Traditional animation generation methods depend on training generative models with human-labelled data, entailing a sophisticated multi-stage pipeline that demands substantial human effort and incurs high training costs. Due to limited prompting plans, these methods typically produce brief, information-poor, and context-incoherent animations. To overcome these limitations and automate the animation process, we pioneer the introduction of large multimodal models (LMMs) as the core processor to build an autonomous animation-making agent, named Anim-Director. This agent mainly harnesses the advanced understanding and reasoning capabilities of LMMs and generative AI tools to create animated videos from concise narratives or simple instructions. Specifically, it operates in three main stages: Firstly, the Anim-Director generates a coherent storyline from user inputs, followed by a detailed director’s script that encompasses settings of character profiles and interior/exterior descriptions, and context-coherent scene descriptions that include appearing characters, interiors or exteriors, and scene events. Secondly, we employ LMMs with the image generation tool to produce visual images of settings and scenes. These images are designed to maintain visual consistency across different scenes using a visual-language prompting method that combines scene descriptions and images of the appearing character and setting. Thirdly, scene images serve as the foundation for producing animated videos, with LMMs generating prompts to guide this process. The whole process is notably autonomous without manual intervention, as the LMMs interact seamlessly with generative tools to generate prompts, evaluate visual quality, and select the best one to optimize the final output. To assess the effectiveness of our framework, we collect varied short narratives and incorporate various Image/video evaluation metrics including visual consistency and video quality. The experimental results and case studies demonstrate the Anim-Director’s versatility and significant potential to streamline animation creation.

⚔️ Overview

🌈 Visual Example

🎨 Comparison

🌰 More Examples

⚡️ Usage

Midjourney and Pika are paid, while Stable Diffusion 3 and PIA are free. If you want to achieve the animation generation effect shown in our paper and demo, please choose Midjourney for T2I and Pika for (T+I)2V.
Welcome to contact us for more details (including how to integrate Pika into our agent).

conda create -n AnimDirector python==3.10.11
conda activate AnimDirector
pip install -r requirements.txt

To use Stable Diffusion 3 for T2I, you need to upgrade your torch version along with all related packages.

conda install git-lfs
git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/

git clone https://huggingface.co/Leoxing/PIA models/PIA/

bash download_bashscripts/2-RcnzCartoon.sh

To use MJ for T2I, you need to prepare stable-diffusion-webui following instructions Here.
After that, run:

bash code/StableDiffusion/webui.sh --nowebui

Sign up for an Imgur account.
Obtain your Imgur client_id, client_secret, access_token, refresh_token following instructions Here.

python code/script_gen.py

• The generated scripts will be saved as code/result/scripts.json.

Since Midjourney does not provide official API services, we use a third-party API platform GoAPI for mj_api_key.
Run the following command to get the (T+I)2V results:

python code/image_gen_mj.py

• The generated images will be saved in code/result/image/mj.

python code/image_gen_pia.py

• The generated images will be saved in code/result/image/sd3.

Inference for (T+I)2V with with PIA

python code/video_gen.py

• The generated videos will be saved in code/result/video.

Citation

@inproceedings{li_animdirector,
author = {Li, Yunxin and Shi, Haoyuan and Hu, Baotian and Wang, Longyue and Zhu, Jiashun and Xu, Jinyi and Zhao, Zhen and Zhang, Min},
title = {Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation},
year = {2024},
isbn = {9798400711312},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3680528.3687688},
doi = {10.1145/3680528.3687688},
booktitle = {SIGGRAPH Asia 2024 Conference Papers},
articleno = {34},
numpages = {11},
series = {SA '24}
}