Home

Awesome

<h1 align='center'>Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation</h1> <div align='center'> <a href='https://github.com/xumingw' target='_blank'>Mingwang Xu</a><sup>1*</sup>&emsp; <a href='https://github.com/crystallee-ai' target='_blank'>Hui Li</a><sup>1*</sup>&emsp; <a href='https://github.com/subazinga' target='_blank'>Qingkun Su</a><sup>1*</sup>&emsp; <a href='https://github.com/NinoNeumann' target='_blank'>Hanlin Shang</a><sup>1</sup>&emsp; <a href='https://github.com/AricGamma' target='_blank'>Liwei Zhang</a><sup>1</sup>&emsp; <a href='https://github.com/cnexah' target='_blank'>Ce Liu</a><sup>3</sup>&emsp; </div> <div align='center'> <a href='https://jingdongwang2017.github.io/' target='_blank'>Jingdong Wang</a><sup>2</sup>&emsp; <a href='https://yoyo000.github.io/' target='_blank'>Yao Yao</a><sup>4</sup>&emsp; <a href='https://sites.google.com/site/zhusiyucs/home' target='_blank'>Siyu Zhu</a><sup>1</sup>&emsp; </div> <div align='center'> <sup>1</sup>Fudan University&emsp; <sup>2</sup>Baidu Inc&emsp; <sup>3</sup>ETH Zurich&emsp; <sup>4</sup>Nanjing University </div> <br> <div align='center'> <a href='https://github.com/fudan-generative-vision/hallo'><img src='https://img.shields.io/github/stars/fudan-generative-vision/hallo?style=social'></a> <a href='https://fudan-generative-vision.github.io/hallo/#/'><img src='https://img.shields.io/badge/Project-HomePage-Green'></a> <a href='https://arxiv.org/pdf/2406.08801'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://huggingface.co/fudan-generative-ai/hallo'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow'></a> <a href='https://huggingface.co/fudan-generative-ai/hallo'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Demo-yellow'></a> <a href='https://www.modelscope.cn/models/fudan-generative-vision/Hallo/summary'><img src='https://img.shields.io/badge/Modelscope-Model-purple'></a> <a href='assets/wechat.jpeg'><img src='https://badges.aleen42.com/src/wechat.svg'></a> </div> <br>

๐Ÿ“ธ Showcase

https://github.com/fudan-generative-vision/hallo/assets/17402682/9d1a0de4-3470-4d38-9e4f-412f517f834c

๐ŸŽฌ Honoring Classic Films

<table class="center"> <tr> <td style="text-align: center"><b>Devil Wears Prada</b></td> <td style="text-align: center"><b>Green Book</b></td> <td style="text-align: center"><b>Infernal Affairs</b></td> </tr> <tr> <td style="text-align: center"><a target="_blank" href="https://cdn.aondata.work/video/short_movie/Devil_Wears_Prada-480p.mp4"><img src="https://cdn.aondata.work/img/short_movie/Devil_Wears_Prada_GIF.gif"></a></td> <td style="text-align: center"><a target="_blank" href="https://cdn.aondata.work/video/short_movie/Green_Book-480p.mp4"><img src="https://cdn.aondata.work/img/short_movie/Green_Book_GIF.gif"></a></td> <td style="text-align: center"><a target="_blank" href="https://cdn.aondata.work/video/short_movie/ๆ— ้—ด้“-480p.mp4"><img src="https://cdn.aondata.work/img/short_movie/Infernal_Affairs_GIF.gif"></a></td> </tr> <tr> <td style="text-align: center"><b>Patch Adams</b></td> <td style="text-align: center"><b>Tough Love</b></td> <td style="text-align: center"><b>Shawshank Redemption</b></td> </tr> <tr> <td style="text-align: center"><a target="_blank" href="https://cdn.aondata.work/video/short_movie/Patch_Adams-480p.mp4"><img src="https://cdn.aondata.work/img/short_movie/Patch_Adams_GIF.gif"></a></td> <td style="text-align: center"><a target="_blank" href="https://cdn.aondata.work/video/short_movie/Tough_Love-480p.mp4"><img src="https://cdn.aondata.work/img/short_movie/Tough_Love_GIF.gif"></a></td> <td style="text-align: center"><a target="_blank" href="https://cdn.aondata.work/video/short_movie/Shawshank-480p.mp4"><img src="https://cdn.aondata.work/img/short_movie/Shawshank_GIF.gif"></a></td> </tr> </table>

Explore more examples.

๐Ÿ“ฐ News

๐Ÿค Community Resources

Explore the resources developed by our community to enhance your experience with Hallo:

Thanks to all of them.

Join our community and explore these amazing resources to make the most out of Hallo. Enjoy and elevate their creative projects!

๐Ÿ”ง๏ธ Framework

abstract framework

โš™๏ธ Installation

prerequisites: 3.11>=python>=3.8, CUDA>=11.3 and ffmpeg.

Python and Git:

Give unrestricted script access to powershell so venv can work:

git clone --recurse-submodules https://github.com/sdbds/hallo-for-windows/

Install with Powershell run install.ps1 or install-cn.ps1(for Chinese)

๐Ÿ—๏ธ๏ธ Usage

The entry point for inference is scripts/inference.py. Before testing your cases, two preparations need to be completed:

  1. Download all required pretrained models.
  2. Prepare source image and driving audio pairs.
  3. Run inference.

or Powershell run with run_inference.ps1

No need Download models manually

## Download pretrained models

You can easily get all pretrained models required by inference from our HuggingFace repo.

Clone the the pretrained models into ${PROJECT_ROOT}/pretrained_models directory by cmd below:

shell~~ ~~git lfs install~~ ~~git clone https://huggingface.co/fudan-generative-ai/hallo pretrained_models~~ ~~

Or you can download them separately from their source repo:

- hallo: Our checkpoints consist of denoising UNet, face locator, image & audio proj. - audio_separator: Kim_Vocal_2 MDX-Net vocal removal model by KimberleyJensen. (Thanks to runwayml) - insightface: 2D and 3D Face Analysis placed into pretrained_models/face_analysis/models/. (Thanks to deepinsight) - face landmarker: Face detection & mesh model from mediapipe placed into pretrained_models/face_analysis/models. - motion module: motion module from AnimateDiff. (Thanks to guoyww). - sd-vae-ft-mse: Weights are intended to be used with the diffusers library. (Thanks to stablilityai) - StableDiffusion V1.5: Initialized and fine-tuned from Stable-Diffusion-v1-2. (Thanks to runwayml) - wav2vec: wav audio to vector model from Facebook.

Finally, these pretrained models should be organized as follows:

./pretrained_models/
|-- audio_separator/
|   |-- download_checks.json
|   |-- mdx_model_data.json
|   |-- vr_model_data.json
|   `-- Kim_Vocal_2.onnx
|-- face_analysis/
|   `-- models/
|       |-- face_landmarker_v2_with_blendshapes.task  # face landmarker model from mediapipe
|       |-- 1k3d68.onnx
|       |-- 2d106det.onnx
|       |-- genderage.onnx
|       |-- glintr100.onnx
|       `-- scrfd_10g_bnkps.onnx
|-- motion_module/
|   `-- mm_sd_v15_v2.ckpt
|-- sd-vae-ft-mse/
|   |-- config.json
|   `-- diffusion_pytorch_model.safetensors
|-- stable-diffusion-v1-5/
|   `-- unet/
|       |-- config.json
|       `-- diffusion_pytorch_model.safetensors
`-- wav2vec/
    `-- wav2vec2-base-960h/
        |-- config.json
        |-- feature_extractor_config.json
        |-- model.safetensors
        |-- preprocessor_config.json
        |-- special_tokens_map.json
        |-- tokenizer_config.json
        `-- vocab.json

๐Ÿ› ๏ธ Prepare Inference Data

Hallo has a few simple requirements for input data:

For the source image:

  1. It should be cropped into squares.
  2. The face should be the main focus, making up 50%-70% of the image.
  3. The face should be facing forward, with a rotation angle of less than 30ยฐ (no side profiles).

For the driving audio:

  1. It must be in WAV format.
  2. It must be in English since our training datasets are only in this language.
  3. Ensure the vocals are clear; background music is acceptable.

We have provided some samples for your reference.

๐ŸŽฎ Run Inference

Simply to run the scripts/inference.py and pass source_image and driving_audio as input:

python scripts/inference.py --source_image examples/reference_images/1.jpg --driving_audio examples/driving_audios/1.wav

Animation results will be saved as ${PROJECT_ROOT}/.cache/output.mp4 by default. You can pass --output to specify the output file name. You can find more examples for inference at examples folder.

For more options:

usage: inference.py [-h] [-c CONFIG] [--source_image SOURCE_IMAGE] [--driving_audio DRIVING_AUDIO] [--output OUTPUT] [--pose_weight POSE_WEIGHT]
                    [--face_weight FACE_WEIGHT] [--lip_weight LIP_WEIGHT] [--face_expand_ratio FACE_EXPAND_RATIO]

options:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
  --source_image SOURCE_IMAGE
                        source image
  --driving_audio DRIVING_AUDIO
                        driving audio
  --output OUTPUT       output video file name
  --pose_weight POSE_WEIGHT
                        weight of pose
  --face_weight FACE_WEIGHT
                        weight of face
  --lip_weight LIP_WEIGHT
                        weight of lip
  --face_expand_ratio FACE_EXPAND_RATIO
                        face region

or Powershell run with run_inference.ps1

You can edit run_inference.ps1 to change some configs.

๐Ÿ“…๏ธ Roadmap

StatusMilestoneETA
โœ…Inference source code meet everyone on GitHub2024-06-15
โœ…Pretrained models on Huggingface2024-06-15
๐ŸšงOptimizing Performance on images with a resolution of 256x256.2024-06-23
๐Ÿš€Improving the model's performance on Mandarin Chinese2024-06-25
๐Ÿš€Releasing data preparation and training scripts2024-06-28
<details> <summary>Other Enhancements</summary> </details>

๐Ÿ“ Citation

If you find our work useful for your research, please consider citing the paper:

@misc{xu2024hallo,
  title={Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation},
  author={Mingwang Xu and Hui Li and Qingkun Su and Hanlin Shang and Liwei Zhang and Ce Liu and Jingdong Wang and Yao Yao and Siyu zhu},
  year={2024},
  eprint={2406.08801},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

๐ŸŒŸ Opportunities Available

Multiple research positions are open at the Generative Vision Lab, Fudan University! Include:

Interested individuals are encouraged to contact us at siyuzhu@fudan.edu.cn for further information.

โš ๏ธ Social Risks and Mitigations

The development of portrait image animation technologies driven by audio inputs poses social risks, such as the ethical implications of creating realistic portraits that could be misused for deepfakes. To mitigate these risks, it is crucial to establish ethical guidelines and responsible use practices. Privacy and consent concerns also arise from using individuals' images and voices. Addressing these involves transparent data usage policies, informed consent, and safeguarding privacy rights. By addressing these risks and implementing mitigations, the research aims to ensure the responsible and ethical development of this technology.

๐Ÿค— Acknowledgements

We would like to thank the contributors to the magic-animate, AnimateDiff, ultimatevocalremovergui, AniPortrait and Moore-AnimateAnyone repositories, for their open research and exploration.

If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.

๐Ÿ‘ Community Contributors

Thank you to all the contributors who have helped to make this project better!

<a href="https://github.com/fudan-generative-vision/hallo/graphs/contributors"> <img src="https://contrib.rocks/image?repo=fudan-generative-vision/hallo" /> </a>