Home

Awesome

DigiHuman

Digihuman is a project which aims to automatically generate <b>whole body pose animation + facial animation</b> on 3D Character models based on the camera input. <br/> This project is my B.Sc thesis of Computer Engineering at Amirkabir University of Technology(AUT).

About DigiHuman

DigiHuman is a system for bringing automation in animation generation on 3D virtual characters. It uses Pose estimation and facial landmark generator models to create entire body and face animation on 3D virtual characters. <br/> DigiHuman is developed with MediaPipe and Unity3D. MediaPipe generates 3D landmarks for the human whole body and face, and Unity3D is used to render the final animation after processing the generated landmarks from MediaPipe. The diagram below, shows the whole architucture of the application.

<div align="center"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/dataFlow.png?raw=true" alt="Logo"> </div>

Sample Outputs of the project

<div align="center"> <a href="https://youtu.be/maUUXfe_EcU">Project demo</a> | <a href="https://youtu.be/L62w5AMaFOk">Tutorial</a> </div>

Hands animations

<div align="center"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/2828_ok.gif"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/2828_ok.gif?raw=true" alt="Logo"> </a> <a href="https://thumbs.gfycat.com/VibrantDearestKomododragon-size_restricted.gif"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/2828_1t05.gif?raw=true" alt="Logo"> </a> </div>

Full body animation

<div align="center"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/figure_headphone.gif"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/figure_headphone.gif" alt="Logo"> </a> <a href="https://gfycat.com/braveglumguanaco"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/hands_greek.gif" alt="Logo"> </a> </div>

Face animation

<div align="center"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/mouth_deform_1_japan.gif"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/mouth_deform_1_japan.gif?raw=true" alt="Logo"> </a> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/blinks_1_japan.gif"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/blinks_1_japan.gif?raw=true" alt="Logo"> </a> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/mouth_1_japan.gif"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/mouth_1_japan.gif?raw=true" alt="Logo"> </a> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/mouth_dir_1_japan.gif"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/mouth_dir_1_japan.gif?raw=true" alt="Logo"> </a> </div> <!-- # Donation Do you want to support me in this project? :D <p align="left"> <a href="https://ko-fi.com/danialkord"> <img src="https://raw.githubusercontent.com/SMotlaq/LoRa/master/bmc.png" width="200" alt="Buy me a Coffee"/> </a> </p> --> <!-- GETTING STARTED -->

Installation

Follow the instructions to run the program!

Backend server installtion

  1. Install MediaPipe python.
 pip install mediapipe
  1. Install OpenCV python.
 pip install opencv-python
  1. Go to backend directory and install other requirements:
 pip install -r requirements.txt
  1. You'll need to download the pre-trained generator model for the COCO dataset and place it into backend/checkpoints/coco_pretrained/.

Unity3D Installation

Install Unity3D and its requirements by the following guidelines(Skip 1-3 if Unity3D is already installed).

  1. Download and install UnityHub
  2. Add a new license in UnityHub and register it
  3. Install a Unity Editor inside UnityHub(LTS versions and a version higher than 2020.3.25f1 are recommended).
  4. In the Unity project setting, allow HTTP connections in the player setting.
<div align="center"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/http.png"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/http.png?raw=true" alt="Logo"> </a> </div>
  1. Download and import the following packages into your project to enable the recording option available with FFmpeg(Download .unitypackage files and drag them to your project).

Usage

Adding new 3D characters

You can add your characters to the project! Characters should have a standard Humanoid rig to show kinematic animations. For rendering face animations, characters should have a facial rig(Blendmesh).</br> Follow these steps to add your character:

  1. Find a 3D character model from Unity asset store or download a free one(You can download them from websites like Mixamo).
  2. Open the character setting and set the rig to humanoid
<div align="left"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/3.png"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/3.png?raw=true" alt="Logo" width="300" height="150"> </a> </div>
  1. Drag and drop your 3D character model to CharacterChooser/CharacterSlideshow/Parent object in Unity main Scene like the image below
<div align="left"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/1.png"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/1.png?raw=true" alt="Logo" width="300" height="400"> </a> </div>
  1. Add BlendShapeController and QualityData components to the character object in the scene(which is dragged inside the Parent object in the last step).
  2. Set BlendShapeController values
<div align="left"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/5.png"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/5.png?raw=true" alt="Logo" > </a> </div> <div align="left"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/6.png"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/6.png?raw=true" alt="Logo" width="300" height="400"> </a> </div>
  1. Open CharacterSlideshow Object on CharacterChooser/CharacterSlideshow path inside the scene hierarchy, then add a new dragged character to the nodes property(all characters should be referenced inside nodes).
<div align="left"> <a href="https://github.com/Danial-Kord/DigiHuman/blob/images/images/8.jpg"> <img src="https://github.com/Danial-Kord/DigiHuman/blob/images/images/8.jpg?raw=true" alt="Logo" width="500" height="300"> </a> </div>
  1. Run the application and you can now select your character for rendering animation!

Features

<!-- ROADMAP --> <!-- ## Available features --> <!-- ## TODO -->

Resources

Licenses & Citations

DigiHuman Licence

Application License: GPL-3.0 license Non-commercial use only. If you distribute or communicate copies of the modified or unmodified Program, or any portion thereof, you must provide appropriate credit to Danial Kordmodanlou as the original author of the Program. This attribution should be included in any location where the Program is used or displayed.

FFmpeg</br>

GauGan

@inproceedings{park2019SPADE,
  title={Semantic Image Synthesis with Spatially-Adaptive Normalization},
  author={Park, Taesung and Liu, Ming-Yu and Wang, Ting-Chun and Zhu, Jun-Yan},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2019}
}

3D Characters

Unity-chan model & mixamo models

<!-- CONTACT -->

Contact

Danial Kordmodanlou - kordmodanloo@gmail.com

Website : danial-kord.github.io

Project Link: github.com/Danial-Kord/DigiHuman

Telegram ID: @Danial_km