Awesome

GANHead: Towards Generative Animatable Neural Head Avatars

Paper | Project Page | Video

Official implentation of CVPR 2023 paper GANHead: Towards Generative Animatable Neural Head Avatars.

Getting Started

Installation

1. Clone the repository and set up a conda environment with all dependencies as follows:

git clone https://github.com/wsj-sjtu/GANHead.git
cd GANHead
conda env create -f env.yml
conda activate ganhead
python setup.py install

2. Download the FLAME model (FLAME 2020), and put the generic_model.pkl file to /lib/flame/flame_model/.

3. Install kaolin for fast occupancy query from meshes:

git clone https://github.com/NVIDIAGameWorks/kaolin
cd kaolin
python setup.py develop

4. If PyTorch3D can not be installed correctly, install it manually following their instructions.

Run demos

1. Download pretrained models and test motion sequence (taken from IMavatar data) from Google drive and organize them according to the fllowing structure:

GANHead
│
└─── data
│    └─── yufeng_1812
└─── outputs
     └─── stage1_model
     └─── stage2_model

Training

Prepare Data

First download FaceVerse dataset following their instructions, and the fitted FLAME parameters form Google drive. Organize all files into the following structure:

GANHead
│
└─── data
     │
     └─── faceverse
          │
          └─── faceverse_dataset
          |    └─── 001_01, 001_02, ...
          └─── flame_params
               └─── 001_01_flame.pkl, 001_02_flame.pkl, ...

Next, run the pre-processing script to get ground truth occupancy, surface color and normal, and rendered multiview images and normal maps:

python preprocess.py --tot 1 --id 0

You can run multiple instances of the script in parallel by simply specifying --tot to be the number of total instances and --id to be the rank of current instance. For example, divide the task into 4 instances and run the first instance on GPU0: CUDA_VISIBLE_DEVICES=0 python preprocess.py --tot 4 --id 0.

Training

Our model is trained in two stages following the steps below:

1. Stage1: train the geometry network and deformation module.

python train.py expname=coarse datamodule=faceverse

This step takes around 12 hours on 4 NVIDIA 3090 GPUs.

2. Precompute

python precompute.py expname=coarse datamodule=faceverse agent_tot=1 agent_id=0

Multiple instances of the script can also be runned in parallel by specifying --agent_tot and --agent_id.

3. Stage2: train the normal and texture networks.

python train.py expname=fine datamodule=faceverse +experiments=fine starting_path='./outputs/coarse/checkpoints/last.ckpt'

This step takes around 12 hours on 4 NVIDIA 3090 GPUs.

Inference

Sampling

Face reenactment

Raw scan fitting

Acknowledgements

This project is built upon gdna. Some code snippets are also borrowed from DECA and IMavatar. Thanks for these great projects. We thank all the authors for their great work and repos.

Citation

If you find our code or paper useful, please cite as:

@inproceedings{wu2023ganhead,
  title={GANHead: Towards Generative Animatable Neural Head Avatars},
  author={Wu, Sijing and Yan, Yichao and Li, Yunhao and Cheng, Yuhao and Zhu, Wenhan and Gao, Ke and Li, Xiaobo and Zhai, Guangtao},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={437--447},
  year={2023}
}