Home

Awesome

GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians

<div align="center"> <img src="media/demo.gif"> <br>

project / arxiv / video / face tracker / bibtex

</div>

Licenses

This work is made available under CC-BY-NC-SA-4.0 and is subject to the following statement:

Toyota Motor Europe NV/SA and its affiliated companies retain all intellectual property and proprietary rights in and to this software and related documentation. Any commercial use, reproduction, disclosure or distribution of this software and related documentation without an express license agreement from Toyota Motor Europe NV/SA is strictly prohibited.

This project uses Gaussian Splatting, which carries its original license. The GUI is inspired by INSTA. The mesh rendering operations are adapted from NVDiffRec and NVDiffRast.

Method

Setup

1. Installation

2. Download

Usage

0. Demo

You can play with a trained GaussianAvatar without downloading the dataset:

python local_viewer.py --point_path media/306/point_cloud.ply

1. Training

SUBJECT=306

python train.py \
-s data/UNION10_${SUBJECT}_EMO1234EXP234589_v16_DS2-0.5x_lmkSTAR_teethV3_SMOOTH_offsetS_whiteBg_maskBelowLine \
-m output/UNION10EMOEXP_${SUBJECT}_eval_600k \
--eval --bind_to_mesh --white_background --port 60000
<details> <summary><span style="font-weight: bold;">Command Line Arguments</span></summary> </details>

NOTE: During training, a complete evaluation are conducted on both the validation set (novel-view synthesis) and test set (self-reenactment) every --interval iterations. You can check the metrics in the commandline or Tensorboard. The metrics are computed on all images, although we only save partial images in Tensorboard.

2. Interactive Viewers

Remote Viewer

remote viewer

During training, one can monitor the training progress with the remote viewer

python remote_viewer.py --port 60000

NOTE:

Local Viewer

local viewer

After training, one can load and render the optimized 3D Gaussians with the local viewer

SUBJECT=306
ITER=300000

python local_viewer.py \
--point_path output/UNION10EMOEXP_${SUBJECT}_eval_600k/point_cloud/iteration_${ITER}/point_cloud.ply
<details> <summary><span style="font-weight: bold;">Command Line Arguments</span></summary> </details>

3. Offline Rendering

Cite

If you find our paper or code useful in your research, please cite with the following BibTeX entry:

@article{qian2023gaussianavatars,
  title={GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians},
  author={Qian, Shenhan and Kirschstein, Tobias and Schoneveld, Liam and Davoli, Davide and Giebenhain, Simon and Nie{\ss}ner, Matthias},
  journal={arXiv preprint arXiv:2312.02069},
  year={2023}
}