Awesome
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians
<div align="center"> <img src="media/demo.gif"> <br>project / arxiv / video / face tracker / bibtex
</div>Licenses
This work is made available under CC-BY-NC-SA-4.0 and is subject to the following statement:
Toyota Motor Europe NV/SA and its affiliated companies retain all intellectual property and proprietary rights in and to this software and related documentation. Any commercial use, reproduction, disclosure or distribution of this software and related documentation without an express license agreement from Toyota Motor Europe NV/SA is strictly prohibited.
This project uses Gaussian Splatting, which carries its original license. The GUI is inspired by INSTA. The mesh rendering operations are adapted from NVDiffRec and NVDiffRast.
Setup
1. Installation
2. Download
Usage
0. Demo
You can play with a trained GaussianAvatar without downloading the dataset:
python local_viewer.py --point_path media/306/point_cloud.ply
1. Training
SUBJECT=306
python train.py \
-s data/UNION10_${SUBJECT}_EMO1234EXP234589_v16_DS2-0.5x_lmkSTAR_teethV3_SMOOTH_offsetS_whiteBg_maskBelowLine \
-m output/UNION10EMOEXP_${SUBJECT}_eval_600k \
--eval --bind_to_mesh --white_background --port 60000
<details>
<summary><span style="font-weight: bold;">Command Line Arguments</span></summary>
-
--source_path
/-s
Path to the source directory containing a COLMAP or Synthetic NeRF data set.
-
--model_path
/-m
Path where the trained model should be stored (
output/<random>
by default). -
--eval
Add this flag to use a training/val/test split for evaluation. Otherwise, all images are used for training.
-
--bind_to_mesh
Add this flag to bind 3D Gaussians to a driving mesh, e.g., FLAME.
-
--resolution
/-r
Specifies resolution of the loaded images before training. If provided
1, 2, 4
or8
, uses original, 1/2, 1/4 or 1/8 resolution, respectively. For all other values, rescales the width to the given number while maintaining image aspect. If not set and input image width exceeds 1.6K pixels, inputs are automatically rescaled to this target. -
--white_background
/-w
Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset.
-
--sh_degree
Order of spherical harmonics to be used (no larger than 3).
3
by default. -
--iterations
Number of total iterations to train for,
30_000
by default. -
--port
Port to use for GUI server,
60000
by default.
[!NOTE] During training, a complete evaluation are conducted on both the validation set (novel-view synthesis) and test set (self-reenactment) every
--interval
iterations. You can check the metrics in the commandline or Tensorboard. The metrics are computed on all images, although we only save partial images in Tensorboard.
2. Interactive Viewers
Remote Viewer
During training, one can monitor the training progress with the remote viewer
python remote_viewer.py --port 60000
[!NOTE]
The remote viewer can slow down training a lot. You may want to close it or check "pause rendering" when not viewing.
The viewer could get frozen and disconnected the first time you enable "show mesh". You can try switching it on and off or simply wait for a few seconds.
Local Viewer
After training, one can load and render the optimized 3D Gaussians with the local viewer
SUBJECT=306
ITER=300000
python local_viewer.py \
--point_path output/UNION10EMOEXP_${SUBJECT}_eval_600k/point_cloud/iteration_${ITER}/point_cloud.ply
<details>
<summary><span style="font-weight: bold;">Command Line Arguments</span></summary>
-
--point_path
Path to the gaussian splatting file (ply)
-
--motion_path
Path to the motion file (npz). You only need this if you want to load a different motion sequence than the original one for training.
3. Offline Rendering
Cite
If you find our paper or code useful in your research, please cite with the following BibTeX entry:
@inproceedings{qian2024gaussianavatars,
title={Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians},
author={Qian, Shenhan and Kirschstein, Tobias and Schoneveld, Liam and Davoli, Davide and Giebenhain, Simon and Nie{\ss}ner, Matthias},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={20299--20309},
year={2024}
}