Awesome

[CVPR2024] SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting

Paper Arxiv | Paper CVF | Video Youtube | Project Page

Official Repository for CVPR 2024 paper SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting.

Lifted optimization

The embedding points of 3DGS on the triangle mesh are updated by the walking on triangle scheme. See the phongsurface module implemented in c++ and pybind11. <img src="assets/SplattingAvatar-triangle.jpg" width="800"/>

Getting Started

Create conda env with pytorch.

conda create -n splatting python=3.9
conda activate splatting

# pytorch 1.13.1+cu117 is tested
pip install torch==1.13.1 torchvision torchaudio functorch --extra-index-url https://download.pytorch.org/whl/cu117

# pytorch3d
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
pip install -e .

# install other dependencies
pip install tqdm omegaconf opencv-python libigl
pip install trimesh plyfile imageio chumpy lpips
pip install packaging pybind11
pip install numpy==1.23.1

Clone this repo recursively. Install Gaussian Splatting's submodules.

git clone --recursive https://github.com/initialneil/SplattingAvatar.git
cd SplattingAvatar

cd submodules/diff-gaussian-rasterization
pip install .

cd ../submodules/simple-knn
pip install .

cd ..

Install simple_phongsurf for walking on triangles.

cd model/simple_phongsurf
pip install -e .

Download FLAME model, choose FLAME 2020 and unzip it, copy 'generic_model.pkl' into ./model/imavatar/FLAME2020,
Download SMPL model (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding places:

mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl model/smplx_utils/smplx_models/smpl/SMPL_FEMALE.pkl
mv /path/to/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl model/smplx_utils/smplx_models/smpl/SMPL_MALE.pkl

Preparing dataset

We provide the preprocessed data of the 10 subjects used in the paper.

Our preprocessing followed IMavatar and replaced the Segmentation with RobustVideoMatting.
~~Pre-trained checkpoints are provided together with the data.~~
- Please find the data from https://github.com/Zielon/INSTA. We'll update the checkpoint link soon.

Training

python train_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir <path/to/subject>
# for example:
python train_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir /path-to/bala

# you may specify gpu id by adding CUDA_VISIBLE_DEVICES=x before calling python:
CUDA_VISIBLE_DEVICES=0 python train_splatting_avatar.py ...
# to disable network_gui, set ip to 'none'
CUDA_VISIBLE_DEVICES=0 python train_splatting_avatar.py ... --ip none

# use SIBR_remoteGaussian_app.exe from 3DGS to watch the training
SIBR_remoteGaussian_app.exe --path <path/to/output_of_any_standard_3dgs>
# <output_of_any_standard_3dgs> is generated by running the original 3dgs on any original dataset
# SIBR_remoteGaussian_app.exe somehow requires a standard 3dgs output to start

# it is recommended to change "FPS" to "Trackball" in the viewer
# you don't need to change the "path" everytime

Evaluation

python eval_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir <path/to/model_path>
# for example:
python eval_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir /path-to/bala/output-splatting/last_checkpoint

Full-body Avatar

We conducted experiments on PeopleSnapshot.

Please download the parameter files (the same with InstantAvatar) from: Baidu Disk or Google Drive.
Download 4 sequences from PeopleSnapshot (male/female-3/4-casual) and unzip images and masks to corresponding folders from above.
Use scripts/preprocess_PeopleSnapshot.py to preprocess the data.
Training:

# override with instant_avatar.yaml for PeopleSnapshot in InstantAvatar's format
python train_splatting_avatar.py --config "configs/splatting_avatar.yaml;configs/instant_avatar.yaml" --dat_dir <path/to/subject>
# for example:
python train_splatting_avatar.py --config "configs/splatting_avatar.yaml;configs/instant_avatar.yaml" --dat_dir /path-to/female-3-casual

# pretrained checkpoints provided in `output-splatting/last_checkpoint` can be evaluated by `eval_splatting_avatar.py`
# for example:
python eval_splatting_avatar.py --config "configs/splatting_avatar.yaml;configs/instant_avatar.yaml" --dat_dir /path-to/female-3-casual --pc_dir /path-to/female-3-casual/output-splatting/last_checkpoint/point_cloud/iteration_30000

# to animate to noval pose `aist_demo.npz`
python eval_animate.py --config "configs/splatting_avatar.yaml;configs/instant_avatar.yaml" --dat_dir /path-to/female-3-casual --pc_dir /path-to/female-3-casual/output-splatting/last_checkpoint/point_cloud/iteration_30000 --anim_fn /path-to/aist_demo.npz

GPU requirement

We conducted our experiments on a single NVIDIA RTX 3090 with 24GB. Training with less GPU memory can be achieved by setting a maximum number of Gaussians.

# in configs/splatting_avatar.yaml
model:
  max_n_gauss: 300000 # or less as needed

or set by command line

python train_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir <path/to/subject> model.max_n_gauss=300000

Citation

If you find our code or paper useful, please cite as:

@inproceedings{shao2024splattingavatar,
  title = {{SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting}},
  author = {Shao, Zhijing and Wang, Zhaolong and Li, Zhuang and Wang, Duotun and Lin, Xiangru and Zhang, Yu and Fan, Mingming and Wang, Zeyu},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2024}
}

Acknowledgement

We thank the following authors for their excellent works!

License

SplattingAvatar <br> The code is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License for Noncommercial use only. Any commercial use should get formal permission first.

Gaussian Splatting <br> Inria and the Max Planck Institut for Informatik (MPII) hold all the ownership rights on the Software named gaussian-splatting. The Software is in the process of being registered with the Agence pour la Protection des Programmes (APP).