Home

Awesome

<div align="center">

<b>GPS-Gaussian</b>: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis

Shunyuan Zheng<sup>†,1</sup>, Boyao Zhou<sup>2</sup>, Ruizhi Shao<sup>2</sup>, Boning Liu<sup>2</sup>, Shengping Zhang<sup>*,1,3</sup>, Liqiang Nie<sup>1</sup>, Yebin Liu<sup>2</sup>

<p><sup>1</sup>Harbin Institute of Technology &nbsp;&nbsp;<sup>2</sup>Tsinghua Univserity &nbsp;&nbsp;<sup>3</sup>Peng Cheng Laboratory <br><sup>*</sup>Corresponding author &nbsp;&nbsp;<sup>&dagger;</sup>Work done during an internship at Tsinghua Univserity<p>

Projectpage · Video · Paper · Supp.

</div>

Introduction

We propose GPS-Gaussian, a generalizable pixel-wise 3D Gaussian representation for synthesizing novel views of any unseen characters instantly without any fine-tuning or optimization.

https://github.com/ShunyuanZheng/GPS-Gaussian/assets/33752042/54a253ad-012a-448f-8303-168d80d3f594

Installation

To deploy and run GPS-Gaussian, run the following scripts:

conda env create --file environment.yml
conda activate gps_gaussian

Then, compile the diff-gaussian-rasterization in 3DGS repository:

git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive
cd gaussian-splatting/
pip install -e submodules/diff-gaussian-rasterization
cd ..

(optinal) RAFT-Stereo provides a faster CUDA implementation of the correlation sampler to speed up the model without impacting performance:

git clone https://github.com/princeton-vl/RAFT-Stereo.git
cd RAFT-Stereo/sampler && python setup.py install && cd ../..

If compiled this CUDA implementation, set corr_implementation='reg_cuda' in config/stereo_human_config.py else corr_implementation='reg'.

Run on synthetic human dataset

Dataset Preparation

Training

Note: At the first training time, we do stereo rectify and determine the disparity offline, the processed data will be saved at render_data/rectified_local. This process takes several hours and can extremely speed up the following training scheme. If you want to skip this pre-processing, set use_processed_data=False in stage1.yaml and stage2.yaml.

python train_stage1.py
python train_stage2.py

Testing

python test_real_data.py \
--test_data_root 'PATH/TO/REAL_DATA' \
--ckpt_path 'PATH/TO/GPS-GS_stage2_final.pth' \
--src_view 0 1 \
--ratio=0.5
python test_view_interp.py \
--test_data_root 'PATH/TO/RENDER_DATA/val' \
--ckpt_path 'PATH/TO/GPS-GS_stage2_final.pth' \
--novel_view_nums 5

Citation

If you find this code useful for your research, please consider citing:

@inproceedings{zheng2024gpsgaussian,
  title={GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis},
  author={Zheng, Shunyuan and Zhou, Boyao and Shao, Ruizhi and Liu, Boning and Zhang, Shengping and Nie, Liqiang and Liu, Yebin},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}