Awesome
Lightweight Deformable 3D Gaussian
Project page | Paper | Vanilla Code
This repository serves as an enhancement project for the paper 'Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction'. By implementing a densification trick, we can achieve a halved number of Gaussians and higher FPS without compromising rendering quality.
Dataset
In our paper, we use:
- synthetic dataset from D-NeRF.
- real-world dataset from NeRF-DS and Hyper-NeRF.
- The dataset in the supplementary materials comes from DeVRF.
We organize the datasets as follows:
├── data
│ | D-NeRF
│ ├── hook
│ ├── standup
│ ├── ...
│ | NeRF-DS
│ ├── as
│ ├── basin
│ ├── ...
│ | HyperNeRF
│ ├── interp
│ ├── misc
│ ├── vrig
I have identified an inconsistency in the D-NeRF's Lego dataset. Specifically, the scenes corresponding to the training set differ from those in the test set. This discrepancy can be verified by observing the angle of the flipped Lego shovel. To meaningfully evaluate the performance of our method on this dataset, I recommend using the validation set of the Lego dataset as the test set. See more in D-NeRF dataset used in Deformable-GS
Run
Environment
git clone https://github.com/ingra14m/Lightweight-Deformable-GS --recursive
cd Lightweight-Deformable-GS
conda create -n deformable_gaussian_env python=3.8
conda activate deformable_gaussian_env
# install pytorch
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
# install dependencies
pip install -r requirements.txt
Train
D-NeRF:
python train.py -s path/to/your/d-nerf/dataset -m output/exp-name --eval --is_blender
NeRF-DS/HyperNeRF:
python train.py -s path/to/your/real-world/dataset -m output/exp-name --eval --iterations 20000
Render & Evaluation
python render.py -m output/exp-name --mode render
python metrics.py -m output/exp-name
We provide several modes for rendering:
render
: render all the test imagestime
: time interpolation tasks for D-NeRF datasetall
: time and view synthesis tasks for D-NeRF datasetview
: view synthesis tasks for D-NeRF datasetoriginal
: time and view synthesis tasks for real-world dataset
Results
FPS is tested on 3090.
D-NeRF
$\tau_g=0.0007$
PSNR | SSIM | LPIPS(VGG) | FPS | Mem(MB) | Num. | |
---|---|---|---|---|---|---|
bouncing | 41.79 | 0.9959 | 0.0082 | 111 | 15.95 | 67417 |
hell | 41.36 | 0.9869 | 0.0253 | 288 | 4.19 | 17709 |
hook | 36.82 | 0.9852 | 0.0168 | 128 | 14.50 | 61321 |
jump | 37.56 | 0.9895 | 0.0135 | 223 | 7.36 | 31118 |
mutant | 42.10 | 0.9943 | 0.0066 | 109 | 17.04 | 72028 |
standup | 43.85 | 0.9941 | 0.0085 | 197 | 6.84 | 28895 |
trex | 37.58 | 0.9927 | 0.0106 | 74 | 22.90 | 96816 |
average | 40.15 | 0.9912 | 0.0128 | 162 | 12.68 | 53614 |
paper | 40.43 | 0.9918 | 0.0116 | 70 | 26.71 | 131428 |
$\tau_g=0.0006$
PSNR | SSIM | LPIPS(VGG) | FPS | Mem(MB) | Num. | |
---|---|---|---|---|---|---|
bouncing | 41.16 | 0.9955 | 0.0088 | 100 | 18.46 | 78033 |
hell | 41.39 | 0.9866 | 0.0253 | 241 | 5.27 | 22288 |
hook | 36.93 | 0.9853 | 0.0165 | 105 | 17.94 | 75831 |
jump | 37.76 | 0.9896 | 0.0135 | 147 | 9.33 | 39457 |
mutant | 42.27 | 0.9946 | 0.0060 | 93 | 21.55 | 91125 |
standup | 44.17 | 0.9944 | 0.0079 | 163 | 8.74 | 36939 |
trex | 38.03 | 0.9931 | 0.0099 | 59 | 27.56 | 116521 |
average | 40.25 | 0.9913 | 0.0125 | 135 | 15.55 | 65742 |
$\tau_g=0.0005$
PSNR | SSIM | LPIPS(VGG) | FPS | Mem(MB) | Num. | |
---|---|---|---|---|---|---|
bouncing | 40.85 | 0.9952 | 0.0091 | 80 | 23.90 | 101031 |
hell | 41.39 | 0.9866 | 0.0247 | 204 | 6.65 | 27704 |
hook | 37.17 | 0.9859 | 0.0158 | 68 | 23.15 | 97876 |
jump | 37.68 | 0.9894 | 0.0135 | 128 | 11.87 | 49986 |
mutant | 43.32 | 0.9948 | 0.0055 | 64 | 27.5 | 116276 |
standup | 44.18 | 0.9946 | 0.0076 | 129 | 11.15 | 47141 |
trex | 37.76 | 0.9929 | 0.0099 | 47 | 34.7 | 146707 |
average | 40.19 | 0.9913 | 0.0123 | 103 | 19.84 | 83817 |
NeRF-DS
PSNR | SSIM | LPIPS(VGG) | FPS | Mem(MB) | Num. | |
---|---|---|---|---|---|---|
as | 26.42 | 0.8865 | 0.1804 | 86 | 24.34 | 102918 |
basin | 19.72 | 0.7948 | 0.1881 | 63 | 33.81 | 142943 |
bell | 25.49 | 0.8466 | 0.1580 | 53 | 43.29 | 183499 |
cup | 24.78 | 0.8895 | 0.1540 | 58 | 27.97 | 118261 |
plate | 20.54 | 0.8143 | 0.2184 | 71 | 29.26 | 123712 |
press | 25.67 | 0.8637 | 0.1931 | 76 | 27.63 | 116800 |
sieve | 25.37 | 0.8704 | 0.1489 | 71 | 28.45 | 120280 |
average | 24.00 | 0.8523 | 0.1773 | 68 | 30.68 | 129773 |
paper | 24.11 | 0.8524 | 0.1769 | 35 | 56.78 | 221428 |
BibTex
@article{yang2023deformable3dgs,
title={Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction},
author={Yang, Ziyi and Gao, Xinyu and Zhou, Wen and Jiao, Shaohui and Zhang, Yuqing and Jin, Xiaogang},
journal={arXiv preprint arXiv:2309.13101},
year={2023}
}
And thanks to the authors of 3D Gaussians for their excellent code, please consider also cite this repository:
@Article{kerbl3Dgaussians,
author = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
title = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
journal = {ACM Transactions on Graphics},
number = {4},
volume = {42},
month = {July},
year = {2023},
url = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}