Home

Awesome

Lightweight Deformable 3D Gaussian

Project page | Paper | Vanilla Code

Teaser image

This repository serves as an enhancement project for the paper 'Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction'. By implementing a densification trick, we can achieve a halved number of Gaussians and higher FPS without compromising rendering quality.

Dataset

In our paper, we use:

We organize the datasets as follows:

├── data
│   | D-NeRF 
│     ├── hook
│     ├── standup 
│     ├── ...
│   | NeRF-DS
│     ├── as
│     ├── basin
│     ├── ...
│   | HyperNeRF
│     ├── interp
│     ├── misc
│     ├── vrig

I have identified an inconsistency in the D-NeRF's Lego dataset. Specifically, the scenes corresponding to the training set differ from those in the test set. This discrepancy can be verified by observing the angle of the flipped Lego shovel. To meaningfully evaluate the performance of our method on this dataset, I recommend using the validation set of the Lego dataset as the test set. See more in D-NeRF dataset used in Deformable-GS

Run

Environment

git clone https://github.com/ingra14m/Lightweight-Deformable-GS --recursive
cd Lightweight-Deformable-GS

conda create -n deformable_gaussian_env python=3.8
conda activate deformable_gaussian_env

# install pytorch
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

# install dependencies
pip install -r requirements.txt

Train

D-NeRF:

python train.py -s path/to/your/d-nerf/dataset -m output/exp-name --eval --is_blender

NeRF-DS/HyperNeRF:

python train.py -s path/to/your/real-world/dataset -m output/exp-name --eval --iterations 20000

Render & Evaluation

python render.py -m output/exp-name --mode render
python metrics.py -m output/exp-name

We provide several modes for rendering:

Results

FPS is tested on 3090.

D-NeRF

$\tau_g=0.0007$

PSNRSSIMLPIPS(VGG)FPSMem(MB)Num.
bouncing41.790.99590.008211115.9567417
hell41.360.98690.02532884.1917709
hook36.820.98520.016812814.5061321
jump37.560.98950.01352237.3631118
mutant42.100.99430.006610917.0472028
standup43.850.99410.00851976.8428895
trex37.580.99270.01067422.9096816
average40.150.99120.012816212.6853614
paper40.430.99180.01167026.71131428

$\tau_g=0.0006$

PSNRSSIMLPIPS(VGG)FPSMem(MB)Num.
bouncing41.160.99550.008810018.4678033
hell41.390.98660.02532415.2722288
hook36.930.98530.016510517.9475831
jump37.760.98960.01351479.3339457
mutant42.270.99460.00609321.5591125
standup44.170.99440.00791638.7436939
trex38.030.99310.00995927.56116521
average40.250.99130.012513515.5565742

$\tau_g=0.0005$

PSNRSSIMLPIPS(VGG)FPSMem(MB)Num.
bouncing40.850.99520.00918023.90101031
hell41.390.98660.02472046.6527704
hook37.170.98590.01586823.1597876
jump37.680.98940.013512811.8749986
mutant43.320.99480.00556427.5116276
standup44.180.99460.007612911.1547141
trex37.760.99290.00994734.7146707
average40.190.99130.012310319.8483817

NeRF-DS

PSNRSSIMLPIPS(VGG)FPSMem(MB)Num.
as26.420.88650.18048624.34102918
basin19.720.79480.18816333.81142943
bell25.490.84660.15805343.29183499
cup24.780.88950.15405827.97118261
plate20.540.81430.21847129.26123712
press25.670.86370.19317627.63116800
sieve25.370.87040.14897128.45120280
average24.000.85230.17736830.68129773
paper24.110.85240.17693556.78221428

BibTex

@article{yang2023deformable3dgs,
    title={Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction},
    author={Yang, Ziyi and Gao, Xinyu and Zhou, Wen and Jiao, Shaohui and Zhang, Yuqing and Jin, Xiaogang},
    journal={arXiv preprint arXiv:2309.13101},
    year={2023}
}

And thanks to the authors of 3D Gaussians for their excellent code, please consider also cite this repository:

@Article{kerbl3Dgaussians,
      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
      journal      = {ACM Transactions on Graphics},
      number       = {4},
      volume       = {42},
      month        = {July},
      year         = {2023},
      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}