Home

Awesome

Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction

Project page | Paper

Teaser image

This repository contains the official implementation associated with the paper "Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction".

News

Dataset

In our paper, we use:

We organize the datasets as follows:

├── data
│   | D-NeRF 
│     ├── hook
│     ├── standup 
│     ├── ...
│   | NeRF-DS
│     ├── as
│     ├── basin
│     ├── ...
│   | HyperNeRF
│     ├── interp
│     ├── misc
│     ├── vrig

I have identified an inconsistency in the D-NeRF's Lego dataset. Specifically, the scenes corresponding to the training set differ from those in the test set. This discrepancy can be verified by observing the angle of the flipped Lego shovel. To meaningfully evaluate the performance of our method on this dataset, I recommend using the validation set of the Lego dataset as the test set. See more in D-NeRF dataset used in Deformable-GS

Pipeline

Teaser image

Run

Environment

git clone https://github.com/ingra14m/Deformable-3D-Gaussians --recursive
cd Deformable-3D-Gaussians

conda create -n deformable_gaussian_env python=3.7
conda activate deformable_gaussian_env

# install pytorch
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

# install dependencies
pip install -r requirements.txt

Train

D-NeRF:

python train.py -s path/to/your/d-nerf/dataset -m output/exp-name --eval --is_blender

NeRF-DS/HyperNeRF:

python train.py -s path/to/your/real-world/dataset -m output/exp-name --eval --iterations 20000

6DoF Transformation:

We have also implemented the 6DoF transformation of 3D-GS, which may lead to an improvement in metrics but will reduce the speed of training and inference.

# D-NeRF
python train.py -s path/to/your/d-nerf/dataset -m output/exp-name --eval --is_blender --is_6dof

# NeRF-DS & HyperNeRF
python train.py -s path/to/your/real-world/dataset -m output/exp-name --eval --is_6dof --iterations 20000

You can also train with the GUI:

python train_gui.py -s path/to/your/dataset -m output/exp-name --eval --is_blender

Render & Evaluation

python render.py -m output/exp-name --mode render
python metrics.py -m output/exp-name

We provide several modes for rendering:

Results

D-NeRF Dataset

Quantitative Results

<img src="assets/results/D-NeRF/Quantitative.jpg" alt="Image1" style="zoom:50%;" />

Qualitative Results

<img src="assets/results/D-NeRF/bouncing.gif" alt="Image1" style="zoom:25%;" /> <img src="assets/results/D-NeRF/hell.gif" alt="Image1" style="zoom:25%;" /> <img src="assets/results/D-NeRF/hook.gif" alt="Image3" style="zoom:25%;" /> <img src="assets/results/D-NeRF/jump.gif" alt="Image4" style="zoom:25%;" />

<img src="assets/results/D-NeRF/lego.gif" alt="Image5" style="zoom:25%;" /> <img src="assets/results/D-NeRF/mutant.gif" alt="Image6" style="zoom:25%;" /> <img src="assets/results/D-NeRF/stand.gif" alt="Image7" style="zoom:25%;" /> <img src="assets/results/D-NeRF/trex.gif" alt="Image8" style="zoom:25%;" />

400x400 Resolution

PSNRSSIMLPIPS (VGG)FPSMemNum. (k)
bouncing41.460.99580.004611213.1655622
hell42.110.98850.01533753.7215733
hook37.770.98970.010312811.7449613
jump39.100.99300.00902176.8128808
mutant43.730.99690.002912411.4548423
standup45.380.99670.00322105.9425102
trex38.400.99590.00418518.678624
Average41.140.99380.007017910.2043132

NeRF-DS Dataset

<img src="assets/results/NeRF-DS/Quantitative.jpg" alt="Image1" style="zoom:50%;" />

See more visualization on our project page.

HyperNeRF Dataset

Since the camera pose in HyperNeRF is less precise compared to NeRF-DS, we use HyperNeRF as a reference for partial visualization and the display of Failure Cases, but do not include it in the calculation of quantitative metrics. The results of the HyperNeRF dataset can be viewed on the project page.

Real-Time Viewer

https://github.com/ingra14m/Deformable-3D-Gaussians/assets/63096187/ec26d0b9-c126-4e23-b773-dcedcf386f36

Acknowledgments

We sincerely thank the authors of 3D-GS, D-NeRF, HyperNeRF, NeRF-DS, and DeVRF, whose codes and datasets were used in our work. We thank Zihao Wang for the debugging in the early stage, preventing this work from sinking. We also thank the reviewers and AC for not being influenced by PR, and fairly evaluating our work. This work was mainly supported by ByteDance MMLab.

BibTex

@article{yang2023deformable3dgs,
    title={Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction},
    author={Yang, Ziyi and Gao, Xinyu and Zhou, Wen and Jiao, Shaohui and Zhang, Yuqing and Jin, Xiaogang},
    journal={arXiv preprint arXiv:2309.13101},
    year={2023}
}

And thanks to the authors of 3D Gaussians for their excellent code, please consider also cite this repository:

@Article{kerbl3Dgaussians,
      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
      journal      = {ACM Transactions on Graphics},
      number       = {4},
      volume       = {42},
      month        = {July},
      year         = {2023},
      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}