Home

Awesome

E-D3DGS : Embedding-Based Deformable 3D Gaussian Splatting (ECCV 2024)

arXiv project_page

Jeongmin Bae<sup>1*</sup>, Seoha Kim<sup>1*</sup>, Youngsik Yun<sup>1</sup>, </br> Hahyun Lee<sup>2 </sup>, Gun Bang<sup>2</sup>, Youngjung Uh<sup>1†</sup>

<sup>1</sup>Yonsei University   <sup>2</sup>Electronics and Telecommunications Research Institute (ETRI) <br><sup>*</sup> Equal Contributions   <sup></sup> Corresponding Author


Official repository for <a href="https://arxiv.org/abs/2404.03613">"Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting"</a><be>. <br> Our approach employs per-Gaussian latent embeddings to predict deformation for each Gaussian and achieves a clearer representation of dynamic motion.

Alt Text

Environmental Setup

Please follow the 3DGS to install the relative packages.

git clone https://github.com/JeongminB/E-D3DGS.git
cd E-D3DGS
git submodule update --init --recursive

conda create -n ed3dgs python=3.7 
conda activate ed3dgs

# If submodules fail to be downloaded, refer to the repository of 3DGS  
pip install -r requirements.txt
pip install -e submodules/diff-gaussian-rasterization/
pip install -e submodules/simple-knn/ 

We use pytorch=1.13.1+cu116 in our environment.

Data Preparation

Downloading Datasets:
Please download datasets from their official websites : HyperNerf, Neural 3D Video and Technicolor <br><br>

<br>

Extracting point clouds from COLMAP:

# setup COLMAP 
bash script/colmap_setup.sh
conda activate colmapenv 

# automatically extract the frames and reorginize them
python script/pre_n3v.py --videopath <dataset>/<scene>
python script/pre_technicolor.py --videopath <dataset>/<scene>
python script/pre_hypernerf.py --videopath <dataset>/<scene>

# downsample dense point clouds
python script/downsample_point.py \
<location>/<scene>/colmap/dense/workspace/fused.ply <location>/<scene>/points3D_downsample.ply

After running COLMAP, Neural 3D Video and Technicolor datasets are orginized as follows:

├── data
│   | n3v
│     ├── cook_spinach
│       ├── colmap
│       ├── images
│           ├── cam01
│               ├── 0000.png
│               ├── 0001.png
│               ├── ...
│           ├── cam02
│               ├── 0000.png
│               ├── 0001.png
│               ├── ...
│     ├── cut_roasted_beef
|     ├── ...

Training

To resize the training image, modify -r 2 in the command line.

# Train
python train.py -s $GT_PATH/$SCENE --configs arguments/$DATASET/$CONFIG.py --model_path $OUTPUT_PATH --expname $DATASET/$SCENE -r 2

Rendering

# Render test view only
python render.py --model_path $OUTPUT_PATH --configs arguments/$DATASET/$CONFIG.py --skip_train --skip_video

# Render train view, test view, and spiral path
python render.py --model_path $OUTPUT_PATH --configs arguments/$DATASET/$CONFIG.py

Evaluation

Note: In our paper, we calculate FPS by measuring rendering time only (except for save_image, etc.).

# Evaluate
python metrics.py --model_path $SAVE_PATH/$DATASET/$CONFIG

Note

Acknowledgements

This code is based on 3DGS, 4DGaussians and STG. In particular, we used 4DGaussians as a starting point for our study. We would like to thank the authors of these papers for their hard work. 😊

BibTex

@inproceedings{bae2024ed3dgs,
    title={Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting}, 
    author={Bae, Jeongmin and Kim, Seoha and Yun, Youngsik and Lee, Hahyun and Bang, Gun and Uh, Youngjung}, 
    booktitle = {European Conference on Computer Vision (ECCV)},
    year={2024}
}