Awesome

E-D3DGS : Embedding-Based Deformable 3D Gaussian Splatting (ECCV 2024)

Jeongmin Bae^1*, Seoha Kim^1*, Youngsik Yun¹, </br> Hahyun Lee², Gun Bang², Youngjung Uh^1†

¹Yonsei University   ²Electronics and Telecommunications Research Institute (ETRI) <br>^* Equal Contributions   ^† Corresponding Author

Official repository for <a href="https://arxiv.org/abs/2404.03613">"Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting"</a><be>. <br> Our approach employs per-Gaussian latent embeddings to predict deformation for each Gaussian and achieves a clearer representation of dynamic motion. <br> We uploaded the checkpoints, configs, and rendered videos for paper results here.

Environmental Setup

Please follow the 3DGS to install the relative packages.

git clone https://github.com/JeongminB/E-D3DGS.git
cd E-D3DGS
git submodule update --init --recursive

conda create -n ed3dgs python=3.7 
conda activate ed3dgs

# If submodules fail to be downloaded, refer to the repository of 3DGS  
pip install -r requirements.txt
pip install -e submodules/diff-gaussian-rasterization/
pip install -e submodules/simple-knn/ 

We use pytorch=1.13.1+cu116 in our environment.

Data Preparation

Downloading Datasets:
Please download datasets from their official websites : HyperNerf, Neural 3D Video and Technicolor <br><br>

• Please remove 'cam13.mp4' and corresponding pose from <i>coffee_martini</i> scene in the Neural 3D Video dataset. <br>
• We split the entire <i>flame_salmon_1_split</i> scene into four 300-frame scenes.

Extracting point clouds from COLMAP:

# setup COLMAP 
bash script/colmap_setup.sh
conda activate colmapenv 

# automatically extract the frames and reorginize them
python script/pre_n3v.py --videopath <dataset>/<scene>
python script/pre_technicolor.py --videopath <dataset>/<scene>
python script/pre_hypernerf.py --videopath <dataset>/<scene>

# downsample dense point clouds
python script/downsample_point.py \
<location>/<scene>/colmap/dense/workspace/fused.ply <location>/<scene>/points3D_downsample.ply

After running COLMAP, Neural 3D Video and Technicolor datasets are orginized as follows:

├── data
│   | n3v
│     ├── cook_spinach
│       ├── colmap
│       ├── images
│           ├── cam01
│               ├── 0000.png
│               ├── 0001.png
│               ├── ...
│           ├── cam02
│               ├── 0000.png
│               ├── 0001.png
│               ├── ...
│     ├── cut_roasted_beef
|     ├── ...

Training

To resize the training image, modify -r 2 in the command line.

# Train
python train.py -s $GT_PATH/$SCENE --configs arguments/$DATASET/$CONFIG.py --model_path $OUTPUT_PATH --expname $DATASET/$SCENE -r 2

Rendering

# Render test view only
python render.py --model_path $OUTPUT_PATH --configs arguments/$DATASET/$CONFIG.py --skip_train --skip_video

# Render train view, test view, and spiral path
python render.py --model_path $OUTPUT_PATH --configs arguments/$DATASET/$CONFIG.py

Evaluation

Note: In our paper, we calculate FPS by measuring rendering time only (except for save_image, etc.).

# Evaluate
python metrics.py --model_path $SAVE_PATH/$DATASET/$CONFIG

Note

• We provide scripts that collectively perform training, rendering, and evaluation. See the train_<dataset_name>.sh.
• You will need to configure the dataset path according to your system.
• In the config file, make sure that the total_num_frames and maxtime are equal to the total number of training frames.

Acknowledgements

This code is based on 3DGS, 4DGaussians and STG. In particular, we used 4DGaussians as a starting point for our study. We would like to thank the authors of these papers for their hard work. 😊

BibTex

@inproceedings{bae2024ed3dgs,
    title={Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting}, 
    author={Bae, Jeongmin and Kim, Seoha and Yun, Youngsik and Lee, Hahyun and Bang, Gun and Uh, Youngjung}, 
    booktitle = {European Conference on Computer Vision (ECCV)},
    year={2024}
}