Awesome

RAIN-GS: Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting

This is our official implementation of the paper "Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting"!

by Jaewoo Jung:umbrella:, Jisang Han:umbrella:, Honggyu An:umbrella:, Jiwon Kang:umbrella:, Seonghoon Park:umbrella:, Seungryong Kim†

:umbrella:: Equal Contribution †: Corresponding Author

News

We found that the current version of our code has minor bugs including the rendering code 😢. We will fix everything as soon as possible.

Introduction

We introduce a novel optimization strategy (RAIN-GS) for 3D Gaussian Splatting!

We show that our simple yet effective strategy consisting of sparse-large-variance (SLV) random initialization, progressive Gaussian low-pass filter control, and the Adaptive Bound-Expanding Split (ABE-Split) algorithm robustly guides 3D Gaussians to model the scene even when starting from random point cloud.

❗️Update (2024/05/29): We have updated our paper and codes which significantly improve our previous results! 😴 TL;DR for our update is as follows:

We added a modification to the original split algorithm of 3DGS which enables the Gaussians to model scenes further from the viewpoints! This new splitting algorithm is named Adaptive Bound-Expanding Split algorithm (ABE-Split algorithm).
Now with our three key components (SLV initialization, progressive Gaussians low-pass filtering, ABE-Split), we perform on-par or even better compared to 3DGS trainied with SfM initialized point cloud.
As RAIN-GS only requires the initial point cloud to be sparse (SLV initialization), we now additionally apply our strategy to SfM/Noisy SfM point cloud by choosing a sparse set of points from the point cloud.

For further details and visualization results, please check out our updated paper and our new project page.

Installation

We implement RAIN-GS above the official implementation of 3D Gaussian Splatting. For environmental setup, we kindly guide you to follow the original requirements of 3DGS.

Training

To train 3D Gaussians Splatting with our updated RAIN-GS novel strategy, all you need to do is:

python train.py -s {dataset_path} --exp_name {exp_name} --eval --ours_new

You can train from various initializations by adding --train_from ['random', 'reprojection', 'cluster', 'noisy_sfm'] (random is default) To train with Mip-NeRF360 dataset, you can add argument --images images_4 for outdoor scenes and --images images_2 for indoor scenes to modify the resolution of the input images.

<details> <summary>Toggle to find more details for training from various initializations.</summary>

Random Initialization (Default)

python train.py -s {dataset_path} --exp_name {exp_name} --eval --ours_new --train_from 'random'

SfM (Structure-from-Motion) Initialization In order to apply RAIN-GS to SfM Initialization, we need to start with a sparse set of points (SLV Initialization). To choose the sparse set of points, you can choose several options:
- Clustering : Apply clustering to the initial point cloud using the HDBSCAN algorithm.
```
python train.py -s {dataset_path} --exp_name {exp_name} --eval --ours_new --train_from 'cluster'
```
- Top 10% : Each of the points from SfM comes with a confidence value, which is the reprojection error. Select the top 10% most confident points from the point cloud.
```
python train.py -s {dataset_path} --exp_name {exp_name} --eval --ours_new --train_from 'reprojection'
```
Noisy SfM Initialization In real-world scenarios, the point cloud from SfM can contain noise. To simulate this scenario, we add a random noise sampled from a normal distribution to the SfM point cloud. If you run with this option, we apply the clustering algorithm to the Noisy SfM point cloud.

python train.py -s {dataset_path} --exp_name {exp_name} --eval --ours_new --train_from 'noisy_sfm'

</details>

To train 3D Gaussian Splatting with our original RAIN-GS, all you need to do is:

python train.py -s {dataset_path} --exp_name {exp_name} --eval --ours

For dense-small-variance (DSV) random initialization (used in the original 3D Gaussian Splatting), you can simply run with the following command:

python train.py -s {dataset_path} --exp_name {exp_name} --eval --paper_random

For SfM (Structure-from-Motion) initialization (used in the original 3D Gaussian Splatting), you can simply run with the following command:

python train.py -s {dataset_path} --exp_name {exp_name} --eval

For Noisy SfM initialization (used in the original 3D Gaussian Splatting), you can simply run with the following command:

python train.py -s {dataset_path} --exp_name {exp_name} --eval --train_from 'noisy_sfm'

Acknowledgement

We would like to acknowledge the contributions of 3D Gaussian Splatting for open-sourcing the official codes for 3DGS!

Citation

If you find our work helpful, please cite our work as:

@article{jung2024relaxing,
  title={Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting},
  author={Jung, Jaewoo and Han, Jisang and An, Honggyu and Kang, Jiwon and Park, Seonghoon and Kim, Seungryong},
  journal={arXiv preprint arXiv:2403.09413},
  year={2024}
}