Awesome

OMEGAS

<a href="https://arxiv.org/abs/2404.15891">arXiv</a>

<img src="./media/examples/bear.gif" alt="bear.gif" width="350"/> <br> <b>Our method extracts meshes from 2D Gaussian Splatting reconstructions and build hybrid representations <br>that enable easy composition and animation in Gaussian Splatting scenes by manipulating the mesh.</b>

Abstract

Recent advancements in 3D reconstruction technologies have paved the way for high-quality and real-time rendering of complex 3D scenes. Despite these achievements, a notable challenge persists: it is difficult to precisely reconstruct specific objects from large scenes. Current scene reconstruction techniques frequently result in the loss of object detail textures and are unable to reconstruct object portions that are occluded or unseen in views. To address this challenge, we delve into the meticulous 3D reconstruction of specific objects within large scenes and propose a framework termed OMEGAS: Object Mesh Extraction from Large Scenes Guided by Gaussian Segmentation. Specifically, we proposed a novel 3D target segmentation technique based on 2D Gaussian Splatting, which segments 3D consistent target masks in multi-view scene images and generates a preliminary target model. Moreover, to reconstruct the unseen portions of the target, we propose a novel target replenishment technique driven by large-scale generative diffusion priors. We demonstrate that our method can accurately reconstruct specific targets from large scenes, both quantitatively and qualitatively. Our experiments show that OMEGAS significantly outperforms existing reconstruction methods across various scenarios.

BibTeX

@misc{wang2024omegasobjectmeshextraction,
      title={OMEGAS: Object Mesh Extraction from Large Scenes Guided by Gaussian Segmentation}, 
      author={Lizhi Wang and Feng Zhou and Bo yu and Pu Cao and Jianqin Yin},
      year={2024},
      eprint={2404.15891},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2404.15891}, 
}

Overview

Installation

0. Requirements

The software requirements are the following:

• Conda (recommended for easy setup)
• C++ Compiler for PyTorch extensions
• CUDA toolkit 11.7 for PyTorch extensions
• C++ Compiler and CUDA SDK must be compatible

1. Clone the repository

Start by cloning this repository:

# HTTPS
git clone https://github.com/CrystalWlz/OMEGAS.git --recursive

or

# SSH
git clone git@github.com:CrystalWlz/OMEGAS.git --recursive

2. Install the required Python packages

To install the required Python packages and activate the environment, go inside the OMEGAS/ directory and run the following commands:

conda env create -f environment.yml
conda activate omg
pip install -r requirements.txt
conda install pytorch3d::pytorch3d

Then you can try to install the required packages manually by running the following commands:

conda create --name omg -y python=3.9
conda activate omg
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install pytorch3d -c pytorch3d==0.7.5
conda install -c plotly plotly
conda install -c conda-forge rich
conda install -c conda-forge plyfile==0.8.1
conda install -c conda-forge jupyterlab
conda install -c conda-forge nodejs
conda install -c conda-forge ipywidgets
pip install --upgrade PyMCubes
conda install pytorch3d::pytorch3d

3. Install the Gaussian Splatting rasterizer

Run the following commands inside the OMEGAS directory to install the additional Python submodules required for Gaussian Splatting:

cd gaussian_splatting/submodules/diff-surfel-rasterization/
pip install -e .
cd ../diff-surfel-rasterization/
pip install .
cd ../simple-knn/
pip install -e .
cd ../../../

Please refer to the <a href="https://github.com/graphdeco-inria/gaussian-splatting">3D Gaussian Splatting repository</a> for more details.

4. Install the DEVA

cd Tracking-Anything-with-DEVA
pip install -e .
bash scripts/download_models.sh     # Download the pretrained models
cd ..

git clone https://github.com/hkchengrex/Grounded-Segment-Anything.git
cd Grounded-Segment-Anything
export AM_I_DOCKER=False
export BUILD_WITH_CUDA=True
python -m pip install -e segment_anything
python -m pip install -e GroundingDINO

cd ..

Quick Start

conda activate omg

bash script/prepare_pseudo_label.sh bear 1

python run_gaussian_2D.py -s data/truck -m output/truck_2D --depth_ratio 1.0

python gs_extract.py -s data/truck -m output/truck_2D/ --config_file configs/gaussian_dataset/truck.json --iteration 30000 --prompt "truck"

python dream_gaussian_2D.py --config configs/dream/dream_truck.yaml

python run_gaussian_2D.py -s output/truck_2D/object_104/refined -m output/truck_2D/object_104/refined --no_obj_loss --depth_ratio 1.0

You can also manually select the ID of the target object and extract the Gaussian.

python extract_object.py -m output/truck_2D --config_file configs/gaussian_dataset/truck.json --iteration 30000 -s data/truck

# python render_obj.py -m output/truck_2d/object_104/ --config_file configs/gaussian_dataset/truck.json

python dream_gaussian_2D.py --config configs/dream/dream_truck.yaml

python run_gaussian_2D.py -s output/truck_2D/object_104/refined -m output/truck_2D/object_104/refined --no_obj_loss --depth_ratio 1.0 --lambda_dist 1000

python render_simple.py -m output/truck_2D/object_104/refined

python mesh_2D.py -s output/truck_2D/object_104/refined -m output/truck_2D/object_104/refined --iteration 30000 --depth_ratio 1.0 --voxel_size 0.004 --sdf_trunc 0.016 --depth_trunc 3.0

Tips for using SuGaR on your own data and obtain better reconstructions

Estimate camera poses with COLMAP

Please first install a recent version of COLMAP (ideally CUDA-powered) and make sure to put the images you want to use in a directory <location>/input. Then, run the script gaussian_splatting/convert.py from the original Gaussian splatting implementation to compute the camera poses from the images using COLMAP. Please refer to the original <a href="https://github.com/graphdeco-inria/gaussian-splatting">3D Gaussian Splatting repository</a> for more details.

python gaussian_splatting/convert.py -s <location>

Sometimes COLMAP fails to reconstruct all images into the same model and hence produces multiple sub-models. The smaller sub-models generally contain only a few images. However, by default, the script convert.py will apply Image Undistortion only on the first sub-model, which may contain only a few images.

If this is the case, a simple solution is to keep only the largest sub-model and discard the others. To do this, open the source directory containing your input images, then open the sub-directory <Source_directory>/distorted/sparse/. You should see several sub-directories named 0/, 1/, etc., each containing a sub-model. Remove all sub-directories except the one containing the largest files, and rename it to 0/. Then, run the script convert.py one more time but skip the matching process:

python gaussian_splatting/convert.py -s <location> --skip_matching

Note: If the sub-models have common registered images, they could be merged into a single model as post-processing step using COLMAP; However, merging sub-models requires to run another global bundle adjustment after the merge, which can be time consuming.

Evaluation

To evaluate the quality of the reconstructions, we provide a script metrics.py that computes the PSNR, SSIM and LPIPS metrics on test images. Start by optimizing SuGaR models for the desired scenes and a regularization method ("density" or "sdf"), then create a .json config file containing the paths to the scenes in the following format: {source_images_dir_path: vanilla_gaussian_splatting_checkpoint_path}.

Finally, run the script as follows:

python run_gaussian_2D.py -s data/teatime -m result/lerf_mask_2D/teatime --eval --train_split

python render_lerf_mask.py -m result/lerf_mask_2D/teatime --skip_train

python script/eval_lerf_mask.py teatime

Results are saved in a .json file in the output/metrics/ directory. Please refer to the script for more details on the command line arguments.