Awesome
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians
Project Page | Paper
Kerui Ren*, Lihan Jiang*, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, Bo Dai βοΈ <br />
News
[2024.09.25] πWe propose Octree-AnyGS, a general anchor-based framework that supports explicit Gaussians (2D-GS, 3D-GS) and neural Gaussians (Scaffold-GS). Additionally, Octree-GS has been adapted to the aforementioned Gaussian primitives, enabling Level-of-Detail representation for large-scale scenes. This framework holds potential for application to other Gaussian-based methods, with relevant SIBR visualizations forthcoming.
Overview
<p align="center"> <img src="assets/pipeline.png" width=100% height=100% class="center"> </p>Abstract: The recently proposed 3D Gaussian Splatting (3D-GS) demonstrates superior rendering fidelity and efficiency compared to NeRF-based scene representations. However, it struggles in large-scale scenes due to the high number of Gaussian primitives, particularly in zoomed-out views, where all primitives are rendered regardless of their projected size. This often results in inefficient use of model capacity and difficulty capturing details at varying scales. To address this, we introduce Octree-GS, a Level-of-Detail (LOD) structured approach that dynamically selects appropriate levels from a set of multi-scale Gaussian primitives, ensuring consistent rendering performance. To adapt the design of LOD, we employ an innovative grow-and-prune strategy for densification and also propose a progressive training strategy to arrange Gaussians into appropriate LOD levels. Additionally, our LOD strategy generalizes to other Gaussian-based methods, such as 2D-GS and Scaffold-GS, reducing the number of primitives needed for rendering while maintaining scene reconstruction accuracy. Experiments on diverse datasets demonstrate that our method achieves real-time speeds, with even 10 Γ faster than state-of-the-art methods in large-scale scenes, without compromising visual quality.
<p align="center"> <img src="assets/teaser.png" width=100% height=100% class="center"> </p>Installation
We tested on a server configured with Ubuntu 20.04, cuda 11.8 and gcc 9.4.0. Other similar configurations should also work, but we have not verified each one individually.
- Clone this repo:
git clone https://github.com/city-super/Octree-AnyGS.git --recursive
cd Octree-GS
- Install dependencies
SET DISTUTILS_USE_SDK=1 # Windows only
conda env create --file environment.yml
conda activate octree_gs
Data
First, create a data/
folder inside the project path by
mkdir data
The data structure will be organised as follows:
data/
βββ dataset_name
β βββ scene1/
β β βββ images
β β β βββ IMG_0.jpg
β β β βββ IMG_1.jpg
β β β βββ ...
β β βββ sparse/
β β βββ0/
β βββ scene2/
β β βββ images
β β β βββ IMG_0.jpg
β β β βββ IMG_1.jpg
β β β βββ ...
β β βββ sparse/
β β βββ0/
...
Public Data
- The MipNeRF360 scenes are provided by the paper author here.
- The SfM data sets for Tanks&Temples and Deep Blending are hosted by 3D-Gaussian-Splatting here.
- The BungeeNeRF dataset is available in Google Drive/ηΎεΊ¦η½η[ζεη :4whv].
- The MatrixCity dataset can be downloaded from Hugging Face/Openxlab/ηΎεΊ¦η½η[ζεη :hqnn]. Point clouds used for training in our paper: pcd
Download and uncompress them into the
data/
folder.
Custom Data
For custom data, you should process the image sequences with Colmap to obtain the SfM points and camera poses. Then, place the results into data/
folder.
Training
For training a single scene with the base model, modify the path and configurations in config/<method>/base_model.yaml
accordingly and run it:
python train.py --config config/<method>/base_model.yaml
For training a single scene with the lod model, modify the path and configurations in config/<method>/lod_model.yaml
accordingly and run it:
python train.py --config config/<method>/lod_model.yaml
This command will store the configuration file and log (with running-time code) into outputs/dataset_name/scene_name/cur_time
automatically.
In addition, we use gsplat to unify the rendering process of different Gaussians. Considering the adaptation for 2D-GS, we choose gsplat version which supports 2DGS.
Evaluation
We keep the manual rendering function with a similar usage of the counterpart in 3D-GS, one can run it by
python render.py -m <path to trained model> # Generate renderings
python metrics.py -m <path to trained model> # Compute error metrics on renderings
Results
Mip-NeRF360
Method | PSNR β | SSIM β | LPIPS β | #GS(k)/Mem |
---|---|---|---|---|
Mip-NeRF360 | 27.69 | 0.792 | 0.237 | - |
2D-GS | 26.93 | 0.800 | 0.251 | 397/440.8M |
3D-GS | 27.54 | 0.815 | 0.216 | 937/786.7M |
Mip-Splatting | 27.61 | 0.816 | 0.215 | 1013/838.4M |
Scaffold-GS | 27.90 | 0.815 | 0.220 | 666/197.5M |
Anchor-2D-GS | 26.98 | 0.801 | 0.241 | 547/392.7M |
Anchor-3D-GS | 27.59 | 0.815 | 0.220 | 707/492.0M |
Our-2D-GS | 27.02 | 0.801 | 0.241 | 397/371.6M |
Our-3D-GS | 27.65 | 0.815 | 0.220 | 504/418.6M |
Our-Scaffold-GS | 28.05 | 0.819 | 0.214 | 657/139.6M |
Tanks and Temples
Method | PSNR β | SSIM β | LPIPS β | #GS(k)/Mem |
---|---|---|---|---|
Mip-NeRF360 | 23.14 | 0.841 | 0.183 | - |
2D-GS | 23.25 | 0.830 | 0.212 | 352/204.4M |
3D-GS | 23.91 | 0.852 | 0.172 | 765/430.1M |
Mip-Splatting | 23.96 | 0.856 | 0.171 | 832/500.4M |
Scaffold-GS | 24.48 | 0.864 | 0.156 | 626/167.5M |
Anchor-2D-GS | 23.52 | 0.835 | 0.199 | 465/279.0M |
Anchor-3D-GS | 24.02 | 0.847 | 0.184 | 572/349.2M |
Our-2D-GS | 23.62 | 0.842 | 0.187 | 330/191.2M |
Our-3D-GS | 24.17 | 0.858 | 0.161 | 424/383.9M |
Our-Scaffold-GS | 24.68 | 0.866 | 0.153 | 443/88.5M |
DeepBlending
Method | PSNR β | SSIM β | LPIPS β | #GS(k)/Mem |
---|---|---|---|---|
Mip-NeRF360 | 29.40 | 0.901 | 0.245 | - |
2D-GS | 29.32 | 0.899 | 0.257 | 196/335.3M |
3D-GS | 29.46 | 0.903 | 0.242 | 398/705.6M |
Mip-Splatting | 29.56 | 0.901 | 0.243 | 410/736.8M |
Scaffold-GS | 30.28 | 0.909 | 0.239 | 207/125.5M |
Anchor-2D-GS | 29.35 | 0.896 | 0.264 | 162/289.0M |
Anchor-3D-GS | 29.66 | 0.899 | 0.260 | 150/272.9M |
Our-2D-GS | 29.44 | 0.897 | 0.264 | 84/202.3M |
Our-3D-GS | 29.65 | 0.901 | 0.257 | 79/180.0M |
Our-Scaffold-GS | 30.49 | 0.912 | 0.241 | 112/71.7M |
Viewer
The SIBR viewers is coming soon.
Contact
- Kerui Ren: renkerui@pjlab.org.cn
- Lihan Jiang: mr.lhjiang@gmail.com
Citation
If you find our work helpful, please consider citing:
@article{ren2024octree,
title={Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians},
author={Ren, Kerui and Jiang, Lihan and Lu, Tao and Yu, Mulin and Xu, Linning and Ni, Zhangkai and Dai, Bo},
journal={arXiv preprint arXiv:2403.17898},
year={2024}
}
LICENSE
Please follow the LICENSE of 3D-GS.
Acknowledgement
We thank all authors from 2D-GS, 3D-GS and Scaffold-GS for presenting such an excellent work. We also thank all authors from gsplat for presenting a generic and efficient Gaussian splatting framework.