Home

Awesome

<!-- # Towards building an object-centric autonomous driving system based on Gaussians -->

logo

We build object-centric autonomous driving systems based on Gaussians!

This repository contains the implementation of the following methods.

GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction<br> Yuanhui Huang, Wenzhao Zheng<sup>*</sup>, Yunpeng Zhang, Jie Zhou, Jiwen Lu<sup>+</sup><br> European Conference on Computer Vision (ECCV), 2024<br> Paper | Project Page

GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction<br> Yuanhui Huang, Amonnut Thammatadatrakoon, Wenzhao Zheng<sup>*</sup>, Yunpeng Zhang, Dalong Du, Jiwen Lu<br> Paper

<sup>*</sup> Project leader. <sup>+</sup> Corresponding author.

News.

GaussianFormer

demo

legend

GaussianFormer proposes the 3D semantic Gaussians as a more efficient object-centric representation for driving scenes compared with 3D occupancy.

teaser

comparisons

Considering the universal approximating ability of Gaussian mixture, we propose an object-centric 3D semantic Gaussian representation to describe the fine-grained structure of 3D scenes without the use of dense grids. We propose a GaussianFormer model consisting of sparse convolution and cross-attention to efficiently transform 2D images into 3D Gaussian representations. To generate dense 3D occupancy, we design a Gaussian-to-voxel splatting module that can be efficiently implemented with CUDA. With comparable performance, our GaussianFormer reduces memory consumption of existing 3D occupancy prediction methods by 75.2% - 82.2%.

overview

GaussianFormer-2

demo

legend

<!-- ![comparisons](./assets/comparisons_gs2.png) --> <p align = "center"> <img src="./assets/comparisons_gs2.png" width="60%" /> </p>

We propose a probabilistic Gaussian superposition model which interprets each Gaussian as a probability distribution of its neighborhood being occupied and conforms to probabilistic multiplication to derive the overall geometry. Furthermore, we adopt the exact Gaussian mixture model for semantics calculation to avoid unnecessary overlapping of Gaussians. To effectively initialize Gaussians in non-empty region, we design a distribution-based initialization module which learns the pixel-aligned occupancy distribution instead of the depth of surfaces. We conduct extensive experiments on nuScenes and KITTI-360 datasets and our GaussianFormer-2 achieves state-of-the-art performance with high efficiency.

overview

Getting Started

Installation

Follow instructions HERE to prepare the environment.

<!-- The environment is almost the same as [SelfOcc](https://github.com/huang-yh/SelfOcc) except for two additional CUDA operations. ``` 1. Follow instructions in SelfOcc to prepare the environment. Not that we do not need packages related to NeRF, so feel safe to skip them. 2. cd model/encoder/gaussian_encoder/ops && pip install -e . # deformable cross attention with image features 3. cd model/head/localagg && pip install -e . # Gaussian-to-Voxel splatting ``` -->

Data Preparation

  1. Download nuScenes V1.0 full dataset data HERE.

  2. Download the occupancy annotations from SurroundOcc HERE and unzip it.

  3. Download pkl files HERE.

Folder structure

GaussianFormer
├── ...
├── data/
│   ├── nuscenes/
│   │   ├── maps/
│   │   ├── samples/
│   │   ├── sweeps/
│   │   ├── v1.0-test/
|   |   ├── v1.0-trainval/
│   ├── nuscenes_cam/
│   │   ├── nuscenes_infos_train_sweeps_occ.pkl
│   │   ├── nuscenes_infos_val_sweeps_occ.pkl
│   │   ├── nuscenes_infos_val_sweeps_lid.pkl
│   ├── surroundocc/
│   │   ├── samples/
│   │   |   ├── xxxxxxxx.pcd.bin.npy
│   │   |   ├── ...

Inference

We provide the following checkpoints trained on the SurroundOcc dataset:

NameType#GaussiansmIoUConfigWeight
BaselineGaussianFormer14400019.10configweight
NonEmptyGaussianFormer2560019.31configweight
Prob-64GaussianFormer-2640020.04configweight
Prob-128GaussianFormer-21280020.08configweight
Prob-256GaussianFormer-22560020.33configweight
python eval.py --py-config config/xxxx.py --work-dir out/xxxx/ --resume-from out/xxxx/state_dict.pth

Train

Download the pretrained weights for the image backbone HERE and put it inside ckpts.

python train.py --py-config config/xxxx.py --work-dir out/xxxx

Stay tuned for more exciting work and models!🤗

Visualize

Install packages for visualization according to the documentation. Here is an example command where you can change --num-samples and --vis-index.

CUDA_VISIBLE_DEVICES=0 python visualize.py --py-config config/nuscenes_gs25600_solid.py --work-dir out/nuscenes_gs25600_solid --resume-from out/nuscenes_gs25600_solid/state_dict.pth --vis-occ --vis-gaussian --num-samples 3 --model-type base

Related Projects

Our work is inspired by these excellent open-sourced repos: TPVFormer PointOcc SelfOcc SurroundOcc OccFormer BEVFormer

Our code is originally based on Sparse4D and migrated to the general framework of SelfOcc.

Citation

If you find this project helpful, please consider citing the following paper:

@article{huang2024gaussian,
    title={GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction},
    author={Huang, Yuanhui and Zheng, Wenzhao and Zhang, Yunpeng and Zhou, Jie and Lu, Jiwen},
    journal={arXiv preprint arXiv:2405.17429},
    year={2024}
}
@article{huang2024probabilisticgaussiansuperpositionefficient,
      title={GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction}, 
      author={Yuanhui Huang and Amonnut Thammatadatrakoon and Wenzhao Zheng and Yunpeng Zhang and Dalong Du and Jiwen Lu},
      journal={arXiv preprint arXiv:2412.04384},
      year={2024}
}