Awesome

Created by Martin Hahner at the Computer Vision Lab of ETH Zurich.

🌨 LiDAR Snowfall Simulation for Robust 3D Object Detection

by Martin Hahner, Christos Sakaridis, Mario Bijelic, Felix Heide, Fisher Yu, Dengxin Dai, and Luc van Gool

📣 Oral at CVPR 2022. Please visit our paper website for more details.

Overview

.
├── calib                     # contains the LiDAR sensor calibration file used in STF
│   └── ...
├── lib                       # contains external libraries as submodules
│   └── ...
├── splits                    # contains the splits we used for our experiments
│   └── ...
├── tools                     # contains our snowfall and wet ground simulation code
│   ├── snowfall
│   │   ├── geometry.py
│   │   ├── precompute.py
│   │   ├── sampling.py
│   │   └── simulation.py
│   └── wet_ground
│       ├── augmentation.py
│       ├── phy_equations.py
│       ├── planes.py
│       └── utils.py
├── .gitignore
├── .gitmodules
├── LICENSE
├── pointcloud_viewer.py      # to visualize LiDAR point clouds and apply various augmentations
├── README.md
└── teaser.gif

Datasets supported by pointcloud_viewer.py:

H3D
A2D2
KITTI
LyftL5
Pandaset
nuScenes
Argoverse
ApolloScape
SeeingThroughFog :arrow_left: works best
WaymoOpenDataset (via waymo_kitti_converter)

Note: The snowfall and wet ground simulation is only tested on the SeeingThroughFog (STF) dataset. To support other datasets as well, code changes are required.

License

This software is made available for non-commercial use under a Creative Commons License. A summary of the license can be found here.

Citation(s)

If you find this work useful, please consider citing our paper.

@inproceedings{HahnerCVPR22,
  author = {Hahner, Martin and Sakaridis, Christos and Bijelic, Mario and Heide, Felix and Yu, Fisher and Dai, Dengxin and Van Gool, Luc},
  title = {{LiDAR Snowfall Simulation for Robust 3D Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2022},
}

You may also want to check out our earlier work Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather.

@inproceedings{HahnerICCV21,
  author = {Hahner, Martin and Sakaridis, Christos and Dai, Dengxin and Van Gool, Luc},
  title = {{Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather}},
  booktitle = {IEEE International Conference on Computer Vision (ICCV)},
  year = {2021},
}

Getting Started

Setup

Install anaconda.
Execute the following commands.

# Create a new conda environment.
conda create --name snowy_lidar python=3.9 -y

# Activate the newly created conda environment.
conda activate snowy_lidar

# Install dependencies.
conda install matplotlib pandas plyfile pyaml pyopengl pyqt pyqtgraph scipy scikit-learn tqdm -c conda-forge -y
pip install PyMieScatt pyquaternion

# Clone this repository (including submodules!).
git clone git@github.com:SysCV/LiDAR_snow_sim.git --recursive
cd LiDAR_snow_sim

If you want to use our precomputed snowflake patterns, you can download them (2.3GB) as mentioned below.

wget https://www.trace.ethz.ch/publications/2022/lidar_snow_simulation/snowflakes.zip
unzip snowflakes.zip
rm snowflakes.zip

If you want to use DROR as well, you need to install PCL or download the point indices (215MB) as mentioned below.

wget https://www.trace.ethz.ch/publications/2022/lidar_snow_simulation/DROR.zip
unzip DROR.zip
rm DROR.zip

Enjoy pointcloud_viewer.py.

python pointcloud_viewer.py

If you also want to run inference on the STF dataset, a couple of extra steps are required. Note: For unknown reasons, this can roughly slow down the augmentation(s) by a factor of two.

# Download our checkpoints (265MB)
wget https://www.trace.ethz.ch/publications/2022/lidar_snow_simulation/experiments.zip
unzip experiments.zip
rm experiments.zip

# Install PyTorch.
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c conda-forge -c pytorch -y

# Install spconv
pip install spconv-cu113

# build pcdet
cd lib/OpenPCDet
python setup.py develop
cd ../..

Disclaimer

The code has been successfully tested on

Ubuntu 18.04.6 LTS + CUDA 11.3 + conda 4.13.0
Debian GNU/Linux 10 (buster) + conda 4.13.0
MacOS Big Sur 11.6.6 + conda 4.13.0

Contributions

Please feel free to suggest improvements to this repository. We are always open to merge useful pull request.

Acknowledgments

This work is supported by Toyota via the TRACE project.

The work also received funding by the AI-SEE project with national funding from

We also thank the Federal Ministry for Economic Affairs and Energy for support within VVM-Verification and Validation Methods for Automated Vehicles Level 4 and 5, a PEGASUS family project.

Felix Heide was supported by an NSF CAREER Award (2047359), a Sony Young Faculty Award, and a Project X Innovation Award.

We thank Emmanouil Sakaridis for verifying our derivation of occlusion angles in our snowfall simulation.