Awesome

Abstract

Accurate mapping of large-scale environments is an essential building block of most outdoor autonomous systems. Challenges of traditional mapping methods include the balance between memory consumption and mapping accuracy. This paper addresses the problems of achieving large-scale 3D reconstructions with implicit representations using 3D LiDAR measurements. We learn and store implicit features through an octree-based hierarchical structure, which is sparse and extensible. The features can be turned into signed distance values through a shallow neural network. We leverage binary cross entropy loss to optimize the local features with the 3D measurements as supervision. Based on our implicit representation, we design an incremental mapping system with regularization to tackle the issue of catastrophic forgetting in continual learning. Our experiments show that our 3D reconstructions are more accurate, complete, and memory-efficient than current state-of-the-art 3D mapping methods.

Installation

1. Clone SHINE Mapping repository

git clone git@github.com:PRBonn/SHINE_mapping.git
cd SHINE_mapping

2. Set up conda environment

conda create --name shine python=3.7
conda activate shine

3. Install the key requirement kaolin

Kaolin depends on Pytorch (>= 1.8, <= 1.13.1), please install the corresponding Pytorch for your CUDA version (can be checked by nvcc --version). You can find the installation commands here.

For example, for CUDA version >=11.6, you can use:

pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116

Kaolin now supports installation with wheels. For example, to install kaolin 0.12.0 over torch 1.12.1 and cuda 11.6:

pip install kaolin==0.12.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.12.1_cu116.html

Follow the instructions to install kaolin. Firstly, clone kaolin to a local directory:

git clone --recursive https://github.com/NVIDIAGameWorks/kaolin
cd kaolin

Then install kaolin by:

python setup.py develop

Use python -c "import kaolin; print(kaolin.__version__)" to check if kaolin is successfully installed.

4. Install the other requirements

pip install open3d scikit-image wandb tqdm natsort pyquaternion

Containerized installation

Note that you CUDA version must be >=11.6.2 to be compatible with the container.

1. Install docker

https://docs.docker.com/engine/install/ubuntu/

2. Install nvidia container runtime

https://developer.nvidia.com/nvidia-container-runtime

3. Clone SHINE Mapping repository

git clone git@github.com:PRBonn/SHINE_mapping.git
cd SHINE_mapping

4. Build container

docker build --tag shine .

5. Run container with example

mkdir /tmp/shine_test_data
docker run --rm -v ${pwd}/:/repository -v /tmp/shine_test_data:/data -it --gpus all shine

Results will be produced in /tmp/shine_test_data/results.

6. Run container on your own data

docker run --rm -v .:/repository -v ${MY_DATA_DIR}:/data -it --gpus all shine bash

where ${MY_DATA_DIR} is the directory on the host with data in the format described in config/kitti/docker_kitti_batch.yaml. Once inside the container RUN as described below. Results will be found on host in ${MY_DATA_DIR}/results .

Prepare data

Generally speaking, you only need to provide:

1. pc_path : the folder containing the point cloud (.bin, .ply or .pcd format) for each frame.
2. pose_path : the pose file (.txt) containing the transformation matrix of each frame.
3. calib_path : the calib file (.txt) containing the static transformation between sensor and body frames (optional, would be identity matrix if set as '').

They all follow the KITTI odometry data format.

After preparing the data, you need to correctly set the data path (pc_path, pose_path and calib_path) in the config files under config folder. You may also set a path output_root to store the experiment results and logs.

Here, we provide the link to several publicly available datasets for testing SHINE Mapping:

MaiCity synthetic LiDAR dataset

Download the dataset from here or use the following script to download (3.4GB):

sh ./scripts/download_maicity.sh

KITTI real-world LiDAR dataset

Download the full dataset from here.

If you want to use an example part of the dataset (seq 00) for the test, you can use the following script to download (117 MB):

sh ./scripts/download_kitti_example.sh

Newer College real-world LiDAR dataset

Download the full dataset from here.

If you want to use an example part of the dataset (Quad) for the test, you can use the following script to download (634 MB):

sh ./scripts/download_ncd_example.sh

RGB-D datasets

SHINE Mapping also supports the mapping on RGB-D datasets. You may first try the synthetic dataset from NeuralRGB-D. You can download the full dataset from here or use the following script to download (7.25 GB).

sh ./scripts/download_neural_rgbd_data.sh

After downloading the data, you need to convert the dataset to the KITTI format by using for each sequence:

sh ./scripts/convert_rgbd_to_kitti_format.sh

Mapping without ground truth pose

Our method is currently a mapping-with-known-pose system. If you do not have the ground truth pose file, you may use a LiDAR odometry system such as KISS-ICP to easily estimate the pose.

You can simply install KISS-ICP by:

pip install kiss-icp

And then run KISS-ICP with your data path pc_path

kiss_icp_pipeline <pc_path>

The estimated pose file can be found in ./results/latest/velodyne.txt. You can directly use it as your pose_path. In this case, you do not need a calib file, so just set calib_path: "" in the config file.

Generate colorized mesh

Check the repository Color-SHINE-MAPPING (thanks @ZorAttC for the contribution) for using SHINE Mapping to reconstruct colorized mesh using colorized point cloud.

Run

We take the MaiCity dataset as an example to show how SHINE Mapping works. You can simply replace maicity with your dataset name in the config file path, such as ./config/[dataset]/[dataset]_[xxx].yaml.

The results will be stored with your experiment name with the starting timestamp in the output_root directory as what you set in the config file. You can find the reconstructed mesh (*.ply format) and optimized model in mesh and model folder, respectively. If the save_map option is turned on, then you can find the grid sdf map in map folder.

For mapping based on offline batch processing, use:

python shine_batch.py ./config/maicity/maicity_batch.yaml

For incremental mapping with replay strategy (within a local bounding box), use:

python shine_incre.py ./config/maicity/maicity_incre_replay.yaml

An interactive visualizer would pop up if you set o3d_vis_on: True (by default) in the config file. You can press space to pause and resume.

For the sake of efficiency, we sacrifice a bit mapping quality to use a 50cm leaf voxel size for the feature octree here.

To only visualize the mesh in a local bounding box for faster operation, you can set mc_local: True and mc_with_octree: False in the config file.

For incremental mapping with a regularization strategy, use:

python shine_incre.py ./config/maicity/maicity_incre_reg.yaml

For the sake of efficiency, we sacrifice a bit of mapping quality to use a 50cm leaf voxel size for the feature octree.

KITTI

Newer College

Apollo

Wild Place Forests

IPB Office

Replica

ICL Living Room

The logs can be monitored via Weights & Bias online if you turn the wandb_vis_on option on. If it's your first time to use Weights & Bias, you would be requested to register and login to your wandb account.

Evaluation

To evaluate the reconstruction quality, you need to provide the (reference) ground truth point cloud and your reconstructed mesh. The ground truth point cloud can be found (or sampled from) the downloaded folder of MaiCity, Newer College and Neural RGBD datasets.

Please change the data path and evaluation set-up in ./eval/evaluator.py and then run:

python ./eval/evaluator.py

to get the reconstruction metrics such as Chamfer distance, completeness, F-score, etc.

As mentioned in the paper, we also compute a fairer accuracy metric using the ground truth point cloud masked by the intersection of the reconstructed meshes of all the compared methods. To generate such masked ground truth point clouds, you can configure the data path in ./eval/crop_intersection.py and then run it.

To reproduce the quantitative results on MaiCity and Newer College dataset in the paper, you can use the config files in ./config/config_icra2023/. The reconstructed meshes can also be downloaded from here. Note that these numbers are obtained using the batch mapping mode. You can achieve similar results using the incremental mapping mode with the replay strategy.

Tips

1. You can play with different loss functions for SHINE Mapping. With the ray_loss: False option, the loss would be calculated from the sdf at each sample point. In this case, you can then select from sdf_bce (the proposed method), sdf_l1 and sdf_l2 loss as the main_loss_type. With the ray_loss: True option, the loss would be calculated from each ray containing multiple point samples as a depth rendering procedure. In this case, you can select from dr and dr_neus as the main_loss_type. According to our experiments, using our proposed sdf_bce loss can achieve the best reconstruction efficiently. We can get a decent reconstruction of a scene with several hundred frames in just one minute. Additionally, you can use the ekional_loss_on option to turn on/off the Ekional loss and use weight_e as its weight.

2. The feature octree is built mainly according to leaf_vox_size, tree_level_world and tree_level_feat. leaf_vox_size represents the size of the leaf voxel size in meter. tree_level_world and tree_level_feat represent the total tree level and the tree levels with latent feature codes, respectively. tree_level_world should be large enough to guarantee all the map data lies inside the cube with the size leaf_vox_size**(tree_level_world+1).

3. SHINE Mapping supports both offline batch mapping and incremental sequential mapping. For incremental mapping, one can either load a fixed pre-trained decoder from the batching mapping on a similar dataset (set load_model: True) or train the decoder for freeze_after_frame frames on-the-fly and then freeze it afterwards (set load_model: False). The first option would lead to better mapping performance.

4. You can use the mc_vis_level parameter to have a trade-off between the scene completion and the exact measurement accuracy. This parameter indicates at which level of the octree the marching cubes reconstruction would be conducted. The larger the value of mc_vis_level (but not larger than tree_level_feat), the more scene completion ability you would gain (but also some artifacts such as a double wall may appear). And with the small value, SHINE mapping would only reconstruct the part with actual measurements without filling the holes. The safest way to avoid the holes on the ground is to set mc_mask_on: False to disable the masking for marching cubes. By turning on the mc_with_octree option, you can achieve a faster marching cubes reconstruction only in the region within the octree nodes.

5. The incremental mapping with regularization strategy (setting continual_learning_reg: True) can achieve incremental neural mapping without storing an ever-growing data pool which would be a burden for the memory. The coefficient lambda_forget needs to be fine-tuned under different feature octree and point sampling settings. The recommended value is from 1e5 to 1e8. A pre-trained decoder is also recommended to be loaded during incremental mapping with regularization for better performance.

6. We also provide an option to conduct incremental mapping with a replay strategy in a local sliding window. You can turn this on by setting window_replay_on: True with a valid window_radius_m indicating the size of the sliding window.

7. It's also possible to incorporate semantic information in our SHINE-Mapping framework. You may set semantic_on = True in the utils/config.py file to enable semantic mapping and also provide semantic supervision by setting the label_path in the config file. The labels should be in Semantic KITTI format. An example semantic reconstruction results using Semantic KITTI can be downloaded from here.

Citation

If you use SHINE Mapping for any academic work, please cite our original paper.

@inproceedings{zhong2023icra,
  title={SHINE-Mapping: Large-Scale 3D Mapping Using Sparse Hierarchical Implicit NEural Representations},
  author={Zhong, Xingguang and Pan, Yue and Behley, Jens and Stachniss, Cyrill},
  booktitle = {Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)},
  year={2023}
}

Contact

If you have any questions, please contact:

• Xingguang Zhong {zhong@igg.uni-bonn.de}
• Yue Pan {yue.pan@igg.uni-bonn.de}

Acknowledgment

This work has partially been funded by the European Union’s HORIZON programme under grant agreement No 101070405 (DigiForest) and grant agreement No 101017008 (Harmony).

Additionally, we thank greatly for the authors of the following opensource projects:

• NGLOD (octree based hierarchical feature structure built based on kaolin)
• VDBFusion (comparison baseline)
• Voxblox (comparison baseline)
• Puma (comparison baseline and the MaiCity dataset)
• KISS-ICP (simple yet effective pose estimation)