Awesome

RO-MAP

Paper | Video

Overview

RO-MAP consists of two parts, including an object SLAM based on ORB-SLAM2 and a Multi-Object NeRF system based on tiny-cuda-nn. The two are decoupled, and the multi-object NeRF system is compiled into a dynamic link library, which provides interfaces for offline and online operation. Its code is located at:

 ./dependencies/Multi-Object-NeRF/Core

Pipeline

License

This repo is GPLv3 Licensed (inherit ORB-SLAM2). The Multi-Object NeRF system based on tiny-cuda-nn (BSD 3-clause license). Our implementation refers to instant-ngp (Nvidia Source Code License-NC) and uses its marching cubes algorithm directly.

Prerequisites

Our prerequisites include ORB-SLAM2 and tiny-cuda-nn requirements. If you encounter compilation problems, please refer to their issues first.

Eigen3 (test version: 3.4.0)
OpenCV (test version: 3.4.16)
Pangolin (test version: 0.8)
CMake (test version: 3.25)
CUDA (test version: 11.8)

Test system: ubuntu (docker) 20.04, GPU: RTX 4090

Building

Begin by cloning this repository and all its submodules using the following command:

git clone --recursive https://github.com/XiaoHan-Git/RO-MAP.git
cd RO-MAP

Then, build the multi-object NeRF system:

cd dependencies/Multi-Object-NeRF
sh build.sh

Finally, return to the main path and build RO-MAP:

cd ../../
sh build.sh

Examples

Dataset

We provide the dataset used in our paper.

synthetic sequence room (including blender files, which come from the Cube Diorama Dataset).
two collected real-world sequences scene1, scene2.

Please note that due to experimental conditions, the real-world sequences cannot be used for quantitative analysis.

Offline Multi-Object NeRF

The offline version is only available to the synthetic sequence. It is recommended to verify the offline version first.

cd dependencies/Multi-Object-NeRF
# Specify which GPU to use (one or two are recommended)
export CUDA_VISIBLE_DEVICES=0
# Since the visualization is implemented using OpenGL, set the environment variable to make it run on the GPU.
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia ./build/OfflineNeRF ./Core/configs/base.json [path_to_sequence] [Use_GTdepth(0 or 1)]

Online RO-MAP

Execute the following command:

# Specify which GPU to use (one or two are recommended)
export CUDA_VISIBLE_DEVICES=0, 1
# Since the visualization is implemented using OpenGL, set the environment variable to make it run on the GPU.
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia ./examples/Monocular/mono_tum ./vocabulary/ORBvoc.bin ./dependencies/Multi-Object-NeRF/Core/configs/base.json [path_to_sequence]

If the GPU performance is relatively low, you can try to reduce the number of iterations:

path_to_sequence/config.yaml -> NeRF.TrainStepIterations : 500 ↓

Note: Due to the randomness of the multi-threading and outlier removal algorithm, sometimes the estimated 3D bounding box of objects may deviate significantly. You can run it several times.

Acknowledgments

We thank Jad Abou-Chakra for his support on the dataset. Many thanks to the following brilliant works!

SLAM: ORB-SLAM2, CubeSLAM, EAO-SLAM
NeRF: instant-ngp, ngp_pl

Citation

If you found this code/work to be useful in your own research, please considering citing the following:

@ARTICLE{RO-MAP,
  author={Han, Xiao and Liu, Houxuan and Ding, Yunchao and Yang, Lu},
  journal={IEEE Robotics and Automation Letters}, 
  title={RO-MAP: Real-Time Multi-Object Mapping With Neural Radiance Fields}, 
  year={2023},
  volume={8},
  number={9},
  pages={5950-5957},
  doi={10.1109/LRA.2023.3302176}
}