Awesome
#DeepI2P: Image-to-Point Cloud Registration via Deep Classification
Summary
Video
PyTorch implementation for our CVPR 2021 paper DeepI2P. DeepI2P solves the problem of cross modality registration, i.e, solve the relative rotation R
and translation t
between the camera and the lidar.
DeepI2P: Image-to-Point Cloud Registration via Deep Classification<br> Jiaxin Li <sup>1</sup>, Gim Hee Lee <sup>2</sup> <br> <sup>1</sup>ByteDance, <sup>2</sup>National University of Singapore
Method
The intuition is to perform the Inverse Camera Projection
, as shown in the images below.
Repo Structure
data
: Generate and process datasetsevaluation
: Registration codes, include Inverse Camera Projection, ICP, PnPfrustum_reg
: C++ codes of the Inverse Camera Projection, using Gauss-Newton Optimization. Installation method is shown below. It requires the Ceres Solver.
python evaluation/frustum_reg/setup.py install
icp
: codes for ICP (Iterative Closest Point)registration_lsq.py
: Python code for Inverse Camera Projection, which utilizes the per-point coarse classification prediction, and thefrustum_reg
solver.registration_pnp.py
: Python code for PnP solver utilizing the per-point fine classification prediction.
kitti
: Training codes for KITTInuscenes
: Training codes for nuscenesoxford
: Training codes for Oxford Robotcar datasetmodels
: Networks and layers- 'index_max_ext': This is a custom operation from SO-Net, which is the backbone of our network. Installation:
python models/index_max_ext/setup.py install
networks_img.py
: Network to process images. It is a resnet-like structure.networks_pc.py
: Network to process point clouds, it is from SO-Netnetwork_united.py
: Network to fuse information between point clouds and images.