Awesome

Pose Correction for Highly Accurate Visual Localization in Large-scale Indoor Spaces (ICCV 2021)

Janghun Hyeon^1*, JooHyung Kim^1*, Nakju Doh^{1, 2}

¹ Korea University, ² TeeLabs

* Equally contributed to this work.

​ <img src="images/paper.png" width="18%"/> <img src="images/poster.png" width=37.0%> <img src="images/slides.png" width=39%>

Abstract

Indoor visual localization is significant for various applications such as autonomous robots, augmented reality, and mixed reality. Recent advances in visual localization have demonstrated their feasibility in large-scale indoor spaces through coarse-to-fine methods that typically employ three steps: image retrieval, pose estimation, and pose selection. However, further research is needed to improve the accuracy of large-scale indoor visual localization. We demonstrate that the limitations in the previous methods can be attributed to the sparsity of image positions in the database, which causes view-differences between a query and a retrieved image from the database. In this paper, to address this problem, we propose a novel module, named pose correction, that enables re-estimation of the pose with local feature matching in a similar view by reorganizing the local features. This module enhances the accuracy of the initially estimated pose and assigns more reliable ranks. Furthermore, the proposed method achieves a new state-of-the-art performance with an accuracy of more than 90 %within 1.0 m in the challenging indoor benchmark dataset InLoc for the first time.

Dependencies

• Python 3

• Pytorch >= 1.1

• Tensorflow >= 1.13

• openCV >= 3.4

• Matplotlib >= 3.1

• Numpy >= 1.18

• scipy >= 1.4.1

• open3d >= 0.7.0.0

• vlfeat >= 0.9.20

• vlfeat-ctypes >= 0.1.5

Prerequisite: Model Parameters

PCLoc is based on coarse-to-fine localization, which uses NetVLAD, SuperPoint, and SuperGlue. Thus, the model parameter should be downloaded from the original code.

NetVLAD: Download Model

Download parameter from the above URL, and unzip the file at:

./thirdparty/netvlad_tf/checkpoints/vd16_pitts30k_conv5_3_vlad_preL2_intra_white.data-00000-of-00001
./thirdparty/netvlad_tf/checkpoints/vd16_pitts30k_conv5_3_vlad_preL2_intra_white.index
./thirdparty/netvlad_tf/checkpoints/vd16_pitts30k_conv5_3_vlad_preL2_intra_white.meta

SuperPoint and SuperGlue: Download Model

./thirdparty/SuperGluePretrainedNetwork/models/weights/superglue_outdoor.pth
./thirdparty/SuperGluePretrainedNetwork/models/weights/superpoint_v1.pth

Prerequisite: Dataset

Dataset

To test our model using the InLoc dataset, the dataset should be downloaded. Downloading takes a while (Dataset is about 1.0TB). Click here to download dataset.

Quick Start

• Clone repository
  • git clone --recurse-submodules https://github.com/JanghunHyeon/PCLoc.git
• Download InLoc dataset
• Install dependencies
• Download model parameters and unzip the files at each of above path
• Modify database_setup.py
  • line 21 (--db_dir) : path to inloc dataset
  • line 22 (--save_dir) : path to save directory of database features
• Execute database_setup.py in python, which prepares database features.
• Modify main_inference.py
  • line 39 (--query_dir) : path to query directory.
  • line 40 (--db_dir) : path to database features, which was generated by running database_setup.py.
• Execute main_inference.py in python, and results are saved at --log_dir.

Database Description

• netvlad_feats.npy: Global descriptors (NetVLAD) of the database images.
• local_feats: Local features and the corresponding 3D coordinates to the keypoints of each database image.
• pc_feats: Local feature map used for the pose correction.
• scans_npy.npy: RGB-D scan data from the dataset (InLoc), which is used for pose verification.

Contents

The provided sample code (06_main_inference.py) runs pose correction. This code provides three options:

• --opt_div_matching: Usage of Divided Matching

Example: --opt_div_matching

• False: Table 5 (b-1) from the paper
• True: Table 5 (b-2) from the paper

Results

After running the code, results are shown in the --log_dir.

Example: ./log/202103241833/IMG_0738/mpv

• 00_query_img.jpg: image used for the query.
• 01_final_pose.jpg: rendered image at the final pose.
• 02_final_err_30.837045.jpg: error image between the query and the rendered image.
• pred_IMG_0738.txt: estimated final pose.
• all/*: top-k candidates from the pose correction.

Every evaluation was conudcted with the online viusal localization benchmark server. visuallocalization.net/benchmark

BibTeX Citation

If you use any ideas from the paper or code from this repo, please consider citing:

@inproceedings{hyeon2021pose,
  title={Pose Correction for Highly Accurate Visual Localization in Large-Scale Indoor Spaces},
  author={Hyeon, Janghun and Kim, Joohyung and Doh, Nakju},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={15974--15983},
  year={2021}
}