Home

Awesome

MeshLoc

This repository contains the code for our publication MeshLoc: Mesh-Based Visual Localization, presented at ECCV 2022.

License

This repository is licensed under the 3-Clause BSD License. See the LICENSE file for full text.

Citation

If you are using the code in this repository, please cite the following paper:

@inproceedings{Panek2022ECCV,
  author = {Panek, Vojtech and Kukelova, Zuzana and Sattler, Torsten},
  title = {{MeshLoc: Mesh-Based Visual Localization}},
  booktitle={European Conference on Computer Vision (ECCV)},
  year = {2022},
}

Installation

Make sure you have COLMAP installed on your system. If not, please follow the instructions on the COLMAP website. The current version of MeshLoc was tested with COLMAP 3.11, but should be compatible with any version >= 3.9.

This version of MeshLoc uses the Image Matching Toolbox for feature matching. Immatch package unfortunately cannot be directly installed as a package from GitHub due to the way it downloads the individual matching methods as submodules. Therefore, it has to be installed from the source. The full installation process is as follows:

git clone https://github.com/GrumpyZhou/image-matching-toolbox.git
cd image-matching-toolbox
git submodule update --init
conda create --name immatch python=3.7
conda activate immatch
pip install Cython loguru jupyter scipy matplotlib opencv-python torch==1.13.1 torchvision pytorch-lightning faiss-gpu h5py==3.6.0
pip install -e .
cd pretrained
bash download.sh
cd ../..
git clone --recurse-submodules https://github.com/tsattler/meshloc_release.git
cd meshloc_release
pip install -e .

In case the Image Matching Toolbox fails to download the pretrained models, you might need to switch to a more recent branch of the submodules. Example for Patch2Pix:

cd image-matching-toolbox/third_party/patch2pix
git checkout main

Usage

Current implementation contains localization scripts for two datasets - Aachen v1.1 and 12 Scenes. The scripts should be launched from image-matching-toolbox directory so the scripts have direct access to the feature matching configuration files. Also activate the prepared immatch conda environment with all necessary packages.

Example on Aachen v1.1 dataset:

cd <image_matching_toolbox_dir>

conda activate immatch

python3 <mesh_loc_dir>/localize.py \
--db_image_dir <meshloc_dataset_path>/aachen_day_night_v11/images/images_db_undist_800 \
--db_depth_image_dir <meshloc_dataset_path>/aachen_day_night_v11/db_renderings/AC14_depth_800_undist \
--colmap_model_dir <meshloc_dataset_path>/aachen_day_night_v11/db_colmap_models/800_undist_underscores \
--query_dir <meshloc_dataset_path>/aachen_day_night_v11/images/images_q_night_800 \
--query_list <meshloc_dataset_path>/aachen_day_night_v11/night_time_queries_with_intrinsics_800_basenames.txt \
--out_prefix <experiment_outputs_dir_path> \
--match_prefix <experiment_matches_dir_path> \
--method_name patch2pix \
--method_config aachen_v1.1 \
--method_string patch2pix_aachen_v1_1_ \
--retrieval_pairs <meshloc_dataset_path>/aachen_day_night_v11/retrieval_pairs/NetVLAD_top50_underscores.txt \
--top_k 50 \
--max_side_length -1 \
--ransac_type POSELIB+REF \
--min_ransac_iterations 10000 \
--max_ransac_iterations 100000 \
--reproj_error 20.0 \
--use_orig_db_images \
--cluster_keypoints

Example on 12 Scenes dataset:

cd <image_matching_toolbox_dir>

conda activate immatch

python3 <mesh_loc_dir>/localize_12scenes.py \
--db_image_dir <12_scenes_dataset_path>/office1/manolis/data \
--db_depth_image_dir <meshloc_dataset_path>/12_scenes/db_renderings/12_scenes_apt1_kitchen_depth \
--colmap_model_dir <meshloc_dataset_path>/12_scenes/db_colmap_models/apt1_kitchen \
--query_dir <12_scenes_dataset_path>/office1/manolis/data \
--query_list <meshloc_dataset_path>/12_scenes/queries_with_intrinsics/apt1_kitchen_queries_with_intrinsics.txt \
--out_prefix <experiment_outputs_dir_path> \
--match_prefix <experiment_matches_dir_path> \
--method_name loftr \
--method_config default \
--method_string loftr_default_ \
--retrieval_pairs <meshloc_dataset_path>/12_scenes/retrieval_pairs/apt1_kitchen_DVLAD_top20.txt \
--top_k 20 \
--max_side_length -1 \
--ransac_type POSELIB+REF \
--min_ransac_iterations 10000 \
--max_ransac_iterations 100000 \
--reproj_error 20.0 \
--use_orig_db_images

List of localization script arguments:

The pose estimates from the localization pipeline on Aachen v1.1 can be evaluated by benchmark at https://www.visuallocalization.net/. The pose estimates on 12 Scenes can be evaluated by the https://github.com/tsattler/visloc_pseudo_gt_limitations repository, which also contains links to 12 Scenes COLMAP models.

Data

We used Aachen v1.1 and 12 Scenes datasets for the evaluation in our paper.

The data derived from the above mentioned datasets (such as meshes, renderings, or COLMAP models) can be found in our data repository at https://data.ciirc.cvut.cz/public/projects/2022MeshLoc.

You can use the download script to easily get the meshes and renderings. The repository also contains resized and undistorted images for Aachen v1.1 dataset and corresponding COLMAP models.

Script parameters:

Note that all the images and renderings for Aachen v1.1 in our repository use a flattened data structure - all database images are in a single directory and their original relative path is in their names (db/1883.jpg --> db_1883.jpg), query images use only their basenames (query/night/nexus5x_additional_night/IMG_20170702_005927.jpg --> IMG_20170702_005927.jpg). We provide database COLMAP models and retrieval files for both the original data structure (800_undist, NetVLAD_top50.txt) and flattened data structure (800_undist_underscores, NetVLAD_top50_underscores.txt). All the provided images and COLMAP models are undistorted and some are resized to 800px on the longer side.

Acknowledgements

This repository is heavily using PoseLib, RansacLib and Image Matching Toolbox. We would like to thank all the contributors of these repositories.