Awesome

Camera Pose Auto-Encoders (PAEs)

Overview

This repository provides an official implementation for:

• Camera Pose Auto-Encoders (PAEs), accepted to ECCV 2022
• Iterative Absolute Pose Regression (iAPR), which extends our ECCV22 work, a new class of APRs, combining absolute pose regression and relative pose regression, without extra image or pose storage.

Introduction

Camera Pose Auto-Encoders (PAEs) are multi-layer perceptrons (MLPs), trained via a Teacher-Student approach to encode camera poses, using Absolute Pose Regressors (APRs) as their teachers (Fig. 1). Once trained, PAEs can closely reproduce their teachers' performance, across outdoor and indoor environments and when learning from multi- and single- scene APR teachers with different architectures.

Below we provide instructions for running our code in order to train teacher APRs and student PAEs and for evaluating them. We also provide pre-trained models.

Once a PAE is trained, we can use it to as a means for extending pose regression with visual and spatial information at a minimal cost.

Iterative Absolute Pose Regression (iAPR) is a new class of APRs, which combines absolute pose regression and relative pose regression, without additional image or pose storage. Specifically, it applies a PAE-based RPR on the initial APR estimate for one or more iterations (Fig. 2). iAPR achieves a new state-of-the-art (SOTA) localization accuracy for APRs on the 7Scenes dataset, even when trained with only 30% of the data.

As for PAEs, we provide instructions for training and testing our iAPR model.

Prerequisites

In order to run this repository you will need:

1. Python3 (tested with Python 3.7.7)
2. PyTorch deep learning framework (tested with version 1.0.0)
3. Download the Cambridge Landmarks dataset and the 7Scenes dataset
4. You can also download pre-trained models to reproduce reported results (see below)
5. For a quick set up you can run: pip install -r requirments.txt Note: All experiments reported in our paper were performed with an 8GB 1080 NVIDIA GeForce GTX GPU

PAEs

Training and Testing Teacher APRs

Our code allows training and testing of single-scene and multi-scene APR teachers. Specifically, we use PoseNet with different CNN backbones as our single-scene APRs and MS-Transformer as our multi-scene APR.

For example, in order to train PoseNet with EfficientNet-B0 on the KingsCollege scene, run:

python main_train_test_apr.py posenet train models/backbones/efficient-net-b0.pth
<path to the CambridgeLandmarks dataset>
datasets/CambridgeLandmarks/abs_cambridge_pose_sorted.csv_KingsCollege_train.csv
CambridgeLandmarks_config.json

In order to train with a different backbone, change the path to the backbone (third argument) and the value of 'backbone_type', under the 'posenet' dictionary, in the json configuraion file. We support MobileNet and ResNet50.

After training, you can test your trained model by running:

python main_train_test_apr.py posenet test models/backbones/efficient-net-b0.pth <path to the CambridgeLandmarks dataset> datasets/CambridgeLandmarks/abs_cambridge_pose_sorted.csv_KingsCollege_test.csv
CambridgeLandmarks_config.json --checkpoint_path posenet_effnet_apr_kings_college.pth

In order to train and test MS-Transformer, please follow the instructions at our MS-Transformer repository

Training and Testing Student PAEs

Single-scene PAEs

To train a single-scene PAE with a PoseNet Teacher (with an EfficientNet-B0 backbone), run the following command, using the same configuration used for the teacher APR:

python main_learn_pose_encoding.py posenet train models/backbones/efficient-net-b0.pth
<path to dataset> datasets/CambridgeLandmarks/abs_cambridge_pose_sorted.csv_KingsCollege_train.csv
CambridgeLandmarks_config.json posenet_effnet_apr_kings_college.pth

You can then evaluate it and compare it to its teacher, by running:

python main_learn_pose_encoding.py posenet test models/backbones/efficient-net-b0.pth
<path to dataset> datasets/CambridgeLandmarks/abs_cambridge_pose_sorted.csv_KingsCollege_test.csv CambridgeLandmarks_config.json posenet_effnet_apr_kings_college.pth
--encoder_checkpoint_path posenet_effnet_apr_kings_college.pth

Multi-scene PAEs

Similarly, you can train a multi-scene PAE with MS-Transformer. For example, training on the 7Scenes dataset:

python main_learn_multiscene_pose_encoding.py
ems-transposenet
train
models/backbones/efficient-net-b0.pth
<path to dataset>
datasets/7Scenes/7scenes_all_scenes.csv
7scenes_config.json 
ems_transposenet_7scenes_pretrained.pth

and then evaluate it by running:

python main_learn_multiscene_pose_encoding.py
ems-transposenet
test
/models/backbones/efficient-net-b0.pth
<path to dataset>
datasets/7Scenes/abs_7scenes_pose.csv_fire_test.csv
7scenes_config.json
ems_transposenet_7scenes_pretrained.pth
--encoder_checkpoint_path 
mstransformer_7scenes_pose_encoder.pth

Pre-trained Models

iAPR

Training and Testing

We propose a PAE-based RPR model (Fig. 3) to estimate the relative motion between an encoded pose and a query image.

In order to train our model, run:

python main_iapr.py train <path to dataset> 7scenes_training_pairs.csv
7scenes_iapr_config.json pretrained_models/ems_transposenet_7scenes_pretrained.pth
models/backbones/efficient-net-b0.pth pretrained_models/mstransformer_7scenes_pose_encoder.pth 

Note: links to data are available below.

In order to test iAPR with our model, for example with the chess scene, run:

python main_iapr.py test <path to dataset> datasets/7Scenes/abs_7scenes_pose.csv_chess_test.csv
7scenes_iapr_config.json pretrained_models/ems_transposenet_7scenes_pretrained.pth
models/backbones/efficient-net-b0.pth pretrained_models/mstransformer_7scenes_pose_encoder.pth 
--checkpoint_path <path to iapr model>             

You can change the number of iterations in the configuration file. Pretrained models are available below.

our iAPR achieves SOTA performance on the 7Scenes dataset and improves performance even when trained on a much smaller subset of the training data.

The following table shows the pose error (in meters/degrees) of MS-Transformer and iAPR, for the 7Scenes dataset, when training with 100%, 70%, 50% and 30% of the train set:

Data

Pre-trained Models

Other Applications of PAEs

Decoding Images from Encoded Train Poses

To train an image decoder for a PAE for the ShopFacade scene:

python main_reconstruct_img.py train <path to cambridge dataset>
datasets/CambridgeLandmarks/abs_cambridge_pose_sorted.csv_ShopFacade_train.csv
reconstruct_config.json pretrained_models/mstransformer_cambridge_pose_encoder.pth

To test our Decoder (decoding the train images for their PAE encoded poses):

python main_reconstruct_img.py
demo <path to cambridge dataset> datasets/CambridgeLandmarks/abs_cambridge_pose_sorted.csv_ShopFacade_train.csv
reconstruct_config.json pretrained_models/mstransformer_cambridge_pose_encoder.pth
--decoder_checkpoint_path pretrained_models/img_decoder_shop_facade.pth

You can download the pre-trained ShopFacade decoder from here