Home

Awesome

KFNet

This is a Tensorflow implementation of our CVPR 2020 Oral paper - "KFNet: Learning Temporal Camera Relocalization using Kalman Filtering" by Lei Zhou, Zixin Luo, Tianwei Shen, Jiahui Zhang, Mingmin Zhen, Yao Yao, Tian Fang, Long Quan.

This paper addresses the temporal camera relocalization of time-series image data by folding the scene coordinate regression problem into the principled Kalman filter framework.

If you find this project useful, please cite:

@inproceedings{zhou2020kfnet,
  title={KFNet: Learning Temporal Camera Relocalization using Kalman Filtering},
  author={Zhou, Lei and Luo, Zixin and Shen, Tianwei and Zhang, Jiahui and Zhen, Mingmin and Yao, Yao and Fang, Tian and Quan, Long},
  booktitle={Computer Vision and Pattern Recognition (CVPR)},
  year={2020}
}

Contents

About

Network architecture

<p align="center"> <img src=doc/architecture.jpg alt="drawing" width="700"/> </p>

Sample results on 7scenes and 12scenes

KFNet simultaneously predicts the mapping points and camera poses in a temporal fashion within the coordinate system defined by a known scene.

DSAC++KFNet
7scenes-fireAlt TextAlt Text
12scenes-office2-5aAlt TextAlt Text
DescriptionBlue - ground truth posesRed - estimated poses

Intermediate uncertainty predictions

Below we visualize the measurement and process noise.

DataMeasurement noiseProcess noise
7scenes-fireAlt TextAlt Text
12scenes-office2-5aAlt TextAlt Text
DescriptionThe brighter color means smaller noise.The figure bar measures the inverse of the covariances (in centimeters)

Intermediate optical flow results on 7scenes, 12scenes, Cambridge and DeepLoc

As an essential component of KFNet, the process system of KFNet (i.e., OFlowNet) delineates pixel transitions across frames through optical flow reasoning yet without recourse to grourd truth optical flow labelling. We visualize the predicted optical flow fields below while suppressing the predictions with too large uncertainties.

DataDescriptionOptical flow
7scenes-fireIndoor; hand-held; small shaky motions<img src="doc/fire_flow.gif" width="375">
12scenes-office2-5aIndoor; hand-held; larger movements<img src="doc/office2_5a_flow.gif" width="375">
Cambridge-KingsCollegeOutdoor; hand-held; large random motions<img src="doc/KingsCollege_flow.gif" width="375">
DeepLocOutdoor; vehicle-mounted; forward motions<img src="doc/DeepLoc_flow.gif" width="375">

Remark For DeepLoc, since OFlowNet is trained only on one scene included in DeepLoc, the flow predictions appear somewhat messy due to the lack of training data. Training with a larger amount and variety of data would improve the results.

Usage

File format

Environment

sudo pip install -r requirements.txt
conda create -f environment.yml
conda activate KFNet

Testing

You can download the trained models of 7scenes from the Google drive link (3G).

git checkout SCoordNet
python SCoordnet/eval.py --input_folder <input_folder> --output_folder <output_folder> --model_folder <model_folder> --scene <scene>
# <scene> = chess/fire/heads/office/pumpkin/redkitchen/stairs, i.e., one of the scene names of 7scenes dataset
git checkout OFlowNet
python OFlowNet/eval --input_folder <input_folder> --output_folder <output_folder> --model_folder <model_folder>

The testing program of OFlowNet will save the 2-d optical flows and 1-d uncertainties of consecutive image pairs as npy files of the dimension 60x80x3. You can visualize the flow results by running scripts vis/vis_optical_flow.py and vis/vis_optical_flow_list.py.

git checkout master
python KFNet/eval.py --input_folder <input_folder> --output_folder <output_folder> --model_folder <model_folder> --scene <scene>
unzip PnP.zip && cd PnP
python main.py <path_to_output_file_list> <output_folder> --gt <path_to_ground_truth_pose_list> --thread_num <32>
// Please note that you need to install git-lfs before cloning to get PnP.zip, since the zip file is stored via LFS.

Training

The training procedure has 3 stages.

  1. Train SCoordNet for each scene independently.
git checkout SCoordnet
python SCoordNet/train.py --input_folder <input_folder> --model_folder <scoordnet_model_folder> --scene <scene>
  1. Train OFlowNet using all the image sequences that are not limited to any specific scenes, for example, concatenating all the image_list.txt and label_list.txt of 7scenes for training.
git checkout OFlowNet
python OFlowNet/train.py --input_folder <input_folder> --model_folder <oflownet_model_folder>
  1. Train KFNet for each scene from the pre-trained SCoordNet and OFlowNet models to jointly finetune their parameters.
git checkout master
python KFNet/train.py --input_folder <input_folder> --model_folder <model_folder> --scoordnet <scoordnet_model_folder> --oflownet <oflownet_model_folder> --scene <scene>

Credit

This implementation was developed by Lei Zhou. Feel free to contact Lei for any enquiry.