Awesome

Dense Matching

A general dense matching library based on PyTorch.

For any questions, issues or recommendations, please contact Prune at prune.truong@vision.ee.ethz.ch

If you're interested in training a probabilistic correspondence network unsupervised (mix of PDCNet and WarpC), check out our recent work Refign, with code. It is also integrated in this code base as UAWarpC.

Updates

06/03/2022: We found that significantly better performance and reduced training time are obtained when initializing with bilinear interpolation weights the weights of the transposed convolutions used to upsample the predicted flow fields between the different pyramid levels. We have integrated this initialization as the default. We might provide updated pre-trained weights as well. Alternatively, one can directly simply use bilinear interpolation for upsampling with similar (maybe a bit better) performance, which is also now an option proposed.

Highlights

Libraries for implementing, training and evaluating dense matching networks. It includes

• Common dense matching validation datasets for geometric matching (MegaDepth, RobotCar, ETH3D, HPatches), optical flow (KITTI, Sintel) and semantic matching (TSS, PF-Pascal, PF-Willow, Spair).

• Scripts to analyse network performance and obtain standard performance scores for matching and pose estimation.

• General building blocks, including deep networks, optimization, feature extraction and utilities.

• General training framework for training dense matching networks with

  • Common training datasets for matching networks.
  • Functions to generate random image pairs and their corresponding ground-truth flow, as well as to add moving objects and modify the flow accordingly.
  • Functions for data sampling, processing etc.
  • And much more...

• Official implementation of GLU-Net (CVPR 2020), GLU-Net-GOCor (NeurIPS 2020), PWC-Net-GOCor (NeurIPS 2020), PDC-Net (CVPR 2021), WarpC models (ICCV 2021), PWarpC models (CVPR 2022) including trained models and respective results.

Dense Matching Networks

The repo contains the implementation of the following matching models. We provide pre-trained model weights, data preparation, evaluation commands, and results for each dataset and method. If you find this library useful, please consider citing the relevant following research papers.

[6] PWarpC: Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences. (CVPR 2022)

Authors: Prune Truong, Martin Danelljan, Fisher Yu, Luc Van Gool<br />

[Paper] [Website] [Poster] [Video] [Citation]

We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. Our approach directly supervises the dense matching scores predicted by the network, encoded as a conditional probability distribution. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our probabilistic learning objectives are then derived using the constraints arising from the resulting image triplet. We further account for occlusion and background clutter present in real image pairs by extending our probabilistic output space with a learnable unmatched state. To supervise it, we design an objective between image pairs depicting different object classes. We validate our method by applying it to four recent semantic matching architectures. Our weakly-supervised approach sets a new state-of-the-art on four challenging semantic matching benchmarks. Lastly, we demonstrate that our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.

[5] PDC-Net+: Enhanced Probabilistic Dense Correspondence Network. (TPAMI 2023)

Authors: Prune Truong, Martin Danelljan, Radu Timofte, Luc Van Gool <br />

[Paper] [Website] [Citation]

[4] WarpC: Warp Consistency for Unsupervised Learning of Dense Correspondences. (ICCV 2021 - ORAL)

Authors: Prune Truong, Martin Danelljan, Fisher Yu, Luc Van Gool<br />

[Paper] [Website] [Poster] [Slides] [Video] [Citation]

Warp Consistency Graph	Results

The key challenge in learning dense correspondences lies in the lack of ground-truth matches for real image pairs. While photometric consistency losses provide unsupervised alternatives, they struggle with large appearance changes, which are ubiquitous in geometric and semantic matching tasks. Moreover, methods relying on synthetic training pairs often suffer from poor generalisation to real data. We propose Warp Consistency, an unsupervised learning objective for dense correspondence regression. Our objective is effective even in settings with large appearance and view-point changes. Given a pair of real images, we first construct an image triplet by applying a randomly sampled warp to one of the original images. We derive and analyze all flow-consistency constraints arising between the triplet. From our observations and empirical results, we design a general unsupervised objective employing two of the derived constraints. We validate our warp consistency loss by training three recent dense correspondence networks for the geometric and semantic matching tasks. Our approach sets a new state-of-the-art on several challenging benchmarks, including MegaDepth, RobotCar and TSS.

[3] PDC-Net: Learning Accurate Correspondences and When to Trust Them. (CVPR 2021 - ORAL)

Authors: Prune Truong, Martin Danelljan, Luc Van Gool, Radu Timofte<br />

[Paper] [Website] [Poster] [Slides] [Video] [Citation]

Dense flow estimation is often inaccurate in the case of large displacements or homogeneous regions. For most applications and down-stream tasks, such as pose estimation, image manipulation, or 3D reconstruction, it is crucial to know when and where to trust the estimated matches. In this work, we aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map indicating the reliability and accuracy of the prediction. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training.

[2] GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network. (NeurIPS 2020)

Authors: Prune Truong *, Martin Danelljan *, Luc Van Gool, Radu Timofte<br />

[Paper] [Website] [Video] [Citation]

The feature correlation layer serves as a key neural network module in numerous computer vision problems that involve dense correspondences between image pairs. It predicts a correspondence volume by evaluating dense scalar products between feature vectors extracted from pairs of locations in two images. However, this point-to-point feature comparison is insufficient when disambiguating multiple similar regions in an image, severely affecting the performance of the end task. This work proposes GOCor, a fully differentiable dense matching module, acting as a direct replacement to the feature correlation layer. The correspondence volume generated by our module is the result of an internal optimization procedure that explicitly accounts for similar regions in the scene. Moreover, our approach is capable of effectively learning spatial matching priors to resolve further matching ambiguities.

[1] GLU-Net: Global-Local Universal Network for dense flow and correspondences (CVPR 2020 - ORAL).

Authors: Prune Truong, Martin Danelljan and Radu Timofte <br /> [Paper] [Website] [Poster] [Oral Video] [Teaser Video] [Citation]

Pre-trained weights

The pre-trained models can be found in the model zoo

Table of Content

1. Installation
2. Test on your own image pairs!
   1. Test on image pairs
   2. Various demos
3. Overview
4. Benchmarks and results
   1. Correspondence evaluation
      1. MegaDepth
      2. RobotCar
      3. ETH3D
      4. HPatches
      5. KITTI
      6. Sintel
      7. TSS
      8. PF-Pascal
      9. PF-Willow
      10. Spair-71k
      11. Caltech-101
   2. Pose estimation
      1. YFCC100M
      2. ScanNet
   3. Sparse evaluation on HPatches
5. Training
6. Acknowledgement
7. Changelog

1. Installation <a name="installation"></a>

Inference runs for torch version >= 1.0

• Create and activate conda environment with Python 3.x

conda create -n dense_matching_env python=3.7
conda activate dense_matching_env

• Install all dependencies (except for cupy, see below) by running the following command:

pip install numpy opencv-python torch torchvision matplotlib imageio jpeg4py scipy pandas tqdm gdown pycocotools timm

Note: CUDA is required to run the code. Indeed, the correlation layer is implemented in CUDA using CuPy, which is why CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the provided binary packages as outlined in the CuPy repository. The code was developed using Python 3.7 & PyTorch 1.0 & CUDA 9.0, which is why I installed cupy for cuda90. For another CUDA version, change accordingly.

pip install cupy-cuda90 --no-cache-dir 

• This repo includes GOCor as git submodule. You need to pull submodules with

git submodule update --init --recursive
git submodule update --recursive --remote

• Create admin/local.py by running the following command and update the paths to the dataset. We provide an example admin/local_example.py where all datasets are stored in data/.

python -c "from admin.environment import create_default_local_file; create_default_local_file()"

• Download pre-trained model weights with the command bash assets/download_pre_trained_models.sh. See more in model zoo

2. Test on your own image pairs! <a name="test"></a>

Possible model choices are :

• SFNet, PWarpCSFNet_WS, PWarpCSFNet_SS, NCNet, PWarpCNCNet_WS, PWarpCNCNet_SS, CATs, PWarpCCATs_SS, CATs_ft_features, PWarpCCATs_ft_features_SS,
• UAWarpC (no pre-trained models available),
• WarpCGLUNet, GLUNet_star, WarpCSemanticGLUNet,
• PDCNet_plus, PDCNet, GLUNet_GOCor_star,
• SemanticGLUNet, GLUNet, GLUNet_GOCor, PWCNet, PWCNet_GOCor

Possible pre-trained model choices are: static, dynamic, chairs_things, chairs_things_ft_sintel, megadepth, megadepth_stage1, pfpascal, spair

PDC-Net and PDC-Net+ have multiple inference alternative options. if model is PDC-Net, add options:

• --confidence_map_R, for computation of the confidence map p_r, default is 1.0
• --multi_stage_type in
  • 'D' (or 'direct')
  • 'H' (or 'homography_from_quarter_resolution_uncertainty')
  • 'MS' (or 'multiscale_homo_from_quarter_resolution_uncertainty')
• --ransac_thresh, used for homography and multiscale multi-stages type, default is 1.0
• --mask_type, for thresholding the estimated confidence map and using the confident matches for internal homography estimation, for homography and multiscale multi-stage types, default is proba_interval_1_above_5
• --homography_visibility_mask, default is True
• --scaling_factors', used for multi-scale, default are [0.5, 0.6, 0.88, 1, 1.33, 1.66, 2]

Use direct ('D') when image pairs only show limited view-point changes (for example consecutive images of a video, like in the optical flow task). For larger view-point changes, use homography ('H') or multi-scale ('MS').

For example, to run PDC-Net or PDC-Net+ with homography, add at the end of the command

PDCNet --multi_stage_type H --mask_type proba_interval_1_above_10

Test on a specific image pair <a name="test1"></a>

You can test the networks on a pair of images using test_models.py and the provided trained model weights. You must first choose the model and pre-trained weights to use. The inputs are the paths to the query and reference images. The images are then passed to the network which outputs the corresponding flow field relating the reference to the query image. The query is then warped according to the estimated flow, and a figure is saved.

For this pair of MegaDepth images (provided to check that the code is working properly) and using PDCNet (MS) trained on the megadepth dataset, the output is:

python test_models.py --model PDCNet --pre_trained_model megadepth --path_query_image images/piazza_san_marco_0.jpg --path_reference_image images/piazza_san_marco_1.jpg --save_dir evaluation/ PDCNet --multi_stage_type MS --mask_type proba_interval_1_above_10

additional optional arguments: --path_to_pre_trained_models (default is pre_trained_models/)

Using GLU-Net-GOCor trained on the dynamic dataset, the output for this image pair of eth3d is:

python test_models.py --model GLUNet_GOCor --pre_trained_model dynamic --path_query_image images/eth3d_query.png --path_reference_image images/eth3d_reference.png --save_dir evaluation/

For baseline GLU-Net, the output is instead:

python test_models.py --model GLUNet --pre_trained_model dynamic --path_query_image images/eth3d_query.png --path_reference_image images/eth3d_reference.png --save_dir evaluation/

And for PWC-Net-GOCor and baseline PWC-Net:

python test_models.py --model PWCNet_GOCor --pre_trained_model chairs_things --path_query_image images/kitti2015_query.png --path_reference_image images/kitti2015_reference.png --save_dir evaluation/

python test_models.py --model PWCNet --pre_trained_model chairs_things --path_query_image images/kitti2015_query.png --path_reference_image images/kitti2015_reference.png --save_dir evaluation/

Demos with videos <a name="test2"></a>

• demos/demo_single_pair.ipynb: Play around with our models on different image pairs, compute the flow field relating an image pair and visualize the warped images and confident matches.

• demos/demo_pose_estimation_and_reconstruction.ipynb: Play around with our models on different image pairs (with intrinsic camera parameters known), compute the flow field and confidence map, then the relative pose.

• Run the online demo with a webcam or video to reproduce the result shown in the GIF above. We compute the flow field between the target (middle) and the source (left). We plot the 1000 top confident matches as well. The warped source is represented on the right (and should resemble the middle). Only the regions for which the matches were predicted as confident are visible.

We modify the utils code from SuperGlue, so you need to adhere to their license in order to use that. Then run:

bash demo/run_demo_confident_matches.sh

• From a video, warp each frame of the video (left) to the middle frame like in the GIF below. The warped frame is represented on the right (and should resemble the middle). Only the regions for which the matches were predicted as confident are visible.

We modify the utils code from SuperGlue, so you need to adhere to their license in order to use that. Then run:

bash demo/run_demo_warping_videos.sh

• More to come!

3. Overview <a name="overview"></a>

The framework consists of the following sub-modules.

• training:
  • actors: Contains the actor classes for different trainings. The actor class is responsible for passing the input data through the network and calculating losses. Here are also pre-processing classes, that process batch tensor inputs to the desired inputs needed for training the network.
  • trainers: The main class which runs the training.
  • losses: Contain the loss classes
• train_settings: Contains settings files, specifying the training of a network.
• admin: Includes functions for loading networks, tensorboard etc. and also contains environment settings.
• datasets: Contains integration of a number of datasets. Additionally, it includes modules to generate synthetic image pairs and their corresponding ground-truth flow as well as to add independently moving objects and modify the flow accordingly.
• utils_data: Contains functions for processing data, e.g. loading images, data augmentations, sampling frames.
• utils_flow: Contains functions for working with flow fields, e.g. converting to mapping, warping an array according to a flow, as well as visualization tools.
• third_party: External libraries needed for training. Added as submodules.
• models: Contains different layers and network definitions.
• validation: Contains functions to evaluate and analyze the performance of the networks in terms of predicted flow and uncertainty.

4. Benchmark and results <a name="results"></a>

All paths to the datasets must be provided in file admin/local.py. We provide an example admin/local_example.py where all datasets are stored in data/. You need to update the paths of admin/local.py before running the evaluation.

PDC-Net and PDC-Net+ has multiple inference alternative options. if model if PDC-Net, add options:

• --confidence_map_R, for computation of the confidence map p_r, default is 1.0
• --multi_stage_type in
  • 'D' (or 'direct')
  • 'H' (or 'homography_from_quarter_resolution_uncertainty')
  • 'MS' (or 'multiscale_homo_from_quarter_resolution_uncertainty')
• --ransac_thresh, used for homography and multiscale multi-stages type, default is 1.0
• --mask_type, for thresholding the estimated confidence map and using the confident matches for internal homography estimation, for homography and multiscale multi-stage types, default is proba_interval_1_above_5
• --homography_visibility_mask, default is True
• --scaling_factors', used for multi-scale, default are [0.5, 0.6, 0.88, 1, 1.33, 1.66, 2]

For example, to run PDC-Net or PDC-Net+ with homography, add at the end of the command

PDCNet --multi_stage_type H --mask_type proba_interval_1_above_10

Results using PDC-Net with multi-stage (homography_from_quarter_resolution_uncertainty, H) or multi-scale (multiscale_homo_from_quarter_resolution_uncertainty, MS) employ RANSAC internally. Therefore results may vary a bit but should remain within 1-2 %. For pose estimation, we also compute the pose with RANSAC, which leads to some variability in the results.

4.1. Correspondence evaluation <a name="correspondence_eval"></a>

Metrics are computed with,

python -u eval_matching.py --dataset dataset_name --model model_name --pre_trained_models pre_trained_model_name --optim_iter optim_step  --local_optim_iter local_optim_iter --save_dir path_to_save_dir 

Optional argument: --path_to_pre_trained_models:

• default is pre_trained_models/
• if it is a path to a directory: it is the path to the directory containing the model weights, the path to the model weights will be path_to_pre_trained_models + model_name + '_' + pre_trained_model_name
• if it is a path to a checkpoint directly, it is used as the path to the model weights directly, and pre_trained_model_name is only used as the name to save the metrics.

Data preparation: We use the test set provided in RANSAC-Flow. It is composed of 1600 pairs and also includes a csv file ('test1600Pairs.csv') containing the name of image pairs to evaluate and the corresponding ground-truth correspondences. Download everything with

bash assets/download_megadepth_test.sh

The resulting file structure is the following

megadepth_test_set/
└── MegaDepth/
    └── Test/
        └── test1600Pairs/  
        └── test1600Pairs.csv

<br /><br /> Evaluation: After updating the path of 'megadepth' and 'megadepth_csv' in admin/local.py, evaluation is run with

python eval_matching.py --dataset megadepth --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type MS

Similar results should be obtained:

Data preparation: Images can be downloaded from the Visual Localization Challenge (at the bottom of the site), or more precisely here. The CSV file with the ground-truth correspondences can be downloaded from here. The file structure should be the following:

RobotCar
├── img/
└── test6511.csv

<br /><br /> Evaluation: After updating the path of 'robotcar' and 'robotcar_csv' in admin/local.py, evaluation is run with

python eval_matching.py --dataset robotcar --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type MS

Similar results should be obtained:

Data preparation: execute 'bash assets/download_ETH3D.sh' from our GLU-Net repo. It does the following:

• Create your root directory ETH3D/, create two sub-directories multiview_testing/ and multiview_training/
• Download the "Low rew multi-view, training data, all distorted images" here and unzip them in multiview_training/
• Download the "Low rew multi-view, testing data, all undistorted images" here and unzip them in multiview_testing/
• We directly provide correspondences for pairs of images taken at different intervals. There is one bundle file for each dataset and each rate of interval, for example "lakeside_every_5_rate_of_3". This means that we sampled the source images every 5 images and the target image is taken at a particular rate from each source image. Download all these files here and unzip them.

As illustration, your root ETH3D directory should be organised as follows:

The organisation of your directories is important, since the bundle files contain the relative paths to the images, from the ETH3D root folder.

<br /><br /> Evaluation: for each interval rate (3,5,7,9,11,13,15), we compute the metrics for each of the sub-datasets (lakeside, delivery area and so on). The final metrics are the average over all datasets for each rate. After updating the path of 'eth3d' in admin/local.py, evaluation is run with

python eval_matching.py --dataset robotcar --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type D

PCK-1 for different rates of intervals between image pairs:

Note that the PCKs are computed per image, and then averaged per sequence. The final metrics is the average over all sequences. It corresponds to the results '_per_image' in the outputted metric file. Note that this is not the metrics used in the PDC-Net paper, where the PCKs are c omputed per sequence instead, using the PDC-Net direct approach (corresponds to results '-per-rate' in outputted metric file).

PCK-5 for different rates of intervals between image pairs:

Data preparation: Download the data with

bash assets/download_hpatches.sh

The corresponding csv files for each viewpoint ID with the path to the images and the homography parameters relating the pairs are listed in assets/.

<br /><br /> Evaluation: After updating the path of 'hp' in admin/local.py, evaluation is run with

python eval_matching.py --dataset hp --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: Both KITTI-2012 and 2015 datasets are available here

Evaluation: After updating the path of 'kitti2012' and 'kitti2015' in admin/local.py, evaluation is run with

python eval_matching.py --dataset kitti2015 --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 PDCNet --multi_stage_type direct

Similar results should be obtained:

Data preparation: Download the data with

bash assets/download_sintel.sh

Evaluation: After updating the path of 'sintel' in admin/local.py, evaluation is run with

python eval_matching.py --dataset sintel --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type direct

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_tss.sh

Evaluation: After updating the path of 'tss' in admin/local.py, evaluation is run with

python eval_matching.py --dataset TSS --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --flipping_condition True --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_pf_pascal.sh

Evaluation: After updating the path of 'PFPascal' in admin/local.py, evaluation is run with

python eval_matching.py --dataset PFPascal --model WarpCSemanticGLUNet --pre_trained_models pfpascal --flipping_condition False --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_pf_willow.sh

Evaluation: After updating the path of 'PFWillow' in admin/local.py, evaluation is run with

python eval_matching.py --dataset PFWillow --model WarpCSemanticGLUNet --pre_trained_models pfpascal --flipping_condition False --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_spair.sh

Evaluation: After updating the path of 'spair' in admin/local.py, evaluation is run with

python eval_matching.py --dataset spair --model WarpCSemanticGLUNet --pre_trained_models pfpascal  --flipping_condition False --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_caltech.sh

Evaluation: After updating the path of 'spair' in admin/local.py, evaluation is run with

python eval_matching.py --dataset caltech --model WarpCSemanticGLUNet --pre_trained_models pfpascal  --flipping_condition False --save_dir path_to_save_dir

4.2 Pose estimation <a name="pose_estimation"></a>

Metrics are computed with

python -u eval_pose_estimation.py --dataset dataset_name --model model_name --pre_trained_models pre_trained_model_name --optim_iter optim_step  --local_optim_iter local_optim_iter --estimate_at_quarter_reso True --mask_type_for_pose_estimation proba_interval_1_above_10 --save_dir path_to_save_dir 

Data preparation: The groundtruth for YFCC is provided the file assets/yfcc_test_pairs_with_gt_original.txt (from SuperGlue repo). Images can be downloaded from the OANet repo and moved to the desired location

bash assets/download_yfcc.sh

File structure should be

YFCC
└──  images/
       ├── buckingham_palace/
       ├── notre_dame_front_facade/
       ├── reichstag/
       └── sacre_coeur/

<br /><br /> Evaluation: After updating the path 'yfcc' in admin/local.py, compute metrics on YFCC100M with PDC-Net homography (H) using the command:

python -u eval_pose_estimation.py --dataset YFCC --model PDCNet --pre_trained_models megadepth --optim_iter 3  --local_optim_iter 7 --estimate_at_quarter_reso True --mask_type_for_pose_estimation proba_interval_1_above_10 --save_dir path_to_save_dir PDCNet --multi_stage_type H --mask_type proba_interval_1_above_10 

You should get similar metrics (not exactly the same because of RANSAC):

Data preparation: The images of the ScanNet test set (100 scenes, scene0707_00 to scene0806_00) are provided here. They were extracted from ScanNet github repo and processed. We use the groundtruth provided by in the SuperGlue repo provided here in the file assets/scannet_test_pairs_with_gt.txt.

<br /><br /> Evaluation: After updating the path 'scannet_test' in admin/local.py, compute metrics on ScanNet with PDC-Net homography (H) using the command:

python -u eval_pose_estimation.py --dataset scannet --model PDCNet --pre_trained_models megadepth --optim_iter 3  --local_optim_iter 7 --estimate_at_quarter_reso True --mask_type_for_pose_estimation proba_interval_1_above_10 --save_dir path_to_save_dir PDCNet --multi_stage_type H --mask_type proba_interval_1_above_10 

You should get similar metrics (not exactly the same because of RANSAC):

4.3 Sparse evaluation on HPatches <a name="sparse_hp"></a>

We provide the link to the cache results here for the sparse evaluation on HPatches. Check PDC-Net+ for more details.

5. Training <a name="training"></a>

Quick Start

The installation should have generated a local configuration file "admin/local.py". In case the file was not generated, run python -c "from admin.environment import create_default_local_file; create_default_local_file()"to generate it. Next, set the paths to the training workspace, i.e. the directory where the model weights and checkpoints will be saved. Also set the paths to the datasets you want to use (and which should be downloaded beforehand, see below). If all the dependencies have been correctly installed, you can train a network using the run_training.py script in the correct conda environment.

conda activate dense_matching_env
python run_training.py train_module train_name

Here, train_module is the sub-module inside train_settings and train_name is the name of the train setting file to be used.

For example, you can train using the included default train_PDCNet_stage1 settings by running:

python run_training.py PDCNet train_PDCNet_stage1

Training datasets downloading <a name="scanNet"></a>

This is the same image pairs used in GLU-Net repo. For the training, we use a combination of the DPED, CityScapes and ADE-20K datasets. The DPED training dataset is composed of only approximately 5000 sets of images taken by four different cameras. We use the images from two cameras, resulting in around 10,000 images. CityScapes additionally adds about 23,000 images. We complement with a random sample of ADE-20K images with a minimum resolution of 750 x 750. It results in 40.000 original images, used to create pairs of training images by applying geometric transformations to them. The path to the original images as well as the geometric transformation parameters are given in the csv files 'assets/csv_files/homo_aff_tps_train_DPED_CityScape_ADE.csv' and 'assets/csv_files/homo_aff_tps_test_DPED_CityScape_ADE.csv'.

1. Download the original images

• Download the DPED dataset (54 GB) ==> images are created in original_images/

• Download the CityScapes dataset

  • download 'leftImg8bit_trainvaltest.zip' (11GB, left 8-bit images - train, val, and test sets', 5000 images) ==> images are created in CityScape/
  • download leftImg8bit_trainextra.zip (44GB, left 8-bit images - trainextra set, 19998 images) ==> images are created in CityScape_extra/

• Download the ADE-20K dataset (3.8 GB, 20.210 images) ==> images are created in ADE20K_2016_07_26/

Put all the datasets in the same directory. As illustration, your root training directory should be organised as follows:

training_datasets/
    ├── original_images/
    ├── CityScape/
    ├── CityScape_extra/
    └── ADE20K_2016_07_26/

2. Save the synthetic image pairs and flows to disk
   During training, from this set of original images, the pairs of synthetic images could be created on the fly at each epoch. However, this dataset generation takes time and since no augmentation is applied at each epoch, one can also create the dataset in advance and save it to disk. During training, the image pairs composing the training datasets are then just loaded from the disk before passing through the network, which is a lot faster. To generate the training dataset and save it to disk:

python assets/save_training_dataset_to_disk.py --image_data_path /directory/to/original/training_datasets/ 
--csv_path assets/homo_aff_tps_train_DPED_CityScape_ADE.csv --save_dir /path/to/save_dir --plot True

It will create the images pairs and corresponding flow fields in save_dir/images and save_dir/flow respectively.

3. Add the paths in admin/local.py as 'training_cad_520' and 'validation_cad_520'

This is useful for adding moving objects. Download the images along with annotations from here. The root folder should be organized as follows. The add the paths in admin/local.py as 'coco'.

coco_root
    └── annotations
        └── instances_train2014.json
    └──images
        └── train2014

We use the reconstructions provided in the D2-Net repo. You can download the undistorted reconstructions and aggregated scene information folder directly here - Google Drive.

File structure should be the following:

MegaDepth
├── Undistorted_Sfm
└── scene_info

Them add the paths in admin/local.py as 'megadepth_training'.

Training scripts

The framework currently contains the training code for the following matching networks. The setting files can be used train the networks, or to know the exact training details.

This is adapted from our recent work Refign, with code. It allows training a probabilistic correspondence network unsupervised (mix of PDCNet and WarpC). It uses a simplifided version of PDCNet, without GOCor and which is predicting a uni-modal Gaussian probability distribution at every pixel instead of a Mixture of Laplacians.

• UAWarpC.train_UAWarpC_PDCNet_stage1: The default settings used for first stage network training without visibility mask. We train on real image pairs of the MegaDepth dataset.

• UAWarpC.train_UAWarpC_PDCNet_stage2: We further finetune the network trained with stage1, by including our visibility mask.

• PWarpC.train_weakly_supervised_PWarpC_SFNet_pfpascal: The default settings used to train the weakly-supervised PWarpC-SF-Net on PF-Pascal.

• PWarpC.train_weakly_supervised_PWarpC_SFNet_spair_from_pfpascal: The default settings used to train the weakly-supervised PWarpC-SF-Net on SPair. More precisely, the network is first trained on PF-Pascal (above) and further finetuned on SPair-71K.

• PWarpC.train_strongly_supervised_PWarpC_SFNet_pfpascal: The default settings used to train the strongly-supervised PWarpC-SF-Net on PF-Pascal.

• PWarpC.train_strongly_supervised_PWarpC_SFNet_spair_from_pfpascal: The default settings used to train the strongly-supervised PWarpC-SF-Net on Spair-71K.

• The rest to come

• WarpC.train_WarpC_GLUNet_stage1: The default settings used for first stage network training without visibility mask. We train on real image pairs of the MegaDepth dataset.

• WarpC.train_WarpC_GLUNet_stage2: We further finetune the network trained with stage1, by including our visibility mask. The network corresponds to our final WarpC-GLU-Net (see WarpC paper).

• WarpC.train_ft_WarpCSemanticGLUNet: The default settings used for training the final WarpC-SemanticGLU-Net (see WarpC paper). We finetune the original SemanticGLUNet (trained on the static/CAD synthetic data) on PF-Pascal using Warp Consistency.

• PDCNet.train_PDCNet_plus_stage1: The default settings used for first stage network training with fixed backbone weights. We train first on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add MULTIPLE independently moving objects and perturbations. We also train by applying our object reprojection mask.

• PDCNet.train_PDCNet_plus_stage2: The default settings used for training the final PDC-Net+ model (see PDC-Net+ paper). This setting fine-tunes all layers in the model trained using PDCNet_stage1 (including the feature backbone). As training dataset, we use a combination of the same dataset than in stage 1 as well as image pairs from the MegaDepth dataset and their sparse ground-truth correspondence data. We also apply the reprojection mask.

• PDCNet.train_PDCNet_stage1: The default settings used for first stage network training with fixed backbone weights. We initialize the backbone VGG-16 with pre-trained ImageNet weights. We train first on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add independently moving objects and perturbations.

• PDCNet.train_PDCNet_stage2: The default settings used for training the final PDC-Net model (see PDC-Net paper). This setting fine-tunes all layers in the model trained using PDCNet_stage1 (including the feature backbone). As training dataset, we use a combination of the same dataset than in stage 1 as well as image pairs from the MegaDepth dataset and their sparse ground-truth correspondence data.

• PDCNet.train_GLUNet_GOCor_star_stage1: Same settings than for PDCNet_stage1, with different model (non probabilistic baseline). The loss is changed accordingly to the L1 loss instead of the negative log likelihood loss.

• PDCNet.train_GLUNet_GOCor_star_stage2: The default settings used for training the final GLU-Net-GOCor* (see PDCNet paper).

• GLUNet.train_GLUNet_with_synthetically_generated_data: This is a simple example of how to generate random transformations on the fly, and to apply them to original images, in order to create training image pairs and their corresponding ground-truth flow. Here, the random transformations are applied to MegaDepth images. On the created image pairs and ground-truth flows, we additionally add a randomly moving object.

• GLUNet.train_GLUNet_static: The default settings used training the final GLU-Net (of the paper GLU-Net).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), which is later (GOCor paper) referred to as 'static' dataset.

• GLUNet.train_GLUNet_dynamic: The default settings used training the final GLU-Net trained on the dynamic dataset (of the paper GOCor).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add one independently moving object. This dataset is referred to as 'dynamic' dataset in GOCor paper.

• GLUNet.train_GLUNet_GOCor_static: The default settings used training the final GLU-Net-GOCor (of the paper GOCor).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), which is later (GOCor paper) referred to as 'static' dataset.

• GLUNet.train_GLUNet_GOCor_dynamic: The default settings used training the final GLU-Net-GOCor trained on the dynamic dataset (of the paper GOCor).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add one independently moving object. This dataset is referred to as 'dynamic' dataset in GOCor paper.

Training your own networks

To train a custom network using the toolkit, the following components need to be specified in the train settings. For reference, see train_GLUNet_static.py.

• Datasets: The datasets to be used for training. A number of standard matching datasets are already available in the datasets module. The dataset class can be passed a processing function, which should perform the necessary processing of the data before batching it, e.g. data augmentations and conversion to tensors.
• Dataloader: Determines how to sample the batches. Can use specific samplers.
• Network: The network module to be trained.
• BatchPreprocessingModule: The pre-processing module that takes the batch and will transform it to the inputs required for training the network. Depends on the different networks and training strategies.
• Objective: The training objective.
• Actor: The trainer passes the training batch to the actor who is responsible for passing the data through the network correctly, and calculating the training loss. The batch preprocessing is also done within the actor class.
• Optimizer: Optimizer to be used, e.g. Adam.
• Scheduler: Scheduler to be used.
• Trainer: The main class which runs the epochs and saves checkpoints.

6. Acknowledgement <a name="acknowledgement"></a>

We borrow code from public projects, such as pytracking, GLU-Net, DGC-Net, PWC-Net, NC-Net, Flow-Net-Pytorch, RAFT, CATs...

7. ChangeLog <a name="changelog"></a>

• 06/21: Added evaluation code
• 07/21: Added training code and more options for evaluation
• 08/21: Fixed memory leak in mixture dataset + added other sampling for megadepth dataset
• 10/21: Added pre-trained models of WarpC
• 12/21: Added training code for WarpC and PDC-Net+, + randomly generated data + Caltech evaluation, + pre-trained models of PDC-Net+ + demo on notebook
• 02/22: Small modifications
• 03/22: Major refactoring, added video demos, code for PWarpC, default initialization of deconv to bilinear weights.